Failed to allocate NNN bytes in slab allocator for memtx_tuple
186 views
Skip to first unread message
Ruslan Aromatov
Mar 18, 2020, 9:06:14 AM3/18/20
to tarantool-ru
Добрый день!
Использую тарантул на бою как сервис кэша сессий.
Записей в таблице около 300000, rps до 400, в пиках 600.
Версия 2.3.1-2-g92750c828
Настройки
log level 5
set 'memtx_memory' configuration option to 1048576000
set 'listen' configuration option to "3301"
set 'force_recovery' configuration option to true
set 'memtx_max_tuple_size' configuration option to 5242880
set 'checkpoint_count' configuration option to 3
Использую java-connctor для работы. С версии 2.3.1 начал замечать в логах такие записи (до этого кажется версия 2.1.3 была):
Failed to allocate 202251 bytes in slab allocator for memtx_tuple
До этого специально устанавливал опцию memtx_max_tuple_size в 5242880, после чего проверял строкой в 10 мегабайт, на что получил корректную ошибку
main/159/main memtx_engine.c:1136 E> ER_MEMTX_MAX_TUPLE_SIZE: Failed to allocate 10000085 bytes for tuple: tuple is too large. Check memtx_max_tuple_size' configuration option.
Проверка на строке в 4Мб отработала правильно, все данные записались.
Тем не менее, сейчас примерно 4-5 раз в час получаю такие исключения при попытке записи гораздо менее объемных данных. При этом исключение вижу в java-коннекторе, а в самом логе тарантула это никак не отражено.
В этой таблице примерно 300000 записей и раз в минуту в отдельном файбере вызывается метод очистки просроченных записей.Добавление новых записей происходит около 200 раз в секунду.Этот же самый код работал и на предыдущих версиях тарантула, ошибок замечено не было.
В чем может быть проблема?
Ruslan Aromatov
Mar 18, 2020, 9:22:04 AM3/18/20
to tarantool-ru
В дополнение.
Сейчас порылся в логах и обнаружил, что эти ошибки появились только после того, как я установил опцию memtx_max_tuple_size при старте тарантула.
А установил я ее потому, что до этого использовалось какое-то значение по умолчанию, а потом в логах я встретил текст
TarantoolException: Failed to allocate 332977 bytes in slab allocator for memtx_tuple
После этого я установил значение memtx_max_tuple_size в 5242880 и перезапустил сервис.
Alexander Turenko
Mar 18, 2020, 9:39:51 AM3/18/20
to tarantool-ru
Добрый день!
Советую помониторить значения из box.info.memory() и box.slab.info(). Возможно,
нужно просто добавить memtx_memory.
WBR, Alexander Turenko.
Konstantin Osipov
Mar 18, 2020, 9:57:00 AM3/18/20
to tarant...@googlegroups.com
* Ruslan Aromatov <r.v.ar...@gmail.com> [20/03/18 16:07]:
1) Память действительно кончилась
2) Память фрагментирована.
Пришлите пожалуйста box.slab.info() и box.slab.stats().
Это, скорее всего, фрагментация. попробуйте покрутить
slab_alloc_factor (поставить 2).
Konstantin Osipov, Moscow, Russia
1) Память действительно кончилась
2) Память фрагментирована.
Пришлите пожалуйста box.slab.info() и box.slab.stats().
slab_alloc_factor (поставить 2).
>
>
> В чем может быть проблема?
--
>
> В чем может быть проблема?
Konstantin Osipov, Moscow, Russia
Ruslan Aromatov
Mar 19, 2020, 4:45:37 AM3/19/20
to tarantool-ru
Спасибо за ответ.
Крутить параметры в боевом режиме опасаюсь. На своей машине организовал нагрузочное тестирование, но никаких ошибок не удалось повторить.
Разве что после вашего анализа информации ниже.
localhost:3301> box.slab.info()
---
- items_size: 647289224
items_used_ratio: 41.43%
quota_size: 1048576000
quota_used_ratio: 99.20%
arena_used_ratio: 32.2%
items_used: 268151048
quota_used: 1040187392
arena_size: 1040187392
arena_used: 335292680
...
localhost:3301>box.slab.stats()
- - mem_free: 16256
mem_used: 24
item_count: 1
item_size: 24
slab_count: 1
slab_size: 16384
- mem_free: 16184
mem_used: 96
item_count: 3
item_size: 32
slab_count: 1
slab_size: 16384
- mem_free: 12560
mem_used: 3720
item_count: 93
item_size: 40
slab_count: 1
slab_size: 16384
- mem_free: 15560
mem_used: 720
item_count: 15
item_size: 48
slab_count: 1
slab_size: 16384
- mem_free: 12864
mem_used: 3416
item_count: 61
item_size: 56
slab_count: 1
slab_size: 16384
- mem_free: 38496
mem_used: 26624
item_count: 416
item_size: 64
slab_count: 4
slab_size: 16384
- mem_free: 15128
mem_used: 1152
item_count: 16
item_size: 72
slab_count: 1
slab_size: 16384
- mem_free: 330200
mem_used: 2095520
item_count: 26194
item_size: 80
slab_count: 149
slab_size: 16384
- mem_free: 3589872
mem_used: 3443088
item_count: 39126
item_size: 88
slab_count: 432
slab_size: 16384
- mem_free: 1387456
mem_used: 7729344
item_count: 80514
item_size: 96
slab_count: 560
slab_size: 16384
- mem_free: 2192344
mem_used: 13355056
item_count: 128414
item_size: 104
slab_count: 955
slab_size: 16384
- mem_free: 16168
mem_used: 112
item_count: 1
item_size: 112
slab_count: 1
slab_size: 16384
- mem_free: 10160
mem_used: 6120
item_count: 51
item_size: 120
slab_count: 1
slab_size: 16384
- mem_free: 14360
mem_used: 1920
item_count: 15
item_size: 128
slab_count: 1
slab_size: 16384
- mem_free: 16008
mem_used: 272
item_count: 2
item_size: 136
slab_count: 1
slab_size: 16384
- mem_free: 20752
mem_used: 11808
item_count: 82
item_size: 144
slab_count: 2
slab_size: 16384
- mem_free: 15976
mem_used: 304
item_count: 2
item_size: 152
slab_count: 1
slab_size: 16384
- mem_free: 16120
mem_used: 160
item_count: 1
item_size: 160
slab_count: 1
slab_size: 16384
- mem_free: 32664
mem_used: 0
item_count: 0
item_size: 168
slab_count: 1
slab_size: 32768
- mem_free: 32136
mem_used: 528
item_count: 3
item_size: 176
slab_count: 1
slab_size: 32768
- mem_free: 29720
mem_used: 2944
item_count: 16
item_size: 184
slab_count: 1
slab_size: 32768
- mem_free: 29592
mem_used: 3072
item_count: 16
item_size: 192
slab_count: 1
slab_size: 32768
- mem_free: 32464
mem_used: 200
item_count: 1
item_size: 200
slab_count: 1
slab_size: 32768
- mem_free: 32664
mem_used: 0
item_count: 0
item_size: 208
slab_count: 1
slab_size: 32768
- mem_free: 32232
mem_used: 432
item_count: 2
item_size: 216
slab_count: 1
slab_size: 32768
- mem_free: 32440
mem_used: 224
item_count: 1
item_size: 224
slab_count: 1
slab_size: 32768
- mem_free: 32664
mem_used: 0
item_count: 0
item_size: 232
slab_count: 1
slab_size: 32768
- mem_free: 67896
mem_used: 226080
item_count: 942
item_size: 240
slab_count: 9
slab_size: 32768
- mem_free: 1162488
mem_used: 7787448
item_count: 31401
item_size: 248
slab_count: 274
slab_size: 32768
- mem_free: 32152
mem_used: 512
item_count: 2
item_size: 256
slab_count: 1
slab_size: 32768
- mem_free: 74616
mem_used: 88704
item_count: 336
item_size: 264
slab_count: 5
slab_size: 32768
- mem_free: 132832
mem_used: 781760
item_count: 2792
item_size: 280
slab_count: 28
slab_size: 32768
- mem_free: 72096
mem_used: 352536
item_count: 1191
item_size: 296
slab_count: 13
slab_size: 32768
- mem_free: 43920
mem_used: 86736
item_count: 278
item_size: 312
slab_count: 4
slab_size: 32768
- mem_free: 48376
mem_used: 17056
item_count: 52
item_size: 328
slab_count: 1
slab_size: 65536
- mem_free: 53736
mem_used: 11696
item_count: 34
item_size: 344
slab_count: 1
slab_size: 65536
- mem_free: 47768
mem_used: 17664
item_count: 48
item_size: 368
slab_count: 1
slab_size: 65536
- mem_free: 124984
mem_used: 5880
item_count: 15
item_size: 392
slab_count: 2
slab_size: 65536
- mem_free: 125872
mem_used: 4992
item_count: 12
item_size: 416
slab_count: 2
slab_size: 65536
- mem_free: 61032
mem_used: 4400
item_count: 10
item_size: 440
slab_count: 1
slab_size: 65536
- mem_free: 108160
mem_used: 284432
item_count: 613
item_size: 464
slab_count: 6
slab_size: 65536
- mem_free: 158104
mem_used: 627080
item_count: 1285
item_size: 488
slab_count: 12
slab_size: 65536
- mem_free: 102392
mem_used: 224768
item_count: 439
item_size: 512
slab_count: 5
slab_size: 65536
- mem_free: 64496
mem_used: 66368
item_count: 122
item_size: 544
slab_count: 2
slab_size: 65536
- mem_free: 99184
mem_used: 31680
item_count: 55
item_size: 576
slab_count: 2
slab_size: 65536
- mem_free: 113840
mem_used: 17024
item_count: 28
item_size: 608
slab_count: 2
slab_size: 65536
- mem_free: 135496
mem_used: 60800
item_count: 95
item_size: 640
slab_count: 3
slab_size: 65536
- mem_free: 275760
mem_used: 510048
item_count: 759
item_size: 672
slab_count: 6
slab_size: 131072
- mem_free: 229776
mem_used: 425064
item_count: 597
item_size: 712
slab_count: 5
slab_size: 131072
- mem_free: 155152
mem_used: 106784
item_count: 142
item_size: 752
slab_count: 2
slab_size: 131072
- mem_free: 472464
mem_used: 1884960
item_count: 2380
item_size: 792
slab_count: 18
slab_size: 131072
- mem_free: 899896
mem_used: 4993664
item_count: 6002
item_size: 832
slab_count: 45
slab_size: 131072
- mem_free: 255896
mem_used: 660880
item_count: 751
item_size: 880
slab_count: 7
slab_size: 131072
- mem_free: 200120
mem_used: 454720
item_count: 490
item_size: 928
slab_count: 5
slab_size: 131072
- mem_free: 101872
mem_used: 160064
item_count: 164
item_size: 976
slab_count: 2
slab_size: 131072
- mem_free: 232032
mem_used: 291840
item_count: 285
item_size: 1024
slab_count: 4
slab_size: 131072
- mem_free: 747000
mem_used: 2527200
item_count: 2340
item_size: 1080
slab_count: 25
slab_size: 131072
- mem_free: 505896
mem_used: 1458624
item_count: 1284
item_size: 1136
slab_count: 15
slab_size: 131072
- mem_free: 1504672
mem_used: 8448896
item_count: 7088
item_size: 1192
slab_count: 76
slab_size: 131072
- mem_free: 1871376
mem_used: 12928008
item_count: 10293
item_size: 1256
slab_count: 113
slab_size: 131072
- mem_free: 750760
mem_used: 3703920
item_count: 2806
item_size: 1320
slab_count: 17
slab_size: 262144
- mem_free: 590520
mem_used: 1767840
item_count: 1270
item_size: 1392
slab_count: 9
slab_size: 262144
- mem_free: 515200
mem_used: 1581120
item_count: 1080
item_size: 1464
slab_count: 8
slab_size: 262144
- mem_free: 1716080
mem_used: 6145120
item_count: 3980
item_size: 1544
slab_count: 30
slab_size: 262144
- mem_free: 618480
mem_used: 1477840
item_count: 910
item_size: 1624
slab_count: 8
slab_size: 262144
- mem_free: 306760
mem_used: 479360
item_count: 280
item_size: 1712
slab_count: 3
slab_size: 262144
- mem_free: 321720
mem_used: 464400
item_count: 258
item_size: 1800
slab_count: 3
slab_size: 262144
- mem_free: 259032
mem_used: 527088
item_count: 278
item_size: 1896
slab_count: 3
slab_size: 262144
- mem_free: 709168
mem_used: 2173272
item_count: 1091
item_size: 1992
slab_count: 11
slab_size: 262144
- mem_free: 1290160
mem_used: 6046960
item_count: 2885
item_size: 2096
slab_count: 28
slab_size: 262144
- mem_free: 929040
mem_used: 4573800
item_count: 2079
item_size: 2200
slab_count: 21
slab_size: 262144
- mem_free: 1496424
mem_used: 9509256
item_count: 4113
item_size: 2312
slab_count: 42
slab_size: 262144
- mem_free: 933728
mem_used: 4307072
item_count: 1771
item_size: 2432
slab_count: 20
slab_size: 262144
- mem_free: 879480
mem_used: 4623360
item_count: 1806
item_size: 2560
slab_count: 21
slab_size: 262144
- mem_free: 1309224
mem_used: 2360064
item_count: 878
item_size: 2688
slab_count: 7
slab_size: 524288
- mem_free: 4883264
mem_used: 24995224
item_count: 8851
item_size: 2824
slab_count: 57
slab_size: 524288
- mem_free: 1391192
mem_used: 4374832
item_count: 1474
item_size: 2968
slab_count: 11
slab_size: 524288
- mem_free: 1447944
mem_used: 4318080
item_count: 1384
item_size: 3120
slab_count: 11
slab_size: 524288
- mem_free: 1623584
mem_used: 6763360
item_count: 2062
item_size: 3280
slab_count: 16
slab_size: 524288
- mem_free: 1608440
mem_used: 8351056
item_count: 2422
item_size: 3448
slab_count: 19
slab_size: 524288
- mem_free: 2015688
mem_used: 7943808
item_count: 2192
item_size: 3624
slab_count: 19
slab_size: 524288
- mem_free: 1546760
mem_used: 4219264
item_count: 1108
item_size: 3808
slab_count: 11
slab_size: 524288
- mem_free: 1538024
mem_used: 4228000
item_count: 1057
item_size: 4000
slab_count: 11
slab_size: 524288
- mem_free: 1970280
mem_used: 8513400
item_count: 2027
item_size: 4200
slab_count: 20
slab_size: 524288
- mem_free: 1762944
mem_used: 6624000
item_count: 1500
item_size: 4416
slab_count: 16
slab_size: 524288
- mem_free: 1566552
mem_used: 5247840
item_count: 1131
item_size: 4640
slab_count: 13
slab_size: 524288
- mem_free: 3034392
mem_used: 7449288
item_count: 1529
item_size: 4872
slab_count: 20
slab_size: 524288
- mem_free: 1490824
mem_used: 4275200
item_count: 835
item_size: 5120
slab_count: 11
slab_size: 524288
- mem_free: 1952400
mem_used: 4338432
item_count: 807
item_size: 5376
slab_count: 6
slab_size: 1048576
- mem_free: 2688944
mem_used: 5698832
item_count: 1009
item_size: 5648
slab_count: 8
slab_size: 1048576
- mem_free: 2022848
mem_used: 4267984
item_count: 719
item_size: 5936
slab_count: 6
slab_size: 1048576
- mem_free: 1665192
mem_used: 3577168
item_count: 574
item_size: 6232
slab_count: 5
slab_size: 1048576
- mem_free: 2639280
mem_used: 3651552
item_count: 558
item_size: 6544
slab_count: 6
slab_size: 1048576
- mem_free: 2741936
mem_used: 4597368
item_count: 669
item_size: 6872
slab_count: 7
slab_size: 1048576
- mem_free: 2687896
mem_used: 2554464
item_count: 354
item_size: 7216
slab_count: 5
slab_size: 1048576
- mem_free: 2643792
mem_used: 2598568
item_count: 343
item_size: 7576
slab_count: 5
slab_size: 1048576
- mem_free: 1909368
mem_used: 2284520
item_count: 287
item_size: 7960
slab_count: 4
slab_size: 1048576
- mem_free: 1431616
mem_used: 1713800
item_count: 205
item_size: 8360
slab_count: 3
slab_size: 1048576
- mem_free: 2085728
mem_used: 2108160
item_count: 240
item_size: 8784
slab_count: 4
slab_size: 1048576
- mem_free: 1826384
mem_used: 1319032
item_count: 143
item_size: 9224
slab_count: 3
slab_size: 1048576
- mem_free: 1527520
mem_used: 1617896
item_count: 167
item_size: 9688
slab_count: 3
slab_size: 1048576
- mem_free: 2117640
mem_used: 1027776
item_count: 101
item_size: 10176
slab_count: 3
slab_size: 1048576
- mem_free: 3168048
mem_used: 1026048
item_count: 96
item_size: 10688
slab_count: 2
slab_size: 2097152
- mem_free: 1143008
mem_used: 954040
item_count: 85
item_size: 11224
slab_count: 1
slab_size: 2097152
- mem_free: 1200856
mem_used: 896192
item_count: 76
item_size: 11792
slab_count: 1
slab_size: 2097152
- mem_free: 1354008
mem_used: 743040
item_count: 60
item_size: 12384
slab_count: 1
slab_size: 2097152
- mem_free: 1537704
mem_used: 559344
item_count: 43
item_size: 13008
slab_count: 1
slab_size: 2097152
- mem_free: 1605144
mem_used: 491904
item_count: 36
item_size: 13664
slab_count: 1
slab_size: 2097152
- mem_free: 1666488
mem_used: 430560
item_count: 30
item_size: 14352
slab_count: 1
slab_size: 2097152
- mem_free: 1675032
mem_used: 422016
item_count: 28
item_size: 15072
slab_count: 1
slab_size: 2097152
- mem_free: 1717080
mem_used: 379968
item_count: 24
item_size: 15832
slab_count: 1
slab_size: 2097152
- mem_free: 1681448
mem_used: 415600
item_count: 25
item_size: 16624
slab_count: 1
slab_size: 2097152
- mem_free: 1800296
mem_used: 296752
item_count: 17
item_size: 17456
slab_count: 1
slab_size: 2097152
- mem_free: 1895440
mem_used: 201608
item_count: 11
item_size: 18328
slab_count: 1
slab_size: 2097152
- mem_free: 1981560
mem_used: 115488
item_count: 6
item_size: 19248
slab_count: 1
slab_size: 2097152
- mem_free: 1834240
mem_used: 262808
item_count: 13
item_size: 20216
slab_count: 1
slab_size: 2097152
- mem_free: 3981880
mem_used: 212320
item_count: 10
item_size: 21232
slab_count: 1
slab_size: 4194304
- mem_free: 3993536
mem_used: 200664
item_count: 9
item_size: 22296
slab_count: 1
slab_size: 4194304
- mem_free: 3936624
mem_used: 257576
item_count: 11
item_size: 23416
slab_count: 1
slab_size: 4194304
- mem_free: 4071240
mem_used: 122960
item_count: 5
item_size: 24592
slab_count: 1
slab_size: 4194304
- mem_free: 4039256
mem_used: 154944
item_count: 6
item_size: 25824
slab_count: 1
slab_size: 4194304
- mem_free: 4139960
mem_used: 54240
item_count: 2
item_size: 27120
slab_count: 1
slab_size: 4194304
- mem_free: 4051800
mem_used: 142400
item_count: 5
item_size: 28480
slab_count: 1
slab_size: 4194304
- mem_free: 4074584
mem_used: 119616
item_count: 4
item_size: 29904
slab_count: 1
slab_size: 4194304
- mem_free: 4131400
mem_used: 62800
item_count: 2
item_size: 31400
slab_count: 1
slab_size: 4194304
- mem_free: 4161224
mem_used: 32976
item_count: 1
item_size: 32976
slab_count: 1
slab_size: 4194304
- mem_free: 4124952
mem_used: 69248
item_count: 2
item_size: 34624
slab_count: 1
slab_size: 4194304
- mem_free: 3976040
mem_used: 218160
item_count: 6
item_size: 36360
slab_count: 1
slab_size: 4194304
- mem_free: 4117832
mem_used: 76368
item_count: 2
item_size: 38184
slab_count: 1
slab_size: 4194304
- mem_free: 4194200
mem_used: 0
item_count: 0
item_size: 40096
slab_count: 1
slab_size: 4194304
- mem_free: 8220088
mem_used: 168416
item_count: 4
item_size: 42104
slab_count: 1
slab_size: 8388608
- mem_free: 8300072
mem_used: 88432
item_count: 2
item_size: 44216
slab_count: 1
slab_size: 8388608
- mem_free: 8342072
mem_used: 46432
item_count: 1
item_size: 46432
slab_count: 1
slab_size: 8388608
- mem_free: 8339744
mem_used: 48760
item_count: 1
item_size: 48760
slab_count: 1
slab_size: 8388608
- mem_free: 8337304
mem_used: 51200
item_count: 1
item_size: 51200
slab_count: 1
slab_size: 8388608
- mem_free: 8388504
mem_used: 0
item_count: 0
item_size: 53760
slab_count: 1
slab_size: 8388608
- mem_free: 8388504
mem_used: 0
item_count: 0
item_size: 56448
slab_count: 1
slab_size: 8388608
- mem_free: 8388504
mem_used: 0
item_count: 0
item_size: 59272
slab_count: 1
slab_size: 8388608
- mem_free: 8326264
mem_used: 62240
item_count: 1
item_size: 62240
slab_count: 1
slab_size: 8388608
- mem_free: 8388504
mem_used: 0
item_count: 0
item_size: 65352
slab_count: 1
slab_size: 8388608
- mem_free: 8388504
mem_used: 0
item_count: 0
item_size: 68624
slab_count: 1
slab_size: 8388608
- mem_free: 8316448
mem_used: 72056
item_count: 1
item_size: 72056
slab_count: 1
slab_size: 8388608
- mem_free: 8388504
mem_used: 0
item_count: 0
item_size: 75664
slab_count: 1
slab_size: 8388608
- mem_free: 16675696
mem_used: 101416
item_count: 1
item_size: 101416
slab_count: 1
slab_size: 16777216
- mem_free: 16641192
mem_used: 135920
item_count: 1
item_size: 135920
slab_count: 1
slab_size: 16777216
- mem_free: 16777112
mem_used: 0
item_count: 0
item_size: 343560
slab_count: 1
slab_size: 16777216
- mem_free: 16338616
mem_used: 438496
item_count: 1
item_size: 438496
slab_count: 1
slab_size: 16777216
- mem_free: 16777112
mem_used: 0
item_count: 0
item_size: 507624
slab_count: 1
slab_size: 16777216
- mem_free: 16777112
mem_used: 0
item_count: 0
item_size: 1005120
slab_count: 1
slab_size: 16777216
- mem_free: 8352096
mem_used: 67141632
item_count: 4098
item_size: 16384
slab_count: 36
slab_size: 2097152
среда, 18 марта 2020 г., 16:57:00 UTC+3 пользователь Konstantin Osipov написал:
* Ruslan Aromatov <r.v.a...@gmail.com> [20/03/18 16:07]:
Reply all
Reply to author
Forward
0 new messages