slave lag after enabling tokudb on large table

Vinay Malkani

unread,

Oct 30, 2015, 1:11:58 PM10/30/15

to tokudb-user

I just changed one of my slaves to tokudb from innodb and started getting slave replication lag (that is ever growing) after changing our largest table to tokudb. In Innodb this table is 14TB and in TokuDB the table is approx 2.73TB.

I've tried enabling RBR and parallel slave workers and neither of these have worked. The slave server hardware is significantly better than the master and when the table was innodb there was 0 slave lag. We have another slave that show 0 slave lag and uses innodb from the same master.

binlog_format is set to ROW

table_open_cache = 2000

562 tables total in the system.

We are using Percona 5.6.26-74.0.

I enabled log_slow_slave_statements and there have not been any statements in the log.

long_query_time is currently set to 0 (usually we set this to 2);

Does anyone have any ideas on how to approach this issue and hopefully how to fix?

show mater status\G

mysql> show master status \G

*************************** 1. row ***************************

File: mysql-bin.427957

Position: 49085263

Binlog_Do_DB:

Binlog_Ignore_DB:

Executed_Gtid_Set:

1 row in set (0.00 sec)

slave status \G

*************************** 1. row ***************************

Slave_IO_State: Waiting for master to send event

Master_Host: prod-db-w2-xx

Master_User: repl_client

Master_Port: 3306

Connect_Retry: 60

Master_Log_File: mysql-bin.427955

Read_Master_Log_Pos: 85412709

Relay_Log_File: mysqld-relay-bin.128865

Relay_Log_Pos: 91674637

Relay_Master_Log_File: mysql-bin.427499

Slave_IO_Running: Yes

Slave_SQL_Running: Yes

Replicate_Do_DB:

Replicate_Ignore_DB:

Replicate_Do_Table:

Replicate_Ignore_Table:

Replicate_Wild_Do_Table:

Replicate_Wild_Ignore_Table:

Last_Errno: 0

Last_Error:

Skip_Counter: 0

Exec_Master_Log_Pos: 91674474

Relay_Log_Space: 47902908239

Until_Condition: None

Until_Log_File:

Until_Log_Pos: 0

Master_SSL_Allowed: No

Master_SSL_CA_File:

Master_SSL_CA_Path:

Master_SSL_Cert:

Master_SSL_Cipher:

Master_SSL_Key:

Seconds_Behind_Master: 22435

Master_SSL_Verify_Server_Cert: No

Last_IO_Errno: 0

Last_IO_Error:

Last_SQL_Errno: 0

Last_SQL_Error:

Replicate_Ignore_Server_Ids:

Master_Server_Id: xx

Master_UUID: a5a8cd81-236a-11e5-b477-22000a230ad2

Master_Info_File: /data/mysql/master.info

SQL_Delay: 0

SQL_Remaining_Delay: NULL

Slave_SQL_Running_State: Reading event from the relay log

Master_Retry_Count: 86400

Master_Bind:

Last_IO_Error_Timestamp:

Last_SQL_Error_Timestamp:

Master_SSL_Crl:

Master_SSL_Crlpath:

Retrieved_Gtid_Set:

Executed_Gtid_Set:

Auto_Position: 0

Best,

Vinay

jon....@percona.com

unread,

Oct 30, 2015, 4:00:50 PM10/30/15

to tokudb-user

Vinay,

Can you post a 'show create table'

Vinay Malkani

unread,

Oct 30, 2015, 11:27:25 PM10/30/15

to tokudb-user

Jon,

The output of

show create table healthservice_log \G

*************************** 1. row ***************************

Table: healthservice_log

Create Table: CREATE TABLE `healthservice_log` (

`id` bigint(20) NOT NULL AUTO_INCREMENT,

`hostname` varchar(50) COLLATE utf8_unicode_ci NOT NULL,

`msg_id` varchar(250) COLLATE utf8_unicode_ci DEFAULT NULL,

`msg_timestamp` datetime NOT NULL,

`create_timestamp` timestamp NULL DEFAULT CURRENT_TIMESTAMP,

`msg` text CHARACTER SET latin1,

PRIMARY KEY (`id`),

KEY `healthservice_log__h_ct__idx` (`hostname`,`create_timestamp`),

KEY `healthservice_log__h_mt__idx` (`hostname`,`msg_timestamp`)

) ENGINE=TokuDB AUTO_INCREMENT=4407595275 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci

Jon Tobin

unread,

Oct 31, 2015, 4:18:01 PM10/31/15

to Vinay Malkani, tokudb-user

Is this a "time series" use case? What I mean is, are you deleting data out of the table as it ages?

--
You received this message because you are subscribed to the Google Groups "tokudb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tokudb-user...@googlegroups.com.
To post to this group, send email to tokud...@googlegroups.com.
Visit this group at http://groups.google.com/group/tokudb-user.
For more options, visit https://groups.google.com/d/optout.

Vinay Malkani

unread,

Oct 31, 2015, 4:26:18 PM10/31/15

to Jon Tobin, tokudb-user

It is similar to time-series; however we do not delete data at all.

Thanks,

Vinay

ᐧ

Vinay Malkani

unread,

Oct 31, 2015, 5:22:39 PM10/31/15

to tokudb-user, j...@tokutek.com

Hi Jon,

We don't delete data from this table.

Is the following information useful:

I tried RFR and use ROW based replication. However RFR did not help.

the master is a 16GB ram, raid10 , xfs, , the slave is a 60GB ram, raid 10 ssd, xfs

I've attached 100 runs of pt-pmp as well.

+--------+---------------------------------------------------------------------------------+---------------------------------------------------+

| Type | Name | Status |

+--------+---------------------------------------------------------------------------------+---------------------------------------------------+

| TokuDB | disk free space | more than 10 percent of total file system space |

| TokuDB | time of environment creation | Mon Oct 12 02:46:35 2015 |

| TokuDB | time of engine startup | Sat Oct 31 05:39:51 2015 |

| TokuDB | time now | Sat Oct 31 20:54:47 2015 |

| TokuDB | db opens | 160 |

| TokuDB | db closes | 0 |

| TokuDB | num open dbs now | 160 |

| TokuDB | max open dbs | 160 |

| TokuDB | period, in ms, that recovery log is automatically fsynced | 0 |

| TokuDB | dictionary inserts | 22356478 |

| TokuDB | dictionary inserts fail | 0 |

| TokuDB | dictionary deletes | 0 |

| TokuDB | dictionary deletes fail | 0 |

| TokuDB | dictionary updates | 0 |

| TokuDB | dictionary updates fail | 0 |

| TokuDB | dictionary broadcast updates | 0 |

| TokuDB | dictionary broadcast updates fail | 0 |

| TokuDB | dictionary multi inserts | 80788152 |

| TokuDB | dictionary multi inserts fail | 0 |

| TokuDB | dictionary multi deletes | 6 |

| TokuDB | dictionary multi deletes fail | 0 |

| TokuDB | dictionary updates multi | 6358270 |

| TokuDB | dictionary updates multi fail | 0 |

| TokuDB | le: max committed xr | 4 |

| TokuDB | le: max provisional xr | 2 |

| TokuDB | le: expanded | 0 |

| TokuDB | le: max memsize | 2900114 |

| TokuDB | le: size of leafentries before garbage collection (during message application) | 114222889234 |

| TokuDB | le: size of leafentries after garbage collection (during message application) | 102688293814 |

| TokuDB | le: size of leafentries before garbage collection (outside message application) | 0 |

| TokuDB | le: size of leafentries after garbage collection (outside message application) | 0 |

| TokuDB | checkpoint: period | 60 |

| TokuDB | checkpoint: footprint | 0 |

| TokuDB | checkpoint: last checkpoint began | Sat Oct 31 20:53:52 2015 |

| TokuDB | checkpoint: last complete checkpoint began | Sat Oct 31 20:53:52 2015 |

| TokuDB | checkpoint: last complete checkpoint ended | Sat Oct 31 20:53:57 2015 |

| TokuDB | checkpoint: time spent during checkpoint (begin and end phases) | 6720 |

| TokuDB | checkpoint: time spent during last checkpoint (begin and end phases) | 5 |

| TokuDB | checkpoint: last complete checkpoint LSN | 34655095382 |

| TokuDB | checkpoint: checkpoints taken | 915 |

| TokuDB | checkpoint: checkpoints failed | 0 |

| TokuDB | checkpoint: waiters now | 0 |

| TokuDB | checkpoint: waiters max | 0 |

| TokuDB | checkpoint: non-checkpoint client wait on mo lock | 1090 |

| TokuDB | checkpoint: non-checkpoint client wait on cs lock | 0 |

| TokuDB | checkpoint: checkpoint begin time | 129033079 |

| TokuDB | checkpoint: long checkpoint begin time | 0 |

| TokuDB | checkpoint: long checkpoint begin count | 0 |

| TokuDB | checkpoint: checkpoint end time | 6900905448 |

| TokuDB | checkpoint: long checkpoint end time | 0 |

| TokuDB | checkpoint: long checkpoint end count | 0 |

| TokuDB | cachetable: miss | 9203 |

| TokuDB | cachetable: miss time | 8028461 |

| TokuDB | cachetable: prefetches | 0 |

| TokuDB | cachetable: size current | 31538770338 |

| TokuDB | cachetable: size limit | 32075487232 |

| TokuDB | cachetable: size writing | 0 |

| TokuDB | cachetable: size nonleaf | 1389909560 |

| TokuDB | cachetable: size leaf | 30148857042 |

| TokuDB | cachetable: size rollback | 3736 |

| TokuDB | cachetable: size cachepressure | 1379854067 |

| TokuDB | cachetable: size currently cloned data for checkpoint | 0 |

| TokuDB | cachetable: evictions | 1580 |

| TokuDB | cachetable: cleaner executions | 105196 |

| TokuDB | cachetable: cleaner period | 1 |

| TokuDB | cachetable: cleaner iterations | 5 |

| TokuDB | cachetable: number of waits on cache pressure | 0 |

| TokuDB | cachetable: time waiting on cache pressure | 0 |

| TokuDB | cachetable: number of long waits on cache pressure | 0 |

| TokuDB | cachetable: long time waiting on cache pressure | 0 |

| TokuDB | cachetable: number of threads in pool | 32 |

| TokuDB | cachetable: number of currently active threads in pool | 0 |

| TokuDB | cachetable: number of currently queued work items | 0 |

| TokuDB | cachetable: largest number of queued work items | 2 |

| TokuDB | cachetable: total number of work items processed | 923023 |

| TokuDB | cachetable: total execution time of processing work items | 223201 |

| TokuDB | cachetable: number of threads in pool | 64 |

| TokuDB | cachetable: number of currently active threads in pool | 0 |

| TokuDB | cachetable: number of currently queued work items | 0 |

| TokuDB | cachetable: largest number of queued work items | 51 |

| TokuDB | cachetable: total number of work items processed | 26491 |

| TokuDB | cachetable: total execution time of processing work items | 52095 |

| TokuDB | cachetable: number of threads in pool | 8 |

| TokuDB | cachetable: number of currently active threads in pool | 0 |

| TokuDB | cachetable: number of currently queued work items | 0 |

| TokuDB | cachetable: largest number of queued work items | 39 |

| TokuDB | cachetable: total number of work items processed | 92105 |

| TokuDB | cachetable: total execution time of processing work items | 2853180 |

| TokuDB | locktree: memory size | 184 |

| TokuDB | locktree: memory size limit | 3942327040 |

| TokuDB | locktree: number of times lock escalation ran | 0 |

| TokuDB | locktree: time spent running escalation (seconds) | 0.000000 |

| TokuDB | locktree: latest post-escalation memory size | 0 |

| TokuDB | locktree: number of locktrees open now | 162 |

| TokuDB | locktree: number of pending lock requests | 0 |

| TokuDB | locktree: number of locktrees eligible for the STO | 2 |

| TokuDB | locktree: number of times a locktree ended the STO early | 42 |

| TokuDB | locktree: time spent ending the STO early (seconds) | 0.959773 |

| TokuDB | locktree: number of wait locks | 0 |

| TokuDB | locktree: time waiting for locks | 0 |

| TokuDB | locktree: number of long wait locks | 0 |

| TokuDB | locktree: long time waiting for locks | 0 |

| TokuDB | locktree: number of lock timeouts | 0 |

| TokuDB | locktree: number of waits on lock escalation | 0 |

| TokuDB | locktree: time waiting on lock escalation | 0 |

| TokuDB | locktree: number of long waits on lock escalation | 0 |

| TokuDB | locktree: long time waiting on lock escalation | 0 |

| TokuDB | ft: dictionary updates | 0 |

| TokuDB | ft: dictionary broadcast updates | 0 |

| TokuDB | ft: descriptor set | 0 |

| TokuDB | ft: messages ignored by leaf due to msn | 760841 |

| TokuDB | ft: total search retries due to TRY_AGAIN | 4350 |

| TokuDB | ft: searches requiring more tries than the height of the tree | 0 |

| TokuDB | ft: searches requiring more tries than the height of the tree plus three | 0 |

| TokuDB | ft: leaf nodes flushed to disk (not for checkpoint) | 0 |

| TokuDB | ft: leaf nodes flushed to disk (not for checkpoint) (bytes) | 0 |

| TokuDB | ft: leaf nodes flushed to disk (not for checkpoint) (uncompressed bytes) | 0 |

| TokuDB | ft: leaf nodes flushed to disk (not for checkpoint) (seconds) | 0.000000 |

| TokuDB | ft: nonleaf nodes flushed to disk (not for checkpoint) | 0 |

| TokuDB | ft: nonleaf nodes flushed to disk (not for checkpoint) (bytes) | 0 |

| TokuDB | ft: nonleaf nodes flushed to disk (not for checkpoint) (uncompressed bytes) | 0 |

| TokuDB | ft: nonleaf nodes flushed to disk (not for checkpoint) (seconds) | 0.000000 |

| TokuDB | ft: leaf nodes flushed to disk (for checkpoint) | 87630 |

| TokuDB | ft: leaf nodes flushed to disk (for checkpoint) (bytes) | 28227416064 |

| TokuDB | ft: leaf nodes flushed to disk (for checkpoint) (uncompressed bytes) | 199616467535 |

| TokuDB | ft: leaf nodes flushed to disk (for checkpoint) (seconds) | 20.093304 |

| TokuDB | ft: nonleaf nodes flushed to disk (for checkpoint) | 145635 |

| TokuDB | ft: nonleaf nodes flushed to disk (for checkpoint) (bytes) | 25301552640 |

| TokuDB | ft: nonleaf nodes flushed to disk (for checkpoint) (uncompressed bytes) | 155809347253 |

| TokuDB | ft: nonleaf nodes flushed to disk (for checkpoint) (seconds) | 27.004770 |

| TokuDB | ft: uncompressed / compressed bytes written (leaf) | 7.071723 |

| TokuDB | ft: uncompressed / compressed bytes written (nonleaf) | 6.158094 |

| TokuDB | ft: uncompressed / compressed bytes written (overall) | 6.639878 |

| TokuDB | ft: nonleaf node partial evictions | 299733 |

| TokuDB | ft: nonleaf node partial evictions (bytes) | 1980404210 |

| TokuDB | ft: leaf node partial evictions | 1419903 |

| TokuDB | ft: leaf node partial evictions (bytes) | 115408705137 |

| TokuDB | ft: leaf node full evictions | 1577 |

| TokuDB | ft: leaf node full evictions (bytes) | 1307033300 |

| TokuDB | ft: nonleaf node full evictions | 1 |

| TokuDB | ft: nonleaf node full evictions (bytes) | 200472 |

| TokuDB | ft: leaf nodes created | 20659 |

| TokuDB | ft: nonleaf nodes created | 2539 |

| TokuDB | ft: leaf nodes destroyed | 0 |

| TokuDB | ft: nonleaf nodes destroyed | 0 |

| TokuDB | ft: bytes of messages injected at root (all trees) | 4879571203 |

| TokuDB | ft: bytes of messages flushed from h1 nodes to leaves | 6280974467 |

| TokuDB | ft: bytes of messages currently in trees (estimate) | 18446744072308148352 |

| TokuDB | ft: messages injected at root | 70069432 |

| TokuDB | ft: broadcast messages injected at root | 0 |

| TokuDB | ft: basements decompressed as a target of a query | 1 |

| TokuDB | ft: basements decompressed for prelocked range | 0 |

| TokuDB | ft: basements decompressed for prefetch | 0 |

| TokuDB | ft: basements decompressed for write | 66 |

| TokuDB | ft: buffers decompressed as a target of a query | 1124 |

| TokuDB | ft: buffers decompressed for prelocked range | 0 |

| TokuDB | ft: buffers decompressed for prefetch | 0 |

| TokuDB | ft: buffers decompressed for write | 285828 |

| TokuDB | ft: pivots fetched for query | 1721 |

| TokuDB | ft: pivots fetched for query (bytes) | 51726848 |

| TokuDB | ft: pivots fetched for query (seconds) | 0.645271 |

| TokuDB | ft: pivots fetched for prefetch | 0 |

| TokuDB | ft: pivots fetched for prefetch (bytes) | 0 |

| TokuDB | ft: pivots fetched for prefetch (seconds) | 0.000000 |

| TokuDB | ft: pivots fetched for write | 216 |

| TokuDB | ft: pivots fetched for write (bytes) | 4032000 |

| TokuDB | ft: pivots fetched for write (seconds) | 0.053538 |

| TokuDB | ft: basements fetched as a target of a query | 2565 |

| TokuDB | ft: basements fetched as a target of a query (bytes) | 17899008 |

| TokuDB | ft: basements fetched as a target of a query (seconds) | 0.744471 |

| TokuDB | ft: basements fetched for prelocked range | 0 |

| TokuDB | ft: basements fetched for prelocked range (bytes) | 0 |

| TokuDB | ft: basements fetched for prelocked range (seconds) | 0.000000 |

| TokuDB | ft: basements fetched for prefetch | 0 |

| TokuDB | ft: basements fetched for prefetch (bytes) | 0 |

| TokuDB | ft: basements fetched for prefetch (seconds) | 0.000000 |

| TokuDB | ft: basements fetched for write | 600368 |

| TokuDB | ft: basements fetched for write (bytes) | 5267368960 |

| TokuDB | ft: basements fetched for write (seconds) | 25.146308 |

| TokuDB | ft: buffers fetched as a target of a query | 0 |

| TokuDB | ft: buffers fetched as a target of a query (bytes) | 0 |

| TokuDB | ft: buffers fetched as a target of a query (seconds) | 0.000000 |

| TokuDB | ft: buffers fetched for prelocked range | 0 |

| TokuDB | ft: buffers fetched for prelocked range (bytes) | 0 |

| TokuDB | ft: buffers fetched for prelocked range (seconds) | 0.000000 |

| TokuDB | ft: buffers fetched for prefetch | 0 |

| TokuDB | ft: buffers fetched for prefetch (bytes) | 0 |

| TokuDB | ft: buffers fetched for prefetch (seconds) | 0.000000 |

| TokuDB | ft: buffers fetched for write | 33966 |

| TokuDB | ft: buffers fetched for write (bytes) | 377777152 |

| TokuDB | ft: buffers fetched for write (seconds) | 1.671078 |

| TokuDB | ft: leaf compression to memory (seconds) | 3108.895240 |

| TokuDB | ft: leaf serialization to memory (seconds) | 240.171434 |

| TokuDB | ft: leaf decompression to memory (seconds) | 61.992943 |

| TokuDB | ft: leaf deserialization to memory (seconds) | 44.029845 |

| TokuDB | ft: nonleaf compression to memory (seconds) | 2408.443803 |

| TokuDB | ft: nonleaf serialization to memory (seconds) | 316.572925 |

| TokuDB | ft: nonleaf decompression to memory (seconds) | 6.427383 |

| TokuDB | ft: nonleaf deserialization to memory (seconds) | 10.775047 |

| TokuDB | ft: promotion: roots split | 1 |

| TokuDB | ft: promotion: leaf roots injected into | 116468 |

| TokuDB | ft: promotion: h1 roots injected into | 14745516 |

| TokuDB | ft: promotion: injections at depth 0 | 9556009 |

| TokuDB | ft: promotion: injections at depth 1 | 47642274 |

| TokuDB | ft: promotion: injections at depth 2 | 5505345 |

| TokuDB | ft: promotion: injections at depth 3 | 909956 |

| TokuDB | ft: promotion: injections lower than depth 3 | 875691 |

| TokuDB | ft: promotion: stopped because of a nonempty buffer | 54796055 |

| TokuDB | ft: promotion: stopped at height 1 | 4474 |

| TokuDB | ft: promotion: stopped because the child was locked or not at all in memory | 1041 |

| TokuDB | ft: promotion: stopped because the child was not fully in memory | 18 |

| TokuDB | ft: promotion: stopped anyway, after locking the child | 5255 |

| TokuDB | ft: basement nodes deserialized with fixed-keysize | 566483 |

| TokuDB | ft: basement nodes deserialized with variable-keysize | 36530 |

| TokuDB | ft: promotion: succeeded in using the rightmost leaf shortcut | 27602406 |

| TokuDB | ft: promotion: tried the rightmost leaf shorcut but failed (out-of-bounds) | 3716 |

| TokuDB | ft: promotion: tried the rightmost leaf shorcut but failed (child reactive) | 260337 |

| TokuDB | ft: cursor skipped deleted leaf entries | 0 |

| TokuDB | ft flusher: total nodes potentially flushed by cleaner thread | 55937 |

| TokuDB | ft flusher: height-one nodes flushed by cleaner thread | 35839 |

| TokuDB | ft flusher: height-greater-than-one nodes flushed by cleaner thread | 20098 |

| TokuDB | ft flusher: nodes cleaned which had empty buffers | 17715 |

| TokuDB | ft flusher: nodes dirtied by cleaner thread | 70611 |

| TokuDB | ft flusher: max bytes in a buffer flushed by cleaner thread | 3986160 |

| TokuDB | ft flusher: min bytes in a buffer flushed by cleaner thread | 43 |

| TokuDB | ft flusher: total bytes in buffers flushed by cleaner thread | 14161483890 |

| TokuDB | ft flusher: max workdone in a buffer flushed by cleaner thread | 1639630 |

| TokuDB | ft flusher: min workdone in a buffer flushed by cleaner thread | 0 |

| TokuDB | ft flusher: total workdone in buffers flushed by cleaner thread | 240522708 |

| TokuDB | ft flusher: times cleaner thread tries to merge a leaf | 0 |

| TokuDB | ft flusher: cleaner thread leaf merges in progress | 0 |

| TokuDB | ft flusher: cleaner thread leaf merges successful | 0 |

| TokuDB | ft flusher: nodes dirtied by cleaner thread leaf merges | 0 |

| TokuDB | ft flusher: total number of flushes done by flusher threads or cleaner threads | 958879 |

| TokuDB | ft flusher: number of in memory flushes | 958879 |

| TokuDB | ft flusher: number of flushes that read something off disk | 0 |

| TokuDB | ft flusher: number of flushes that triggered another flush in child | 203 |

| TokuDB | ft flusher: number of flushes that triggered 1 cascading flush | 172 |

| TokuDB | ft flusher: number of flushes that triggered 2 cascading flushes | 26 |

| TokuDB | ft flusher: number of flushes that triggered 3 cascading flushes | 5 |

| TokuDB | ft flusher: number of flushes that triggered 4 cascading flushes | 0 |

| TokuDB | ft flusher: number of flushes that triggered 5 cascading flushes | 0 |

| TokuDB | ft flusher: number of flushes that triggered over 5 cascading flushes | 0 |

| TokuDB | ft flusher: leaf node splits | 20659 |

| TokuDB | ft flusher: nonleaf node splits | 2538 |

| TokuDB | ft flusher: leaf node merges | 0 |

| TokuDB | ft flusher: nonleaf node merges | 0 |

| TokuDB | ft flusher: leaf node balances | 0 |

| TokuDB | hot: operations ever started | 0 |

| TokuDB | hot: operations successfully completed | 0 |

| TokuDB | hot: operations aborted | 0 |

| TokuDB | hot: max number of flushes from root ever required to optimize a tree | 0 |

| TokuDB | txn: begin | 71616585 |

| TokuDB | txn: begin read only | 0 |

| TokuDB | txn: successful commits | 71616584 |

| TokuDB | txn: aborts | 0 |

| TokuDB | logger: next LSN | 34655327944 |

| TokuDB | logger: writes | 23096723 |

| TokuDB | logger: writes (bytes) | 78949443940 |

| TokuDB | logger: writes (uncompressed bytes) | 78949443940 |

| TokuDB | logger: writes (seconds) | 179.407285 |

| TokuDB | logger: number of long logger write operations | 2 |

| TokuDB | indexer: number of indexers successfully created | 0 |

| TokuDB | indexer: number of calls to toku_indexer_create_indexer() that failed | 0 |

| TokuDB | indexer: number of calls to indexer->build() succeeded | 0 |

| TokuDB | indexer: number of calls to indexer->build() failed | 0 |

| TokuDB | indexer: number of calls to indexer->close() that succeeded | 0 |

| TokuDB | indexer: number of calls to indexer->close() that failed | 0 |

| TokuDB | indexer: number of calls to indexer->abort() | 0 |

| TokuDB | indexer: number of indexers currently in existence | 0 |

| TokuDB | indexer: max number of indexers that ever existed simultaneously | 0 |

| TokuDB | loader: number of loaders successfully created | 0 |

| TokuDB | loader: number of calls to toku_loader_create_loader() that failed | 0 |

| TokuDB | loader: number of calls to loader->put() succeeded | 0 |

| TokuDB | loader: number of calls to loader->put() failed | 0 |

| TokuDB | loader: number of calls to loader->close() that succeeded | 0 |

| TokuDB | loader: number of calls to loader->close() that failed | 0 |

| TokuDB | loader: number of calls to loader->abort() | 0 |

| TokuDB | loader: number of loaders currently in existence | 0 |

| TokuDB | loader: max number of loaders that ever existed simultaneously | 0 |

| TokuDB | memory: number of malloc operations | 0 |

| TokuDB | memory: number of free operations | 0 |

| TokuDB | memory: number of realloc operations | 0 |

| TokuDB | memory: number of malloc operations that failed | 0 |

| TokuDB | memory: number of realloc operations that failed | 2 |

| TokuDB | memory: number of bytes requested | 0 |

| TokuDB | memory: number of bytes used (requested + overhead) | 0 |

| TokuDB | memory: number of bytes freed | 0 |

| TokuDB | memory: largest attempted allocation size | 0 |

| TokuDB | memory: size of the last failed allocation attempt | 0 |

| TokuDB | memory: estimated maximum memory footprint | 0 |

| TokuDB | memory: mallocator version | 3.6.0-0-g46c0af68bd248b04df75e4f92d5fb804c3d75340 |

| TokuDB | memory: mmap threshold | 4194304 |

| TokuDB | filesystem: ENOSPC redzone state | 0 |

| TokuDB | filesystem: threads currently blocked by full disk | 0 |

| TokuDB | filesystem: number of operations rejected by enospc prevention (red zone) | 0 |

| TokuDB | filesystem: most recent disk full | Thu Jan 1 00:00:00 1970 |

| TokuDB | filesystem: number of write operations that returned ENOSPC | 0 |

| TokuDB | filesystem: fsync time | 53058474146 |

| TokuDB | filesystem: fsync count | 23234113 |

| TokuDB | filesystem: long fsync time | 2235827 |

| TokuDB | filesystem: long fsync count | 2 |

| TokuDB | context: tree traversals blocked by a full fetch | 0 |

| TokuDB | context: tree traversals blocked by a partial fetch | 0 |

| TokuDB | context: tree traversals blocked by a full eviction | 0 |

| TokuDB | context: tree traversals blocked by a partial eviction | 0 |

| TokuDB | context: tree traversals blocked by a message injection | 0 |

| TokuDB | context: tree traversals blocked by a message application | 0 |

| TokuDB | context: tree traversals blocked by a flush | 13 |

| TokuDB | context: tree traversals blocked by a the cleaner thread | 5 |

| TokuDB | context: tree traversals blocked by something uninstrumented | 2 |

| TokuDB | context: promotion blocked by a full fetch (should never happen) | 1 |

| TokuDB | context: promotion blocked by a partial fetch (should never happen) | 0 |

| TokuDB | context: promotion blocked by a full eviction (should never happen) | 0 |

| TokuDB | context: promotion blocked by a partial eviction (should never happen) | 0 |

| TokuDB | context: promotion blocked by a message injection | 11 |

| TokuDB | context: promotion blocked by a message application | 0 |

| TokuDB | context: promotion blocked by a flush | 0 |

| TokuDB | context: promotion blocked by the cleaner thread | 8 |

| TokuDB | context: promotion blocked by something uninstrumented | 1 |

| TokuDB | context: something uninstrumented blocked by something uninstrumented | 0 |

| TokuDB | handlerton: primary key bytes inserted | 67394770498 |

+--------+---------------------------------------------------------------------------------+---------------------------------------------------+

mysql> show variables like '%toku%';

+-----------------------------------------+-------------+

| Variable_name | Value |

+-----------------------------------------+-------------+

| tokudb_alter_print_error | OFF |

| tokudb_analyze_delete_fraction | 1.000000 |

| tokudb_analyze_time | 5 |

| tokudb_block_size | 4194304 |

| tokudb_bulk_fetch | ON |

| tokudb_cache_size | 31538616320 |

| tokudb_cachetable_pool_threads | 0 |

| tokudb_check_jemalloc | 1 |

| tokudb_checkpoint_lock | OFF |

| tokudb_checkpoint_on_flush_logs | OFF |

| tokudb_checkpoint_pool_threads | 0 |

| tokudb_checkpointing_period | 60 |

| tokudb_cleaner_iterations | 5 |

| tokudb_cleaner_period | 1 |

| tokudb_client_pool_threads | 0 |

| tokudb_commit_sync | ON |

| tokudb_compress_buffers_before_eviction | ON |

| tokudb_create_index_online | ON |

| tokudb_data_dir | |

| tokudb_debug | 0 |

| tokudb_directio | OFF |

| tokudb_disable_hot_alter | OFF |

| tokudb_disable_prefetching | OFF |

| tokudb_disable_slow_alter | OFF |

| tokudb_disable_slow_update | OFF |

| tokudb_disable_slow_upsert | OFF |

| tokudb_empty_scan | rl |

| tokudb_enable_partial_eviction | ON |

| tokudb_fanout | 16 |

| tokudb_fs_reserve_percent | 5 |

| tokudb_fsync_log_period | 0 |

| tokudb_hide_default_row_format | ON |

| tokudb_killed_time | 4000 |

| tokudb_last_lock_timeout | |

| tokudb_load_save_space | ON |

| tokudb_loader_memory_size | 100000000 |

| tokudb_lock_timeout | 4000 |

| tokudb_lock_timeout_debug | 1 |

| tokudb_log_dir | |

| tokudb_max_lock_memory | 3942327040 |

| tokudb_optimize_index_fraction | 1.000000 |

| tokudb_optimize_index_name | |

| tokudb_optimize_throttle | 0 |

| tokudb_pk_insert_mode | 2 |

| tokudb_prelock_empty | ON |

| tokudb_read_block_size | 65536 |

| tokudb_read_buf_size | 131072 |

| tokudb_read_status_frequency | 10000 |

| tokudb_row_format | tokudb_zlib |

| tokudb_rpl_check_readonly | ON |

| tokudb_rpl_lookup_rows | OFF |

| tokudb_rpl_lookup_rows_delay | 0 |

| tokudb_rpl_unique_checks | OFF |

| tokudb_rpl_unique_checks_delay | 0 |

| tokudb_support_xa | ON |

| tokudb_tmp_dir | |

| tokudb_version | 5.6.26-74.0 |

| tokudb_write_status_frequency | 1000 |

+-----------------------------------------+-------------+

On Saturday, October 31, 2015 at 1:26:18 PM UTC-7, Vinay Malkani wrote:

It is similar to time-series; however we do not delete data at all.

Thanks,

Vinay
ᐧ

On Sat, Oct 31, 2015 at 1:18 PM, Jon Tobin wrote:

Is this a "time series" use case? What I mean is, are you deleting data out of the table as it ages?

To unsubscribe from this group and stop receiving emails from it, send an email to tokudb-user+unsubscribe@googlegroups.com.

pt-pmp.log.gz

jon....@percona.com

unread,

Nov 3, 2015, 9:47:33 AM11/3/15

to tokudb-user, j...@tokutek.com

Vinay,

Can you post your .cnf file and a detailed description of your workload?

...

Vinay Malkani

unread,

Nov 3, 2015, 2:01:15 PM11/3/15

to tokudb-user, j...@tokutek.com

Hi Jon,

The machine happens to be a slave of our workload (no other connectivity other than being a slave currently). The overall use case is a high write load for log data on a second by second basis. The slave has 60GB ram and the master has 16GB ram. We thought increasing the ram on the slave to 60GB ram would help with conversion of data from innodb to tokudb (it seemed to helped). Our ultimate goal is to make the tokudb database a master so that we are running tokudb on both master and slave.

the my.cnf is currently set to this. I believe tokudb uses 50% of ram by default

---

[client]

port = 3306

socket = /var/run/mysqld/mysqld.sock

[mysqld_safe]

thp-setting=never

open_files_limit = 1024000

socket = /var/run/mysqld/mysqld.sock

nice = 0

syslog

[mysqld]

open_files_limit = 1024000

user = mysql

pid-file = /var/run/mysqld/mysqld.pid

socket = /var/run/mysqld/mysqld.sock

port = 3306

basedir = /opt/mysql/Percona-Server-5.6.26-rel74.0

datadir = /var/lib/mysql

tmpdir = /tmp

lc-messages-dir = /opt/mysql/Percona-Server-5.6.26-rel74.0/share

slave-parallel-workers = 12

skip-external-locking

bind-address = <readacted hostname>

max_heap_table_size = 32M

tmp_table_size = 32M

key_buffer = 384M

max_allowed_packet = 16M

thread_stack = 192K

thread_cache_size = 8

myisam-recover = BACKUP

max_connections = 1200

query_cache_limit = 1M

query_cache_size = 16M

slow-query-log = 1

slow_query_log_file = /mysql_logs/mysql-slow.log

long_query_time = 2

server-id = 51

auto_increment_increment= 2

auto_increment_offset = 2

log_bin = /mysql_logs/mysql-bin.log

expire_logs_days = 10

max_binlog_size = 100M

binlog_format=ROW

log-slave-updates=TRUE

skip-slave-start=TRUE

ssl = on

ssl-ca = <redacted>

ssl-cert = <redacted>

ssl-key = <redacted>

innodb_additional_mem_pool_size = 12M

innodb_buffer_pool_size = 4G

innodb_file_per_table = 1

innodb_flush_log_at_trx_commit = 2

innodb_lock_wait_timeout = 180

innodb_log_buffer_size = 16M

innodb_open_files = 1024

innodb_thread_concurrency = 0

innodb_log_file_size = 500M

innodb_flush_log_at_trx_commit=2

[mysqldump]

quick

quote-names

max_allowed_packet = 16M

[mysql]

[isamchk]

key_buffer = 384M

Reply all

Reply to author

Forward