Eviction threads vs worker threads

[Puneet M]

Hi, 

I have a heavy insertion workload, where each application thread inserts rows into the same table. My machine supports a maximum of 28 threads. I am setting the WT_CACHE size to 10GB, and the data I am trying to insert is 1.7GB (ASCII format). 

When I use all the threads available on the system, I can see that it takes longer to finish insertion than when I use 20 threads. I suspected that it might be because of eviction, and I tried to set the  eviction=(threads_max=8)when opening the connection. This does not seem to help either. 

When we specify the eviction thread counts, does it mean that out of the N threads an application uses, threads_max number of them would get used for eviction, or does it mean that WT will use up to threads_max additional threads for eviction?

Also, is there any recommendation on deciding how much cache to use for the connection and guidelines on choosing the eviction_dirty_trigger and the eviction_dirty_target parameters?

thanks,

Puneet

[Will Korteland]

Hi Puneet,

Thanks for your interest in WiredTiger :-)

Without having access to the test, it's a little hard to make a diagnosis here. For an insert-heavy workload, eviction becoming a bottleneck does make sense. The "threads_max" configuration refers to the maximum number of standalone eviction threads that WiredTiger will create - if we get to the "eviction_dirty_trigger", then any threads calling into WiredTiger will also be roped into eviction. The WiredTiger documentation goes into a little more detail here: https://source.wiredtiger.com/develop/group__wt.html#gacbe8d118f978f5bfc8ccb4c77c9e8813

The "right" amount of cache to use will depend on your needs - if WiredTiger is the only thing on the machine, go forth and give it half of the machine's RAM (and let the OS the rest for e.g. the page cache). Otherwise, it will depend on the memory requirements of whatever else you're trying to run as well.

Regarding the "eviction_dirty_target" and "eviction_dirty_trigger": this also depends on your use case. If there's a less write-intensive steady state that gets reached after the insert, then it possibly isn't worth optimising this insert phase. If you expect to sustain 20+ threads all inserting as fast as they can, the disk will become a bottleneck (visible via eviction) no matter what you configure those values to.

Hopefully that helps - if you've got more questions or something didn't make sense, feel free to get back to me.

Cheers,

 - Will

--
You received this message because you are subscribed to the Google Groups "wiredtiger-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wiredtiger-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wiredtiger-users/77bd90f9-2bdf-4480-bdf3-4bdf894149e4n%40googlegroups.com.