WiredTiger with QAT-ZSTD-Plugin

[Microcraft]

Hi!

I want to ask how to compile WiredTiger so that it can use Intel's QAT-ZSTD-Plugin. Here is its Github address:

QAT-ZSTD-Plugin

In the README file, it is mentioned that if you want to integrate it into the application, you need to call three high-level interfaces of the zstd library. However, after checking some of the WiredTiger code, I think it only calls some basic interfaces. I am having a headache on this matter.

[Marc Butler]

Hi Microcraft,

Thank you for your interest in WiredTiger.

You'll find WiredTiger has support for zstd as a plugin, the plugin code is located in: ext/compressors/zstd/ in the source tree.

To enable the plugin you'll need to run cmake with either -DENABLE_ZSTD=ON or -DHAVE_BUILTIN_EXTENSION_ZSTD=ON. 

Something to consider is that when enabled the cmake build will look for the zstd.h header and the zstd library in the current environment.

Hint: look for zstd in cmake/configs/auto.cmake and the config_lib functionality is defined in: cmake/helpers.cmake

So a potential problem is the build finding the wrong zstd to link against: for instance the default zstd development libraries installed in the system and not the the QAT version.

In terms of modifying the zstd plugin you could copy and rename (for example zstdqat) the existing zstd plugin code in: ext/compressors/zstd as well as duplicating support for building the plugin in the cmake build system, where can also copy and rename the existing zstd support.

Alternatively you could modify the existing plugin, to support QAT which could be enabled using a configuration parameter. I suspect this will take longer to get working, as it requires more intricate changes.

I am not familiar with the QAT extension, but do note from the readme in the repository you referenced that it entails configuring the zstd compression context ZSTD_CCtx.

The zstd plugin uses a pool of pre-allocated contexts of size CONTEXT_POOL_SIZE, and falls back on compressing without context when there are no free contexts in that pool.

I would suggest that this part of the plugin likely needs close review to successfully support the QAT extension.

Regards,

Marc

--
You received this message because you are subscribed to the Google Groups "wiredtiger-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wiredtiger-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wiredtiger-users/3232f172-bdd0-4955-b347-733683044cf4n%40googlegroups.com.