Compile WiredTiger Extensions/Collator Windows

[David Seide]

Hi everyone,

I need help in compiling a custom collator on Windows.

On Linux I just went to ext/collators/reverse and executed make which creates ext/collators/reverse/.libs/libwiredtiger_reverse_collator.so which can be loaded as an extension.

Can someone help me to do the same on Windows with SCons?

Thanks you!

[Luke Pearson]

Hi David,

Thanks for your interest, we will investigate and get back to you shortly.

Thanks,
Luke

[Luke Pearson]

Hi David,

I don't think its possible to compile the example reverse collator or a custom one on windows without modifying the SConstruct file, I have written a patch that adds the reverse collator to libwiredtiger.lib and a reverse_collator.lib. Could you give that a try?

Patch:

diff --git a/SConstruct b/SConstruct
index 5eddc4e35..807c5296a 100644
--- a/SConstruct
+++ b/SConstruct
@@ -28,6 +28,9 @@ AddOption("--enable-diagnostic", dest="diagnostic", action="store_true", default
 AddOption("--enable-lz4", dest="lz4", type="string", nargs=1, action="store",
           help="Use LZ4 compression")

+AddOption("--enable-reverse-collator", dest="collator", type="string", nargs=1, action="store",
+          help="Build the reverse collator library")
+
 AddOption("--enable-java", dest="lang-java", type="string", nargs=1, action="store",
           help="Build java extension, specify location of swig.exe binary and Java JDK dir separated by comma")

@@ -110,7 +113,9 @@ useZlib = GetOption("zlib")
 useSnappy = GetOption("snappy")
 useLz4 = GetOption("lz4")
 useTcmalloc = GetOption("tcmalloc")
+useCollator = GetOption("collator")
 wtlibs = []
+reverse_collatorlibs = []

 conf = Configure(env)
 if not conf.CheckCHeader('stdlib.h'):
@@ -147,6 +152,12 @@ if useLz4:
         print('lz4.h must be installed!')
         Exit(1)

+if useCollator:
+    conf.env.Append(CPPPATH=[useCollator + "/include"])
+    conf.env.Append(LIBPATH=[useCollator + "/lib"])
+    reverse_collatorlibs.append("reverse_collator")
+    wtlibs.append("reverse_collator")
+
 if useTcmalloc:
     conf.env.Append(CPPPATH=[useTcmalloc + "/include"])
     conf.env.Append(LIBPATH=[useTcmalloc + "/lib"])
@@ -256,8 +267,15 @@ if useSnappy:
 if useLz4:
     wtsources.append("ext/compressors/lz4/lz4_compress.c")

+if useCollator:
+    wtsources.append("ext/collators/reverse/reverse_collator.c")
+
 wt_objs = [env.Object(a) for a in wtsources]

+reverse_collator = env.Library(
+target="reverse_collator",
+source="ext/collators/reverse/reverse_collator.c", LIBS=reverse_collatorlibs)
+
 # Static Library - libwiredtiger.lib
 #
 wtlib = env.Library(
@@ -274,6 +292,8 @@ wtdll = env.SharedLibrary(

 env.Depends(wtdll, [filelistfile, version_file])

+Default(reverse_collator)
+
 Default(wtlib, wtdll)

 wtbin = env.Program("wt", [

Regards,

Luke

[David Seide]

Hi Luke,

Thanks so much for your help, I was editing the SConstruct file as well but didn't get that far alone.

I applied the patch and compiled everything, but I am not sure about which parameter to provide to enable-reverse-collator.

python F:\wt_win\scons-local-4.0.1\scons.py --enable-java="F:\wt_win\swigwin-3.0.12\swig.exe","C:\Program Files\Java\jdk1.8.0_201" --enable-lz4="F:\wt_win\lz4\forwt" --enable-reverse-collator="F:\wt_win\wiredtiger\ext\collators\reverse"   1> scons.log 2>&1

My next question is how to load and use the extension on Windows as a Java library.

On Linux I do:

connection.load_extension("reverse_collator.so", null);

Session session = ...

session.create(table:index-a prefix_compression=true,block_compressor=lz4,key_format=ur,value_format=,collator=reverse);

I am assuming that on Windows load_extension is obsolete because there is no .dll to load and the lz4 extension is also loaded without any further doing.

However, when trying to open the session, WT complains:

[1601456368:747393][1176:140737080939760], file:index-a.wt, WT_SESSION.create: unknown collator 'reverse': Invalid argument

What am I missing?

Best,

David

[Luke Pearson]

Hi David, 

It seems that building on windows requires you to build the extensions as builtins, which means there's some code add and an additional line to add to the SConstruct. I've attached a patch that gets the reverse collator working on my windows machine.

To build I use the following command:
scons --enable-reverse-collator=1

Let me know if that works for you.

Regards,
Luke

[David Seide]

It works as expected, your help is much appreciated!

Now I can implement my own collator and use it on Macos, Linux and Windows.

I hope people find this post when fiddling with the same problem.

Thanks again!