Compile WiredTiger Extensions/Collator Windows

63 views
Skip to first unread message

David Seide

unread,
Sep 28, 2020, 8:22:28 AM9/28/20
to wiredtiger-users
Hi everyone,

I need help in compiling a custom collator on Windows.

On Linux I just went to ext/collators/reverse and executed make which creates ext/collators/reverse/.libs/libwiredtiger_reverse_collator.so which can be loaded as an extension.

Can someone help me to do the same on Windows with SCons?

Thanks you!

Luke Pearson

unread,
Sep 29, 2020, 2:31:29 AM9/29/20
to wiredtiger-users
Hi David,

Thanks for your interest, we will investigate and get back to you shortly.

Thanks,
Luke

Luke Pearson

unread,
Sep 30, 2020, 3:33:52 AM9/30/20
to wiredtiger-users
Hi David,

I don't think its possible to compile the example reverse collator or a custom one on windows without modifying the SConstruct file, I have written a patch that adds the reverse collator to libwiredtiger.lib and a reverse_collator.lib. Could you give that a try?

Patch:
diff --git a/SConstruct b/SConstruct
index 5eddc4e35..807c5296a 100644
--- a/SConstruct
+++ b/SConstruct
@@ -28,6 +28,9 @@ AddOption("--enable-diagnostic", dest="diagnostic", action="store_true", default
 AddOption("--enable-lz4", dest="lz4", type="string", nargs=1, action="store",
           help="Use LZ4 compression")

+AddOption("--enable-reverse-collator", dest="collator", type="string", nargs=1, action="store",
+          help="Build the reverse collator library")
+
 AddOption("--enable-java", dest="lang-java", type="string", nargs=1, action="store",
           help="Build java extension, specify location of swig.exe binary and Java JDK dir separated by comma")

@@ -110,7 +113,9 @@ useZlib = GetOption("zlib")
 useSnappy = GetOption("snappy")
 useLz4 = GetOption("lz4")
 useTcmalloc = GetOption("tcmalloc")
+useCollator = GetOption("collator")
 wtlibs = []
+reverse_collatorlibs = []

 conf = Configure(env)
 if not conf.CheckCHeader('stdlib.h'):
@@ -147,6 +152,12 @@ if useLz4:
         print('lz4.h must be installed!')
         Exit(1)

+if useCollator:
+    conf.env.Append(CPPPATH=[useCollator + "/include"])
+    conf.env.Append(LIBPATH=[useCollator + "/lib"])
+    reverse_collatorlibs.append("reverse_collator")
+    wtlibs.append("reverse_collator")
+
 if useTcmalloc:
     conf.env.Append(CPPPATH=[useTcmalloc + "/include"])
     conf.env.Append(LIBPATH=[useTcmalloc + "/lib"])
@@ -256,8 +267,15 @@ if useSnappy:
 if useLz4:
     wtsources.append("ext/compressors/lz4/lz4_compress.c")

+if useCollator:
+    wtsources.append("ext/collators/reverse/reverse_collator.c")
+
 wt_objs = [env.Object(a) for a in wtsources]

+reverse_collator = env.Library(
+target="reverse_collator",
+source="ext/collators/reverse/reverse_collator.c", LIBS=reverse_collatorlibs)
+
 # Static Library - libwiredtiger.lib
 #
 wtlib = env.Library(
@@ -274,6 +292,8 @@ wtdll = env.SharedLibrary(

 env.Depends(wtdll, [filelistfile, version_file])

+Default(reverse_collator)
+
 Default(wtlib, wtdll)

 wtbin = env.Program("wt", [

Regards,
Luke

David Seide

unread,
Sep 30, 2020, 8:54:44 AM9/30/20
to wiredtiger-users
Hi Luke,

Thanks so much for your help, I was editing the SConstruct file as well but didn't get that far alone.
I applied the patch and compiled everything, but I am not sure about which parameter to provide to enable-reverse-collator.
python F:\wt_win\scons-local-4.0.1\scons.py --enable-java="F:\wt_win\swigwin-3.0.12\swig.exe","C:\Program Files\Java\jdk1.8.0_201" --enable-lz4="F:\wt_win\lz4\forwt" --enable-reverse-collator="F:\wt_win\wiredtiger\ext\collators\reverse"   1> scons.log 2>&1

My next question is how to load and use the extension on Windows as a Java library.

On Linux I do:
connection.load_extension("reverse_collator.so", null);
Session session = ...
session.create(table:index-a prefix_compression=true,block_compressor=lz4,key_format=ur,value_format=,collator=reverse);

I am assuming that on Windows load_extension is obsolete because there is no .dll to load and the lz4 extension is also loaded without any further doing.
However, when trying to open the session, WT complains:
[1601456368:747393][1176:140737080939760], file:index-a.wt, WT_SESSION.create: unknown collator 'reverse': Invalid argument

What am I missing?

Best,
David
scons.log

Luke Pearson

unread,
Oct 1, 2020, 1:19:48 AM10/1/20
to wiredtiger-users
Hi David, 

It seems that building on windows requires you to build the extensions as builtins, which means there's some code add and an additional line to add to the SConstruct. I've attached a patch that gets the reverse collator working on my windows machine.

To build I use the following command:
scons --enable-reverse-collator=1

Let me know if that works for you.

Regards,
Luke
patch

David Seide

unread,
Oct 1, 2020, 4:21:27 AM10/1/20
to wiredtiger-users
It works as expected, your help is much appreciated!
Now I can implement my own collator and use it on Macos, Linux and Windows.

I hope people find this post when fiddling with the same problem.
Thanks again!
Reply all
Reply to author
Forward
0 new messages