Including new library into mongodb using scons

[SH SHin]

Hi,

I have already included my new library libtto.so in /usr/lib and libtto.h in /usr/include. 

After modifying some part of codes to utilize the functions included in libtto, I tried scons -j16 all, and what I saw was this.

src/mongo/db/ftdc/controller.cpp:154: error: undefined reference to 'tto_get_bg_task()'

src/mongo/db/ftdc/controller.cpp:212: error: undefined reference to 'tto_begin(unsigned long)'

src/mongo/db/ftdc/controller.cpp:222: error: undefined reference to 'tto_end(unsigned long)'

src/mongo/db/instance.cpp:237: error: undefined reference to 'tto_begin(unsigned long)'

src/mongo/db/instance.cpp:256: error: undefined reference to 'tto_end(unsigned long)'

src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp:106: error: undefined reference to 'taio_begin(unsigned long)'

src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp:111: error: undefined reference to 'taio_begin(unsigned long)'

collect2: error: ld returned 1 exit status

scons: *** [build/opt/mongo/db/exec/sort_test] Error 1

What I have tried so far is, (library hwloc is required for libtto)

in src/mongo/SConscript, 

.

.

.

env.Append(LIBPATH=['tto'])

env.Append(LIBPATH=['hwloc'])

env.Append(LIBPATH=['/usr/lib'])

env.Append(CPPPATH=['tto'])

env.Append(CPPPATH=['hwloc'])

env.Append(CPPPATH=['/usr/include'])

.

.

.

mongod = env.Program(

     target="mongod",

     source=[

         "db/db.cpp",

         "db/mongod_options_init.cpp",

     ],

     LIBDEPS=mongodLibDeps,

     SYSLIBDEPS=[

         "tto",

         "hwloc",

     ], 

 )

.

.

.

env.Library(

     target='base',

     source=baseSource,

     LIBDEPS=baseLibDeps,

     SYSLIBDEPS=[

         "tto",

         "hwloc"

     ],  

 )

.

.

in src/third_party/wiredtiger/SConstruct

.

.

conf = Configure(env)

conf.env.Append(LIBPATH=['tto']) 

conf.env.Append(LIBPATH=['hwloc']) 

conf.env.Append(CPPPATH=['tto']) 

conf.env.Append(CPPPATH=['hwloc']) 

.

.

in src/third_party/wiredtiger/SConscript

.

.

env.Append(LIBPATH=['tat'])

env.Append(LIBPATH=['hwloc'])

env.Append(LIBPATH=['/usr/lib'])

env.Append(CPPPATH=['tto'])

env.Append(CPPPATH=['hwloc'])

env.Append(CPPPATH=['/usr/include'])

 

baseLibDeps=[

..

.

wtlib = env.Library(

     target='wiredtiger,

     source=wtsources,

     LIBDEPS=[

         '$BUILD_DIR/third_party/shim_snappy',

         '$BUILD_DIR/third_party/shim_zlib',

     ],

     SYSLIBDEPS=[

         "tto",

         "hwloc"

     ],  

 )

 .

.

.

And also tried similar modification on the top SConscript but always same result as shown above.

Is there any more modification that I should put in to let scons recognize my new library??

I've been stuck with this problem for several days and no progress....

[Andrew Morrow]

The part where you are adding tto and hwloc to the SYSLIBDEPS of various things seems fine, but the other changes are not correct. First, you shouldn't be hardcoding changes to CPPPATH and LIBPATH into the SConscript like this. Second, you seem to be mixing paths and library names when you do so. And finally, you shouldn't need to add /usr/include and /usr/lib to the include and library search paths, since they are automatically searched by the toolchain unless you are doing something very unusual.

Instead you can pass additional paths by adding them to the SCons invocation command line when you call it, and those directories will be searched as needed. So if you have a package foo in /opt/foo/{include,lib} and a package bar in /opt/bar/{include, lib} that you want to be found, then add foo and bar to SYSLIBDEPS as needed, and then add CPPATH="/opt/foo/include /opt/bar/include" LIBPATH="/opt/foo/lib /opt/bar/lib" to the command line when you invoke SCons. Note that I wouldn't expect you to need to do this for hwloc as I would expect it to appear in the system include and library search paths. I don't know what tto is, so I can't speak as to the correct way to locate it, but the general idea is the same.

Thanks,

Andrew

--
You received this message because you are subscribed to the Google Groups "mongodb-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-dev+unsubscribe@googlegroups.com.
To post to this group, send email to mongo...@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-dev.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-dev/16c9deee-6df9-4033-8c41-a966260453c5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[SH SHin]

Thanks for your reply. Here's what I've done.

libtto is implemented by ourselves, and it is successfully installed in our /usr/lib and /usr/include as libtto, libtto.so, libtto.so.1, and libtto.so.1.0.1 in /usr/lib and libtaio.h in /usr/include.

I got rid of all lines manually adding CPPPATH and LIBPATH in all SConstruct and SConscript in mongodb, /src/mongo, and /src/third_pary/wiredtiger. Didn't modify SYSLIBDEPS part since you mentioned it is okay.

Since the recommendation was there is nothing to do for /usr/lib, and /usr/include, and the target library libtto is installed in those folders, then I have nothing to modify in my cmd-line. instead I tried to include /usr/include/x86_64-linux-gnu and /usr/lib/x86_64-linux-gnu for CPPPATH and LIBPATH repectively, where libhwloc is located at. And same results come out.

src/mongo/db/ftdc/controller.cpp:154: error: undefined reference to 'tto_get_bg_task_id()'

src/mongo/db/ftdc/controller.cpp:212: error: undefined reference to 'tto_task_begin(unsigned long)'

src/mongo/db/ftdc/controller.cpp:222: error: undefined reference to 'tto_task_end(unsigned long)'

src/mongo/db/instance.cpp:237: error: undefined reference to 'tto_task_begin(unsigned long)'

src/mongo/db/instance.cpp:256: error: undefined reference to 'tto_task_end(unsigned long)'

src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp:106: error: undefined reference to 'tto_task_begin(unsigned long)'

src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp:111: error: undefined reference to 'tto_task_begin(unsigned long)'

cmd-line was

scons CPPPATH="/usr/include/x86_64-linux-gnu" LIBPATH="/usr/lib/x86_86-linux-gnu" -j16 all

If it is true that scons automatically collect the library information from CPPPATH and LIBPATH(I think this is right because we checked through CheckLib that the program acknowledges hwloc, but not tto, our self-implemented library), does that mean that there is a problem on our self-implemented library libtto? then, do I need to put the source code for libtto into mongodb manually?

To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-dev...@googlegroups.com.

[Andrew Morrow]

Several follow-up questions, please see below

On Wed, Apr 25, 2018 at 6:12 AM, SH SHin <horn...@gmail.com> wrote:

Thanks for your reply. Here's what I've done.

libtto is implemented by ourselves, and it is successfully installed in our /usr/lib and /usr/include as libtto, libtto.so, libtto.so.1, and libtto.so.1.0.1 in /usr/lib and libtaio.h in /usr/include.

OK.

 

I got rid of all lines manually adding CPPPATH and LIBPATH in all SConstruct and SConscript in mongodb, /src/mongo, and /src/third_pary/wiredtiger. Didn't modify SYSLIBDEPS part since you mentioned it is okay.

Since the recommendation was there is nothing to do for /usr/lib, and /usr/include, and the target library libtto is installed in those folders, then I have nothing to modify in my cmd-line. instead I tried to include /usr/include/x86_64-linux-gnu and /usr/lib/x86_64-linux-gnu for CPPPATH and LIBPATH repectively, where libhwloc is located at.

I'd be surprised if you need those either. Unless there is something funky with your toolchain I'd expect the architecture specific include and library search paths to be automatically searched. Are you using the system toolchain, or one you rolled yourself? Can it still locate the hwloc.h header if you don't add those?

 

And same results come out.

src/mongo/db/ftdc/controller.cpp:154: error: undefined reference to 'tto_get_bg_task_id()'

src/mongo/db/ftdc/controller.cpp:212: error: undefined reference to 'tto_task_begin(unsigned long)'

src/mongo/db/ftdc/controller.cpp:222: error: undefined reference to 'tto_task_end(unsigned long)'

src/mongo/db/instance.cpp:237: error: undefined reference to 'tto_task_begin(unsigned long)'

src/mongo/db/instance.cpp:256: error: undefined reference to 'tto_task_end(unsigned long)'

src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp:106: error: undefined reference to 'tto_task_begin(unsigned long)'

src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp:111: error: undefined reference to 'tto_task_begin(unsigned long)'

- Have you added 'tto' as a SYSLIBDEP to each of the libraries that contain controller.cpp, instance.cpp, and wiredtiger_kv_engine.cpp? You should be.

- Have you verified that libtto.so.1 actually exports the symbols that are not found here? You should be able to check that with nm.

- Can you post here the modified declarations for the libraries?

 

cmd-line was

scons CPPPATH="/usr/include/x86_64-linux-gnu" LIBPATH="/usr/lib/x86_86-linux-gnu" -j16 all

I'll re-iterate that I find it odd that you would need to specify the arch specific directories unless something odd with your toolchain.

 

If it is true that scons automatically collect the library information from CPPPATH and LIBPATH(I think this is right because we checked through CheckLib that the program acknowledges hwloc, but not tto, our self-implemented library), does that mean that there is a problem on our self-implemented library libtto? then, do I need to put the source code for libtto into mongodb manually?

- Can you show the call to CheckLib that you are doing?

- You should try a little more to get it to work the way you are now. Vendoring the code comes with its own set of problems. I'm pretty sure we can get it to work.

 

To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-dev+unsubscribe@googlegroups.com.

To post to this group, send email to mongo...@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-dev.

To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-dev/f0e36a4c-38bc-4152-949d-a3076adc7ed6%40googlegroups.com.