hard time with SConstruct

[vkuznet]

Hi,
I'm still experience different problems with a Mongo build system, in
particular how SConstruct creates compiler statements. I'm working in
environment where many versions of the same software can coexists,
both in system and user areas. So, I invoke scons as following:

scons --64 --extrapath=$PCRE_ROOT,$BOOST_ROOT,$SPIDERMONKEY_ROOT all

and compilation part is fine, while at link time I'm having problems:
g++ -o mongodump -fPIC -pthread -rdynamic ....
-L/lib -L/build/vk/DAS/slc5_amd64_gcc434/external/boost/1.40.0/lib -L/
build/vk/DAS/slc5_amd64_gcc434/external/spidermonkey/1.8.0-rc1/lib -L/
usr/lib64 -L/lib64 -lpthread -lstdc++ -lboost_system /usr/lib64/
libboost_thread.a /usr/lib64/libboost_filesystem.a /usr/lib64/
libboost_program_options.a /usr/lib64/libpcrecpp.a /usr/lib64/
libpcre.a /build/vk/DAS/slc5_amd64_gcc434/external/spidermonkey/1.8.0-
rc1/lib/libjs.a

As you can see, SConsturct properly supplied -lboost_system, while
other boost libraries it picked up from the system area. My boost
area, provided via $BOOST_ROOT, is coming first in a pass for linker,
but since SConstruct used full path names to static libraries it
picked libraries from there, instead of providing -l<lib> to the
linker.

The part of SConstruct which doing this is the following:
def myCheckLib( poss , failIfNotFound=False , java=False ,
staticOnly=False):

if type( poss ) != types.ListType :
poss = [poss]

allPlaces = [];
if nix and release:
allPlaces += myenv["LIBPATH"]
if not force64:
allPlaces += [ "/usr/lib" , "/usr/local/lib" ]

for p in poss:
for loc in allPlaces:
fullPath = loc + "/lib" + p + ".a"
if os.path.exists( fullPath ):
myenv['_LIBFLAGS']='${_stripixes
(LIBLINKPREFIX, LIBS, LIBLINKSUFFIX, LIBPREFIXES, LIBSUFFIXES,
__env__)} $SLIBS'
myenv.Append( SLIBS=" " + fullPath + " " )
return True

It explicitly look at static libraries and constructs a full path with
static library. Can someone have a look and fix that to use -l<lib>
instead.
Thanks,
Valentin.

[Eliot Horowitz]

We can't use -l for various reasons including doing releases, but I
did just make a change that should fix this.
It now looks in extrapath directories first for .a files, see if this
fixes your issue.
(in master)

> --
>
> You received this message because you are subscribed to the Google Groups "mongodb-user" group.
> To post to this group, send email to mongod...@googlegroups.com.
> To unsubscribe from this group, send email to mongodb-user...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.
>
>
>

[vkuznet]

Unfortunately I force to use boost shared libraries. We share software
components among different projects. So the look-up should be done
first for shared libraries then for static ones.

[Eliot Horowitz]

We can't change the default behavior unfortunately.
I added another option
--extrapathdyn
that should do what you want, so just use that instead of --extrapath

[vkuznet]

Cool, will try.

[vkuznet]

Eliot,
the extra option you add, doesn't do much, it just add a path, rather
specify the libraries. I look at SConstruct and made appropriate
changes. Below you can find a patch. I was able to build with it the
mogno linked against shared boost libraries. Please have a look. The
change also handle fully static build if you need one, which is
specific via --static option, in this case section with shared library
check is skipped. I also found another problem with spidermonkey
check, which fixed as well.


diff --git a/SConstruct b/SConstruct
index 50ea9bd..7c6a413 100644
--- a/SConstruct
+++ b/SConstruct
@@ -156,7 +156,7 @@ AddOption( "--extrapathdyn",
type="string",
nargs=1,
action="store",
- help="comma seperated list of add'l paths (--extrapath /
opt/foo/,/foo) dynamic linking" )
+ help="comma seperated list of add'l paths (--
extrapathdyn /opt/foo/,/foo) dynamic linking" )


AddOption( "--extralib",
@@ -731,6 +731,12 @@ def doConfigure( myenv , needJava=True ,
needPcre=True , shell=False ):

for p in poss:
for loc in allPlaces:

+ if not static:
+ fullPath = loc + "/lib" + p + ".so"
+ if os.path.exists( fullPath ):
+ myenv['_LIBFLAGS']='${_stripixes
(LIBLINKPREFIX, LIBS, LIBLINKSUFFIX, LIBPREF
+ myenv.Append( SLIBS=" -l%s" % p )
+ return True

fullPath = loc + "/lib" + p + ".a"
if os.path.exists( fullPath ):
myenv['_LIBFLAGS']='${_stripixes
(LIBLINKPREFIX, LIBS, LIBLINKSUFFIX, LIBPREFIXES

@@ -819,6 +825,8 @@ def doConfigure( myenv , needJava=True ,
needPcre=True , shell=False ):
else:
print( "no spider monkey headers!" )
Exit(1)
+ else:
+ myenv.Append(CPPDEFINES=[ "OLDJS" ])

if usev8:
if debugBuild:

[Eliot Horowitz]

Why do you need that?
If the .so are in that path, then you shouldn't need to specify them
on the command line like that.

[Andy Madsen]

Hi everyone:

Sorry, but I'm haveing trouble with an E11000 error. I use Mongo for
session management on a pretty high-traffic site, and have had issues
with users dropping sessions (and session data). Prior to today, I
assumed it was an application error, but tailing Mongo's forked log, I
see a number of these:

Thu Dec 10 07:15:02 User Exception E11000 duplicate key errorindex:
some_session_db.session.$_id_ dup key: { :
ObjId(4b2110768e3d1f56007a5cd1) }

I'm letting Mongo generate it's own ObjId, so I'm puzzled as to how
we're generating duplicates. This is a low-probability event, right?
Should I use "upsert" as opposed to "insert" when creating new sessions?
Session data is written multiple times during a request, so I'm assuming
that's the source of the collisions.

[Eliot Horowitz]

You need to use upsert or save if you want to overwrite an old session
(which is probably what you're doing)
insert will fail if the _id already exists
you want
.update( { _id : xxx } , { ... } )
most drivers have a save() method which is smart and will do an insert
if there isn't an _id and a update if there is.

[Michael Dirolf]

Andy,

Can you give us a bit more info about your setup? What driver are you using? What version?

On Thu, Dec 10, 2009 at 10:38 AM, Andy Madsen <ama...@gmail.com> wrote:

[Andy Madsen]

Hi Michael:

I'm using Mongo 1.1.3 in Centos5, PHP 5.2 with the 0.9.5 driver. Machine has four Dual 3G Proc with 12 Gigs RAM.

Just FYI, I switched from insert to update and still get this error.  Heres' the specific lines of code. Note this is about all there is to it, other than a GC function:

function::Init

$this -> m = new Mongo( somehost );
$this -> collection = $this -> m ->selectDB( "some_session" )->selectCollection( "session" );
$this -> collection-> ensureIndex( "session_guid" );

...

function:Read
$query = array( "session_guid" => $guid );
$result = $this -> collection->findOne( $query );
   
if ($result){
    return $result["session_data"];
} else {
    $doc = array (
    "session_guid"=>$guid,
    "session_data"=>"",
    "session_time"=>time());
   
    //Removed the "insert" here, replaced with update
    if ($this -> collection->update( $query,
                                 $doc,
                                 true ))
    { return '' }
}

...

function:Write
$query = array( "session_guid" => $guid );
$doc = array (
      "session_guid"=>$guid,
      "session_data"=>$data,
      "session_time"=>time());

if ($this -> collection)
{
  $this -> collection-> update($query,$doc,true);
  return true;

[Kristina Chodorow]

You should upgrade to the latest version if the PHP driver (1.0.1).  MongoId generation was not guaranteed unique in old versions of the driver... it used 8 bytes of random numbers, but there could be collisions, especially with high loads.

[vkuznet]

Eliot,
according to code in SConstruct it will pick up .a if it found it. So
on my system there are system level boost libraries sitting in /usr/
lib64. What's bad that they installed incomplete set (either by
mistake or any other reason). My path supplied via --extrapath or --
extrapathdyn just added to the path search list. This path is used to
look-at static libraries, e.g. fullPath = loc + "/lib" + p + ".a", but
not dynamic ones. Therefore I endup with statment
g++ .... -L$MYPATH -L/usr/lib64 ... -lboot_system /usr/lib64/
libboost_thread.a ...

So it's a mixture. The SConstruct look-up explicitly only
boost_system, but not the others.

[Eliot Horowitz]

if you use scons --extrapathdym it shouldn't try to add any .a files at all.
it should just do -L and -l not foo.a

[vkuznet]

Ok,
remove the change and run it again. Here is the output
scons: Reading SConscript files ...
scons version: 1.2.0
python version: 2 6 4 'final' 0
Checking whether the C++ compiler works(cached) yes
error: no result
Checking for C library stdc++... (cached) yes
Checking for C++ header file pcrecpp.h... (cached) yes
Checking for C++ header file boost/filesystem/operations.hpp...
(cached) yes
Checking for C++ header file boost/asio.hpp... (cached) yes
Checking for C library boost_thread-mt... no
Checking for C library boost_thread... yes
Checking for C library boost_filesystem-mt... no
Checking for C library boost_filesystem... no
can't find library ['boost_filesystem-mt', 'boost_filesystem'] in ['/
lib', '/build/vk/DAS/slc5_amd64_gcc434/external/boost/1.40.0/lib', '/
build/vk/DAS/slc5_amd64_gcc434/external/spidermonkey/1.8.0_rc1-cmp/
lib', '/usr/lib64', '/lib64']

And here is my libraries:
ls /build/vk/DAS/slc5_amd64_gcc434/external/boost/1.40.0/lib/
libboost_filesystem.so*
/build/vk/DAS/slc5_amd64_gcc434/external/boost/1.40.0/lib/
libboost_filesystem.so
/build/vk/DAS/slc5_amd64_gcc434/external/boost/1.40.0/lib/
libboost_filesystem.so.1.40.0

Weird part that it found boost_thread.

and I invoked scons as
scons --64 --cxx=$CXX --extrapathdyn=$PCRE_ROOT,$BOOST_ROOT,
$SPIDERMONKEY_ROOT all

[Eliot Horowitz]

Can you send config.log
Not sure why that's failing, doesn't make sense to me yet.

[vkuznet]

Ahh, here we go:

scons: Configure: Checking for C library boost_filesystem...
.sconf_temp/conftest_8.c <-
|
|
|
|int
|main() {
|
|return 0;
|}
|
/build/vk/DAS/slc5_amd64_gcc434/external/gcc/4.3.4/bin/g++ -
o .sconf_temp/conftest_8.o -c -fPIC -fno-strict-aliasing -ggdb -
pthread -Wall -Wsign-compare -Wno-unknown-pragmas -Winvalid-pch -O3 -
D_SCONS -DUSE_ASIO_OFF -I. -I/include -I/build/vk/DAS/
slc5_amd64_gcc434/external/boost/1.40.0/include -I/build/vk/DAS/
slc5_amd64_gcc434/external/spidermonkey/1.8.0_rc1-cmp/
include .sconf_temp/conftest_8.c
/build/vk/DAS/slc5_amd64_gcc434/external/gcc/4.3.4/bin/g++ -
o .sconf_temp/conftest_8 -fPIC -pthread -rdynamic .sconf_temp/
conftest_8.o -L/lib -L/build/vk/DAS/slc5_amd64_gcc434/external/boost/
1.40.0/lib -L/build/vk/DAS/slc5_amd64_gcc434/external/spidermonkey/
1.8.0_rc1-cmp/lib -L/usr/lib64 -L/lib64 -lpthread -lstdc++ -
lboost_thread -lboost_filesystem
/build/vk/DAS/slc5_amd64_gcc434/external/gcc/4.3.4/bin/../lib/gcc/
x86_64-unknown-linux-gnu/4.3.4/../../../../x86_64-unknown-linux-gnu/
bin/ld: warning: libboost_system.so.1.40.0, needed by /build/vk/DAS/
slc5_amd64_gcc434/external/boost/1.40.0/lib/libboost_filesystem.so,
not found (try using -rpath or -rpath-link)
/build/vk/DAS/slc5_amd64_gcc434/external/boost/1.40.0/lib/
libboost_filesystem.so: undefined reference to
`boost::system::get_generic_category()'
/build/vk/DAS/slc5_amd64_gcc434/external/boost/1.40.0/lib/
libboost_filesystem.so: undefined reference to
`boost::system::get_system_category()'
collect2: ld returned 1 exit status
scons: Configure: no

while if I do with my change and invoke scons as
scons --64 --cxx=$CXX --extrapath=$PCRE_ROOT,$BOOST_ROOT,
$SPIDERMONKEY_ROOT all
I get everything build

/build/vk/DAS/slc5_amd64_gcc434/external/gcc/4.3.4/bin/g++ -o mongod -
fPIC -pthread -rdynamic stdafx.o buildinfo.o db/jsobj.o db/json.o db/
commands.o db/lasterror.o db/nonce.o db/queryutil.o shell/mongo.o util/
background.o util/mmap.o util/sock.o util/util.o util/message.o util/
assert_util.o util/httpclient.o util/md5main.o util/base64.o util/
debug_util.o util/md5.o client/connpool.o client/dbclient.o client/
model.o client/parallel.o scripting/engine.o util/mmap_posix.o util/
processinfo_linux2.o scripting/engine_spidermonkey.o db/query.o db/
update.o db/introspect.o db/btree.o db/clientcursor.o db/tests.o db/
repl.o db/btreecursor.o db/cloner.o db/namespace.o db/matcher.o db/
dbeval.o db/dbwebserver.o db/dbinfo.o db/dbhelpers.o db/instance.o db/
pdfile.o db/cursor.o db/security_commands.o db/client.o db/security.o
util/miniwebserver.o db/storage.o db/reccache.o db/queryoptimizer.o db/
extsort.o db/mr.o s/d_util.o db/dbcommands.o db/dbcommands_admin.o s/
d_logic.o db/module.o db/modules/mms.o db/db.o -L/lib -L/build/vk/DAS/
slc5_amd64_gcc434/external/boost/1.40.0/lib -L/build/vk/DAS/
slc5_amd64_gcc434/external/spidermonkey/1.8.0_rc1-cmp/lib -L/usr/lib64
-L/lib64 -lpthread -lstdc++ -lboost_thread -lboost_filesystem -
lboost_program_options -lboost_system -lpcrecpp -lpcre -ljs

So the test failed because it misses other boost libraries.

> ...
>
> read more »

[Eliot Horowitz]

Great.
That makes a whole lot more sense now :)

[vkuznet]

Hi,
did you get change to fix this issue in master? I'll be willing to
test it within our build environment.
Thanks,
Valentin

> >> >> > according to code inSConstructit will pick up .a if it found it. So

> >> >> > on my system there are system level boost libraries sitting in /usr/
> >> >> > lib64. What's bad that they installed incomplete set (either by
> >> >> > mistake or any other reason). My path supplied via --extrapath or --
> >> >> > extrapathdyn just added to the path search list. This path is used to
> >> >> > look-at static libraries, e.g. fullPath = loc + "/lib" + p + ".a", but
> >> >> > not dynamic ones. Therefore I endup with statment
> >> >> > g++ .... -L$MYPATH -L/usr/lib64 ... -lboot_system /usr/lib64/
> >> >> > libboost_thread.a ...
>

> >> >> > So it's a mixture. TheSConstructlook-up explicitly only

> >> >> > boost_system, but not the others.
>
> >> >> > On Dec 10, 10:29 am, Eliot Horowitz <eliothorow...@gmail.com> wrote:
> >> >> >> Why do you need that?
> >> >> >> If the .so are in that path, then you shouldn't need to specify them
> >> >> >> on the command line like that.
>
> >> >> >> On Thu, Dec 10, 2009 at 10:20 AM, vkuznet <vkuz...@gmail.com> wrote:
> >> >> >> > Eliot,
> >> >> >> > the extra option you add, doesn't do much, it just add a path, rather

> >> >> >> > specify the libraries. I look atSConstructand made appropriate

> ...
>
> read more »

[Eliot Horowitz]

I thought we had made all the changes you need...

[vkuznet]

Hmm,
I got the opposite thought. The option you added, --extrapathdyn,
e.g.

scons --64 --cxx=$CXX --extrapathdyn=$PCRE_ROOT,$BOOST_ROOT,
$SPIDERMONKEY_ROOT all

produce error during during configuration, which you requested to look
at and I posted earlier, see

Checking for C library boost_filesystem-mt... no
Checking for C library boost_filesystem... no
can't find library ['boost_filesystem-mt', 'boost_filesystem'] in ['/
lib', '/build/vk/DAS/slc5_amd64_gcc434/external/boost/1.40.0/lib', '/
build/vk/DAS/slc5_amd64_gcc434/external/spidermonkey/1.8.0_rc1-cmp/
lib', '/usr/lib64', '/lib64']

and config.log output.

I said that if I apply my patch everything works just fine. So I got
the impression that you can look at this. Sorry for mis-understanding.
Bottom line that I still can't compile the code with SConstruct and
shared boost libraries.

> ...
>
> read more »

[Eliot Horowitz]

Can you do:
ls /build/vk/DAS/slc5_amd64_gcc434/external/boost/1.40.0/lib/

[vkuznet]

ls /build/vk/DAS/slc5_amd64_gcc434/external/boost/1.40.0/lib/
libboost_date_time.so libboost_program_options.so
libboost_date_time.so.1.40.0 libboost_program_options.so.
1.40.0
libboost_filesystem.so libboost_python.so
libboost_filesystem.so.1.40.0 libboost_python.so.1.40.0
libboost_graph.so libboost_regex.so
libboost_graph.so.1.40.0 libboost_regex.so.1.40.0
libboost_iostreams.so libboost_serialization.so
libboost_iostreams.so.1.40.0 libboost_serialization.so.1.40.0
libboost_math_c99f.so libboost_signals.so
libboost_math_c99f.so.1.40.0 libboost_signals.so.1.40.0
libboost_math_c99l.so libboost_system.so
libboost_math_c99l.so.1.40.0 libboost_system.so.1.40.0
libboost_math_c99.so libboost_thread.so
libboost_math_c99.so.1.40.0 libboost_thread.so.1.40.0
libboost_math_tr1f.so libboost_unit_test_framework.so
libboost_math_tr1f.so.1.40.0 libboost_unit_test_framework.so.
1.40.0
libboost_math_tr1l.so libboost_wave.so
libboost_math_tr1l.so.1.40.0 libboost_wave.so.1.40.0
libboost_math_tr1.so libboost_wserialization.so
libboost_math_tr1.so.1.40.0 libboost_wserialization.so.1.40.0
libboost_prg_exec_monitor.so libs
libboost_prg_exec_monitor.so.1.40.0 python2.6

> ...
>
> read more »

[vkuznet]

Elliot,
did you get chance to look at this? Upon your request I posted the
list of libs in my area and certainly I have all the required ones.
The problem is that the tests do not use all required shared libs
during the linking step. In the config.log I posted above the test
fail because of this

-lstdc++ -lboost_thread -lboost_filesystem

while it should do
-lstdc++ -lboost_thread -lboost_filesystem -lboost_system

May be better if I file a jira request?

> ...
>
> read more »

[Eliot Horowitz]

Just pushed a change that may fix.
I can't reproduce, so hard for me to test.

[vkuznet]

It works!!! I pull changes from the master and rebuilt mongo within
our environment. I checked with ldd that it does link to shared boost
libraries and once setup LD_LIBRARY_PATH I was able to run mongod.
Thank you very much.

> ...
>
> read more »