Compile errors with -O0.

133 views
Skip to first unread message

Kenneth Heafield

unread,
Jun 29, 2013, 1:00:10 PM6/29/13
to jane-...@googlegroups.com
Hi Jane,

It's me again. The problem I'm currently having is that you don't compile with optimizations turned off (-O0) so it's hard to run gdb (and therefore hard to explain why you're segfaulting).

Steps to reproduce: export CPPFLAGS="-O0 -g -I/home/zngu/sri/installed/include" export LDFLAGS="-L/home/zngu/sri/installed/lib" scons -j3 debug=1 Output: scons: Reading SConscript files ... Checking whether the C++ compiler works(cached) yes Checking whether the C++ compiler works(cached) yes Checking for C++ library oolm_c... (cached) yes Checking for C++ library NumericalRecipes... (cached) no Checking for C++ library cppunit... (cached) no Not building #/bin/x86_64/optimizer.x86_64-standard because library NumericalRecipes is missing. Not building #/bin/x86_64/unitTests.x86_64-standard because library cppunit is missing. scons: done reading SConscript files. scons: Building targets ... scons: `options.bash' is up to date. scons: `options.zsh' is up to date. scons: `queueSettings.bash' is up to date. scons: `queueSettings.zsh' is up to date. build/x86_64-standard/Translation/libTranslation.a(RuleSet.o): In function `Translation::RuleSet::getMan()': /home/zngu/jane2.2-debug/src/Translation/RuleSet.cc:76: undefined reference to `Core::param2Man(Core::AbstractParameter const&)' /home/zngu/jane2.2-debug/src/Translation/RuleSet.cc:76: undefined reference to `Core::param2Man(Core::AbstractParameter const&)' /home/zngu/jane2.2-debug/src/Translation/RuleSet.cc:76: undefined reference to `Core::param2Man(Core::AbstractParameter const&)' /home/zngu/jane2.2-debug/src/Translation/RuleSet.cc:76: undefined reference to `Core::param2Man(Core::AbstractParameter const&)' /home/zngu/jane2.2-debug/src/Translation/RuleSet.cc:76: undefined reference to `Core::param2Man(Core::AbstractParameter const&)' build/x86_64-standard/Translation/libTranslation.a(RuleSet.o):/home/zngu/jane2.2-debug/src/Translation/RuleSet.cc:76: more undefined references to `Core::param2Man(Core::AbstractParameter const&)' follow build/x86_64-standard/Translation/libTranslation.a(DependencyInformation.o): In function `Translation::DependencyRuleReader::readPlain(std::vector<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) const': /home/zngu/jane2.2-debug/src/Translation/DependencyInformation.cc:33: undefined reference to `string2Dependency(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)' /home/zngu/jane2.2-debug/src/Translation/DependencyInformation.cc:37: undefined reference to `string2Dependency(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)' build/x86_64-standard/Translation/libTranslation.a(DependencyInformation.o): In function `Translation::DependencyInformation::writePlain(std::basic_ostream<char, std::char_traits<char> >&) const': /home/zngu/jane2.2-debug/src/Translation/DependencyInformation.cc:54: undefined reference to `dependency2String(int)' /home/zngu/jane2.2-debug/src/Translation/DependencyInformation.cc:55: undefined reference to `dependencyVector2String(std::vector<int, std::allocator<int> > const&)' collect2: ld returned 1 exit status scons: *** [bin/x86_64/optimizeNbest.x86_64-standard] Error 1 scons: building terminated because of errors.

Hopefully, we can work out our issues before our dates in Sofia and Prague.

Kenneth

Kenneth Heafield

unread,
Jun 30, 2013, 9:35:02 AM6/30/13
to jane-...@googlegroups.com
Oh and, even without optimization, it doesn't compile on

g++ (Ubuntu/Linaro 4.6.3-1ubuntu5) 4.6.3
Copyright (C) 2011 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

because scons is providing the wrong linking order.

build/x86_64-standard/Core/libCore.a(CompressedStream.o): In function `Core::CompressedStreamBuf::underflow()':
CompressedStream.cc:(.text+0x7e): undefined reference to `gzread'
build/x86_64-standard/Core/libCore.a(CompressedStream.o): In function `Core::CompressedStreamBuf::open(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::_Ios_Openmode)':
CompressedStream.cc:(.text+0x135): undefined reference to `gzopen64'
build/x86_64-standard/Core/libCore.a(CompressedStream.o): In function `Core::CompressedStreamBuf::close()':
CompressedStream.cc:(.text+0x21a): undefined reference to `gzclose'
build/x86_64-standard/Core/libCore.a(CompressedStream.o): In function `Core::CompressedStreamBuf::flush_buffer()':
CompressedStream.cc:(.text+0x2c3): undefined reference to `gzwrite'
build/x86_64-standard/Syntax/libSyntax.a(SyntaxTreeEnjuXMLReader.o): In function `Extraction::SyntaxTreeEnjuXMLReader::readTree(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, SyntaxTreeNode*&, std::vector<SyntaxTreeNode*, std::allocator<SyntaxTreeNode*> >&, std::vector<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&, Core::Ref<Fsa::StaticAlphabet>)':
SyntaxTreeEnjuXMLReader.cc:(.text+0xf7): undefined reference to `xmlParseMemory'
SyntaxTreeEnjuXMLReader.cc:(.text+0x102): undefined reference to `xmlDocGetRootElement'
SyntaxTreeEnjuXMLReader.cc:(.text+0x110): undefined reference to `xmlStrcmp'
SyntaxTreeEnjuXMLReader.cc:(.text+0x12c): undefined reference to `xmlFreeDoc'
build/x86_64-standard/KenLM/libKenLM.a(file_piece.o): In function `util::GZException::GZException(void*)':
file_piece.cc:(.text+0xbe): undefined reference to `gzerror'
build/x86_64-standard/KenLM/libKenLM.a(file_piece.o): In function `util::FilePiece::~FilePiece()':
file_piece.cc:(.text+0x164): undefined reference to `gzclose'
build/x86_64-standard/KenLM/libKenLM.a(file_piece.o): In function `util::FilePiece::TransitionToRead()':
file_piece.cc:(.text+0x2e6): undefined reference to `gzdopen'
build/x86_64-standard/KenLM/libKenLM.a(file_piece.o): In function `util::FilePiece::ReadShift()':
file_piece.cc:(.text+0x64a): undefined reference to `gzread'

collect2: ld returned 1 exit status
scons: *** [bin/jane.x86_64-standard] Error 1

scons: building terminated because of errors.

Specifically, it's running

g++ -o bin/jane.x86_64-standard -fopenmp build/x86_64-standard/Jane/RunMode.o build/x86_64-standard/Jane/Socket.o build/x86_64-standard/Jane/Jane.o build/x86_64-standard/Jane/SingleBestRunMode.o build/x86_64-standard/Jane/NBestRunMode.o build/x86_64-standard/Jane/OptimizationServerRunMode.o build/x86_64-standard/Jane/ForcedAlignmentRunMode.o -L/raid0/heafield/sri-jane-install/lib -Lbuild/x86_64-standard/Core -Lsrc/Core -Lbuild/x86_64-standard/Translation -Lsrc/Translation -Lbuild/x86_64-standard/ForcedAlignment -Lsrc/ForcedAlignment -Lbuild/x86_64-standard/Syntax -Lsrc/Syntax -Lbuild/x86_64-standard/Training -Lsrc/Training -Lbuild/x86_64-standard/RandLM -Lsrc/RandLM -Lbuild/x86_64-standard/KenLM -Lsrc/KenLM -lz -lxml2 -lz -lm -lpthread -lCore -lTranslation -lForcedAlignment -lSyntax -lRandLM -lCore -lKenLM -loolm_c -ldstruct_c -lflm_c -lmisc_c

when it should be running

g++ -o bin/jane.x86_64-standard -fopenmp build/x86_64-standard/Jane/RunMode.o build/x86_64-standard/Jane/Socket.o build/x86_64-standard/Jane/Jane.o build/x86_64-standard/Jane/SingleBestRunMode.o build/x86_64-standard/Jane/NBestRunMode.o build/x86_64-standard/Jane/OptimizationServerRunMode.o build/x86_64-standard/Jane/ForcedAlignmentRunMode.o -L/raid0/heafield/sri-jane-install/lib -Lbuild/x86_64-standard/Core -Lsrc/Core -Lbuild/x86_64-standard/Translation -Lsrc/Translation -Lbuild/x86_64-standard/ForcedAlignment -Lsrc/ForcedAlignment -Lbuild/x86_64-standard/Syntax -Lsrc/Syntax -Lbuild/x86_64-standard/Training -Lsrc/Training -Lbuild/x86_64-standard/RandLM -Lsrc/RandLM -Lbuild/x86_64-standard/KenLM -Lsrc/KenLM  -lm -lpthread -lCore -lTranslation -lForcedAlignment -lSyntax -lRandLM -lCore -lKenLM -loolm_c -ldstruct_c -lflm_c -lmisc_c -lxml2 -lz

Jan

unread,
Jul 1, 2013, 5:16:37 AM7/1/13
to jane-...@googlegroups.com
Hi,
this problem seems to appear with GCC 4.5, I don't know why it works with GCC 4.4. Since the compiler doesn't care in how often the static libraries are presented you can avoid this problem by simply passing them twice.
Change line 220 in the SConstruct file from:
   env.Program(target, sources, LIBS = env["LIBS"] + additionalLibs)
to:
  env.Program(target, sources, LIBS = env["LIBS"] + additionalLibs + env["LIBS"])

Kenneth Heafield

unread,
Jul 3, 2013, 7:17:57 AM7/3/13
to jane-...@googlegroups.com
Progress.  That fixed the second post (normal compilation with newer gcc) but not the first post (does not compile with -O0). 

Jan-Thorsten Peter

unread,
Jul 3, 2013, 10:57:24 AM7/3/13
to jane-...@googlegroups.com
Hi, I can't reproduce the problem. Jane compiles fine for me on Ubuntu 12.04 with gcc 4.6.3. Try to run:

scons -c
scons OPTIMIZE=no

or 

scons -c
scons COMPILE=debug

Jan

Kenneth Heafield

unread,
Jul 3, 2013, 11:50:22 AM7/3/13
to jane-...@googlegroups.com
Still fails. 

g++ (GCC) 4.6.1

Copyright (C) 2011 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

I did a clean et al then
[zngu@trestles-login2 jane2.2-debug]$ scons OPTIMIZE=no VERBOSE=full

scons: Reading SConscript files ...
Compile without optimization (O0)

Checking whether the C++ compiler works(cached) yes
Checking whether the C++ compiler works(cached) yes
Checking for C++ library oolm_c... (cached) yes
Checking for C++ library NumericalRecipes... (cached) no
Checking for C++ library cppunit... (cached) no
Not building #/bin/x86_64/optimizer.x86_64-standard because library NumericalRecipes is missing.
Not building #/bin/x86_64/unitTests.x86_64-standard because library cppunit is missing.
scons: done reading SConscript files.
scons: Building targets ...
echo unknown, could not run git > bin/gitId

scons: `options.bash' is up to date.
scons: `options.zsh' is up to date.
scons: `queueSettings.bash' is up to date.
scons: `queueSettings.zsh' is up to date.
scons: `jane' is up to date.
g++ -o bin/x86_64/optimizeNbest.x86_64-standard -fopenmp build/x86_64-standard/Optimization/OptimizeNBest.o build/x86_64-standard/Optimization/DownhillSimplexMethod.o build/x86_64-standard/Optimization/Spsa.o build/x86_64-standard/Optimization/SecondOrder.o build/x86_64-standard/Optimization/NBestScorer.o build/x86_64-standard/Optimization/ObjectiveFunction.o build/x86_64-standard/Optimization/ScriptObjectiveFunction.o build/x86_64-standard/Optimization/ReferenceObjectiveFunction.o build/x86_64-standard/Optimization/NBestProcessor.o build/x86_64-standard/Optimization/Mert.o build/x86_64-standard/Optimization/Mira.o build/x86_64-standard/Optimization/Pro.o build/x86_64-standard/Optimization/BleuScorer.o build/x86_64-standard/Optimization/TERScorer.o build/x86_64-standard/Optimization/BigramFMeasure.o build/x86_64-standard/Optimization/ExternScorer.o -L/home/zngu/sri/installed/lib -L/home/zngu/prefix/lib64 -Lbuild/x86_64-standard/Core -Lsrc/Core -Lbuild/x86_64-standard/Translation -Lsrc/Translation -Lbuild/x86_64-standard/ForcedAlignment -Lsrc/ForcedAlignment -Lbuild/x86_64-standard/Syntax -Lsrc/Syntax -Lbuild/x86_64-standard/Training -Lsrc/Training -Lbuild/x86_64-standard/RandLM -Lsrc/RandLM -Lbuild/x86_64-standard/KenLM -Lsrc/KenLM -lxml2 -lz -lm -lpthread -lCore -lTranslation -lxml2 -lz -lm -lpthread -lCore

build/x86_64-standard/Translation/libTranslation.a(DependencyInformation.o): In function `Translation::DependencyRuleReader::readPlain(std::vector<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) const':
/home/zngu/jane2.2-debug/src/Translation/DependencyInformation.cc:33: undefined reference to `string2Dependency(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)'
/home/zngu/jane2.2-debug/src/Translation/DependencyInformation.cc:37: undefined reference to `string2Dependency(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)'
build/x86_64-standard/Translation/libTranslation.a(DependencyInformation.o): In function `Translation::DependencyInformation::writePlain(std::basic_ostream<char, std::char_traits<char> >&) const':
/home/zngu/jane2.2-debug/src/Translation/DependencyInformation.cc:54: undefined reference to `dependency2String(int)'
/home/zngu/jane2.2-debug/src/Translation/DependencyInformation.cc:55: undefined reference to `dependencyVector2String(std::vector<int, std::allocator<int> > const&)'
collect2: ld returned 1 exit status
scons: *** [bin/x86_64/optimizeNbest.x86_64-standard] Error 1
scons: building terminated because of errors.

If I run that command manually indeed the error happens:
 g++ -o bin/x86_64/optimizeNbest.x86_64-standard -fopenmp build/x86_64-standard/Optimization/OptimizeNBest.o build/x86_64-standard/Optimization/DownhillSimplexMethod.o build/x86_64-standard/Optimization/Spsa.o build/x86_64-standard/Optimization/SecondOrder.o build/x86_64-standard/Optimization/NBestScorer.o build/x86_64-standard/Optimization/ObjectiveFunction.o build/x86_64-standard/Optimization/ScriptObjectiveFunction.o build/x86_64-standard/Optimization/ReferenceObjectiveFunction.o build/x86_64-standard/Optimization/NBestProcessor.o build/x86_64-standard/Optimization/Mert.o build/x86_64-standard/Optimization/Mira.o build/x86_64-standard/Optimization/Pro.o build/x86_64-standard/Optimization/BleuScorer.o build/x86_64-standard/Optimization/TERScorer.o build/x86_64-standard/Optimization/BigramFMeasure.o build/x86_64-standard/Optimization/ExternScorer.o -L/home/zngu/sri/installed/lib -L/home/zngu/prefix/lib64 -Lbuild/x86_64-standard/Core -Lsrc/Core -Lbuild/x86_64-standard/Translation -Lsrc/Translation -Lbuild/x86_64-standard/ForcedAlignment -Lsrc/ForcedAlignment -Lbuild/x86_64-standard/Syntax -Lsrc/Syntax -Lbuild/x86_64-standard/Training -Lsrc/Training -Lbuild/x86_64-standard/RandLM -Lsrc/RandLM -Lbuild/x86_64-standard/KenLM -Lsrc/KenLM -lxml2 -lz -lm -lpthread -lCore -lTranslation -lCore -lxml2 -lz -lm -lpthread -lCore

build/x86_64-standard/Translation/libTranslation.a(DependencyInformation.o): In function `Translation::DependencyRuleReader::readPlain(std::vector<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) const':
/home/zngu/jane2.2-debug/src/Translation/DependencyInformation.cc:33: undefined reference to `string2Dependency(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)'
/home/zngu/jane2.2-debug/src/Translation/DependencyInformation.cc:37: undefined reference to `string2Dependency(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)'
build/x86_64-standard/Translation/libTranslation.a(DependencyInformation.o): In function `Translation::DependencyInformation::writePlain(std::basic_ostream<char, std::char_traits<char> >&) const':
/home/zngu/jane2.2-debug/src/Translation/DependencyInformation.cc:54: undefined reference to `dependency2String(int)'
/home/zngu/jane2.2-debug/src/Translation/DependencyInformation.cc:55: undefined reference to `dependencyVector2String(std::vector<int, std::allocator<int> > const&)'
collect2: ld returned 1 exit status

If I fix up your linkage of "-lCore -lTranslation -lCore" and turn it into the more sane "-lTranslation -lCore" then it works:

g++ -o bin/x86_64/optimizeNbest.x86_64-standard -fopenmp build/x86_64-standard/Optimization/OptimizeNBest.o build/x86_64-standard/Optimization/DownhillSimplexMethod.o build/x86_64-standard/Optimization/Spsa.o build/x86_64-standard/Optimization/SecondOrder.o build/x86_64-standard/Optimization/NBestScorer.o build/x86_64-standard/Optimization/ObjectiveFunction.o build/x86_64-standard/Optimization/ScriptObjectiveFunction.o build/x86_64-standard/Optimization/ReferenceObjectiveFunction.o build/x86_64-standard/Optimization/NBestProcessor.o build/x86_64-standard/Optimization/Mert.o build/x86_64-standard/Optimization/Mira.o build/x86_64-standard/Optimization/Pro.o build/x86_64-standard/Optimization/BleuScorer.o build/x86_64-standard/Optimization/TERScorer.o build/x86_64-standard/Optimization/BigramFMeasure.o build/x86_64-standard/Optimization/ExternScorer.o -L/home/zngu/sri/installed/lib -L/home/zngu/prefix/lib64 -Lbuild/x86_64-standard/Core -Lsrc/Core -Lbuild/x86_64-standard/Translation -Lsrc/Translation -Lbuild/x86_64-standard/ForcedAlignment -Lsrc/ForcedAlignment -Lbuild/x86_64-standard/Syntax -Lsrc/Syntax -Lbuild/x86_64-standard/Training -Lsrc/Training -Lbuild/x86_64-standard/RandLM -Lsrc/RandLM -Lbuild/x86_64-standard/KenLM -Lsrc/KenLM -lxml2 -lz -lm -lpthread -lTranslation -lCore -lxml2 -lz -lm -lpthread

But the real is the segfault.  Talk to Matthias Huck. 

Matthias Huck

unread,
Jul 3, 2013, 12:14:54 PM7/3/13
to jane-...@googlegroups.com
Hi Kenneth,

> But the real is the segfault. Talk to Matthias Huck.

This is just a guess, but could you please check whether the tilde
symbol ("~") is present anywhere in your data?

Cheers,
Matthias



Kenneth Heafield

unread,
Jul 3, 2013, 4:30:37 PM7/3/13
to jane-...@googlegroups.com
Not in the Moses version of the phrase table nor in the input.  I converted it to Jane with https://github.com/kpu/misc/blob/master/misc/moses_to_jane_main.cc which may have bugs.  Also escaped #.  You can download all of it from http://www.statmt.org/jane/

Matthias Huck

unread,
Jul 4, 2013, 1:27:39 PM7/4/13
to jane-...@googlegroups.com
Hi Kenneth,

Your phrase table contains flawed entries. I found 2240 rules where the
tilde symbol (which is the standard indicator character for
non-terminals in Jane) does not appear equally often on source and
target side.

Here are the first five ones (lines 1097, 1099, 1101, 1159, 1161):

5.338647815230077 6.7204627645625825 4.658918069363252 4.6376210662586 -0.999896315728952 1 # X # " ( # X~0 # 1 1 1 1 1
1.5518107195993072 0.9005354999296487 3.962716479743729 5.589217946175199 -0.999896315728952 2 # X # " ( # " X~0 # 1 1 1 1 1
2.1663598320930646 1.5856969720523029 4.658918069363252 5.594090752230854 -0.999896315728952 2 # X # " ( # ' X~0 # 1 1 1 1 1
9.80626647239264 6.934669231923187 6.855381068994397 4.85317267622413 -0.999896315728952 3 # X # " ( X~0 ( # X~0 X~0 ( # 1 1 1 1 1
4.17369950645914 1.114738621822153 7.179734620506631 5.804769235804997 -0.999896315728952 4 # X # " ( X~0 ( # " X~0 X~0 ( # 1 1 1 1 1

I did not have a detailed look into your Moses-to-Jane phrase table
converter implementation. It seems very likely that the tool has some
small bug, though.

Cheers,
Matthias

Kenneth Heafield

unread,
Jul 6, 2013, 5:09:29 AM7/6/13
to jane-...@googlegroups.com
Hi,

Ugg sorry.  Moses uses [X] notation and there were some stray [ characters. 

Kenneth
Reply all
Reply to author
Forward
0 new messages