Compile errors with -O0.

[Kenneth Heafield]

Hi Jane,

   It's me again.  The problem I'm currently having is that you don't compile with optimizations turned off (-O0) so it's hard to run gdb (and therefore hard to explain why you're segfaulting).  

Steps to reproduce:

export CPPFLAGS="-O0 -g -I/home/zngu/sri/installed/include"
export LDFLAGS="-L/home/zngu/sri/installed/lib"
scons -j3 debug=1

Output:

scons: Reading SConscript files ...
Checking whether the C++ compiler works(cached) yes
Checking whether the C++ compiler works(cached) yes
Checking for C++ library oolm_c... (cached) yes
Checking for C++ library NumericalRecipes... (cached) no
Checking for C++ library cppunit... (cached) no
Not building #/bin/x86_64/optimizer.x86_64-standard because library
NumericalRecipes is missing.
Not building #/bin/x86_64/unitTests.x86_64-standard because library
cppunit is missing.
scons: done reading SConscript files.
scons: Building targets ...
scons: `options.bash' is up to date.
scons: `options.zsh' is up to date.
scons: `queueSettings.bash' is up to date.
scons: `queueSettings.zsh' is up to date.
build/x86_64-standard/Translation/libTranslation.a(RuleSet.o): In
function `Translation::RuleSet::getMan()':
/home/zngu/jane2.2-debug/src/Translation/RuleSet.cc:76: undefined
reference to `Core::param2Man(Core::AbstractParameter const&)'
/home/zngu/jane2.2-debug/src/Translation/RuleSet.cc:76: undefined
reference to `Core::param2Man(Core::AbstractParameter const&)'
/home/zngu/jane2.2-debug/src/Translation/RuleSet.cc:76: undefined
reference to `Core::param2Man(Core::AbstractParameter const&)'
/home/zngu/jane2.2-debug/src/Translation/RuleSet.cc:76: undefined
reference to `Core::param2Man(Core::AbstractParameter const&)'
/home/zngu/jane2.2-debug/src/Translation/RuleSet.cc:76: undefined
reference to `Core::param2Man(Core::AbstractParameter const&)'
build/x86_64-standard/Translation/libTranslation.a(RuleSet.o):/home/zngu/jane2.2-debug/src/Translation/RuleSet.cc:76:
more undefined references to `Core::param2Man(Core::AbstractParameter
const&)' follow
build/x86_64-standard/Translation/libTranslation.a(DependencyInformation.o):
In function
`Translation::DependencyRuleReader::readPlain(std::vector<std::basic_string<char,
std::char_traits<char>, std::allocator<char> >,
std::allocator<std::basic_string<char, std::char_traits<char>,
std::allocator<char> > > > const&) const':
/home/zngu/jane2.2-debug/src/Translation/DependencyInformation.cc:33:
undefined reference to `string2Dependency(std::basic_string<char,
std::char_traits<char>, std::allocator<char> > const&)'
/home/zngu/jane2.2-debug/src/Translation/DependencyInformation.cc:37:
undefined reference to `string2Dependency(std::basic_string<char,
std::char_traits<char>, std::allocator<char> > const&)'
build/x86_64-standard/Translation/libTranslation.a(DependencyInformation.o):
In function
`Translation::DependencyInformation::writePlain(std::basic_ostream<char,
std::char_traits<char> >&) const':
/home/zngu/jane2.2-debug/src/Translation/DependencyInformation.cc:54:
undefined reference to `dependency2String(int)'
/home/zngu/jane2.2-debug/src/Translation/DependencyInformation.cc:55:
undefined reference to `dependencyVector2String(std::vector<int,
std::allocator<int> > const&)'
collect2: ld returned 1 exit status
scons: *** [bin/x86_64/optimizeNbest.x86_64-standard] Error 1
scons: building terminated because of errors.

Hopefully, we can work out our issues before our dates in Sofia and Prague.    

Kenneth

[Kenneth Heafield]

Oh and, even without optimization, it doesn't compile on

g++ (Ubuntu/Linaro 4.6.3-1ubuntu5) 4.6.3
Copyright (C) 2011 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

because scons is providing the wrong linking order.

build/x86_64-standard/Core/libCore.a(CompressedStream.o): In function `Core::CompressedStreamBuf::underflow()':
CompressedStream.cc:(.text+0x7e): undefined reference to `gzread'
build/x86_64-standard/Core/libCore.a(CompressedStream.o): In function `Core::CompressedStreamBuf::open(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::_Ios_Openmode)':
CompressedStream.cc:(.text+0x135): undefined reference to `gzopen64'
build/x86_64-standard/Core/libCore.a(CompressedStream.o): In function `Core::CompressedStreamBuf::close()':
CompressedStream.cc:(.text+0x21a): undefined reference to `gzclose'
build/x86_64-standard/Core/libCore.a(CompressedStream.o): In function `Core::CompressedStreamBuf::flush_buffer()':
CompressedStream.cc:(.text+0x2c3): undefined reference to `gzwrite'
build/x86_64-standard/Syntax/libSyntax.a(SyntaxTreeEnjuXMLReader.o): In function `Extraction::SyntaxTreeEnjuXMLReader::readTree(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, SyntaxTreeNode*&, std::vector<SyntaxTreeNode*, std::allocator<SyntaxTreeNode*> >&, std::vector<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&, Core::Ref<Fsa::StaticAlphabet>)':
SyntaxTreeEnjuXMLReader.cc:(.text+0xf7): undefined reference to `xmlParseMemory'
SyntaxTreeEnjuXMLReader.cc:(.text+0x102): undefined reference to `xmlDocGetRootElement'
SyntaxTreeEnjuXMLReader.cc:(.text+0x110): undefined reference to `xmlStrcmp'
SyntaxTreeEnjuXMLReader.cc:(.text+0x12c): undefined reference to `xmlFreeDoc'
build/x86_64-standard/KenLM/libKenLM.a(file_piece.o): In function `util::GZException::GZException(void*)':
file_piece.cc:(.text+0xbe): undefined reference to `gzerror'
build/x86_64-standard/KenLM/libKenLM.a(file_piece.o): In function `util::FilePiece::~FilePiece()':
file_piece.cc:(.text+0x164): undefined reference to `gzclose'
build/x86_64-standard/KenLM/libKenLM.a(file_piece.o): In function `util::FilePiece::TransitionToRead()':
file_piece.cc:(.text+0x2e6): undefined reference to `gzdopen'
build/x86_64-standard/KenLM/libKenLM.a(file_piece.o): In function `util::FilePiece::ReadShift()':
file_piece.cc:(.text+0x64a): undefined reference to `gzread'

collect2: ld returned 1 exit status

scons: *** [bin/jane.x86_64-standard] Error 1

scons: building terminated because of errors.

Specifically, it's running

g++ -o bin/jane.x86_64-standard -fopenmp build/x86_64-standard/Jane/RunMode.o build/x86_64-standard/Jane/Socket.o build/x86_64-standard/Jane/Jane.o build/x86_64-standard/Jane/SingleBestRunMode.o build/x86_64-standard/Jane/NBestRunMode.o build/x86_64-standard/Jane/OptimizationServerRunMode.o build/x86_64-standard/Jane/ForcedAlignmentRunMode.o -L/raid0/heafield/sri-jane-install/lib -Lbuild/x86_64-standard/Core -Lsrc/Core -Lbuild/x86_64-standard/Translation -Lsrc/Translation -Lbuild/x86_64-standard/ForcedAlignment -Lsrc/ForcedAlignment -Lbuild/x86_64-standard/Syntax -Lsrc/Syntax -Lbuild/x86_64-standard/Training -Lsrc/Training -Lbuild/x86_64-standard/RandLM -Lsrc/RandLM -Lbuild/x86_64-standard/KenLM -Lsrc/KenLM -lz -lxml2 -lz -lm -lpthread -lCore -lTranslation -lForcedAlignment -lSyntax -lRandLM -lCore -lKenLM -loolm_c -ldstruct_c -lflm_c -lmisc_c

when it should be running

g++ -o bin/jane.x86_64-standard -fopenmp build/x86_64-standard/Jane/RunMode.o build/x86_64-standard/Jane/Socket.o build/x86_64-standard/Jane/Jane.o build/x86_64-standard/Jane/SingleBestRunMode.o build/x86_64-standard/Jane/NBestRunMode.o build/x86_64-standard/Jane/OptimizationServerRunMode.o build/x86_64-standard/Jane/ForcedAlignmentRunMode.o -L/raid0/heafield/sri-jane-install/lib -Lbuild/x86_64-standard/Core -Lsrc/Core -Lbuild/x86_64-standard/Translation -Lsrc/Translation -Lbuild/x86_64-standard/ForcedAlignment -Lsrc/ForcedAlignment -Lbuild/x86_64-standard/Syntax -Lsrc/Syntax -Lbuild/x86_64-standard/Training -Lsrc/Training -Lbuild/x86_64-standard/RandLM -Lsrc/RandLM -Lbuild/x86_64-standard/KenLM -Lsrc/KenLM  -lm -lpthread -lCore -lTranslation -lForcedAlignment -lSyntax -lRandLM -lCore -lKenLM -loolm_c -ldstruct_c -lflm_c -lmisc_c -lxml2 -lz

[Jan]

Hi,

this problem seems to appear with GCC 4.5, I don't know why it works with GCC 4.4. Since the compiler doesn't care in how often the static libraries are presented you can avoid this problem by simply passing them twice.

Change line 220 in the SConstruct file from:

   env.Program(target, sources, LIBS = env["LIBS"] + additionalLibs)

to:

  env.Program(target, sources, LIBS = env["LIBS"] + additionalLibs + env["LIBS"])

[Kenneth Heafield]

Progress.  That fixed the second post (normal compilation with newer gcc) but not the first post (does not compile with -O0). 

[Jan-Thorsten Peter]

Hi, I can't reproduce the problem. Jane compiles fine for me on Ubuntu 12.04 with gcc 4.6.3. Try to run:

scons -c

scons OPTIMIZE=no

or 

scons -c

scons COMPILE=debug

Jan

[Kenneth Heafield]

Still fails. 

g++ (GCC) 4.6.1

Copyright (C) 2011 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

I did a clean et al then
[zngu@trestles-login2 jane2.2-debug]$ scons OPTIMIZE=no VERBOSE=full

scons: Reading SConscript files ...

Compile without optimization (O0)

Checking whether the C++ compiler works(cached) yes
Checking whether the C++ compiler works(cached) yes
Checking for C++ library oolm_c... (cached) yes
Checking for C++ library NumericalRecipes... (cached) no
Checking for C++ library cppunit... (cached) no
Not building #/bin/x86_64/optimizer.x86_64-standard because library NumericalRecipes is missing.
Not building #/bin/x86_64/unitTests.x86_64-standard because library cppunit is missing.
scons: done reading SConscript files.
scons: Building targets ...

echo unknown, could not run git > bin/gitId

scons: `options.bash' is up to date.
scons: `options.zsh' is up to date.
scons: `queueSettings.bash' is up to date.
scons: `queueSettings.zsh' is up to date.

scons: `jane' is up to date.
g++ -o bin/x86_64/optimizeNbest.x86_64-standard -fopenmp build/x86_64-standard/Optimization/OptimizeNBest.o build/x86_64-standard/Optimization/DownhillSimplexMethod.o build/x86_64-standard/Optimization/Spsa.o build/x86_64-standard/Optimization/SecondOrder.o build/x86_64-standard/Optimization/NBestScorer.o build/x86_64-standard/Optimization/ObjectiveFunction.o build/x86_64-standard/Optimization/ScriptObjectiveFunction.o build/x86_64-standard/Optimization/ReferenceObjectiveFunction.o build/x86_64-standard/Optimization/NBestProcessor.o build/x86_64-standard/Optimization/Mert.o build/x86_64-standard/Optimization/Mira.o build/x86_64-standard/Optimization/Pro.o build/x86_64-standard/Optimization/BleuScorer.o build/x86_64-standard/Optimization/TERScorer.o build/x86_64-standard/Optimization/BigramFMeasure.o build/x86_64-standard/Optimization/ExternScorer.o -L/home/zngu/sri/installed/lib -L/home/zngu/prefix/lib64 -Lbuild/x86_64-standard/Core -Lsrc/Core -Lbuild/x86_64-standard/Translation -Lsrc/Translation -Lbuild/x86_64-standard/ForcedAlignment -Lsrc/ForcedAlignment -Lbuild/x86_64-standard/Syntax -Lsrc/Syntax -Lbuild/x86_64-standard/Training -Lsrc/Training -Lbuild/x86_64-standard/RandLM -Lsrc/RandLM -Lbuild/x86_64-standard/KenLM -Lsrc/KenLM -lxml2 -lz -lm -lpthread -lCore -lTranslation -lxml2 -lz -lm -lpthread -lCore

build/x86_64-standard/Translation/libTranslation.a(DependencyInformation.o): In function `Translation::DependencyRuleReader::readPlain(std::vector<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) const':
/home/zngu/jane2.2-debug/src/Translation/DependencyInformation.cc:33: undefined reference to `string2Dependency(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)'
/home/zngu/jane2.2-debug/src/Translation/DependencyInformation.cc:37: undefined reference to `string2Dependency(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)'
build/x86_64-standard/Translation/libTranslation.a(DependencyInformation.o): In function `Translation::DependencyInformation::writePlain(std::basic_ostream<char, std::char_traits<char> >&) const':
/home/zngu/jane2.2-debug/src/Translation/DependencyInformation.cc:54: undefined reference to `dependency2String(int)'
/home/zngu/jane2.2-debug/src/Translation/DependencyInformation.cc:55: undefined reference to `dependencyVector2String(std::vector<int, std::allocator<int> > const&)'
collect2: ld returned 1 exit status
scons: *** [bin/x86_64/optimizeNbest.x86_64-standard] Error 1
scons: building terminated because of errors.

If I run that command manually indeed the error happens:
 g++ -o bin/x86_64/optimizeNbest.x86_64-standard -fopenmp build/x86_64-standard/Optimization/OptimizeNBest.o build/x86_64-standard/Optimization/DownhillSimplexMethod.o build/x86_64-standard/Optimization/Spsa.o build/x86_64-standard/Optimization/SecondOrder.o build/x86_64-standard/Optimization/NBestScorer.o build/x86_64-standard/Optimization/ObjectiveFunction.o build/x86_64-standard/Optimization/ScriptObjectiveFunction.o build/x86_64-standard/Optimization/ReferenceObjectiveFunction.o build/x86_64-standard/Optimization/NBestProcessor.o build/x86_64-standard/Optimization/Mert.o build/x86_64-standard/Optimization/Mira.o build/x86_64-standard/Optimization/Pro.o build/x86_64-standard/Optimization/BleuScorer.o build/x86_64-standard/Optimization/TERScorer.o build/x86_64-standard/Optimization/BigramFMeasure.o build/x86_64-standard/Optimization/ExternScorer.o -L/home/zngu/sri/installed/lib -L/home/zngu/prefix/lib64 -Lbuild/x86_64-standard/Core -Lsrc/Core -Lbuild/x86_64-standard/Translation -Lsrc/Translation -Lbuild/x86_64-standard/ForcedAlignment -Lsrc/ForcedAlignment -Lbuild/x86_64-standard/Syntax -Lsrc/Syntax -Lbuild/x86_64-standard/Training -Lsrc/Training -Lbuild/x86_64-standard/RandLM -Lsrc/RandLM -Lbuild/x86_64-standard/KenLM -Lsrc/KenLM -lxml2 -lz -lm -lpthread -lCore -lTranslation -lCore -lxml2 -lz -lm -lpthread -lCore

build/x86_64-standard/Translation/libTranslation.a(DependencyInformation.o): In function `Translation::DependencyRuleReader::readPlain(std::vector<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) const':
/home/zngu/jane2.2-debug/src/Translation/DependencyInformation.cc:33: undefined reference to `string2Dependency(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)'
/home/zngu/jane2.2-debug/src/Translation/DependencyInformation.cc:37: undefined reference to `string2Dependency(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)'
build/x86_64-standard/Translation/libTranslation.a(DependencyInformation.o): In function `Translation::DependencyInformation::writePlain(std::basic_ostream<char, std::char_traits<char> >&) const':
/home/zngu/jane2.2-debug/src/Translation/DependencyInformation.cc:54: undefined reference to `dependency2String(int)'
/home/zngu/jane2.2-debug/src/Translation/DependencyInformation.cc:55: undefined reference to `dependencyVector2String(std::vector<int, std::allocator<int> > const&)'
collect2: ld returned 1 exit status

If I fix up your linkage of "-lCore -lTranslation -lCore" and turn it into the more sane "-lTranslation -lCore" then it works:

g++ -o bin/x86_64/optimizeNbest.x86_64-standard -fopenmp build/x86_64-standard/Optimization/OptimizeNBest.o build/x86_64-standard/Optimization/DownhillSimplexMethod.o build/x86_64-standard/Optimization/Spsa.o build/x86_64-standard/Optimization/SecondOrder.o build/x86_64-standard/Optimization/NBestScorer.o build/x86_64-standard/Optimization/ObjectiveFunction.o build/x86_64-standard/Optimization/ScriptObjectiveFunction.o build/x86_64-standard/Optimization/ReferenceObjectiveFunction.o build/x86_64-standard/Optimization/NBestProcessor.o build/x86_64-standard/Optimization/Mert.o build/x86_64-standard/Optimization/Mira.o build/x86_64-standard/Optimization/Pro.o build/x86_64-standard/Optimization/BleuScorer.o build/x86_64-standard/Optimization/TERScorer.o build/x86_64-standard/Optimization/BigramFMeasure.o build/x86_64-standard/Optimization/ExternScorer.o -L/home/zngu/sri/installed/lib -L/home/zngu/prefix/lib64 -Lbuild/x86_64-standard/Core -Lsrc/Core -Lbuild/x86_64-standard/Translation -Lsrc/Translation -Lbuild/x86_64-standard/ForcedAlignment -Lsrc/ForcedAlignment -Lbuild/x86_64-standard/Syntax -Lsrc/Syntax -Lbuild/x86_64-standard/Training -Lsrc/Training -Lbuild/x86_64-standard/RandLM -Lsrc/RandLM -Lbuild/x86_64-standard/KenLM -Lsrc/KenLM -lxml2 -lz -lm -lpthread -lTranslation -lCore -lxml2 -lz -lm -lpthread

But the real is the segfault.  Talk to Matthias Huck. 

[Matthias Huck]

Hi Kenneth,

> But the real is the segfault. Talk to Matthias Huck.

This is just a guess, but could you please check whether the tilde
symbol ("~") is present anywhere in your data?

Cheers,
Matthias

[Kenneth Heafield]

Not in the Moses version of the phrase table nor in the input.  I converted it to Jane with https://github.com/kpu/misc/blob/master/misc/moses_to_jane_main.cc which may have bugs.  Also escaped #.  You can download all of it from http://www.statmt.org/jane/ . 

[Matthias Huck]

Hi Kenneth,

Your phrase table contains flawed entries. I found 2240 rules where the
tilde symbol (which is the standard indicator character for
non-terminals in Jane) does not appear equally often on source and
target side.

Here are the first five ones (lines 1097, 1099, 1101, 1159, 1161):

5.338647815230077 6.7204627645625825 4.658918069363252 4.6376210662586 -0.999896315728952 1 # X # " ( # X~0 # 1 1 1 1 1
1.5518107195993072 0.9005354999296487 3.962716479743729 5.589217946175199 -0.999896315728952 2 # X # " ( # " X~0 # 1 1 1 1 1
2.1663598320930646 1.5856969720523029 4.658918069363252 5.594090752230854 -0.999896315728952 2 # X # " ( # ' X~0 # 1 1 1 1 1
9.80626647239264 6.934669231923187 6.855381068994397 4.85317267622413 -0.999896315728952 3 # X # " ( X~0 ( # X~0 X~0 ( # 1 1 1 1 1
4.17369950645914 1.114738621822153 7.179734620506631 5.804769235804997 -0.999896315728952 4 # X # " ( X~0 ( # " X~0 X~0 ( # 1 1 1 1 1

I did not have a detailed look into your Moses-to-Jane phrase table
converter implementation. It seems very likely that the tool has some
small bug, though.

Cheers,
Matthias

[Kenneth Heafield]

Hi,

Ugg sorry.  Moses uses [X] notation and there were some stray [ characters. 

Kenneth