Issues building cdec
Randy West
I'm having some trouble building cdec on Debian Lenny 2.6.26-26
(x86_64). The issue starts with a "bug" with autoreconf reported in
various places, e.g. http://www.gnu.org/software/hello/manual/automake/Error-required-file-ltmain_002esh-not-found.html.
The issue cascades afterwards during make because libtool cannot be
found. I'm not very familiar with automake, so it may be that there's
a simple workaround to be had, but after several hours of looking and
guessing I decided it was time to ask.
Here are the last two lines of stderr from autoreconf -ifv:
configure.ac:4: installing `./config.guess'
configure.ac:4: installing `./config.sub'
configure.ac:4: required file `./ltmain.sh' not found
Makefile.am:4: directory should not contain `/'
decoder/Makefile.am: installing `./depcomp'
Makefile.am:4: directory should not contain `/'
autoreconf: automake failed with exit status: 1
The result is the same for autoreconf v2.61 and v2.68 (the latest).
I'm happy to provide any relevant details or full command output if
needed.
Thanks very much in advance.
Randy
Chris Dyer
Is this a recent github.com version of cdec? (The reason I ask is
there have been a couple of forks being merged around, and only
recently did things get unified back at github.com). I'm not sure what
this error indicates-- I haven't encountered it. Is there anything
unusual about the file system where you're trying to make cdec? (i.e.,
inside of a symlink or something?)
-Chris
Randy West
Thanks for your response, and sorry for the delay in mine. I was using
the HEAD version of cdec from github.com as of 12/15 (commit
491848839c0340f6c629ebae7ed6b6dc1a3842ad), and the issue persists with
the most recent version as of this writing. We are using a network file
system, so most of root is aliased to /net/<file server name>/, but I
don't *think* that would cause this issue.
My suspicion is that this is a problem somewhere in the
autoconf/automake/libtool tool chain. The /usr/bin/ versions on the
machine are:
autoconf: 2.61
automake: 1.7.9
libtool: 1.5.26
As I was able to use autoreconf successfully on my home computer (but
not make just yet due apparently to boost issues), I thought I would
match up the shared machine versions with my home versions. These are
now at:
autoconf: 2.65
automake: 1.11.1
libtool: 2.6.6b
Unfortunately, using these does not seem to change the behavior. Do you
have any other possible avenues of investigation? It would also be
interesting to see which versions of automake/boost/etc. people are
using successfully.
Thanks,
Randy
Chris Dyer
likely the problem. I think the boost m4 macros rely quite heavily on
libtool, and it looks like the one you've got installed may be
somewhat out of date, so perhaps upgrading that will work. I've built
cdec using an autotools package that was installed in a non-standard
location, so it should be possible to get new versions installed even
if you don't have root.
Randy West
after these things get sorted out), but installing the latest version of
the autotools packages in my home directory worked just fine through the
configure step. Thanks for helping me through that.
However, I'm now having issues at the make stage that I imagine might
similarly stem from a versioning issue. I'm using srilm 1.5.11, which I
believe is the latest stable version, but it seems to be causing some
issues. I've included the last bit of the make output below for your
reference. Before slogging through the specifics of the errors though,
could you please recommend boost/srilm versions known to work with the
latest version of cdec?
Thanks,
Randy
Tail of make output. Note that I have full permissions on
/local/contrib/cpl dirs under my ownership and read only otherwise. I
own /local/contrib/cpl/cdec but not /local/contrib/cpl/srilm-1.5.11:
g++ -DHAVE_CONFIG_H -I. -I.. -W -Wall -Wno-sign-compare -I../utils
-pthread -I/local/contrib/cpl/srilm-1.5.11//include -g -O2 -MT
fast_score.o -MD -MP -MF .deps/fast_score.Tpo -c -o fast_score.o
fast_score.cc
mv -f .deps/fast_score.Tpo .deps/fast_score.Po
/bin/sh ../libtool --tag=CXX --mode=link g++ -g -O2 -L/usr/local/lib
-R/usr/local/lib -L/usr/local/lib -R/usr/local/lib
-L/local/contrib/cpl/srilm-1.5.11//lib/i686-m64 -o fast_score
fast_score.o libmteval.a ../utils/libutils.a -lz
-lboost_program_options-mt -lboost_thread-mt -pthread -loolm -ldstruct
-lmisc
libtool: link: g++ -g -O2 -o fast_score fast_score.o -pthread
-L/usr/local/lib -L/local/contrib/cpl/srilm-1.5.11//lib/i686-m64
libmteval.a ../utils/libutils.a -lz -lboost_program_options-mt
-lboost_thread-mt -loolm -ldstruct -lmisc -pthread -Wl,-rpath
-Wl,/usr/local/lib
/local/contrib/cpl/srilm-1.5.11//lib/i686-m64/liboolm.a(Vocab.o): In
function `LHash<char const*, unsigned int>::dump() const':
Vocab.cc:(.text._ZNK5LHashIPKcjE4dumpEv[LHash<char const*, unsigned
int>::dump() const]+0x121): undefined reference to
`std::ctype<char>::_M_widen_init() const'
/local/contrib/cpl/srilm-1.5.11//lib/i686-m64/liboolm.a(LMStats.o): In
function `LHash<unsigned int, unsigned int>::dump() const':
LMStats.cc:(.text._ZNK5LHashIjjE4dumpEv[LHash<unsigned int, unsigned
int>::dump() const]+0x111): undefined reference to
`std::ctype<char>::_M_widen_init() const'
/local/contrib/cpl/srilm-1.5.11//lib/i686-m64/liboolm.a(TextStats.o): In
function `std::ctype<char>::widen(char) const':
/net/truffle/local/contrib/cpl/gcc4.4.3/bin/../lib/gcc/x86_64-unknown-linux-gnu/4.4.3/../../../../include/c++/4.4.3/bits/locale_facets.h:869:
undefined reference to `std::ctype<char>::_M_widen_init() const'
/net/truffle/local/contrib/cpl/gcc4.4.3/bin/../lib/gcc/x86_64-unknown-linux-gnu/4.4.3/../../../../include/c++/4.4.3/bits/locale_facets.h:869:
undefined reference to `std::ctype<char>::_M_widen_init() const'
collect2: ld returned 1 exit status
make[2]: *** [fast_score] Error 1
make[2]: Leaving directory `/net/truffle/local/contrib/cpl/cdec/mteval'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/net/truffle/local/contrib/cpl/cdec'
make: *** [all] Error 2
Chris Dyer
different version of g++ or one of the standard libraries or on
another machine?
Randy West
seemed to clear up the error that I was referring to previously.
Unfortunately, I'm now having issues with boost. When I don't specify
the --with-boost flag to configure, it successfully creates a Makefile,
but then I start getting errors on make during the decoder build process:
Making all in decoder
make[2]: Entering directory `/net/truffle/local/contrib/cpl/cdec/decoder'
g++ -DHAVE_CONFIG_H -I. -I.. -W -Wall -Wno-sign-compare -I.. -I../mteval
-I../utils -I../klm -pthread -I/home/rdwest/programs/srilm/include
-g -O2 -MT cdec_ff.o -MD -MP -MF .deps/cdec_ff.Tpo -c -o cdec_ff.o
cdec_ff.cc
In file included from cdec_ff.cc:15:
ff_wordset.h:6:45: error: boost/unordered/unordered_set.hpp: No such
file or directory
In file included from ff_fsa_data.h:7,
from ff_fsa_dynamic.h:6,
from ff_factory.h:23,
from cdec_ff.cc:9:
feature_accum.h:38: warning: unused parameter �fids�
In file included from ff_factory.h:23,
from cdec_ff.cc:9:
... (snip)
Thinking that perhaps this was similar to the srilm issue, I tried
building my own copy of boost (1.45) with standard options. Specifying
the new build directory via --with-boost, however, breaks configure as
follows:
checking for Boost headers version >= 0...
/home/rdwest/programs/boost_1_45_0
checking for Boost's header version... 1_45
checking for the toolset name used by Boost for g++... gcc43 -gcc
checking boost/program_options.hpp usability... yes
checking boost/program_options.hpp presence... yes
checking for boost/program_options.hpp... yes
checking for the Boost program_options library... no
configure: error: cannot not find the flags to link with Boost
program_options
I was unsure that I had built boost properly at first, but manually
specifying a different build (1.41) on the same file system yields the
same results.
Given all of this, do you have any further ideas? Perhaps I should be
building boost in a certain way to get configure to recognize it?
Thanks,
Randy
Chris Dyer
then not be able to find it when compiling. You might look in your
config.log to see what the g++ command looks like that succeeds when
looking for boost is.
Also, when building boost, you need to install it somewhere or the m4
macros have trouble using it. I think the command is "bjam install
--prefix=/somewhere". This is a bit unclear, and I've been bit by it
in the past.
-Chris
On Mon, Dec 27, 2010 at 11:32 PM, Randy West <randy...@gmail.com> wrote:
> It is possible. I tried building srilm in my home directory, and that seemed
> to clear up the error that I was referring to previously.
>
> Unfortunately, I'm now having issues with boost. When I don't specify the
> --with-boost flag to configure, it successfully creates a Makefile, but then
> I start getting errors on make during the decoder build process:
>
> Making all in decoder
> make[2]: Entering directory `/net/truffle/local/contrib/cpl/cdec/decoder'
> g++ -DHAVE_CONFIG_H -I. -I.. -W -Wall -Wno-sign-compare -I.. -I../mteval
> -I../utils -I../klm -pthread -I/home/rdwest/programs/srilm/include
> -g -O2 -MT cdec_ff.o -MD -MP -MF .deps/cdec_ff.Tpo -c -o cdec_ff.o
> cdec_ff.cc
> In file included from cdec_ff.cc:15:
> ff_wordset.h:6:45: error: boost/unordered/unordered_set.hpp: No such file or
> directory
> In file included from ff_fsa_data.h:7,
> from ff_fsa_dynamic.h:6,
> from ff_factory.h:23,
> from cdec_ff.cc:9:
> feature_accum.h:38: warning: unused parameter ‘fids’
Randy West
configure now works, and it seemed for a while that make was going
along it's merry way, until this:
g++ -DHAVE_CONFIG_H -I. -I../../.. -W -Wall -Wno-sign-compare
-funroll-loops -I../../../utils -I/home/rdwest/usr//include -pthread
-I/home/rdwest/programs/srilm/inc
lude -g -O2 -MT contexts_corpus.o -MD -MP -MF
.deps/contexts_corpus.Tpo -c -o contexts_corpus.o contexts_corpus.cc
mv -f .deps/contexts_corpus.Tpo .deps/contexts_corpus.Po
/bin/sh ../../../libtool --tag=CXX --mode=link g++ -g -O2
-L/home/rdwest/usr//lib -R/home/rdwest/usr//lib
-L/home/rdwest/usr//lib -R/home/rdwest/usr//lib -L/home
/rdwest/programs/srilm/lib/i686-m64 -o pyp-topics-train mt19937ar.o
corpus.o gzstream.o pyp-topics.o train.o contexts_lexer.o
contexts_corpus.o ../../../utils/libuti
ls.a -lz -lboost_program_options -lboost_thread-mt -pthread -loolm
-ldstruct -lmisc
libtool: link: g++ -g -O2 -o pyp-topics-train mt19937ar.o corpus.o
gzstream.o pyp-topics.o train.o contexts_lexer.o contexts_corpus.o
-pthread -L/home/rdwest/usr//lib
-L/home/rdwest/programs/srilm/lib/i686-m64 ../../../utils/libutils.a
-lz -lboost_program_options -lboost_thread-mt -loolm -ldstruct -lmisc
-pthread -Wl,-rpath -Wl,/home/rdwest/usr//lib
pyp-topics.o: In function
`boost::thread::thread<boost::packaged_task<long double>
>(boost::detail::thread_move_t<boost::packaged_task<long double> >)':
pyp-topics.cc:(.text._ZN5boost6threadC1INS_13packaged_taskIeEEEENS_6detail13thread_move_tIT_EE[boost::thread::thread<boost::packaged_task<long
double> >(boost::detail::thread_move_t<boost::packaged_task<long
double> >)]+0x4b): undefined reference to `vtable for
boost::detail::thread_data_base'
pyp-topics.cc:(.text._ZN5boost6threadC1INS_13packaged_taskIeEEEENS_6detail13thread_move_tIT_EE[boost::thread::thread<boost::packaged_task<long
double> >(boost::detail::thread_move_t<boost::packaged_task<long
double> >)]+0x1a9): undefined reference to
`boost::thread::start_thread()'
pyp-topics.o: In function
`boost::condition_variable::wait(boost::unique_lock<boost::mutex>&)':
pyp-topics.cc:(.text._ZN5boost18condition_variable4waitERNS_11unique_lockINS_5mutexEEE[boost::condition_variable::wait(boost::unique_lock<boost::mutex>&)]+0x2e):
undefined reference to `boost::detail::get_current_thread_data()'
pyp-topics.cc:(.text._ZN5boost18condition_variable4waitERNS_11unique_lockINS_5mutexEEE[boost::condition_variable::wait(boost::unique_lock<boost::mutex>&)]+0x99):
undefined reference to `boost::this_thread::interruption_point()'
pyp-topics.o: In function `boost::unique_lock<boost::shared_mutex>::lock()':
pyp-topics.cc:(.text._ZN5boost11unique_lockINS_12shared_mutexEE4lockEv[boost::unique_lock<boost::shared_mutex>::lock()]+0x2a):
undefined reference to
`boost::this_thread::disable_interruption::disable_interruption()'
pyp-topics.cc:(.text._ZN5boost11unique_lockINS_12shared_mutexEE4lockEv[boost::unique_lock<boost::shared_mutex>::lock()]+0x108):
undefined reference to `boost::detail::get_current_thread_data()'
pyp-topics.cc:(.text._ZN5boost11unique_lockINS_12shared_mutexEE4lockEv[boost::unique_lock<boost::shared_mutex>::lock()]+0x1cd):
undefined reference to `boost::this_thread::interruption_point()'
pyp-topics.cc:(.text._ZN5boost11unique_lockINS_12shared_mutexEE4lockEv[boost::unique_lock<boost::shared_mutex>::lock()]+0x25e):
undefined reference to
`boost::this_thread::disable_interruption::~disable_interruption()'
pyp-topics.cc:(.text._ZN5boost11unique_lockINS_12shared_mutexEE4lockEv[boost::unique_lock<boost::shared_mutex>::lock()]+0x348):
undefined reference to
`boost::this_thread::disable_interruption::~disable_interruption()'
pyp-topics.o: In function `Queuemt<boost::function<long double ()()> >::pop()':
pyp-topics.cc:(.text._ZN7QueuemtIN5boost8functionIFevEEEE3popEv[Queuemt<boost::function<long
double ()()> >::pop()]+0x71): undefined reference to
`boost::detail::get_current_thread_data()'
pyp-topics.cc:(.text._ZN7QueuemtIN5boost8functionIFevEEEE3popEv[Queuemt<boost::function<long
double ()()> >::pop()]+0x1d9): undefined reference to
`boost::this_thread::interruption_point()'
pyp-topics.o: In function
`boost::detail::thread_data<boost::packaged_task<long double>
>::~thread_data()':
pyp-topics.cc:(.text._ZN5boost6detail11thread_dataINS_13packaged_taskIeEEED0Ev[boost::detail::thread_data<boost::packaged_task<long
double> >::~thread_data()]+0x28): undefined reference to
`boost::detail::thread_data_base::~thread_data_base()'
pyp-topics.cc:(.text._ZN5boost6detail11thread_dataINS_13packaged_taskIeEEED0Ev[boost::detail::thread_data<boost::packaged_task<long
double> >::~thread_data()]+0x49): undefined reference to
`boost::detail::thread_data_base::~thread_data_base()'
pyp-topics.o: In function
`boost::detail::thread_data<boost::packaged_task<long double>
>::~thread_data()':
pyp-topics.cc:(.text._ZN5boost6detail11thread_dataINS_13packaged_taskIeEEED1Ev[boost::detail::thread_data<boost::packaged_task<long
double> >::~thread_data()]+0x41):
undefined reference to `boost::detail::thread_data_base::~thread_data_base()'
pyp-topics.cc:(.text._ZN5boost6detail11thread_dataINS_13packaged_taskIeEEED1Ev[boost::detail::thread_data<boost::packaged_task<long
double> >::~thread_data()]+0x36):
undefined reference to `boost::detail::thread_data_base::~thread_data_base()'
(snip)
collect2: ld returned 1 exit status
make[2]: *** [pyp-topics-train] Error 1
make[2]: Leaving directory
`/net/truffle/local/contrib/cpl/cdec/gi/pyp-topics/src'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/net/truffle/local/contrib/cpl/cdec'
make: *** [all] Error 2
So it would appear that the linker is having trouble with some
undefined boost references. Any ideas?
Thanks,
Randy
Randy West
someone who has successfully installed cdec could tell me exactly how
they compiled boost. I realize that many users are probably using
precompiled versions, but since I don't have root access to use the
package manager on my system, and moreover since the precompiled
version on my system is also generating errors, it would be a great
help if someone who has compiled boost manually could outline their
method in detail.
Thanks,
Randy
> double> >)]+0x4b): undefined reference to `vtable for
> boost::detail::thread_data_base'
> double> >)]+0x1a9): undefined reference to
> `boost::thread::start_thread()'
> pyp-topics.o: In function
> `boost::condition_variable::wait(boost::unique_lock<boost::mutex>&)':
> pyp-topics.o: In function `boost::unique_lock<boost::shared_mutex>::lock()':
> `boost::this_thread::disable_interruption::disable_interruption()'
> `boost::this_thread::disable_interruption::~disable_interruption()'
> `boost::this_thread::disable_interruption::~disable_interruption()'
> pyp-topics.o: In function `Queuemt<boost::function<long double ()()> >::pop()':
> `boost::detail::get_current_thread_data()'
> `boost::this_thread::interruption_point()'
> pyp-topics.o: In function
> `boost::detail::thread_data<boost::packaged_task<long double>>::~thread_data()':
>
> `boost::detail::thread_data_base::~thread_data_base()'
> `boost::detail::thread_data_base::~thread_data_base()'
> pyp-topics.o: In function
> `boost::detail::thread_data<boost::packaged_task<long double>>::~thread_data()':
>
> undefined reference to `boost::detail::thread_data_base::~thread_data_base()'
> undefined reference to `boost::detail::thread_data_base::~thread_data_base()'
> (snip)
> collect2: ld returned 1 exit status
> make[2]: *** [pyp-topics-train] Error 1
> make[2]: Leaving directory
> `/net/truffle/local/contrib/cpl/cdec/gi/pyp-topics/src'
> make[1]: *** [all-recursive] Error 1
> make[1]: Leaving directory `/net/truffle/local/contrib/cpl/cdec'
> make: *** [all] Error 2
>
> So it would appear that the linker is having trouble with some
> undefined boost references. Any ideas?
>
> Thanks,
> Randy
>
>
>
>
>
>
>
> On Tue, Dec 28, 2010 at 9:28 AM, Chris Dyer <cd...@cs.cmu.edu> wrote:
> > I'm not sure why it would detect boost when you're configuring but
> > then not be able to find it when compiling. You might look in your
> > config.log to see what the g++ command looks like that succeeds when
> > looking for boost is.
>
> > Also, when building boost, you need to install it somewhere or the m4
> > macros have trouble using it. I think the command is "bjam install
> > --prefix=/somewhere". This is a bit unclear, and I've been bit by it
> > in the past.
>
> > -Chris
>
>
> >>>> Well, I feel silly for not trying this right up front (don't we always
> >>>> after
> >>>> these things get sorted out), but installing the latest version of the
> >>>> autotools packages in my home directory worked just fine through the
> >>>> configure step. Thanks for helping me through that.
>
> >>>> However, I'm now having issues at the make stage that I imagine might
> >>>> similarly stem from a versioning issue. I'm using srilm 1.5.11, which I
> >>>> believe is the latest stable version, but it seems to be causing some
> >>>> issues. I've included the last bit of the make output below for your
> >>>> reference. Before slogging through the specifics of the errors though,
> >>>> could
> >>>> you please recommend boost/srilm versions known to work with the latest
> >>>> version of cdec?
>
> >>>> Thanks,
> >>>> Randy
>
> >>>> Tail of make output. Note that I have full permissions on
> >>>> /local/contrib/cpl
> >>>> dirs under my ownership and read only otherwise. I own
> >>>> /local/contrib/cpl/cdec but not /local/contrib/cpl/srilm-1.5.11:
>
> >>>> g++ -DHAVE_CONFIG_H -I. -I.. -W -Wall -Wno-sign-compare -I../utils
> >>>> -pthread -I/local/contrib/cpl/srilm-1.5.11//include
>
>
> read more »
Randy West
questions/3584365/boost-thread-error) with the same error (undefined
reference to `vtable for boost::detail::thread_data_base'), but the
code in that post compiles and runs without issue on my system. I
still don't know what's going on, but at the very least we can rule
out the issues mentioned in that thread.
Randy
>
> read more »
Chris Dyer
Thanks for the suggestions. I agree that the boost dependencies makes
things complicated. My immediate plan is to remove the dependency on
boost.thread, which causes the majority of the problems.
Glad you've got things compiling.
Chris
Randy West
Sorry if I was unclear, but I don't actually have cdec compiling yet.
What I meant in my last post was that the stackoverflow example
compiled, indicating that I'm not having the particular issues mentioned
in that post.
Removing the boost.thread dependency sounds like a good idea though if
you think there are issues within cdec and not just with my system. I
would still like to try building boost once more with whatever options
you/others have used. If you know how boost was compiled on your system,
then I would appreciate if you could detail the process.
Thanks,
Randy
>>>>>> feature_accum.h:38: warning: unused parameter �fids�
>>> read more �
Chris Dyer
box" for me, but we did have some trouble with boost thread in some
environments this summer. This dependency is a relatively new (and
experimental) so it has not been compiled in nearly as many
environments, so I haven't had a chance to become familiar with the
kinds of problems that can crop up. If you wish, you can compile
everything but the gi "package" and the decoder will work. The stuff
you're having trouble with is some undocumented grammar induction
code.
-Chris
>>>>>>> feature_accum.h:38: warning: unused parameter ‘fids’
>>>> read more »
>
Randy West
Removing the gi package as you suggested worked very nicely, and the
decoder now seems to be in working order (it can run cdec-demo anyhow).
Just to make sure I am interpreting "compile everything but the 'gi'
package" as you intended, what I did was to comment out the end of the
SUBDIRS line in my Makefile as follows:
SUBDIRS = utils mteval klm/util klm/lm decoder training vest extools
#gi/pyp-topics/src gi/clda/src gi/posterior-regularisation/prjava
FYI, there was a slight discrepancy in the cdec-demo scoring output.
It's most likely the result of some machine-specific fp precision
differences and nothing to worry about, but I thought I'd mention it anyhow:
Loading references (4 files)
Loaded reference translations for 919 sentences.
Loaded 919 references for scoring with ibm_bleu
BLEU = 32.20, 76.3|42.8|24.2|13.8 (brev=0.997)
0.322045
You specify the following on the wiki:
BLEU = 32.20, 76.2|42.8|24.2|13.8 (brev=0.997)
0.321954
So, thank you for all of your help. To avoid these issues for others in
the future, it might be worthwhile to have configure check for *all* of
the required boost headers and/or list package dependencies along with
their versions on the installation page. The summary of my boost
experience is as follows:
v1.34.1 (the old system version): Required
boost/unordered/unordered_set.hpp not present. Doesn't work for decoder
package at all.
v1.40 (the new system version that my admin just added for me): Required
boost/thread/future.hpp not present, but works for decoder package.
v1.45 (the version I built in my home directory): Works for decoder
package, but gi packages generate the following error:
pyp-topics.cc:(.text._ZN5boost6threadC1INS_13packaged_taskIeEEEENS_6detail1
3thread_move_tIT_EE[boost::thread::thread<boost::packaged_task<long
double> >(boost::detail::thread_move_t<boost::packaged_task<long
double> >)]+0x4b): undefined reference to `vtable for
boost::detail::thread_data_base'
If there's anything that I can do to help you debug that last one on
Debian then I'd be more than happy to oblige. I'm sure that I'll have
plenty more questions once I start actually using cdec, but that's it
for now. Thanks again!
Randy
>>>>>>>> feature_accum.h:38: warning: unused parameter �fids�
>>>>> read more �
>>>>>