Joshua 6.0.5 KenLM Build Error

[lesman...@gmail.com]

Dear Joshua devs,

First, congratulations for the 6.0.5 release. I'm very happy to see that the 6.0.5 release also publish the light runtime version. This is gonna be fun to work with.

So, I try to build the Joshua 6.0.5 release, but I find an error message regarding the query, lmplz, and build_binary unable to be copied because unable to be found.
At first, I ignore it and try to work through the pipeline. But, as I suspected, since I know that lmplz and build_binary is crucial part of one of the pipeline step, 

it stop at the after the [lm-sort-uniq] phase. 

Here is the error message when I build Joshua. Please note the "No such file or directory" messages.
=============================================================

root@tesis:/datadrive/tools/joshua-6.0.5# ant -f $JOSHUA/build.xml kenlm

Buildfile: /datadrive/tools/joshua-6.0.5/build.xml

check-joshua-home:

     [echo] JOSHUA = /datadrive/tools/joshua-6.0.5 basedir = /datadrive/tools/joshua-6.0.5

kenlm:

     [exec] -- Boost version: 1.54.0

     [exec] -- Found the following Boost libraries:

     [exec] --   program_options

     [exec] --   system

     [exec] --   thread

     [exec] --   unit_test_framework

     [exec] -- Configuring done

     [exec] -- Generating done

     [exec] -- Build files have been written to: /datadrive/tools/joshua-6.0.5/src/kenlm/build

     [exec] [ 35%] Built target kenlm_util

     [exec] Linking CXX executable ../bin/bit_packing_test

     [exec] CMakeFiles/kenlm_util.dir/read_compressed.cc.o: In function `util::(anonymous namespace)::StreamCompressed<util::(anonymous namespace)::GZip>::~StreamCompressed()':

     [exec] read_compressed.cc:(.text+0x219): undefined reference to `inflateEnd'

     [exec] CMakeFiles/kenlm_util.dir/read_compressed.cc.o: In function `util::(anonymous namespace)::StreamCompressed<util::(anonymous namespace)::GZip>::~StreamCompressed()':

     [exec] read_compressed.cc:(.text+0x439): undefined reference to `inflateEnd'

     [exec] CMakeFiles/kenlm_util.dir/read_compressed.cc.o: In function `util::(anonymous namespace)::ReadFactory(int, unsigned long&, void const*, unsigned long, bool)':

     [exec] read_compressed.cc:(.text+0xa49): undefined reference to `inflateInit2_'

     [exec] CMakeFiles/kenlm_util.dir/read_compressed.cc.o: In function `util::(anonymous namespace)::StreamCompressed<util::(anonymous namespace)::GZip>::Read(void*, unsigned long, util::ReadCompressed&)':

     [exec] read_compressed.cc:(.text+0xe8f): undefined reference to `inflate'

     [exec] collect2: error: ld returned 1 exit status

     [exec] make[2]: *** [bin/bit_packing_test] Error 1

     [exec] make[1]: *** [util/CMakeFiles/bit_packing_test.dir/all] Error 2

     [exec] make: *** [all] Error 2

     [exec] cp: cannot stat ‘bin/query’: No such file or directory

     [exec] cp: cannot stat ‘bin/lmplz’: No such file or directory

     [exec] cp: cannot stat ‘bin/build_binary’: No such file or directory

     [exec] g++: error: lm/CMakeFiles/kenlm.dir/*.o: No such file or directory

     [exec] Result: 1

BUILD SUCCESSFUL

=================================================

And, here is the error message from when I invoke the pipeline using this command : 

$JOSHUA/bin/pipeline.pl --type hiero --corpus input/train --tune input/tune --test input/test --source en --target id --tmp /datadrive/tmp --joshua-mem 10g --hadoop-mem 42g --threads 8

===========================================================

[lm-sort-uniq] cached, skipping...

* FATAL: /datadrive/tools/joshua-6.0.5/bin/lmplz (for building LMs) does not exist.

  This is often a problem with the boost libraries (particularly threaded

  versus unthreaded).

===========================================================

I try to work my way around it by adding  "--lm-gen berkeleylm --buildlm-mem 20g" to the pipeline command, but as I suspected, stop at the [compile-kenlm] phase with this error message

============================================================

[compile-kenlm] rebuilding...

  dep=lm.gz [CHANGED]

  dep=lm.kenlm [NOT FOUND]

  cmd=/datadrive/tools/joshua-6.0.5/bin/build_binary lm.gz lm.kenlm

  JOB FAILED (return code 127)

/bin/bash: /datadrive/tools/joshua-6.0.5/bin/build_binary: No such file or directory

============================================================


I already try to build Joshua using the latest source code from Github, but it seems the error is still the same, unable to copy those kenlm lib items.

Here is my setup :

• Ubuntu Server 14.04 LTS
• OpenJDK 7
• Installing Boost from apt-get -> apt-get install libboost-all-dev
• Joshua 6.0.5 (both from the tarball and the source code, same error)

Looking forward for anyone to help me out here. Thanks for your time.

Regards,
Reza Lesmana

[Matt Post]

It looks like cmake is not finding the zlib libraries. Do you have these installed? What system are you on?

--
You received this message because you are subscribed to the Google Groups "Joshua Technical Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to joshua_suppor...@googlegroups.com.
To post to this group, send email to joshua_...@googlegroups.com.
Visit this group at http://groups.google.com/group/joshua_support.
For more options, visit https://groups.google.com/d/optout.

[Reza Lesmana]

Hi, Matt

Yes, I think I have the zlib installed. I'm on Ubuntu 14.04 Server LTS, installed as VM on Azure.

=================================================

root@tesis:/home/reza# apt-get install zlib1g-dev

Reading package lists... Done

Building dependency tree

Reading state information... Done

zlib1g-dev is already the newest version.

0 upgraded, 0 newly installed, 0 to remove and 4 not upgraded.

=================================================

[Matt Post]

Is it in your LD_LIBRARY_PATH? Where is libz.so?

I do not encounter this problem on any of the four systems I have tried building kenlm on, so unfortunately, there's not much I can say. You might try downloading KenLM directly (github.com/kpu/kenlm) and compiling it there (same cmake build system), and seeing if you have the same problem. If you don't, then there's something wrong with Joshua, but if you do, then probably there's a fix needed for their build system. Please do keep us updated.

matt

[Reza Lesmana]

Hi, Matt

I don't know about LD_LIBRARY_PATH, and LD_LIBRARY_PATH is not in my environment variables. 

According to this : http://joshua-decoder.org/6.0/faq.html , I should set LD_LIBRARY_PATH to my boost library directory. But, BOOST_ROOT environment variable should be enough, right? And according to the kenlm build log, it's able to find my boost lib directory, so it shouldn't be a problem. 

Anyway, I try to set LD_LIBRARY_PATH to my boost library directory, but kenlm build still has the same error. Please let me know if I did something wrong.

I try to find my libz.so, and the path is : 

/lib/x86_64-linux-gnu/libz.so.1.2.8

What should I do with this libz.so path?

I try to build kenlm in two locations. First, the kenlm source on joshua-6.0.5 ( $JOSHUA/src/kenlm ), and it succeeded to build those kenlm lib items I'm missing in Joshua KenLM build. Second, I clone the kenlm git source, and then build it. The same result, succeeded to build those kenlm lib items. I build both kenlm using ./bjam command (as stated in the BUILDING file). If I succeeded to build the kenlm from the git source, can I just copy those kenlm lib items? 

If you want an access to my server, please let me know, I'll give you the information. 

Regards,
Reza Lesmana

[Reza Lesmana]

Hi, Matt

I try to copy the build_binary, lmplz, and query I've succeeded to build from kenlm git source, to $JOSHUA/bin folder. And it seems that everything works OK.

Will let you know more soon.

Regards,
Reza Lesmana

[Matt Post]

Yes, please let me know more. At the very least, the sequence of commands you used to download and compile KenLM from its source would be useful to help us update KenLM inside Joshua.

matt

[Reza Lesmana]

Hi, Matt

Sorry for the late reply. 

I'm only using this to download and compile KenLM from its source :

$>clone https://github.com/kpu/kenlm.git

$>cd kenlm
$>./bjam

That's it. I've managed to get the lmplz, build_binary, and query, and then copy it to $JOSHUA/bin folder. 

The pipeline goes through until [mert] phase :

===============================

[mert-1] rebuilding...

  dep=/datadrive/workdir_baseline/data/tune/corpus.en [CHANGED]

  dep=/datadrive/workdir_baseline/tune/joshua.config [CHANGED]

  dep=tune/model/grammar.filtered.gz.packed/slice_00000.source [CHANGED]

  dep=/datadrive/workdir_baseline/tune/joshua.config.final [NOT FOUND]

  cmd=/datadrive/tools/joshua-6.0.5/scripts/training/run_tuner.py /datadrive/workdir_baseline/data/tune/corpus.en /datadrive/workdir_baseline/data/tune/corpus.id --tunedir /datadrive/workdir_baseline/tune --tuner mert --decoder /datadrive/workdir_baseline/tune/decoder_command --decoder-config /datadrive/workdir_baseline/tune/joshua.config --decoder-output-file /datadrive/workdir_baseline/tune/output.nbest --decoder-log-file /datadrive/workdir_baseline/tune/joshua.log --iterations 15 --metric 'BLEU 4 closest'

  JOB FAILED (return code 1)

* FATAL: Can't find libken.so (libken.dylib on OS X) in $JOSHUA/lib

*        This probably means that the KenLM library didn't compile.

*        Make sure that BOOST_ROOT is set to the root of your boost

*        installation (it's not /opt/local/, the default), change to

*        $JOSHUA, and type 'ant kenlm'. If problems persist, see the

*        website (joshua-decoder.org).

Traceback (most recent call last):

  File "/datadrive/tools/joshua-6.0.5/scripts/training/run_tuner.py", line 537, in <module>

    main(sys.argv)

  File "/datadrive/tools/joshua-6.0.5/scripts/training/run_tuner.py", line 520, in main

    run_zmert(opts.tunedir, opts.source, opts.target, opts.decoder, opts.decoder_config, opts.decoder_output_file, opts)

  File "/datadrive/tools/joshua-6.0.5/scripts/training/run_tuner.py", line 401, in run_zmert

    opts.metric, opts.iterations or 10)

  File "/datadrive/tools/joshua-6.0.5/scripts/training/run_tuner.py", line 383, in setup_configs

    for feature,weight in get_features(config):

  File "/datadrive/tools/joshua-6.0.5/scripts/training/run_tuner.py", line 335, in get_features

    output = check_output("%s/bin/joshua-decoder -c %s -show-weights -v 0" % (JOSHUA, config_file), shell=True)

  File "/usr/lib/python2.7/subprocess.py", line 573, in check_output

    raise CalledProcessError(retcode, cmd, output=output)

subprocess.CalledProcessError: Command '/datadrive/tools/joshua-6.0.5/bin/joshua-decoder -c /datadrive/workdir_baseline/tune/joshua.config -show-weights -v 0' returned non-zero exit status 1

==================================================================

Apparently, I should generate the libken.so too. But, using ./bjam is not generating the libken.so file. How should I generate this file when I want to build KenLM separately?

Btw, it seems other user also generate the same problem, but it seems 6.0.5 release should resolve this problem (reference : https://github.com/joshua-decoder/joshua/issues/225 ). But, apparently in my case it's not. My cmake is newest version. I also try on Ubuntu 15.10 server, and it's the same problem. 

Matt, I'm gonna send you an email to access my server and test it out. Is it okay for you?

Regards,
Reza Lesmana

[Matt Post]

Matt, I'm gonna send you an email to access my server and test it out. Is it okay for you?

Sure, that would be a simple way to resolve this. Email me offline and we can set this up.

[Matt Post]

Okay, you are using the bjam build system in KenLM. We can't include that in Joshua because it is not robust enough. You could try using the cmake instructions within KenLM to see if that works.

If you look in jni/build_kenlm.sh, you see there is a command that builds the KenLM library and places it in lib/. You could try running that command manually, but if all the object files have not compiled, it's not going to work properly.

matt

[joa.b...@gmail.com]

Hi,

I've been experiencing the same problems as Reza, and having compiled KenLM myself, I'm stuck at the point where Joshua says libken.so is missing in $JOSHUA/lib.
What exactly has been your solution for this?

By the way, part of the reason Joshua fails to build KenLM is that no *.o files are present in $JOSHUA/src/kenlm/build/lm/CMakeFiles/kenlm.dir (that is after manually adding the KenLM binaries to $JOSHUA/lib). These files are required by build_kenlm.sh.

Best wishes,
Joachim

[Matt Post]

Hi,

Those *.o files should be *produced* by build_kenlm.sh (via the call to make).

What OS are you working on? Can you tell me exactly what commands you typed after downloading Joshua?

matt

[joa.b...@gmail.com]

Thanks for your quick answer! I've tried it on Ubuntu 14.04 and 15.04. On both systems, I followed the install instructions at http://joshua-decoder.org/6.0/install.html and ran the commands

$ export JAVA_HOME=...
$ wget -q http://cs.jhu.edu/~post/files/joshua-runtime-6.0.5.tgzwget
$ tar xzf joshua-runtime-6.0.5.tgz
$ cd joshua-runtime-6.0.5
$ export JOSHUA=$PWD
$ ant

... and when I realised KenLM did not install properly, I also ran

$ ant kenlm

... wherupon I copied the KenLM executables from a separate KenLM installation into $JOSHUA/bin, but to no avail. No I'm stuck where Joshua wants to see libken.so

Thank you very much for helping out!

[Matt Post]

Can you send me the output of

cd $JOSHUA

bash ./jni/build_kenlm.sh

matt

[joa.b...@gmail.com]

The output is

-- Boost version: 1.54.0

-- Found the following Boost libraries:

--   program_options
--   system
--   thread
--   unit_test_framework
-- Configuring done
-- Generating done
-- Build files have been written to: /home/joachim/nlp/joshua-6.0.5/src/kenlm/build
[ 35%] Built target kenlm_util
Linking CXX executable ../bin/bit_packing_test
CMakeFiles/kenlm_util.dir/read_compressed.cc.o: In function `util::(anonymous namespace)::StreamCompressed<util::(anonymous namespace)::GZip>::~StreamCompressed()':

read_compressed.cc:(.text+0x219): undefined reference to `

inflateEnd'

CMakeFiles/kenlm_util.dir/read_compressed.cc.o: In function `util::(anonymous namespace)::StreamCompressed<util::(anonymous namespace)::GZip>::~StreamCompressed()'

:

read_compressed.cc:(.text+0x439): undefined reference to

`inflateEnd'
CMakeFiles/kenlm_util.dir/read_compressed.cc.o: In function `util::(anonymous namespace)::ReadFactory(int, unsigned long&, void const*, unsigned long, bool)':

read_compressed.cc:(.text+0xa49): undefined reference to `inflateInit2_'

CMakeFiles/kenlm_util.dir/read_compressed.cc.o: In function `util::(anonymous namespace)::StreamCompressed<util::(anonymous namespace)::GZip>::Read(void*, unsigned long, util::ReadCompressed&)':

read_compressed.cc:(.text+0xe8f): undefined reference to `

inflate'

collect2: error: ld returned 1 exit status

make[2]: *** [bin/bit_packing_test] Error 1

make[1]: *** [util/CMakeFiles/bit_packing_test.dir/all] Error 2

make: *** [all] Error 2

cp: cannot stat ‘bin/query’: No such file or directory

cp: cannot stat ‘bin/lmplz’: No such file or directory

cp: cannot stat ‘bin/build_binary’: No such file or directory

g++: error: lm/CMakeFiles/kenlm.dir/*.o: No such file or directory

After copying the KenLM excutables to $JOSHUA/src/kenlm/build/bin, the 'cp: cannot stat...' errors obviously disappear, but the last line remains.
 

[Matt Post]

What happens if you do this?

cd $JOSHUA/src

rm -rf kenlm

git clone https://github.com/kpu/kenlm.git

cd $JOSHUA

bash jni/build_kenlm.sh

matt

[joa.b...@gmail.com]

Same error as before... Here's the full output. Does this make any sense to you?
Thanks!

-- The C compiler identification is GNU 4.9.2
-- The CXX compiler identification is GNU 4.9.2
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/g++
-- Check for working CXX compiler: /usr/bin/g++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Boost version: 1.55.0

-- Found the following Boost libraries:
--   program_options
--   system
--   thread
--   unit_test_framework
-- Configuring done
-- Generating done

-- Build files have been written to: /home/joachim/tools/joshua-6.0.5/src/kenlm/build
Scanning dependencies of target kenlm_util
[  1%] Building CXX object util/CMakeFiles/kenlm_util.dir/double-conversion/bignum-dtoa.cc.o
[  2%] Building CXX object util/CMakeFiles/kenlm_util.dir/double-conversion/bignum.cc.o
[  3%] Building CXX object util/CMakeFiles/kenlm_util.dir/double-conversion/cached-powers.cc.o
[  5%] Building CXX object util/CMakeFiles/kenlm_util.dir/double-conversion/diy-fp.cc.o
[  6%] Building CXX object util/CMakeFiles/kenlm_util.dir/double-conversion/double-conversion.cc.o
[  7%] Building CXX object util/CMakeFiles/kenlm_util.dir/double-conversion/fast-dtoa.cc.o
[  8%] Building CXX object util/CMakeFiles/kenlm_util.dir/double-conversion/fixed-dtoa.cc.o
[ 10%] Building CXX object util/CMakeFiles/kenlm_util.dir/double-conversion/strtod.cc.o
[ 11%] Building CXX object util/CMakeFiles/kenlm_util.dir/stream/chain.cc.o
[ 12%] Building CXX object util/CMakeFiles/kenlm_util.dir/stream/io.cc.o
[ 14%] Building CXX object util/CMakeFiles/kenlm_util.dir/stream/line_input.cc.o
[ 15%] Building CXX object util/CMakeFiles/kenlm_util.dir/stream/multi_progress.cc.o
[ 16%] Building CXX object util/CMakeFiles/kenlm_util.dir/stream/rewindable_stream.cc.o
[ 17%] Building CXX object util/CMakeFiles/kenlm_util.dir/bit_packing.cc.o
[ 19%] Building CXX object util/CMakeFiles/kenlm_util.dir/ersatz_progress.cc.o
[ 20%] Building CXX object util/CMakeFiles/kenlm_util.dir/exception.cc.o
[ 21%] Building CXX object util/CMakeFiles/kenlm_util.dir/file.cc.o
[ 23%] Building CXX object util/CMakeFiles/kenlm_util.dir/file_piece.cc.o
[ 24%] Building CXX object util/CMakeFiles/kenlm_util.dir/float_to_string.cc.o
[ 25%] Building CXX object util/CMakeFiles/kenlm_util.dir/integer_to_string.cc.o
[ 26%] Building CXX object util/CMakeFiles/kenlm_util.dir/mmap.cc.o
[ 28%] Building CXX object util/CMakeFiles/kenlm_util.dir/murmur_hash.cc.o
[ 29%] Building CXX object util/CMakeFiles/kenlm_util.dir/parallel_read.cc.o
[ 30%] Building CXX object util/CMakeFiles/kenlm_util.dir/pool.cc.o
[ 32%] Building CXX object util/CMakeFiles/kenlm_util.dir/read_compressed.cc.o
[ 33%] Building CXX object util/CMakeFiles/kenlm_util.dir/scoped.cc.o
[ 34%] Building CXX object util/CMakeFiles/kenlm_util.dir/string_piece.cc.o
[ 35%] Building CXX object util/CMakeFiles/kenlm_util.dir/usage.cc.o
[ 35%] Built target kenlm_util
Scanning dependencies of target bit_packing_test
[ 37%] Building CXX object util/CMakeFiles/bit_packing_test.dir/bit_packing_test.cc.o

Linking CXX executable ../bin/bit_packing_test
CMakeFiles/kenlm_util.dir/read_compressed.cc.o: In function `util::(anonymous namespace)::StreamCompressed<util::(anonymous namespace)::GZip>::~StreamCompressed()':
read_compressed.cc:(.text+0x219): undefined reference to `inflateEnd'
CMakeFiles/kenlm_util.dir/read_compressed.cc.o: In function `util::(anonymous namespace)::StreamCompressed<util::(anonymous namespace)::GZip>::~StreamCompressed()':

read_compressed.cc:(.text+0x359): undefined reference to `inflateEnd'
CMakeFiles/kenlm_util.dir/read_compressed.cc.o: In function `util::(anonymous namespace)::ReadFactory(int, unsigned long&, void const*, unsigned long, bool)':
read_compressed.cc:(.text+0x931): undefined reference to `inflateInit2_'
CMakeFiles/kenlm_util.dir/read_compressed.cc.o: In function `util::(anonymous namespace)::StreamCompressed<util::(anonymous namespace)::GZip>::Read(void*, unsigned long, util::ReadCompressed&)':
read_compressed.cc:(.text+0xc5f): undefined reference to `inflate'

collect2: error: ld returned 1 exit status

util/CMakeFiles/bit_packing_test.dir/build.make:146: recipe for target 'bin/bit_packing_test' failed

make[2]: *** [bin/bit_packing_test] Error 1

CMakeFiles/Makefile2:964: recipe for target 'util/CMakeFiles/bit_packing_test.dir/all' failed

make[1]: *** [util/CMakeFiles/bit_packing_test.dir/all] Error 2

Makefile:127: recipe for target 'all' failed

make: *** [all] Error 2
cp: cannot stat '

bin/query': No such file or directory
cp: cannot stat 'bin/lmplz': No such file or directory
cp: cannot stat 'bin/build_binary': No such file or directory

[Reza Lesmana]

Hi, 

Sorry for late reply. I've been switching from Ubuntu to Centos 7.1 and the build is working great. So, the problem only persists for Ubuntu (14.04, 15.04, 15.10)

However, thrax is giving me trouble since no final directory available so the get_merge command won't be able to execute, still working out what's the problem. I'll post it on new thread. 

Regards,
Reza Lesmana

...

[Matt Post]

Thanks for this update, Reza. I don't understand what the issue is with Ubuntu, but it's good to have the problem isolated, somewhat. 

The Hadoop issue likely has something to do with disk space, but I'll watch for your other post.

matt

[joa.b...@gmail.com]

I also tried a different OS now, and the build works like a charm for Debian 3.2.68.

Thanks for helping!

[Matt Post]

Great to hear, thanks for the followup. 

If anyone is inclined, it would be great to submit this as a bug to KenLM (doesn't compile with cmake on Ubuntu).

e.g., https://github.com/kpu/kenlm/issues/45

matt

On Nov 24, 2015, at 10:07 AM, joa.b...@gmail.com wrote:

I also tried a different OS now, and the build works like a charm for Debian 3.2.68.

Thanks for helping!

[Gideon]

Dear Matt and others,
I've also struggled with these issues recently.
I run Kubuntu 14.04 on my work computer and we have an outdated CentOS
installation on some of our servers, which necessitates local installation of Boost
and static linking against it , causing more problems.
I've made a patch that fixes the issues I encountered:
1. (Not) Linking to Zlib (at least not in a way working on ubuntu)
2. Not dealing properly with static binding against boost
3. Not adding linking to the "real time" library ( often added as option"-lrt")

My fix is available on my Joshua fork https://github.com/gwenniger/joshua/tree/gideon/fixJoshuaKenlmCMakeFiles
and will possibly be integrated in the Joshua main branch in one form or another at some point.
Cheers.

Gideon

[Matt Post]

Thanks very much, Gideon! I just made a note on your PR that you should submit the patch to KenLM instead, since we just pull in a copy of their code. We also plan to soon switch to using them as a proper submodule, so it's better for you to get it fixed upstream. From my own frustration with this issue, I imagine others (on other decoders) have the same problems and will appreciate the fix.

matt

Visit this group at https://groups.google.com/group/joshua_support.