Error Build on KenLM

[Reza Lesmana]

Dear developers,

Currently I'm trying out Joshua Decoder for my thesis. Per installation document and FAQ, I've tried to build the Joshua Decoder.
I have encountered some error build on KenLM. At first, it was the Boost library not installed, after installing, still not able to detect
the Boost. FAQ gives me information about the MT option on the Makefile and I edit the Makefile to solve it. It is finally able to detect Boost.

Then, the JDK (regarding the jni.h) is not able to be found. Before, I use the openjdk 8, but it seems to have problem on the jni.h, so I installed the
Oracle JDK 8, and (I think) it is solved. Or at least it's not showing error message related to jni.h

But, still, it gives me error.

This is the partial printed log on terminal.
-------------------------------------------------------------
kenlm:
     [exec] make: Entering directory '/home/reza/Downloads/joshua-v6.0.2/src/joshua/decoder/ff/lm/kenlm'
     [exec] Detected Boost
     [exec] g++ -I. -DNO_ICU  -I/opt/local/include -I. -O3 -DKENLM_MAX_ORDER=9 -DHAVE_ZLIB -DNDEBUG jni/wrap.cc -I/usr/lib/jvm/java-8-oracle/include -I/usr/lib/jvm/java-8-oracle/include/linux lm/bhiksha.o lm/binary_format.o lm/config.o lm/lm_exception.o lm/model.o lm/quantize.o lm/read_arpa.o lm/search_hashed.o lm/search_trie.o lm/sizes.o lm/trie.o lm/trie_sort.o lm/value_build.o lm/virtual_interface.o lm/vocab.o util/bit_packing.o util/ersatz_progress.o util/exception.o util/file.o util/file_piece.o util/mmap.o util/murmur_hash.o util/pool.o util/read_compressed.o util/scoped.o util/usage.o util/double-conversion/bignum.o util/double-conversion/bignum-dtoa.o util/double-conversion/cached-powers.o util/double-conversion/diy-fp.o util/double-conversion/double-conversion.o util/double-conversion/fast-dtoa.o util/double-conversion/fixed-dtoa.o util/double-conversion/strtod.o util/parallel_read.o -fpic -shared -Wl,-soname,libken.so -o libken.so -lz -Wno-deprecated -pthread -lrt
     [exec] Makefile:62: recipe for target 'libken.so' failed
     [exec] make: Leaving directory '/home/reza/Downloads/joshua-v6.0.2/src/joshua/decoder/ff/lm/kenlm'
     [exec] lm/bhiksha.o: file not recognized: File format not recognized
     [exec] collect2: error: ld returned 1 exit status
     [exec] make: *** [libken.so] Error 1
     [exec] Result: 2
---------------------------------------------------------------
Now, I'm stuck, not able to think what should I do to solve this.

Can anyone help me?

Thanks alot.

[Reza Lesmana]

Dear developers,

A friend of mine (who has used Linux for quite some time), just now told me to "make clean" first on the KenLM source folder (where Makefile for KenLM reside).
And (I think) KenLM is built successfully. It seems the bhiksha.o is having error, and make clean able to resolve it. But, when I try to run the pipeline (using Quickstart documentation), it gives me error. Try to google to solve it, but can't find others that has the same problem (or probably my keyword is not good enough).

This is the printed log in terminal.

---------------------------------------------------
reza@thesis:~/Downloads$ $JOSHUA/bin/pipeline.pl --source bn --target en     --no-prepare --aligner berkeley     --corpus input/bn-en/tok/training.bn-en     --tune input/bn-en/tok/dev.bn-en     --test input/bn-en/tok/devtest.bn-en
[source-numlines] cached, skipping...
[source-numlines] retrieved cached result => 20788
[berkeley-aligner-chunk-0] cached, skipping...
[aligner-combine] cached, skipping...
[lm-sort-uniq] cached, skipping...
[kenlm] cached, skipping...
[compile-kenlm] rebuilding...
  dep=lm.gz [CHANGED]
  dep=lm.kenlm [NOT FOUND]
  cmd=/home/reza/Downloads/joshua-v6.0.2/src/joshua/decoder/ff/lm/kenlm/build_binary lm.gz lm.kenlm
  JOB FAILED (return code 1)
Reading lm.gz
----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
****************************************************************************************************
End of file Byte: 0
ERROR
------------------------------------------------------------------------------------

Does anyone have any idea what is the error and how to solve it??

Thanks alot.

Regards,
Reza Lesmana

[Reza Lesmana]

Dear developers,

From here (after trying to change google keyword search) : https://github.com/joshua-decoder/joshua/issues/154

It said that the problem is KenLM still not build properly. So, I try once again to run "ant kenlm", but it seems everything is okay.

Here is the printed log on "ant kenlm".

------------------------------------
reza@thesis:~/Downloads/joshua-v6.0.2$ ant kenlm
Picked up JAVA_TOOL_OPTIONS: -javaagent:/usr/share/java/jayatanaag.jar
Buildfile: /home/reza/Downloads/joshua-v6.0.2/build.xml

check-joshua-home:
     [echo] JOSHUA = /home/reza/Downloads/joshua-v6.0.2 basedir = /home/reza/Downloads/joshua-v6.0.2

kenlm:
     [exec] make: Entering directory '/home/reza/Downloads/joshua-v6.0.2/src/joshua/decoder/ff/lm/kenlm'
     [exec] Detected Boost

     [exec] cp -f libken.so /home/reza/Downloads/joshua-v6.0.2/lib/libken.so
     [exec] cp -f build_binary ngram_query lmplz  /home/reza/Downloads/joshua-v6.0.2/bin/

     [exec] make: Leaving directory '/home/reza/Downloads/joshua-v6.0.2/src/joshua/decoder/ff/lm/kenlm'

BUILD SUCCESSFUL
Total time: 0 seconds
--------------------------------------------------------------------------------

If anyone can point me to the right direction, it would be really nice.

Thanks. :)

[Matt Post]

What platform are you on?

lmplz now builds on OS X.

You can test whether KenLM built successfully by checking whether $JOSHUA/lib/libken.so (Linux) or $JOSHUA/lib/libken.dylib (OS X) exists.

--
You received this message because you are subscribed to the Google Groups "Joshua Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to joshua_develop...@googlegroups.com.
To post to this group, send email to joshua_d...@googlegroups.com.
Visit this group at http://groups.google.com/group/joshua_developers.
For more options, visit https://groups.google.com/d/optout.

[Reza Lesmana]

Hi, Matt

I'm using Ubuntu 15.04 64bit on VirtualBox.

I can confirm that libken.so exists in $JOSHUA/lib, but I'm still not able to run the pipeline script example on Quickstart documentation.

On the log message, there is "dep=lm.kenlm [NOT FOUND]" message. What is this lm.kenlm?

Thanks.

Regards,
Reza Lesmana

[Reza Lesmana]

Hi, Matt,

After stuck for quite a while, I try to do everything from scratch, including downloading the Joshua Decoder.
And I'm happy to say that I've managed to run the pipeline script example on Quickstart documentation successfully.

I want to share some of my experience here for other users who try to do the same thing.

1. I work on Ubuntu 15.04 64 bit on VirtualBox with 2 cores and 4GB of memory.
2. I download the Joshua Decoder 6.0.2 from this link ( http://joshua-decoder.org/releases/6.0/ ).Note: the version might change in the future.
Extract it and set the $JOSHUA environment variable to the extracted folder.
3. I have to install JDK first. I install the Oracle JDK 8 via PPA http://www.webupd8.org/2012/09/install-oracle-java-8-in-ubuntu-via-ppa.html
Note: Don't forget to use the JAVA_HOME variable setup script at the end of the article.
4. I have to install Boost lib first. http://stackoverflow.com/questions/12578499/how-to-install-boost-on-ubuntu
5. I do "make clean" on the folder $JOSHUA/src/joshua/decoder/ff/lm/kenlm/ , since somehow it's like there is an error there, and make clean is able to solve it.
I forgot what is the error, but I think it's related to the build_binary file.
6. I need to edit the Makefile on the folder $JOSHUA/src/joshua/decoder/ff/lm/kenlm/ ,change the BOOST_ROOT to point to my Boost
installation directory. Mine is : /usr/include/boost
7. I need to edit the same Makefile on step 6, and comment the BOOST_MT option. I know that in the FAQ it's said to uncomment if you encounter an error
regarding this one, but I comment it and no more error. Note: this depends on your Boost lib installation.
8. Run 'ant' on the $JOSHUA folder. Check printed log if you found any suspicious error.
9. Run the Quickstart tutorial, I think mine run in about 6-8 hours.

And, if you are stuck, try to do it from scratch :D

Btw, I'm trying to use Joshua for my graduate thesis for Indonesian language. I most likely have more questions after this, hopefully I can have some help from here. Thanks alot.

Regards,
Reza Lesmana

[Matt Post]

Hi Reza,

Thanks for this awesome summary. If you'd like you could issue this information as a pull request against https://github.com/joshua-decoder/joshua-decoder.github.com, but I am also happy to pull it in myself (let me know!)

And please do feel free to continue to post questions here.

matt

[Matt Post]

FYI, I also just pushed up a change that lets you set BOOST_ROOT from your local environment, which then gets passed down to the KenLM makefile automatically. This will be included in Joshua 6.0.3. I also hope to find a way to automatically determine whether to set BOOST_MT, which will eliminate many of these troubles.

matt

[Reza Lesmana]

Hi, Matt

Sorry for the long wait. Still work on weekdays, and only able to work on my thesis at weekend.

Not that I don't want to send pull request, but I'm still working around my way on github (and git), so it's still a strange area for me. :D
I would love to contribute someday though :).

Btw, please see another question from me on other thread that I'll be writing after this.

Regards,
Reza Lesmana