Memory limitation error

[Prakash Pimpale]

Hi,

Trying to run the pipeline using the recent Joshua version 6.0. I am getting following error. I have tried setting the all the Xmx values to 2g by default in joshua-6.0.5/scripts/training/run_tuner.py. But that is also not helping.

I am on 4GB RAM ubuntu system. 

 cmd=/home/prakash/smt/joshua/joshua-6.0.5/scripts/training/run_tuner.py /home/prakash/smt/joshua/joshua-6.0.5/examples/data/tune/corpus.hi /home/prakash/smt/joshua/joshua-6.0.5/examples/data/tune/corpus.en --tunedir /home/prakash/smt/joshua/joshua-6.0.5/examples/tune --tuner mert --decoder /home/prakash/smt/joshua/joshua-6.0.5/examples/tune/decoder_command --decoder-config /home/prakash/smt/joshua/joshua-6.0.5/examples/tune/joshua.config --decoder-output-file /home/prakash/smt/joshua/joshua-6.0.5/examples/tune/output.nbest --decoder-log-file /home/prakash/smt/joshua/joshua-6.0.5/examples/tune/joshua.log --iterations 15 --metric 'BLEU 4 closest'

  JOB FAILED (return code 1)

Invalid maximum heap size: -Xmx4g

The specified size exceeds the maximum representable size.

Error: Could not create the Java Virtual Machine.

Error: A fatal exception has occurred. Program will exit.

Traceback (most recent call last):

  File "/home/prakash/smt/joshua/joshua-6.0.5/scripts/training/run_tuner.py", line 537, in <module>

    main(sys.argv)

  File "/home/prakash/smt/joshua/joshua-6.0.5/scripts/training/run_tuner.py", line 520, in main

    run_zmert(opts.tunedir, opts.source, opts.target, opts.decoder, opts.decoder_config, opts.decoder_output_file, opts)

  File "/home/prakash/smt/joshua/joshua-6.0.5/scripts/training/run_tuner.py", line 401, in run_zmert

    opts.metric, opts.iterations or 10)

  File "/home/prakash/smt/joshua/joshua-6.0.5/scripts/training/run_tuner.py", line 383, in setup_configs

    for feature,weight in get_features(config):

  File "/home/prakash/smt/joshua/joshua-6.0.5/scripts/training/run_tuner.py", line 335, in get_features

    output = check_output("%s/bin/joshua-decoder -c %s -show-weights -v 0" % (JOSHUA, config_file), shell=True)

  File "/usr/lib/python2.7/subprocess.py", line 544, in check_output

    raise CalledProcessError(retcode, cmd, output=output)

I am using this to run the pipeline:

$JOSHUA/bin/pipeline.pl --rundir .  --type hiero --tune input/t --test input/t --corpus input/t --source hi  --target en --lmfile english50k.lm.blm --no-corpus-lm --joshua-mem 2g

Please help.

Thank you.

----

प्रकाश बा. पिंपळे । Prakash B Pimpale 

मुंबई, महाराष्ट्र, भारत । Mumbai, Maharashtra, India
==

मला माहित नाही मी जग बदलू शकेल की नाही, पण त्यासाठी प्रयत्न करायचा निश्चय तरी मी केलाय!
I don't know if I can change the world, but I have taken a firm decision to give it a try!

[Matt Post]

What version of Joshua do you have installed?

It looks like run_tuner.py has hard-coded 4gb for MERT. Change that line in the file and see if that helps.

matt

--
You received this message because you are subscribed to the Google Groups "Joshua Technical Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to joshua_suppor...@googlegroups.com.
To post to this group, send email to joshua_...@googlegroups.com.
Visit this group at https://groups.google.com/group/joshua_support.
For more options, visit https://groups.google.com/d/optout.

[pbpi...@gmail.com]

I am using joshua 6. I have changed all hardcoded 4g to 2g. But not worked. I think the value is being picked up from tune/joshua.config but there is no such value as 4g. Please suggest.

On Wednesday, December 16, 2015 at 7:36:01 PM UTC+5:30, Matt Post wrote:

What version of Joshua do you have installed?

It looks like run_tuner.py has hard-coded 4gb for MERT. Change that line in the file and see if that helps

[Prakash Pimpale]

I figured out that and it was hard coded in $JOSHUA/bin/joshua-decoder

## memory usage; default is 4 GB

mem=2g

That particular error has went away. Thank you for help.