Another non-utf8 locales problem

[Mohammad Mumin]

Sir,

Thank you very much for your help.

By fixing previous non-utf8 locales problem in /nematus/setup.py, I became able to run dockerfile.gpu.

Anyway, now, I face following errors when I run nematus/train.py for my model:

(This may also be same kind problem of issue with non-utf8 locales)

Traceback (most recent call last):
  File "nematus/train.py", line 447, in <module>
    train(config, sess)
  File "nematus/train.py", line 149, in train
    text_iterator, valid_text_iterator = load_data(config)
  File "nematus/train.py", line 53, in load_data
    keep_data_in_memory=config.keep_train_set_in_memory)
  File "/playground/nematus/nematus/data_iterator.py", line 64, in __init__
    self.source, self.target = shuffle.main([self.source_orig, self.target_orig], temporary=True)
  File "/playground/nematus/nematus/shuffle.py", line 15, in main
    line = [l.strip()] + [ff.readline().strip() for ff in fds[1:]]
  File "/playground/nematus/nematus/shuffle.py", line 15, in <listcomp>
    line = [l.strip()] + [ff.readline().strip() for ff in fds[1:]]
  File "/usr/lib/python3.5/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe0 in position 0: ordinal not in range(128)

I seek your kind help.

Thanks in advance.

[Rico Sennrich]

Hello Mohammad,

thanks for reporting this. I've pushed a fix that opens all files in UTF-8, which should fix any issues with varying locale settings. Just make sure that the text files you work with are using UTF-8.

best wishes,
Rico

--
You received this message because you are subscribed to the Google Groups "Nematus Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nematus-suppo...@googlegroups.com.
To post to this group, send email to nematus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/nematus-support/c7a9e64c-9cfe-439e-9503-98f121018e30%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.