OSError: [Errno 12] Cannot allocate memory

[Mohammad Mumin]

Dear Sir,

After running for more than 3 days, I have got the following error:

......

......

......

INFO: [2019-08-07 13:19:06] Epoch: 423 Update: 289000 Loss/word: 0.01086953039962727 Words/sec: 5990.283468987167 Sents/sec: 310.57984805119486
INFO: Starting epoch 424
INFO: Starting epoch 425
INFO: [2019-08-07 13:34:34] Epoch: 425 Update: 290000 Loss/word: 0.0111569994173202 Words/sec: 5972.707581318338 Sents/sec: 310.33315214868344
INFO: Seen 25
INFO: Seen 58
INFO: Seen 98
INFO: Seen 146
INFO: Seen 201
INFO: Seen 267
INFO: Seen 351
INFO: Seen 484
INFO: Seen 500
INFO: Validation cross entropy (AVG/SUM/N_SENTS/N_TOKENS): 66.95999866014719 33479.999330073595 500 11461
INFO: Starting external validation.
INFO: NOTE: Length of translations is capped to 100
INFO: Translated 40 sents
INFO: Translated 80 sents
INFO: Translated 120 sents
INFO: Translated 160 sents
INFO: Translated 200 sents
INFO: Translated 240 sents
INFO: Translated 280 sents
INFO: Translated 320 sents
INFO: Translated 360 sents
INFO: Translated 400 sents
INFO: Translated 440 sents
INFO: Translated 480 sents
INFO: Translated 500 sents
INFO: Translated 500 sents in 36.85555028915405 sec. Speed 13.56647766963722 sents/sec
Traceback (most recent call last):
  File "/home/mumin-cse/nematus//nematus/train.py", line 454, in <module>
    train(config, sess)
  File "/home/mumin-cse/nematus//nematus/train.py", line 254, in train
    score = validate_with_script(sess, replicas[0], config)
  File "/home/mumin-cse/nematus//nematus/train.py", line 358, in validate_with_script
    stderr=subprocess.PIPE)
  File "/usr/lib/python3.6/subprocess.py", line 729, in __init__
    restore_signals, start_new_session)
  File "/usr/lib/python3.6/subprocess.py", line 1295, in _execute_child
    restore_signals, start_new_session, preexec_fn)
OSError: [Errno 12] Cannot allocate memory

I have attached the full training output for your convenience.

At the beginning of the file you will find my model configuration and at the end is the error.

Thanks for your all cooperation.

Best wishes,

[Rico Sennrich]

Hello Mohammad,

running out of memory after this long is highly unusual. Is it possible that you (or somebody else with access to the machine) started another process that competed for the available memory? It's also a bit unusual in that you may have been running out of main memory, rather than GPU memory - does your machine have sufficient RAM, and swap?

best wishes,
Rico

--
You received this message because you are subscribed to the Google Groups "Nematus Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nematus-suppo...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/nematus-support/a314d018-64e0-4f85-831c-472b6fe1a409%40googlegroups.com.

[Mohammad Mumin]

Thank you very much Sir for your thoughtful response.

Perhaps, the first case is happened for me.

Thanks again.

Best wishes.