CUDA OOM

Nannan Zou

unread,

Mar 21, 2021, 2:46:25 AM3/21/21

to Workshop and Challenge on Learned Image Compression (CLIC)

Dear organizers,

For our submission #1070, we find CUDA is out of memory when the process is starting to load decoder model. We have tested our decoder on the same GPU memory locally, and it works fine. Also, we cannot believe only loading decoder model allocates 16GB.

So, could you help us to look into this problem?

Best,

Nannan

Nannan Zou

unread,

Mar 21, 2021, 2:56:07 AM3/21/21

to Workshop and Challenge on Learned Image Compression (CLIC)

And also, it happens to #1066.

sumayiw...@gmail.com

unread,

Mar 21, 2021, 3:28:43 AM3/21/21

to Workshop and Challenge on Learned Image Compression (CLIC)

I was in a similar situation, submission #1033 and submission #1064 have the same decoder(with the same Decoder hash), but submission #1064 failed to load the decoder: " tensorflow.python.framework.errors_impl.InternalError: Dst tensor is not initialized " while submission #1064 work well

fra jem

unread,

Mar 21, 2021, 1:27:12 PM3/21/21

to Workshop and Challenge on Learned Image Compression (CLIC)

Dear Organizers,

Could you please check the issue raised by Nannan in this thread? This is very important for us to understand whether we can proceed in the competition.

Thank you in advance!

Best regards,

Fra

Lucas Theis

unread,

Mar 22, 2021, 7:08:05 AM3/22/21

to Workshop and Challenge on Learned Image Compression (CLIC)

There's not much we can do to help with decoders running out of memory, especially close to/after the submission deadline. These issues are highly dependent on your code and the frameworks/libraries you use. Tensorflow's memory consumption can be difficult to predict and there's several settings that affect its memory management, but that's not necessarily something we can help with.

Reply all

Reply to author

Forward