how to keep a python script running in notebook (AI Platform)

34 views
Skip to first unread message

Oumayma Abdesslem

unread,
May 21, 2021, 10:38:21 AM5/21/21
to gce-discussion
Good evening, I am working in a deep learning project, so I use the AIplatform service, but I find a problem with the training model spending a lot of time (up to 3 days), there is a break in the chapter notebook
 here is an image that explains my problem
I need the help And thank you in advanceimage.png

Fady (Google Cloud Platform)

unread,
May 21, 2021, 4:19:50 PM5/21/21
to gce-discussion
From the error message is seems that many connection attempts fail through a websocket. This might be for different reasons related to connectivity or the VM. As this happens after several days of training, the most probable theory is "VM related "such as if the disk is out of space, utilization, or an issue with the proxy. Here is a guide for troubleshooting that should get you started. To explain, " Inverting Proxy agent runs inside your Notebooks instance as a Docker container" and is responsible for the connectivity. The factors above can affect it and cause such issues. I hope this puts you on the right track. I suggest posting on serverfault.com if you find more specific logs if you SSH to the instance (from the guide above) and to help the community troubleshoot. 
Reply all
Reply to author
Forward
0 new messages