Running derivative containers on Workbench through Terraform

[Markus Toivonen]

Hello! Hopefully this is the correct place to post this question, as I didn't find such group for Workbench in general. If not, any direction to the correct forum is appreciated.

I am trying to run a derivative container in our Workbench environment that gets deployed with terraform, but am having some problems. I tried to do a similar operation to the tutorial here: https://cloud.google.com/deep-learning-containers/docs/derivative-container, but without success.

I downloaded a DL notebooks image (gcr.io/deeplearning-platform-release/pytorch-gpu.1-13.py310) to my local environment, pushed it to our private GCR and set it as the image to use in the relevant terraform configs.

The Workbench instances get deployed, and appear in the Vertex Web UI. The UI shows the correct image name under the "Custom container" field. However, the instances don't seem to launch JupyterLab, as the logs show the following messages:

WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by \'NewConnectionError(\'<urllib3.connection.HTTPConnection object at 0x7ff86c7923d0>: Failed to establish a new connection: [Errno 111] Connection refused\')\': /api/terminals\r\n"

notebooks_collection_agent. Unable to contact Jupyter API: HTTPConnectionPool(host=\'127.0.0.1\', port=8080): Max retries exceeded with url: /api/terminals (Caused by NewConnectionError(\'<urllib3.connection.HTTPConnection object at 0x7ff86c71fdd0>: Failed to establish a new connection: [Errno 111] Connection refused\'))

So I am wondering should the derivative image include some commands for launching JupyterLab? Its not mentioned in the documentation I linked, since it just mentions about updating the image and pushing it to GCR.

Thanks!