Setting up JupyterHub with HTCondor batchspawner

[Nikita Balashov]

Hi,

I'm trying to make batchspawner work with HTCondor but I'm stuck with the following error:

[I 2018-11-09 13:35:39.816 JupyterHub batchspawner:242] Spawner submitting job using sudo -i -u testuser condor_submit
[I 2018-11-09 13:35:39.816 JupyterHub batchspawner:243] Spawner submitted script:
   
    Executable = /bin/sh
    RequestMemory = 4gb
    RequestCpus = 1
    Arguments = "-c 'exec batchspawner-singleuser --ip=""0.0.0.0""'"
    Remote_Initialdir = /home/testuser
    Output = /home/testuser/.jupyterhub.condor.out
    Error = /home/testuser/.jupyterhub.condor.err
    ShouldTransferFiles = False
    GetEnv = True
    Universe = vanilla
    Queue
   
[I 2018-11-09 13:35:40.119 JupyterHub batchspawner:246] Job submitted. cmd: sudo -i -u testuser condor_submit output: Submitting job(s).
    1 job(s) submitted to cluster 19.
[D 2018-11-09 13:35:40.120 JupyterHub batchspawner:269] Spawner querying job: sudo -i -u testuser condor_q 19 -format "%s, " JobStatus -format "%s, " RemoteHost -format "

    " True
[E 2018-11-09 13:35:40.356 JupyterHub batchspawner:215] Subprocess returned exitcode 1
[E 2018-11-09 13:35:40.357 JupyterHub batchspawner:216] Stdout:
[E 2018-11-09 13:35:40.357 JupyterHub batchspawner:217] b''
[E 2018-11-09 13:35:40.357 JupyterHub batchspawner:218] Stderr:
[E 2018-11-09 13:35:40.357 JupyterHub batchspawner:219] Error: -format requires format and attribute parameters
[E 2018-11-09 13:35:40.357 JupyterHub batchspawner:274] Error querying job 19
[W 2018-11-09 13:35:40.358 JupyterHub batchspawner:372] Job  neither pending nor running.
   
[E 2018-11-09 13:35:40.359 JupyterHub user:477] Unhandled error starting testuser's server: The Jupyter batch job has disappeared while pending in the queue or died immediately after starting.
[D 2018-11-09 13:35:40.373 JupyterHub user:578] Deleting oauth client jupyterhub-user-testuser
[E 2018-11-09 13:35:40.410 JupyterHub web:1670] Uncaught exception GET /hub/user/testuser/ (159.93.40.25)
    HTTPServerRequest(protocol='http', host='jupyterhub.jinr.ru', method='GET', uri='/hub/user/testuser/', version='HTTP/1.1', remote_ip='159.93.40.25')
    Traceback (most recent call last):
      File "/usr/share/anaconda3/lib/python3.7/site-packages/tornado/web.py", line 1592, in _execute
        result = yield result
      File "/usr/share/anaconda3/lib/python3.7/site-packages/jupyterhub/handlers/base.py", line 1052, in get
        await self.spawn_single_user(user)
      File "/usr/share/anaconda3/lib/python3.7/site-packages/jupyterhub/handlers/base.py", line 705, in spawn_single_user
        timedelta(seconds=self.slow_spawn_timeout), finish_spawn_future
      File "/usr/share/anaconda3/lib/python3.7/site-packages/jupyterhub/handlers/base.py", line 626, in finish_user_spawn
        await spawn_future
      File "/usr/share/anaconda3/lib/python3.7/site-packages/jupyterhub/user.py", line 489, in spawn
        raise e
      File "/usr/share/anaconda3/lib/python3.7/site-packages/jupyterhub/user.py", line 409, in spawn
        url = await gen.with_timeout(timedelta(seconds=spawner.start_timeout), f)
      File "/usr/share/anaconda3/lib/python3.7/site-packages/batchspawner/batchspawner.py", line 373, in start
        raise RuntimeError('The Jupyter batch job has disappeared'
    RuntimeError: The Jupyter batch job has disappeared while pending in the queue or died immediately after starting.

The condor_q command succeeds if ran manually:

# sudo -i -u testuser condor_q 19 -format "%s, " JobStatus -format "%s, " RemoteHost -format "\n" True

1,

# echo $?
0

I'm using the latest batchspawner (from the master):

# pip list |grep batchspawner
batchspawner                       0.9.0.dev0

And the spawner configuration:

c.JupyterHub.spawner_class = 'batchspawner.CondorSpawner'

c.Spawner.http_timeout = 120

c.BatchSpawnerBase.req_nprocs = '1'
c.BatchSpawnerBase.req_memory = '1gb'

c.BatchSpawnerBase.req_runtime = '12:00:00'

c.CondorSpawner.exec_prefix = 'sudo -i -u {username}'

What can be the cause of this error?

Thanks,

Nikita Balashov

[Michael Milligan]

Hi Nikita,

This question will get a better response as an issue at the batchspawner GitHub repo: https://github.com/jupyterhub/batchspawner/issues

If you create an issue report there with this information, the appropriate developers (including myself) will see it and give you a response as soon as they are able.

Thanks,

Michael

--
You received this message because you are subscribed to the Google Groups "Project Jupyter" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jupyter+u...@googlegroups.com.
To post to this group, send email to jup...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/c7891051-3231-4cab-894e-9af3a9c129fa%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Chris Holdgraf]

You could also give a shot at asking on the JupyterHub discourse pilot! https://discourse.jupyter.org 

To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/CAPuy8gpZZbytyoYSwHWo6LJckjAtd-0ghqX7fTuMs_a7Nk3xHg%40mail.gmail.com.