Custom docker mounts?

[CMV]

Hi all, 

Right before the job starts, I would like to create some folders on the execution node (named in a particular way using amongst other, Owner, ClusterID and JobID) and make them available inside the job's container. I used HOOK_PRE to create the folders but then how can I add a new DOCKER_VOLUME_DIR_ from within the HOOK_PRE script? 

[Todd Tannenbaum]

On 7/8/2026 8:17 AM, CMV wrote:

Hi all, 

Right before the job starts, I would like to create some folders on the execution node (named in a particular way using amongst other, Owner, ClusterID and JobID) and make them available inside the job's container. I used HOOK_PRE to create the folders but then how can I add a new DOCKER_VOLUME_DIR_ from within the HOOK_PRE script?  -- 

Hello,

Using the HOOK_PREPARE_JOB is a good choice to create the folders.  Note that the stdout from this hook is a set of classad attributes (key-value pairs) that will be inserted in the copy of the job classad at the EP.  So your prepare job hook could output a job attribute that says what should be mounted, then you would reference this attribute(s) in EP config knob DOCKER_VOLUME_DIR_xxx_MOUNT_IF.  This config knob is a classad expression (so you can use string manipulation, regular expressions, whatever) that can reference attributes in the job classad to state what docker mounts should happen.  For an example see:
   https://htcondor.readthedocs.io/en/latest/admin-manual/ep-policy-configuration.html#docker-universe


regards,
Todd

[CMV]

On 7/13/26 9:51 PM, 'Todd Tannenbaum' via HTCondor Users wrote:

On 7/8/2026 8:17 AM, CMV wrote:

Hi all, 

Right before the job starts, I would like to create some folders on the execution node (named in a particular way using amongst other, Owner, ClusterID and JobID) and make them available inside the job's container. I used HOOK_PRE to create the folders but then how can I add a new DOCKER_VOLUME_DIR_ from within the HOOK_PRE script?  -- 

Note that the stdout from this hook is a set of classad attributes (key-value pairs) that will be inserted in the copy of the job classad at the EP.

I must be doing something badly wrong because it didn't work. 

My HOOK_PRE script ends with 

cat <<EOF

Hook_SetupTimestamp = $(date +%s)

NewPath = '/some/path'

EOF

exit 0


... but a 

cat $(echo $_CONDOR_MACHINE_AD) | grep -i new

... from condor_ssh shows nothing. 

  So your prepare job hook could output a job attribute that says what should be mounted, then you would reference this attribute(s) in EP config knob DOCKER_VOLUME_DIR_xxx_MOUNT_IF.  This config knob is a classad expression (so you can use string manipulation, regular expressions, whatever) that can reference attributes in the job classad to state what docker mounts should happen.  For an example see:
   https://htcondor.readthedocs.io/en/latest/admin-manual/ep-policy-configuration.html#docker-universe

The example reads:

DOCKER_VOLUMES = SOME_DIR, ANOTHER_DIR
DOCKER_VOLUME_DIR_SOME_DIR = /path1
DOCKER_VOLUME_DIR_SOME_DIR_MOUNT_IF = WantSomeDirMounted && Owner == "smith"


... so it was my understanding that you can't DOCKER_VOLUME_DIR_xxx_MOUNT_IF unless the "xxx" part is previously defined using DOCKER_VOLUMES and DOCKER_VOLUME_DIR_SOME_DIR; yet DOCKER_VOLUMES = SOME_DIR, ANOTHER_DIR  and DOCKER_VOLUME_DIR_SOME_DIR = /path1 appear to be static, they don't seem to be able to reference attributes in the hob classad.  That knob appears to decide only which of the _previously defined_ paths should be mounted or not. Am I wrong?


After some discussion within the team, a colleague suggested that we could mount the parent folder (of the custom paths) and then simply "ln -s /mnt/user_jobId_randomNum   /xtraspace" as the very first thing ran inside the container. 

Sounded like a feasible idea (the fact that the user might see all other custom folders under "/mnt" is just a minor inconvenience, they're 700 anyway) but then I bumped into the same problem described above: echoing stuff from the HOOK_PRE doesn't arrive as job's classad inside the container so the container doesn't know which of the paths to chose. 

[CMV]

On 7/15/26 6:31 PM, CMV wrote:

 

  So your prepare job hook could output a job attribute that says what should be mounted, then you would reference this attribute(s) in EP config knob DOCKER_VOLUME_DIR_xxx_MOUNT_IF.  This config knob is a classad expression (so you can use string manipulation, regular expressions, whatever) that can reference attributes in the job classad to state what docker mounts should happen.  For an example see:
   https://htcondor.readthedocs.io/en/latest/admin-manual/ep-policy-configuration.html#docker-universe

Tried even bypassing the _HOOK_PREPARE_JOB completely and went for 

include : docker_volume.sh |


... but although I can 


XTRA=$( random_generator.sh )

mkdir /mnt/${XTRA}

echo "DOCKER_VOLUME_DIR_xtra = /mnt/${XTRA}:/X"

exit 0

... from within docker_volume.sh and have that volume mounted, the amount of information available to the script is not enough to make a correlation to the job itself! So I could create extra mounts on the fly right before the job starts but then how would the _HOOK_JOB_EXIT know what to delete when the job is done? 

Apparently I can not create a custom docker mount where the path is specific/unique to each job and have it then deleted upon job's exit.