Abstractions for storage hierarchy in cylc

[Whitcomb, Mr. Tim]

We are making heavier use of the $CYLC_TASK_WORK_DIR and $CYLC_SUITE_SHARE_DIR directories that are automatically created by Cylc for tasks running within the suite, eliminating the need for a lot of tasks to manage their own work directories and simplifying the storage layout for where things go.

As it stands now, we have then two levels of a storage hierarchy for jobs:
1. task work dir and suite share dir, both on a parallel filesystem - assumed that things can be cleaned, scrubbed, etc. - this is more of a "scratch" space. Persists beyond a single task execution.
2. external archival directories for long-term storage (a separate location on the same parallel filesystem or a separate archive server, for example). Persists beyond a single suite execution.

However, for some jobs it's useful to have an additional level in the storage hierarchy:
0. Fast local storage (e.g. on-node ramdisk or SSD), only persists for duration of the task

Questions:
1. Do other sites currently make use of this additional storage level? If so, how is it managed?
2. Is this something that should be configurable at a Cylc level? Maybe something like:
a) $CYLC_TASK_TEMP_DIR defined as an alias for $CYLC_TASK_WORK DIR by default
b) if, in the task, a "temp directory" entry is set, it's sent to the task via $CYLC_TASK_TEMP_DIR
c) optionally, $CYLC_TASK_TEMP_DIR is cleaned up on job success (similar to the way that $CYLC_TASK_WORK_DIR is, but more aggressive than "rmdir")
3. Is this a use case for an "always run this at the end of the task, be it succeed or fail" as opposed to duplicating something in exit-script and err-script?

I hope this is clear - I'm trying to synthesize several sets of notes from different discussions.

Tim

[Hilary Oliver]

Hi Tim,

People definitely do this sort of thing, but manage it on an ad hoc basis.  I suppose the idea of automatic directory creation and clean up at job end might justify built-in management by Cylc.  I'm not sure that "temporary directory" would be the right name for it, as that suggests a scratch space doesn't it?  (For files needed internally by a process but not relevant externally).  Local fast storage could be needed for important files too (well, presumably that's your use case) e.g. for a job that generates a huge number of small files before aggregating them in some way for external use.

There's a somewhat related discussion on https://github.com/cylc/cylc/pull/2935 and linked Issues, about locating entire single-node suites on fast/local storage, to make "sub-suites" more useful.

Hilary

--

---
You received this message because you are subscribed to the Google Groups "cylc" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cylc+uns...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Matt Shin]

Hi Tim and Hilary,

This is very much related to https://github.com/cylc/cylc/issues/2199 which I am currently trying to kick start. (I have been saying this for the last 2 years, but delaying this is no longer an option.)

On a future system that has much more awareness of the site's task job platforms (or clusters), we should be able to configure ad-hoc settings or environments for task jobs targeted for specific platforms.

Matt