execution node and copyback

[Anna Kostikova]

Hello everyone,

This is me again, with more questios about saga:)
We keep learing saga and using it for our web-application, but we are
stuck with one thing which prevent us from the next step. If you could
help us with some ideas/suggestions on how to solve it - we would
greately appreciate it.

We want to do copy to/copy back of file to/from the execution node
when we launch a job. Basically, the idea is that we have some storage
nodes, where users put their files that need to be processed, but for
actual processing we copy these files to the execution node(s) and
then we copy back the resulting files to the user folder.

The main issue is that we don't know which node will be assigned by
slurm as an execution node. Thus we don't know how to implement
copy/copyback. Are there any options in saga so that we could do some
actions after job.rub()?

We investigated three options:
1. callbacks (http://saga-python.readthedocs.org/en/latest/library/attribute.html
), but unsure if it would do what we want?
2. maybe it is possible with jd.executable? if yes, then how?
3. maybe we could use a function which would be executed after
job.run() , but before an execution of task set in jd.executable?

So, in essense, how can we get that saga.filesystem.Directory and
saga.filesystem.File would "know" an execution node ?

For instance, we tried:

_dir = "output-work"
baseDir = saga.filesystem.Directory("sftp://NODE_NAME/home/user1")
baseDir.make_dir(_dir)

But now we need to have NODE_NAME in order to copy files to the execution node.

Any ideas/suggestions would be really greatly appreciated. Thanks a
lot guys for helping us with your great project

Anna

[Ole Weidner]

Hi Anna,

let me start with a naive question: does your SLURM cluster not use a shared filesystem between compute nodes? At least all HPC clusters that I have ever worked with followed this design.

If that is the case, you don’t need to now the execution node. Instead of:

sftp://NODE_NAME/home/user1

you would just use:

sftp://CLUSTERNAME/home/user1

And "/home/user1” would be shared across all compute nodes.

If this is not the case for your cluster, would you mind sharing its design details with us?

Thanks!
Ole

> --
> You received this message because you are subscribed to the Google Groups "saga-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to saga-users+...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[trollius]

hello Ole,

Thanks for your reply!

as you correctly pointed out we don't use a shared filesystem, because at some point it may lead to exsessive usage of the central server to which all our nodes are mounted. Also, we are planning to use external storage (cloud) for keeping large datasets and we really need to find another solution instead of shared filesystem. 

If Saga could have an option of knowing to which node the task is going to be sent by SLURM, so it would be possible to do other actions (copy-in/copy-back for instance) - it would be really great. Do you think there might be other ways of achieving what we need (as I described above)? 

Thanks a lot for any advice,

Anna

[Andre Merzky]

I don't think it is possible to know where the task will land before
it actually goes there -- that is, after all, the whole point of using
a scheduler on a cluster, to take care of task placement... So I am
not really sure how to handle that use case, independently on how that
could be expressed in SAGA.

Did you run your workload without SAGA before? How did you do the
data placement in that case? Are there any other users on that
cluster who found a solution to the problem?

One somewhat clumsy approach would be to wrap the job into a shell
script which stages the data once slurm placed it onto a compute node.
Not sure if that would be viable, and also, that would imply some
magic outside of SAGA (i.e. you would need to create and submit those
wrapper scripts). I'm afraid I can't think of anything more sensible,
really...

Best, Andre.

--
It was a sad and disappointing day when I discovered that my Universal
Remote Control did not, in fact, control the Universe. (Not even
remotely.)

[Mark Santcroos]

Hi all,

Don't the transfer directives exactly address this use-case?

Gr,

Mark

[Andre Merzky]

From the original mail (emphasis mine):

> If Saga could have an option of knowing to which node the task is going to

> be sent by SLURM [...]

At the point where we could enact the stage-in, we do not yet know
onto which compute node the job will be placed on, so how can we stage
to that node? Once it is places, it will start running quickly, so
its too late for SAGA to do any staging at that point.

Am I missing something?

Thanks, Andre.

[Mark Santcroos]

> On 29 Oct 2014, at 13:33 , Andre Merzky <an...@merzky.net> wrote:
>
> From the original mail (emphasis mine):
>
>> If Saga could have an option of knowing to which node the task is going to
>> be sent by SLURM [...]
>
> At the point where we could enact the stage-in, we do not yet know
> onto which compute node the job will be placed on, so how can we stage
> to that node? Once it is places, it will start running quickly, so
> its too late for SAGA to do any staging at that point.
>
> Am I missing something?

Not sure, but I would argue that this is the task of the LRMS.
The latter being SLURM in this case, that would be a combination of salloc, sbcast, srun.

Gr,

Mark

[Andre Merzky]

Dear Anna,

we have been discussing your use case in our group. It seems that
Mark is right, this could in principle be implemented. It is
difficult to judge at this point how much effort that would be though.
Since clusters without shared filesystems are relatively rare (at
least amongst our current set of users), we probably won't be able to
put a high priority on this item.

We opened a ticket for this feature request, at [1], to keep track of
the issue. Is not having this supported in SAGA a showstopper for
you? Is your group managing the cluster on your own? If not, could
we get in contact with the admins, to discuss other potential options?

Best, Andre.



[1] https://github.com/radical-cybertools/saga-python/issues/396

On Wed, Oct 29, 2014 at 2:01 PM, Mark Santcroos
<mark.sa...@rutgers.edu> wrote:
>
>> On 29 Oct 2014, at 13:33 , Andre Merzky <an...@merzky.net> wrote:
>>
>> From the original mail (emphasis mine):
>>
>>> If Saga could have an option of knowing to which node the task is going to
>>> be sent by SLURM [...]
>>
>> At the point where we could enact the stage-in, we do not yet know
>> onto which compute node the job will be placed on, so how can we stage
>> to that node? Once it is places, it will start running quickly, so
>> its too late for SAGA to do any staging at that point.
>>
>> Am I missing something?
>
> Not sure, but I would argue that this is the task of the LRMS.
> The latter being SLURM in this case, that would be a combination of salloc, sbcast, srun.
>
> Gr,
> Mark

[trollius]

Dear Andre,

That's very exciting - thanks a lot for your kind reply and attention to our case. I will check with our admin the questions you've asked and will write here a follow-up.

Thanks a lot,

Anna

[trollius]

Dear Andre,

Our admins has tried different approaches over last week, and, indeed, it's a stopover.

Just to summarize again: the task is to copy files from master server to execution node via slurm+ssh adaptor before main task (processing of copied files) have be triggered and copy resulting file from execution node to main(master) server via the same channel.

We have tried to use following approach (see code below), which worked on master, but failed on any of the remote node. Probably because pty_shell does not accept slurm+ssh connect where ssh works fine.

Can we use following functions via slurm+ssh adapter?

self.shell = saga.utils.pty_shell.PTYShell ("ssh://us...@remote.host.net/", contexts, self._logger)

and following functiones:

run_copy_to(src, tgt, cp_flags='')

run_copy_from(src, tgt, cp_flags='')[

read_from_remote(src)

stage_to_remote

stage_from_remote(src, tgt, cp_flags='')

One more idea we tried is to use saga.filesystem.Directory. However we do need to know the host (node address) which is unknown at the moment of myjob.run(). Can we somehow trace the host before an actual execution (similar to myjob.execution_hosts) and use the host address with saga.filesystem.Directory function to do file copyig before running the main command?

Anyways, indeed, it seems like we are having hard time making it work - so any help/advice or discussion with you would be really great. Admins are more than happy to discuss any time that would be convenient to you and via any channel (email, skype, etc) 

Thanks a lot for your great help with all this!

Anna

On Thursday, 30 October 2014 13:04:30 UTC+1, Andre Merzky wrote:

[Anna Kostikova]

Dear Andre and list,

Any news on the thread?
Sorry to bother you guys again, but we are quite stuck without your help.

Anna

> --
> You received this message because you are subscribed to a topic in the Google Groups "saga-users" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/saga-users/-PB1Z9t_bnQ/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to saga-users+...@googlegroups.com.

[Ole Weidner]

Dear Anna,

as Andre already pointed out, this is a relatively low priority item which would require a significant amount of work. So realistically speaking, I don’t think that anyone from our team at Rutgers will be able to work on this over the next couple of months. 

But — SAGA is by no means “our” project. it is an open source project (managed by us) that is open to everyone’s ideas and contribution. The development process explicitly encourages external contributions. The source code is publicly available on GitHub: https://github.com/radical-cybertools/saga-python. So if you or someone in your group would want to try implementing a solution for your specific use case, we would be delighted to help you getting started with the development process. Just let us know. 

Best regards,

Ole

You received this message because you are subscribed to the Google Groups "saga-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to saga-users+...@googlegroups.com.

[Andre Merzky]

On Wed, Nov 5, 2014 at 6:34 PM, trollius <anna.ko...@gmail.com> wrote:
> Anyways, indeed, it seems like we are having hard time making it work - so
> any help/advice or discussion with you would be really great. Admins are
> more than happy to discuss any time that would be convenient to you and via
> any channel (email, skype, etc)

Anna,

I think a chat or call via skype or hangout would help to get us on
the same page. If I am not mistaken, you are on Central European
time, is that correct? If so, would, say, Friday 11am work for you
and the admins? If not, could you suggest a time between 8:30am and
1pm CET on weekdays?

Best, Andre.