directory.copy returning NoneType

[Michael Afanasiev]

Hello,

I'm trying to use SAGA to submit a few hundred jobs, where each job requires some filesystem manipulation (i.e. copy from /store to /scratch) before running. I thought the best way to do this would be to use a job/task container. The job container works as expected, but I'm a bit puzzled by the behaviour of the task container.

For example, if I write something like

bulkPreC = saga.task.Container()
preDstDir=saga.filesystem.Directory("sftp://localhost%" %(storeDir))
preSrcDir=saga.filesystem.Directory("sftp://localhost%s" %(scratchDir))
for f in preSrcDir.list():
   bulkPreC.add(preSrcDir.copy(f, preDstDir.get_url(), saga.task.TASK))

it errors out with

Signature Mismatch

  in function        : preprocessData

  in file            : /users/afanasm/src/codeDevelopment/ovalOffice2/ovalOffice/project_management.py +196

  on line            : bulkPreC.add(preSrcDir.copy(f, preDstDir.get_url(), saga.task.TASK))

  method             : add

  argument           : #1

  has incorrect type : NoneType

  instead of         : Task

Am i making some mistake here? Should it be possible for the copy command to return a SAGA task which can be run later using a .run() on the container? If I do not place this within a Container.add(), the copy completes.

Thanks for you help,

Mike.

[Andre Merzky]

Hi Mike,

The signature of the directory.copy call is [1]:

def copy (self, url_1, url_2=None, flags=0, ttype=None)

so in the way you use it, the 3rd parameter is interpreted as a flag,
not as a ttype (task type). saga.task.TASK is defined as an int, and
thus it passes the type check for flags. You probably want to use:

preSrcDir.copy(f, preDstDir.get_url(), ttype=saga.task.TASK)

Calling out the ttype parameter may be a good idea in general -- it
is usually the last parameter, and on many calls you would otherwise
get signature mismatches.

I want to add that running too many sftp commands concurrently in the
background may fail at some point: many systems limit the number of
open sftp connections. SAGA tries to be clever about connection
handling, but it still might run into troubles, depending on the
actual limits. We would be glad to get some feedback how this works
out for you! :)

HtH, and thanks, Andre.


PS: I think you have a typo in the preDstDir definition: "localhost%"
should probably be "localhost%s".

[1] https://github.com/radical-cybertools/saga-python/blob/devel/src/saga/namespace/directory.py#L329

> --
> You received this message because you are subscribed to the Google Groups
> "saga-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to saga-users+...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



--
99 little bugs in the code.
99 little bugs in the code.
Take one down, patch it around.

127 little bugs in the code...

[Michael Afanasiev]

Hi Andre,

Thanks so much for the response. You're right, specifying ttype in the function call fixed the issue. I did a small test where I append 20 Directory.copy tasks to a task container, and then run the task container. I've now run into another error, copied below. In this case, 19 out of the 20 copies did actually complete, whereas one thread (I guess the one with the exception) did not. Interestingly enough, the reason I switched to using task containers is because I had originally tried to do this with threads, and had encountered the same error. Could this be due to some limitation on the system?

thread got exception: RuntimeError:LeaseObject is already leased: <radical.utils.lease_manager._LeaseObject object at 0x2aaaccc751d0>Traceback (most recent call last):

  File "/users/afanasm/anaconda/lib/python2.7/site-packages/radical/utils/threads.py", line 143, in run

    self._result    = self._call (*self._args, **self._kwargs)

  File "/users/afanasm/anaconda/lib/python2.7/site-packages/saga/adaptors/cpi/decorators.py", line 57, in wrap_function

    return sync_function (self, *args, **kwargs)

  File "/users/afanasm/anaconda/lib/python2.7/site-packages/saga/adaptors/shell/shell_file.py", line 524, in copy

    ret, out, err = self._command (" cp %s '%s' '%s'\n" % (rec_flag, src.path, tgt.path))

  File "/users/afanasm/anaconda/lib/python2.7/site-packages/saga/adaptors/shell/shell_file.py", line 302, in _command

    with self.lm.lease (lease_tgt, self.shell_creator, location) \

  File "/users/afanasm/anaconda/lib/python2.7/site-packages/radical/utils/lease_manager.py", line 362, in lease

    obj.lease ()

  File "/users/afanasm/anaconda/lib/python2.7/site-packages/radical/utils/lease_manager.py", line 60, in lease

    raise RuntimeError ("LeaseObject is already leased: %s" % self)

RuntimeError: LeaseObject is already leased: <radical.utils.lease_manager._LeaseObject object at 0x2aaaccc751d0>

All the best,

Mike.