suggestion for saving state

[nbecker]

I'm trying to decide how to adapt my current code to use with picloud.  Everything is currently running fine, but I'm not sure about how to save results.

The code is long-running monte-carlo simulation.  As it's currently structured, it can periodically save the info I need (includes all simulation parameters and results) in pickle  format to a file.  I notice that running in picloud, this model (saving to a file) doesn't work (permission denied writing the file).  Also, I believe such a file would not persist after the job completes?

I'm not sure what's a good alternative.  Some ideas:

1. Write periodic results to /tmp.  (Would  this persist after the job completes?)

2. As it is now, there is 1 function run in picloud, which would return a pickle of results if/when it completes.  However, I don't always want to wait that long, and might want intermediate results.  I could restructure this so the function returns more often with intermediate results - but this would be a bit of work.

Suggestions?

[Ken Elkabany]

I've responded inline. Do let us know how you end up designing your function to provide intermediate results.

Ken

On Sun, Nov 6, 2011 at 1:17 PM, nbecker <ndbe...@gmail.com> wrote:

I'm trying to decide how to adapt my current code to use with picloud.  Everything is currently running fine, but I'm not sure about how to save results.

The code is long-running monte-carlo simulation.  As it's currently structured, it can periodically save the info I need (includes all simulation parameters and results) in pickle  format to a file.  I notice that running in picloud, this model (saving to a file) doesn't work (permission denied writing the file).  Also, I believe such a file would not persist after the job completes?

Where were you attempting to write the file? You can open a file for writing in the /home/picloud (default current working directory). You're right that the file does not persist after the job completes.

 

I'm not sure what's a good alternative.  Some ideas:

1. Write periodic results to /tmp.  (Would  this persist after the job completes?)

No it does not.

 

2. As it is now, there is 1 function run in picloud, which would return a pickle of results if/when it completes.  However, I don't always want to wait that long, and might want intermediate results.  I could restructure this so the function returns more often with intermediate results - but this would be a bit of work.

Suggestions?

You could design your function to regularly store intermediate results using the cloud.files module (http://docs.picloud.com/moduledoc.html#module-cloud.files).

Roughly, for your use case, you would include the following in the function that you offload to PiCloud:

cloud.files.putf('somekey', pickle.dumps(obj))

And then from anywhere else, you could see the intermediate results using:

pickle.dumps(cloud.files.getf('somekey'))

[Neal Becker]

I can't write a file in the working directory, it gets 'permission denied'.  It seems picloud does not own the stuff.

When I build the env, it seems everything is initially root.root and when I ssh in, I do chown -R picloud.picloud.  But after finishing setting up the env, I suspect once again picloud cannot write to his home dir or subdir of it. (Actually, I'm running in a subdir of /home/picloud).

[Neal Becker]

If I ssh to my env, and try to test I get:

ImportError: No module named cloud

You were saying on the _server_ (cloud) side to use cloud.files.putf?

[Neal Becker]

I tried:
on server (cloud side):
periodically:
cloud.files.putf(opt.logname, dumps (results, -1))

The function is running and no exceptions. On the client side, in a seperate
process (not the one that is waiting on the function result), I do:
cloud.setkey(api_key=1478, api_secretkey='xxx)
print cloud.files.list() <<<< this line prints '[]', an empty list

So my file is not found, and for that matter, no files are listed.

from cPickle import loads
res = loads
(cloud.files.getf('test_8psk_cancel_lms_olddemod.py.nbecker6.hughes.com.bbd8025fa4bf2f8ebf1f6c281b7a1fa1.err'))

python ./read_cloud.py
[]
[Tue Nov 08 12:04:18 2011] - [ERROR] - Cloud.HTTPConnection: rawquery:
received error from server: Status 492: The specified file was not found.
Traceback (most recent call last):
File "./read_cloud.py", line 28, in <module>
res = loads
(cloud.files.getf('test_8psk_cancel_lms_olddemod.py.nbecker6.hughes.com.bbd8025fa4bf2f8ebf1f6c281b7a1fa1.err'))
File "/usr/lib/python2.7/site-packages/cloud/files.py", line 419, in getf
resp = conn.send_request(_file_get_query, {'name': file_name})
File "/usr/lib/python2.7/site-packages/cloud/transport/network.py", line
412, in send_request
log_cloud_excp = log_cloud_excp)
File "/usr/lib/python2.7/site-packages/cloud/transport/network.py", line
307, in send_request_helper
retry=resp['retry'], logger=cloudLog)
cloud.cloud.CloudException: Status 492: The specified file was not found.

[Aaron Staley]

Hi Neal,

We are still looking into the issues you are having with the setup server (environment creator).

As for cloud.files, it unfortunately looks like the argument order provided to you earlier was incorrect.  The correct command to save your result is:

cloud.files.putf(pickle.dumps(obj), 'somekey')

Please let me know if this change fixes the issue.  If not, can you please file support ticket letting us know the jid of the job that attempted to save to cloud.files?

Regards,

Aaron Staley

PiCloud, Inc.

[Neal Becker]

I seem to have it working now, after correcting a couple of minor errors.

One thing though:

"cloud.setkey() must be called before using any functions in this module."

On the server (cloud) side, I had forgotten to include cloud.setkey(). I
would have expected an exception, but I don't think I got one.

[Aaron Staley]

Sorry about that; bad documentation. Before your job runs on PiCloud, we call setkey for you. :)

docs are now:

Api keys must be configured before using any functions in this module.

(via cloudconf.py, cloud.setkey, or being on PiCloud server)