First swamp run with custom ncks, soap_client.py problems

[Jeff]

For the benefit of the community, I'm posting a personal
correspondence between Daniel and myself.

Hi Jeff,

> I can apparently get a swamp service started. I assumed the
> "slave.conf" was an appropriate config file, so I copied it to
> "swamp.conf" which is what the swamp/config.py was expecting. After
> editing the paths to my data, NCO, and my server information, 'python
> swamp_soapinterface.py' starts just fine.

Great! Actually, the intent was to start from demo/sample-swamp.conf
(renamed to swamp.conf for installations), but I may have forgotten to
update it after some recent code changes. The slave.conf is supposed
to be used for worker-instances, which receive directives from a
master swamp instance.

> I've written a very simple swamp script. Curious? It's just 'ncks -d
> level,1 r5_pressure-0.nc jefftest.nc'. When I ran 'python
> swamp_client.py jefftest.swamp' I can see the DEBUG messages of the
> swamp server process my request up through publishing the output file
> to my semi-persistent directory. I can even navigate to the
> semi-persistent directory to ncdump the output file. However, the
> client is having the problem of "No such file or directory: 'd'".
> I've narrowed down the problem to the urllib.urlretrieve call thanks
> to the strack trace. Commenting out that line I get the following output:
> Fetching d and writing to /
> Fetching t and writing to h
> That makes me think that server.pollOutputs(token) is at fault.

This may be due to some issues with using what was intended to be a
'slave' configuration. Please have a look at the other sample, and
see if it makes sense to you. If you send me your current config, I
can look over it and see if it looks reasonable.


> But hey, the ipcctest.swamp worked great and if I don't call
> server.discardFlow(token) at least the file(s) can be found in my
> semi-persistent directory. I guess this isn't a huge issue since
> transferring the files directly to the client isn't our first goal
> necessarily. I plan on using a web form to construct a swamp script,
> submit the script for the user, then send the user the URLs to the
> files when the process completes. The goal is not making the user
> wait for the script to finish running. I guess in the end I do need
> to have access to the output filenames to construct correct URLs...

This should be fine. If your web form calls acts as a client, it can
certainly download files from the swamp instance and write them to
local known filenames and a known path. The user can then download
files from that path. The swamp service will make up filenames on its
own, but the client python code does the mapping work for you.


> I have to say, fantastic service you've created, this swamp. I don't
> even know the half of its details and inner workings, but I can
> already tell its going to give us what we need and save us
> implementation time (we were going to have to do something similar
> ourselves someday...)
>
> Oh, wait, one more thing. We have a custom ncks build that adds a
> custom argument. The argument is '-g' and takes an argument string.
> For example)
> ncks -g -180,-5,5,180
> Our data is on a geodesic grid so we can't have monotonically
> increasing coordinates. Our hack computes the needed indices into the
> appropriate dimension by essentially reading through the entire
> coordinate variable for both lat and lon. (Aside: It'd be nice if NCO worked on "auxiliary"
> coordinate variables AND we could turn off the monotonic requirement.)
> Question: How can I modify swamp to allow a custom ncks argument like
> the one I've defined? I tried adding a 'g' to the string of ncks
> short options inside the Parser, but it interpreted the next "-180" as a "1"
> argument and bailed.

When you add it to the list of short options, be sure to add it with a
colon after the 'g'. That way, our call to the getopt library will
expect a parameter for the 'g' option.



> Thanks again, Daniel!
You're quite welcome! Keep the comments and questions coming...
though we should probably move this to the swamp-users list so that it
gets archived for others' benefit. ( http://groups.google.com/group/swamp-users
)

-Daniel