Problems with swamp input script

[Jeff]

I can run the swamp_client.py using the ipcctest.swamp file without
any problems. I printed out the raw object during the "Fetching"
phase of the client and see lots of lines like the the following (note
the "Url returned" is my addition):

Url returned: <SOAPpy.Types.typedArrayType item at -1212440116>:
['sresa1b_cccma_cgcm3_1_t63_anm.nc', 'http://pbs.calit2.uci.edu:8080/
pub/f4595_341BBAE6sresa1b_cccma_cgcm3_1_t63_anm.nc']

When I run my simple script, I get the following:

Url returned: r5_pressure-0_test.nc
Url returned: http://<auto>:28080/pub/f9783_72ADC95Lr5_pressure-0_test.nc

Notice that in the latter cae the URLs are not wrapped in a SOAPpy
type. Here's my input file, for reference. The triple quotes were
added to show the first and last lines of the file.

"""
#!/bin/bash

TESTROOT=/data/nc

ncks -d level,1 $TESTROOT/r5_pressure-0.nc r5_pressure-0_test.nc
"""

I am running revision 97 of the code from SVN, in case anyone needs to
know.

Jeff

[Daniel]

The <auto> designation for hostname doesn't work for standalone
instances or master instances in clusters. This is because the code
detects the proper host address by making an outbound connection to
whichever host it wishes to receive from, and then checking which
network interface got used.

Please change the hostname field in the [service] section of your
configuration file appropriately.

Please let us know if this doesn't work.
-Daniel

[Jeff]

I changed the swamp.conf file in two places.
1) I noticed that the swamp/config.py file was looking for "mode"
instead of "level" as it was currently defined in the swamp.conf.
"level" should refer to the debug/testing/production value whereas
"mode" refers to master (the default).
2) Replaced <auto> with "localhost".

Unfortunately, the object returned from server.pollOutputs(token) is
still not the SOAPpy array type that is expected in the subsequent for
loop. I'm still using my simple swamp script described earlier.

[Daniel]

I can see that I have some work to do in preparing a package for
server installations. :) The sample-swamp.conf is not consistent
with the current code. I'll fix that.

I haven't been able to reproduce your problem. I tested a script
similar to yours:
'''
TESTROOT=/pcmdi
ncks -d lat,1 $TESTROOT/
pcmdi.ipcc4.cccma_cgcm3_1.sresb1.run1.daily.pr_a2_sresb1_1_cgcm3.1_t47_2046_2065.nc
r5_pressure-0_test.nc
'''
...which gave me expected results as far as I could tell. I changed
the path/filename to fit my data, and the variable to lat, because it
doesn't have a 'level' variable. I reused your output file name to
test whether there's some strange file naming problem.

It's strange that the return types are different. My version
returned:
<SOAPpy.Types.typedArrayType Result at 46912541042376>:
[['r5_pressure-0_test.nc', 'http://localhost:8081/pub/
f26183_F98BB326r5_pressure-0_test.nc']]
Fetching http://localhost:8081/pub/f26183_F98BB326r5_pressure-0_test.nc
and writing to r5_pressure-0_test.nc

I'm inclined to believe that the client is okay, since it works
properly against our test instance. pollOutputs(token) should only be
called once per script, where your output seems to show that it's
been called twice, receiving one list element of the correct return
value each time.

Here's the swamp.conf of my single-node development instance. I'm
sure you've doublechecked everything, but you may have been waylaid by
my incorrect sample earlier. It's got a lot of vestigial things from
my testing, but perhaps you'll see something.

Hope this helps,
-Daniel
--------------------
## Logging-related settings
[log]

# A place to write logs of what the handler sees.
location = ./ssdap_cli.log

# True turns off logging
disable = False

# Log level: one of CRITICAL, ERROR, WARNING, INFO, DEBUG,
NOTSET
level = DEBUG

########################################
## Service-related settings (for servers)
########################################

[service]
hostname = localhost
port = 8081
path = SOAP
filePath = data
# mode is one of: master, worker
mode = master
# level is one of: production, testing, debug
level = debug

########################################
## Execution-related settings
########################################
[exec]

# path to NCO binaries compiled with NetCDF4 support
nco4 = /home/wangd/bin

# path to NCO binaries compiled with OPeNDAP support
ncodap = /home/wangd/bin

# scratchPath: temporary scratchspace
scratchPath = /dev/shm/

# semi-persistent space
resultPath = /data/wangd/swamp/publish

bulkPath = /data/wangd/swamp

# root for source data (will support catalog later)
sourcePath = /data/wangd

# exec slots:
localSlots = 3
slaveNodes = 0

# if you have slave nodes with the same source file space,
# define slave?Url and slave?Slots for each of them.
slave1Url = http://localhost:8082/SOAP
slave1Slots = 4

slave2Url = http://localhost:8083/SOAP
slave2Slots = 4

########################################
## Database settings
## (for db-backed asynchronous scheduling)
[db]
#/dev/shm is a ram-based temporary filesystem.
dbFilename = /home/wangd/ssdapSpace/swamp_db.sqlite

[Jeff]

It turns out the problem was with the version of SOAPpy I installed.
I was using 0.12.0, but switching to 0.11.6 fixed the problem.

On Dec 11 2007, 5:55 pm, Daniel <daniel2...@gmail.com> wrote:
> I can see that I have some work to do in preparing a package for
> server installations. :) The sample-swamp.conf is not consistent
> with the current code. I'll fix that.
>
> I haven't been able to reproduce your problem. I tested a script
> similar to yours:
> '''
> TESTROOT=/pcmdi
> ncks -d lat,1 $TESTROOT/
> pcmdi.ipcc4.cccma_cgcm3_1.sresb1.run1.daily.pr_a2_sresb1_1_cgcm3.1_t47_2046_2065.nc
> r5_pressure-0_test.nc
> '''
> ...which gave me expected results as far as I could tell. I changed
> the path/filename to fit my data, and the variable to lat, because it
> doesn't have a 'level' variable. I reused your output file name to
> test whether there's some strange file naming problem.
>
> It's strange that the return types are different. My version
> returned:
> <SOAPpy.Types.typedArrayType Result at 46912541042376>:
> [['r5_pressure-0_test.nc', 'http://localhost:8081/pub/
> f26183_F98BB326r5_pressure-0_test.nc']]

> Fetchinghttp://localhost:8081/pub/f26183_F98BB326r5_pressure-0_test.nc

> slave1Url =http://localhost:8082/SOAP
> slave1Slots = 4
>
> slave2Url =http://localhost:8083/SOAP