CML - CICSTART Macro Language & Macro Resource, first experiences documented

[Roddi]

I have been working on a macro language that allows for easy access to CICSTART resources.  The language offers a variety of mechanisms for getting access to catalogued files, files on your VFS, and running commands on the OS the script is running on.  I was able to generate about 250 quicklook images of MACCS data the other night with the following script and curl commands:

on $localhost {

// Get the gnuplot script from my VFS.

gnuPlotScript = getVFS($CICSTART.session,"/maccs.gp");

// get some data from MACCS

maccsData = getCataloguedFiles({

project=("MACCS"),

observatories=("PG"),

dateRange=("2010-01-01","2011-01-01")

});

// Loop over each MACCS data file we grabbed

foreach file in $maccsData {

// run gnuplot (assuming it's on the path)

run("gnuplot -e \"filename='$file'\" $gnuPlotScript", "3");

// convert eps to png files.

run("convert -density 300 -alpha off $file.eps $file.png", "3");

// put the png back onto my VFS

putVFS($CICSTART.session, "/$JOBID","$file.png");

}

}

the gnuplot script I used was

set terminal postscript eps size 16,9 enhanced color font 'Helvetica,20' linewidth 2

set output filename.".eps"

set timefmt "%Y| %m| %d| %H| %M| %S|"

set xdata time

plot filename every ::13 using 1:7 with lines linestyle 1

Notice the CML script gnuplot run command "-e" parameter which gives the filename variable a value.  That variable is referenced in the gnuplot script on the second line and the last line.

First, I logged into CICSTART with my user id:

curl --user myemail@email:mypassword -X POST http://208.75.74.81/auth/api/session.json

and saved the session token it returned.  

I had previously uploaded the gnuplot script to my VFS instance on CICSTART.  Check out the github wiki for info on how to do that.

Next, I used the Macro service to generate a binary client of the CML script (Change the paths in the following command to where you have saved your cml and gnuplot scripts. Also change the session token to the result of the login command above):

curl -H CICSTART.session:"f74ee76a-abb9-4b66-9336-34a912387daf" -H Content-Type:"application/octet-stream" --data-binary @/home/rpotter/workspaces/cicstart/cml/src/test/resources/plot_maccs.cml -X POST http://208.75.74.81/macro/api/macro.json/bin?include_jre=false > client.tar.gz 

I extracted the client.tar.gz tarball and ran bin/run.  About 1.1 GB of MACCS data downloaded to my machine, gnuplot then generated a .ps image for each file, and then image magic converted those .ps files into .png files.  Each .png file was then uploaded to my VFS where I could access it via other REST commands (I almost have the FTP service working, just a bit more firewall tweaking to do).  I could have left out the putVFS command, but I wanted to simulate my machine being a cloud resource.

I imagine these scripts can be shared among users of CICSTART.  A following piece of work might be to create a repository for CML scripts to be shared and expanded upon by CICSTART end users.

Next piece of work to do is to have the Macro service start VMs in the cloud and have those VMs get the CML binary client to run.  This is where the putVFS is handy ... because now I can run the script on private VMs (i.e., those VMs would be started without an external IP associated with them) but I still gain access to my data and I can still provide input data.

One other thing is to enhance the putVFS command to allow for other external VFS instances to be accessed by the script to put data.  The CICSTART DAIR install limits the VFS for each user to about 256MB due to current resource limitations.

Rod