GSoC 2015 JGrasstools MODFLOW and ZOO-WPS ideas

[Alexander Kmoch]

Hi Andrea, hi all,

as Andrea suggested I am switching the mailing list from the Osgeo SoC to this one.

I'll try to check out https://github.com/moovida/jgrasstools and just run the maven build I guess. The main pom.xml doesn't look too complicated?

The wiki states [1] Maven 2.2, can I also build with Maven 3?

Is there any documentation regarding the ZOO work?

I have heard from OMS3 before, I wonder if distributed integrated modelling is more scalable via WPS services or OMS3 :-) ?

Cheers,

Alex

[1] : https://code.google.com/p/jgrasstools/wiki/Building

Am 25/02/2015 um 11:02 a.m. andrea antonello wrote:> 

Hi Alex,

> 

>> my name is Alex, I am studying Geoinformatics in New Zealand, water is a big

>> issue here (like probably in most places nowadays). I have been working with

>> the 52North WPS for a while. In my research work I am aiming to apply OGC

>> standards in Hydrogeology. Lately I started experimenting with encapsulating

>> SWAT (Soil Water Assessment Tool) within a 52N WPS algorithm and to "feed"

>> it with OGC geodata (aka WFS features, WCS coverages and SOS hydro-climate

>> time-series).

>>

>> At this stage I saw the OSGeo GSoC Jgrasstools ideas, amazing :-) As of now

> 

> nice to hear you like them.

> And I like the experiments you write about a lot. :-)

> 

>> I'd like to ask, why ZOO-WPS is chosen as the WPS? Just as a question,

>> JGrassTools as Java-API would also play nicely with the 52N WPS? I don't

>> have much experiences with ZOO, but I got it installed and could get a JVM

>> process running.

> 

> This all comes from the fact that some initial work has been done to

> create a Zoo WPS modules generator for the JGrasstools project and we

> had contact with the ZooWPS community.

> 

>> I'd would be intrigued to explore these topics further with you.

> 

> Feel free to stop in the JGrasstools mailinglist and discuss this further.

> 

> All the best,

> Andrea

> _______________________________________________

> SoC mailing list

> S...@lists.osgeo.org

> http://lists.osgeo.org/mailman/listinfo/soc

[andrea antonello]

Hi Alexander,

> as Andrea suggested I am switching the mailing list from the Osgeo SoC to
> this one.

nice to find you here. Welcome! :-)

> I'll try to check out https://github.com/moovida/jgrasstools and just run
> the maven build I guess. The main pom.xml doesn't look too complicated?
>
> The wiki states [1] Maven 2.2, can I also build with Maven 3?

Yes, I am using that one. Guess the docs are lagging beind :-/

> Is there any documentation regarding the ZOO work?

I am afraid not much, or better no more than:

The generator code I wrote as long as I was able to:
https://github.com/moovida/jgrasstools/tree/nf_zooproject

And then the docs on the zoo site: http://www.zoo-project.org/
for java:
http://www.zoo-project.org/docs/services/howtos.html#java

There were some discussions in the zoo mailinglist. Look in the ones
where Andrea Antonello is present:
http://lists.osgeo.org/pipermail/zoo-discuss/2014-November/thread.html

When I started on this, I directly tried the jgrasstools branch and to
convert the modules.
Then I tried them in Zoo, the problem was that the Zoo kernel with
java support is not all that keen about giving out decent error
messages, so I went one step back and started with a more complex
example than the one in the Zoo docs, which obviously works because it
is simple simple. :-)
So I started to work on this:
https://github.com/moovida/zoo-java-example-service

I fiddled around and never really got it working, time ran out for me
and now everything is stalled. That is why the gsoc proposal came up
also this year :-)

> I have heard from OMS3 before, I wonder if distributed integrated modelling
> is more scalable via WPS services or OMS3 :-) ?

LOL, good point. Sure OMS3, but it would be nice to have the whole
JGrasstools library supported through a OGC standard framework.

On a quick analysis, working on this would require:
- get zoo working with java
- get the complex java example working (or make one even more complex)
- tweak the generator if necessary
- understand how input and output data should be handled. People
working on JGrasstools often work with large rasters both in input and
output. Since the birth of the LESTO part that handles LiDAR data,
this is even more an important issue, so I am not keen to have
something that only works with WCS or WFS data. There should be a way
to make it work with server-local data reference. This would need to
be discussed with in the Zoo list. i think I can have a co-mentor from
Zoo register for this project, if it is picked.
- maybe create a simple client for it in the www.geopaparazzi.eu
project (ok, that might be too much, maybe a next year's project :-D )

Hope I have cleared some of the issues.
Cheers,
Andrea

[Alexander Kmoch]

Hi, me again,

regarding MODFLOW, how is it intended to integrate (with) MODFLOW? Isn't MODFLOW a 'C' program? I perceive it a bit like the SWAT codes (Fortran): Basically wrapping the execution of the actual program by pre-processing the input files in their designated formats and then running it and harvesting the outputs?

Certainly step by step everything, but wouldn't it be amazing if like a surface water modelling tools could exchange data on time-step basis with a groundwater modelling tool :-) ? But therefore the modelling processes would need to be able to talk to each other. I see OMS3 might do the trick? Is there any reason why OMS3 is supported more prominently than e.g. OpenMI?

Cheers,

Alex

[Alexander Kmoch]

Hi Andrea,

thanks a lot for the resources, I'll look through it in the next days.

Cheers,

Alex

[Alexander Kmoch]

Hi Andrea,

I got in touch with Gerald Fenoy from the ZOO-WPS project. I'd like to ask how best we possibly would stay in touch to flesh out the idea JGrasstools integration in ZOO-WPS? Via the OSGEO-SOC mailing list, JGrasstools or ZOO-Project mailing lists :-) ? That said, personal emails are also possible, but I think not encouraged, right?

Cheers,

Alex

On Wednesday, 25 February 2015 21:56:25 UTC+13, Alexander Kmoch wrote:

[andrea antonello]

Hi Alexander,

> I got in touch with Gerald Fenoy from the ZOO-WPS project. I'd like to ask
> how best we possibly would stay in touch to flesh out the idea JGrasstools
> integration in ZOO-WPS? Via the OSGEO-SOC mailing list, JGrasstools or
> ZOO-Project mailing lists :-) ? That said, personal emails are also
> possible, but I think not encouraged, right?

Well, yes, usually it is better to go through the mailinglists :-) But
every year I also had a whole pile of private emails. Sometimes it
just seems more appropriate.

Anyways, in this case I think the responsibilities of the two projects
are quite clean. I think we could start in the discussions in the Zoo
list, since some problems need to be fixed there.
In fact it could be started as a Zoo-Project, since a work in that
direction would open up Zoo finally for all java projects (you might
be a hero :-) ). If they do not have an active gsoc project, it might
be simpler to get it.

We should talk about it with Gerald.
What do you think?
Andrea

> --
> You received this message because you are subscribed to the Google Groups
> "jgrasstools" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to jgrasstools...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Alex K]

Hi Andrea,

you might remember me, I am Alex, who was applying for the GSoC 2015
with ZOO and JGrassTools.

I still like the idea to make JGrasstools accessible via a WPS.

So I had a little time to reconcile what I have done in preparation for
GSoC.

One thing was to develop against JGrasstools, to see if the generated
code actually compiles against JGrasstools. And the Java API of ZOO
doesn't really impose any dependencies itself on the java wps processes.
But all the JGrass dependencies of course need to be available in the
class path of the JVM that is started by ZOO.

So I created a fork of JGrasstools and added a "wpsgen" maven submodule,
which makes it easy to integrate the WPS process generating with the
jgrass snapshot modules etc, with changing really anything in the jgrass
modules.

https://github.com/allixender/jgrasstools

https://github.com/allixender/jgrasstools/tree/master/wpsgen

Additionally I looked into generating 52North WPS processes, just
because I am more familiar with those.

I used your ZooGenerator example and built one for 52North WPS

https://github.com/allixender/jgrasstools/blob/master/wpsgen/src/main/java/org/jgrasstools/wps/utils/Generator52N.java

I haven't tested the ZOO ones yet again, but here you can see the
slightly different mapping of the conf/inputs/outputs hashmap that come
from ZOO-Project java API

https://github.com/allixender/jgrasstools/blob/master/wpsgen/src/main/java/org/jgrasstools/wps/utils/RasterReprojectorWps.java

The package structure is basically

package org.jgrasstools.wps.utils (for the generators)
package org.jgrasstools.wps.n52; (where the 52N generated process go)
package org.jgrasstools.wps.zoo; (where the zoo generated classes go)

the zoo config zcfg files go into ./src/main/resources

wpsgen is a proper maven module, so it's nice in the IDE.

I'll try to keep my jgrasstools fork up to date while making the
generation process a bit better.

Main challenge is actually getting the JGrasstools outputs right. I
realised, that output collection from componentaccess can be empty, when
for example the result is a file. The file will be written, where the
input parameter "outfile" says, but the result in the collection is
empty. This is something that the WPS can't handle, so we need some glue
to get the outputs filled with either the resulting filename again or
the file itself or so.

Cheers,
Alex

[andrea antonello]

Hi Alex,

> you might remember me, I am Alex, who was applying for the GSoC 2015 with
> ZOO and JGrassTools.

I sure do :-)

> I still like the idea to make JGrasstools accessible via a WPS.

Which is great!

> So I had a little time to reconcile what I have done in preparation for
> GSoC.
>

[...]

> I haven't tested the ZOO ones yet again, but here you can see the slightly
> different mapping of the conf/inputs/outputs hashmap that come from
> ZOO-Project java API
>
> https://github.com/allixender/jgrasstools/blob/master/wpsgen/src/main/java/org/jgrasstools/wps/utils/RasterReprojectorWps.java
>
> The package structure is basically
>
> package org.jgrasstools.wps.utils (for the generators)
> package org.jgrasstools.wps.n52; (where the 52N generated process go)
> package org.jgrasstools.wps.zoo; (where the zoo generated classes go)
>
> the zoo config zcfg files go into ./src/main/resources
>
> wpsgen is a proper maven module, so it's nice in the IDE.
>
> I'll try to keep my jgrasstools fork up to date while making the generation
> process a bit better.

ok, I will try to look into it. right now we are preparing for Foss4g
Europe and have a lot of stuff at hand, so I might be delaying a
little bit. But I sure need a clear head to look into your work :-)

> Main challenge is actually getting the JGrasstools outputs right. I
> realised, that output collection from componentaccess can be empty, when for
> example the result is a file. The file will be written, where the input
> parameter "outfile" says, but the result in the collection is empty. This is

That is strange, if I am not mistaken, if the output collection is
empty, it should not write an empty file. There should even be an
output message telling you so? Can you supply an example to better
understand?

> something that the WPS can't handle, so we need some glue to get the outputs
> filled with either the resulting filename again or the file itself or so.

I noticed that problem on my first tests.

As you might have noticed, the "modules" project is a wrapper to all
the available modules. It handles simple parameters and inputs/outputs
only in file path format. Would those work at the moment? Sure, that
means that intermediate data will be written to disk and read again,
which is not the best.

Andrea



>
> Cheers,
> Alex

[Alex Kmoch]

Hi Andrea,

sorry for late replay, I am also quite busy :-)

I will look into the outputs collection. Yes, the modules wrapper seems
a quite convenient solution for now.

Regarding outputs, I believe that if the result is a file, the parameter
might be called outfile and already would have to be mentioned in the
input collection, to tell jgrass where to write the resulting file, right?

However, as I/jgrass then already knows where the result goes, the
output collection map is empty, as no computed value needs to be
returned (at least that's my suspicion). If the output is a computed
value, like a double computed as a Mean over a raster or so, then I bet
that value would be returned in the output collection. But I didn't test
these two cases enough yet.

What I can say, in both cases starting the jgrass process via WPS
provided input paramters works already anyway :-)

But officially the WPS process doesn't complete successfully (in both
cases ZOO and 52N WPS) because there were no proper results returned for
the wrapper (ZOO or 52N) to read and return to the user.

But as said, I have to test a little more to confirm my suspicion. And
then I could just add some logic based on the FILE_UI things to handle
output files appropriately.

Now that I am understanding JGrass modules wrapper and the WPS servers a
little better, might even hardcode a few things and provide the ZOO
classes and 52N WPS algorithm classes as full as separate packages and
use the generator only for the first raw skeleton generations?

Keep in touch.

Cheers,
Alex