Considerations regarding raster formats in GeoServer/Geonode
Ole Nielsen
Hi all
Thanks for discussing this topic as it is very important both for Risiko and Tsudat. The Arc ASCII format is probably the most common raster format around as it is what is produces by ESRI products and also many scientific modelling packages (in order to be compatible with ESRI). This includes ANUGA, python-FALL3d and others.
Being able to easily use the ASCII format with Risiko & TsuDAT is therefore of critical importance to both projects.
As you know, in Risiko, we do the conversion from ASCII to GTIFF by shelling out and using the GDAL command line utilities. The reason for this is that the AAIGRID driver in the GDAL bindings doesn't seem to work: In the Risiko code where rasters are written (and which works for GTIFF) we get the following error when replacing the driver:
Got driver <osgeo.gdal.Driver; proxy of <Swig Object of type 'GDALDriverShadow *' at 0x9e199c8> >
ERROR 6: GDALDriver::Create() ... no create method implemented for this format.
This would suggest that the driver is recognized, but not fully implemented yet. I have included the pertinent bits of code and the full stack trace below.
Options, in my mind therefore include
· Accept that we have to ‘shell-out’ for the time being. As we in Risiko only care for Ubuntu or Mac OSX this is not a show stopper.
· Contribute to GDAL by finishing the AAIGRID driver
· Write our own routine for writing ASCII grids. I have done this many times before in other projects and have the code at hand. It is important to remember to also generate the .prj file for this format, but that is not hard either.
I’d be interested in everyone’s thoughts on these options, but I can say that I am not one for letting standards getting in my way.
Second issue, which is far more sinister in my mind, is the notion that reprojection is from what I can gather prohibitively slow in GeoServer. The design of Risiko relies on GeoServer to take care of file formats, spatial references, visualization and interoperability.
If GeoServer does not provide reprojection on-the-fly adequately we really need to rethink our design. For example, to do reprojections in Risiko and Tsudat using GDAL.
A related issue that I have observed a few times, but never really nailed yet is that GeoServer tends to provide data with slightly different geo_transforms (bounding box and resolution) than what is requested. This would not present a problem for visualization, but sure as eggs does when it comes to scientific calculations. I am working to create a reproducible example of this.
Again, I’d welcome your thoughts.
Cheers and many thanks
Ole
(and don’t forget the code and stack trace snippets at the very end of this mail J)
-----Original Message-----
From: Jeffrey Johnson [mailto:jjoh...@opengeo.org]
Sent: Wednesday, 1 June 2011 4:38 AM
To: Ariel Nunez; Ole Nielsen
Subject: Fwd: Fwd: Raster uploader for Geonode
Maybe you guys can fill him in on the problems you have had using
GeoServer for this kind of thing?
---------- Forwarded message ----------
From: Jeffrey Johnson <jjoh...@opengeo.org>
Date: Tue, May 31, 2011 at 2:21 PM
Subject: Re: Fwd: Raster uploader for Geonode
To: Andreas Hocevar <ahoc...@opengeo.org>
Andreas,
There are a whole number of reasons why GeoServer wont do what we
need, first of which is that it doesnt support ASC files natively very
well. Ole, Ariel and I have discussed this extensively and all decided
rather than trying to fight with geoserver, its easier just to have a
policy of storing and retrieving ALL rasters in GeoServer as 4326
GeoTIFFs and doing what we need with GDAL ... and once things actually
work, we can go back and see whats up with GeoServer and perhaps take
our issues to the GeoServer folks.
I can say that the largest blocker for me is that if the layers are
stored projected in GeoServer, the WCS support for requesting them
back in another projection is very very very slow (even with JAI) and
there is a problem that you have to manually covert the units in your
application because the WCS will advertise the resx and resx in one
unit and if the request projection is in another unit, geoserver
cannot (or does not) provide the ability to do the conversion.
As much as I understand our organizations reasons for wanting to do
everything that can be done with GeoServer and with 'standards' ...
there are often customer use cases and timelines where its just not
feasible and instead the most efficacious approach is the one taken.
Jeff
On Tue, May 31, 2011 at 2:04 PM, Andreas Hocevar <ahoc...@opengeo.org> wrote:
> Hey Jeff,
>
> what are you trying to reproject with GDAL? GeoServer will do all
> reprojection for you, you just have to tell it the native SRS of the
> raster.
>
> Andreas.
>
> On Tue, May 31, 2011 at 6:18 PM, Jeffrey Johnson <jjoh...@opengeo.org> wrote:
>> Ole,
>>
>> I think we can hook it up to the UI in standard GeoNode, but Im also
>> thinking to add an upload button/form etc into the "Add Data" dialog
>> in the actual TsuDAT UI
>> (https://img.skitch.com/20110208-e42uwfyxk1mjsapp6gwf3krsju.medium.jpg)
>> ... but need to coordinate with Andreas for that.
>>
>> My hesitation/delay in hooking it up to the normal upload UI in
>> GeoNode is that I havent thought of a good way to do that without
>> modifying GeoNode itself, and I've really tried to follow Ariels lead
>> and do all mods outside of the geonode tree so its easily upgradeable.
>> Ariel, do you have any bright ideas here?
>>
>> Also, I'm actually more concerned about the transfer itself timing out
>> or otherwise failing on very large uploads. Im not sure we have a good
>> solution for that at this point. We discussed this with the folks at
>> NCI last week and they pointed me at this solution
>> http://monalisa.cern.ch/FDT/ ... but I dont see any way of hooking
>> that up in the browser. Perhaps we could/should make some kind of Java
>> Web Start uploader that used this tech on the back end.
>>
>> Also, I wrote (what I thought was) most of the code to do the
>> reprojection/conversion with the gdal python bindings, but it didnt
>> quite work, and I just put it aside. At some point I can take it up
>> with folks in the #gdal channel and see if they can figure out what Im
>> doing wrong. That way we can do it without shelling out.
>>
>> Jeff
>>
>> On Sat, May 28, 2011 at 6:41 AM, Ole Nielsen
>> <ole.molle...@gmail.com> wrote:
>>> Got it. Would you be able to hook the UI upload of elevation data to the
>>> Risiko function?
>>> It is just a simple matter of doing the conversion but the way we do it is
>>> shelling out, which means it is not platform independent. For Risiko that is
>>> not and issue but this approach was not considered appropriate for GeoNode.
>>>
>>> But if we know that TsuDAT will alway be deployed on Unix/Linux it would be
>>> OK too, right?
>>>
>>> Cheers
>>> Ole
>>>
>>> On Sat, May 28, 2011 at 12:02 PM, Nick Horspool <nick.h...@gmail.com>
>>> wrote:
>>>>
>>>> Just to get everyone on the same wavelength, there are two places the
>>>> ascii files are handled. The first is in the UI when the user uploads their
>>>> high res elevation data. At present the geonode uploader only allows
>>>> geotiffs. We would like to upload asciis and other formats. This elevation
>>>> data gets converted to asciis then bundled up and sent to ANUGA for the
>>>> simulation.
>>>>
>>>> The second time asciis are handled is what Ole is referring to, getting
>>>> the results from anuga (which is the max inundation depth etc). Jeff has
>>>> written code to convert from ascii to geotiff which then automatically gets
>>>> pushed to the geonode.
>>>>
>>>> The main problem is the first example where we need to upload elevation
>>>> data in non-geotiff format.
>>>>
>>>> Nick
>>>>
>>>> On Sat, May 28, 2011 at 12:36 PM, Ole Nielsen
>>>> <ole.molle...@gmail.com> wrote:
>>>>>
>>>>> OK - I see.
>>>>> I am not sure how you want that to work, though.
>>>>> Can you tell me a bit about the tsudat workflow from the time the ANUGA
>>>>> runscript has been generated till the final result, which I assume is an
>>>>> ASCII file with max inundation depth (plus optionally other quantities) is
>>>>> available.
>>>>> I probably thought the output would be uploaded programmatically to the
>>>>> geonode that I assume is available within tsudat and that could be done with
>>>>> the code I pointed to.
>>>>> But, best if you outline the workflow.
>>>>> Thanks
>>>>> Ole
>>>>>
>>>>> On Sat, May 28, 2011 at 7:25 AM, Jeffrey Johnson <jjoh...@opengeo.org>
>>>>> wrote:
>>>>>>
>>>>>> Ole,
>>>>>>
>>>>>> we are planning to use this same basic concept of conversion and
>>>>>> reprojection in tsudat, but what Nick is asking about, and what we've
>>>>>> discussed is how this is to be handled in the UI .. specifically when the
>>>>>> files are very large.
>>>>>>
>>>>>> Jeff
>>>>>>
>>>>>> On May 28, 2011 8:31 AM, "Ole Nielsen" <ole.molle...@gmail.com>
>>>>>> wrote:
>>>>>> > Copy for the general tsudat mailing list
>>>>>> > O
>>>>>> >
>>>>>> > ---------- Forwarded message ----------
>>>>>> > From: Ole Nielsen <ole.molle...@gmail.com>
>>>>>> > Date: Sat, May 28, 2011 at 5:30 AM
>>>>>> > Subject: Re: Raster uploader for Geonode
>>>>>> > To: Nick Horspool <nick.h...@gmail.com>
>>>>>> >
>>>>>> >
>>>>>> > Thanks for this comment.
>>>>>> >
>>>>>> > I can confirm that the upload of ASCII files have been available in
>>>>>> > some
>>>>>> > form in all incarnations of Risiko pretty much from day one as it is
>>>>>> > probably the most important format for the kind of work we do. Our
>>>>>> > upload
>>>>>> > simple does the conversion and then uploads it as a GeoTiff. If there
>>>>>> > is no
>>>>>> > associated prj file an Exception is raised.
>>>>>> >
>>>>>> > The code for this save_to_geonode line 188 in
>>>>>> > https://github.com/AIFDR/riab/blob/master/risiko/utilities.py
>>>>>> > and the tests are in
>>>>>> >
>>>>>> > https://github.com/AIFDR/riab/blob/master/risiko/tests/test_utilities.py
>>>>>> >
>>>>>> > Have a look and let's discuss how we can do this in both projects.
>>>>>> > Gotta run as my daughter just woke up and wants my undivided attention
>>>>>> > after
>>>>>> > two weeks awaye
>>>>>> >
>>>>>> > Chers
>>>>>> > O
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> > On Thu, May 26, 2011 at 6:26 AM, Nick Horspool
>>>>>> > <nick.h...@gmail.com>wrote:
>>>>>> >
>>>>>> >> Ole,
>>>>>> >>
>>>>>> >> As you are aware the Geonode uploader at present only allows the user
>>>>>> >> to
>>>>>> >> upload raster files as geotiffs. However for both Risiko and TsuDAT
>>>>>> >> we
>>>>>> >> require other formats to be uploaded (e.g esri asc, etc). In addition
>>>>>> >> it
>>>>>> >> appears that for large files we may need another way to do it to
>>>>>> >> handle
>>>>>> >> larger files so the http connection doesnt time-out.
>>>>>> >>
>>>>>> >> I was wondering if the Risiko dev team has considered this or started
>>>>>> >> building any such functionality.
>>>>>> >>
>>>>>> >> I see this task as critical to both projects and would like to
>>>>>> >> discuss
>>>>>> >> further with you.
>>>>>> >>
>>>>>> >> Cheers
>>>>>> >> Nick
>>>>>> >>
>>>>>
>>>>
>>>
>>>
>>
>
>
>
> --
> Andreas Hocevar
> OpenGeo - http://opengeo.org/
> Expert service straight from the developers.
>
=========== CODE SNIPPET FOR WRITING ASCII FILES =======================
...
# Get raster
data
A = self.get_data()
# Get Dimensions. Note numpy and
Gdal swap order
N, M = A.shape
# Create empty file
driver =
gdal.GetDriverByName(format)
print
print 'Got driver', driver
fid = driver.Create(filename, M, N,
1, gdal.GDT_Float64)
if fid is None:
msg = ('Gdal
could not create filename %s using '
'format %s' % (filename, format))
raise
Exception(msg)
# Write metada
fid.SetProjection(str(self.projection))
fid.SetGeoTransform(self.geotransform)
# Write data
fid.GetRasterBand(1).WriteArray(A)
===================== ERROR MESSAGES (first one is using TIF, the second ASC) ===========
(riab_env)nielso@risiko:~/dev/riab/impact/tests$ python test_io.py
Rasters can be read and written correctly in different formats ...
Got driver <osgeo.gdal.Driver; proxy of <Swig Object of type
'GDALDriverShadow *' at 0x9e199b0> >
Got driver <osgeo.gdal.Driver; proxy of <Swig Object of type
'GDALDriverShadow *' at 0x9e199c8> >
ERROR 6: GDALDriver::Create() ... no create method implemented for this format.
ERROR
======================================================================
ERROR: Rasters can be read and written correctly in different formats
----------------------------------------------------------------------
Traceback (most recent call last):
File "test_io.py", line 381, in test_reading_and_writing_of_real_rasters
out_filename)
File "/home/nielso/dev/riab/impact/storage/io.py", line 52, in
write_raster_data
R.write_to_file(filename)
File "/home/nielso/dev/riab/impact/storage/raster.py", line
198, in write_to_file
raise Exception(msg)
Exception: Gdal could not create filename /tmp/tmpNzIQ2G.asc using format
AAIGRID