GDAL Wrapper

[Will Kearney]

Hey everyone,

I threw together a Julia wrapper for GDAL this weekend: https://github.com/wkearn/GDAL.jl

Right now, it should be able to open a raster and read that raster into a newly-defined Raster type and write that Raster into a file using either of GDAL's two methods (CreateCopy() and Create()).

There are some big holes in the implementation, some of which are detailed in a Todo section in the README file. The goal was really to provide some minimal functionality for Julia interactions with GDAL to get the conversation about geodata going.

One thing I'm a little worried about is the number of pointers that are flying around to prevent the wrapper from having to interact directly with GDAL-defined types for datasets, raster bands, drivers, etc. Maybe someone with a bit more knowledge of C and memory management could tell me if this is particularly dangerous and a better way to deal with these types.

What do you think good next steps would be?

Of course, send me any questions, comments or pull requests you come up with!

Thanks,

Will Kearney

[Fabian Gans]

This is great news,

I also started some work on this that might be useful to you. If you don't want to write out all the ccall's I used Clang.jl to parse the gdal header files. So feel free to grab whatever you need from here: https://github.com/meggart/GDAL-funs.jl

Did you think about visualization options? The more I think about it, I think an easy conversion Raster -> Image would be cool to make use of the functionlity that is already in the Images package. I will look into this.

Fabian

[pablo...@gmail.com]

Hi Will,

Thanks for starting this project. However, it is unclear how to install and use this unregistered package.

I grabed using the Pkg.clone("https://github.com/wkearn/GDAL.jl.git"), and it was added to .julia package folder. But then what... I didn't find anything useful in docs.julialang.org/en/release-0.2/manual/packages/. It reads like you will have access inmediately, but I don't get it (using GDAL or require("GDAL") doesn't work).

julia> using GDAL
ERROR: GDAL not found
 in require at loading.jl:39

julia> require("GDAL")
ERROR: GDAL not found
 in require at loading.jl:39

Just if it helps someone, Pkg functions will work better with git version 1.9 (I was having multiple problems with 1.7, and after upgrade it was fine, at least no errors).

Cheers, Pablo

[J Luis]

Did you think about visualization options? The more I think about it, I think an easy conversion Raster -> Image would be cool to make use of the functionlity that is already in the Images package. I will look into this.

Indeed, this is excellent news. Regarding the visualization, I happen to have a very nice tool that relies a lot on GDAL (and several other libs) but the problem is that it's written entirely on Matlab. I do want to try to migrate it do Julia but it's huge task and requires a lot of time to learn new GUI building (Gtk.jl) techniques that are themselves still under development.
If any of you would consider this interesting and feel like trying to start I would be more than pleased.
And the tool:
http://w3.ualg.pt/~jluis/mirone

Joaqum

[Fabian Gans]

I just gave your Matlab tool a try, this is a great tool and would most of our problems would be solved to have something equivalent in Julia. I don't know if the GUI is really necessary, just providing the functionality would already be a big step, on which a GUI could be built later.

Fabian

[J Luis]

Fabian, I'm not sure I follow you. Mirone is built around the fact that it's a GUI tool with many functions that don't even have return values and communicate via the handles structure and/or appdata. What useful functionality are referring that could be detached from the GUI and to be used in julia?

[Mauro]

From my point of view (and I think that is what Fabian means too), the
important bit will be to provide all the functions necessary to put
together processing pipelines to work with and transform geodata.

A GUI is a secondary step which would use those functions underneath.
In fact, nicest would be to eventually have a GUI tool which can
translate ones GUI-actions into a (readable) script. Running this
script would then exactly replicate what was GUI-ed. This would ensure
reproducability.

[J Luis]

I agree on the importance of command line tools but my reply was more to the question of the visualization. Another option to "visualization by command line" would be GMT. It has the ability to call GDAL directly to read data and it can produce very nice maps. However, GMT is more directed to grid manipulations and images are not treated so efficiently. I t happens that I'm trying to see if I could have some minimalist GMT wrapper but I'm not getting away from the point where I crash Julia.
Unfortunately, I also failed in the attempts to install Clang.jl on both Win and Mac so I can't on its help for the task.
Still another source to consider, though I never looked closer to their contents, are the python scripts that GDAL uses to several tasks/tests.

[Mauro]

For visualisation, maybe a first step could be to use basemap, a
matplotlib toolbox, through pyplot.jl?

On Thu, 2014-03-06 at 20:11, jmf...@gmail.com wrote:
> I agree on the importance of command line tools but my reply was more to
> the question of the visualization. Another option to "visualization by
> command line" would be GMT. It has the ability to call GDAL directly to
> read data and it can produce very nice maps. However, GMT is more directed
> to grid manipulations and images are not treated so efficiently. I t
> happens that I'm trying to see if I could have some minimalist GMT wrapper
> but I'm not getting away from the point where I crash Julia.
> Unfortunately, I also failed in the attempts to install Clang.jl on both
> Win and Mac so I can't on its help for the task.
> Still another source to consider, though I never looked closer to their
> contents, are the python scripts that GDAL uses to several tasks/tests.
>
> Quinta-feira, 6 de Março de 2014 19:54:16 UTC, Mauro escreveu:
>>
>> From my point of view (and I think that is what Fabian means too), the
>> important bit will be to provide all the functions necessary to put
>> together processing pipelines to work with and transform geodata.
>>
>> A GUI is a secondary step which would use those functions underneath.
>> In fact, nicest would be to eventually have a GUI tool which can
>> translate ones GUI-actions into a (readable) script. Running this
>> script would then exactly replicate what was GUI-ed. This would ensure
>> reproducability.
>>

--

[Fabian Gans]

I also thought more of command line tools than of visualization, however, both would be good to have. If it helps anyone I just parsed the gmt4 header files using Clang, you can find them here : https://github.com/meggart/GMT-julia-headers completely untested. If you want gmt5 rather than gmt4, or some other library, just let me know. Once you have Clang running parsing a library is done in no longer than 10 Minutes.

Fabian

[J Luis]

Fabian, yes please for GMT5. GMT4 is a dead end (I'm one of the GMT developers).

[J Luis]

Sorry, I should have been more specific. GMT5 is a library but the header separation is not totally finished. The API is documented and with declarations in gmt_dev.h and probably some others, while gmt.h is for much more functions, also exported by the shared lib, but are mostly undocumented (though very useful).

A difficulty (to my reduced julia knowledge) with GMT5 API is that it relies a lot on opaque structures that one might need to 'open' in the julia side to interact with.

[Fabian Gans]

Hello,

Ok, some progress on GMT5 wrappers. I wrapped the new GMT5 gmt.h which is documented here: http://gmt.soest.hawaii.edu/doc/5.1.0/GMT_API.html#the-gmt-c-application-program-interface.  and these functions should be callable now. Also the C structs are translated to julia types and should be accessible from within julia. Just have a look at the new version: https://github.com/meggart/GMT-julia-headers.

Regarding gmt-dev.h: Where can I find the documentation of this API? Possibly with examples so that testing would be easier...
I did not search extensively, could you provide a link or place to start reading?

Fabian

[J Luis]

OK, I had a very small success in using them but several issues:

First, and this is a general problem with wrappers, the shared libraries often do not have the same names in unix and Windows. For example in GMT the DLLs are called `gmt_w32` and `gmt_w64` while on unix the name is `libgmt`. So I tried to replace in `gmt.jl`

```
lib = "gmt_w64"
@c Ptr{None} GMT_Create_Session (Ptr{Uint8}, Uint32, Uint32, Ptr{Void}) lib
```

but got error

ERROR: error compiling GMT_Create_Session: could not load module lib: The specified module could not be found.

so the macro c is not able to ingest the shared library name as variable? Strange. Any idea for a workaround?

I noticed also that, at least the

```
@c Ptr{None} GMT_Create_Data (Ptr{None}, Uint32, Uint32, Uint32, Ptr{uint64_t} ...
```
but `uint64_t`is not known. I had to change it to `Culonglong`

Regarding the documentation, there is more than the link you posted which is intended to document the gmt_dev.h (the gmt.h is for lower level functions).

I don't remember if these test codes are shipped with the official releases but if not you can find them (there just a couple more)

http://gmtrac.soest.hawaii.edu/projects/gmt/repository/changes/trunk/src/testapi.c

longer examples are the modules themselves, but they are too complex for the moment.

With a small GMT.jl module wrapper I managed to had these to run and not error

```
using GMT

API=GMT_Create_Session("test",uint32(2), uint32(0), C_NULL)
dim=uint64([2,9])
V = GMT_Create_Data (API, 5, 1, 0, dim, C_NULL, C_NULL, 0, 0, C_NULL)
```