First experiments with transformation plugins

[Jochen Topf]

Hi!

I have started implementing generic "transformation plugins". Just like
datasources they are implemented as shared libraries and are configured with
parameters that depend on the plugin. Each layer can have zero or more of those
transformations configured. After features have been read from the data source
and before they are rendered they are sent through all configured
transformations. Transformations can optionally return false to indicate that
this feature should be ignored alltogether, so they can also work as filters.

I have implemented the framework and two transformation plugins, one called
"foo" is just a very simple demo, it just changes all text in an attribute to
"foo". The other called "icu" allows any kind of text transformations that the
icu library provides, this includes changing text to uppercase, lowercase,
and titlecase, but also transliterations, say from latin to greek letters.
The c++ demo program shows you how to use it.

I have not yet implemented the XML reading or writing and there are no Python
bindings yet.

The code is here: https://github.com/joto/mapnik

What do you think? Does this sound useful? Is this the right interface?

Jochen
--
Jochen Topf joc...@remote.org http://www.remote.org/jochen/ +49-721-388298

[Artem Pavlenko]

Hi Jochen,

Looks very cool! Interface is very simple, too.
Artem

[Jochen Topf]

I have added Python bindings now and support for loading and saving XML files.

It mostly seems to work, but I can't figure out one problem: When calling
everything from C++ in demo/c++/rundemo it works. But when I try to run the
tests, the "load_map_test.py" test fails on me if I enable the "icu" plugin
in tests/data/good_maps/transformations.xml:

python: symbol lookup error: /usr/local/lib/mapnik/transform/icu.transform: undefined symbol: _ZN6icu_4414Transliterator14createInstanceERKNS_13UnicodeStringE15UTransDirectionR10UErrorCode

The icu.transform lib does need the icu lib and it does ask for it:
ldd /usr/local/lib/mapnik/transform/icu.transform
...
libicuuc.so.44 => /usr/lib/libicuuc.so.44 (0x00007fee7ed44000)
...

And that works fine in C++. But I guess when calling from Python there is
something missing.

Jochen

On Tue, Dec 18, 2012 at 10:27:02AM +0000, Artem Pavlenko wrote:
> Date: Tue, 18 Dec 2012 10:27:02 +0000
> From: Artem Pavlenko <ar...@mapnik.org>
> To: "map...@googlegroups.com" <map...@googlegroups.com>
> Subject: Re: [mapnik] First experiments with transformation plugins

[Dane Springmeyer]

On Dec 18, 2012, at 12:59 PM, Jochen Topf wrote:

> python: symbol lookup error: /usr/local/lib/mapnik/transform/icu.transform: undefined symbol: _ZN6icu_4414Transliterator14createInstanceERKNS_13UnicodeStringE15UTransDirectionR10UErrorCode
>
> The icu.transform lib does need the icu lib and it does ask for it:
> ldd /usr/local/lib/mapnik/transform/icu.transform
> ...
> libicuuc.so.44 => /usr/lib/libicuuc.so.44 (0x00007fee7ed44000)
> ...
>
> And that works fine in C++. But I guess when calling from Python there is
> something missing.

maybe try linking the plugin explicitly to -licui18n?

[Jochen Topf]

Ah, that was it. Silly me. Should have checked that that function was not in
icuuc but in icui18n. Who knew that these jumbled letters all actually mean
something. :-)

[Jochen Topf]

So, I did some cleanups etc. and the current state of the transformation
plugins code doesn't look too bad. But there are some questions that turned
up:

* What about not only changing existing attributes, but adding new attributes
(or remove existing ones)? I tried this, but get an error message. It seems
I can add a new attribute to the feature (with put_new()), but the expression
parser complains that this attribute is not in the datasource. Adding new
attributes could be very useful, I think we should support this. Either we
have to somehow associate the information with attributes are available
with the transformations and change existing code to check that instead of
the datasource. Or we have to ignore all this and just dynamically see
what attributes are there and just return empty string or whatever when
non-existing attributes are accessed.

* What about geometries on vector layers? I have only manipulated attributes
so far? There are already other places in Mapnik that do geometry
transformations. Maybe they should be in these plugins, too? Instead?
Do we need extra checks, say that datasource, transformation, style all
fit together with regards to the geometry type?

* What about raster layers? I haven't taken them into account at all. Do
we need transformation plugins for raster layers? Maybe have two different
classes of transformations, one for vector and one for raster layers?
Should the code make sure that you can't add a raster transformation to
a vector layer or so?

Any thoughts on these issues?

[Jochen Topf]

On Wed, Dec 19, 2012 at 05:14:26PM +0100, Jochen Topf wrote:
> * What about not only changing existing attributes, but adding new attributes
> (or remove existing ones)? I tried this, but get an error message. It seems
> I can add a new attribute to the feature (with put_new()), but the expression
> parser complains that this attribute is not in the datasource. Adding new
> attributes could be very useful, I think we should support this. Either we
> have to somehow associate the information with attributes are available
> with the transformations and change existing code to check that instead of
> the datasource. Or we have to ignore all this and just dynamically see
> what attributes are there and just return empty string or whatever when
> non-existing attributes are accessed.

Somewhat related to this issue is the following idea: Many people now use
osm2pgsql with hstore support to render OSM maps. This allows for more
flexibility, but is sometimes a bit cumbersome, because to access attributes
you'll have to use the hstore syntax like tags->"highway" or whatever in your
SQL queries or add views to your database.

If attributes on each feature are completely dynamic, we could add support to
the postgis input driver for this. The driver could unpack the hstore column(s)
for us automatically and always return all the attributes there are. In many
cases this would mean SQL queries would only look like "SELECT geom, tags FROM
...", no attribute list needed. (Well, there could be a performance impact in
this case, because more attributes are transmitted than actually needed, but
I don't expect this to be actually that large. Of course we'd have to benchmark
it, though.)