GeoNode work for MapStory project

[David Winslow]

Hi all,

As I think many of the developers have already heard, OpenGeo is working on a project called MapStory based on GeoNode.  Feature-wise, we are focusing on adding time support to GeoNode's map composer and data import tools, and improving the look and interactivity of the GeoNode site to be more approachable.  While some of the changes we are making are likely unsuitable for inclusion in GeoNode, I would like to outline the changes we are making to foster some discussion about which ones are worthy candidates.

Code is in OpenGeo's GeoNode fork under the 'mapstory' branch, with a custom theme in a separate repository: 

https://github.com/opengeo/geonode/tree/mapstory/

https://github.com/opengeo/mapstory/

Build/Structural Changes

• The geonode-client subproject is now a submodule rather than a separately published artifact.  This facilitates development of features requiring tight server-side integration (ie, when GeoNode exposes a custom service that the GeoExt client needs to access) and allows versioning of the client along with the rest of GeoNode.  Since our current publishing scheme does not provide versioning, this is effectively necessary for development of GeoNode variants with modifications to the core client code.
  Note: as I recall one reason that we moved geonode-client out of the main repository in the first place was to avoid requiring a third build tool (Ant, along with Paver and Maven) to build GeoNode.  Recent changes in geonode-client allow building with Maven or Ant so this concern at least is addressed.
• GeoNode's "paired" GeoServer now requires an additional dependency - OpenGeo's Importer extension.  This allows the upload process to have a verification/feedback step with the user between completing the actual file upload and having the layer actually published in the GeoNode site.  We are using this step to prompt the user for details about the temporal aspect of his/her data, but it is open-ended for other uses. For example, we could use this to prompt the user when the projection is not detected for a Shapefile.
• GeoServer updated to GeoServer 2.2 (2.2 is currently not released, but we need it for the time-related features it adds to WMS.)

Behavioral Changes

• GeoNode's map composer now configures itself via Ajax request at page load time, rather than having the configuration embedded in the HTML for the page.  This endpoint should be accessible to non-JavaScript clients as well, making for the beginnings of a GeoNode map composition and query API.
• The geonode.maps.models.Map model class now includes 'tool_params' and 'portal_params' fields to accommodate storage of map configuration unrelated to map layers.  For mapstory we're using this to store animation details (what time to display initially, frame rate, step size...) but it is open-ended to accommodate other configurations as well.
• The upload and "replace this layer" forms now use the aforementioned Importer module instead of just the GeoServer REST API.
• Two new ad-hoc endpoints: get_time_info to get some information about the temporal aspect of a layer that is not exposed in the WMS Capabilities, and create_layer to create new layers.  The latter especially is subject to revision; we are using it to support an 'annotation' layer which may be created to add free-hand marks and notes to a GeoNode map.

There are a number of minor tweaks (moving buttons around etc.) that might be reasonable additions as well.  Also on the roadmap for MapStory:

• General editing features
• Visual/usability improvements, especially to search and user profiles
• Customizability improvements, things like the ability to have multiple, user-selected themes within a single GeoNode site.

--

David Winslow

OpenGeo - http://opengeo.org/

[Ariel Nunez]

David,

Thanks for sharing this detailed description of the changes, please find my replies in line below.

Build/Structural Changes

• The geonode-client subproject is now a submodule rather than a separately published artifact.  This facilitates development of features requiring tight server-side integration (ie, when GeoNode exposes a custom service that the GeoExt client needs to access) and allows versioning of the client along with the rest of GeoNode.  Since our current publishing scheme does not provide versioning, this is effectively necessary for development of GeoNode variants with modifications to the core client code.
  Note: as I recall one reason that we moved geonode-client out of the main repository in the first place was to avoid requiring a third build tool (Ant, along with Paver and Maven) to build GeoNode.  Recent changes in geonode-client allow building with Maven or Ant so this concern at least is addressed.

+1 On making the same change in our main repo. 

• GeoNode's "paired" GeoServer now requires an additional dependency - OpenGeo's Importer extension.  This allows the upload process to have a verification/feedback step with the user between completing the actual file upload and having the layer actually published in the GeoNode site.  We are using this step to prompt the user for details about the temporal aspect of his/her data, but it is open-ended for other uses. For example, we could use this to prompt the user when the projection is not detected for a Shapefile.

Is this just a change in the maven profile for geoserver-geonode-dev (adding that extenstion)? Or does it require other steps? Does this mean the restconfig api is not used anymore to create the stores / layers? Would this make it easier to support other GeoServer supported file formats appart from .shp and .tiff?

I would like to know more about this extension. 

• GeoServer updated to GeoServer 2.2 (2.2 is currently not released, but we need it for the time-related features it adds to WMS.)

This does not look like a problem to me, GeoServer's trunk is usually in good shape and when it breaks it is not for more than one day.

 

Behavioral Changes

• GeoNode's map composer now configures itself via Ajax request at page load time, rather than having the configuration embedded in the HTML for the page.  This endpoint should be accessible to non-JavaScript clients as well, making for the beginnings of a GeoNode map composition and query API.

I have heard about this alternative from Andreas before and I believe it makes a lot of sense.

+1 for inclusion in GeoNode.

• The geonode.maps.models.Map model class now includes 'tool_params' and 'portal_params' fields to accommodate storage of map configuration unrelated to map layers.  For mapstory we're using this to store animation details (what time to display initially, frame rate, step size...) but it is open-ended to accommodate other configurations as well.

This may be useful for Risk in a Box as well if we are able to achieve tighter integration with the map composer. i.e. by allowing to just add the 'Risiko' widget to any map composer in order to create new layers based on loaded layers.

I am not sure if we should have these two fields in GeoNode or if we should try to make it easier for downstream projects to achieve the same functionality (adding new utility fields).

I am curious to hear other opinions here.

• The upload and "replace this layer" forms now use the aforementioned Importer module instead of just the GeoServer REST API.

Sounds good, this looks like an opportunity to remove the need to deal with coveragestores and featurestores, etc. Was the new code added to gsconfig or GeoNode? Is it much simpler than the REST API?

 

• Two new ad-hoc endpoints: get_time_info to get some information about the temporal aspect of a layer that is not exposed in the WMS Capabilities, and create_layer to create new layers.  The latter especially is subject to revision; we are using it to support an 'annotation' layer which may be created to add free-hand marks and notes to a GeoNode map.

Is this information about all layers (like getcaps) or per layer (like getlayerdetails)? 

 

There are a number of minor tweaks (moving buttons around etc.) that might be reasonable additions as well.  Also on the roadmap for MapStory:

• General editing features
• Visual/usability improvements, especially to search and user profiles

Great!  

• Customizability improvements, things like the ability to have multiple, user-selected themes within a single GeoNode site.

Both WorldMap and our Events app have done this sort of 'internal' theming. It would be great to start another thread and share the approaches already taken so that we can all benefit from the same implementation in the future.

-Ariel

[David Winslow]

On Mon, Nov 14, 2011 at 5:11 PM, Ariel Nunez <ingenie...@gmail.com> wrote:

Build/Structural Changes

• GeoNode's "paired" GeoServer now requires an additional dependency - OpenGeo's Importer extension.  This allows the upload process to have a verification/feedback step with the user between completing the actual file upload and having the layer actually published in the GeoNode site.  We are using this step to prompt the user for details about the temporal aspect of his/her data, but it is open-ended for other uses. For example, we could use this to prompt the user when the projection is not detected for a Shapefile.

Is this just a change in the maven profile for geoserver-geonode-dev (adding that extenstion)? Or does it require other steps? Does this mean the restconfig api is not used anymore to create the stores / layers? Would this make it easier to support other GeoServer supported file formats appart from .shp and .tiff?

I would like to know more about this extension.

It is both a change in the Maven dependency listing and the Python code in GeoNode that talks to GeoServer.  The whole point of using the Importer extension is that it allows us a more complex workflow than what is provided in RESTConfig - in particular we can get GeoServer to report on how it is interpreting data before we "finalize" the import and put the data into Postgres.  This allows us to do things like let the user select a date format to use when interpreting string fields that contain formatted date information, and eventually preview the data during import (think of MS Excel when you try to load a CSV file and it asks you to verify the separator char/escape conventions/character set.)

It does not especially affect GeoNode's format support, beyond providing one avenue we might use to support more ad-hoc formats such as CSV - while providing a basic upload form for those would basically be unusable, if we can guide the user through a conversion process before importing then it's much more likely to be of use.

While binaries are currently available in the OpenGeo Maven repository (http://repo.opengeo.org/) the source repository is currently password-protected.  If you'd like to get a snapshot of those sources please let me know.

There is also a gsuploader module (similar to gsconfig.py) for interacting with the new service in the mapstory branch of OpenGeo's GeoNode fork.