[opentreemap-user] Customization of OpenTreeMap for India and other locations

[Robert Cheetham]

Jitendra,

I am the president/ceo of Azavea, the company that has been doing much of the development work on OpenTreeMap over the past two years.  I do not respond to the list much, but I really appreciate your detailed and helpful suggestions with the database schema and feature suggestions.

First, let me start out by saying that we share your frustrations with the limitations of the database structure as well as the difficulty of implementing the software with a language other than English.  Some people on the list may be aware of the history behind OpenTreeMap, but for those who are not, I'll try to give a brief rundown.

OpenTreeMap was not originally developed in a way that would enable it to be implemented in any arbitrary location.  The original version was implemented through the hard work of Josh Livni, Kelaine Vargas and others to support the UrbanForestMap project in San Francisco.  Not knowing about this work, Azavea had applied for a Small Business Innovation Research (SBIR) grant from the US Dept of Agriculture and was awarded a "Phase 1" feasibility grant to create a prototype that would support crowdsourced tree inventories while overcoming some of the data quality issues that had plagued previous efforts.

UrbanForestMap was released to the public about the time we were informed we had won this feasibility grant from USDA.  The San Francisco project had many of the core features that we knew we would need to eventually build, but did not overlap with the technology development that we were going to do for USDA.  So, rather than build rival systems, we reached out to Kelaine and asked if she would be willing to collaborate with us in order to extend what she and Josh were developing.  Fortunately for all concerned, she agreed to join forces.  Azavea set out to accomplish several things:

 -> Replace the GoogleMaps API requirement with an open source mapping system while not removing the ability to use GoogleMaps - we went with OpenLayers

 -> Simplify the mapping system to only use one map server (GeoServer)

 -> Retain the overall database structure but extend it with some new fields

 -> Implement some quality control tools for data managers

 -> Implement some data import tools for reconciling multiple data sets for the same location

 -> Generalize the software enough to enable it to be used in other North American cities.

 -> Release the software under and open source license (a commitment Kelaine had made to her funders at CalFire)

The result of these efforts was PhillyTreeMap and then the later release of the OpenTreeMap software on GitHub.  In the meantime, both Azavea and other folks have implemented OpenTreeMap instances in many locations, including Sacramento, San Diego, Grand Rapids, Asheville, Washington DC, UK and others.

This was far more than we had originally promised USDA and they rewarded the effort with a Phase II grant that would support a number of additional improvements, including:

 * an API that could be used to support alternative user workflows and integration with other systems - released in summer 2012

 * an iOS version - released in late 2012

 * an Android version - released in July 2013

 * add social media features - released in 2012

 * add competition features

 * implement a more scalable version that could be run on a cloud infrastructure, such as Amazon

 * develop curricular materials for using OpenTreeMap to teach urban ecology in schools - currently under development by Kelaine

We have continued to work with Kelaine on the design of new features as well as sharing the cost of implementing new features not covered by our USDA grant.

The API, iOS and Android work were far more difficult and time-consuming than we expected and this delayed implementation of a version that would support simpler and more scalable deployment scenarios.

In the meantime, we have learned a lot from both implementing the software ourselves for other cities, from people doing field data entry, and from all of the people on this list.  A brief summary of what we have learned includes the following:

 * OTM is complex to implement and has a lot of moving parts

 * While the Django framework supports localization and internationalization, OTM was not designed to support this and does not make this easy

 * It is not easy to implement new fields, and we need a more flexible data structure

 * the current workflow for the web app is ok for a desktop, but it cumbersome on a tablet in the field

 * the application is complex to style

 * very large data sets slow down the application

We have tried to mitigate some of these issues with some better documentation for the installation process, but this doesn't change the underlying structure of the software, and we don't think we can solve these problems easily.

So we have decided to re-architect the application with several objectives in mind:

 * Retain most of the underlying toolkits (Django, Python, PostgreSQL, etc.)

 * Retain the current API design so that we don't have to make significant changes to the iOS and Android apps

 * Re-design the user experience with both faster response times and tablets in mind - in particular to reduce the number of page loads required when making many edits in the field

 * Add support for other languages using the Django features for this

 * Have a few core fields, but support a much more flexible idea of user-defined fields

 * Implement some responsive design features that cause the application to reconfigure its layout based on the size of the screen

 * Make it easier to apply visual styles

 * Improve performance with very large data sets

So we are in the midst of a redesign of the application that will support these features (and others).

We've been developing what we are calling OTM2 in the open, on GitHub, at https://github.com/azavea/OTM2 but we have been reluctant to say much about it on the list because it is definitely not ready for prime time yet.  Our entire team is focused on this new version, and we cannot support answering what we expect will be a lot of questions and problems when people initially try to implement it.  So it's there, but please don't touch it, of, if you do, please understand that it's in a state of flux, and will not be easy to implement right now.

But your messages to the list have caused me to think that this redesign of the application may address at least some of your suggestions, and that you should be aware of this ongoing activity.

We'll likely have more to announce in several weeks, as the software gets closer to release.

Best regards,

Robert

------------------
Robert Cheetham

Azavea  |  340 N 12th St, Ste 402, Philadelphia, PA
chee...@azavea.com  | T 215.701.7713  | F 215.925.2663
Web azavea.com  |  Blog azavea.com/blogs  |  Twitter @rcheetham  and @azavea

Azavea is a B Corporation - we apply geospatial technology to create better communities 

while advancing the state-of-the-art through research. Join us in creating a better world.

On Tue, Aug 20, 2013 at 1:57 AM, jitendra shah <jitu...@gmail.com> wrote:

Dear Andrew
I am attaching the database schema ( tables have been updated and more uptodate schema image will be prepared soon) to clarify what we propose to do in our treemap.

One crucial table of plot or area with geographic details and administrative info will need to be added which will also link to property (tax department, building permission department, development plan department land survey department etc ) database of the corporation..

I had asked a question: 'it it worthwhile?' and I suppose you wondered what I meant.

1. Can we try to abstract from many (all) specific cases and generalise and allow for customization .
2. can Azavea and OTM group (developer +users) on one hand and group with similar interest in India join hands and develop a solution that is more generic if not universal
3. Can we create a document with such intention and then
   
4. can we should we  scout for sponsorship   .

 I expect governments may easily support such an effort

 



On Saturday, August 3, 2013 3:13:13 PM UTC+5:30, jitendra shah wrote:

Here I wish to put forth some comparison of what are functional requirements of a Treemap in Indian context
Some initial steps are visible on
www.treemapindia.in

The OTM has a model that differs from the Indian situation as follows : draft (3Aug2013) by Jitendra Shah

All collaborators are invited to suggest and if required fiercely disagree.

	US	India	Remarks
legal mandate	not known	Under an act of 1975, mandate to carry out census every 5 years. Digital records not part of the mandate as of now. Tree Authority act to regulate tree cutting, not linked to digital records.
Data surveyors	Volunteers	Assigned largely on payment Garden department inspectors Community participants like NGO Academic institutions with college students as NSS participants with some academic reward Volunteers Property owners undertaking their own survey Property owners wishing to apply for tree felling and replantation
Training of surveyors	un-certified, un-trained	To be trained To be certified
Rating of Surveyors	Reputation Model : based on volunteers. is there no hierarchy?	Because tree survey is going to be paid activity, the reputation will be linked to remuneration in terms of rate of remuneration per tree data added. Can have two options decided by a hierarchy appointed/defined by local government garden superintendent horticulture supervisor academic body appointed to certify survey agency selected with experts including IT+Botany Crowdsourcing based on no hierarchy, but only on registered users
Field of survey	Only public areas, like streets	All urban areas, including private plots
Time frames	un-defined	Defined for assignments by authorities
GIS	well developed, to know a location by address in text zip codes or pincodes coincide with administrative boundaries. neighbourhoods based on google map or other sources, defined for all locations	zip codes or pincodes do not coincide with administrative boundaries. Prabhags which are electoral boundaries keep shifting every few years Plot boundaries not digitized while legal mandate requires data of plot owners Poorly developed in most cities. approximated as boundaries of clusters defined using road network , available in many cities with reasonable accuracy which means fitting on google map. names of localities not standardized neighbourhoods poorly defined.
species	a small number	a large number
Eco considerations	Powerful mandate to evaluate eco impact of urban forest. But seen in only a few cities. And does not cover private plots, so incomplete, evolving. i-tree software developed with decades of research to back it,	No mandate. Climate Change is a mission of Ministry of Environment and Forests with little impetus.
Tree characteristics for eco impact	Well documented in Resource Units for different regions of US for all trees in those regions	Not known to many Botanists and forest officers yet.
Objective of urban tree survey	Ecological impact	for preservation of trees helping regulation Make tree data manageable Provide opportunity to students and community in general to assess urban forests identify and promote native trees preserve biodiversity remove nuisance on trees, prevent damage to trees
Language	only english	Need for many indian languages including in English Need to develop tools for localization in indian scripts. Interface needs localization Messages need localization
local names	synonyms not known if documented	Need for documenting and Need for dis-ambiguation
hierarchy	stewards , data collector	Municipal authorities garden department, inspectors syperintendent Tree Officer Tree authority members others survey team survey organization head team lead surveyor Community participant registered un-registered property owner (choosing to self-survey) registered unregistered

Data and code changes expected :

users and role definitions	surveyors, authority municipal survey organization tree authority
app _ server integration ( an android app called treemapindia has been made available on app store ref : www.github.com/pune-lug/treecensus )	registering of users on both login-passwords to be on devices data transport between (to and from) device and server exporting data from device to server in a dump pushing data from dump to database
incorporating changes in app from time to time	As app progresses from version 0.2 to 1.0, the intergration will have to keep pace