GPX logging extensions

[Marko Mäkelä]

Hi,

I joined the OpenStreetMap project when I got my first Garmin Edge 705, my first GPS. Last November, it partly broke and I got a SonyEricsson Xperia Active, possibly the only outdoor-proof Android device that supports ANT+Sport sensors.

I would like to improve the GPX recording capabilities of OsmAnd. So far, I have recorded a few traces with MyTracks, but I do not like the bundling to Google services nor the way the ANT+Sport support has been implemented (for example, it throws away the speed data from the wheel sensor).

I would like to log as much raw data from my rides as possible. For example, in areas with bad GPS reception, it might be useful to compute an alternative GPX file from the compass reading and the wheel sensor pulses. The wheel circumference and the compass deviation could be calibrated on the longest straight run of the trace. The GPX XML schema allows custom extensions, so technically it should be possible to record things like the following:

• barometric pressure (for more accurate elevation data)
• compass bearing
• accelerometer data (for determining the smoothness of the surface when the devices is mounted on bicycle handlebars)
• ANT+Sport sensor data (mainly for applications like MyTourbook)
• temperature
• battery level

Some of these settings would require some user interface. One might want to enable or disable these extra attributes on a case-by-case basis. Also, one might want to display some of these on the screen, similar how the configureable data screens work on the Garmin Edge 705 and 800. From a thread about a year ago, I got the impression that there is some reluctance to turn OsmAnd into ‘everything and the kitchen sink.’ If that is still the case (and if OsmAnd only saves GPX while displaying the map), I understand and will try to transform My Tracks into my preferred direction.

With best regards,

 Marko Mäkelä (OSM user skela)

[Pavol Zibrita]

Hi Marko!

   If I understand it, your proposal is it that OsmAnd could support ANT+ devices and when logging gpx track, it could log also data from this devices.
A kind of feature request. Did I got it right?

Best regards,
Pavol

[Marko Mäkelä]

Hi Pavol,

Yes, this is a kind of a feature request that I would be willing to implement myself, if there is support for this idea. This is not only about ANT+Sport sensors, but also about the built-in sensors in many devices: temperature, pressure, battery voltage (or current), compass, accelerometer. ANT+Sport sensors support things like heart rate, pedal cadence, wheel rotation speed, pedalling power.

Does OsmAnd currently record the GPX trace while it is not displaying the map? I think that I would prefer a background service, like the one that exists in My Tracks.

Best regards,
Marko

2012/3/26 Pavol Zibrita <pavol....@gmail.com>

[Pavol Zibrita]

Hi!

   OsmAnd has background service. You will find it in settings. And what do you mean by that you are willing to
implement it if there is support. Support from whom? Developers, users, what kind of support do you have in mind?

Best regards,
Pavol

[Marko Mäkelä]

I mean, if a core developer shots down the idea and says ‘we are never going to merge this feature’ then I will not bother. I would be doing this as a hobby project, as my first Android development exercise. I write C and C++ for living.

Thanks Pavol, I found the background service and it looks useable. Most of the needed infrastructure seems to be in place.

[Pavol Zibrita]

Hi!

   I understand. For this reason, we wanted to create a plugin capability for Osmand. So that such features as
this one, for example, would be a separate product that can be installed to Osmand, and people who want it
will install it. You can imagine, that implementing features from all areas in OsmAnd would probably make
it harder to use, bigger, and also less maintainable in future.
   You always can create your fork of OsmAnd and maintain it, and people could use it if they want ANT+ and similar
features, we can at least support you by creating daily builds on the build server. And if it is ok, it might be merged, or the plugin creation would be great also. However, I cannot tell what the future will bring (fortunately :-) ).
   Probably Victor could tell if such feature will for sure not be merged.

Best regards,
Pavol

[Marko Mäkelä]

Pavol,

Sorry for taking so long to respond, but I was extremely busy at work
until last week. Now I finally have summer vacation.

Today, I updated to OsmAnd 0.8 (actually the latest nightly build, and
then downloaded vector maps for offline use). It feels great. Bicycle
routing could be better, but it is already mostly on par with my
Garmin Edge 705, which will be facing retirement.

Can you explain me how the GPX trace collection currently works in
OsmAnd? I did not find any related menu item where to enable or
disable it. Is there already a plugin for logging GPX, and is there a
document explaining how to build OsmAnd? I am familiar with
distributed version control systems (although not git) and Java ant.

I think that it would be easiest for me to fork an existing plugin and
add extra logging to it (written as GPX extensions, which the file
format does allow). For starters, I would log things like temperature,
barometric pressure, battery charge level, ANT+Sport sensor packets,
mainly for postprocessing. Later, there could be a UI for selecting
what to record, and for viewing heart rate, cadence and so on, similar
to the configureable "bicycle computer" screens on the Garmin Edge
series.

Marko

[Onkar Shinde]

For GPX logging you have to go to Settings -> Plugin Manager and enable 'Tracking'.
After that reopen the settings Ui and you will see a 'Tracking' section where you can choose if you want to log your track and also dump current logged track. Please note that tracking preferences are profile specific.

[Pavol Zibrita]

Hi!

  I updated this page, I hope it is more clear now, about the dev environment.
https://code.google.com/p/osmand/wiki/InstallDevelopmentEnvironment

Best regards,
Pavol

[Gints Polis]

May be it is worth to mention recomended Eclipse startup parameters. With default one my build did not work.

Gints.

2012/7/4 Pavol Zibrita <pavol....@gmail.com>

--
Ginc

[Pavol Zibrita]

Hi, good point, I will mentioned it there.

Best regards,
Pavol

[Marko Mäkelä]

Sorry for being so slow, but I guess I have to admit that my day job as C and C++ programmer is exhausting me.

I would like to get rid of MyTracks and use OsmAnd for GPX logging. But, I would like to record some extra attributes. I do not care about real-time display; it is enough to have the extra data in the GPX file.

I was wondering if this is doable with Android intentions. When recording, OsmAnd would send an event 'gpxext' every 1s. If any logging extension apps are enabled, they would respond with a string that is to be stored with the next GPS sample in the extension tag.

Or would this be better doable with some plugin and registering cal

[Marko Mäkelä]

Sorry, I used the Android gmail client, and it looks like it send a
partial draft instead of the complete message that I intended to send.
Trying again.

Sorry for being so slow, but I guess I have to admit that my day job
as C and C++ programmer is exhausting me.

I would like to get rid of MyTracks and use OsmAnd for GPX logging.
But, I would like to record some extra attributes. I do not care about
real-time display; it is enough to have the extra data in the GPX
file.

I was wondering if this is doable with Android intentions. When
recording, OsmAnd would send an event 'gpxext' every 1s. If any
logging extension apps are enabled, they would respond with a string
that is to be stored with the next GPS sample in the extension tag.

Or would this be better doable with some plugin and registering

callback functions? I would appreciate some pointers. I do have some
Java programming experience, but I am new to Android.

Ideally, there would be an easy framework that would allow anyone with
a little knowledge about Java programming to write their own custom
logging plugins. Preferrably, writing and deploying a plugin should be
a matter of installing a small *.apk, not involving a rebuild of the
whole OsmAnd. Maybe this would even work without installing an Eclipse
environment?

Personally, I would like to try recording as much raw extra data as
possible, so that it would be theoretically possible to experiment
with different postprocessing algorithms, to enhance the resolution of
the GPX trace. (For example, try to get better elevation data by
having access to a barometer reference from an external source, or try
to extrapolate GPS samples from compass heading and bicycle wheel
rotation sensor.)

[Pavol Zibrita]

Hi!

  I think answering to all your questions is yes. However, even so, there is probably no one who has time for it here :-/.

So currently, it is more or less on you. The idea of adding some 'filter' that can filter/add values to gpx, using intents, or new application or whatever, is probably the way to go. But I would need also to dig deeper, still some android parts are mystery to me :-)

Best regards,

Pavol

[Marko Mäkelä]

2012/9/7 Pavol Zibrita <pavol....@gmail.com>:

> Hi!
>
> I think answering to all your questions is yes. However, even so, there is
> probably no one who has time for it here :-/.
> So currently, it is more or less on you. The idea of adding some 'filter'
> that can filter/add values to gpx, using intents, or new application or
> whatever, is probably the way to go. But I would need also to dig deeper,
> still some android parts are mystery to me :-)

OK, thanks. I may do this at some future time. I guess it might be
easiest to start by writing a patch to OsmAnd, and then figure out the
decoupling or plugins from that.

On a somewhat related note, the OsmAnd FAQ says that OsmAnd should
listen to this intent:

geo:0,0?q=my+street+address

If I have a location "Aleksanterinkatu 16, Helsinki" in my calendar
and I click it, sure, OsmAnd will fire up, but it will fail to find
the address. Does it expect it in a different format? Omitting the
street number did not help.

I also tried clicking a postal address in the phone contact list, but
that did not do anything. It would be handy if OsmAnd would search for
the address, when I am going to visit a contact. (No need to create
POIs for friends’ addresses.)

Best regards,

Marko

[Rahul Maindargi]

Hi, 

If you on ICS or 2.3.3  you can find the more details at below thread. 

https://groups.google.com/forum/?fromgroups=#!topic/osmand/YNj5y6EPaVU

Osmand-ContactAddressPlugin.apk can be downloaded. 

The address you enter in Contacts is searchable, as long as results are found on "OsmAnd Search" directly.

it support even lat,long format. 

[Marko Mäkelä]

Thanks, I installed the plugin, and now I can click an address on the
contacts. When I do so, I get the same result as with an address
string in the calendar:

OsmAnd shows a gray screen with white text. The title bar has dark
gray background.

Title bar: Osmand
next line: Search address using offline maps

The rest of the screen is empty. Initially, there is a
dark-gray-background pop-up that says ‘Nothing found’

In which format does OsmAnd geo: handler expect the address? How can I
debug this with adb?

Is there a screenshot feature in ICS or adb? (In 2.3.4, you could push
the power button and select it from a menu.)

Marko

2012/9/7 Rahul Maindargi <rahulma...@gmail.com>:

[Marko Mäkelä]

I studied the code a bit. The GPX track points and waypoints are buffered into two tables of a Sqlite3 database in net/osmand/plus/activities/SavingTrackHelper.java.

I was wondering if this class could listen to a new intent, say, 'log:(timestamp):arbitrary data', and write these to a third Sqlite3 table. Then, the GPX file writer would pick any data from the new table and write it to the extension tag of the nearest preceding track point. I guess that the events could be generated a few times per second, depending on how many apps or sensors are generating them.

Is this feasible, or is there too much overhead in Android intents? Could the helper class technically listen to an intent, or would we require another thread for that?

 Marko

[Pavol Zibrita]

I see the point Yes, in deed, this could be probably possible and also maybe not very hard. OsmAnd would define an intent, which could say what data to log in gpx. The part of the intent would be the data, probably if osmand should write it immediately (with latest location) or immediately with actual location, or if it should wait for its log interval and write down the data it has. This way, any number of programs can be sending data to OsmAnd (well, this seems from user perspective a nightmare to setup however).

So the idea of OsmAnd plugin would be, that user can select what to monitor there, and plugin will be sending the data to OsmAnd with intenet, and OsmAnd will store them (or only last entry) until it logs the event down with the current possition.

Something like this?

Best regards,

Pavol

[Marko Mäkelä]

perjantai, 7. syyskuuta 2012 19.57.46 UTC+3 Pavol Zibrita kirjoitti:

I see the point Yes, in deed, this could be probably possible and also maybe not very hard. OsmAnd would define an intent, which could say what data to log in gpx. The part of the intent would be the data, probably if osmand should write it immediately (with latest location) or immediately with actual location, or if it should wait for its log interval and write down the data it has. This way, any number of programs can be sending data to OsmAnd (well, this seems from user perspective a nightmare to setup however).

So the idea of OsmAnd plugin would be, that user can select what to monitor there, and plugin will be sending the data to OsmAnd with intenet, and OsmAnd will store them (or only last entry) until it logs the event down with the current possition.

Something like this?

Yes, this sounds good. The option whether to associate the data with the latest written sample or whether to write it with the next sample is fine-tuning. For my purposes, the 1-second accuracy is probably fine. Besides, the extra data can include more accurate timestamps (in whatever application-specific format) if needed.

One thing that needs some thought is notifications of starting and stopping of recording. For example, ANT+Sport and Bluetooth radio consume battery, and therefore the additional logging plugins cannot be active all the time (when OsmAnd is not listening). Therefore, I guess that OsmAnd should broadcast an event (or intent) when it starts the GPX recording. If we did not do that, the user would have to start&stop recording in multiple places: OsmAnd and in the plugin(s) that are being used.

I guess that this needs some kind of binding (registration) between OsmAnd and the plugins, so that OsmAnd can notify registered plugins when it is starting or stopping a GPX recording. I will have to read the basics in the Android guide and then get back to this.

 Marko

[Pavol Zibrita]

I think there are some broadcast intents as well. Good point. OsmAnd can also send the logging interval interval in the broadcast start intent. Plugins should probably query OsmAnd if it is still recording and if not, stop automatically (in some interval).

Hm, the registration should be actually to intents, or to some specific schema. Can be also to some specific sender/receiver I think.

Best regards,

Pavol

[Marko Mäkelä]

Hi Pavol,

We could also keep the communication simple and unidirectional, if OsmAnd started recording automatically when receiving the extra logging intents. If the intent stream stops for longer than a few seconds, OsmAnd would stop GPX recording (unless it wants to record itself). The user would initiate the logging from the external application(s), and OsmAnd would act as a ‘slave’. When all the external applications are stopped, OsmAnd would stop recording.

Or, the user could initiate the recording from OsmAnd and explicitly start and stop the external application(s) when the extra logging is needed. If OsmAnd does not receive any of these new intents, it would act as it currently does.

The above should be easy enough for an Android novice like me to implement, provided that the intent mechanism is suitable at the planned sample rate. (All examples of sending intents that I saw in the Android guide so far seemed to be something triggered by a direct UI action.)

In the future, it could be nice to have some UI for this, such as some checkboxes in OsmAnd profiles, to enable specific logging plugins (for example, do not attempt to record bicycle sensor or heart rate sensor data when going by car, but do record something else).

 Marko

[Marko Mäkelä]

lauantai, 8. syyskuuta 2012 9.17.32 UTC+3 Marko Mäkelä kirjoitti:

Hi Pavol,

We could also keep the communication simple and unidirectional, if OsmAnd started recording automatically when receiving the extra logging intents.

I have been thinking about this. The Android intents seem to be intended to be used for launching external applications, not for periodic events (or at least I have not found examples of it).

I guess that it should be the other way around. The external application would take care of all logging, and OsmAnd would generate intents when it wants to start or stop recording a GPX track. The user could disable the GPX logging within OsmAnd. The only thing that might be against this arrangement is anecdotal evidence that suggests that some Android devices in the past have had problems when multiple applications have attempted to use the GPS location service concurrently. I have not faced any such issues when using MyTracks and OsmAnd concurrently on my Xperia Active.

Another reason for writing a small logging application is that it might be more stable. If OsmAnd crashes due to some bug (or running out of memory when navigating or calculating a complex route), the GPX logging would hopefully carry on.

I have no ETA for the small logging application. I will have to read more on Android basics, and then maybe start by castrating the most out of MyTracks.

 Marko

[Alexander Horst]

On 13.09.2012 12:57, Marko Mäkelä wrote:
> I have no ETA for the small logging application. I will have to read more
> on Android basics, and then maybe start by castrating the most out of
> MyTracks.

Maybe you should also have a look at Open GPS Tracker:
http://code.google.com/p/open-gpstracker/

[Marko Mäkelä]

2012/9/14 Alexander Horst <arh...@googlemail.com>:

Thanks for the link! It appears to be built from third-party components.

One thing I have been thinking about is a scrollable widget that would
display some ‘bicycle computer’ type data, such as the recorded
distance and time, current speed, rate of ascend/descend, cadence,
heart rate, etc. If this were part of OsmAnd, it could also display
ETA and remaining distance. But, let’s not start dreaming too much
before writing any code. :-)

Marko