MapsForge on iOS with J2ObjC

[Matthew Hall]

In short, MapsForge core/map/reader + J2ObjC + rendering code in ObjC = MapsForge on iOS

I've used MapsForge on Android about a year ago and always wished such a lib existed on iOS. For iOS offline maps, I use mbtiles storing PNGs for each zoom of a limited area with limited zoom and still big file sizes. An < 1 MB Android application becomes > 20 MB on iOS for a very small map. Map sizes are 230 KB for MapsForge format vs 11 MB for 3 zoom levels of black and white PNG-8s. I read about J2ObjC this past week and wondered, could I make MapsForge work on iOS.

First off, thank you for architecting MapsForge the way it is. The design made my job so much easier. Using J2ObjC on core, map, and reader worked great, ignoring a J2ObjC bug involving Java arrays with iOS ARC. Only InternalRenderTheme.java had issues. Since it is an enum, J2ObjC transpiles it weirdly. Plus, the way it fetches the internal XML resource doesn't work on iOS. Implementing my own XmlRenderTheme in ObjC worked perfect though. All I have to do is implement mapsforge-map-android for iOS in native code using CoreGraphics. I'm still a ways off from finishing the implementation, but I at least can render a tile! After nearly 20 hours into this, getting a correctly colored tile is cause for celebration.

Progress:

graphics:

Bitmap - 85%

Canvas - 65%

GraphicFactory - 60%

Matrix - 80%

Paint - 80%

Path - 100%

PointTextContainer - 0%

ResourceBitmap - 0%

SvgBitmap - 0%

TileBitmap - 75%

input:

MapZoomControls - 0%

ScaleListener - 0%

TouchEventHandler - 0%

TouchEventListener - 0%

TouchGestureDetector - 0%

layer:

MyLocationOverlay - 0%

rendertheme:

AssetRenderTheme - 75%

view:

MapView - 75%

Preferences - 0%

tile caches:

Have not tested the file based tile cache yet, but the InMemoryCache works great.

tile downloads:

have not tested using external tile services. Still need to implement parts of Bitmap needed for it.

[Ludwig]

Thanks for reporting this, this is quite interesting.

The architecture is indeed meant to be portable to different platforms with an implementation of platform specific elements through a library that mirrors mapsforge-map-android or mapsforge-map-awt, but of course only if there is Java.

I have not played with Google's J2ObjC, but I am pretty sure that a large number of people would be interested in mapsforge on iOS and I would also be willing to make changes to the mapsforge code if they would make the process of translation significantly easier (e.g. changing the InternalRendertheme to something different from enum, which seems just too much Java trickery).

Ludwig

--
You received this message because you are subscribed to the Google Groups "mapsforge-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mapsforge-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mapsforge-dev/d2ec8170-3a6e-44b7-b0b0-b6c999da542e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Emux]

I agree with Ludwig on that.

Also the modules of Mapsforge have been a great help for Java and I imagine for similar processes.

Please keep us informed about the progress of the project!

--
Emux
Cruiser - Atlas

[Matthew Hall]

(To be honest, not super familiar with Google Groups, so hopefully replying to this right.)

Thanks for the support. Development of the rendering code is moving along great. Implementing a version of InternalRenderTheme on iOS was super easy, on the order of 10 source lines of code. I can load assets.xml, rendertheme-v4, detailed, and probably all the other ones I haven't tried yet. I still have to figure out SVG support within CoreGraphics, but doing a .replace("svg", "png") is getting me by for now. There's a third party SVGKit I need to look into.

I've got 2 screenshots, one of downtown Houston Texas showing off Rendertheme-v4 and another of a park in my area with assets theme. So far, only a bit over 1200 lines of ObjC along with the 10,000 lines of Java code that is converted.

How should we post the source code to this? I noticed there is a MapsForge account on GitHub, I suppose another repo for mapsforge-ios next to the main mapsforge repo could be created there.

Progress:

graphics:

Bitmap - 90%

Canvas - 95%

GraphicFactory - 100%

Matrix - 100%

Paint - 95%

Path - 100%

PointTextContainer - 90%

ResourceBitmap - 100%

SvgBitmap - 0% (faked with PNGs for now)

TileBitmap - 90%

input:

MapZoomControls - 0%

ScaleListener - 0%

TouchEventHandler - 0%

TouchEventListener - 0%

TouchGestureDetector - 0%

layer:

MyLocationOverlay - 0%

rendertheme:

AssetRenderTheme - 95%

view:

MapView - 75%

Preferences - 0%

tile caches:

Still just using InMemoryCache, no files yet

tile downloads:

Still haven't attempted external tiles

[Ludwig]

Great, well done. 

 

How should we post the source code to this? I noticed there is a MapsForge account on GitHub, I suppose another repo for mapsforge-ios next to the main mapsforge repo could be created there.

I think the easiest would be if you first created a mapsforge-ios project on Github yourself and see how far you get, we could later clone it under the mapsforge account to give it more exposure if you want. 

The original mapsforge project is licensed under LGPL and I think it would be preferable if the IOS version was under the same terms.

I am not quite sure what would really be the best way, mapsforge will be evolving and ideally mapsforge-ios would be recreated from the evolving source, but if it involves fixing certain elements by hand then maybe we would need to store the translation command together with a number of patches that would be applied on top (that way updating to a new mapsforge version would involve retranslating and then reapplying those originally manual patches (hoping they will still work)).

The trick will be to keep both versions at least half-way in sync....but maybe I am ahead of everything now as first a working version will be needed.

Ludwig

[Emux]

Yes that's the best course of actions for now.

It's important as we continue the development on Mapsforge (Android / Java) to be able in the future to "pass" the changes to iOS relatively easily.
As it seems there aren't going to be totally separate (e.g. as Mapbox platforms), but probably a one direction process from Android / Java to iOS.
So Ludwig is right, we should be able to perform this somewhat semi-automatically, (well when we have a finalized code).

I emphasize also the license issue since it's translates (based on) existing LGPL code.

[Andre Höpfner]

If you won't Java on iOS, so you can use RoboVm => http://www.robovm.com

You can run Java from eclipse to a iOS simmulator or real device.

--

You received this message because you are subscribed to the Google Groups "mapsforge-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mapsforge-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/mapsforge-dev/5478AD44.1000609%40gmail.com.

[Matthew Hall]

I will keep it as LGPL3, the same license as MapsForge. Hopefully I can get a Github repo set up this weekend. My hope is to set up the project to include the Java files as is, then use build scripts to translate the code before compiling. It should be able to grab the latest MapsForge code and work with that. If I go this route, the build process will require J2ObjC, but it helps keep the parts in sync with the latest bug fixes. The project will generate an iOS static library in a Framework to be used by others. I will just keep an eye on any changes to the API between the core and Android code.

[Matthew Hall]

I wasn't able to get the project created the way I want to yet. Trying to find the best way to include the Java source as is in the XCode project to make everything as automated as possible while also not requiring constant transpiling of the Java code. The holiday weekend has slowed me down a bit with all the family in town.

On the other hand, progress is moving very well. Most of the major features are in. One major mem leak I need to fix. The TileBitmaps never get free'd. They are removed from the InMemoryCache and the MapsForge code is correct, but somehow the memory is never freed. I suspect it could be an issue with J2ObjC. My best guess right now is it doesn't create a proper AutoRelease Pool for the threads, but that's a guess right now. Need to do some digging to find out for sure.

graphics:

Bitmap - 98% (only scale left)

Canvas - 100%

GraphicFactory - 100%

Matrix - 100%

Paint - 100%

Path - 100%

PointTextContainer - 95% (only applying matrix transforms when drawing left)

ResourceBitmap - 100%

SvgBitmap - 0% (faked with PNGs for now)

TileBitmap - 100%

input:

MapZoomControls - 0%

ScaleListener -100%

PanListener - 100%

DoubleTapListener - 100%

LongPressListener - 0%

layer:

MyLocationOverlay - 0%

rendertheme:

AssetRenderTheme - 100%

view:

MapView - 85%

Preferences - 0%

tile caches:

File caching and memory caching both work perfect!

tile downloads:

External map sources work perfect!

[Ludwig]

Most impressive progress.

Maybe the best way of doing this would be to not import the Java code into the project, but somehow record the build step in the repository including checking out a specific version of mapsforge. 

Then the result of the original transformation could be recorded in git with any manual changes on top. That way the manual changes could be replayed later. 

But without seeing how this works, it is difficult to comment on it, so if this is totally beyond the point, just ignore my comment.

Ludwig

--

You received this message because you are subscribed to the Google Groups "mapsforge-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mapsforge-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/mapsforge-dev/efe5a09a-6c65-4c41-9f72-9ff894df8925%40googlegroups.com.

[Emux]

SvgBitmap - 0% (faked with PNGs for now)

How about trying to translate also AndroidSVG library?
(We use version 1.2.2 beta 1 which has performance improvements)

Also we use kxml2 2.3.0 library for parsing the xml, what's the situation there?

[Matthew Hall]

 

Maybe the best way of doing this would be to not import the Java code into the project, but somehow record the build step in the repository including checking out a specific version of mapsforge. 

Then the result of the original transformation could be recorded in git with any manual changes on top. That way the manual changes could be replayed later. 

This is a discussion I was having with a few others at my workplace today. One of the senior devs suggested a build process on the iOS side where it fetches the latest Java code, compiles it into some form (library or just translated source), then cleans up the Java code. An option that I will explore is if I can do a fetch, compile into a static lib, clean up, then link the static lib into the XCode project with as few manual steps on the user side. A second option is to have a manual build step where a user fetches the mapsforge components, runs a command such as "ant build-ios-lib," uses the generated lib inside the mapsforge-ios project, and builds the final Framework there.

I am hoping to have no modifications to the translated source if at all possible. Manual edits just increase the risk of something going wrong. I'm doing good so far with just two other issues.

The smallest issue involves static INSTANCE objects. The static initializers of a class are not called in ObjC until the first class object is created, though the initialize can be called manually at any time. The ObjC workaround is to just call the initialize. The only place I have hit this so far is org/mapsforge/map/layer/download/tilesource/OpenStreetMapMapnik. A quick search on "INSTANCE" shows 12 classes with static instances. The only way I know right now to fix on the Java side is to have a getter for a static instance. As soon as the static getter is called, the ObjC initialize is called for the class and the instance is properly created.

The bigger issue I have is how ObjC handles memory management. There is an "autorelease pool" that caches objects that are set to autorelease and does a full release at a later time. This way, something can return an autoreleased object and the receiver can claim ownership by retaining it before the object is scheduled for deallocation. The issue comes in with threads as a pool has to be created in the run loop of the thread or just around everything in run() if there is no main loop. The autorelease pool is drained and all objects released when one run cycle is completed this way. J2ObjC doesn't add the pool by default, but does provide a Java Annotation (com.google.j2objc.annotations.AutoreleasePool) to suggest where to add the pool. There's 3 areas I found where a pool would have to be added. The single annotation has no dependencies and can be included on it's own. The issue becomes should the mapsforge-core and mapsforge-map include this annotation or should the changes be made in the translated code? I know 0.5 is rc3 and just around the corner, so I don't want to impose any changes at this time. The 3 areas are:

org/mapsforge/map/layer/renderer/DestroyThread.java - no loop, just added an autorelease pool around everything inside of run().

org/mapsforge/map/model/MapViewPosition.java - there is a for loop, but I added an autorelease pool around everything inside of run(). The for loop isn't a long lived thread.

org/mapsforge/map/util/PausableThread.java - I added an autorelease pool just inside the primary while loop.

 

How about trying to translate also AndroidSVG library?
(We use version 1.2.2 beta 1 which has performance improvements)

There is an SVGKit for iOS, but I am struggling to get it to work right now. Having issues with it complaining about namespaces and non-namespaces. I just have it on the back log right now. I looked at the code for AndroidSVG. There's a number of classes that would transpile, but the main objects are too heavily Android specific code that would not make it. I will just have to figure out this SVG Kit. Sadly, I don't have the free time this week like I did last week. I think I threw 60+ hours at this in the last week, but back to normal work this week. It'll take me some time, but I'll get it working one way or another.

 

Also we use kxml2 2.3.0 library for parsing the xml, what's the situation there?

 

Somehow it just worked without any effort on my end. I knew J2ObjC mentioned XML parsing was part of the features, but didn't dig too deep. Looking at the J2ObjC source, turns out they use kxml2 and transpile it to ObjC.

[Matthew Hall]

Been awhile since I've had a status update, just been busy with normal work.

I have been working to create what I have into a lib and get it on Github soon. I tried to create it into a Framework, but had some issues with how the header directory structure is for the transpiled code. I have a bash script that can take the Java code, transpile it, compile it into all the iOS architectures (i386, x64, armv7, armv7s, arm64), then create one big static library to be used. My next goal is to have an XCode project that can also take the CoreGraphics code and put that in a library as well. If you all are good with it, I can branch MapsForge into my account to include the J2ObjC needed things and just pay close attention to pull updates in. The code changes are very minor and just involve creating the autorelease pools in about 6 locations.

[Martin Crossley]

On Thursday, 18 December 2014 18:53:35 UTC, Matthew Hall wrote:

Been awhile since I've had a status update, just been busy with normal work.

I have been working to create what I have into a lib and get it on Github soon. I tried to create it into a Framework, but had some issues with how the header directory structure is for the transpiled code. I have a bash script that can take the Java code, transpile it, compile it into all the iOS architectures (i386, x64, armv7, armv7s, arm64), then create one big static library to be used. My next goal is to have an XCode project that can also take the CoreGraphics code and put that in a library as well. If you all are good with it, I can branch MapsForge into my account to include the J2ObjC needed things and just pay close attention to pull updates in. The code changes are very minor and just involve creating the autorelease pools in about 6 locations.

Many thanks for all your work on this Matthew and even moreso for taking the time to post about it. I've located the 'Crazy50/MapsForge-ios' repository on Github and plan to start digging into this topic in the next month or so. Before I start that 'voyage of adventure' is this still an active development topic for you? any pitfalls to watch out for?

Cheers, Martin

[Matthew Hall]

Thanks. I haven't touched it in almost 6 months. I had been bogged down changing jobs. Thankfully, my new workplace is more open source and side project friendly. The last few weeks, I had been playing with ReactJS and isomorphic webapp. I'd be happy to pick MapsForge-iOS back up and definitely happy to have contributors.

The biggest hurdle is the lack of a solid build script. I had been experimenting with bash and make to see how I wanted to have a one-click automated build which would automatically pull the Java code, J2ObjC it, add in the iOS specific rendering code, and generate a universal lib. Currently, one can build the Mapsforge core lib with J2ObjC then include all the iOS specific code directly in a project.

It is also missing SVG, a few other TODO's, and some parameter data validation. I'll bring up my Mac tonight and at least start writing some documentation to help out getting it set up.

[Martin Crossley]

[...] I'll bring up my Mac tonight and at least start writing some documentation to help out getting it set up.

Brilliant - thanks Nick. My own 'side project' uses python to pull together a load of hetrogeneous tools (Java, C, scripts). I have a feeling that Apache Ant might be a more elegant way of doing it though :).
If you did get some time to hack together even some outline documentation that would be much appreciated - I really know how much of a PitA it can be writing up something that you haven't worked on for a while; so huge thanks in advance

Martin

[Martin Crossley]

Sorry - I meant to say 'Matthew' not 'Nick' (durr)

[Matthew Hall]

I had hoped to get it into a use-able state this weekend, but not quite there. I didn't get anything done during Father's Day yesterday. The library is buildable using an XCode project and could be directly linked into another XCode project. I'm still working out copying all the Public Headers, cleaning up the j2objc generated names (OrgMapsforgeMapLayerLayerManager is a lot to type) and hoping to get it into a framework. Once I get that working, I will get a Readme in the project with how to use it, maybe throw an example together, and start working on the rest of the documentation. The lib has quite a bit to go and could be a month or so before it has all the features implemented.