Problem with memory management

[Torello Querci]

Hi guys,

I'm developing an application that needs to show a map and a long list view with lot of thumb image, but not at the same time.

Without MapView the application work fine but with MapView, the application crash because ListView not have enought memory to show the thumb image. The application manage the ListView correctly releasing the bitmap that are not more used.

Seems that the problem is related with mapdroid bitmap cache use that cannot be release when I hide MapView.

Looking inside the code I see that there is a bitmap cache that is preallocated on mapView startup. The size of this cache cannot changed. I try to destroy the mapView when this is not necessary but semms not release the memory so when the mapView came up my application crash.

Can somebody help me?

Best Regards to all.

[Ludwig]

I am currently looking into the memory problems and while I cannot provide a solution right now, I can report some progress.

I think one of the problems is that the resources allocated by the rendertheme are not freed in some cases. I am currently particularly observing the situation where the app is stopped (with the back button back), but the process remains alive. The next time the app is started, the process is comes back to front and the resources from the rendertheme are allocated for a second time. There are simply some bits missing where the destroy of the MapActivity does not destroy all the resources associated with the MapView. I am particularly looking at bitmaps at the moment, but there might be other resources as well.

I have managed to get most bitmaps recycled, I am not down to zero yet, so I am still seeing the out of memory crashes. But it is some progress at least. If I find some time tomorrow, I might be able to provide a patch for at least a part of the problem.

Ludwig

[Ludwig]

Hi Torello, 

would that be possible that you make a small code example available that shows what you are doing to hide the map view?

Ludwig

On 15 December 2012 02:55, Torello Querci <tqu...@gmail.com> wrote:

[Torello Querci]

Hi Ludwig,

I'm try to change visibility of destroy MapView method so to hide the map I use:

ViewGroup parent = (ViewGroup) mapView.getParent();

parent.removeView(mapView);

mapView.destroy();

mapView = null;

Following are the mapsforge code changes. Unfortunally after three show/hide round the app crash.

Looking logcat log I find that every time that MapView is showed the external memory usage indicated, by Garbage Collector log, is higher.

This show me that not all the external resource are removed in the destroy method.

Best Regards

Index: mapsforge-map/src/main/java/org/mapsforge/android/maps/MapView.java

===================================================================

--- mapsforge-map/src/main/java/org/mapsforge/android/maps/MapView.java (revisione 2030)

+++ mapsforge-map/src/main/java/org/mapsforge/android/maps/MapView.java (copia locale)

@@ -80,7 +80,7 @@

 

  private static final float DEFAULT_TEXT_SCALE = 1;

  private static final int DEFAULT_TILE_CACHE_SIZE_FILE_SYSTEM = 100;

- private static final int DEFAULT_TILE_CACHE_SIZE_IN_MEMORY = 20;

+ private static final int DEFAULT_TILE_CACHE_SIZE_IN_MEMORY = 10;

 

  private DebugSettings debugSettings;

  private final TileCache fileSystemTileCache;

@@ -714,13 +714,17 @@

  redrawTiles();

  }

 

- void destroy() {

+ public void destroy() {

  this.overlays.clear();

 

  this.mapMover.interrupt();

  this.mapWorker.interrupt();

  this.zoomAnimator.interrupt();

 

+ for (Overlay overlay:overlays) {

+ overlay.interrupt();

+ }

+

  try {

  this.mapWorker.join();

  } catch (InterruptedException e) {

2012/12/16 Ludwig <ludwigbr...@gmail.com>

[Torello Querci]

Sorry .... I move the overaly code interrupt from my application to mapsforge library but I not check it.

This is the new differences with clear/interrupt order right.

Index: mapsforge-map/src/main/java/org/mapsforge/android/maps/MapView.java

===================================================================

--- mapsforge-map/src/main/java/org/mapsforge/android/maps/MapView.java (revisione 2030)

+++ mapsforge-map/src/main/java/org/mapsforge/android/maps/MapView.java (copia locale)

@@ -80,7 +80,7 @@

 

  private static final float DEFAULT_TEXT_SCALE = 1;

  private static final int DEFAULT_TILE_CACHE_SIZE_FILE_SYSTEM = 100;

- private static final int DEFAULT_TILE_CACHE_SIZE_IN_MEMORY = 20;

+ private static final int DEFAULT_TILE_CACHE_SIZE_IN_MEMORY = 10;

 

  private DebugSettings debugSettings;

  private final TileCache fileSystemTileCache;

@@ -714,13 +714,17 @@

  redrawTiles();

  }

 

- void destroy() {

- this.overlays.clear();

-

+ public void destroy() {

  this.mapMover.interrupt();

  this.mapWorker.interrupt();

  this.zoomAnimator.interrupt();

 

+ for (Overlay overlay:overlays) {

+ overlay.interrupt();

+ }

+

+ this.overlays.clear();

+

  try {

  this.mapWorker.join();

  } catch (InterruptedException e) {

2012/12/16 Torello Querci <tqu...@gmail.com>

[Ludwig]

Thanks, 

I have made some further progress in tracking down memory leaks, there were a few (including some I am not sure about as technically all the bitmaps should be recycled by gc once not reachable anymore). One was that the bitmaps for the images in the rendertheme were not cleaned up, but the biggest one was that the bitmap backing the overlaycontroller is being allocated a few times at startup, caused by the mapworkers. 

I hope to be able to produce a patch either today or tomorrow that at least solves the problems that I have been seeing. I will try to test this with your code as well, but I cannot give you any guarantees on this.

Ludwig

[Torello Querci]

Thanks

now I'm using 0.3.0 mapsforge version. Works your patch on this version?

2012/12/16 Ludwig <ludwigbr...@gmail.com>

[Ludwig]

I am working on the head of the current development. 

I guess my work will not benefit you directly as the implementation of the the overlay has changed totally, but you might be able to backport at least some of the fixes.

Ludwig

[Torello Querci]

Ok,

I'm awaiting for your patch.

Thanks

Best Regards

2012/12/16 Ludwig <ludwigbr...@gmail.com>

[emux]

Hi,

Is the work that you currently do connects to the issue with the map view resize and its OOM problems?
I am referring to issue 365: http://code.google.com/p/mapsforge/issues/detail?id=365
where at map view resize the canvas holds its old size and we have gray tiles. If I try to reset canvas at FrameBuffer.onSizeChanged with
this.mapViewCanvas.setMatrix(null); to eliminate gray tiles, we have out of memory errors at FrameBuffer and/or OverlayController bitmaps at resize.
May be it has to do with the mapViewBitmap1, mapViewBitmap2 recycle and recreate at size change?


Thanks.

[Ludwig]

Some good news and some bad: I think I have a fix for some of the memory leak issues, but this has not yet solved all the problems. 

While currently this seems to work when doing manual testing, an automated test that does a lot of device rotating still makes the app run out of memory pretty quickly. But I am running out of time today and wanted at least to share where I got to.

I have made a public branch called 'memory' at http://code.google.com/r/ludwigbrinckmann-mapsforge/source/list?name=memory on my clone. It has a single commit on top of the current head in the official git repository.

There are three core fixes for what appears to be memory leaks:

1. The bitmaps allocated from the rendertheme were never freed and a new rendertheme was created when the app was paused. To fix this, a new destroy() method was added to the DatabaseRenderer and the method gets called when the MapView is destroyed. This method propagates the destroy to the RenderTheme, which propagates it to its elements. Some of the elements (areas, lines and symbols) hold (sometimes indirectly via Paint) bitmaps. The required operations have been added/fixed. For debugging purposes I had added counters for the bitmaps allocated and it seemed that I managed to get all of them cleaned up.
2. The second problem seemed to be in the OverlayController and the way the MapWorkers at startup cause the allocation method to be called multiple times. I have added here a ReadWriteLock where the writeLock is acquired when the size changes, whereas only a readLock is required for access. Additional checks are performed so that the actual resizing of bitmaps only takes place when the size has actually changed. The bitmaps allocated here are quite big, so any reduction is the work here should have significant benefit.
3. There was also a bitmap in the FileSystemCache that did not get cleaned up.

As I said above, this is not a final solution as I still see the out of memory errors in automated testing. 

Comments & problem reports welcome.

Ludwig

[Torello Querci]

Hi Ludwig,

I try to apply your patch on 0.3.0 version, obviusly the part that can be applied, and is little bit better but not solve the problem.

Actualle I need remove and add the mapView again when the map need to be showed because I not have enought memory to continue with the application.

Moreover the mapview is only on portrain so I not have problem with device rotation.

Thare are same problems with old version of MapView? Move to ond MapView version can be a solution for me?

Best Regards

2012/12/16 Ludwig <ludwigbr...@gmail.com>

Some good news and some bad: I think I have a fix for some of the memory leak issues, but this has not yet solved all the problems. 

[Ludwig]

As I said in the comment, the patch does not solve the problem entirely. I have made some further progress today I think, but the issue is still not fully resolved. 

The problem seems to be that the bitmaps are not recycled unless the canvas is nulled out and recreated (as apparently the canvas holds references to the bitmaps). 

I will push out an updated patch tomorrow, as I will need some more time for testing.

Ludwig

[Torello Querci]

I know, moreover I'm using 0.3.0 version and not git version so I need to reapply your patch in different code version so some classes, methods or other can not exist or to be different.

Best Regards

2012/12/18 Ludwig <ludwigbr...@gmail.com>

[Ludwig]

I have pushed another set of changes to my mapsforge-clone on the 'memory' branch, see 

http://code.google.com/r/ludwigbrinckmann-mapsforge/source/list?name=memory

The core of the change here is to null out the canvas when the bitmaps are recycled. Apparently the canvas holds references to the bitmaps, which stops them from being freed despite the call to recycle(). 

There is a bit of discussion on this topic on https://code.google.com/p/android/issues/detail?id=8488

My understanding of the problem is that the bitmaps are not cleared up as the underlying infrastructure still holds some references to them. Any new assignment actually clears those references, but by that time it is already too late as the allocation preceding the assignment fails. One might call this memory spikes due to double allocation. (It would actually be much better if Android offered a way of resizing bitmaps, i.e. change their width/height without actually going through the reallocation process).

My own tests now run to completion on Android versions >= 2.3, I see crashes on because of memory issues on 2.2. I do not think this is actually Android version related, it is probably because the devices just have a bit more memory and are so able to survive the memory spikes.

It would be good if people who had memory issues would checkout the branch and give some feedback if this resolves any of their issues or creates new problems.

Note that the fix provided in this branch does not include the fix for the deadlock problem.

Ludwig

[andrea antonello]

Not sure if that has something to do with it, but as far as I know on
OS before honeycomb, the bitmaps are stored in native heap and it
might be that they are not released in a performant way, while after
HC they are stored on VM_HEAP and garbage collected properly.

I remeber seing a good google IO video about it but I can't find it again.

If the above applies, I assume that the error might appear in some
situations in 2.3.

Just as a sidenote.

Cheers,
Andrea

[Ludwig]

Thanks, if you find the link please post it, would be interesting to see if there is any additional hint in there on how to work around it.

[Gabriel Sanmartín]

About this, I am getting a huge pain in the ass with memory problems. I am trying my app in an emulator with 800x480 screen size and 512mb RAM, and I can't even get two overlays running (it even crashes with the first one). I am using a 17mb offline map, is this mapfile completely stored in memory? Should I use smaller maps instead?

I feel tempted to switch to HEAD revision, but it's lacking the OnTap event which I need for my overlays...

[andrea antonello]

@Ludwig , I think the session was this one:
http://www.youtube.com/watch?v=_CruQY55HOk
Vincent ferries of this list lead me to it :)

> About this, I am getting a huge pain in the ass with memory problems. I am
> trying my app in an emulator with 800x480 screen size and 512mb RAM, and I
> can't even get two overlays running (it even crashes with the first one). I
> am using a 17mb offline map, is this mapfile completely stored in memory?
> Should I use smaller maps instead?
>
> I feel tempted to switch to HEAD revision, but it's lacking the OnTap event
> which I need for my overlays...

Are you sure it is a mapsforge problem Gabriel? I am still on the last
available version that has tile support and work really nice with it.
I had the same issues when I started with mapsforge, but after looking
at the video at the begin of the email, I solved all memory leaks,
which, apart of one (which is referenced in the issue tracker with
solution), it was all my own mess.

Maybe you could have a look with the tools described in the video,
where your leaks are. I am doing fine even on 1024 tablets.
If you want, I can cleanup and open the repository where I had to fork
off the last tile enabled version, and where I backported all the
fixes I saw pass that were bound to memory.

Ciao,
Andrea

[Gabriel Sanmartín]

Maybe you could have a look with the tools described in the video,
where your leaks are. I am doing fine even on 1024 tablets.
If you want, I can cleanup and open the repository where I had to fork
off the last tile enabled version, and where I backported all the
fixes I saw pass that were bound to memory.

I would very much appreciate that. I am currently watching that video, although it seems strange that this is a mistake of mine since I am only showing the map plus two overlays (one has around 10 pois, and the other one is an arraywayoverlay that holds quite a lot of points to trace a route). But maybe I am missing something here. 

I did got it to work on the emulator by increasing the VM heap size, but I keep having the same problem I posted in this issue: if I go back and enter my activity again, It crashes (probably because of what is said in this very post, bitmaps' references are held and thus not collected by the GC). Is there anything I can do to "uncache" the whole map on my onDestroy() callback to free that memory up?

Thanks a lot for your reply.

--Gabriel

[Ludwig]

An overlay always requires a bitmap the size of the screen to draw on, regardless of objects actually drawn on it, so two bitmaps make the memory issue appear much quicker than with just one. 

I think the point we got to in this discussion is that even when cleaning up very carefully, the problem can still appear as the bitmaps do not get freed quickly enough. I will see if the video gives me any new ideas of how to tackle this. (A square mapview might help).

[emux]

I have similar behavior with OOM with every 2nd start of application as mentioned at
http://code.google.com/p/mapsforge/issues/detail?id=378
Also with map view resize.
But these issues happen more rarely with the Ludwig's memory clone.

Personally with the snapshot version (as with the Ludwig's clone) I use only one ListOverlay and add/remove overlays only in it in order to minimize the memory requirements.
Still I come across memory issues with most of them happen at big resolutions - big bitmaps - that many tablets and new phones have these days (~720p).
I think that an issue is which heap size vs screen resolution has each device, combination which as developers cannot control.

[Gabriel Sanmartín]

Turns out you were right after all. I do get high memory usage from using 2 overlays, but thanks to that video I found out the memory leak when exiting activity and going back in is not mapsforge's fault.

What I am using is an OverlayFactory class of mine that condenses both POIs and the user location (so I use only one ItemizedOverlay instead of splitting them in two). The very line responsible for the memory leak is not related to MapsForge, but the GPS location Listener. If I comment this line out:

mgr.requestLocationUpdates(bestProvider, 0, 0, onLocationChange);

I don't get those memory problems at all. I checked with all my overlays minus the GPS location and memory usage is constant between activity launches.

So, if any of you is having similar problems, and is gathering the user position, you should check if this is the root of your problem too.

Now I just have to find out what the problem is with my LocationListener not being garbage-collected, even if I am removing it from updates in the onDestroy callback.

I thus apologize for blaming it on this project :P

[Ludwig]

The problem you are describing (two overlays already eat up all the memory) is only relevant to the 0.3.0 release, the re-architecture of the overlays is, amongst other things, the attempt to fix this. So, there will not be a separate 0.3.0 way to fix this. 

For 0.3.0 (and before) my approach was to put everything on a single overlay (write your own class, it is not difficult). The reason for the memory crash there is that each overlay has its own bitmap, it has nothing to do with the map size, I use much bigger maps without problems.

Ludwig

[Torello Querci]

Hi,

I tryed to reapply patch for gif mapsforge for 0.3.0

I also make that change needs to run mapsforge on android4.2.

In order to try to solve my memory problem I try to remove map and create new one each time I need.

Seems to works if I not use overlay but using it after some switch the application hang up.

Moreover I modify my application to use only one layer. At the beginning I used three layers, one for POI, one for selected POI and one for user position because I had not been able to the the different Drawable resource.

Using ItemizedOverlay.boundCenterBottom(getResources().getDrawable(R.drawable.marker_visitor)) now seems to work.

Best Regards.

2012/12/21 Ludwig <ludwigbr...@gmail.com>

[Tiago Pereira]

Hi... I'm having the same problem. I don't get these memory problems when i only use the mapsforge library to display a mapview and overlayitems. But when i try to listen my location, i don't now why but every time i create a new mapactivity, the memory allocated increses and causes a OOM error. How can i solve the problem? i'm simply using a Class that extends MapActivity and implements LocationListener.

locationManager.requestLocationUpdates(LocationManager.GPS_PROVIDER, 0, 0, this);

and ondrestroy() i remove the locationUpdates

locationManager.removeUpdates(this);

sorry for my english and thank you in advance.

Tiago Pereira

[emux]

Hi,

If I understand correctly you have frequent creation of new MapActivity (and so new MapView and reallocation of new bitmaps)?
Because of device orientation changes?
See below tip #3 and you can read the relevant discussion:
https://groups.google.com/forum/#!topic/mapsforge-dev/7Mem4S1Aotw

Also the removal of location updates doesn't it have to be at onPause?

Just to summarize some tips about memory issues:
- Use current master branch as it has memory fixes
- Merge all overlays into a single overlay, as overlays are memory hungry
- Tell Android to not restart your activity at orientation changes, via android:configChanges at your manifest

Best regards, Emux
https://play.google.com/store/apps/details?id=gr.talent.cruiser

[Tiago Pereira]

Hi. I want to thank you for your reply.

I have blocked the rotation and in the manifest file, i already had android:configChanges to avoid call the onCreate method. I don't have any problem with screen rotation but when I go out of my activity and enter again later and i repeat the process 2 or 3 times, i have a OOM. But if i don't use the methods to get Location, my app runs normally. I only have a Overlay with +- 30 POIs.

Again, I apologize for my English and thanks for your reply.