mapsforge: Multiple Android apps using the same maps

[Michael von Glasow]

Hi all,

I have multiple Android apps which rely on mapsforge and use local maps. Having to keep one copy of each map for each app is not very efficient. Does mapsforge have a best practice for sharing map data between apps?

So far, I have solved this by placing the maps on shared external storage, at the cost of apps needing external storage permission. As the files are only read, not written, concurrent access is not an issue.

However, Android 10 is restricting shared external storage access via file paths:

* Accessing files via `File` with a conventional filesystem path will be restricted to app-specific directories (private and external storage).
* Other files on external storage can only be accessed using the [`MediaStore`](https://developer.android.com/reference/android/provider/MediaStore) API or the [Storage Access Framework](https://developer.android.com/guide/topics/providers/document-provider).
* The path returned by [`Environment.getExternalStorageDirectory()`](https://developer.android.com/reference/android/os/Environment.html#getExternalStorageDirectory()) is no longer directly accessible to apps.
* For the moment, apps can circumvent this by either targeting API 28 (Android 9) or lower, or by [requesting legacy shared storage](https://developer.android.com/training/data-storage/files/external-scoped#opt-out-of-scoped-storage) in their manifest. However, this will be discontinued with the next major platform update (presumably API 30/Android R/Android 11).

All of this is bad news for libraries which need to be portable (JRE and Android) and need to work with shared data—unless they use some kind of abstraction to the file concept, such as working with input/output streams rather than files, with the platform-specific code bridging the gap to file/storage access. However, as far as I can see, Mapsforge does not seem to offer this kind of abstraction yet—unless I am overlooking something.

What is Mapsforge’s take on this?

[Emux]

> Does mapsforge have a best practice for sharing map data between apps?

Mapsforge & VTM are map libraries, expecting map files as input and rendering them on screen.
Map file location - shared or not - is not any library's concern, but belongs to apps' responsibility.

> Android 10 is restricting shared external storage access via file paths

The fact is that Mapsforge and VTM must and will remain multi-platform libraries.
Cannot convert them to only Android libs, which each Android requires a rewrite.

And if I remember correctly map files need random access to work properly.

So a possible course of actions could be to implement a separate Android module for just this exotic Android reading.
But based on above remark, no idea if such implementation can work...

--
Emux

[Emux]

> All of this is bad news for libraries which need to be portable (JRE and Android) and need to work with shared data

There is a misunderstanding: libraries are libraries, they don't need to work with any shared data.

Applications request to work with shared data.
And Android decided that is bad, no secure, etc. so will allow access only to app specific directories.

Depending the app's workflow and how handles user data, may need no changes at all.

--
Emux

[Ludwig]

I think you can define custom permissions to give other apps access to your data. You could define a data provider app to just make the map data available, but then any data access will cross process boundaries, making it much slower 

But I also think that worrying too much about storage is not really appropriate any more. Unless you are targeting particularly low spec phones I think this is not an issue any more.

--
You received this message because you are subscribed to the Google Groups "Mapsforge & VTM" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mapsforge-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mapsforge-dev/5d3f93bc-b302-2702-6a65-733b4bd22c3e%40gmail.com.

[Emux]

More technical info:

https://www.xda-developers.com/android-q-storage-access-framework-scoped-storage/

--
Emux

[Michael von Glasow]

I think you can define custom permissions to give other apps access to your data. You could define a data provider app to just make the map data available, but then any data access will cross process boundaries, making it much slower 

That would be a content provider. Android already has a DocumentProvider, i.e. a generic content provider for files of any type. One would just need to get Mapsforge to work with the interface over which it exposes its data.

 

But I also think that worrying too much about storage is not really appropriate any more. Unless you are targeting particularly low spec phones I think this is not an issue any more.

Well, maps for France or Germany are more than 2 GiB in size each. For the entire EU I would estimate a two-digit figure; close to 25% of the available space on a 64 GiB phone. Having to maintain that separately for two apps costs not only extra space but also extra effort, time and (unless you are connected to a PC and can copy the map data over) bandwidth.

> Mapsforge and VTM must and will remain multi-platform libraries. Cannot convert them to only Android libs, which each Android requires a rewrite.

I know, I am facing similar issues in libraries which I develop. Google isn’t exactly making things easy for us, to say the least. I am solving this by requiring an InputStream rather than a File for input: getting an InputStream that reads from a File is a one-liner in Java (and if earlier API versions expect a File as input, one can always keep them around as convenience methods). In return, this opens up many ways to access the data: from a traditional File, Android’s storage framework, in some use cases this may even be a network URL, and you-name-it.

> And if I remember correctly map files need random access to work properly.

Random access as in “be able to jump to a random position in the file, read a random number of bytes from it, multiple times, in random order”? Or random access as in “read and write to a map file”? The former should not be an issue; the latter would be a deal-breaker for any kind of concurrent access to map files. Though I would be surprised to learn that Mapsforge *writes* to map files; I can’t see a need for that, not have I observed timestamps on map files changing. But you guys know Mapsforge better than I do…

 

There is a misunderstanding: libraries are libraries, they don't need to work with any shared data.

Applications request to work with shared data.

And if they do, they depend on the library supporting that. A bit like thread-safety—the app requests the functionality, but it will only work if the library offers it.

 

And Android decided that is bad, no secure, etc. so will allow access only to app specific directories.

Not quite—as the article to which you linked points out, they are still allowing apps the same level of access to shared data, but only if these apps use the new, custom, Android-only API rather than the well-established portable one. Personally, I suspect the main objective is either vendor lock-in rather than security, or an unfortunate attempt to prevent users from ever getting enough rope to hang themselves.

[Emux]

> Random access as in “be able to jump to a random position in the file, read a random number of bytes from it, multiple times, in random order”?

Random read access, like that.

> I am solving this by requiring an InputStream rather than a File for input: getting an InputStream that reads from a File is a one-liner in Java (and if earlier API versions expect a File as input, one can always keep them around as convenience methods).

MapFile can already read File or FileChannel.
Internally there is somewhere input stream, probably can add an extra constructor.

Note that current public API should remain to not break user applications.

--
Emux

[Michael von Glasow]

On Tuesday, October 22, 2019 at 5:01:42 PM UTC+2, Emux wrote:

MapFile can already read File or FileChannel.
Internally there is somewhere input stream, probably can add an extra constructor.

That sounds like a plan :-)

 

Note that current public API should remain to not break user applications.

Absolutely. I’m definitely not a friend of changes which break compatibility.

[Emux]

I added a new constructor in MapFile using FileInputStream, working similarly to FileChannel constructor.

By the way there is already the StreamRenderTheme for render themes.

--
Emux

[Michael von Glasow]

> I added a new constructor in MapFile using FileInputStream, working similarly to FileChannel constructor.

Any chance of getting the constructor to accept a generic InputStream instead of a FileInputStream? Android supplies a generic InputStream. Since FileInputStream is a subclass of InputStream, moving to InputStream should not be an issue unless Mapsforge were to depend on some specific characteristics of the subclass. Though, as far as the documentation goes, the only difference seems to be in the constructor, while the rest seems to be functionally equivalent.

[Emux]

Mapsforge uses FileChannel provided by FileInputStream.
VTM uses RandomAccessFile, both for the known benefits.

Can InputStream work in such workflows, without rewriting everything?

I can review pull requests from community with different implementations,
but must be via new separate classes, so current code remains unmodified.

--
Emux

[Emux]

https://www.xda-developers.com/google-file-manager-devs-submit-form-broad-file-storage-access-android-11/

--
Emux

[Michael von Glasow]

Thanks for the update. So in a nutshell, select apps will continue to be able to use file access as they used to, but will require individual approval.

From the talk (referenced in the source) I infer that file access will continue to be available at the OS level, but apps relying on it will need special approval to get listed on the Play Store. As far as the OS itself is concerned, apps that rely on file access will continue to work if they target a legacy API version, or Android 10 with scoped storage opt-out. It is not entirely clear if opt-out will remain available on apps targeting Android 11 and up; however, given that Google keeps allowing file access at least for some, it seems likely that there will be a supported way for apps targeting Android 11. In that case, the lockdown would not affect apps which are not released through Play Store.

[Peter Storch]

I think I got it working.

I got an Uri via Storage Access Framework, got an InputStream via ContentResolver, casted it to FileInputStream and fed it to the MapFile:

protected MapDataStore getMapFile() {
if (baseApplication.getMap() == null) {
return null;
}
final Uri mapFile = baseApplication.getMap();
if (mapFile == null || !DocumentFile.isDocumentUri(this, mapFile)) {
return null;
}
try {
FileInputStream inputStream = (FileInputStream) getContentResolver().openInputStream(mapFile);
return new MapFile(inputStream, 0, null);
} catch (Exception e) {
Log.e(TAG, "Can't open mapFile", e);
}
return null;
}