StreamRenderTheme: provide custom resource provider

[eddiemuc]

Hello all,

for usage in c:geo I have the need to use custom themes with mapsforge where I don't have any resource available as File (neither the xml render scheme itself nor the resource files it references to) but can only provide inputstreams for all of this.

Class org.mapsforge.map.rendertheme.StreamRenderTheme solves half the problem because it allows providing the xml render scheme as InputStream. However, it still requires all references (file) resources in the xml to be referenced as File using a relative file path. 

My idea is to provide an optional "custom" InputStream provider which receives a string (the resource string from XML scheme) and returns the InputStream for this ressource. E.g. new constructor of  StreamRenderTheme may look like this:

public interface ResourceProvider {

     InputStream get(final String src);

}

/**
* @param relativePathPrefix the prefix for all relative resource paths.
* @param inputStream an input stream containing valid render theme XML data.

* @param resourceProvider provides inputstreams for resource src
*/
public StreamRenderTheme(String relativePathPrefix, InputStream inputStream, ResourceProvider resourceProvider) {
  ...
}

This resourceProvider can be passed through RenderThemeHandler, Area/Line/... etc until XmlUtils.createInputStream() and used there like this:

/**
* Create InputStream from assets, file or jar resource.
* <p/>
* If the resource has not a location prefix, then the search order is (file, assets, jar).
*/
private static InputStream createInputStream(GraphicFactory graphicFactory, String relativePathPrefix, String src, ResourceProvider resourceProvider) throws IOException {
InputStream inputStream;

if (resourceProvider != null) {

   inputStream = resourceProvider.get(src);

   if (inputStream != null) {

       return inputStream;

   }

}

...(continue with already existing code

if (src.startsWith(PREFIX_ASSETS)) {

...

What do you think about this? This would really help me out. If ok I can code this up and provide a PR for this.

[Emux]

It's not very clear what you mean or the need for that.

Is it about Android's scoped storage where it expects all files to be read as input streams instead of real file paths?

There is work in progress for that, like the ContentRenderTheme.

--
Emux

[eddiemuc]

Thanks emux for the fast reply. You are completely right of course, this is for Android scoped storage.

I took a look at the ContentRenderTheme you posted me, and I noticed that at least in its current state it also only gets the main Xml render file from contentScope and not the resource files linked in it. I also need the later to be retrieved from scoped Storage. I know however that a "base path for relative paths" cannot be easily constructed for lscoped storage.

I would thus propose to add the above scetched solution to the ContentRenderTheme. What do you say?

Some more details to the use case: c:geo allows the user to specify a public dir where he can simply extract complete render themes e.g. donwloaded from the web. When taking such a theme e.g. from openandromaps and extracting it to said folder this may look in the end something like this ("Themes" being the user-selected scoped-storage-accessed folder where themes shall be stored):

* Themes

    * ele_res

       * ... (various resource files for the themes, e.g. p_camp_site.svg)

    * Elements.xml

    * Elevate.xml

Now, providing the Theme Files "Elements.xml" and "Elevate.xml" as InputStream is no problem at all (and if I see it you are further abstracting this in ContentRenderTheme to relieve the user of this burden, which I like).

The problem is when inside these Theme files relative ressources inside the dir "ele_res" are referenced, e.g. like in the following snippet:

   

<rule cat="borders" e="way" k="tourism" v="camp_site" closed="yes">

<rule e="any" k="*" v="*" zoom-max="15" zoom-min="14">

<area src="file:ele_res/p_camp_site.svg" symbol-height="32" />

</rule>

<rule e="any" k="*" v="*" zoom-min="16">

<area src="file:ele_res/p_camp_site_large.svg" symbol-height="64" />

</rule>

</rule>

In our use case these need to be retrieved via scoped storage / contentResolver too!

I dealt with the scoped storage quite a lot the last weeks, and as far as I know there is unfortunately no way to use an Uri as some kind of "base path" to construct Uris for relative paths (if you know a solution for this, I would be very interested!). But I could easily write with a few lines a c:geo-specific "ResourceProvider") to provide the necessary inputstreams. This is why I proposed the above solution.

What do you say?

[eddiemuc]

Something else: would you prefer to put "resoution of relative Uris" into mapsforge itself instead of allowing custom "InputStream-Providers"? That would also be a way where I could contribute. But there will be some code needed in mapsforge for that.

[Emux]

We need a solution that all users can use in their applications.

I added the MapsforgeMapViewer sample via #1185 to open map files and externals themes via scoped storage.

What seems to be missing is the open of external theme resources (via ContentRenderTheme or other theme class).

So what changes need to be done in that sample and Mapsforge library in order to properly open the Elevate theme?

--
Emux

[eddiemuc]

Hi Emux,

>  what changes need to be done in that sample and Mapsforge library in order to properly open the Elevate theme? 

I would personally be satisfied with the RessourceProvider which is imho a very small change for mapsforge and you dont have to deal with the more ugly parts of scoped storage.

However if you want to go this way, here is how it basically works:

* You need the Uri of a folder (not the one of the Theme XML document!) as a starting point. This is one which is directly returned from an ACTION_OPEN_DOCUMENT_TREE Intent issued to the user by the app.

* Having this Uri you can use DocumentFile.fromTree() method to create a "DocumentFile" object representing this folder

* The DocumentFile object has methods similar to "File" (e.g. listFiles()) to navigate through a directory tree

* Especially it has a "getUri()" method to retrieve a file uri which can then be used with contentresolver to read documents

Note that in my coding I found DocumentFile to be a very inperformant class. It is basically a "shell" around usage of DocumentContract as well as the "contentresolver.query" method, and it passes a lot of those queries when you call its methods. For the xml ressource reading use case, when used carefully, performance might be adequate (has to be tried). If not you have to fall back to to the base methods. For c:geo this fallback was necessary because we also do a lot of writing and handling of subfolder structures.

Here are some ressources I found helpful:

* Android Doku: https://developer.android.com/preview/privacy/storage
* * Android Doku on use cases: https://developer.android.com/training/data-storage/use-cases#handle-non-media-file  

Also, if you are interested, you will find here the implementation of Document access planned for c:geo: https://github.com/cgeo/cgeo/blob/storage_framework/main/src/cgeo/geocaching/storage/DocumentContentAccessor.java 

(this implementation does not use DocumentFile for performance reasons. It is work-in-progress and not yet approved by c:geo community)

What do you say?

[Emux]

Thanks for all the details.

The documents are known to not have great performance in the new storage.


If we make such changes in the library, we must also solve completely the reading of external render themes (xml + resources).

That's why I created the new MapsforgeMapViewer sample, to help the development and more the testing.

With an interface or other method, does not seem to show users how to read properly external render themes?
We need the sample to be updated too and demonstrate everything together with any needed library change.

Do you have a complete solution for all that process?

You can update the sample's theme reading too, but we must be able to test first the changes inside library's repository samples app.

--
Emux

[Emux]

You can describe the details here or even post a pull request if there is a complete solution.

But a pull request must also contain and demonstrate the successful opening of Elevate theme within the Samples app.

--
Emux

[eddiemuc]

Hi Emux,

I can try to build up such an example and provide a PR for it against mapsforge master. WOuld that be the right way to proceed?

Btw: I saw in your example that you even consider versions < KITKAT/API19. ACTION_OPEN_DOCUMENT_TREE is afair only supported starting with Lollipop/API21. I would not know of any solution for reading resources using scoped storage working below that.

Thanks and regards

  Eddie

[Emux]

> I can try to build up such an example and provide a PR for it against mapsforge master. WOuld that be the right way to proceed?

Thanks, that would be the best way to proceed.

> I saw in your example that you even consider versions < KITKAT/API19. ACTION_OPEN_DOCUMENT_TREE is afair only supported starting with Lollipop/API21.

We can deactivate the external theme menu on older Android?
So it can only work on the versions where that API is available.

--
Emux

[eddiemuc]

Hi Emux, 

I got mapsforge installed and the example is running. Your project is very easy to set up, and the sample app looks quite good, my congratulations!

Hopefully I can get the PR ready tomorrow or Sunday...

Thanks for your support

  Eddie

[eddiemuc]

Hi Emux,

you will now find a PR containing the proposed change here: https://github.com/mapsforge/mapsforge/pull/1186 

Thanks and best regards

  Eddie

[eddiemuc]

Hi Emux,

I continued discussion of this feature now in the Pull Request  https://github.com/mapsforge/mapsforge/pull/1186   

Would you prefer to do that discussion here in this Google conversation?

Thanks and regards

  Eddie 

[Emux]

Thanks for the pull request and the nice clean work.

We can continue the discussion in the pull request.


Please give me some time to review the code.

Since Mapsforge is developed along with VTM,
I also have to check how all this applies there.

--
Emux

[Emux]

Thanks Eddie for the useful contribution and the careful work / testing.

I merged pull request #1186 with the discussed improvements (see MapsforgeMapViewer sample).


I also transferred via #804 the same functionality in VTM (see MapsforgeActivity sample).

--
Emux