How to read file from res/raw folder

[idea]

We know there is Android NDK API to read file from assets folder.
Is there any way to read file from res/raw folder?

[Alex Cohn]

There is no API, but you can find your APK file, and trust that it is a ZIP file under the hood.

BR,

Alex

[Dan Albert]

Sounds like a reasonable API request. Looks like we're missing a bug component for NDK API requests. I've asked for that to be fixed, but I suppose for now you can file it in the Java category and we'll try to route it appropriately: https://issuetracker.google.com/issues/new?component=328403

--
You received this message because you are subscribed to the Google Groups "android-ndk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to android-ndk...@googlegroups.com.
To post to this group, send email to andro...@googlegroups.com.
Visit this group at https://groups.google.com/group/android-ndk.
To view this discussion on the web visit https://groups.google.com/d/msgid/android-ndk/a76b0327-7ccf-438d-8f0f-d38f9ca55d96%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Alex Cohn]

I beg to differ. I believe that the whole separation between assets and resources exists because the former can be accessed as read-only files, while the latter are used via Android SDK with built-in support for providing alternatives for different languages, OS versions, screen orientations, etc.

There is very little added value in having native access to resources, and many tricky questions to resolve while implementing it.

BR,

Alex

[Dan Albert]

Fair enough. It's certainly easier to not add the API :)

To unsubscribe from this group and stop receiving emails from it, send an email to android-ndk+unsubscribe@googlegroups.com.

To post to this group, send email to andro...@googlegroups.com.
Visit this group at https://groups.google.com/group/android-ndk.

To view this discussion on the web visit https://groups.google.com/d/msgid/android-ndk/b32ab556-c1ad-4dda-8b79-f3d020808e3c%40googlegroups.com.

[Steven Winston]

I have to admit to agreeing with Dan that this might be worthy of consideration.  I think you're right about the underlining intention for Assets to hold assets and resources to hold android sdk stuff that changes per several different factors.  However, a standardized method of accessing it from native would actually prove quite beneficial.

The reasoning I'd give is two fold:

1.) It lowers the barrier for entry into using the NDK.

Everyone that works with the NDK for any longer length of time has in their arsenal of libraries a tool that handles APK file access.  It's common to everyone and a concern of everyone.  Thus it stands to reason that new comers are likely to not have these tools/libraries and will wind up reinventing the wheel.
2.) There's tons of implementations for reading the APK of vastly varying levels of quality.

That point speaks for itself, but suffice it to say, if it's hard to get the API right, then it's likely hard for new comers to get the implementation right.  Lots of bad code out there, can point to some sample code that does incredibly bad things in this area too.

A bonus reason is this is a good common place where caching is quite helpful; but underused by several companies / developers.  If anything I'd love to get a folder to place resources that gets auto expanded to a common location on app install; for convenience if nothing else.

Please note, this is not a concern for me directly as I fall into the category of having an already existing competent toolset.

[Dan Albert]

+some folks that have more useful opinions on this than I do

To view this discussion on the web visit https://groups.google.com/d/msgid/android-ndk/196877bb-05fe-4c8f-b757-9b1d2f33c179%40googlegroups.com.

[Alex Cohn]

@Steven,

Could you point to some sample code that works decently? Or maybe Google can help you opensource your toolset?

OTOH, I would be glad to learn in what use cases your native code needs access specifically to /res and not to /assets.

Sincerely,

Alex

[Steven Winston]

@Alex,

While I can't offer up my companies' code (not a particular hurdle I'd want to jump through for this), it's also not exactly earth shattering.  Zlib as provided by cdep and caching of files as they are extracted.  If you need such a tool open sourced, I can certainly write a new tool and provide that.  There's a ton of samples that aren't always the highest quality of ideas.  https://www.google.com/search?ei=9piyWoyAA4fUjwP57KjoCA&q=ndk+read+files+in+resources+of+apk results in many SO questions that point people to using some form of:

unzip the apk

AAssetManager

Read in Java pass through JNI

etc.

For almost every use case I've seen in those questions, I believe the understanding is incorrect and the developer should place the files within Assets successfully ignoring the res directory when in the ndk. For these use cases selecting AAssetManager is a quite competent solution; however, it fails on one key area, cross platform and requires maintaining the java object in code that doesn't really need it (I've seen some less than optimal solutions caused by accessing the java object from other threads etc).

So most of the time legacy game devs that aren't already on Unity, Unreal or other common game engine here would select the first option, unzip the apk.  These are quite competent devs that understand caching so there's no real problem there that a new API would help.  That said, they are the people I'd most expect to have a valid reason to select the first path and already have toolsets in house that handle it.

For new devs, the challenge should be to instruct that Assets are where you want to place NDK resources and use AAssetManager.  However, I'd propose a relatively new use case where if my understanding is correct, the only available answer facing a new dev would be treat the apk as a zip.

Imagine, having a project that is targeting a VR/AR app that you wish to have an in Vr/Ar GUI.  Now that VR/AR app has a requirement of handling 2d  user interaction (system messages, help screens, or options menu) that must display classic Android GUI elements for.  Now imagine those GUI elements include something that would be useful to both GUI resource sets (logos come to mind).  The natural solution for this use case would be to inform the dev to duplicate the resource in both Assets and Res and stay with AAssetManager.  That works until someone cares about apk size, someone might say "hey how about we do an instant app."  Now we have to care about the size of the apk a whole lot more.  If my understanding is correct, this set of circumstances will transition a dev from being fully able to successfully only use AAssetManager.  They'd now be in the corner case of using the apk as a zip.

Again, this isn't a major requirement as using zlib or other zip library is quite easy, most devs are used to it from other platforms, also even if I weren't in a similar use-case requirement for the above scenario I'd still opt for using zlib to better support handling resources in a common method between Android/iOS/PS4 etc.  Hopefully this provides an idea for when it could prove useful to have such an API for others.

[Alex Cohn]

Thank you for this reply. 

This is not very convincing, I must confess.

When we speak about cross platform, no new API to access resources will really help (unless Android finds a way to bring iOS and Unity to the same ground). An independent 3rd party library could be better positioned in this case.

As for sharing some resources with the "classic Android GUI", we must keep in mind that the API will need all the logic of screen size / orientation / locale / whatnot that is transparent when using Context.getResources(). Not only that, resources use generated id's (e.g. R.drawable.my_background_image), not names. As a result, the best strategy in the rare cases that this is justified, is to call the  native Resources API via JNI. All you need a valid  Context.

I agree that the existing Assets API which requires a Java object for Asset Manager is not very convenient for native developers.

BR,

Alex

[Steven Winston]

@Alex,

Happy to follow up, and please remember these are just thoughts for improvement as I have no problems with the current system.  I wasn't intending to present an argument or present even an advocacy for any particular solution; my apologies as my meaning was muddled with the intention of presenting the use case of a new user using Google only provided technologies and winding up in a situation that warranted a requirement of accessing the /res directory from the NDK.  The goal was merely to show it is a non-zero chance (and I'd wager an increasing chance) that only going down the path of Google best practices results in developers having an unaddressed need.

That said, the points you make a valid.  No Google provided API would be considered cross platform, and naturally there's an xkcd to cover this situation: https://imgs.xkcd.com/comics/standards.png  

However, I would mention that perhaps we're a bit quick to say there isn't a solution that we can craft to address the needs of a cross platform developer having access to apk resources.  There's tradeoffs to other solutions, however, if we first accept that all languages the NDK might support now and in the future can natively read files off the disk with basic file I/O and all other platforms share similar basic file I/O capabilities.  A solution presents itself if we eliminate the zipped nature of the apk.

Obvious trade off with that solution is it would take up more disk space on app install.  This could be mitigated by expanding only portions a developer would care about; which providing an api would allow a developer to say what files should be available or label them in the manifest for easy NDK access then explode to temp directory on either app install, run, or when requested via API.  There's also a chance if one were to be a bit clever of crafting a ramdisk solution in Java that C sees as a file system.

Moving on to the Resources access for shared GUI concerns.  I'd wager that when sharing resources one wouldn't need to have concern about which screen size / orientation etc are used.  Allow me to elaborate this strange assertion with pointing out that if one were to go down the path of having their own GUI and requiring shared resources, then likely it's shared because they're still using the very excellent Android GUI for some interactions.  This would indicate with my wild assumption that the reason they might have a second GUI requirement is because the Android GUI doesn't completely fit their needs.  Now the reason why there's different resources for different orientations, screen densities etc is because the GUI is operating on a different viewport dictated by being different size, oriented or on different screen densities etc.

Assuming my understanding is correct, the time an end user would get into a use case requirement of needing their own GUI is when the frustum are not the same as the full screen one provided by Android's GUI.  This would be the case of VR or AR and some games.  Thus, if you have a custom projection matrix, you likely have need of one version of a resource and wouldn't care what the Java mapping is for a resource that wouldn't match your viewport in the first place; after all, if you did care, you likely would be in a scenario where Android's GUI is a better path forward.

So to clarify the use case here is Android GUI can't fit the needs due to not using the custom viewport (which by the way you guys provide a means to render the Android GUI to a GLTexture and I'll sing the highest of praises for Android's GUI).  However, there are some resources that can be used sans changes between GUIs (logos and other sharable disk guzzling assets that are high res etc).  So my point in all this is it wouldn't likely matter that R.drawable.my_background_image maps to /res/hdpi/xxxx/background.png instead of /res/xxx/yyy/background.jpg.

What would ultimately matter is generic access to the file in the first place as the reasoning behind the correlation is lost.  It would have to be incumbent upon the developer to determine the correct mapping within the resources directory of reusable resources as that developer is the only one capable of knowing the frustum they will be using.

If the dev wants to use a completely custom GUI, then they have all resources in Assets and they have an existing solution.  If the dev wants to use completely Android GUI, then this isn't an issue as they can use Java.  If the dev has need for keeping the apk small (instant apps) and they're looking for cutting file size costs, then providing a means to access resources that might be shared between the two is attractive.

In this case what I'd propose is packaging zlib along with an easy to use interface (API) to return a file for a given path within the APK.  I'm not sure there's a need to be more clever than that for the shared GUI problem.

--
You received this message because you are subscribed to a topic in the Google Groups "android-ndk" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/android-ndk/A9jdEgAqkNw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to android-ndk+unsubscribe@googlegroups.com.

To post to this group, send email to andro...@googlegroups.com.
Visit this group at https://groups.google.com/group/android-ndk.

To view this discussion on the web visit https://groups.google.com/d/msgid/android-ndk/852c4dfe-a28a-4b1e-a6db-1e2dc0dd4dfa%40googlegroups.com.

[Chris Browet]

In XBMC derivatives, we always leave the "res" folder alone in native (only for JAVA), and we plainly access the "assets" folder as a zip file from the apk.

Just my $.02

[Alex Cohn]

@Steven,

My takeaway from your description is that there may be another piece missing in the Android resources API. What if we could easily map some resources (e.g. R.drawable.my_background_image to assets/background.png) to reduce duplication for images and sounds that must be used both through the Resources API (easier in Java) and through Assets API (easier in C)? This could be an extension of the existing alias resources feature, and avoid the custom workarounds. Note that even in Java, getDrawable(int) is now deprecated, for good.

As for the zipped nature of APK, it's under your control. You can force your asset to be stored uncompressed in the APK, e.g. by faking its filename extension, and this means that the asset manager will give you handle to the piece of your APK with no need to cache the unpacked asset on disk (make sure that you don't drop the offset for the asset while working with it). On the other hand, if you can force your asset to be compressed, if you prepare it as whatever.ext.gz. The Asset Manager will provide the unpacked (cached) copy when you ask for it – both in C and in Java.

For resources, the Java API only gives you a file descriptor if the raw resource is stored as uncompressed data. The nice shortcut that Java has for us, though, is to provide an InputStream with no extra boilerplate code.

Sincerely,

Alex

To unsubscribe from this group and all its topics, send an email to android-ndk...@googlegroups.com.

To post to this group, send email to andro...@googlegroups.com.
Visit this group at https://groups.google.com/group/android-ndk.

[Alex Cohn]

Why don't you use the AAsset API? Just curious.

Alex

[Chris Browet]

Good Point.

I guess the API was overlooked.
Is there a performance gain to be expected vs "plain" zip handling?

[Steven Winston]

@alex

Thanks for this; I learned quite a bit of android lore and it made me think about a system I've long since taken for granted.  If nothing else comes of our discussion I view it as useful to me for no other reason than to have thought through it.

I like your thought of extending alias resources.  Admittedly, I didn't know that resources could be aliased to be reproduced in that manner.  Very nifty.  I did know that GetDrawable was deprecated.  Lint has been very helpful in pointing deprecation out but glad to know the why; makes sense.  It would seem that you've found a way to tackle this from the Java side instead of the NDK side.  I think it's pretty cool to be able to utilize assets files as though they were a res however, I worry that it might blur the designed lines of what should be in assets and what should be in res.  I guess shared UIs could be created in Java so would make sense to solve for both places and this is the path of least resistance.

For the compressed nature.  I've seen that solution before.  I was under the impression that this is a "packaging option."  Meaning that it is never compressed and thus the apk is larger.  If our goal is to have the smallest sized apk (instant apps), then compression is good; and likely what we want to keep.  Also when thinking of networking; I might be one of the few devs at the office that worries about end user's data, but why make them download a larger apk when they don't need to?  What would be really cool is an ability to inform that the apk should expand certain files or a directory on install, or some other caching mechanism.  Ideally, the final solution should be better than the current zlib everyone has a different yet common solution scenario.

btw, for all that see this conversation and wish to play with aapt, gradle has a very robust dsl available that is easier than the eclipse method Alex was kind enough to link to:

https://google.github.io/android-gradle-dsl/current/com.android.build.gradle.internal.dsl.AaptOptions.html

To unsubscribe from this group and all its topics, send an email to android-ndk+unsubscribe@googlegroups.com.

To post to this group, send email to andro...@googlegroups.com.
Visit this group at https://groups.google.com/group/android-ndk.

To view this discussion on the web visit https://groups.google.com/d/msgid/android-ndk/7513c7e6-b569-4ed5-acae-904c3f1a96b6%40googlegroups.com.

[Alex Cohn]

On Monday, March 26, 2018 at 6:17:57 PM UTC+3, Chris Browet wrote:

Good Point.

I guess the API was overlooked.
Is there a performance gain to be expected vs "plain" zip handling?

If the asset is stored 'as is' (e.g. an mp3 file), then the Asset Manager will let you use the APK file with offset to the beginning of the asset. If the asset is compressed, the Asset Manager will seemlessly unpack it for you and cache the result. Both tricks can be reproduced with "plain zip". Actually, your custom unzip may use a smarter caching strategy for compressed resources. But "dumb zip" will definitely be less efficient in both cases.

BR,

Alex

[Alex Cohn]

There is a cute fact about the APK size is that Google Play will perform another round of compression for the full file. This means that any unpacked resource will be compressed as tightly as possible while it's downloaded; the APK file in /data/apps will be uncompressed, to let you access the unpacked resources 'in place'. This is not necessarily a waste of user's disk space: you don't need to keep a cache of unpacked assets. From 'M' and up, same logic applies to the native libs: instead of unpacking them to /data/app/package/lib/arm, you simply use them 'in place'. Also, Play Store supports delta updates: it will download only the patch, and produce the new APK from the old one. Such deltas may become smaller when native libraries and other assets are not compressed in the APK. Finally, for asset types that are not compressed by default (e.g. jpg or mp3), zip compression will hardly provide any gain.

Cheers,

Alex Cohn

PS Thanks for the Aapt options link, it's very welcome in the context of this discussion.

To unsubscribe from this group and all its topics, send an email to android-ndk...@googlegroups.com.

To post to this group, send email to andro...@googlegroups.com.
Visit this group at https://groups.google.com/group/android-ndk.

[Steven Winston]

I'm speechless.  If my understanding of what you just said is correct; why is the apk a zip file at all?  The only place there seems to be a benefit, is in disk space.  Disk space is much cheaper than cell phone data.  Even with that consideration; given the fact that data in the apk should be viewed as "read only" why not simply use a partition that's compressed (i.e. brtfs or similar?).  I'm not sure I see a benefit for apk being a zip file at all now.

By the way, that's a really awesome feature; I'm going to have to start using it in projects.  You just made the aapt tool much more intriguing.

To unsubscribe from this group and all its topics, send an email to android-ndk+unsubscribe@googlegroups.com.

To post to this group, send email to andro...@googlegroups.com.
Visit this group at https://groups.google.com/group/android-ndk.

To view this discussion on the web visit https://groups.google.com/d/msgid/android-ndk/b451cc2f-2fe4-4f0f-9e93-51a3bca9e426%40googlegroups.com.

[Alex Cohn]

There are three reasons why APK is and will be a ZIP. First, backwards compatibility and tools that depend on this implementation detail. This format is so widespread in the Java world: JAR, AAR. Second, there still exist scenarios where the APK is served not through Play Store, e.g. during debugging. ZIP compression may be handy in these cases. Third, there are no dramatic disadvantages.

Cheers,

Alex

To unsubscribe from this group and all its topics, send an email to android-ndk...@googlegroups.com.

To post to this group, send email to andro...@googlegroups.com.
Visit this group at https://groups.google.com/group/android-ndk.

[Steven Winston]

Cool, that makes sense.

I question the utility of "during debugging" being a reasoning due to simple availability of rsync over usb or wifi.  The major disadvantage to zip is the entire file is a monolithic item to be sent rather than just a patch delta; and other stores could reasonably be expected to support both a zipped package and an unzipped package with enough time for them to catch  if enough apps take advantage of this functionality.  I also have an expectation that such patch systems would save end user's data on their cell phones, plus battery life given less requires transfer anytime any app updates.

That said, I think this is getting into a rabbit hole and far away from the primary concern of our topic.  What you propose, using aapt to leave files uncompressed is a definite possibility for helping out cross platform concerns.  In practice, there's a few things I'd request to make this easier:

1.) native method of getting file handle to uncompressed file. <-- ideally just make some soft links that maintain in zip directory hierarchy in a reasonable location that's easily referenced.

2.) update the play console to measure the apk after it has been compressed by play store for instant apps (i.e. uncompressed apk is counted if using this route and eliminates this option rapidly).

I think those two things would make this an approachable solution; what are your thoughts?  I do worry about other stores not enjoying the same patching mechanism so Oculus Mobile would still need the same already existing zlib style solutions.  However, this at least makes sense to experiment with.

I also think it's a good idea to extend the resource map alias generation to handle files in the assets directory so Java side can benefit from a shared GUI.

To unsubscribe from this group and all its topics, send an email to android-ndk+unsubscribe@googlegroups.com.

To post to this group, send email to andro...@googlegroups.com.
Visit this group at https://groups.google.com/group/android-ndk.

To view this discussion on the web visit https://groups.google.com/d/msgid/android-ndk/79f6d983-e8d4-41c3-8c95-2e19fcab331c%40googlegroups.com.

[Alex Cohn]

your 1) is in android/asset_manager.h 

Best of luck

Alex

To unsubscribe from this group and all its topics, send an email to android-ndk...@googlegroups.com.

To post to this group, send email to andro...@googlegroups.com.
Visit this group at https://groups.google.com/group/android-ndk.

[Steven Winston]

:) Ideally it'd be something that doesn't require a custom Android only API.  In lieu of that, something that doesn't require a java object and is multi-thread ready.  However, I'll admit AAssetManager is much more powerful than I original thought it was.

To unsubscribe from this group and all its topics, send an email to android-ndk+unsubscribe@googlegroups.com.

To post to this group, send email to andro...@googlegroups.com.
Visit this group at https://groups.google.com/group/android-ndk.

To view this discussion on the web visit https://groups.google.com/d/msgid/android-ndk/512c026f-2326-4c2b-b50a-d803e2d1deb0%40googlegroups.com.