jobb tool generate corrupted OBB files, badly bugged?

[Daniele Segato]

Hi,

I've opened a bug here:

https://code.google.com/p/android/issues/detail?id=61344

A G+ discussion here:

https://plus.google.com/u/0/103362309911306712784/posts/DkKiMtN5hJx

Resumed a topic in the official Android developer group:

https://groups.google.com/forum/#!searchin/android-developers/Are$20Encrypt$20Opaque$20Binary$20Blobs$20(OBBS)$20supported$3F/android-developers/x_kTCgMfGG8/UitGecE8SRQJ

And I'm not alone with this:

https://code.google.com/p/android/issues/list?can=2&q=obb&colspec=ID+Type+Status+Owner+Summary+Stars&cells=tiles

And googling around:

http://stackoverflow.com/questions/14342068/how-to-mount-encrypted-apk-expansion-files

http://stackoverflow.com/questions/12101126/using-encrypted-obb-files-in-android

This are all the same issue!

I'm testing on a Nexus4

I simply cant mount any Obb file generated with the obb tool:

10-22 16:17:49.852      166-195/? E/Vold﹕ Error loading mapping table (Invalid argument)
10-22 16:17:49.852      166-195/? E/Vold﹕ ASEC device mapping failed (Invalid argument)
10-22 16:17:49.852      166-195/? W/Vold﹕ Returning OperationFailed - no handler for errno 22
10-22 16:17:49.852      560-631/? E/MountService﹕ Couldn't mount OBB file: -1

My files have a structure like this:

/dae/

     <long filename>

and are exactly 38 files (19 textures, 19 .dae 3d models) for slightly less then 200MB

Trying to dump it produce an error:

Package Name: com.my.app.package
Package Version: 11
SALT: -63dddd10f3a63bb

29eb26a5c9227f6efeab677fc53a7348
[LFN = dae / SFN = ShortName [/   >
                                    ifl -- 5 2f 1d 10 3 3e c 1d 69 66 6c ]]
    LFN = m11_BBJ94_13WishesDollAsst.dae / SFN = ShortName ["* /  ifl -- 1a b 22 2a 1d 2f 1d 3 69 66 6c ]
Alignment off reading from sector: 2449
Partial read from sector: 2449
Exception in thread "main" java.nio.BufferOverflowException
	at java.nio.HeapByteBuffer.put(HeapByteBuffer.java:183)
	at java.nio.ByteBuffer.put(ByteBuffer.java:832)
	at com.android.jobb.EncryptedBlockFile$EncryptedBlockFileChannel.readDecryptedSector(EncryptedBlockFile.java:292)
	at com.android.jobb.EncryptedBlockFile$EncryptedBlockFileChannel.read(EncryptedBlockFile.java:142)
	at de.waldheinz.fs.util.FileDisk.read(FileDisk.java:118)
	at de.waldheinz.fs.fat.ClusterChain.readData(ClusterChain.java:214)
	at de.waldheinz.fs.fat.FatFile.read(FatFile.java:126)
	at com.android.jobb.Main.dumpDirectory(Main.java:137)
	at com.android.jobb.Main.dumpDirectory(Main.java:112)
	at com.android.jobb.Main.main(Main.java:315)



I'm pretty desperate for help here, sorry to push so much.

Thanks,
Daniele

[Scott Barta]

The response from the author of the JOBB tool is:

That's going to be a hard one to fix.  The filesystem is failing to get created correctly with a random set of input files independent of the encryption layer.  The only thing I can tell the person to do is to install a portable Linux distribution and use the Linux kernel to generate the OBB file, which is likely to work.  (although that tool also will fail in some cases)

--
You received this message because you are subscribed to the Google Groups "adt-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to adt-dev+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Daniele Segato]

Hi Scott,

Where did you found that response? I'd like to read the conversation.
But I guess you asked internally about my issue and gave me his response (thank you for that, I really appreciate)

Are you saying this depends by the environment in which you create the obb file? (where I launch the jobb command )?

I'm on Linux (mint 15 to be precise) and I want to add a bit to this mess. I didn't before because it would have sounded weird.

Friday I created an obb file to start integrating the apk expansion library for the obb download.

That test obb file is always mounted. So I know my code works . I didn't checked the hash sum of the files, but they obb mount and I can see the list.

If I try to generate the obb for the actual resources I need they will not mount. Same jobb command.

The weirdness goes on: if I regenerate the Friday obb, same files, same commands it also fail to mount.

This is what made me think of a bug.

My laptop received some updates since Friday, not many.. But I can remember seeing an update to libc.

Also the Friday files have been moved and I may have changed their permissions.

This is to give you the full picture.

Can you give instructions on which Linux distribution should I use?

Can I use the jobb file in this distribution or should I create the obb manually? How?

Thanks for your time, you are giving me hopes!

I promise I'll give back to the community the knowledge I get from this.

Regards
Daniele Segato

[Scott Barta]

I sent mail to the developer internally; what I had in my previous message is literally as much as I know. He's in the process of activating project member status on his account and should be able to chime in on the bug soon.

[Daniele Segato]

Thank you Scott!

I really hope this can be fixed. I'll find a way to borrow some time and hope to have a workaround or a feedback in time!

I can't express how grateful I am!

Please let me know if you have news!

Regards,

Daniele

[Daniele Segato]

Just one more thing, do you have an idea of how much time will be needed for the jobb author to give some feedback?

I'd really like to have a workaround to the issue.

Thanks

[d...@android.com]

There is script called mkobb distributed with the AOSP project that actually uses the Linux kernel to generate the OBB file and has similar syntax to JOBB.  If you need encrypted OBB support, this tool should be very reliable because it uses the same code that is inside of Android to create the OBB files.

https://github.com/android/platform_frameworks_base/blob/master/tools/obbtool/mkobb.sh

While I'm going to try to figure out why these filesystems are failing to get created correctly, any time encryption is involved it is a serious challenge.

[Daniele Segato]

Hi again..

I updated the bug adding some information...

I downgraded libgfs and procfs in my system and I could generate the OBB file again.

This is far to be a solution since I can't keep my system back in updates forever.

But at least I can now generate OBB file and, hopefully, this will help fixing the tool.

Should I open a bug report to the libgfs /  procfs maintainers?

regards,

Daniele

[Daniele Segato]

Hi Dag,

thanks for the response.

I'll have a look into this mkobb tool you speak about.

But I really hope to be able to use the jobb tool because not all my colleagues run on Linux.

I've added details to the bug I opened here:

https://code.google.com/p/android/issues/detail?id=61344

I was able to generate obb file that could be mounted on devices downgrading some recent upgrade I got in my system.

Specifically the libgfs and procfs (not sure which one of these two). See the bug report for details on the versions and exact packages names.

Is jobb using one of this libraries?

I think the culprit is there somewhere.

I've not much time now, when this project will be less urgent I'll try with mkobb!

Thank you in the meanwhile!

And don't hesitate to ask for anything I can do to help you out!

Regards,

Daniele

[d...@android.com]

JOBB is a pure Java tool, so I wouldn't anticipate system libraries causing issues with file corruption.  It would be interesting to see how the FAT filesystem has been corrupted, although it might not help us find a solution.

- D

[Daniele Segato]

I don't know how this is possible.

I just know that when I downgraded those packages in my system the tool generated valid OBB files again.

That's a fact.

I initially couldn't believe it either.

But since that was the only possible explanation for why it first worked, then suddenly didn't I downgraded my system and found out that the tool generated valid files again.

I do not know if this is related to my files.

But if it doesn't I guess you can reproduce it with any Linux Mint distribution and all the updates.

I also didn't tried without encryption so I'm not sure this issue is related to that.

If you confirm me you want the corrupted files I can try to upgrade my system again and generate them to you. But you'll have to give me some time for that, a few days probably and I can upload it somewhere publicy and share it with you directly.

If I do that I can also try with some other fileset.

Let me know if you think those test can be useful, thanks!

Cheers,

Daniele

[Yasmani Llanes]

Hey guys. Did you guys ever find a solution to this? I'm still having issues with the JOBB tool and it's almost 2016.

[Luigi Rosso]

Same issue here. I'm completely flabbergasted to find this to be an ongoing problem since 2013 with no resolution. 

I am able to mount my un-encrypted OBB created with the JOBB tool but certain files are corrupt when reading from the mount.  I am using Android SDK Tools 24.3.4 on OSX.  I've attempted optimizing files, and as others have found, changing the contents of the OBB (for example adding a new file) causes unpredictable results (different files are now corrupt and ones that were previously corrupt start working).