Multidex bug with dependent jars (legacy multidex mode, no minify)

[Zac Bowling]

So I'm fairly sure there is a multidex bug in the gradle plugin from an assumption of how project are laid out. I post this on the bug tracker in a bit but I wanted to post it here first. 

Imagine a project where your activities and services are actually being built in subprojects or in your jar dependencies and not in the top level app project. The Progaurd step to make the components.jar only passing in the jar for the current Gradle project but not all the dependent ones when it makes that components.jar. 

In BasePlugin.groovy after:

        ProGuardTask proguardComponentsTask = createShrinkingProGuardTask(project,

                    "shrink${config.fullName.capitalize()}MultiDexComponents")

The proguard is step is only using:

       proguardComponentsTask.injars(pcData.inputFiles.call().iterator().next())

If your dependencies are the ones that have all your classes the step will just fail (with "java.io.IOException: The output jar is empty. Did you specify the proper '-keep' options?"). If you get it to sort of work by having just a single class reference work I believe that progaurd will not produce what you expect it to feed into the next step when it that private method in dx.

I think the entire process of asking Progaurd to shink may be an issue. I believe all dependencies should be added or that the allclasses.jar should be used instead. 

Also these two items are forced to always be added to the -keep list and they still warn with the same error:

Note: the configuration refers to the unknown class 'android.app.backup.BackupAgent'

Note: the configuration refers to the unknown class 'java.lang.annotation.Annotation'

Here is an example of what we see with the error in our cases:

:app:shrinkDebugMultiDexComponents

Executing task ':app:shrinkDebugMultiDexComponents' (up-to-date check took 0.002 secs) due to:

  Input file /private/tmp/TestShit/app/build/intermediates/multi-dex/debug/allclasses.jar has changed.

  Input file /private/tmp/TestShit/app/build/intermediates/multi-dex/debug/manifest_keep.txt has changed.

ProGuard, version 4.11

Reading program jar [/private/tmp/TestShit/app/build/intermediates/multi-dex/debug/allclasses.jar]

Note: the configuration refers to the unknown class 'com.apportable.app.VerdeApplication'

Note: the configuration refers to the unknown class 'android.content.Context'

Note: the configuration refers to the unknown class 'com.apportable.activity.VerdeActivity'

Note: the configuration refers to the unknown class 'com.apportable.activity.GdbServerService'

Note: the configuration refers to the unknown class 'com.apportable.media.AudioManagerService'

Note: the configuration refers to the unknown class 'com.apportable.activity.ManageSpaceActivity'

Note: the configuration refers to the unknown class 'com.apportable.notifications.NotificationsBootReceiver'

Note: the configuration refers to the unknown class 'com.apportable.notifications.NotificationsAlarmReceiver'

Note: the configuration refers to the unknown class 'com.apportable.notifications.LocalNotificationService'

Note: the configuration refers to the unknown class 'com.apportable.iap.BillingService'

Note: the configuration refers to the unknown class 'com.apportable.iap.BillingReceiver'

Note: the configuration refers to the unknown class 'com.facebook.LoginActivity'

Note: the configuration refers to the unknown class 'com.adjust.sdk.ReferrerReceiver'

Note: the configuration refers to the unknown class 'com.mobileapptracker.Tracker'

Note: the configuration refers to the unknown class 'com.amazon.device.ads.AdActivity'

Note: the configuration refers to the unknown class 'com.google.android.gms.ads.AdActivity'

Note: the configuration refers to the unknown class 'com.millennialmedia.android.MMActivity'

Note: the configuration refers to the unknown class 'com.millennialmedia.android.VideoPlayerActivity'

Note: the configuration refers to the unknown class 'com.rhythmnewmedia.sdk.RhythmActivity'

Note: the configuration refers to the unknown class 'com.rhythmnewmedia.sdk.video.RhythmVideoActivity'

Note: the configuration refers to the unknown class 'com.applovin.adview.AppLovinInterstitialActivity'

Note: the configuration refers to the unknown class 'com.inmobi.androidsdk.IMBrowserActivity'

Note: the configuration refers to the unknown class 'com.jirbo.adcolony.AdColonyOverlay'

Note: the configuration refers to the unknown class 'com.jirbo.adcolony.AdColonyFullscreen'

Note: the configuration refers to the unknown class 'com.jirbo.adcolony.AdColonyBrowser'

Note: the configuration refers to the unknown class 'com.rfm.sdk.HTMLBrowserView'

Note: the configuration refers to the unknown class 'com.mopub.common.MoPubBrowser'

Note: the configuration refers to the unknown class 'com.mopub.mobileads.MoPubActivity'

Note: the configuration refers to the unknown class 'com.mopub.mobileads.MraidActivity'

Note: the configuration refers to the unknown class 'com.mopub.mobileads.MraidVideoPlayerActivity'

Note: the configuration refers to the unknown class 'com.unity3d.ads.android.view.UnityAdsFullscreenActivity'

Note: the configuration refers to the unknown class 'com.scopely.zendeskhelper.ui.RequestActivity'

Note: the configuration refers to the unknown class 'com.scopely.zendeskhelper.ui.RequestCompletionActivity'

Note: the configuration refers to the unknown class 'android.app.backup.BackupAgent'

Note: the configuration refers to the unknown class 'java.lang.annotation.Annotation'

Note: android.support.multidex.MultiDexExtractor accesses a method 'apply()' dynamically

Note: there were 39 references to unknown classes.

      You should check your configuration for typos.

      (http://proguard.sourceforge.net/manual/troubleshooting.html#unknownclass)

Note: there were 1 accesses to class members by means of introspection.

      You should consider explicitly keeping the mentioned class members

      (using '-keep' or '-keepclassmembers').

      (http://proguard.sourceforge.net/manual/troubleshooting.html#dynamicalclassmember)

Note: You're ignoring all warnings!

:app:shrinkDebugMultiDexComponents FAILED

:app:shrinkDebugMultiDexComponents (Thread[main,5,main]) completed. Took 0.122 secs.

FAILURE: Build failed with an exception.

* What went wrong:

Execution failed for task ':app:shrinkDebugMultiDexComponents'.

> java.io.IOException: The output jar is empty. Did you specify the proper '-keep' options?

* Try:

Run with --stacktrace option to get the stack trace. Run with --debug option to get more log output.

[Xavier Ducrohet]

We have a jar-merging task running before this task that should be merging everything into allclasses.jar

--
You received this message because you are subscribed to the Google Groups "adt-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to adt-dev+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Xavier Ducrohet
Android SDK Tech Lead
Google Inc.
http://developer.android.com | http://tools.android.com

Please do not send me questions directly. Thanks!

[Zac Bowling]

It’s not adding all the dependencies to that allclasses.jar though for the ProGaurd pass. I think I know what to do to fix the logic in BasePlugin/JarMergingTask but I can’t easily rebuild the plugin at the moment. The JarMergingTask is pulling in only inputDir and not any of the inputLibraries. The inputDir only has the out of the java compile step (attaching jdb and dumping the state while stepping through). The dependencies are in inputLibraries so those classes are not pulled in.

That explains why allclasses.jar is almost totally empty in my specific case. Thats why it fails in the progaurd pass (see the error below). 

Without turning on the mutlidex thought all those dependencies do get added to the final dex correctly. Also if I add a test activity to the main project for it to compile with mutlidex on, I get the same progaurd warnings but the final dex will have all the dependencies as well but the main list of expected classes to be kept in the primary dex is not respected (that the manifest scanning step gets from the manifest) since those do not make it in the components.jar. Dx then just picks classes purely on it’s own and some activities, services, broadcast receivers, code called early in JNI, etc that live in my dependencies get put in classes2.dex instead of forcing them to the primary dex.

I can send you a sample if you want one. (I don’t want to post it to the list because it may contain client sensitive data but I can email you directly)

Thanks!

Zac

You received this message because you are subscribed to a topic in the Google Groups "adt-dev" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/adt-dev/4VVvCr8r3EE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to adt-dev+u...@googlegroups.com.

[Zac Bowling]

Oh sorry. Omg. Wow. I have no idea how things have been working. 

So these jars that have been coming from older build system actually have been predex’d. They don’t contain classes but dex files themselves. They work oddly in the rest of gradle except for this step.

Wow. Hahaha. Sorry about that.

[Xavier Ducrohet]

multi-dex should disable pre-dexing unless you are targeting 21+

Are you still have problems?

[Zac Bowling]

No, sorry, this was a bug in our current build system in our SDK at Apportable. Thanks though!

I’m not sure if you are curious about the details and I think I’ve mentioned this a little bit in forum, but basically we are adding support for generating AARs so that our SDK users can consume our platform with their native C++/Objective-C code into their Android Studio/Gradle projects instead of having to use our build system entirely to make APKs. This is mostly for non-game devs that want to develop native Android UIs around their cross platform native core code they share with iOS (and possibly other platforms) without having to get their Java sources, resources, and manifest changes to work our high level system like we used to require. 

Basically our build system is capable of generating APKs on it’s own (and has been for a years) and I’m reusing a lot of that existing code everywhere in our AAR generation target. The pre-dexing step was happening in our system because we do it for APK targets and I didn’t realize it was still happening for AAR targets. The jars being fed into Gradle via our AARs we being pre-dexed on our end. When that wasn’t working I was working a raw sample project without AARs to try and just get our jars in the intermediate phase to work directly in the gradle (not realizing they were being pre-dex’d). 

Our build system is ok at building Java but it’s much better at building cross platform native code (supports things you can’t find in another build system for taking advantage of some big features of gold and clang and we also have native support for native Xcode projects for people trying to port and maintain their shared iOS/Android code in the same IDE they use for iOS). It’s based on a very forked version of SCONS that we modified to use Ninja on the backend and works with a set of native plugins in Xcode for real Xcode integration to target Android directly in the IDE. Before Gradle came around, we created this build system to solve a lot problems we couldn’t solve with ANT, make, or anything else out there that was already in place (I still have nightmares from the many man months spent reverse engineering ANT and to try and get aapt and R working in our platform properly... much undocumented. very pain. wow.)

If you are purely porting from iOS to Android and don’t need to do anything more than the most trivial Java integrations with Android, our existing build system is perfect (which is why we do really well with game developers at Apportable who only care about native code and don’t want to deal with native UI since they are providing one themselves in GL), but as we try to make our platform more useful for the existing Android ecosystem so developers can reuse common backend code in regular apps, we don’t want to fight against the grain. Gradle is becoming ubiquitous and defacto and for solving the problems that the Java devs have to deal with (like R and dex issues like this) and we don’t want to reinvent the wheel and just stick to making the Android native experience better so people can share the same code where possible on the backends of their apps.  

TLDR: it was my mistake on what I was feeding into gradle. :-)