Own implementation of apk and app bundle signing

[Tomáš Procházka]

Hi.

Is there already any more official way how to implement own sign mechanism which will be used for apk and app bundles?

We are using a special server where we send the apk and it returns back signed version.

Currenty we are creating a custom task for every variant which handle remote sign in this way

project.android.applicationVariants.all { ApplicationVariant variant ->

    if (!isReleaseBuildType(variant)) {
        return
    }

    variant.outputs.each { ApkVariantOutput output ->
        project.getLogger().info "Remote signing enabled for build variant: $project.name $variant.name"

        // create a signing task for this
        project.tasks.create(
                "remoteSign${output.name.capitalize()}Apk", RemoteSignApkTask,
                new RemoteSignApkTask.ConfigAction(variant, output)
        )
    }

    variant.testVariant?.outputs?.each { ApkVariantOutput output ->
        project.getLogger().info "Remote signing enabled for build test variant: $project.name $variant.testVariant.name"

        // create a signing task for this
        project.tasks.create(
                "remoteSign${output.name.capitalize()}Apk", RemoteSignApkTask,
                new RemoteSignApkTask.ConfigAction(variant, output)
        )
    }
}

But it starting to be obsolete, it should be rewritten to more lazy way.  So I'm asking if there is any better way already.

[Jerome Dochez]

you should use Lazy  tasks

https://docs.gradle.org/current/userguide/lazy_configuration.html

How do you express the dependency on the Bundle/APK file you want to sign ?

--
You received this message because you are subscribed to the Google Groups "adt-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to adt-dev+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Tomáš Procházka]

Yes. I know that I should use lazy tasks. I'm on the way to rewrite it, it is because I'm asking if there is any better way.

Would be great if build plugin provides some interface and allows to inject own implementation of it. And allows doing that without a care about task creating and it's dependency.

The whole implementation of our tasks is here:

https://gist.github.com/tprochazka/09bc1fdf6da0de7ad0830c911d10028d

But it has several disadvantages currently:

1.) It does not sign app bundles now, just apk

2.) it doesn't work from Android Studio when the Instant run is enabled (project.android.applicationVariants.all is missing for some reason)

3.) It is not lazy enough

Dne středa 14. listopadu 2018 18:40:40 UTC+1 Jerome Dochez napsal(a):

[Jerome Dochez]

sorry for the delay in answering.

We don't have an API that allow you to register interest in either the Bundle or APK at the moment but we are working on it. 

I am not sure why you need to sign in IR mode, can you not use normal debug signatures for debug builds ?

In the meantime, you can use a incubating API so it will definitely change in the future : 

            class APKAccessTask extends DefaultTask {

                @org.gradle.api.tasks.InputFiles

                FileCollection apkFile;

                @javax.inject.Inject

                MappingFileUserTask(FileCollection apkFile) {

                  this.apkFile = apkFile

                  doLast {

                      def file = apkFile.get().getSingleFile()

                      println "APK File  " + file

                      // sign here...

                   }

                }

            }

            android {

                buildTypes {

                  release {

                      minifyEnabled = true

                  }

                }

                // create the apk access task and register it if necessary.

                applicationVariants.all { variant ->

                      def apkFile = variant.getFinalArtifact(com.android.build.gradle.internal.scope.InternalArtifactType.APK)

                                               // use com.android.build.gradle.internal.scope.InternalArtifactType.BUNDLE for bundles.

                      println "Creating signing task for " + variant.name

                      def mappingTask = tasks.create("remoteSign" + variant.name.capitalize(), APKAccessTask, apkFile)                   

                }

            }

Like I said, it will change in 3.4 or 3.5 or both so be ready for some pain when you upgrade. Eventually we will stabilize this.

[Tomáš Procházka]

Thank you very much! I will investigate it. 

And yes we are using default debug sign way for debug builds. This remote sign is used just for release builds.

First few notes:

• I'm expecting that your solution is valid for build plugin 3.3.X, right? It will not work on 3.2.X.
  
• It is interesting that you suggesting to use variant.getFinalArtifact(). I'm currently using variant.outputs.each, because if I know every variant can have multiple outputs if apk split is enabled, or it is already removed in 3.3 (there is still no source code available if I know, but my current solution still works in this way.)
  
• I'm signing also test apk by using  variant.testVariant?.outputs?, it is still possible?
• I still need to setup dependency on this APKAccessTask to handle that it will be run in the right time when apk or/and app bundle is prepared already. Currently, I'm using
  
  

  signTask.dependsOn output.packageApplication
  output.assemble.dependsOn signTask
  

As you can se here. And probably still need to call variant.outputsAreSigned = true.

[Jerome Dochez]

right, it's for 3.3.x, I am honestly not sure it would work or not with 3.2

if you use apk split then it complicate things quite a bit, I need to look. 

I think the same solution should work for test variants. 

for dependencies, you only need the second line as the dependency on the packageApplication is embedded in the FileCollection so your task is implicitly made dependent (as long as you declare the FileCollection as an Input of course). 

 

--

[Tomáš Procházka]

Hi.

I started implementing it according your example code. It looks that it working in this way.

I have just few more question.

apkFile = variant.getFinalArtifact(com.android.build.gradle.internal.scope.InternalArtifactType.APK)

def file = apkFile.get().getSingleFile()

println "APK File  " + file

Actually, it doesn't contain APK file itself, but just folder when APK is located. It quite weird. I can simply search all apks inside of this folder (there is always just one), but I was expecting a file when it is called getFinalArtifact.

And What is the benefit of using variant.getFinalArtifact instead of variant.outputs? The second one is not lazy?

Dne středa 28. listopadu 2018 1:50:27 UTC+1 Jerome Dochez napsal(a):

[Tomáš Procházka]

And one more problem is how to setup dependency on bundle task for variant?

For APK I did simply

variant.assembleProvider.configure {
    dependsOn(signTask)
}

But there is nothing for bundle task.

Dne sobota 5. ledna 2019 0:10:32 UTC+1 Tomáš Procházka napsal(a):

[Tomáš Procházka]

You wrote

for dependencies, you only need the second line as the dependency on the packageApplication is embedded in the FileCollection so your task is implicitly made dependent (as long as you declare the FileCollection as an Input of course). 

But it looks that this is not enough to setup dependency on bundle task.

Dne sobota 5. ledna 2019 2:15:52 UTC+1 Tomáš Procházka napsal(a):

[Tomáš Procházka]

So. I already understand it.

BuildableArtifact provide dependency on PackageApplication task for APK and PackageBundleTask for bundle.

This works correctly. When I call my sign task it automaticaly run PackageApplication  or PackageBundleTask .

But I don't want to run it manually I want to have signed apk or bundle when I call assembleRelease o bundleRelease

So I need to declare this depenency for APK

variant.assembleProvider?.configure {
    dependsOn(signTask)
}

Then my sign taks will be automatical run between package and assemble and will be completely hidden for all developers.

But for app bundle it is not so easy, but I found hack how to get a bundle task:

variant.getVariantData().getTaskContainer().bundleTask?.configure {
    it.dependsOn(bundleSignTask)
}

And another important observation.

As I already wrote,  variant.getFinalArtifact(InternalArtifactType.APK) return just the folder name where APK is stored, but not a APK itself by

println "Single file: " + artifact.get().getSingleFile()

I will get sample\build\outputs\apk\full\release

But for variant.getFinalArtifact(InternalArtifactType.BUNDLE)

I will get sample\build\outputs\bundle\fullRelease\sample.aab

It's bug?

Also, the folder structure for apk and aab is quite non-uniform.

[Tomáš Procházka]

Last thing that I need to have signed is apk used fort connectedAndroidTest

So I mean this one sample\build\outputs\apk\androidTest\full\release\

I tried to use

variant.getFinalArtifact(InternalArtifactType.APK_FOR_LOCAL_TEST)

but it is always empty. Only way hot to get it is still variant.testVariant?.outputs?. Or miss I something?

[Tomáš Procházka]

And it is still not the end :-(

I'm able to get the APK to my SignTask in the new way by using BuildableArtifact, but how to put it to the input of next task I mean assemble, bundle or connectedAndroidTest tasks :-(

Before when I was using variant.outputs. I simply get instance of ApkVariantOutput a and I simply set output.outputFileName = signTask.outputFile.getName()

Whole code looks like this:

File assembleOutputFile = output.outputFile
signTask.outputFile = new File(assembleOutputFile.parent, assembleOutputFile.name.replaceFirst("[,-]unsigned", ""))
signTask.inputFile = signTask.outputFile
// hack to keep proper final name also when assemble task will be skipped
output.outputFileName = signTask.outputFile.getName()




xxxxx

[Tomáš Procházka]

So, here is my final solution:

https://gist.github.com/tprochazka/457c7eebd044c0210dcc8ba49301cda9

It's quite complicated. I'm using not public API, but it looks that it works currently.

I created a new feature request to make it possible in some easier way https://issuetracker.google.com/issues/122883577

[Tomáš Procházka]

Hi. 

Please, I need help again.

My custom sign mechanism is again broken in plugin 4.6.0.

InstallableVariantImpl.getFinalArtifact now return a different value, insead of BuildableArtifact it is there now Provider<FileCollection>.

It is also necessary to call it in this way from Groovy now variant.getFinalArtifact(new InternalArtifactType.APK())

I get Provider instance correctly, but the collection is always empty.

@TaskAction
void sign() {
    println '>>>>>>>>>>>>>> A2 sign task: ' + inputFiles.get().files.size()

This is always 0.

I'm calling get() in my task, which is registered in this way

variant.assembleProvider.configure {
    dependsOn(signTask)
}

Dne čtvrtek 17. ledna 2019 12:40:06 UTC+1 Tomáš Procházka napsal(a):

[Tomáš Procházka]

I will reply myself. The correct form of getFinalArtifact parameter is InternalArtifactType.APK.INSTANCE

Dne neděle 13. října 2019 23:56:14 UTC+2 Tomáš Procházka napsal(a):

[Jerome Dochez]

Hi Tomas

we are getting closer to provide a new API that will be stable so you want have to handle such changes. 

--

You received this message because you are subscribed to the Google Groups "adt-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to adt-dev+u...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/adt-dev/55e89d8f-5216-4ff5-b2e0-765f80025438%40googlegroups.com.

[Tomáš Procházka]

Hi Jerome. Thanks.

Btw. Best would be if there will be a direct way how to replace default sign mechanism, just by implementing some interface ;-)

Without modifying tasks and dependencies between them.

Dne pátek 18. října 2019 19:43:29 UTC+2 Jerome Dochez napsal(a):

To unsubscribe from this group and stop receiving emails from it, send an email to adt...@googlegroups.com.

[Jerome Dochez]

yes there will be : 

1. disable all signing in the DSL (v1 and v2)

2. obtain the built APK with new variant API

3. sign them.

To unsubscribe from this group and stop receiving emails from it, send an email to adt-dev+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/adt-dev/cae178f3-d127-4803-9062-8bacbeebd250%40googlegroups.com.

[Tomáš Procházka]

Hi.

If. Understand it correctly.

New variant API is mentioned here https://youtu.be/OTANozHzgPc?t=995

It is currently possible with 3.6.0 ?

Is there some more advanced article or video about this API or documentation somewhere?

For example, I'm not sure if I should use replace or transform to replace the default sign mechanism?

And I also need one new thing. I would like to count hash of all APK and AAB produced during the build.

So I maybe can use register, but I don't want to force production of APK or AAB. I want just wait when it happens so when user call  bundle or assemble I need to know about every apk or bundle that will be produced.

It is possible?

Dne pondělí 21. října 2019 18:15:13 UTC+2 Jerome Dochez napsal(a):

To view this discussion on the web visit https://groups.google.com/d/msgid/adt-dev/cae178f3-d127-4803-9062-8bacbeebd250%40googlegroups.com.

[Jerome Dochez]

it's not possible to do this in 3.6. We are hoping to deliver most of this during the 4.1 and 4.2 releases. 

for your second question, you probably will just need to get the PublicArtifactType.APK artifacts and have a task that depend on them. 

this should actually already work in 4.0 with relatively (!) stable APIs, I can slap an example next week if you are interested

To unsubscribe from this group and stop receiving emails from it, send an email to adt-dev+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/adt-dev/76a13511-fa1b-4737-9c20-8f5425867438%40googlegroups.com.

[Tomáš Procházka]

Write some extension for Android Gradle plugin is always a nightmare. I really hate it. Maybe convert everything to Kotlin will help, because I historically have everything in Groovy.

Autocompletion doesn't work in most cases and there is no source code of android build plugin, last tag here https://android.googlesource.com/platform/manifest/+refs is 3.4.0.

Most of StackOverflow examples are obsolete already.

Even such an easy operation like collect all finished APK/AAB is really problematic.

Dne pátek 6. března 2020 19:02:36 UTC+1 Jerome Dochez napsal(a):

To view this discussion on the web visit https://groups.google.com/d/msgid/adt-dev/76a13511-fa1b-4737-9c20-8f5425867438%40googlegroups.com.

[Tomáš Procházka]

Hi. Thanks. An example would be really great. So in 3.6 is the only possible way to found last one task which produces APK and do my stuff in the doLast {} closure, right?

Dne pátek 6. března 2020 19:02:36 UTC+1 Jerome Dochez napsal(a):

it's not possible to do this in 3.6. We are hoping to deliver most of this during the 4.1 and 4.2 releases. 

To view this discussion on the web visit https://groups.google.com/d/msgid/adt-dev/76a13511-fa1b-4737-9c20-8f5425867438%40googlegroups.com.

[Tomáš Procházka]

This is super useful https://github.com/android/gradle-recipes 

Would be possible to add there how to

• rename output apk/aab to a custom name
• write custom apk/aab signer

Please.

Dne čtvrtek 21. května 2020 8:56:13 UTC+2 Tomáš Procházka napsal(a):

[Jerome Dochez]

we have examples that show to copy the apk to a new location, I am sure this can be extrapolated to cover what you suggest.

I am not suggesting your examples are bad but I would like to strike a balance between being useful and having too many examples that confuse the users + maintenance costs for us. 

wdyt ?

To unsubscribe from this group and stop receiving emails from it, send an email to adt-dev+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/adt-dev/4aecddaa-33a0-4282-aede-5d076a259176o%40googlegroups.com.

[Tomáš Procházka]

Copy APK to a new location is maybe also possible. But so far I was using just output.outputFileName = ... like a much better way that creates a copy, because I want to keep the same location, I just want to have a better name. I'm currently using this

/**
* Renames APK or AAR name to contain version, build number and commit hash.
*/
project.ext.renameVariant = { BaseVariant variant ->
  variant.outputs.all { BaseVariantOutput output ->
    String gitHash = scmService.getLastCommitHashShort()

    def parentProjectName = (project.parent != null && project.parent.name != project.name) ? project.parent.name + '-' : ''
    def versionCodePart = variant.mergedFlavor.versionCode != null ? variant.mergedFlavor.versionCode + "-" : ""

    def name = output.outputFileName.replace("-", ",").replace(project.name.replace("-", ",") + ",",
    parentProjectName + project.name + "-" +
    versionProvider.get().getVersion() + "-" +
    versionCodePart + gitHash + "-")

    output.outputFileName = name
  }
}

But much more important for us is the possibility to replace the standard mechanism of production APK signing.

This is now completely broken in 4.1. According to one Gradle tooling talk from last year, I was thinking that there will be official API for it finally. Interesting that transition from 3.6.0 to 4.0.0 was complete without change and now transition to 4.1 is so big change.

Dne úterý 16. června 2020 v 0:30:07 UTC+2 uživatel Jerome Dochez napsal:

[Tomáš Procházka]

But it looks that replace Sign mechanism should be done in the same way as replace manifest merger which is shown here

https://github.com/android/gradle-recipes/blob/master/Groovy/manifestReplacementTest/app/build.gradle 

right? 

Dne čtvrtek 11. června 2020 v 20:33:23 UTC+2 uživatel Tomáš Procházka napsal:

[Jerome Dochez]

yes, just replace the artifact type with APK. 

To unsubscribe from this group and stop receiving emails from it, send an email to adt-dev+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/adt-dev/0d56caca-91f4-40d7-8069-055051881948n%40googlegroups.com.

[Tomáš Procházka]

I'm still not sure how to use it exactly, can you help a little bit. Currently, I have this code:

            android.onVariantProperties { ApplicationVariantProperties v ->

                TaskProvider signTaskProducer = tasks.register(

                        "remoteSign" + variant.name.capitalize() + "Apk",

                        RemoteSignTask)

                v.artifacts.use(signTaskProducer)

                .wiredWith( ? )

                .toCreate(ArtifactType.APK.INSTANCE)

            }

But I'm not sure what is responsible for produce unsigned task.

And I need to sign APK, AAB and also test APK

And inside of RemoteSignTask task I need to get location of all files which should be fixed.

My original code working until 4.0.0 was looking like this:

            project.android.applicationVariants.each { ApplicationVariant variant ->

                if (!isReleaseBuildType(variant)) {

                    project.logger.info("Skipping signing, '$variant.name' is not release.")

                    return

                }

                variant.outputs.all { ApkVariantOutput output ->

                    // hack to keep proper final name also when assemble task will be skipped

                    output.outputFileName = RemoteSignTask.renameToSignedName(output.outputFileName)

                }

                variant.setOutputsAreSigned(true)

                // Regular APK signing

                def apk = variant.getFinalArtifact(InternalArtifactType.APK.INSTANCE)

                InternalArtifactType.APK.INSTANCE

                def signTask = project.tasks.register(

                        "remoteSign" + variant.name.capitalize() + "Apk", RemoteSignTask,

                        new RemoteSignTask.ConfigAction(variant, apk))

                // FIXME: IT should be completely outside this task, but it is not possible now

                def folderManifestTaskForApk = project.tasks.register(

                        "folderManifest" + variant.name.capitalize() + "Apk", FolderManifestTask,

                        new FolderManifestTask.ConfigAction(variant, apk))

                variant.assembleProvider.configure {

                    it.dependsOn(signTask)

                    it.dependsOn(folderManifestTaskForApk)

                }

                signTask.configure {

                    folderManifestTaskForApk.get().dependsOn(it)

                }

                // App bundle signing

                def bundle = variant.getFinalArtifact(InternalArtifactType.BUNDLE.INSTANCE)

                def bundleSignTask = project.tasks.register(

                        "remoteSign" + variant.name.capitalize() + "Bundle", RemoteSignTask,

                        new RemoteSignTask.ConfigAction(variant, bundle))

                // FIXME: IT should be completely outside this tak, but it is not possible now

                def folderManifestTaskForAAB = project.tasks.register(

                        "folderManifest" + variant.name.capitalize() + "Bundle", FolderManifestTask,

                        new FolderManifestTask.ConfigAction(variant, bundle))

                variant.getVariantData().getTaskContainer().bundleTask.configure {

                    it.dependsOn(bundleSignTask)

                    it.dependsOn(folderManifestTaskForAAB)

                }

                bundleSignTask.configure {

                    folderManifestTaskForAAB.get().dependsOn(it)

                }

                if (variant.testVariant != null ) {

                    // Test APK signing

                    def testApk = variant.testVariant.getFinalArtifact(InternalArtifactType.APK.INSTANCE)

                    def testSignTask = project.tasks.register(

                            "remoteSign" + variant.name.capitalize() + "AndroidTest", RemoteSignTask,

                            new RemoteSignTask.ConfigAction(variant.testVariant, testApk))

                    variant.testVariant.assembleProvider.configure {

                        dependsOn(testSignTask)

                    }

                    variant.testVariant.connectedInstrumentTestProvider.configure {

                        dependsOn(testSignTask)

                    }

                }

            }

Dne středa 24. června 2020 v 7:00:11 UTC+2 uživatel Jerome Dochez napsal:

[Jerome Dochez]

toCreate API is used to create an artifact from its input, so here that would mean creating the APK file from dex, merged manifests, resources, etc... 

what you want is transform : 

val transformationRequest = artifacts.use(singingTaskProvider)
    .wiredWithDirectories(
        SigningTask::inputApkFolder,
        SingingTas::outputAPKFolder)
    .toTransformMany(ArtifactType.APK)

look at WorkerEnabledTransformation, it should be very similar except you copy and sign the files instead of just copying

you can do AAB but I need to add test APK which right now are not available through the public artifact types. 

To view this discussion on the web visit https://groups.google.com/d/msgid/adt-dev/2bb989c1-97d1-4d7c-9603-53eaf0ce66dcn%40googlegroups.com.

[Tomáš Procházka]

I'm sorry, but this API doesn't make sense at all for me.

This WorkerEnabledTransformation example just takes the final APK and copy it somewhere else.

But I need to replace the default sign mechanism provided by the Android plugin and nobody else (a different plugin) should know about.

So technically I need to tell that my task needs unsigned APK on input and final APK on output.

If I understand correctly this talk

https://youtu.be/OTANozHzgPc?t=1023

With new API I should not care about the actual task which is responsible o signing. I should care just about input and output artifacts.

So I should tell that I have a task that can handle unsigned APK on the input and can sign it. And I want to replace any other task, which would like do the same.

The replace mechanism described in the talk at https://youtu.be/OTANozHzgPc?t=1023 gives much more sense for me.

I would expect something like this:

artifact.use(signTaskProducer).toCreate(ArtifactType.APK.INSTANCE).from(ArtifactType.UNSIGNED_APK.INSTANCE)

Dne středa 24. června 2020 v 21:54:32 UTC+2 uživatel Jerome Dochez napsal:

[Jerome Dochez]

we do not have such a thing as an unsigned APK (even internally because we package and sign simultaneously). 

what you need to do is disable signing alltogether for your app by turning off v1 and v2 signing through the DSL, then your APK would be unsigned...

To view this discussion on the web visit https://groups.google.com/d/msgid/adt-dev/1d22642d-3d74-456e-af8b-8e17eda0d055n%40googlegroups.com.

[Tomáš Procházka]

But it looks that not understanding this API is smaller problem.

I found that android.onVariantProperties is never called when I call it inside of project.afterEvaluate
I'm doing it because I have own configuration extension inside of my plugin.

So setup looks like

apply .....

myConfigExtension {

    mysetup

}

And in the time od signing configuration I need to know about values in myConfigExtension

Without it it at least run the first println, but it never call RemoteSignTask

            android.onVariantProperties { ApplicationVariantProperties v ->

                println 'xxxxx' +  "remoteSign" + v.name.capitalize() + "Apk"

                TaskProvider singingTaskProvider = project.tasks.register(

                        "remoteSign" + v.name.capitalize() + "Apk",

                        RemoteSignTask)

                //def transformationRequest =

                v.artifacts.use(singingTaskProvider)

                        .wiredWithDirectories(

                                { it.apkFolder }, { it.apkFolder }

                        )

                        .toTransformMany(ArtifactType.APK.INSTANCE)

            }

Dne čtvrtek 25. června 2020 v 2:23:37 UTC+2 uživatel Jerome Dochez napsal:

[Jerome Dochez]

yes it will not necessarily work if you call from the afterEvaluate because that's where we hook up the AGP in Gradle and the order matters (you are coming after us)

you need to find a way to move your code out.

btw, do NOT use the same folder for input and output, one is to read the unsigned APK, the other is to write out the signed, you should not use the same folder.

To view this discussion on the web visit https://groups.google.com/d/msgid/adt-dev/4113877e-5987-4d9c-bc5e-d008fdc14e6fn%40googlegroups.com.