Using Resolver to get google play services

[Douglas Jones]

I find myself needing a new version of the Google Play Services fitness library but it's no longer just in the sdk extras, but now is in maven.

Bringing myself up to modern times I find remote_file() which looks just like what I need and in figuring that out learn about Resolver which should build the buck library dependencies for me straight from a maven repository. Great!

Unfortunately when I try what I think needs to be done it just tells me it can't find various dependencies. Can someone point out where I'm going wrong? Are my arguments wrong? Is there something special for Google's maven? Is this all meant to be run via buck run when it can potentially utilize elements of the buckconfig?

java -jar ~/resolver.jar -repo libs -third-party libs -visibility PUBLIC -maven https://maven.google.com/ com.google.android.gms:play-services-fitness:15.0.1

ends up giving me this:

Exception in thread "main" org.eclipse.aether.resolution.DependencyResolutionException: The following artifacts could not be resolved: com.google.android.gms:play-services-fitness:jar:15.0.1, com.android.support:support-compat:jar:26.1.0, android.arch.lifecycle:runtime:jar:1.0.0, com.android.support:support-media-compat:jar:26.1.0, com.android.support:support-core-utils:jar:26.1.0, com.android.support:support-core-ui:jar:26.1.0, com.android.support:support-fragment:jar:26.1.0: Could not find artifact com.google.android.gms:play-services-fitness:jar:15.0.1 in https://maven.google.com/ (https://maven.google.com/)

at org.eclipse.aether.internal.impl.DefaultRepositorySystem.resolveDependencies(DefaultRepositorySystem.java:384)

at com.facebook.buck.maven.Resolver.getRunTimeTransitiveDeps(Resolver.java:554)

at com.facebook.buck.maven.Resolver.resolve(Resolver.java:178)

at com.facebook.buck.maven.Resolver.main(Resolver.java:589)

Caused by: org.eclipse.aether.resolution.ArtifactResolutionException: The following artifacts could not be resolved: com.google.android.gms:play-services-fitness:jar:15.0.1, com.android.support:support-compat:jar:26.1.0, android.arch.lifecycle:runtime:jar:1.0.0, com.android.support:support-media-compat:jar:26.1.0, com.android.support:support-core-utils:jar:26.1.0, com.android.support:support-core-ui:jar:26.1.0, com.android.support:support-fragment:jar:26.1.0: Could not find artifact com.google.android.gms:play-services-fitness:jar:15.0.1 in https://maven.google.com/ (https://maven.google.com/)

at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:444)

at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifacts(DefaultArtifactResolver.java:246)

at org.eclipse.aether.internal.impl.DefaultRepositorySystem.resolveDependencies(DefaultRepositorySystem.java:367)

... 3 more

Caused by: org.eclipse.aether.transfer.ArtifactNotFoundException: Could not find artifact com.google.android.gms:play-services-fitness:jar:15.0.1 in https://maven.google.com/ (https://maven.google.com/)

at org.eclipse.aether.connector.basic.ArtifactTransportListener.transferFailed(ArtifactTransportListener.java:39)

at org.eclipse.aether.connector.basic.BasicRepositoryConnector$TaskRunner.run(BasicRepositoryConnector.java:355)

at org.eclipse.aether.util.concurrency.RunnableErrorForwarder$1.run(RunnableErrorForwarder.java:67)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:745)

If I manually specify an aar format like this:

java -jar ~/resolver.jar -repo libs -third-party libs -visibility PUBLIC -maven https://maven.google.com/ com.google.android.gms:play-services-fitness:aar:15.0.1

then it seems a little happier and gives me this (a little happier in that it doesn't complain about the initial component):

Exception in thread "main" org.eclipse.aether.resolution.DependencyResolutionException: The following artifacts could not be resolved: com.android.support:support-compat:jar:26.1.0, android.arch.lifecycle:runtime:jar:1.0.0, com.android.support:support-media-compat:jar:26.1.0, com.android.support:support-core-utils:jar:26.1.0, com.android.support:support-core-ui:jar:26.1.0, com.android.support:support-fragment:jar:26.1.0: Could not find artifact com.android.support:support-compat:jar:26.1.0 in https://maven.google.com/ (https://maven.google.com/)

at org.eclipse.aether.internal.impl.DefaultRepositorySystem.resolveDependencies(DefaultRepositorySystem.java:384)

at com.facebook.buck.maven.Resolver.getRunTimeTransitiveDeps(Resolver.java:554)

at com.facebook.buck.maven.Resolver.resolve(Resolver.java:178)

at com.facebook.buck.maven.Resolver.main(Resolver.java:589)

Caused by: org.eclipse.aether.resolution.ArtifactResolutionException: The following artifacts could not be resolved: com.android.support:support-compat:jar:26.1.0, android.arch.lifecycle:runtime:jar:1.0.0, com.android.support:support-media-compat:jar:26.1.0, com.android.support:support-core-utils:jar:26.1.0, com.android.support:support-core-ui:jar:26.1.0, com.android.support:support-fragment:jar:26.1.0: Could not find artifact com.android.support:support-compat:jar:26.1.0 in https://maven.google.com/ (https://maven.google.com/)

at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:444)

at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifacts(DefaultArtifactResolver.java:246)

at org.eclipse.aether.internal.impl.DefaultRepositorySystem.resolveDependencies(DefaultRepositorySystem.java:367)

... 3 more

Caused by: org.eclipse.aether.transfer.ArtifactNotFoundException: Could not find artifact com.android.support:support-compat:jar:26.1.0 in https://maven.google.com/ (https://maven.google.com/)

at org.eclipse.aether.connector.basic.ArtifactTransportListener.transferFailed(ArtifactTransportListener.java:39)

at org.eclipse.aether.connector.basic.BasicRepositoryConnector$TaskRunner.run(BasicRepositoryConnector.java:355)

at org.eclipse.aether.util.concurrency.RunnableErrorForwarder$1.run(RunnableErrorForwarder.java:67)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:745)

[Douglas Jones]

Trying one of the listed dependencies it claims it couldn't find appears to have downloaded some of its dependencies before giving up and not finding what was asked for, so at least I know something can get downloaded

How do people using buck go about downloading the various Google support and play services libraries? It seems like that is a very common usage and should be documented somewhere 

[Shawn Wilsher]

Note that `remote_file` does not fetch the dependencies of the target
specified. You have to declare *all* of those dependencies as
`remote_file`s as well.

[Douglas Jones]

I'm okay with declaring all the dependencies as remote_files, but I was kind of hoping the Resolver would generate that part of the BUCK file for me, or at least generate the local prebuilt rules and dependencies. It keeps hitting dead ends in resolution before it gets to the BUCK generation phase. Something download okay but some things pull down a skeleton 

[Douglas Jones]

Okay, back on this now...

Just as an experiment seeing what gets pulled into a local maven repository when trying to run Resolver on a particular target with just a few dependencies:

java -jar ~/resolver.jar -repo repo -local-maven lm2 -visibility PUBLIC -maven https://maven.google.com/ com.android.support:support-compat:aar:26.1.0

It pulled down all dependencies except for the android.arch.lifecycle runtime library (1.0.0). The pom files came down for it but no archive; it died claiming it could find the jar for it. So I tried again just targeting that specific library except I specified an aar...

java -jar ~/resolver.jar -repo repo -local-maven lm2 -visibility PUBLIC -maven https://maven.google.com/ android.arch.lifecycle:runtime:aar:1.0.0

The result was the aar came down and then it again threw an exception that it couldn't find the jar for that artifact. I haven't dug into the pom files to check all their artifact references yet but I'm thinking the Resolver needs some first aid

[Douglas Jones]

I'm giving up on the resolver as it appears to have been built in a time when everything existed as both jar and aar files. Now most google code for Android appears to be aar files so it won't play along

It appears that the aether library is not honoring the <package> tag and so will not grab aar files if they are dependencies while working fine if it is the primary target

If it were to get to the resolveLib code, it would then send the extension (package type) as "jar" which would make it favor jar files over aar files

Finally, the buck file generation, which is the part I was really excited about will only generate targets using prebuilt_jar, again not aware that the world of Android is often populated with aar files now

[Douglas Jones]

Okay, this is my final answer, and I actually have a solution! From a lot of reading, it seems the aether library that Resolver (and Maven) use just can't deal with whatever Google has done on their maven repository. Apparently gradle uses the ivy library which can download all of the Android support and play services libraries and hunt down all or their dependencies.

I found a good example gradle file to pull down the dependencies and put them in a specific directory. I'd link to it but I appear to have lost that window.

apply plugin: 'java'

repositories {

   google()

   jcenter()

   mavenCentral()

}

dependencies {

  compile "com.android.support:support-v4:26.1.0"

  compile "com.android.support:appcompat-v7:26.1.0"

  compile "com.android.support:support-annotations:26.1.0"

  compile "com.android.support:percent:26.1.0"

  compile "com.android.support:design:26.1.0"

  compile "com.google.android.gms:play-services-gcm:15.0.1"

  compile "com.google.android.gms:play-services-maps:15.0.1"

  compile "com.google.android.gms:play-services-fitness:15.0.1"

  compile "com.google.android.gms:play-services-location:15.0.1"

  compile "com.google.android.gms:play-services-identity:15.0.1"

  compile "com.google.firebase:firebase-dynamic-links:16.0.1"

  compile "com.google.firebase:firebase-invites:16.0.0"

}

task copyDependencies(type: Copy) {

  from configurations.compile

  into 'latest_libs'

}

I then drive that gradle file with this bash script which generates a new BUCK file with all the prebuilt targets defined. From there I can use the dependency tree to generate android_lib targets with exported_deps to match the hierarchy

# This will put grab all the libraries and their dependencies in get_libs.gradle
# and put them in a safe space so you can get your BUCK file updated. An  
# initial BUCK file will be generated referncing all of the prebuilt targets downloaded

# nullglob makes sure the for loops before won't act weirdly if there aren't any jar or aar files
shopt -s nullglob

# Get all of the dependencies we want (change that list in get_libs.gradle)
gradle -b get_libs.gradle copyDependencies

# Put a copy of the dependency tree into a file for reference when building the android_library targets
gradle -b get_libs.gradle dependencies | sed -n '/^compile /,/^$/p' | sed -n '2,/^$/p' > dependency_tree

cd latest_libs

for f in *.aar
do
 # Gets the file name without the extension
 rulename=${f%.aar}
 # Removes the version information from the end
 rulename=${rulename%-*}
 echo -e "\nandroid_prebuilt_aar(\n  name = '$rulename',\n  aar = 'libs/$f',\n)\n" >> ../proto_BUCK
done

for f in *.jar
do
 # Gets the file name without the extension
 rulename=${f%.jar}
 # Removes the version information from the end
 rulename=${rulename%-*}
 echo -e "\nprebuilt_jar(\n  name = '$rulename',\n  binary_jar = 'libs/$f',\n)\n" >> ../proto_BUCK
done