Non official versions of Android support libraries starting to circluate

[William Ferguson]

What's the Android team's stance on the non-official versions of the Android support libraries?

Eg https://github.com/Prototik/HoloEverywhere/issues/842#issuecomment-49746122

These libraries have the same GAV (groupId, artifactId, version) as the official versions but have totally different contents. This means that the same project built on 2 different machines can produce radically different outputs (unbeknownst to the developers). Or even 2 libraries both listing the same dependency having very different needs and producing some nightmare when combined (again unbeknownst to the developer doing the combining).

These libraries are being published in the com.android.* namespace, so appear to be official Android team libraries which means developers are going to start coming to you guys for support as things start to break down at the edges.  If the Android team were actually publishing these artifacts into a public repository there wouldn't be a vacuum for incidents like this to occur.

So what's the plan to stop this hole getting bigger and deeper?

William

[Kevin Schultz]

William,

That definitely seems like a major problem with HoloEverywhere and I wouldn't use it. But is he actually publishing artifacts that impact non-HoloEverywhere users? I'm not sure I understand the connection between his misconfiguration & Google's publishing strategy?

[Xavier Ducrohet]

Where are they getting published?

If there were put on MavenCentral or JCenter that would be a problem but it looks like they are just on github? Someone wanting to use them has to go find them there, and that should be a red flag anyway.

--
You received this message because you are subscribed to the Google Groups "adt-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to adt-dev+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Xavier Ducrohet
Android SDK Tech Lead
Google Inc.
http://developer.android.com | http://tools.android.com

Please do not send me questions directly. Thanks!

[William Ferguson]

Xavier, that would be fine if it every developer was a solo developer AND they only ever used libraries directly and never transitively. But both of those cases are clearly false.

Yes, the HoloEverywhere versions are only available via the repo they are publishing on Github. But that it much more visible and more reachable than the version that are being distributed by the Android team which are only visible once you have manually downloaded those versions via the SDK Manager and then manually deployed them to your local Maven repository.

Maven GAVs are supposed to be unique. Once Maven finds a match it stops looking. So if the HoloEverywhere artifacts get into your repository first there is no chance of the real artifacts being downloaded (even if they were available on a public repo). Here's 2 ways that can happen:

1) You use a shared repository manager. Someone else has a project that explicitly references the HE GitHub repo and the HE artifacts. They build their project first and pollute the repository manager with the HE artifacts.

2) You use an Android library X. It (or one of its dependencies) lists an explicit reference to the HE Github repo. If you build a project using X before manually populating your Maven repo with the official support libraries you will have polluted your local Maven repo with the HE artifacts.

So there really isn't any red flag.

Even acting independently with best intent you have made yourself a unknowing victim.

This could largely be avoided if the Android team just published the artifacts through normal channels.

William

--
You received this message because you are subscribed to a topic in the Google Groups "adt-dev" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/adt-dev/2hPuSUYttbg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to adt-dev+u...@googlegroups.com.

[William Ferguson]

Hey Kevin, see my answer to Xavier for how this will impact on non HE users (especially in a shared repo environment).

The fact that the HoloEverywhere project has 2196 stars and 794 forks indicates that it very widely used. That means that as a disease vector it is highly contagious.

My point about Google publishing strategy is (to use the disease analogy again) that if Google were following the standard publishing mechanism and  publishing it's artifacts in Maven central then we would be somewhat inoculated from this rogue vector because 

1) Maven central is the first point of call when resolving dependencies. So you will get the official artifacts ahead of the HE artifacts.

2) Publishing in Maven central doesn't require manually downloading and deployment of each version of the Android artifacts for each developer. They become instantly available as soon as they are published by the Android team. So you remove attack via update latency.

3) Publishing the Android artifacts in central marks ownership of the namespace. It is less likely for someone to attempt to usurp that namespace elsewhere as users can see that the namespace is owned and being used. Ie its territory communication.

William

[Jake Wharton]

On Thu, Aug 7, 2014 at 2:20 PM, William Ferguson <william....@xandar.com.au> wrote:

1) Maven central is the first point of call when resolving dependencies. So you will get the official artifacts ahead of the HE artifacts.

This is not true. You can configure this behavior, but it's not Gradle's default.

Moreover, HE isn't publishing these to Maven central because they do not have the rights to this groupId.

 

2) Publishing in Maven central doesn't require manually downloading and deployment of each version of the Android artifacts for each developer. They become instantly available as soon as they are published by the Android team. So you remove attack via update latency.

This makes no sense to me.

 

3) Publishing the Android artifacts in central marks ownership of the namespace. It is less likely for someone to attempt to usurp that namespace elsewhere as users can see that the namespace is owned and being used. Ie its territory communication.

Both com.google.* and com.android.* are already widely used groupIds by Google for thousands of artifacts in central so this doesn't change anything.

What you are seeing is an old, legacy Maven project doing ugly hacks to try and make things work. It affects neither Gradle nor Ant builds which are the only supported mechanisms of building Android apps.

[Jonathan Steele]

If that is true, Why I see groupid is org.holoeverywhere in the maven central?

[Jake Wharton]

On Thu, Aug 7, 2014 at 3:48 PM, Jonathan Steele <xfsu...@gmail.com> wrote:

If that is true, Why I see groupid is org.holoeverywhere in the maven central?

Because that's their project name.

[William Ferguson]

On Fri, Aug 8, 2014 at 7:47 AM, Jake Wharton <jakew...@gmail.com> wrote:

On Thu, Aug 7, 2014 at 2:20 PM, William Ferguson <william....@xandar.com.au> wrote:

1) Maven central is the first point of call when resolving dependencies. So you will get the official artifacts ahead of the HE artifacts.

This is not true. You can configure this behavior, but it's not Gradle's default.

Correct, it is configurable. But it is by far the largest artifact repo and it is the default in Maven and the natural default for most projects using other dependency management mechanisms such as Ivy (via Ant for instance).

 

Moreover, HE isn't publishing these to Maven central because they do not have the rights to this groupId.

Correct, but I never claimed they were. My point is that because their artifacts are published to a publicly accessible web address their artifacts are more accessible than those being provided by the Android team.

 

 

2) Publishing in Maven central doesn't require manually downloading and deployment of each version of the Android artifacts for each developer. They become instantly available as soon as they are published by the Android team. So you remove attack via update latency.

This makes no sense to me

OK, let me try and explain. This is really a follow up on my last point. The barriers that are imposed by the SDK Manager being the distribution point for Android artifacts are two fold

A) Before you can consume any of the artefacts you need to start SDK manager, choose the appropriate versions of the required artifacts and then download them.

B) You then need to take those artifacts and deploy them to your local (or shared repo). 

Most developers don't use the SDK Manager to download the latest artifacts on a daily basis, they do it at irregular intervals. This means that there is a latency between when the artifacts become available and when the developer first has access to the official artifacts. That gap provides an opportunity for the developers local/shared repo to become polluted with non-official artifacts (potentially without the developer becoming aware of it).

As an aside this also means that you can't just fork an Android project and build and expect to be able to build it because the Android dependencies that it relies upon may not yet have been downloaded to your repo. This is a real pain for open source an example projects.

 .

3) Publishing the Android artifacts in central marks ownership of the namespace. It is less likely for someone to attempt to usurp that namespace elsewhere as users can see that the namespace is owned and being used. Ie its territory communication.

Both com.google.* and com.android.* are already widely used groupIds by Google for thousands of artifacts in central so this doesn't change anything.

Yes, but com.google.android has been dormant since August 2012 aside for 2 example apps. And nature abhors a vacuum to quote Aristotle.

 

What you are seeing is an old, legacy Maven project doing ugly hacks to try and make things work. 

If you are telling me that the support libraries can provide the Holo look and feel without need for HE that's great and I can ditch it from any of my projects that rely on it directly.

It affects neither Gradle nor Ant builds which are the only supported mechanisms of building Android apps.

 

This is not true. Any Gradle build or Ant build using dep management is at risk of this even if they haven't used HoloEverywhere directly themselves.

William

[William Ferguson]

It affects neither Gradle nor Ant builds which are the only supported mechanisms of building Android apps.

 

This is not true. Any Gradle build or Ant build using dep management is at risk of this even if they haven't used HoloEverywhere directly themselves.

I'd also like to point out that there was and still is a very vibrant Maven Android community long before Gradle was mentioned at IO13. If in doubt just listen to the recording of the Android fireside chat at IO12.

One of the things I like about Google's general approach is that it is very open, so that developers can pick up the tool that best fits their needs and apply it without much impediment. Not publishing the Android artifacts has introduced such an impediment and while the community has put up with the situation for the last 2 years, the cracks are starting to get wide enough to do some serious damage.

 

[Jake Wharton]

On Thu, Aug 7, 2014 at 4:45 PM, William Ferguson <william....@xandar.com.au> wrote:

Correct, but I never claimed they were. My point is that because their artifacts are published to a publicly accessible web address their artifacts are more accessible than those being provided by the Android team.

But not Maven central, therefore not a problem.

A) Before you can consume any of the artefacts you need to start SDK manager, choose the appropriate versions of the required artifacts and then download them.

Every developer does this. You need the API level you are compiling against. This isn't a problem

 

B) You then need to take those artifacts and deploy them to your local (or shared repo). 

Not for Gradle nor Ant, therefore not a problem for Google to worry about.

 

Most developers don't use the SDK Manager to download the latest artifacts on a daily basis, they do it at irregular intervals. This means that there is a latency between when the artifacts become available and when the developer first has access to the official artifacts.

I agree. I wrote a Gradle plugin to address this: https://github.com/JakeWharton/sdk-manager-plugin. I'd love for the first-party plugin to do this some day.

Ant users copy jars so they don't matter. And Maven isn't supported so it's not Google's concern. 

That gap provides an opportunity for the developers local/shared repo to become polluted with non-official artifacts (potentially without the developer becoming aware of it).

No, because nobody is publishing these artifacts to central. 

 

As an aside this also means that you can't just fork an Android project and build and expect to be able to build it because the Android dependencies that it relies upon may not yet have been downloaded to your repo. This is a real pain for open source an example projects.

See above plugin.

 

Yes, but com.google.android has been dormant since August 2012 aside for 2 example apps. And nature abhors a vacuum to quote Aristotle.

Because those artifacts weren't sanctioned. In fact, they should be removed.

 

If you are telling me that the support libraries can provide the Holo look and feel without need for HE that's great and I can ditch it from any of my projects that rely on it directly.

AppCompat. Again, HoloEverywhere and Maven are not Google's concern.

 

This is not true. Any Gradle build or Ant build using dep management is at risk of this even if they haven't used HoloEverywhere directly themselves.

Neither uses the local .m2 so no they aren't!

[Jake Wharton]

On Thu, Aug 7, 2014 at 4:53 PM, William Ferguson <william....@xandar.com.au> wrote:

I'd also like to point out that there was and still is a very vibrant Maven Android community long before Gradle was mentioned at IO13. If in doubt just listen to the recording of the Android fireside chat at IO12.

One of the things I like about Google's general approach is that it is very open, so that developers can pick up the tool that best fits their needs and apply it without much impediment. Not publishing the Android artifacts has introduced such an impediment and while the community has put up with the situation for the last 2 years, the cracks are starting to get wide enough to do some serious damage.

And I was a part of it. For years. It was awful, but it was the only way.

It was also completely third-party so not only is it not Google's problem but you also go into it knowing there are risks of problems.

There's an easy fix: stop using Maven.

[William Ferguson]

Jake you appear to be looking at the world through a very blinkered view.

All of what you said works perfectly in a homogenized world, but the world is not that way.

There are many forces dictating build environments, not all them within an individual developer's control. And even when the individual does have control there are economic considerations.

My point is that because their artifacts are published to a publicly accessible web address their artifacts are more accessible than those being provided by the Android team.

But not Maven central, therefore not a problem.

Gradle: If you are using a shared (eg corporate) repo then you are at risk.

Ant: If you are using dep management  then you are at risk.

 

A) Before you can consume any of the artefacts you need to start SDK manager, choose the appropriate versions of the required artifacts and then download them.

Every developer does this. You need the API level you are compiling against. This isn't a problem

API level is an orthogonal issue.  This is about whether the dependencies that you download are the dependencies that you expect.

You seem to be missing the point that these deps can be drawn in transitively without your knowledge, for Gradle and Ant as well as Maven.

I'm aware of your history with the android-maven-plugin. It is 3rd party and open source, no reason it can't also be adopted or at the very least not shut out. It was painful largely because we had to constantly fight with the Android build tools. Now that those tools are being published as Maven artifacts there is less fight and more co-operation.

A lot of us feel that Gradle is where Maven1 was at in 2003. A great concept but far from usable. I recently tried to drag a group of developers who were used to Ant and Maven across to Gradle and after 4 weeks trying to chow down we ended up converting the whole lot to Maven3.

 

Most developers don't use the SDK Manager to download the latest artifacts on a daily basis, they do it at irregular intervals. This means that there is a latency between when the artifacts become available and when the developer first has access to the official artifacts.

I agree. I wrote a Gradle plugin to address this: https://github.com/JakeWharton/sdk-manager-plugin. I'd love for the first-party plugin to do this some day.

 

Right so it's problem. And you have a Gradle work around.

I'm suggesting we fix the problem.

 

[Jake Wharton]

Your problem is one for the Maven community.

Get them to fix it by delegating to the SDK for the API jar and the two embedded m2 repos. ~50 line change in the plugin, problem completely solved.

--

You received this message because you are subscribed to the Google Groups "adt-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to adt-dev+u...@googlegroups.com.

[William Ferguson]

On Fri, Aug 8, 2014 at 10:43 AM, Jake Wharton <jakew...@gmail.com> wrote:

Your problem is one for the Maven community.

Get them to fix it by delegating to the SDK for the API jar and the two embedded m2 repos. ~50 line change in the plugin, problem completely solved.

This does NOT solve the issue. This does not even reference the issue. 

I can't believe believe that someone like yourself can pretend to be so wilfully ignorant of the impact of the spread of GAV equivalent artifacts that have a higher visibility that those produced by the Android team.

William

[Jake Wharton]

You were complaining about non-standard artifacts. It solves it by always using official artifacts.

I don't care that there's slightly higher overhead for Maven users (which the continued fact that it's wholly unsupported falls on your deaf ears). That's the price you pay for doing the right thing.

--

[William Ferguson]

Jake, what you are not hearing is that the non-official artefacts are able to enter a dependency stream through other means not visible to the developer. This is true for Ant, Gradle and Maven users.

Do you have the official Android support artefacts in your corporate repository?

Are you sure?

Did you personally check? For each and every version version of each artefact?

--
You received this message because you are subscribed to a topic in the Google Groups "adt-dev" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/adt-dev/2hPuSUYttbg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to adt-dev+u...@googlegroups.com.

[Jake Wharton]

On Thu, Aug 7, 2014 at 8:46 PM, William Ferguson <william....@xandar.com.au> wrote:

Do you have the official Android support artefacts in your corporate repository?

No.

 

Are you sure?

Yes.

 

Did you personally check? For each and every version version of each artefact?

I don't need to check.

1. They're in the SDK. The SDK's m2 repos are enabled by default in builds and can only be populated from the SDK manager. Safe.
2. Gradle doesn't use the local .m2 repo. Even if someone is dealing with a Maven project on the same machine, they don't share a local artifact cache. Safe.
3. They'll never be there since no artifacts exist at the same coordinates in central to populate them by on a cache miss. Google controls the groupId in which they would be deployed if they ever do show up. Safe.

I fought the (losing) Maven battle for four years. I contributed to the maven-android-plugin. I contributed to the maven-android-sdk-deployer. I built the r7 support-v4 library artifact that is sitting in Maven central right now. I've published tens of libraries in apklib and jar format using Maven (and continue to do so).

The right way forward is delegating to the SDK. Manfred even once said that this was what the plugin would do in the future. I'm surprised to learn it doesn't yet.

I would welcome the artifacts in central. Until that time, the SDK is the canonical source of truth and should be respected as such. It is trivial to change the maven-android-plugin to support it as such and it would alleviate this concern completely and in a manner that is actually endorsed by Google.

[Vyacheslav Blinov]

> Gradle doesn't use the local .m2 repo. Even if someone is dealing with a Maven project on the same machine, they don't share a local artifact cache. Safe.

Unfortunately not so safe if you use 3rd party libs with transitive deps, like, possibly HE, maybe there are others, doesens of them.

[Jake Wharton]

On Thu, Aug 7, 2014 at 10:10 PM, Vyacheslav Blinov <blinov.v...@gmail.com> wrote:

Unfortunately not so safe if you use 3rd party libs with transitive deps, like, possibly HE, maybe there are others, doesens of them.

Except it has nowhere to resolve these from because they do not exist in Maven central. Transitive artifact resolution is no different than an explicitly declared dependency. 

[Bryan Herbst]

I'm not sure why you keep bringing up corporate repositories.

First, if your company's development standards don't discourage the practice of creating artifacts with hijacked GAVs, you should probably update your standards. Naivete is no excuse either- your developers should be aware of what they are publishing to your company's repository and shouldn't publish non-official versions of official artifacts. If they do this, they should lose their right to upload artifacts.

Second, I really don't see how this is the responsibility of the Android team. I could just as easily create my own version of JUnit and upload that to a corporate or personal repository using the official JUnit's GAV. I could even do it transitively like you describe for HE.

It might be a valid issue to bring up, but this is certainly not the place to do it. I do thank you for raising the issue on HE's GitHub project though, because I agree it is poor practice.

[Manfred Moser]

Wow... you guys got a bit of a debate going here ;-) 

I just wanted to clarify and mention a few things:

1. I am happy that the SDK team pushes sdklib and others to the Central Repo.

2. I am happy that they include the proprietary and other components in the SDK in the form of a Maven repo. 

The Android Maven Plugin, my Maven Android SDK Deployer just like many of Jakes and other projects are community efforts. Google and the Android team should imho support them where they can with the limited resources they have. They always have been and still are a imho small team of excellent people with very limited resources. 

Now with regards to the coordinate usage and the such. 

1. Yes.. it would be better if google would publish all components in a repository somewhere. Central would be good, but their own Maven repo or so would be just as good. Publishing them via the SDK is a compromise .. ideally it would at least include all the components including the android jar itself. This would give them a leadership position on what the coordinates should be and reduce confusion for users by simply setting a standard.

2. The Maven Android SDK deployer can publish the repo other artifacts into a repository manager and therefore makes it all available. I had numerous requests from various cloud CI server platform providers and other development infrastructure providers to use it to publish the artifacts and I had to advise against them due to the legal concerns with the EULA. Even Google itself does not publish in a repo without the SDK EUAL click through. There definitely is a need in the community for a repo .. Google just chooses to prioritize it low/ignore it (for now?) forcing workarounds. Same applies to e.g. the Eclipse support at the moment. 

3. The Android Maven plugin could provision the plugins into a repository/repo manager but the Maven Android SDK Deployer already does that. It has to be provisioned into a repo since other plugins also have to be able to access these dependencies. If it is a really easy fix I still always welcome your pull request Jake or anyone else's as well.

Overall I think that there is no really big problem here. 

Yes.. it would be better if Google would take more of a leadership role and participate more in the community even by simple things like setting standards and making recommendations.

And yes... I am still hoping for more good things to come from Google and the SDK team and I am sure there will be.. 

And yes... the SDK team is small and they have a lot on their plate and the community steps up and does their thing.

And no... telling people not to use their tool of choice and is not good taste and not necessary. There are many happy Android Maven Plugin users or Eclipse users and they contribute to the Android eco system just like the in-crowd of Gradle and IntelliJ users do. I have been using Intellij since version 4.x and happily switch to Eclipse and back all the time. Both are great tools with their strengths and weaknesses just like Maven and Gradle. And there are lots of others around that all deserve to be helped and embraced. Choice is good.

Lets all work together even more and we can make things even better.. 

Manfred

PS: Every company uses lots of competing tools. In fact I am sure Google and Square both use Gradle and Maven heavily.. so what? Both are better than make ... hahaha