sbt-pgp should be part of sbt 0.13

[eugene yokota]

Hi guys,

I'm seeing some AttributeKey ID collisions for pgp-signer etc since sbt-pgp 0.8 came out.

What do you guys think about incorporating it as part of sbt 0.13?

sbt-pgp is now an essential part of ecosystem, since many OSS projects who

don't have their own repo publish to Sonatype as signed jar, which is overall a good thing.

sbt-pgp 0.8 came out a few months ago, adding publish-signed instead of just signing everything,

which also was smooth, in the parlance of our times.

But now we have half the OSS projects depending on sbt-pgp, and some plugin or build may even make

explicit reference to a particular version of it to customize the release behavior,

resulting to diamond inheritance problem.

[error] AttributeKey ID collisions detected for: 'pgp-signer' (sbt.Task[com.typesafe.sbt.pgp.PgpSigner], sbt.Task[com.jsuereth.pgp.sbtplugin.PgpSigner]), 'pgp-verifier' (sbt.Task[com.typesafe.sbt.pgp.PgpVerifier], sbt.Task[com.jsuereth.pgp.sbtplugin.PgpVerifier]), 'check-pgp-signatures' (sbt.Task[com.typesafe.sbt.pgp.SignatureCheckReport], sbt.Task[com.jsuereth.pgp.sbtplugin.SignatureCheckReport]), 'signatures-module' (sbt.Task[com.typesafe.sbt.pgp.GetSignaturesModule], sbt.Task[com.jsuereth.pgp.sbtplugin.GetSignaturesModule])

[error] Use 'last' for the full log.

It's not like a project or a plugin needs more feature than the other, they just need to agree on a version.

One solution is to say "only use sbt-pgp from global" but if someone needs to tweak releasing, that's probably not reasonable.

Another solution is to say "then at least use the latest version." That also won't work since binary versioning of sbt,

a build or a plugin can now live a long life, and some build may be using older plugins that references sbt-pgp.

The real solution, I think is that sbt-pgp becomes part of the sbt 0.13. This way, I can do my things in global,

and other projects can do their things in build.sbt, and it'll always be the same version since globals will be version-segregated.

-eugene

[Josh Suereth]

SO, i'd like to say two things:

(1) I personally have little issue adding PGP ot sbt 0.13, besides the fact that It may mean less frequent updates, and also removes a current pain point that is exposing a larger issues.

(2) I think the issue we're running into is fundamental in the plugin model.  That is how do plugins make use of other plugin successfully in the face of User(Global) vs. Project(project/plugins.sbt) configuration.  Not only that, how do you safely use another person's plugin to expose your own.  Right now it is possible (I use it very well in the sbt-ghpages plugin to pull in sbt-git and sbt-site).   However, the mechanisms is dangerous in the face of global plugins *AND* in the face of binary compatibility inside of plugins.

SO -> While I think the current suggestion is the most practical for the time being, can you think of any alternatives to solve the indemic issue completely?

I can think of one:  Removing global plugins and forcing projects to specify all plugins themselves.   Not one I'd be amazingly happy with, but it would remove a whole class of issues from SBT so we could focus on improving our plugin ecosystem with renewed vigor/less confusing situations.

-eugene

--
You received this message because you are subscribed to the Google Groups "simple-build-tool" group.
To unsubscribe from this group and stop receiving emails from it, send an email to simple-build-t...@googlegroups.com.
To post to this group, send email to simple-b...@googlegroups.com.
Visit this group at http://groups.google.com/group/simple-build-tool?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Paul Phillips]

On Mon, May 20, 2013 at 5:24 AM, Josh Suereth <joshua....@gmail.com> wrote:

(2) I think the issue we're running into is fundamental in the plugin model.  That is how do plugins make use of other plugin successfully in the face of User(Global) vs. Project(project/plugins.sbt) configuration.  Not only that, how do you safely use another person's plugin to expose your own.  Right now it is possible (I use it very well in the sbt-ghpages plugin to pull in sbt-git and sbt-site).   However, the mechanisms is dangerous in the face of global plugins *AND* in the face of binary compatibility inside of plugins.

It's almost as if some sort of manager is needed... a manager for dependencies... 

"This is one time where television really fails to capture the true excitement of a large squirrel predicting the weather."

[eugene yokota]

On Sunday, May 19, 2013 1:30:58 PM UTC-4, eugene yokota wrote:

I'm seeing some AttributeKey ID collisions for pgp-signer etc since sbt-pgp 0.8 came out.

I'm actually seeing the AttributeKey ID collision on every build that

was opened using sbt 0.12.3 prior to sbt-pgp 0.7 -> 0.8 upgrade, so this could be just a plain sbt bug.

The impression of 2.7.3 fixing [#272][272] by loading global plugins in its own class loader was strong in my mind,

and because commenting out some plugins fixed the issue, I thought builds and plugins were causing the collision.

In reality, commenting out plugins probably just forced rebuild of plugins build.

The potential for version conflict of commonly globally used plugins still stands,

but it's not as prevalent as it seemed.

Josh wrote:

>> SO -> While I think the current suggestion is the most practical for the time being, can you think of any alternatives to solve the indemic issue completely?

>> I can think of one:  Removing global plugins and forcing projects to specify all plugins themselves. 

How about adding some priority between the global and project settings? If a project senses inconsistent settings, remove only the offending keys.

-eugene

  [272]: https://github.com/sbt/sbt/issues/272 

[Doug Tangren]

Since we're getting closed to that season again ( the next stable minor release! ), I thought it may be a good time to bring up the discussion ( maybe a new thread would make sense ) about the Best Practices ™ for plugin authors to name and scope keys. I times of yore we all scoped keys to configs. We then learned what configs were for :) The times of yesteryear we ( well some people ) started scoping keys to tasks. Mark even went the extra mile and allowed for tab completion of these to make it easier for users to discover features. Both of these were practices were there to help ensure a decreased chance of key collision. Some plugin authors just make a best effort at making really long key names with starting with something like the name of the plugin. 

Now that 0.13.0 is at our door step, what is the direction we, as a plugin community, should take with regards to providing some consistency for users as well as not stomping on the toes of each others keys? I think at this point I think most plugin authors have been in the community long  enough and have seen enough posts to this list to have some idea of what causes pain and what causes joy ( yes I have seen some joyous posts pass through these parts ). 

0.13.0's new killer feature seems to be user friendliness. Plugin authors are really just key smiths™ that plug and unlock features in sbt's platform. What can we be doing do keep the 0.13.0's user friendliness untarnished?

 

-eugene

  [272]: https://github.com/sbt/sbt/issues/272 

[Josh Suereth]

If you follow almost ANY project with a *plugin* system (not just dependencies but plugins), plugin versioning, compatibility and the whole gambit is a serious problem.   The project I know that solves it in the most robust manner (eclipse) suffers from terrible user experience.    I'm curious if you have any viable suggestions here, because I have not seen goodness in the surrounding community.

That, and we already know that Maven/Ivy repositories are pretty anemic when it comes to our binary compatibility constraints.  We're trying to mimic the Linux distributions where you just say "Everything must work together" and prevent too-new packages from entering the world.  Not really an option.

ALSO - there is an sbt specific issue of how to order the "pluggable" portions of the system.  Who gets their settings in first?   If I want to adapt the PGP plugins settings, I need to make sure he's installed first.

It's definitely not as simple as throwing up your hands and saying "dependency management".  I think plugin systems are, in general, the winning and killing aspect of a lot of projects.

[Josh Suereth]

Very good point.  We should probably evolve some best practices here, but I think it's up to the plugin community to define some of this.  E.g. Mark doesn't hit this issue, because core sbt keys are always in every build.  It's inter-plugin dependencies where the true pain starts to show up.   I haven't thought of a good solution to this problem, but we have discussed making a "plugin key" library at one point.

That doesn't solve the second issue of *ordering* of settings from plugins (which we currently make users do).   Personally, I think the plugin aspect of SBT may require its own release (and a lot of discusison/design from us) to improve in the future.   I think sbt 0.13 brings things to a nicer place, but plugins still pose some very tough design decisions.

[Paul Phillips]

On Mon, May 20, 2013 at 8:26 AM, Josh Suereth <joshua....@gmail.com> wrote:

If you follow almost ANY project with a *plugin* system (not just dependencies but plugins), plugin versioning, compatibility and the whole gambit is a serious problem.   The project I know that solves it in the most robust manner (eclipse) suffers from terrible user experience.    I'm curious if you have any viable suggestions here, because I have not seen goodness in the surrounding community.

At the tree branch level I don't have any viable suggestions. My observation, and I didn't mean it as criticism so much as a general acknowledgement, is that it often seems as if we do little but reimplement existing mechanisms in slightly variant contexts. And sbt does seem, to me at least, to be especially prone to this.

[Josh Suereth]

This may be true.   However, I firmly disagree on the plugin aspect specifically.   I've been doing a good bit of research, especially given the conflation of Play plugins into the mix of sbt plugins.   Basically, I have not seen plugins done 'well' in any scenario, however there is a level of 'adequate' that I think we can attain.

As much as we can solve plugin issues on the plugin author side of things, so users only need to declare them, would be ideal.   However, this means we need SBT to provide the right hooks that plugin authors can *anticipate* problems that their own users may encounter when mixing plugins.   All-in-all, it's a teneous fiber that connects us all, and I think there's still room in the world for a better system.  I just don't know what that would be.

When it comes to other areas where sbt deviates and would not need to, perhaps we're in agreement that less of that is better.   The key line being "When does an existing solution show enough pain points that you should try to improve the status quo" vs. "a reimplementation won't improve the status quo".

SO -> onto the PGP and SBT discussions.   Should PGP-related bits be part of sbt core?

Ideal-land says no.  SBT's plugin system shoudl allow all manner of new interactions and the core should be as small as possible.

Pragmatic land says it's the path of least resistance.

What does middle ground say?

[Doug Tangren]

On Mon, May 20, 2013 at 11:30 AM, Josh Suereth <joshua....@gmail.com> wrote:

Very good point.  We should probably evolve some best practices here, but I think it's up to the plugin community to define some of this.  E.g. Mark doesn't hit this issue, because core sbt keys are always in every build.  It's inter-plugin dependencies where the true pain starts to show up.   I haven't thought of a good solution to this problem, but we have discussed making a "plugin key" library at one point.

I'd be a little worried about a key library and friction  that would arise for the maintenance of granting access and administration. It's really cool that someone can just roll up their sleeves, writing a plugin to solve a problem, push it out there, and having others use it with just the *little* amount of friction it is to post to sonatype. Adding more friction can cause a rash. I guess the flip side of that is that user's need to be strapped with more knowledge of what can and may happen when they install a plugin.

[Josh Suereth]

Yeah.   that's why I personally feel the "uber-key-plugin" may be a red herring here.  The focus should be on what do users want from plugins.  (add, minimally configure, WORK), and what plugin authors need (this has some ??? here) and then use that to decide what SBT provides.

Like I said, I think the PGP issue is the tip of the iceberg we've all planted our flags on and called "iceland".   I'm pretty sure the iceberg is melting and our users are sinking their ships against it.

[eugene yokota]

Is plugin system such pain point?

For the most part, I think it's doing a good job of providing popular features like pgp signing

and IDE integration without blocking on sbt dev cycle.

And allowing advanced users to customize or experiment with build environment.

In addition to "add, minimally configure, WORK," the ability to fix problems when it happens could make or break the user experience

(e.g. dynamic vs static typing). Often it's to do with the gap in the mental model of what the user thinks build is doing vs what sbt implements. Here's an example.

A user recently open an issue on sbt-assembly saying he can't disable the test during assembly.

He created a repro project, which had:

  ....

  

test in assembly := {}

assemblySettings

I thought this was interesting because to as a plugin author, it's self-evident that assemblySetting needs to come before rewiring "test in assembly" because I read between the lines, and imagine some m**adic context passed around. To the users, it's innocent mutable variable assignment and a procedure call.

-eugene

[nafg]

I haven't followed this discussion too closely, just wondering if it should inform the design of adept at all.

[Josh Suereth]

Yeah, I think not.  The key issue is specifically in how sbt plugins hook into (a) sbt projects (b) other sbt plugins.   It's less about how to specify them as dependencies and more about how to *use* them as dependencies.

On Mon, May 20, 2013 at 10:56 PM, nafg <nafto...@gmail.com> wrote:

I haven't followed this discussion too closely, just wondering if it should inform the design of adept at all.

--