RE: Erlang Packaging
Tim Watson
1. Support for snapshot builds?
2. How does code signing work?
3. How do people publish their public key(s)?
4. How do we package the tool(s)?
1 - I guess what I'm thinking about is whether or not we allow people to publish and overwrite the most current (bleeding edge) version of their code. I don't know if this is a 'good' thing or not, but we should think about where we stand on it.
2 - I'm assuming that the publisher generates a hash (not the word I'm looking for, but I've not slept!) using their private key and that goes into the metadata. Then they presumably have to have published their public key, which we (in turn) use to verify the binary against the expected output specified in the metadata?
3 - Do organisations' public keys go into a (remote) index, or do they publish it on the web (i.e., a URL) and point to it from the metadata?
4 - Thoughts?
Cheers,
Tim
Eric B Merritt
Tim Watson wrote:
>
> Some outstanding questions I've got in my mind...
>
> 1. Support for snapshot builds?
I dont like snapshot builds where the the version is just 'SNAPSHOT'
it breaks the addativeness of the repo. That said I have no problem
with <vsn>-SNAPSHAT or anything like that as long as the version
number increments.
> 2. How does code signing work?
We could probably just sign the hash of the binary. I am not too
familiar with code signing. lets look at what others do out there and
probably just emulate it.
> 3. How do people publish their public key(s)?
how about as part of the metadata. Perhaps we have a publishers
directory that contains the public key for folks.
> 4. How do we package the tool(s)?
I suspect in the system itself and any other way we can find. I would
probably create debs for them as well. We should probably also
download tarballs.
> 1 - I guess what I'm thinking about is whether or not we allow
> people to publish and overwrite the most current (bleeding edge)
> version of their code. I don't know if this is a 'good' thing or
> not, but we should think about where we stand on it.
I dont want to break the addative nature of the system. That is
information added never deleted. That said we may have too. I would
still rather encourage them to do something like increment a version
with a -snapshot or -alpha rather then publishing the latest and
greatest. That said we might have to give them a tool to help them
with that.
>
> 2 - I'm assuming that the publisher generates a hash (not the word
> I'm looking for, but I've not slept!) using their private key and
> that goes into the metadata. Then they presumably have to have
> published their public key, which we (in turn) use to verify the
> binary against the expected output specified in the metadata?
Exactly. Though we should think about how to make that secure. Given
the fact that we have their public key it should be pretty easy to
do. We just make sure every action is signed, including adding a new
public key.
> 3 - Do organisations' public keys go into a (remote) index, or do
> they publish it on the web (i.e., a URL) and point to it from the
> metadata?
I think they go into the metadata directly.
>
> 4 - Thoughts?
inline -> :)
> Cheers,
>
> Tim
>
> --
> You received this message because you are subscribed to the Google Groups "erlware-questions" group.
> To post to this group, send email to erlware-...@googlegroups.com.
> To unsubscribe from this group, send email to erlware-questi...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/erlware-questions?hl=en.
>
Tim Watson
On 27 Feb 2012, at 19:45, Eric B Merritt wrote:
> At Mon, 27 Feb 2012 19:11:18 +0000,
> Tim Watson wrote:
>>
>> Some outstanding questions I've got in my mind...
>>
>> 1. Support for snapshot builds?
>
> I dont like snapshot builds where the the version is just 'SNAPSHOT'
> it breaks the addativeness of the repo. That said I have no problem
> with <vsn>-SNAPSHAT or anything like that as long as the version
> number increments.
>
Yeah, it was the latter that I had in mind. What I was wondering is, from the publishing point of view, if it makes sense to overwrite 1.0.2-SNAPSHOT instead of forcing the user to increment the final part of the version number with every single build. By way of example, what we do at work is basically your standard CI process in which every commit triggers a build and every successful build generates an artefact and deploys it to the repository. The problem with enforcing every artefact upload to have a new version number is that it makes it rather unmanageable for the client to bind to the right version (as it is constantly incrementing every hour or so).
Now perhaps this is a hangover from using maven, where each dependency requires an explicit version number. Is the solution to this to say 'libfoobar >= 1.0.2' and have the snapshot builds do something like 1.0.2-1, 1.0.2-2, 1.0.2-3, 1.0.2-4, etc?
>> 2. How does code signing work?
>
> We could probably just sign the hash of the binary. I am not too
> familiar with code signing. lets look at what others do out there and
> probably just emulate it.
I think the standard seems to be to produce, as you said previously, a one way hash of the binary. This is encrypted with the private key (by the publisher) and to verify the signatory at the other end, we reproduce the hash and use the public key to verify the artefact.
>
>> 3. How do people publish their public key(s)?
>
> how about as part of the metadata. Perhaps we have a publishers
> directory that contains the public key for folks.
>
I think it should be a URL. If people want to sign their code, they'll have to go to the effort of generating the key pair anyway. If we do provide 'public key hosting' as it were, I think it should either go into the directory that maps to the organisation or into a public keys directory. The latter sounds like a hassle as we'd have to map organisations to keys, so putting the key into the root directory for that organisation's metadata sounds preferable. Third party signing will have to be handled differently somehow I guess.
>> 4. How do we package the tool(s)?
>
> I suspect in the system itself and any other way we can find. I would
> probably create debs for them as well. We should probably also
> download tarballs.
>
Yes that sounds about right. We also need to think about how the executables are provided. Personally I favour un-compiled escripts as the entry point, as these make no assumptions about what Erlang/OTP environment is present on the user's machine. We've also got to bear in mind that we want to be able to use the various modules from other code (effectively using them as library code) as well. Presumably the easiest way to do this is to keep all the functionality in the library code and have the escript(s) just call into it.
What are your thoughts about this?
>> 1 - I guess what I'm thinking about is whether or not we allow
>> people to publish and overwrite the most current (bleeding edge)
>> version of their code. I don't know if this is a 'good' thing or
>> not, but we should think about where we stand on it.
>
> I dont want to break the addative nature of the system. That is
> information added never deleted. That said we may have too. I would
> still rather encourage them to do something like increment a version
> with a -snapshot or -alpha rather then publishing the latest and
> greatest. That said we might have to give them a tool to help them
> with that.
>
Sounds reasonable.
>>
>> 2 - I'm assuming that the publisher generates a hash (not the word
>> I'm looking for, but I've not slept!) using their private key and
>> that goes into the metadata. Then they presumably have to have
>> published their public key, which we (in turn) use to verify the
>> binary against the expected output specified in the metadata?
>
> Exactly. Though we should think about how to make that secure. Given
> the fact that we have their public key it should be pretty easy to
> do. We just make sure every action is signed, including adding a new
> public key.
>
+1.
>> 3 - Do organisations' public keys go into a (remote) index, or do
>> they publish it on the web (i.e., a URL) and point to it from the
>> metadata?
>
> I think they go into the metadata directly.
>
+1.
Eric B Merritt
Tim Watson wrote:
>
>
> On 27 Feb 2012, at 19:45, Eric B Merritt wrote:
>
> > At Mon, 27 Feb 2012 19:11:18 +0000,
> > Tim Watson wrote:
> >>
> >> Some outstanding questions I've got in my mind...
> >>
> >> 1. Support for snapshot builds?
> >
> > I dont like snapshot builds where the the version is just 'SNAPSHOT'
> > it breaks the addativeness of the repo. That said I have no problem
> > with <vsn>-SNAPSHAT or anything like that as long as the version
> > number increments.
> >
>
> Yeah, it was the latter that I had in mind. What I was wondering is,
> from the publishing point of view, if it makes sense to overwrite
> 1.0.2-SNAPSHOT instead of forcing the user to increment the final
> part of the version number with every single build.
I am not a fan of the same version number pointing to two different
things over time. That just seems like a bad bad thing. That being the
case I tend towards correctness more then usability sometimes.
> By way of example, what we do at work is basically your standard CI
> process in which every commit triggers a build and every successful
> build generates an artefact and deploys it to the repository. The
> problem with enforcing every artefact upload to have a new version
> number is that it makes it rather unmanageable for the client to
> bind to the right version (as it is constantly incrementing every
> hour or so).
I know. I would still rather exploring making the above trivial and
transparent versus going with SNAPSHOT builds. Incrementing versions
automatically should be pretty easy. This is, however, a topic I dont
claim to have a definitive answer on.
>
> Now perhaps this is a hangover from using maven, where each
> dependency requires an explicit version number. Is the solution to
> this to say 'libfoobar >= 1.0.2' and have the snapshot builds do
> something like 1.0.2-1, 1.0.2-2, 1.0.2-3, 1.0.2-4, etc?
This is the solution I would rather have actually. If we can work it
in such a way as to make it easy.
> >> 2. How does code signing work?
> >
> > We could probably just sign the hash of the binary. I am not too
> > familiar with code signing. lets look at what others do out there and
> > probably just emulate it.
>
> I think the standard seems to be to produce, as you said previously,
> a one way hash of the binary. This is encrypted with the private key
> (by the publisher) and to verify the signatory at the other end, we
> reproduce the hash and use the public key to verify the artefact.
That seems like the right way to go to me.
> >
> >> 3. How do people publish their public key(s)?
> >
> > how about as part of the metadata. Perhaps we have a publishers
> > directory that contains the public key for folks.
> >
>
> I think it should be a URL. If people want to sign their code,
> they'll have to go to the effort of generating the key pair
> anyway. If we do provide 'public key hosting' as it were, I think it
> should either go into the directory that maps to the organisation or
> into a public keys directory. The latter sounds like a hassle as
> we'd have to map organisations to keys, so putting the key into the
> root directory for that organisation's metadata sounds
> preferable.
Thats what I am recommending (storing it in the metadata). I think it
should be part of the metadata. Forever, there so to speak.
As to where I dont really have a preference at the moment. I think the
key will be by the organizations name (org names must be unique and
durable in this model) so naming in a public key directory should not
be a problem. Then again having an organization metadata directory
(for info about an org) doesnt sound like a bad idea either.
> Third party signing will have to be handled differently
> somehow I guess.
Hmm, yes. but I think it can follow the same model. The user will have
to pick a name for himself there is no reason that cant be in the same
namespace as an organization.
>
> >> 4. How do we package the tool(s)?
> >
> > I suspect in the system itself and any other way we can find. I would
> > probably create debs for them as well. We should probably also
> > download tarballs.
> >
>
> Yes that sounds about right. We also need to think about how the
> executables are provided. Personally I favour un-compiled escripts
> as the entry point, as these make no assumptions about what
> Erlang/OTP environment is present on the user's machine. We've also
> got to bear in mind that we want to be able to use the various
> modules from other code (effectively using them as library code) as
> well. Presumably the easiest way to do this is to keep all the
> functionality in the library code and have the escript(s) just call
> into it.
>
> What are your thoughts about this?
I am not a big fan of strait scripts. It makes it hard to do a drop in
executable (if there are any dependencies at all) and it also makes it
a bit harder to provide a library. Unless you ship a OTP app with the
script and then you suddenly that makes it more difficult. Better to
ship a binary (compiled escript) then a human readable script.
We can get around the OTP problems by just building on an older
version of erlang. Make sure its one or two versions behind and we get
the best of both worlds.
Tim Watson
I am not a big fan of strait scripts. It makes it hard to do a drop in
executable (if there are any dependencies at all) and it also makes it
a bit harder to provide a library. Unless you ship a OTP app with the
script and then you suddenly that makes it more difficult. Better to
ship a binary (compiled escript) then a human readable script.
We can get around the OTP problems by just building on an older
version of erlang. Make sure its one or two versions behind and we get
the best of both worlds.
Eric B Merritt
Tim Watson wrote:
>
> [1 <text/plain; ISO-8859-1 (quoted-printable)>]
>
> [2 <text/html; ISO-8859-1 (quoted-printable)>]
> On 28 Feb 2012, at 16:21, Eric B Merritt wrote:
>
> I am not a big fan of strait scripts. It makes it hard to do a drop in
> executable (if there are any dependencies at all) and it also makes it
> a bit harder to provide a library. Unless you ship a OTP app with the
> script and then you suddenly that makes it more difficult. Better to
> ship a binary (compiled escript) then a human readable script.
>
> I'm not sure why you'd want to do the former (drop in executable),
> and the latter seems to me to be a case of bundling the application
> and dependent libraries together, putting the escript in a 'priv'
> folder and sticking that on the path (or symlinking on unix). Still
> I'm open to other approaches.
because it makes bootstrapping so much easier. You have to have erlang
installed but then to get started you just download a binary. It makes
a huge difference, not having this approach is one of the many reasons
that faxien/sinan did not take off. Making it easy to get started is
hugely important for these things.
I have come to the conclusion that projects need to be somewhat like
crack. There has to be as little friction as possible. Also remember
that in that first use there is no such thing as bundling unless you
are talking about a tarball.
>
> We can get around the OTP problems by just building on an older
> version of erlang. Make sure its one or two versions behind and we get
> the best of both worlds.
>
> Ok, I guess we can provide downloads for different versions (or
> version ranges more likely) of OTP once it moves along a version or
> two.
I dont think we actually need to. If we just build on R14 through say
R16, basically stay one or two versions behind I suspect we would be
golden.
Tim Watson
> At Tue, 28 Feb 2012 16:33:37 +0000,
> Tim Watson wrote:
>>
>> [1 <text/plain; ISO-8859-1 (quoted-printable)>]
>>
>> [2 <text/html; ISO-8859-1 (quoted-printable)>]
>> On 28 Feb 2012, at 16:21, Eric B Merritt wrote:
>>
>> I am not a big fan of strait scripts. It makes it hard to do a drop in
>> executable (if there are any dependencies at all) and it also makes it
>> a bit harder to provide a library. Unless you ship a OTP app with the
>> script and then you suddenly that makes it more difficult. Better to
>> ship a binary (compiled escript) then a human readable script.
>>
>> I'm not sure why you'd want to do the former (drop in executable),
>> and the latter seems to me to be a case of bundling the application
>> and dependent libraries together, putting the escript in a 'priv'
>> folder and sticking that on the path (or symlinking on unix). Still
>> I'm open to other approaches.
>
> because it makes bootstrapping so much easier. You have to have erlang
> installed but then to get started you just download a binary. It makes
> a huge difference, not having this approach is one of the many reasons
> that faxien/sinan did not take off. Making it easy to get started is
> hugely important for these things.
>
Ok I can buy this.
> I have come to the conclusion that projects need to be somewhat like
> crack. There has to be as little friction as possible. Also remember
> that in that first use there is no such thing as bundling unless you
> are talking about a tarball.
>
Yes I agree it needs to be addictive. :)
>>
>> We can get around the OTP problems by just building on an older
>> version of erlang. Make sure its one or two versions behind and we get
>> the best of both worlds.
>>
>> Ok, I guess we can provide downloads for different versions (or
>> version ranges more likely) of OTP once it moves along a version or
>> two.
>
> I dont think we actually need to. If we just build on R14 through say
> R16, basically stay one or two versions behind I suspect we would be
> golden.
I still have applications in life that use R13, so I'd say that's likely to be the case for other users too. Folks who're already in production will take their time testing and appraising a new release before upgrading, and that can take time. I'd suggest we build an R13 binary for these 'legacy' users and then have another 'main' binary with which we do as you say and follow 2 releases behind the latest and greatest.
Eric Merritt
sure I have no problem with that. We can build for R13 as easily as
R14. We can keep building for that as long as erlang retains backwards
compatibility.