> I have taking my 'story so far post' and put it in the github wiki. > Its just a start but it should give us a base move forward.
> The goal here is to get something we can present to erlang-questions > for feedback, and alternatively use as a very high level spec for > requirements.
On the wiki. Lets send questions to this list and just update the wiki directly once we come to some conclusion or just update the wiki if you are comfortable with that.
> Question(s):
> do we want to tie ourselves to git+github in the spec? > My thoughts are that people may use any mechanism they like > to build alternative repository implementations (which we should > support via a plugins mechanism) - not everyone uses git
> In light of this, can we rephrase item 2, as > "Repositories will initially be stored in git and targeted at github"?
I think in version on we do target git and github. Down the road we will have other tasks to branch that out. However, I think having a hard target initially is a good thing.
This is not a hard and fast requirement for me though.
> Question:
> is this really optional? I certain want the tools that *we* > build to do this, as I don't want to download <package-x>-1.0.1 every > time I use it.
In my mind it is. I have a lot of bandwidth and I am nearly always connected. I would rather just download the things I need when I need them rather then have some 3rd location that I have to know about and maintain. For me its a trade off of bandwidth for increased simplicity.
I probably wont have a global anything, no global index no local repo etc. It will all be stored on a project by project basis. One of the reasons that I want to have command line options for everything so so that I can do that. That removes things hanging out it some global directory isolates problems to a single project and makes it very easy to clean up. With that model in mind a local repo is very redundant and actually mostly just adds unneeded overhead. Disk space is cheap, band with is cheap I want to leverage both.
> I'm not convinced this is really an optimization rather than the only > sane way to do things. Think about a build plugin for example, or a > logging library that you use in all your applications/systems. Do you > really want to resolve this to some local app directory each time you > start a new project?
Yes actually.
> This is what rebar does with dependencies that I > actually hate - the local repository (along with a good index and binary > artefacts) is part of the main reason I want to do this stuff.
Its one of the few things that I think rebar got right. I like things to be self contained. That maybe why I like the rox and osx application model so much. Its also the reason I distribute my otp releases with everything the need. Its a space trade off that I think is well worth it.
We might be able to do this transparently in something like ~/.repo but I absolutely want the option to turn it off.
> Question:
> I agree that we need the namespace flattening, but I'm > wondering why we need do implement it in this way. On windows, for > example, this is a royal pain as there are no symlinks.
> If we make the local repository a core part of the solution, then > there is another option which IMO is viable.
> All the use-cases for the dependencies involve getting a running > emulator to 'know about' them, for example compile time ones (such > as parse transforms and build plugins) need to be on the code path > during the 'build tool' operations. Runtime ones need to be available > during testing, local development and pacakging a release. Test scoped > ones need to be on the path during testing. In all cases, the following > things need to work
> So why not just add the correct directories to the code path at the > appropriate time? This saves copying (or symlinking) and avoids > cluttering the local directory structure.
lol, I think we have very different outlooks here. We are going to need to resolve that.
> It also avoid having to > manage an 'app dir' when dependencies are removed for example.
nuking the directory?
> Personally I find this approach preferable, as long as the code > path modification process is documented clearly.
This is doable. There are two things I dont like about it. The first and biggest is that its much more difficult to plug in arbitrary tools that are not part of the chain. That is things that use the packages now need to know about this non-erlang special repo structure that we have. This may actualy not be a problem, but we need to make sure that none of the packaging semantics make it into running code. The process being as vanilla as possible I think is important.
On dependencies.
I am not sure what you intend for 'build-support'. I guess that is the tools to do the compile themselves. I am not sure how that can be worked into the process.
Also on packaging. At the moment we are not allowing user code to run as packaging. that is the packaging tools are complete unto themselves. Well better said that, if you are doing packaging the packaging tool is already there. I may be missing something here.
On Wed, Feb 29, 2012 at 2:54 PM, Tim Watson <watson.timo...@gmail.com> wrote: > Thanks for this Eric. I will start to use this for collaborative working now.
> On 29 Feb 2012, at 19:16, Eric Merritt wrote:
>> I have taking my 'story so far post' and put it in the github wiki. >> Its just a start but it should give us a base move forward.
>> The goal here is to get something we can present to erlang-questions >> for feedback, and alternatively use as a very high level spec for >> requirements.
On 1 March 2012 16:14, Eric Merritt <ericbmerr...@gmail.com> wrote:
> Tim,
> On the wiki. Lets send questions to this list and just update the wiki > directly once we come to some conclusion or just update the wiki if > you are comfortable with that.
Sure I wasn't sure which model to go for, but from now on I'll only post updates to the wiki and keep discusions to the list. That's actually cleaner anyway, from a random visitor's point of view not to see lots of questions inline.
>> do we want to tie ourselves to git+github in the spec? >> My thoughts are that people may use any mechanism they like >> to build alternative repository implementations (which we should >> support via a plugins mechanism) - not everyone uses git
>> In light of this, can we rephrase item 2, as >> "Repositories will initially be stored in git and targeted at github"?
> I think in version on we do target git and github. Down the road we > will have other tasks to branch that out. However, I think having a > hard target initially is a good thing.
> This is not a hard and fast requirement for me though.
I am fine with it being github initially, but for work there are some cases where I can't publish internal/private metadata to github because it is offsite. We are a former state run monopoly and the largest telecoms outfit in the UK, so we not only have a lot of rules around privacy but also lots of government as well as industry regulation that we have to comply with. In some cases, even the notion that an artefact exists must not reside outside of the corporate LAN, for legal reasons.
I will write an alternative remote index handler that doesn't use github (based on subversion or nexus probably) but I will need a way to wire this alternative module in - that's what build-support dependencies are for. More on this later.
>> Question:
>> is this really optional? I certain want the tools that *we* >> build to do this, as I don't want to download <package-x>-1.0.1 every >> time I use it.
> In my mind it is. I have a lot of bandwidth and I am nearly always > connected. I would rather just download the things I need when I need > them rather then have some 3rd location that I have to know about and > maintain. For me its a trade off of bandwidth for increased > simplicity.
Yes ok, but a few points here:
1. you don't have to manage anything yourself, the tool does it for you 2. once you've already downloaded something, why do you want to do it again and again, etc?
We're both looking at the local repo as a cache, and caching downloads so you don't have to repeatedly get them is simple and obvious. If the local index has support for marking items as having already been downloaded - perhaps simply by putting a file://<path> url next to the item - then everything else should be able to remain blissfully ignorant of the local cache.
> I probably wont have a global anything, no global index no local repo > etc. It will all be stored on a project by project basis. One of the > reasons that I want to have command line options for everything so so > that I can do that. That removes things hanging out it some global > directory isolates problems to a single project and makes it very easy > to clean up. With that model in mind a local repo is very redundant > and actually mostly just adds unneeded overhead. Disk space is cheap, > band with is cheap I want to leverage both.
I really don't like the idea of having an index for every project. I cannot think of any other tool that does this, not any of the apt/yum/etc package managers, not any of the varietous build tools (except autotools, which makes most people go insane anyway).
That said, if you're happy for us to provide both options via command line flags (or user defined configuration files) then I have no issues with it if you want to re-download the index for every project you're working on.
>> I'm not convinced this is really an optimization rather than the only >> sane way to do things. Think about a build plugin for example, or a >> logging library that you use in all your applications/systems. Do you >> really want to resolve this to some local app directory each time you >> start a new project?
> Yes actually.
This is fair enough. The designers of rebar think the same way and I guess can understand the benefits of isolation. I don't want it for myself but if we're going to provide both options then we should hopefully be able to meet the needs of a wider audience anyway.
>> This is what rebar does with dependencies that I >> actually hate - the local repository (along with a good index and binary >> artefacts) is part of the main reason I want to do this stuff.
> Its one of the few things that I think rebar got right. I like things > to be self contained. That maybe why I like the rox and osx > application model so much. Its also the reason I distribute my otp > releases with everything the need. Its a space trade off that I think > is well worth it.
Which OSX applications re-bundle libraries that are used in multiple places each time? Or don't get installed into either a global or user-wide directory? Or fail to put commonly used things into a standard and common Framework directory? I see what you're getting at, but there are numerous angles, an I suppose the way in which people perceive and/or interpret the principles is where the split tends to occur.
I agree with the principle of having things self contained, but I don't see this as an important example of self-containment. I completely agree with you about releases needing to contain everything you need - that is an example that stands out to me. Having said that, we've already agreed that in order to make our lives 'manageable' we are not going to bundle runtime OTP libraries in packaged releases, so clearly 'self-containment' isn't an overriding principle in the face of other forces.
> We might be able to do this transparently in something like ~/.repo > but I absolutely want the option to turn it off.
I am absolutely fine with people being able to choose between using a project local app dir (and index) and having the dependencies resolved from the local repo. I am also absolutely fine with writing the local repo handling code myself, as long as I have an API to integrate it with the assembler/packager. I want to make this endevour
1. easy for you and I as well as other potential contributors to work on 2. as simple to understand as possible, inside and out 3. as configurable and extensible as possible (without violating 1 and 2 greatly)
>> Question:
>> I agree that we need the namespace flattening, but I'm >> wondering why we need do implement it in this way. On windows, for >> example, this is a royal pain as there are no symlinks.
> I wasn't planning to use symlinks at all.
Fair enough. I tend to err on the side of performance, and my dislike of copying files and directories around (and/or repeatedly fetching things from the internel) is not about space usage or redundancy, it's about making sure that my build+test cycle takes seconds and not minutes. Bare in mind that for me, a full integration test requires that the release is packaged and deployed to a VM before the end-2-end test suites (common_test and other things besides) are run, and most CI servers do this with a fresh checkout from version control, not a pull/update - this means that if everything has to be fetched and copied and so on, the build can take 10 minutes which is a very long lapse to then discover that you've broken someone else's code.
>> If we make the local repository a core part of the solution, then >> there is another option which IMO is viable.
>> All the use-cases for the dependencies involve getting a running >> emulator to 'know about' them, for example compile time ones (such >> as parse transforms and build plugins) need to be on the code path >> during the 'build tool' operations. Runtime ones need to be available >> during testing, local development and pacakging a release. Test scoped >> ones need to be on the path during testing. In all cases, the following >> things need to work
>> So why not just add the correct directories to the code path at the >> appropriate time? This saves copying (or symlinking) and avoids >> cluttering the local directory structure.
> lol, I think we have very different outlooks here. We are going to > need to resolve that.
Yes I can see that! :)
I'm not worried about resolving it - I will go along with implementing your preferred method(s) as long as there are extensibility points for me to exploit so I can implement my alternative approaches re global local repository and index handling. I don't think you're wrong, I just think people have different opionions about this (mainly based on the tools they're familiar and comfortable with) and having both options strikes me as a good thing.
>> It also avoid having to >> manage an 'app dir' when dependencies are removed for example.
> nuking the directory?
Yes you're right - that's the simplest way to refresh the dependencies whenever anything other than an insert takes place in the dependency graph. I guess we've made an implementation decision for the app-dir then.
>> Personally I find this approach preferable, as long as the code >> path modification process is documented clearly.
> This is doable. There are two things I dont like about it. The first > and biggest is that its much more difficult to plug in arbitrary tools > that are not part of the chain. That is things that use the packages > now need to know about
> On 1 March 2012 16:14, Eric Merritt <ericbmerr...@gmail.com> wrote: >> Tim,
>> On the wiki. Lets send questions to this list and just update the wiki >> directly once we come to some conclusion or just update the wiki if >> you are comfortable with that.
> Sure I wasn't sure which model to go for, but from now on I'll only > post updates to the wiki and keep discusions to the list. That's > actually cleaner anyway, from a random visitor's point of view not to > see lots of questions inline.
>>> Question(s):
>>> do we want to tie ourselves to git+github in the spec? >>> My thoughts are that people may use any mechanism they like >>> to build alternative repository implementations (which we should >>> support via a plugins mechanism) - not everyone uses git
>>> In light of this, can we rephrase item 2, as >>> "Repositories will initially be stored in git and targeted at github"?
>> I think in version on we do target git and github. Down the road we >> will have other tasks to branch that out. However, I think having a >> hard target initially is a good thing.
>> This is not a hard and fast requirement for me though.
> I am fine with it being github initially, but for work there are some > cases where I can't publish internal/private metadata to github > because it is offsite. We are a former state run monopoly and the > largest telecoms outfit in the UK, so we not only have a lot of rules > around privacy but also lots of government as well as industry > regulation that we have to comply with. In some cases, even the notion > that an artefact exists must not reside outside of the corporate LAN, > for legal reasons.
> I will write an alternative remote index handler that doesn't use > github (based on subversion or nexus probably) but I will need a way > to wire this alternative module in - that's what build-support > dependencies are for. More on this later.
>>> Question:
>>> is this really optional? I certain want the tools that *we* >>> build to do this, as I don't want to download <package-x>-1.0.1 every >>> time I use it.
>> In my mind it is. I have a lot of bandwidth and I am nearly always >> connected. I would rather just download the things I need when I need >> them rather then have some 3rd location that I have to know about and >> maintain. For me its a trade off of bandwidth for increased >> simplicity.
> Yes ok, but a few points here:
> 1. you don't have to manage anything yourself, the tool does it for you > 2. once you've already downloaded something, why do you want to do it > again and again, etc?
> We're both looking at the local repo as a cache, and caching downloads > so you don't have to repeatedly get them is simple and obvious. If the > local index has support for marking items as having already been > downloaded - perhaps simply by putting a file://<path> url next to the > item - then everything else should be able to remain blissfully > ignorant of the local cache.
>> I probably wont have a global anything, no global index no local repo >> etc. It will all be stored on a project by project basis. One of the >> reasons that I want to have command line options for everything so so >> that I can do that. That removes things hanging out it some global >> directory isolates problems to a single project and makes it very easy >> to clean up. With that model in mind a local repo is very redundant >> and actually mostly just adds unneeded overhead. Disk space is cheap, >> band with is cheap I want to leverage both.
> I really don't like the idea of having an index for every project. I > cannot think of any other tool that does this, not any of the > apt/yum/etc package managers, not any of the varietous build tools > (except autotools, which makes most people go insane anyway).
> That said, if you're happy for us to provide both options via command > line flags (or user defined configuration files) then I have no issues > with it if you want to re-download the index for every project you're > working on.
>>> I'm not convinced this is really an optimization rather than the only >>> sane way to do things. Think about a build plugin for example, or a >>> logging library that you use in all your applications/systems. Do you >>> really want to resolve this to some local app directory each time you >>> start a new project?
>> Yes actually.
> This is fair enough. The designers of rebar think the same way and I > guess can understand the benefits of isolation. I don't want it for > myself but if we're going to provide both options then we should > hopefully be able to meet the needs of a wider audience anyway.
>>> This is what rebar does with dependencies that I >>> actually hate - the local repository (along with a good index and binary >>> artefacts) is part of the main reason I want to do this stuff.
>> Its one of the few things that I think rebar got right. I like things >> to be self contained. That maybe why I like the rox and osx >> application model so much. Its also the reason I distribute my otp >> releases with everything the need. Its a space trade off that I think >> is well worth it.
> Which OSX applications re-bundle libraries that are used in multiple > places each time? Or don't get installed into either a global or > user-wide directory? Or fail to put commonly used things into a > standard and common Framework directory? I see what you're getting at, > but there are numerous angles, an I suppose the way in which people > perceive and/or interpret the principles is where the split tends to > occur.
> I agree with the principle of having things self contained, but I > don't see this as an important example of self-containment. I > completely agree with you about releases needing to contain everything > you need - that is an example that stands out to me. Having said that, > we've already agreed that in order to make our lives 'manageable' we > are not going to bundle runtime OTP libraries in packaged releases, so > clearly 'self-containment' isn't an overriding principle in the face > of other forces.
>> We might be able to do this transparently in something like ~/.repo >> but I absolutely want the option to turn it off.
> I am absolutely fine with people being able to choose between using a > project local app dir (and index) and having the dependencies resolved > from the local repo. I am also absolutely fine with writing the local > repo handling code myself, as long as I have an API to integrate it > with the assembler/packager. I want to make this endevour
> 1. easy for you and I as well as other potential contributors to work on > 2. as simple to understand as possible, inside and out > 3. as configurable and extensible as possible (without violating 1 and > 2 greatly)
>>> Question:
>>> I agree that we need the namespace flattening, but I'm >>> wondering why we need do implement it in this way. On windows, for >>> example, this is a royal pain as there are no symlinks.
>> I wasn't planning to use symlinks at all.
> Fair enough. I tend to err on the side of performance, and my dislike > of copying files and directories around (and/or repeatedly fetching > things from the internel) is not about space usage or redundancy, it's > about making sure that my build+test cycle takes seconds and not > minutes. Bare in mind that for me, a full integration test requires > that the release is packaged and deployed to a VM before the end-2-end > test suites (common_test and other things besides) are run, and most > CI servers do this with a fresh checkout from version control, not a > pull/update - this means that if everything has to be fetched and > copied and so on, the build can take 10 minutes which is a very long > lapse to then discover that you've broken someone else's code.
>>> If we make the local repository a core part of the solution, then >>> there is another option which IMO is viable.
>>> All the use-cases for the dependencies involve getting a running >>> emulator to 'know about' them, for example compile time ones (such >>> as parse transforms and build plugins) need to be on the code path >>> during the 'build tool' operations. Runtime ones need to be available >>> during testing, local development and pacakging a release. Test scoped >>> ones need to be on the path during testing. In all cases, the following >>> things need to work
>>> So why not just add the correct directories to the code path at the >>> appropriate time? This saves copying (or symlinking) and avoids >>> cluttering the local directory structure.
>> lol, I think we have very different outlooks here. We are going to >> need to resolve that.
> Yes I can see that! :)
> I'm not worried about resolving it - I will go along with implementing > your preferred method(s) as long as there are extensibility points for > me to exploit so I can implement my alternative approaches re global > local repository and index handling. I don't think you're wrong, I > just think people have different opionions about this (mainly based on > the tools they're familiar and comfortable with) and having both > options strikes me as a good thing.
>>> It also avoid having to >>> manage an 'app dir' when dependencies are removed for example.
>> nuking the directory?
> Yes you're right - that's the simplest way to refresh the dependencies > whenever anything other than an insert takes place
> On 1 March 2012 16:14, Eric Merritt <ericbmerr...@gmail.com> wrote: > > Tim,
> > On the wiki. Lets send questions to this list and just update the wiki > > directly once we come to some conclusion or just update the wiki if > > you are comfortable with that.
> Sure I wasn't sure which model to go for, but from now on I'll only > post updates to the wiki and keep discusions to the list. That's > actually cleaner anyway, from a random visitor's point of view not to > see lots of questions inline.
> >> Question(s):
> >> do we want to tie ourselves to git+github in the spec? > >> My thoughts are that people may use any mechanism they like > >> to build alternative repository implementations (which we should > >> support via a plugins mechanism) - not everyone uses git
> >> In light of this, can we rephrase item 2, as > >> "Repositories will initially be stored in git and targeted at github"?
> > I think in version on we do target git and github. Down the road we > > will have other tasks to branch that out. However, I think having a > > hard target initially is a good thing.
> > This is not a hard and fast requirement for me though.
> I am fine with it being github initially, but for work there are some > cases where I can't publish internal/private metadata to github > because it is offsite. We are a former state run monopoly and the > largest telecoms outfit in the UK, so we not only have a lot of rules > around privacy but also lots of government as well as industry > regulation that we have to comply with. In some cases, even the notion > that an artefact exists must not reside outside of the corporate LAN, > for legal reasons.
> I will write an alternative remote index handler that doesn't use > github (based on subversion or nexus probably) but I will need a way > to wire this alternative module in - that's what build-support > dependencies are for. More on this later.
Understandable. This is going to be a bit of a chicken and the egg problem. If you need the package manager to manage packages but you need the packages to get the package manager what do you do?
In this model I think we have to assume that the package manager is not available. I dont see how it could be. If that is the case then it still obviates the need for that class of packages.
> >> is this really optional? I certain want the tools that *we* > >> build to do this, as I don't want to download <package-x>-1.0.1 every > >> time I use it.
> > In my mind it is. I have a lot of bandwidth and I am nearly always > > connected. I would rather just download the things I need when I need > > them rather then have some 3rd location that I have to know about and > > maintain. For me its a trade off of bandwidth for increased > > simplicity.
> Yes ok, but a few points here:
> 1. you don't have to manage anything yourself, the tool does it for you
this never quite seems to be the reality of things.
> 2. once you've already downloaded something, why do you want to do it > again and again, etc?
Because downloading it again is simpler and less complex then keeping it around. (this is an assumption that is worth questioning in the same way that having a cache is an assumption worth questioning).
> We're both looking at the local repo as a cache, and caching downloads > so you don't have to repeatedly get them is simple and obvious. If the > local index has support for marking items as having already been > downloaded - perhaps simply by putting a file://<path> url next to the > item - then everything else should be able to remain blissfully > ignorant of the local cache.
If we go this route we should probably just check the local cache to see if its been downloaded. That is the true canonical location for that information. No need for additional metadata in the index.
> > I probably wont have a global anything, no global index no local repo > > etc. It will all be stored on a project by project basis. One of the > > reasons that I want to have command line options for everything so so > > that I can do that. That removes things hanging out it some global > > directory isolates problems to a single project and makes it very easy > > to clean up. With that model in mind a local repo is very redundant > > and actually mostly just adds unneeded overhead. Disk space is cheap, > > band with is cheap I want to leverage both.
> I really don't like the idea of having an index for every project. I > cannot think of any other tool that does this, not any of the > apt/yum/etc package managers, not any of the varietous build tools > (except autotools, which makes most people go insane anyway).
Of course not. they are system package managers not build or language package managers. There is a very big difference. I can think of a couple of language package managers that do do this though.
> That said, if you're happy for us to provide both options via command > line flags (or user defined configuration files) then I have no issues > with it if you want to re-download the index for every project you're > working on.
Thats just the point. Providing the command line flags cost us very little and add enough flexibility to the process to meet everyones needs.
> >> I'm not convinced this is really an optimization rather than the only > >> sane way to do things. Think about a build plugin for example, or a > >> logging library that you use in all your applications/systems. Do you > >> really want to resolve this to some local app directory each time you > >> start a new project?
> > Yes actually.
> This is fair enough. The designers of rebar think the same way and I > guess can understand the benefits of isolation. I don't want it for > myself but if we're going to provide both options then we should > hopefully be able to meet the needs of a wider audience anyway.
Again, we should be able to support both really. Even having a non-optional local cache doesn't change that. For me its just a redundant extra step.
> >> This is what rebar does with dependencies that I > >> actually hate - the local repository (along with a good index and binary > >> artefacts) is part of the main reason I want to do this stuff.
> > Its one of the few things that I think rebar got right. I like things > > to be self contained. That maybe why I like the rox and osx > > application model so much. Its also the reason I distribute my otp > > releases with everything the need. Its a space trade off that I think > > is well worth it.
> Which OSX applications re-bundle libraries that are used in multiple > places each time?
I should have said historically. I guess that may not be the case so much any more.
> Or don't get installed into either a global or > user-wide directory? Or fail to put commonly used things into a > standard and common Framework directory?
the apps are self contained. If you drop an app into the 'applications' directory depending on your permissions it goes into the global or yours. Now with more windows like installers I guess this has started to change. Its been awhile since I used a mac.
> I see what you're getting at, > but there are numerous angles, an I suppose the way in which people > perceive and/or interpret the principles is where the split tends to > occur.
Thats exactly true.
> I agree with the principle of having things self contained, but I > don't see this as an important example of self-containment. I > completely agree with you about releases needing to contain everything > you need - that is an example that stands out to me. Having said that, > we've already agreed that in order to make our lives 'manageable' we > are not going to bundle runtime OTP libraries in packaged releases, so > clearly 'self-containment' isn't an overriding principle in the face > of other forces.
Not for our use case of providing packages. However, I will absolutely still do that when I go to deploy a release. I seriously doubt I will be deploying directly from our packages.
> > We might be able to do this transparently in something like ~/.repo > > but I absolutely want the option to turn it off.
> I am absolutely fine with people being able to choose between using a > project local app dir (and index) and having the dependencies resolved > from the local repo.
I think I might be confused by this. Let me restate and see if I misunderstood.
The local repo isnt really usable directly by erlang. Not without some tool to munge the paths and its certainly not usable by any tool that isnt in this toolchain or hasn't been modified to support this tool chain. So I would like the dependencies to end up in some usable place. I dont actually care where they come from be it a local repo or directly from a remote repo.
I worry that if we dont have some 'normal' directory structure we will be getting away from the idea that the end points and outpoints of each tool are generally consumable. That actually bugs me a bit from an interoperability standpoint.
> I am also absolutely fine with writing the local > repo handling code myself, as long as I have an API to integrate it > with the assembler/packager. I want to make this endevour
> 1. easy for you and I as well as other potential contributors to work on > 2. as simple to understand as possible, inside and out > 3. as configurable and extensible as possible (without violating 1 and > 2 greatly)
I think we have the same goals we are just a bit off on thoughts about how each applies. I suspect we can come to some agreement.
> No that's not so much it. Think about it like this - I want to write a > custom 'remote index' handler that'll resolve/fetch the remote index > from subversion or nexus instead of github. The default remote index > handler is a module which is probably called something like > 'erlware_remote_index' and is on the code path because it is > distributed with the tool (being the default handler). I want to put > my 'bt_nexus_remote_index' module into an OTP application and resolve > it just like any other dependency. Once it is resolved (via the > default 'remote index' on github) I want it downloaded and then added > to the code path. I want this to happen before any further > dependencies are resolved, because I want this 'extension' to kick in > and add its own index metadata so that private artefacts which are > hosted internally on nexus can be resolved alongside public open > source components hosted on github/bitbucket.
So this issue of getting hold of extensions and loading them dynamically is a real pain in rebar, and is one of the things that has driven me to look for an external solution that I can plug into any build tool (including rebar) to make my sanity come back. So I have a couple of approaches to working with dependencies that 'cause rebar problems' for one reason or another. One of them is alt_deps, which resolves dependencies in a different way and along with my 'alien plugin' supports deps that are rely on a non-rebar tool chain to be built.
I want this plugin to work in the normal way, so that people go 'rebar get-deps compile' and it just works. But this isn't the case because first of all the person who has downloaded my project needs to go and get alt_deps and alien_plugin! To solve this problem, I distribute these extensions as dependencies.
Now here come the rub. For rebar to recognise my plugin's extension hook, it needs to find rebar_alt_deps:'get-deps'/2 on the code path. In order for that to work, the plugin needs to be compiled, so I rely on rebar fetching the plugin and then compiling it. But now I have a problem because once 'get-deps' is finished it runs 'compile' and then stops. Now the user is forced to do some crazy nonsense like
$ rebar get-deps compile install-deps
That just looks (and is) plain wrong. I don't have any of this crap to deal with if I write and distribute a maven plugin. I write the plugin and publish it to a public OSS repository (like Sonatype Nexus OSS) and then in the projects that use it, I just put a reference to it in my pom.xml:
And that's it!!! Maven downloads the plugin the first time this project is built by running 'mvn compile' and after that it is business as usual. A plugin in this model is basically a kind of internally/specially scoped dependency.
> Just one quick note/clarification inline below, which I thought might > be useful background.
> On 1 March 2012 19:54, Tim Watson <watson.timo...@gmail.com> wrote:
> > No that's not so much it. Think about it like this - I want to write a > > custom 'remote index' handler that'll resolve/fetch the remote index > > from subversion or nexus instead of github. The default remote index > > handler is a module which is probably called something like > > 'erlware_remote_index' and is on the code path because it is > > distributed with the tool (being the default handler). I want to put > > my 'bt_nexus_remote_index' module into an OTP application and resolve > > it just like any other dependency. Once it is resolved (via the > > default 'remote index' on github) I want it downloaded and then added > > to the code path. I want this to happen before any further > > dependencies are resolved, because I want this 'extension' to kick in > > and add its own index metadata so that private artefacts which are > > hosted internally on nexus can be resolved alongside public open > > source components hosted on github/bitbucket.
> So this issue of getting hold of extensions and loading them > dynamically is a real pain in rebar, and is one of the things that has > driven me to look for an external solution that I can plug into any > build tool (including rebar) to make my sanity come back. So I have a > couple of approaches to working with dependencies that 'cause rebar > problems' for one reason or another. One of them is alt_deps, which > resolves dependencies in a different way and along with my 'alien > plugin' supports deps that are rely on a non-rebar tool chain to be > built.
> I want this plugin to work in the normal way, so that people go 'rebar > get-deps compile' and it just works. But this isn't the case because > first of all the person who has downloaded my project needs to go and > get alt_deps and alien_plugin! To solve this problem, I distribute > these extensions as dependencies.
> Now here come the rub. For rebar to recognise my plugin's extension > hook, it needs to find rebar_alt_deps:'get-deps'/2 on the code path. > In order for that to work, the plugin needs to be compiled, so I rely > on rebar fetching the plugin and then compiling it. But now I have a > problem because once 'get-deps' is finished it runs 'compile' and then > stops. Now the user is forced to do some crazy nonsense like
> $ rebar get-deps compile install-deps
> That just looks (and is) plain wrong. I don't have any of this crap to > deal with if I write and distribute a maven plugin. I write the plugin > and publish it to a public OSS repository (like Sonatype Nexus OSS) > and then in the projects that use it, I just put a reference to it in > my pom.xml:
> And that's it!!! Maven downloads the plugin the first time this > project is built by running 'mvn compile' and after that it is > business as usual. A plugin in this model is basically a kind of > internally/specially scoped dependency.
I think we are going to end up supporting this case. We should be able to certainly get around the problems you are specifying here with rebar.
I think you have talked me into this. The only caveat to the list below is that I think we should have a simple tool that takes a list of paths and copies the OTP apps to a directory.
It would be cool if each of these could read their inputs on standard in and write the outputs to standard out as well.
**manager**:
Inputs: a list of repos? outputs: local index + basic local repo structure
**solver**:
Input: Constraints are by their nature not OTP but we should try to fit them in as seamlessly as possible
Output: Should be a rel or relup file with organization annotation
**fetcher**: Inputs: a rel or relup file Outputs: A list of code paths for the various dependencies
**builder**: Inputs: A project that contains a rel+/relup file, a list of code paths for dependencies Outputs: A built otp application(s) + updated list of codepaths
**assembler**: Inputs: A rel/+relup, A list of codepaths Outputs: An OTP Release directory structure
**packager**: Inputs: An OTP Release directory structure Outputs: A dist tarball (debian package, rpm etc)
I was just starting to think I needed to reconsider my position in light of the 'lots of little tools' approach - not be so monolithic in my perception of the process and all that - and it occurs to me that if some of our outputs are lists of file system paths we might be ok. I come on here to suggest it and voila, you've read my mind.
On 1 March 2012 21:01, Eric B Merritt <ericbmerr...@gmail.com> wrote:
> I think you have talked me into this. The only caveat to the list > below is that I think we should have a simple tool that takes a list > of paths and copies the OTP apps to a directory.
I completely agree with that.
> It would be cool if each of these could read their inputs on standard > in and write the outputs to standard out as well.
Yes definitely. I'm almost inclined to say that should be the required unix-y interface, so that people who're using Make (and similar tools) can still get value out of each and every one of these tools as they see fit.
> **manager**:
> Inputs: a list of repos? > outputs: local index + basic local repo structure
Inputs: you mean a list of 'remote indexes' here right? The word 'repo' is overloaded and implies storage more than anything else. But yes, I think a list of repos is fine.
Flags: -f <filename> read the list of repos from <filename> instead of stdin?
Question: how are the repos specified? As a URL or as something else?
Question: this guy is presumably not writing to stdout, as it is creating directories on disk etc. So perhaps the stdout stream should contain the root directories and/or locations of these?
> **solver**:
> Input: Constraints are by their nature not OTP but we should try to > fit them in as seamlessly as possible
> Output: Should be a rel or relup file with organization annotation
Yes that output sounds very reasonable. This is the area we've spent the least time thinking about the inputs for.
> **fetcher**: > Inputs: a rel or relup file > Outputs: A list of code paths for the various dependencies
Absolutely - this is a great idea. What about where you're in an OTP application/library project and want to fetch your dependencies? Should we still work off a rel file out of the solver? I guess it doesn't matter that the rel/relup file is not needed, as we can still use it as an artefact in our process if we want to.
> **builder**: > Inputs: A project that contains a rel+/relup file, a list of code paths for dependencies > Outputs: A built otp application(s) + updated list of codepaths
Cool, I like this. As we've discussed before though, not every project is a release - sometimes it's just an application or a library. We need to support those use cases as well, so the rel/+relup can't be mandatory.
> **assembler**: > Inputs: A rel/+relup, A list of codepaths > Outputs: An OTP Release directory structure
> **packager**: > Inputs: An OTP Release directory structure > Outputs: A dist tarball (debian package, rpm etc)
Ok I'm fine with these. Also the packager needs to support packaging plain applications/libraries as well as releases.
> Inputs: A dist tarball and choice of index to publish to + the users personal configuration (passwords, etc) > Outputs-1: an HTTP request to github/bitbucket/nexus/etc to upload/store the publication metadata in an index > Outputs-2: an HTTP request to github/bitbucket/nexus/etc to upload/store the binary artefact online
> I was just starting to think I needed to reconsider my position in > light of the 'lots of little tools' approach - not be so monolithic in > my perception of the process and all that - and it occurs to me that > if some of our outputs are lists of file system paths we might be ok. > I come on here to suggest it and voila, you've read my mind.
Its good to be on the same page.
> > It would be cool if each of these could read their inputs on standard > > in and write the outputs to standard out as well.
> Yes definitely. I'm almost inclined to say that should be the required > unix-y interface, so that people who're using Make (and similar tools) > can still get value out of each and every one of these tools as they > see fit.
I agree lets make that a requirement.
> > **manager**:
> > Inputs: a list of repos? > > outputs: local index + basic local repo structure
> Inputs: you mean a list of 'remote indexes' here right?
Yes. lets adopt that terminology.
> The word > 'repo' is overloaded and implies storage more than anything else. But > yes, I think a list of repos is fine.
> Flags: -f <filename> read the list of repos from <filename> instead of stdin?
Sure. It should also read configuration files as well.
> Question: how are the repos specified? As a URL or as something else?
I think all the DVCSs support urls, so lets just use those. We can fit the plugin concept around those urls.
> Question: this guy is presumably not writing to stdout, as it is > creating directories on disk etc. So perhaps the stdout stream should > contain the root directories and/or locations of these?
Thats a good idea actually.
> > **solver**:
> > Input: Constraints are by their nature not OTP but we should try to > > fit them in as seamlessly as possible
> > Output: Should be a rel or relup file with organization annotation
> Yes that output sounds very reasonable. This is the area we've spent > the least time thinking about the inputs for.
It might also take the directory of the repo as part of its input. That way it could be linked to the above. Along with command line options as well of course.
> > **fetcher**: > > Inputs: a rel or relup file > > Outputs: A list of code paths for the various dependencies
> Absolutely - this is a great idea. What about where you're in an OTP > application/library project and want to fetch your dependencies? > Should we still work off a rel file out of the solver? I guess it > doesn't matter that the rel/relup file is not needed, as we can still > use it as an artefact in our process if we want to.
In this case the rel file just serves as a carrier of dependency information. I think that works whether you have a single app or an entire release. I dont think there is any need to invent something ourselves that carries the exact same information.
> > **builder**: > > Inputs: A project that contains a rel+/relup file, a list of code paths for dependencies > > Outputs: A built otp application(s) + updated list of codepaths
> Cool, I like this. As we've discussed before though, not every project > is a release - sometimes it's just an application or a library. We > need to support those use cases as well, so the rel/+relup can't be > mandatory.
As I said above we need something to carry that information. Using a dotrel makes more sense then inventing something ourselves. You can have one without actually using it for anything else.
> > **assembler**: > > Inputs: A rel/+relup, A list of codepaths > > Outputs: An OTP Release directory structure
> Ok I'm fine with these. Also the packager needs to support packaging > plain applications/libraries as well as releases.
if by applications you mean OTP Applications of the active and inactive variety (curse erlang's name overloading) then I am on board.
> **publisher** > > Inputs: A dist tarball and choice of index to publish to + the users personal configuration (passwords, etc) > > Outputs-1: an HTTP request to github/bitbucket/nexus/etc to upload/store the publication metadata in an index > > Outputs-2: an HTTP request to github/bitbucket/nexus/etc to upload/store the binary artefact online
Absolutely.
There are probably tools for a repository maintainer as well. For pulling merge requests, verifying signatures and the like. To come up with something we forward to the erlang-questions list we dont need to detail that. It is something we will want to keep in mind though.
At Thu, 1 Mar 2012 22:10:05 +0000, top posting again.
I just realized that we could turn our 'list of code paths' to a viable source in and of itself and then use that is inputs and outputs to everything even a rel/relup compiler. The reality is that we need a bunch of stuff in addition to whats in the so having that in this spec and making the rel a compile out put would work. Though we loose the completely vanilla otpishness of some of the tools.
> I was just starting to think I needed to reconsider my position in > light of the 'lots of little tools' approach - not be so monolithic in > my perception of the process and all that - and it occurs to me that > if some of our outputs are lists of file system paths we might be ok. > I come on here to suggest it and voila, you've read my mind.
> On 1 March 2012 21:01, Eric B Merritt <ericbmerr...@gmail.com> wrote:
> > I think you have talked me into this. The only caveat to the list > > below is that I think we should have a simple tool that takes a list > > of paths and copies the OTP apps to a directory.
> I completely agree with that.
> > It would be cool if each of these could read their inputs on standard > > in and write the outputs to standard out as well.
> Yes definitely. I'm almost inclined to say that should be the required > unix-y interface, so that people who're using Make (and similar tools) > can still get value out of each and every one of these tools as they > see fit.
> > **manager**:
> > Inputs: a list of repos? > > outputs: local index + basic local repo structure
> Inputs: you mean a list of 'remote indexes' here right? The word > 'repo' is overloaded and implies storage more than anything else. But > yes, I think a list of repos is fine.
> Flags: -f <filename> read the list of repos from <filename> instead of stdin?
> Question: how are the repos specified? As a URL or as something else?
> Question: this guy is presumably not writing to stdout, as it is > creating directories on disk etc. So perhaps the stdout stream should > contain the root directories and/or locations of these?
> > **solver**:
> > Input: Constraints are by their nature not OTP but we should try to > > fit them in as seamlessly as possible
> > Output: Should be a rel or relup file with organization annotation
> Yes that output sounds very reasonable. This is the area we've spent > the least time thinking about the inputs for.
> > **fetcher**: > > Inputs: a rel or relup file > > Outputs: A list of code paths for the various dependencies
> Absolutely - this is a great idea. What about where you're in an OTP > application/library project and want to fetch your dependencies? > Should we still work off a rel file out of the solver? I guess it > doesn't matter that the rel/relup file is not needed, as we can still > use it as an artefact in our process if we want to.
> > **builder**: > > Inputs: A project that contains a rel+/relup file, a list of code paths for dependencies > > Outputs: A built otp application(s) + updated list of codepaths
> Cool, I like this. As we've discussed before though, not every project > is a release - sometimes it's just an application or a library. We > need to support those use cases as well, so the rel/+relup can't be > mandatory.
> > **assembler**: > > Inputs: A rel/+relup, A list of codepaths > > Outputs: An OTP Release directory structure
> Ok I'm fine with these. Also the packager needs to support packaging > plain applications/libraries as well as releases.
> **publisher** > > Inputs: A dist tarball and choice of index to publish to + the users personal configuration (passwords, etc) > > Outputs-1: an HTTP request to github/bitbucket/nexus/etc to upload/store the publication metadata in an index > > Outputs-2: an HTTP request to github/bitbucket/nexus/etc to upload/store the binary artefact online
I'm not sure I fully understand what this means, some of the tools don't care about the list of code paths. I suppose each of the tools cares about a list of something (repositories, dependencies, code paths, etc). I think we are going to end up passing around data that isn't pure OTP (app and rel files) anyway. If the OTP metadata was rich enough to describe all of what we're trying to achieve, we wouldn't have a job to do because it'd already be written.
I have some outstanding questions about configuration. Where do the dependency lists go?In the .app file? Doesn't adding the ">=" and version number screw other tools up? I know that my 'appstart' utility will puke, although I'll happily change it if it's the only thing that does.
> At Thu, 1 Mar 2012 22:10:05 +0000, > top posting again.
> I just realized that we could turn our 'list of code paths' to a > viable source in and of itself and then use that is inputs and outputs > to everything even a rel/relup compiler. The reality is that we need a > bunch of stuff in addition to whats in the so having that in this spec > and making the rel a compile out put would work. Though we loose the > completely vanilla otpishness of some of the tools.
> Tim Watson wrote:
>> I was just starting to think I needed to reconsider my position in >> light of the 'lots of little tools' approach - not be so monolithic in >> my perception of the process and all that - and it occurs to me that >> if some of our outputs are lists of file system paths we might be ok. >> I come on here to suggest it and voila, you've read my mind.
>> On 1 March 2012 21:01, Eric B Merritt <ericbmerr...@gmail.com> wrote:
>>> I think you have talked me into this. The only caveat to the list >>> below is that I think we should have a simple tool that takes a list >>> of paths and copies the OTP apps to a directory.
>> I completely agree with that.
>>> It would be cool if each of these could read their inputs on standard >>> in and write the outputs to standard out as well.
>> Yes definitely. I'm almost inclined to say that should be the required >> unix-y interface, so that people who're using Make (and similar tools) >> can still get value out of each and every one of these tools as they >> see fit.
>>> **manager**:
>>> Inputs: a list of repos? >>> outputs: local index + basic local repo structure
>> Inputs: you mean a list of 'remote indexes' here right? The word >> 'repo' is overloaded and implies storage more than anything else. But >> yes, I think a list of repos is fine.
>> Flags: -f <filename> read the list of repos from <filename> instead of stdin?
>> Question: how are the repos specified? As a URL or as something else?
>> Question: this guy is presumably not writing to stdout, as it is >> creating directories on disk etc. So perhaps the stdout stream should >> contain the root directories and/or locations of these?
>>> **solver**:
>>> Input: Constraints are by their nature not OTP but we should try to >>> fit them in as seamlessly as possible
>>> Output: Should be a rel or relup file with organization annotation
>> Yes that output sounds very reasonable. This is the area we've spent >> the least time thinking about the inputs for.
>>> **fetcher**: >>> Inputs: a rel or relup file >>> Outputs: A list of code paths for the various dependencies
>> Absolutely - this is a great idea. What about where you're in an OTP >> application/library project and want to fetch your dependencies? >> Should we still work off a rel file out of the solver? I guess it >> doesn't matter that the rel/relup file is not needed, as we can still >> use it as an artefact in our process if we want to.
>>> **builder**: >>> Inputs: A project that contains a rel+/relup file, a list of code paths for dependencies >>> Outputs: A built otp application(s) + updated list of codepaths
>> Cool, I like this. As we've discussed before though, not every project >> is a release - sometimes it's just an application or a library. We >> need to support those use cases as well, so the rel/+relup can't be >> mandatory.
>>> **assembler**: >>> Inputs: A rel/+relup, A list of codepaths >>> Outputs: An OTP Release directory structure
>>> **packager**: >>> Inputs: An OTP Release directory structure >>> Outputs: A dist tarball (debian package, rpm etc)
>> Ok I'm fine with these. Also the packager needs to support packaging >> plain applications/libraries as well as releases.
>> **publisher** >>> Inputs: A dist tarball and choice of index to publish to + the users personal configuration (passwords, etc) >>> Outputs-1: an HTTP request to github/bitbucket/nexus/etc to upload/store the publication metadata in an index >>> Outputs-2: an HTTP request to github/bitbucket/nexus/etc to upload/store the binary artefact online
> I'm not sure I fully understand what this means, some of the tools > don't care about the list of code paths. I suppose each of the tools > cares about a list of something (repositories, dependencies, code > paths, etc).
I was thinking about just those tools that take code paths. That is everything but the manager and packager really.
> I think we are going to end up passing around data that > isn't pure OTP (app and rel files) anyway. If the OTP metadata was > rich enough to describe all of what we're trying to achieve, we > wouldn't have a job to do because it'd already be written.
I am not sure thats the case. The OTP guys never tackled and so simply never addressed these issues. That doesn't mean the OTP metadata isn't sufficient.
It also does not mean that we should either, but I would like to avoid redundancies if possible.
> I have some outstanding questions about configuration. Where do the > dependency lists go?In the .app file?
I suspect thats where they would go by default. Though I suspect we need to be able to specify them directly as well.
> Doesn't adding the ">=" and > version number screw other tools up?
Yup. It would screw up everything. However, we can add another tuple there without a problem and that is the approach I would take. That extra tuple would give us the information we would need.
> I know that my 'appstart' > utility will puke, although I'll happily change it if it's the only > thing that does.
Again none of this is set in stone. We could also stick it in a separate file in priv or any number of things.
> > At Thu, 1 Mar 2012 22:10:05 +0000, > > top posting again.
> > I just realized that we could turn our 'list of code paths' to a > > viable source in and of itself and then use that is inputs and outputs > > to everything even a rel/relup compiler. The reality is that we need a > > bunch of stuff in addition to whats in the so having that in this spec > > and making the rel a compile out put would work. Though we loose the > > completely vanilla otpishness of some of the tools.
> > Tim Watson wrote:
> >> I was just starting to think I needed to reconsider my position in > >> light of the 'lots of little tools' approach - not be so monolithic in > >> my perception of the process and all that - and it occurs to me that > >> if some of our outputs are lists of file system paths we might be ok. > >> I come on here to suggest it and voila, you've read my mind.
> >> On 1 March 2012 21:01, Eric B Merritt <ericbmerr...@gmail.com> wrote:
> >>> I think you have talked me into this. The only caveat to the list > >>> below is that I think we should have a simple tool that takes a list > >>> of paths and copies the OTP apps to a directory.
> >> I completely agree with that.
> >>> It would be cool if each of these could read their inputs on standard > >>> in and write the outputs to standard out as well.
> >> Yes definitely. I'm almost inclined to say that should be the required > >> unix-y interface, so that people who're using Make (and similar tools) > >> can still get value out of each and every one of these tools as they > >> see fit.
> >>> **manager**:
> >>> Inputs: a list of repos? > >>> outputs: local index + basic local repo structure
> >> Inputs: you mean a list of 'remote indexes' here right? The word > >> 'repo' is overloaded and implies storage more than anything else. But > >> yes, I think a list of repos is fine.
> >> Flags: -f <filename> read the list of repos from <filename> instead of stdin?
> >> Question: how are the repos specified? As a URL or as something else?
> >> Question: this guy is presumably not writing to stdout, as it is > >> creating directories on disk etc. So perhaps the stdout stream should > >> contain the root directories and/or locations of these?
> >>> **solver**:
> >>> Input: Constraints are by their nature not OTP but we should try to > >>> fit them in as seamlessly as possible
> >>> Output: Should be a rel or relup file with organization annotation
> >> Yes that output sounds very reasonable. This is the area we've spent > >> the least time thinking about the inputs for.
> >>> **fetcher**: > >>> Inputs: a rel or relup file > >>> Outputs: A list of code paths for the various dependencies
> >> Absolutely - this is a great idea. What about where you're in an OTP > >> application/library project and want to fetch your dependencies? > >> Should we still work off a rel file out of the solver? I guess it > >> doesn't matter that the rel/relup file is not needed, as we can still > >> use it as an artefact in our process if we want to.
> >>> **builder**: > >>> Inputs: A project that contains a rel+/relup file, a list of code paths for dependencies > >>> Outputs: A built otp application(s) + updated list of codepaths
> >> Cool, I like this. As we've discussed before though, not every project > >> is a release - sometimes it's just an application or a library. We > >> need to support those use cases as well, so the rel/+relup can't be > >> mandatory.
> >>> **assembler**: > >>> Inputs: A rel/+relup, A list of codepaths > >>> Outputs: An OTP Release directory structure
> >> Ok I'm fine with these. Also the packager needs to support packaging > >> plain applications/libraries as well as releases.
> >> **publisher** > >>> Inputs: A dist tarball and choice of index to publish to + the users personal configuration (passwords, etc) > >>> Outputs-1: an HTTP request to github/bitbucket/nexus/etc to upload/store the publication metadata in an index > >>> Outputs-2: an HTTP request to github/bitbucket/nexus/etc to upload/store the binary artefact online
> At Sat, 3 Mar 2012 18:29:48 +0000, > Tim Watson wrote:
>> I'm not sure I fully understand what this means, some of the tools >> don't care about the list of code paths. I suppose each of the tools >> cares about a list of something (repositories, dependencies, code >> paths, etc).
> I was thinking about just those tools that take code paths. That is > everything but the manager and packager really.
>> I think we are going to end up passing around data that >> isn't pure OTP (app and rel files) anyway. If the OTP metadata was >> rich enough to describe all of what we're trying to achieve, we >> wouldn't have a job to do because it'd already be written.
> I am not sure thats the case. The OTP guys never tackled and so simply > never addressed these issues. That doesn't mean the OTP metadata isn't > sufficient.
> It also does not mean that we should either, but I would like to avoid > redundancies if possible.
>> I have some outstanding questions about configuration. Where do the >> dependency lists go?In the .app file?
> I suspect thats where they would go by default. Though I suspect we > need to be able to specify them directly as well.
>> Doesn't adding the ">=" and >> version number screw other tools up?
> Yup. It would screw up everything. However, we can add another tuple > there without a problem and that is the approach I would take. That > extra tuple would give us the information we would need.
And that extra tuple doesn't screw things up? Baring in mind that the design decision to break up the tool chain into lots of little chunks is as much about giving people choice as any aesthetic judgement, I'm assuming there may well be a class of users who continue to use reltool, opting only to utilise the resolver and one or two other bits. So we can't even break reltool compatibility, and the requirement to not break existing code is tough to meet given the amount of tools coverage we'd need to deal with.
>> I know that my 'appstart' >> utility will puke, although I'll happily change it if it's the only >> thing that does.
> Again none of this is set in stone. We could also stick it in a > separate file in priv or any number of things.
For my reason above - not risking breaking any existing code in OTP or elsewhere - I prefer the option of a separate file, despite the obvious redundancy. It is a safe decision: we will know for sure we are not going to clash with anything else.
>>> At Thu, 1 Mar 2012 22:10:05 +0000, >>> top posting again.
>>> I just realized that we could turn our 'list of code paths' to a >>> viable source in and of itself and then use that is inputs and outputs >>> to everything even a rel/relup compiler. The reality is that we need a >>> bunch of stuff in addition to whats in the so having that in this spec >>> and making the rel a compile out put would work. Though we loose the >>> completely vanilla otpishness of some of the tools.
>>> Tim Watson wrote:
>>>> I was just starting to think I needed to reconsider my position in >>>> light of the 'lots of little tools' approach - not be so monolithic in >>>> my perception of the process and all that - and it occurs to me that >>>> if some of our outputs are lists of file system paths we might be ok. >>>> I come on here to suggest it and voila, you've read my mind.
>>>> On 1 March 2012 21:01, Eric B Merritt <ericbmerr...@gmail.com> wrote:
>>>>> I think you have talked me into this. The only caveat to the list >>>>> below is that I think we should have a simple tool that takes a list >>>>> of paths and copies the OTP apps to a directory.
>>>> I completely agree with that.
>>>>> It would be cool if each of these could read their inputs on standard >>>>> in and write the outputs to standard out as well.
>>>> Yes definitely. I'm almost inclined to say that should be the required >>>> unix-y interface, so that people who're using Make (and similar tools) >>>> can still get value out of each and every one of these tools as they >>>> see fit.
>>>>> **manager**:
>>>>> Inputs: a list of repos? >>>>> outputs: local index + basic local repo structure
>>>> Inputs: you mean a list of 'remote indexes' here right? The word >>>> 'repo' is overloaded and implies storage more than anything else. But >>>> yes, I think a list of repos is fine.
>>>> Flags: -f <filename> read the list of repos from <filename> instead of stdin?
>>>> Question: how are the repos specified? As a URL or as something else?
>>>> Question: this guy is presumably not writing to stdout, as it is >>>> creating directories on disk etc. So perhaps the stdout stream should >>>> contain the root directories and/or locations of these?
>>>>> **solver**:
>>>>> Input: Constraints are by their nature not OTP but we should try to >>>>> fit them in as seamlessly as possible
>>>>> Output: Should be a rel or relup file with organization annotation
>>>> Yes that output sounds very reasonable. This is the area we've spent >>>> the least time thinking about the inputs for.
>>>>> **fetcher**: >>>>> Inputs: a rel or relup file >>>>> Outputs: A list of code paths for the various dependencies
>>>> Absolutely - this is a great idea. What about where you're in an OTP >>>> application/library project and want to fetch your dependencies? >>>> Should we still work off a rel file out of the solver? I guess it >>>> doesn't matter that the rel/relup file is not needed, as we can still >>>> use it as an artefact in our process if we want to.
>>>>> **builder**: >>>>> Inputs: A project that contains a rel+/relup file, a list of code paths for dependencies >>>>> Outputs: A built otp application(s) + updated list of codepaths
>>>> Cool, I like this. As we've discussed before though, not every project >>>> is a release - sometimes it's just an application or a library. We >>>> need to support those use cases as well, so the rel/+relup can't be >>>> mandatory.
>>>>> **assembler**: >>>>> Inputs: A rel/+relup, A list of codepaths >>>>> Outputs: An OTP Release directory structure
>>>>> **packager**: >>>>> Inputs: An OTP Release directory structure >>>>> Outputs: A dist tarball (debian package, rpm etc)
>>>> Ok I'm fine with these. Also the packager needs to support packaging >>>> plain applications/libraries as well as releases.
>>>> **publisher** >>>>> Inputs: A dist tarball and choice of index to publish to + the users personal configuration (passwords, etc) >>>>> Outputs-1: an HTTP request to github/bitbucket/nexus/etc to upload/store the publication metadata in an index >>>>> Outputs-2: an HTTP request to github/bitbucket/nexus/etc to upload/store the binary artefact online
> >> Doesn't adding the ">=" and > >> version number screw other tools up?
> > Yup. It would screw up everything. However, we can add another tuple > > there without a problem and that is the approach I would take. That > > extra tuple would give us the information we would need.
> And that extra tuple doesn't screw things up?
I have been using it for a long time in sinan and it doesn't break anything at all.
> Baring in mind that the design decision to break up the tool chain > into lots of little chunks is as much about giving people choice as > any aesthetic judgement, I'm assuming there may well be a class of > users who continue to use reltool, opting only to utilise the > resolver and one or two other bits. So we can't even break reltool > compatibility, and the requirement to not break existing code is > tough to meet given the amount of tools coverage we'd need to deal > with.
I have never tested this with reltool. However, I suspect it will work. The fact that it is a association list usually implies that things only pay attention to the key value pairs they care about.
However, I agree 100% that we should not break compatibility and need to check this.
> >> I know that my 'appstart' > >> utility will puke, although I'll happily change it if it's the only > >> thing that does.
> > Again none of this is set in stone. We could also stick it in a > > separate file in priv or any number of things.
> For my reason above - not risking breaking any existing code in OTP > or elsewhere - I prefer the option of a separate file, despite the > obvious redundancy. It is a safe decision: we will know for sure we > are not going to clash with anything else.
I am ok with that. We can do the same thing for releases. Lets document it will though and make it trivially parsable in erlang. Basically as an erlang term file.
The only downside is it will not be accessible from escript archives.
Sorry about top posting - using someone else's machine. Maybe putting it in priv/ would be a good idea, just so that it can be included in escripts. IIRC Ulf has a patch for OTP (erl_prim_loader) that fixes the accessibility of files in .ez archives. For escripts it's another story though - but that's another thing we'll need to think about whether we support or not - bundled executable escripts as a packaging type!?
On 05/03/2012, Eric B Merritt <ericbmerr...@gmail.com> wrote:
> At Sat, 3 Mar 2012 23:59:59 +0000, > Tim Watson wrote:
>> >> Doesn't adding the ">=" and >> >> version number screw other tools up?
>> > Yup. It would screw up everything. However, we can add another tuple >> > there without a problem and that is the approach I would take. That >> > extra tuple would give us the information we would need.
>> And that extra tuple doesn't screw things up?
> I have been using it for a long time in sinan and it doesn't break > anything at all.
>> Baring in mind that the design decision to break up the tool chain >> into lots of little chunks is as much about giving people choice as >> any aesthetic judgement, I'm assuming there may well be a class of >> users who continue to use reltool, opting only to utilise the >> resolver and one or two other bits. So we can't even break reltool >> compatibility, and the requirement to not break existing code is >> tough to meet given the amount of tools coverage we'd need to deal >> with.
> I have never tested this with reltool. However, I suspect it will > work. The fact that it is a association list usually implies that > things only pay attention to the key value pairs they care about.
> However, I agree 100% that we should not break compatibility and need > to check this.
>> >> I know that my 'appstart' >> >> utility will puke, although I'll happily change it if it's the only >> >> thing that does.
>> > Again none of this is set in stone. We could also stick it in a >> > separate file in priv or any number of things.
>> For my reason above - not risking breaking any existing code in OTP >> or elsewhere - I prefer the option of a separate file, despite the >> obvious redundancy. It is a safe decision: we will know for sure we >> are not going to clash with anything else.
> I am ok with that. We can do the same thing for releases. Lets > document it will though and make it trivially parsable in > erlang. Basically as an erlang term file.
> The only downside is it will not be accessible from escript archives.
On Mon, Mar 5, 2012 at 2:26 PM, Tim Watson <watson.timo...@gmail.com> wrote: > Sorry about top posting - using someone else's machine. Maybe putting > it in priv/ would be a good idea, just so that it can be included in > escripts.
Would be or would *not* be? I am a bit confused.
> IIRC Ulf has a patch for OTP (erl_prim_loader) that fixes > the accessibility of files in .ez archives. For escripts it's another > story though - but that's another thing we'll need to think about > whether we support or not - bundled executable escripts as a packaging > type!?
To be fair *app files are not available in escript archives either.
Hmmm, I think we are probably going to have to support escripts and come up with some way to spec dependencies there too.
So for apps, something in the priv dir. for releases something somewhere else. same for rel files. we just have to define something else.
> On 05/03/2012, Eric B Merritt <ericbmerr...@gmail.com> wrote: >> At Sat, 3 Mar 2012 23:59:59 +0000, >> Tim Watson wrote:
>>> >> Doesn't adding the ">=" and >>> >> version number screw other tools up?
>>> > Yup. It would screw up everything. However, we can add another tuple >>> > there without a problem and that is the approach I would take. That >>> > extra tuple would give us the information we would need.
>>> And that extra tuple doesn't screw things up?
>> I have been using it for a long time in sinan and it doesn't break >> anything at all.
>>> Baring in mind that the design decision to break up the tool chain >>> into lots of little chunks is as much about giving people choice as >>> any aesthetic judgement, I'm assuming there may well be a class of >>> users who continue to use reltool, opting only to utilise the >>> resolver and one or two other bits. So we can't even break reltool >>> compatibility, and the requirement to not break existing code is >>> tough to meet given the amount of tools coverage we'd need to deal >>> with.
>> I have never tested this with reltool. However, I suspect it will >> work. The fact that it is a association list usually implies that >> things only pay attention to the key value pairs they care about.
>> However, I agree 100% that we should not break compatibility and need >> to check this.
>>> >> I know that my 'appstart' >>> >> utility will puke, although I'll happily change it if it's the only >>> >> thing that does.
>>> > Again none of this is set in stone. We could also stick it in a >>> > separate file in priv or any number of things.
>>> For my reason above - not risking breaking any existing code in OTP >>> or elsewhere - I prefer the option of a separate file, despite the >>> obvious redundancy. It is a safe decision: we will know for sure we >>> are not going to clash with anything else.
>> I am ok with that. We can do the same thing for releases. Lets >> document it will though and make it trivially parsable in >> erlang. Basically as an erlang term file.
>> The only downside is it will not be accessible from escript archives.
Yes - really not sure what to do with binary escripts. Rebar handles fetching files out of ./priv in its escript by running escript:extract/2 and then calling zip:foldl/3, which works ok.
On 5 Mar 2012, at 20:39, Eric Merritt wrote:
> On Mon, Mar 5, 2012 at 2:26 PM, Tim Watson <watson.timo...@gmail.com> wrote: >> Sorry about top posting - using someone else's machine. Maybe putting >> it in priv/ would be a good idea, just so that it can be included in >> escripts.
> Would be or would *not* be? I am a bit confused.
I guess I'm thinking it might be a good thing - I'm not really sure. I'm torn between wanted to keep the dependency config separate and at the project top level for a cleaner looking build environment, and wanting to obey OTP habbits in the distribution by putting the config into either the app file or the priv directory.
>> IIRC Ulf has a patch for OTP (erl_prim_loader) that fixes >> the accessibility of files in .ez archives. For escripts it's another >> story though - but that's another thing we'll need to think about >> whether we support or not - bundled executable escripts as a packaging >> type!?
> To be fair *app files are not available in escript archives either.
Really? I didn't know that.
> Hmmm, I think we are probably going to have to support escripts and > come up with some way to spec dependencies there too.
Yes.
> So for apps, something in the priv dir. for releases something > somewhere else. same for rel files. we just have to define something > else.
On Mon, Mar 5, 2012 at 4:58 PM, Tim Watson <watson.timo...@gmail.com> wrote: > Yes - really not sure what to do with binary escripts. Rebar handles fetching files out of ./priv in its escript by running escript:extract/2 and then calling zip:foldl/3, which works ok.
I suspect we shoud treat escripts as on output. That is they are something the packager produces but are not stored in the system for use as dependencies. That would solve the problem.
> On 5 Mar 2012, at 20:39, Eric Merritt wrote:
>> On Mon, Mar 5, 2012 at 2:26 PM, Tim Watson <watson.timo...@gmail.com> wrote: >>> Sorry about top posting - using someone else's machine. Maybe putting >>> it in priv/ would be a good idea, just so that it can be included in >>> escripts.
>> Would be or would *not* be? I am a bit confused.
> I guess I'm thinking it might be a good thing - I'm not really sure. I'm torn between wanted to keep the dependency config separate and at the project top level for a cleaner looking build environment, and wanting to obey OTP habbits in the distribution by putting the config into either the app file or the priv directory.
well the dependency for an app cant be at the project top level simply because a project can mean different things to different tools. For me a project is often several related OTP apps. If you mean at the OTP app top level I probably wouldn't want it there simply from an organizational standpoint.
>>> IIRC Ulf has a patch for OTP (erl_prim_loader) that fixes >>> the accessibility of files in .ez archives. For escripts it's another >>> story though - but that's another thing we'll need to think about >>> whether we support or not - bundled executable escripts as a packaging >>> type!?
>> To be fair *app files are not available in escript archives either.
> Really? I didn't know that.
>> Hmmm, I think we are probably going to have to support escripts and >> come up with some way to spec dependencies there too.
> Yes.
>> So for apps, something in the priv dir. for releases something >> somewhere else. same for rel files. we just have to define something >> else.
