The Collaboration Wiki for Packaging
Eric Merritt
Its just a start but it should give us a base move forward.
The goal here is to get something we can present to erlang-questions
for feedback, and alternatively use as a very high level spec for
requirements.
you can use emacs (or lesser editors ;) by directly accessing it via git
g...@github.com:erlware/erlware.github.com.wiki.git
Tim Watson
Eric Merritt
On the wiki. Lets send questions to this list and just update the wiki
directly once we come to some conclusion or just update the wiki if
you are comfortable with that.
> Question(s):
>
> do we want to tie ourselves to git+github in the spec?
> My thoughts are that people may use any mechanism they like
> to build alternative repository implementations (which we should
> support via a plugins mechanism) - not everyone uses git
>
> In light of this, can we rephrase item 2, as
> "Repositories will initially be stored in git and targeted at github"?
I think in version on we do target git and github. Down the road we
will have other tasks to branch that out. However, I think having a
hard target initially is a good thing.
This is not a hard and fast requirement for me though.
> Question:
>
> is this really optional? I certain want the tools that *we*
> build to do this, as I don't want to download <package-x>-1.0.1 every
> time I use it.
In my mind it is. I have a lot of bandwidth and I am nearly always
connected. I would rather just download the things I need when I need
them rather then have some 3rd location that I have to know about and
maintain. For me its a trade off of bandwidth for increased
simplicity.
I probably wont have a global anything, no global index no local repo
etc. It will all be stored on a project by project basis. One of the
reasons that I want to have command line options for everything so so
that I can do that. That removes things hanging out it some global
directory isolates problems to a single project and makes it very easy
to clean up. With that model in mind a local repo is very redundant
and actually mostly just adds unneeded overhead. Disk space is cheap,
band with is cheap I want to leverage both.
> I'm not convinced this is really an optimization rather than the only
> sane way to do things. Think about a build plugin for example, or a
> logging library that you use in all your applications/systems. Do you
> really want to resolve this to some local app directory each time you
> start a new project?
Yes actually.
> This is what rebar does with dependencies that I
> actually hate - the local repository (along with a good index and binary
> artefacts) is part of the main reason I want to do this stuff.
Its one of the few things that I think rebar got right. I like things
to be self contained. That maybe why I like the rox and osx
application model so much. Its also the reason I distribute my otp
releases with everything the need. Its a space trade off that I think
is well worth it.
We might be able to do this transparently in something like ~/.repo
but I absolutely want the option to turn it off.
> Question:
>
> I agree that we need the namespace flattening, but I'm
> wondering why we need do implement it in this way. On windows, for
> example, this is a royal pain as there are no symlinks.
I wasn't planning to use symlinks at all.
> If we make the local repository a core part of the solution, then
> there is another option which IMO is viable.
>
> All the use-cases for the dependencies involve getting a running
> emulator to 'know about' them, for example compile time ones (such
> as parse transforms and build plugins) need to be on the code path
> during the 'build tool' operations. Runtime ones need to be available
> during testing, local development and pacakging a release. Test scoped
> ones need to be on the path during testing. In all cases, the following
> things need to work
>
> $ code:which(AModuleInTheDependency) /= non_existing
> $ code:lib_dir(TheAppName) /= {error,bad_name}
>
> -include_lib("dependency/include/header.hrl").
>
> So why not just add the correct directories to the code path at the
> appropriate time? This saves copying (or symlinking) and avoids
> cluttering the local directory structure.
lol, I think we have very different outlooks here. We are going to
need to resolve that.
> It also avoid having to
> manage an 'app dir' when dependencies are removed for example.
nuking the directory?
> Personally I find this approach preferable, as long as the code
> path modification process is documented clearly.
This is doable. There are two things I dont like about it. The first
and biggest is that its much more difficult to plug in arbitrary tools
that are not part of the chain. That is things that use the packages
now need to know about this non-erlang special repo structure that we
have. This may actualy not be a problem, but we need to make sure that
none of the packaging semantics make it into running code. The process
being as vanilla as possible I think is important.
On dependencies.
I am not sure what you intend for 'build-support'. I guess that is the
tools to do the compile themselves. I am not sure how that can be
worked into the process.
Also on packaging. At the moment we are not allowing user code to run
as packaging. that is the packaging tools are complete unto
themselves. Well better said that, if you are doing packaging the
packaging tool is already there. I may be missing something here.
Tim Watson
> Tim,
>
> On the wiki. Lets send questions to this list and just update the wiki
> directly once we come to some conclusion or just update the wiki if
> you are comfortable with that.
>
Sure I wasn't sure which model to go for, but from now on I'll only
post updates to the wiki and keep discusions to the list. That's
actually cleaner anyway, from a random visitor's point of view not to
see lots of questions inline.
>> Question(s):
>>
>> do we want to tie ourselves to git+github in the spec?
>> My thoughts are that people may use any mechanism they like
>> to build alternative repository implementations (which we should
>> support via a plugins mechanism) - not everyone uses git
>>
>> In light of this, can we rephrase item 2, as
>> "Repositories will initially be stored in git and targeted at github"?
>
>
> I think in version on we do target git and github. Down the road we
> will have other tasks to branch that out. However, I think having a
> hard target initially is a good thing.
>
> This is not a hard and fast requirement for me though.
>
I am fine with it being github initially, but for work there are some
cases where I can't publish internal/private metadata to github
because it is offsite. We are a former state run monopoly and the
largest telecoms outfit in the UK, so we not only have a lot of rules
around privacy but also lots of government as well as industry
regulation that we have to comply with. In some cases, even the notion
that an artefact exists must not reside outside of the corporate LAN,
for legal reasons.
I will write an alternative remote index handler that doesn't use
github (based on subversion or nexus probably) but I will need a way
to wire this alternative module in - that's what build-support
dependencies are for. More on this later.
>> Question:
>>
>> is this really optional? I certain want the tools that *we*
>> build to do this, as I don't want to download <package-x>-1.0.1 every
>> time I use it.
>
> In my mind it is. I have a lot of bandwidth and I am nearly always
> connected. I would rather just download the things I need when I need
> them rather then have some 3rd location that I have to know about and
> maintain. For me its a trade off of bandwidth for increased
> simplicity.
>
Yes ok, but a few points here:
1. you don't have to manage anything yourself, the tool does it for you
2. once you've already downloaded something, why do you want to do it
again and again, etc?
We're both looking at the local repo as a cache, and caching downloads
so you don't have to repeatedly get them is simple and obvious. If the
local index has support for marking items as having already been
downloaded - perhaps simply by putting a file://<path> url next to the
item - then everything else should be able to remain blissfully
ignorant of the local cache.
> I probably wont have a global anything, no global index no local repo
> etc. It will all be stored on a project by project basis. One of the
> reasons that I want to have command line options for everything so so
> that I can do that. That removes things hanging out it some global
> directory isolates problems to a single project and makes it very easy
> to clean up. With that model in mind a local repo is very redundant
> and actually mostly just adds unneeded overhead. Disk space is cheap,
> band with is cheap I want to leverage both.
>
I really don't like the idea of having an index for every project. I
cannot think of any other tool that does this, not any of the
apt/yum/etc package managers, not any of the varietous build tools
(except autotools, which makes most people go insane anyway).
That said, if you're happy for us to provide both options via command
line flags (or user defined configuration files) then I have no issues
with it if you want to re-download the index for every project you're
working on.
>> I'm not convinced this is really an optimization rather than the only
>> sane way to do things. Think about a build plugin for example, or a
>> logging library that you use in all your applications/systems. Do you
>> really want to resolve this to some local app directory each time you
>> start a new project?
>
> Yes actually.
>
This is fair enough. The designers of rebar think the same way and I
guess can understand the benefits of isolation. I don't want it for
myself but if we're going to provide both options then we should
hopefully be able to meet the needs of a wider audience anyway.
>> This is what rebar does with dependencies that I
>> actually hate - the local repository (along with a good index and binary
>> artefacts) is part of the main reason I want to do this stuff.
>
> Its one of the few things that I think rebar got right. I like things
> to be self contained. That maybe why I like the rox and osx
> application model so much. Its also the reason I distribute my otp
> releases with everything the need. Its a space trade off that I think
> is well worth it.
>
Which OSX applications re-bundle libraries that are used in multiple
places each time? Or don't get installed into either a global or
user-wide directory? Or fail to put commonly used things into a
standard and common Framework directory? I see what you're getting at,
but there are numerous angles, an I suppose the way in which people
perceive and/or interpret the principles is where the split tends to
occur.
I agree with the principle of having things self contained, but I
don't see this as an important example of self-containment. I
completely agree with you about releases needing to contain everything
you need - that is an example that stands out to me. Having said that,
we've already agreed that in order to make our lives 'manageable' we
are not going to bundle runtime OTP libraries in packaged releases, so
clearly 'self-containment' isn't an overriding principle in the face
of other forces.
> We might be able to do this transparently in something like ~/.repo
> but I absolutely want the option to turn it off.
>
I am absolutely fine with people being able to choose between using a
project local app dir (and index) and having the dependencies resolved
from the local repo. I am also absolutely fine with writing the local
repo handling code myself, as long as I have an API to integrate it
with the assembler/packager. I want to make this endevour
1. easy for you and I as well as other potential contributors to work on
2. as simple to understand as possible, inside and out
3. as configurable and extensible as possible (without violating 1 and
2 greatly)
>> Question:
>>
>> I agree that we need the namespace flattening, but I'm
>> wondering why we need do implement it in this way. On windows, for
>> example, this is a royal pain as there are no symlinks.
>
> I wasn't planning to use symlinks at all.
>
Fair enough. I tend to err on the side of performance, and my dislike
of copying files and directories around (and/or repeatedly fetching
things from the internel) is not about space usage or redundancy, it's
about making sure that my build+test cycle takes seconds and not
minutes. Bare in mind that for me, a full integration test requires
that the release is packaged and deployed to a VM before the end-2-end
test suites (common_test and other things besides) are run, and most
CI servers do this with a fresh checkout from version control, not a
pull/update - this means that if everything has to be fetched and
copied and so on, the build can take 10 minutes which is a very long
lapse to then discover that you've broken someone else's code.
>> If we make the local repository a core part of the solution, then
>> there is another option which IMO is viable.
>>
>> All the use-cases for the dependencies involve getting a running
>> emulator to 'know about' them, for example compile time ones (such
>> as parse transforms and build plugins) need to be on the code path
>> during the 'build tool' operations. Runtime ones need to be available
>> during testing, local development and pacakging a release. Test scoped
>> ones need to be on the path during testing. In all cases, the following
>> things need to work
>>
>> $ code:which(AModuleInTheDependency) /= non_existing
>> $ code:lib_dir(TheAppName) /= {error,bad_name}
>>
>> -include_lib("dependency/include/header.hrl").
>>
>> So why not just add the correct directories to the code path at the
>> appropriate time? This saves copying (or symlinking) and avoids
>> cluttering the local directory structure.
>
> lol, I think we have very different outlooks here. We are going to
> need to resolve that.
>
Yes I can see that! :)
I'm not worried about resolving it - I will go along with implementing
your preferred method(s) as long as there are extensibility points for
me to exploit so I can implement my alternative approaches re global
local repository and index handling. I don't think you're wrong, I
just think people have different opionions about this (mainly based on
the tools they're familiar and comfortable with) and having both
options strikes me as a good thing.
>> It also avoid having to
>> manage an 'app dir' when dependencies are removed for example.
>
> nuking the directory?
>
Yes you're right - that's the simplest way to refresh the dependencies
whenever anything other than an insert takes place in the dependency
graph. I guess we've made an implementation decision for the app-dir
then.
>> Personally I find this approach preferable, as long as the code
>> path modification process is documented clearly.
>
> This is doable. There are two things I dont like about it. The first
> and biggest is that its much more difficult to plug in arbitrary tools
> that are not part of the chain. That is things that use the packages
> now need to know about this non-erlang special repo structure that we
> have. This may actualy not be a problem, but we need to make sure that
> none of the packaging semantics make it into running code. The process
> being as vanilla as possible I think is important.
>
Let me just be really clear. When the packaging takes place, I expect
the applications and libraries which are listed as dependencies to be
copied into the 'lib' directory inside the release folder, just the
same as you.
I agree that providing a seamless API for working with both 'styles'
is going to be a bit more complex. I am spending a fair bit of time
thinking about what underlying abstractions can support both these
processes, whilst fitting nicely into a unix toolchain phiolosophy. I
think we can do it, it'll just take a bit of thinking about. I propose
that we decide on the API(s) between the tools for the general case,
where there is an app-dir, and then I'll think about how I can expose
the local repository paths that matter in a fashion that suits this
API.
> On dependencies.
>
> I am not sure what you intend for 'build-support'. I guess that is the
> tools to do the compile themselves. I am not sure how that can be
> worked into the process.
>
No that's not so much it. Think about it like this - I want to write a
custom 'remote index' handler that'll resolve/fetch the remote index
from subversion or nexus instead of github. The default remote index
handler is a module which is probably called something like
'erlware_remote_index' and is on the code path because it is
distributed with the tool (being the default handler). I want to put
my 'bt_nexus_remote_index' module into an OTP application and resolve
it just like any other dependency. Once it is resolved (via the
default 'remote index' on github) I want it downloaded and then added
to the code path. I want this to happen before any further
dependencies are resolved, because I want this 'extension' to kick in
and add its own index metadata so that private artefacts which are
hosted internally on nexus can be resolved alongside public open
source components hosted on github/bitbucket.
Another example is resolving a rebar plugin. Rebar plugins need to be
downloaded and put on the code path before rebar runs (or before the
rebar commands they add/exploit are run) and this often needs to
happen way before 'compile' is executed. Now in both those cases, the
dependencies are 'before compile' and might also be used 'during
compile' but in either case they're not required (i.e., included) when
I assemble a release.
Hope that makes sense.
> Also on packaging. At the moment we are not allowing user code to run
> as packaging. that is the packaging tools are complete unto
> themselves. Well better said that, if you are doing packaging the
> packaging tool is already there. I may be missing something here.
>
>
>
I've not explained these clearly, which is poor form on my part. I've
also failed, in my rather shabby analysis of this, to delineate the
scope of dependencies from the packaging types. Let me try again.
There are, in my mind, the following scopes for a dependency:
1. things that extend the erlware tool (or toolset - possibly even the
running code that fetches them) like custom indexes and/or publishers
2. things that extend the functionality of the build tool (whether it
is sinan or rebar or make or whatever)
3. things that are required during compilation (such as parse transforms)
4. things that are required during testing only
5. things that are required at runtime (not just for testing)
6. things that are required at runtime but not for compile or test (I
suspect this scope can be ignored as we can build it from the others)
Now I've based these scopes on the different ways in which the
dependency is being used. In each case, I'm asking myself 'does it
need to be available to do X or not'. When one does this in rebar,
it's almost irrelevant because to generate a release using rebar you
have to have reltool.config which specifies which applications are to
be included. I am working on the basis that we will replace reltool
with our own 'assembly' implementation, which I'm assuming will scan
the dependencies to figure out what to include in the release. Based
on this, my thoughts are that
- I have things like 'bt_nexus_remote_index' which I need on the path
to resolve dependencies via nexus, but not included in the release
- I have things like 'esl/parse_trans' which I need on the path to
compile, but not included in the release
- I have things like 'rebar_cucumber_plugin' which I need on the path
to run tests, but not included in the release
- I have things like PropEr and hamcrest-erlang and cucumberl that I
want on the path for testing, but not included in the release
- I have libraries which need to be available for testing and are also
included in the release (making these compile time would solve it)
- I have libraries which need to be available only when
building/packaging the release (for example, esl/setup or erlctrl)
- I have things like 'bt_nexus_artefact_publisher' which I need on the
path to publish, but have nothing to do with the release
So maybe based on this, having a 'packaging' scope isn't really
necessary, as you can make things 'compile' scope to achieve the same
effect. The 'build-support' scope is for being able to differentiate
between dependencies such as 'bt_nexus_remote_index' which are
required for everything to work. It is a poorly chosen term, as it
isn't necessarily just 'build' that the dependency is supporting - it
is 'resolve-dependencies' scoped but shouldn't be included in the
release. Because 'bt_nexus_artefact_publisher' is 'publish' scoped I
figured I'd combine the two, but it doesn't matter either way.
Hopefully what you'll have picked up from this is that I want to be
able to write customisations as separate libraries/applications and
include then as dependencies. This is because
a. I don't like bundling half a million extensions with the main tool sources
b. I don't like having to fork a project when I've built an extension
that the maintainers may or may not want to include
c. I don't like having to manage huge pull request queues for my
project when lots of loonies (like me!) get overexcited about
customising
Again I'm coloured by many years of experience with maven. It resolves
plugins on the fly and downloads them in ~/.m2/repository/... and then
makes them available during the build. All you need to do is declare
the plugin (and sometimes the repository it resides in if it's not a
standard one) and maven does the rest. This makes adding scala or
clojure or groovy of even erlang support to your maven build, very
simple - just configure the project xml and run 'mvn clean compile'
and everything gets downloaded (just once) and then magically works
and your source code (and project folders) remain completely
unaltered.
Because I've become accustomed to this philosophy, wherein compile,
test and runtime 'artefacts' are completely managed for me, I'm
slightly OCD about being forced to deal with them when using rebar.
Even adding rebar's 'deps' app-dir to .gitignore for every project
drives me over the edge.
As maven mainly 'just works' I'd probably use the maven-erlang-plugin
for my open source projects but the reality is that many Erlangers
aren't going to install the JVM just to build my sources and I don't
want to maintain multiple configurations, so I really want to make
rebar do things the way I like them. That's why I initially wrote
https:/github.com/hyperthunk/rebar_alt_deps and later decided to
externalise the dependency management was better. This multi-tool idea
is even better still, as people can use it from sinan and make and
whatever else, but ultimately my end goal is for a user to download my
project and run
$ rebar clean package
and for all the dependencies to get properly resolved and downloaded
at the right time and for the entire tool chain to just work without
further intervention and for it to be blazingly fast from then on when
the project(s) are in CI.
Tim Watson
covering them in this thread! :)
Eric B Merritt
My thoughts about this changed as I moved through the reply. So you
might want to read the whole thing before replying inline.
At Thu, 1 Mar 2012 19:54:49 +0000,
Understandable. This is going to be a bit of a chicken and the egg
problem. If you need the package manager to manage packages but you
need the packages to get the package manager what do you do?
In this model I think we have to assume that the package manager is
not available. I dont see how it could be. If that is the case then it
still obviates the need for that class of packages.
>
> >> Question:
> >>
> >> is this really optional? I certain want the tools that *we*
> >> build to do this, as I don't want to download <package-x>-1.0.1 every
> >> time I use it.
> >
> > In my mind it is. I have a lot of bandwidth and I am nearly always
> > connected. I would rather just download the things I need when I need
> > them rather then have some 3rd location that I have to know about and
> > maintain. For me its a trade off of bandwidth for increased
> > simplicity.
> >
>
> Yes ok, but a few points here:
>
> 1. you don't have to manage anything yourself, the tool does it for you
this never quite seems to be the reality of things.
> 2. once you've already downloaded something, why do you want to do it
> again and again, etc?
Because downloading it again is simpler and less complex then keeping
it around. (this is an assumption that is worth questioning in the
same way that having a cache is an assumption worth questioning).
>
> We're both looking at the local repo as a cache, and caching downloads
> so you don't have to repeatedly get them is simple and obvious. If the
> local index has support for marking items as having already been
> downloaded - perhaps simply by putting a file://<path> url next to the
> item - then everything else should be able to remain blissfully
> ignorant of the local cache.
If we go this route we should probably just check the local cache to
see if its been downloaded. That is the true canonical location for
that information. No need for additional metadata in the index.
>
> > I probably wont have a global anything, no global index no local repo
> > etc. It will all be stored on a project by project basis. One of the
> > reasons that I want to have command line options for everything so so
> > that I can do that. That removes things hanging out it some global
> > directory isolates problems to a single project and makes it very easy
> > to clean up. With that model in mind a local repo is very redundant
> > and actually mostly just adds unneeded overhead. Disk space is cheap,
> > band with is cheap I want to leverage both.
> >
>
> I really don't like the idea of having an index for every project. I
> cannot think of any other tool that does this, not any of the
> apt/yum/etc package managers, not any of the varietous build tools
> (except autotools, which makes most people go insane anyway).
Of course not. they are system package managers not build or language
package managers. There is a very big difference. I can think of a
couple of language package managers that do do this though.
> That said, if you're happy for us to provide both options via command
> line flags (or user defined configuration files) then I have no issues
> with it if you want to re-download the index for every project you're
> working on.
Thats just the point. Providing the command line flags cost us very
little and add enough flexibility to the process to meet everyones
needs.
>
> >> I'm not convinced this is really an optimization rather than the only
> >> sane way to do things. Think about a build plugin for example, or a
> >> logging library that you use in all your applications/systems. Do you
> >> really want to resolve this to some local app directory each time you
> >> start a new project?
> >
> > Yes actually.
> >
>
> This is fair enough. The designers of rebar think the same way and I
> guess can understand the benefits of isolation. I don't want it for
> myself but if we're going to provide both options then we should
> hopefully be able to meet the needs of a wider audience anyway.
Again, we should be able to support both really. Even having a
non-optional local cache doesn't change that. For me its just a
redundant extra step.
> >> This is what rebar does with dependencies that I
> >> actually hate - the local repository (along with a good index and binary
> >> artefacts) is part of the main reason I want to do this stuff.
> >
> > Its one of the few things that I think rebar got right. I like things
> > to be self contained. That maybe why I like the rox and osx
> > application model so much. Its also the reason I distribute my otp
> > releases with everything the need. Its a space trade off that I think
> > is well worth it.
> >
>
> Which OSX applications re-bundle libraries that are used in multiple
> places each time?
I should have said historically. I guess that may not be the case so
much any more.
> Or don't get installed into either a global or
> user-wide directory? Or fail to put commonly used things into a
> standard and common Framework directory?
the apps are self contained. If you drop an app into the
'applications' directory depending on your permissions it goes into
the global or yours. Now with more windows like installers I guess
this has started to change. Its been awhile since I used a mac.
> I see what you're getting at,
> but there are numerous angles, an I suppose the way in which people
> perceive and/or interpret the principles is where the split tends to
> occur.
Thats exactly true.
>
> I agree with the principle of having things self contained, but I
> don't see this as an important example of self-containment. I
> completely agree with you about releases needing to contain everything
> you need - that is an example that stands out to me. Having said that,
> we've already agreed that in order to make our lives 'manageable' we
> are not going to bundle runtime OTP libraries in packaged releases, so
> clearly 'self-containment' isn't an overriding principle in the face
> of other forces.
Not for our use case of providing packages. However, I will absolutely
still do that when I go to deploy a release. I seriously doubt I will
be deploying directly from our packages.
>
> > We might be able to do this transparently in something like ~/.repo
> > but I absolutely want the option to turn it off.
> >
>
> I am absolutely fine with people being able to choose between using a
> project local app dir (and index) and having the dependencies resolved
> from the local repo.
I think I might be confused by this. Let me restate and see if I
misunderstood.
The local repo isnt really usable directly by erlang. Not without some
tool to munge the paths and its certainly not usable by any tool that
isnt in this toolchain or hasn't been modified to support this tool
chain. So I would like the dependencies to end up in some usable
place. I dont actually care where they come from be it a local repo or
directly from a remote repo.
I worry that if we dont have some 'normal' directory structure we will
be getting away from the idea that the end points and outpoints of
each tool are generally consumable. That actually bugs me a bit from
an interoperability standpoint.
> I am also absolutely fine with writing the local
> repo handling code myself, as long as I have an API to integrate it
> with the assembler/packager. I want to make this endevour
>
> 1. easy for you and I as well as other potential contributors to work on
> 2. as simple to understand as possible, inside and out
> 3. as configurable and extensible as possible (without violating 1 and
> 2 greatly)
I think we have the same goals we are just a bit off on thoughts about
how each applies. I suspect we can come to some agreement.
>
> >> Question:
> >>
> >> I agree that we need the namespace flattening, but I'm
> >> wondering why we need do implement it in this way. On windows, for
> >> example, this is a royal pain as there are no symlinks.
> >
> > I wasn't planning to use symlinks at all.
> >
>
> Fair enough. I tend to err on the side of performance, and my dislike
> of copying files and directories around (and/or repeatedly fetching
> things from the internel) is not about space usage or redundancy, it's
> about making sure that my build+test cycle takes seconds and not
> minutes.
It gets done infrequently. so I tend to not worry about it. If it was
happening every time I needed to build or something like that it would
be a very different story.
> Bare in mind that for me, a full integration test requires
> that the release is packaged and deployed to a VM before the end-2-end
> test suites (common_test and other things besides) are run, and most
> CI servers do this with a fresh checkout from version control, not a
> pull/update - this means that if everything has to be fetched and
> copied and so on, the build can take 10 minutes which is a very long
> lapse to then discover that you've broken someone else's code.
Fair enough. We have a slightly different model. I test with nearly
every build but that test is unit/cucumber tests. Integration tests
happen much less frequently maybe once a day or so. Usually before I
publish my changes. So again I tend not to mind a bit of extra time there.
>
> >> If we make the local repository a core part of the solution, then
> >> there is another option which IMO is viable.
> >>
> >> All the use-cases for the dependencies involve getting a running
> >> emulator to 'know about' them, for example compile time ones (such
> >> as parse transforms and build plugins) need to be on the code path
> >> during the 'build tool' operations. Runtime ones need to be available
> >> during testing, local development and pacakging a release. Test scoped
> >> ones need to be on the path during testing. In all cases, the following
> >> things need to work
> >>
> >> $ code:which(AModuleInTheDependency) /= non_existing
> >> $ code:lib_dir(TheAppName) /= {error,bad_name}
> >>
> >> -include_lib("dependency/include/header.hrl").
> >>
> >> So why not just add the correct directories to the code path at the
> >> appropriate time? This saves copying (or symlinking) and avoids
> >> cluttering the local directory structure.
> >
> > lol, I think we have very different outlooks here. We are going to
> > need to resolve that.
> >
>
> Yes I can see that! :)
>
> I'm not worried about resolving it - I will go along with implementing
> your preferred method(s) as long as there are extensibility points for
> me to exploit so I can implement my alternative approaches re global
> local repository and index handling.
I would like us to come to consensus. We both have decent experience
and decent approaches. Coming up with a way to simply and elegantly
support them both is worth the effort I think.
> I don't think you're wrong, I
> just think people have different opionions about this (mainly based on
> the tools they're familiar and comfortable with) and having both
> options strikes me as a good thing.
I agree!
>
> >> It also avoid having to
> >> manage an 'app dir' when dependencies are removed for example.
> >
> > nuking the directory?
> >
>
> Yes you're right - that's the simplest way to refresh the dependencies
> whenever anything other than an insert takes place in the dependency
> graph. I guess we've made an implementation decision for the app-dir
> then.
Not necessarily. I will break this conversation out into its own
thread so we can discuss the points. I think it worth doing and it
gets a bit long having it intermingled with others.
>
> >> Personally I find this approach preferable, as long as the code
> >> path modification process is documented clearly.
> >
> > This is doable. There are two things I dont like about it. The first
> > and biggest is that its much more difficult to plug in arbitrary tools
> > that are not part of the chain. That is things that use the packages
> > now need to know about this non-erlang special repo structure that we
> > have. This may actualy not be a problem, but we need to make sure that
> > none of the packaging semantics make it into running code. The process
> > being as vanilla as possible I think is important.
> >
>
> Let me just be really clear. When the packaging takes place, I expect
> the applications and libraries which are listed as dependencies to be
> copied into the 'lib' directory inside the release folder, just the
> same as you.
Right, as part of packaging. I understand. however, there are several
'pre-packaging' steps that take place.
>
> I agree that providing a seamless API for working with both 'styles'
> is going to be a bit more complex. I am spending a fair bit of time
> thinking about what underlying abstractions can support both these
> processes, whilst fitting nicely into a unix toolchain phiolosophy. I
> think we can do it, it'll just take a bit of thinking about. I propose
> that we decide on the API(s) between the tools for the general case,
> where there is an app-dir, and then I'll think about how I can expose
> the local repository paths that matter in a fashion that suits this
> API.
I think the base APIs are their inputs and outputs and those should be
as close to OTP as possible.
>
> > On dependencies.
> >
> > I am not sure what you intend for 'build-support'. I guess that is the
> > tools to do the compile themselves. I am not sure how that can be
> > worked into the process.
> >
>
> No that's not so much it. Think about it like this - I want to write a
> custom 'remote index' handler that'll resolve/fetch the remote index
> from subversion or nexus instead of github. The default remote index
> handler is a module which is probably called something like
> 'erlware_remote_index' and is on the code path because it is
> distributed with the tool (being the default handler). I want to put
> my 'bt_nexus_remote_index' module into an OTP application and resolve
> it just like any other dependency. Once it is resolved (via the
> default 'remote index' on github) I want it downloaded and then added
> to the code path. I want this to happen before any further
> dependencies are resolved, because I want this 'extension' to kick in
> and add its own index metadata so that private artefacts which are
> hosted internally on nexus can be resolved alongside public open
> source components hosted on github/bitbucket.
I just realized that is long as bt_nexus_remote_index is in the normal
repository pulling it down for further interaction is probably not a
problem.
Though this is probably unique to the manager.
> Another example is resolving a rebar plugin. Rebar plugins need to be
> downloaded and put on the code path before rebar runs (or before the
> rebar commands they add/exploit are run) and this often needs to
> happen way before 'compile' is executed. Now in both those cases, the
> dependencies are 'before compile' and might also be used 'during
> compile' but in either case they're not required (i.e., included) when
> I assemble a release.
>
> Hope that makes sense.
It does and its just naming so I dont really have a problem with any
of it. I would like to recommend that we dont think of this as one
large 'build' process. But a lot of small processes that can be strung
together arbitrarily or not strung together at all.
> > Also on packaging. At the moment we are not allowing user code to run
> > as packaging. that is the packaging tools are complete unto
> > themselves. Well better said that, if you are doing packaging the
> > packaging tool is already there. I may be missing something here.
> >
> >
> >
>
> I've not explained these clearly, which is poor form on my part. I've
> also failed, in my rather shabby analysis of this, to delineate the
> scope of dependencies from the packaging types. Let me try again.
This thread did well. I am with you on it.
> There are, in my mind, the following scopes for a dependency:
>
> 1. things that extend the erlware tool (or toolset - possibly even the
> running code that fetches them) like custom indexes and/or publishers
> 2. things that extend the functionality of the build tool (whether it
> is sinan or rebar or make or whatever)
> 3. things that are required during compilation (such as parse transforms)
> 4. things that are required during testing only
> 5. things that are required at runtime (not just for testing)
> 6. things that are required at runtime but not for compile or test (I
> suspect this scope can be ignored as we can build it from the others)
>
> Now I've based these scopes on the different ways in which the
> dependency is being used. In each case, I'm asking myself 'does it
> need to be available to do X or not'. When one does this in rebar,
> it's almost irrelevant because to generate a release using rebar you
> have to have reltool.config which specifies which applications are to
> be included. I am working on the basis that we will replace reltool
> with our own 'assembly' implementation, which I'm assuming will scan
> the dependencies to figure out what to include in the release. Based
> on this, my thoughts are that
I agree with this across the board. However, lets be careful about
unifying ths process. As you say this is a suite of related
tools. However, each should be self contained. So your
'pre-manager-fetching' class is actually just a list of plugins for
'erlang-manager'. Its not a 'pre-build-tools' scope because there is
no such thing as a 'pre-process'. I hope I am being articulate enough
here. I worry that I am not.
> - I have things like 'bt_nexus_remote_index' which I need on the path
> to resolve dependencies via nexus, but not included in the release
Right, these are 'manager' plugins and have little or nothing to do
with any of the other tools.
> - I have things like 'esl/parse_trans' which I need on the path to
> compile, but not included in the release
compile_time for the build tool.
> - I have things like 'rebar_cucumber_plugin' which I need on the path
> to run tests, but not included in the release
test time for the build tool.
> - I have things like PropEr and hamcrest-erlang and cucumberl that I
> want on the path for testing, but not included in the release
same.
> - I have libraries which need to be available for testing and are also
> included in the release (making these compile time would solve it)
so these are run time really? test time I assume needs all of the
runtime tools. Where as compile time tools do not have to be there at
test or run time.
> - I have libraries which need to be available only when
> building/packaging the release (for example, esl/setup or erlctrl)
so these are plugins to the assembler?
> - I have things like 'bt_nexus_artefact_publisher' which I need on the
> path to publish, but have nothing to do with the release
again plugins to the manager I think.
> So maybe based on this, having a 'packaging' scope isn't really
> necessary, as you can make things 'compile' scope to achieve the same
> effect. The 'build-support' scope is for being able to differentiate
> between dependencies such as 'bt_nexus_remote_index' which are
> required for everything to work. It is a poorly chosen term, as it
> isn't necessarily just 'build' that the dependency is supporting - it
> is 'resolve-dependencies' scoped but shouldn't be included in the
> release. Because 'bt_nexus_artefact_publisher' is 'publish' scoped I
> figured I'd combine the two, but it doesn't matter either way.
As I said I am not opposed to that scope. I think in the context of
having mulitple small focused tools thinking about them as a scope
might not be the right way to go.
> Hopefully what you'll have picked up from this is that I want to be
> able to write customisations as separate libraries/applications and
> include then as dependencies. This is because
>
> a. I don't like bundling half a million extensions with the main tool sources
I am with you there.
> b. I don't like having to fork a project when I've built an extension
> that the maintainers may or may not want to include
I am with you here too.
> c. I don't like having to manage huge pull request queues for my
> project when lots of loonies (like me!) get overexcited about
> customising
I am with you here too.
>
> Again I'm coloured by many years of experience with maven. It resolves
> plugins on the fly and downloads them in ~/.m2/repository/... and then
> makes them available during the build. All you need to do is declare
> the plugin (and sometimes the repository it resides in if it's not a
> standard one) and maven does the rest. This makes adding scala or
> clojure or groovy of even erlang support to your maven build, very
> simple - just configure the project xml and run 'mvn clean compile'
> and everything gets downloaded (just once) and then magically works
> and your source code (and project folders) remain completely
> unaltered.
Thinking about these plugins having the local repo starts to make more
sense and it doesn't violate the OTPness of things I dont think. I
have no problem if that is user local either. So for example, lets say
the manager. It needs to pull down code for your nexus plugin. I
actually have no problem at all having it look in the local repo for
its dependencies because its not part of its public api so to speak. I
want OTPness in the inputs and outputs that are designed to be
consumed by the next tool. I dont actually mind things are not OTP in
the between input and output.
>
> Because I've become accustomed to this philosophy, wherein compile,
> test and runtime 'artefacts' are completely managed for me, I'm
> slightly OCD about being forced to deal with them when using rebar.
> Even adding rebar's 'deps' app-dir to .gitignore for every project
> drives me over the edge.
I am somewhat there for you. I hate the actual project dir to be
polluted by non-project things even beam code. If its generated it
shouldn't be in there with the not generated code period. Thats why
sinan takes the approach of building everything into _build. It sounds
like you take it even a step further and dont want *anything* in the
project dir. I am not sure that makes a ton of sense to me. But we can
probably sort it out.
>
> As maven mainly 'just works' I'd probably use the maven-erlang-plugin
> for my open source projects but the reality is that many Erlangers
> aren't going to install the JVM just to build my sources and I don't
> want to maintain multiple configurations, so I really want to make
> rebar do things the way I like them.
I am one of them really. I spent ten years working mostly in java and
was very happy when I got away from that. I cringe even starting the
jvm anymore unless its to run clojure. Even then the startup times
always piss me off. So this is understandable to me.
> That's why I initially wrote
> https:/github.com/hyperthunk/rebar_alt_deps and later decided to
> externalise the dependency management was better. This multi-tool idea
> is even better still, as people can use it from sinan and make and
> whatever else, but ultimately my end goal is for a user to download my
> project and run
>
> $ rebar clean package
>
> and for all the dependencies to get properly resolved and downloaded
> at the right time and for the entire tool chain to just work without
> further intervention and for it to be blazingly fast from then on when
> the project(s) are in CI.
I will start a new thread about this so we can focus on this topic.
Tim Watson
be useful background.
On 1 March 2012 19:54, Tim Watson <watson....@gmail.com> wrote:
>
> No that's not so much it. Think about it like this - I want to write a
> custom 'remote index' handler that'll resolve/fetch the remote index
> from subversion or nexus instead of github. The default remote index
> handler is a module which is probably called something like
> 'erlware_remote_index' and is on the code path because it is
> distributed with the tool (being the default handler). I want to put
> my 'bt_nexus_remote_index' module into an OTP application and resolve
> it just like any other dependency. Once it is resolved (via the
> default 'remote index' on github) I want it downloaded and then added
> to the code path. I want this to happen before any further
> dependencies are resolved, because I want this 'extension' to kick in
> and add its own index metadata so that private artefacts which are
> hosted internally on nexus can be resolved alongside public open
> source components hosted on github/bitbucket.
>
So this issue of getting hold of extensions and loading them
dynamically is a real pain in rebar, and is one of the things that has
driven me to look for an external solution that I can plug into any
build tool (including rebar) to make my sanity come back. So I have a
couple of approaches to working with dependencies that 'cause rebar
problems' for one reason or another. One of them is alt_deps, which
resolves dependencies in a different way and along with my 'alien
plugin' supports deps that are rely on a non-rebar tool chain to be
built.
I want this plugin to work in the normal way, so that people go 'rebar
get-deps compile' and it just works. But this isn't the case because
first of all the person who has downloaded my project needs to go and
get alt_deps and alien_plugin! To solve this problem, I distribute
these extensions as dependencies.
Now here come the rub. For rebar to recognise my plugin's extension
hook, it needs to find rebar_alt_deps:'get-deps'/2 on the code path.
In order for that to work, the plugin needs to be compiled, so I rely
on rebar fetching the plugin and then compiling it. But now I have a
problem because once 'get-deps' is finished it runs 'compile' and then
stops. Now the user is forced to do some crazy nonsense like
$ rebar get-deps compile install-deps
That just looks (and is) plain wrong. I don't have any of this crap to
deal with if I write and distribute a maven plugin. I write the plugin
and publish it to a public OSS repository (like Sonatype Nexus OSS)
and then in the projects that use it, I just put a reference to it in
my pom.xml:
<packaging>erlang-otp</packaging>
<build>
<plugins>
<plugin>
<groupId>eu.lindenbaum</groupId>
<artifactId>maven-erlang-plugin</artifactId>
<version>2.1.0</version>
<extensions>true</extensions>
</plugin>
</plugins>
</build>
And that's it!!! Maven downloads the plugin the first time this
project is built by running 'mvn compile' and after that it is
business as usual. A plugin in this model is basically a kind of
internally/specially scoped dependency.
Eric B Merritt
I think we are going to end up supporting this case. We should be able
to certainly get around the problems you are specifying here with
rebar.
Eric B Merritt
I think you have talked me into this. The only caveat to the list
below is that I think we should have a simple tool that takes a list
of paths and copies the OTP apps to a directory.
It would be cool if each of these could read their inputs on standard
in and write the outputs to standard out as well.
**manager**:
Inputs: a list of repos?
outputs: local index + basic local repo structure
**solver**:
Input: Constraints are by their nature not OTP but we should try to
fit them in as seamlessly as possible
Output: Should be a rel or relup file with organization annotation
**fetcher**:
Inputs: a rel or relup file
Outputs: A list of code paths for the various dependencies
**builder**:
Inputs: A project that contains a rel+/relup file, a list of code paths for dependencies
Outputs: A built otp application(s) + updated list of codepaths
**assembler**:
Inputs: A rel/+relup, A list of codepaths
Outputs: An OTP Release directory structure
**packager**:
Inputs: An OTP Release directory structure
Outputs: A dist tarball (debian package, rpm etc)
Tim Watson
light of the 'lots of little tools' approach - not be so monolithic in
my perception of the process and all that - and it occurs to me that
if some of our outputs are lists of file system paths we might be ok.
I come on here to suggest it and voila, you've read my mind.
On 1 March 2012 21:01, Eric B Merritt <ericbm...@gmail.com> wrote:
>
> I think you have talked me into this. The only caveat to the list
> below is that I think we should have a simple tool that takes a list
> of paths and copies the OTP apps to a directory.
>
I completely agree with that.
> It would be cool if each of these could read their inputs on standard
> in and write the outputs to standard out as well.
>
Yes definitely. I'm almost inclined to say that should be the required
unix-y interface, so that people who're using Make (and similar tools)
can still get value out of each and every one of these tools as they
see fit.
> **manager**:
>
> Inputs: a list of repos?
> outputs: local index + basic local repo structure
>
Inputs: you mean a list of 'remote indexes' here right? The word
'repo' is overloaded and implies storage more than anything else. But
yes, I think a list of repos is fine.
Flags: -f <filename> read the list of repos from <filename> instead of stdin?
Question: how are the repos specified? As a URL or as something else?
Question: this guy is presumably not writing to stdout, as it is
creating directories on disk etc. So perhaps the stdout stream should
contain the root directories and/or locations of these?
> **solver**:
>
> Input: Constraints are by their nature not OTP but we should try to
> fit them in as seamlessly as possible
>
> Output: Should be a rel or relup file with organization annotation
>
>
Yes that output sounds very reasonable. This is the area we've spent
the least time thinking about the inputs for.
> **fetcher**:
> Inputs: a rel or relup file
> Outputs: A list of code paths for the various dependencies
>
Absolutely - this is a great idea. What about where you're in an OTP
application/library project and want to fetch your dependencies?
Should we still work off a rel file out of the solver? I guess it
doesn't matter that the rel/relup file is not needed, as we can still
use it as an artefact in our process if we want to.
>
> **builder**:
> Inputs: A project that contains a rel+/relup file, a list of code paths for dependencies
> Outputs: A built otp application(s) + updated list of codepaths
>
>
Cool, I like this. As we've discussed before though, not every project
is a release - sometimes it's just an application or a library. We
need to support those use cases as well, so the rel/+relup can't be
mandatory.
>
> **assembler**:
> Inputs: A rel/+relup, A list of codepaths
> Outputs: An OTP Release directory structure
>
> **packager**:
> Inputs: An OTP Release directory structure
> Outputs: A dist tarball (debian package, rpm etc)
Ok I'm fine with these. Also the packager needs to support packaging
plain applications/libraries as well as releases.
**publisher**
> Inputs: A dist tarball and choice of index to publish to + the users personal configuration (passwords, etc)
> Outputs-1: an HTTP request to github/bitbucket/nexus/etc to upload/store the publication metadata in an index
> Outputs-2: an HTTP request to github/bitbucket/nexus/etc to upload/store the binary artefact online
Eric B Merritt
Tim Watson wrote:
>
> I was just starting to think I needed to reconsider my position in
> light of the 'lots of little tools' approach - not be so monolithic in
> my perception of the process and all that - and it occurs to me that
> if some of our outputs are lists of file system paths we might be ok.
> I come on here to suggest it and voila, you've read my mind.
Its good to be on the same page.
>
> > It would be cool if each of these could read their inputs on standard
> > in and write the outputs to standard out as well.
> >
>
> Yes definitely. I'm almost inclined to say that should be the required
> unix-y interface, so that people who're using Make (and similar tools)
> can still get value out of each and every one of these tools as they
> see fit.
I agree lets make that a requirement.
> > **manager**:
> >
> > Inputs: a list of repos?
> > outputs: local index + basic local repo structure
> >
>
> Inputs: you mean a list of 'remote indexes' here right?
Yes. lets adopt that terminology.
> The word
> 'repo' is overloaded and implies storage more than anything else. But
> yes, I think a list of repos is fine.
>
> Flags: -f <filename> read the list of repos from <filename> instead of stdin?
Sure. It should also read configuration files as well.
> Question: how are the repos specified? As a URL or as something else?
I think all the DVCSs support urls, so lets just use those. We can fit
the plugin concept around those urls.
>
> Question: this guy is presumably not writing to stdout, as it is
> creating directories on disk etc. So perhaps the stdout stream should
> contain the root directories and/or locations of these?
Thats a good idea actually.
>
> > **solver**:
> >
> > Input: Constraints are by their nature not OTP but we should try to
> > fit them in as seamlessly as possible
> >
> > Output: Should be a rel or relup file with organization annotation
> >
> >
>
> Yes that output sounds very reasonable. This is the area we've spent
> the least time thinking about the inputs for.
It might also take the directory of the repo as part of its
input. That way it could be linked to the above. Along with command
line options as well of course.
>
> > **fetcher**:
> > Inputs: a rel or relup file
> > Outputs: A list of code paths for the various dependencies
> >
>
> Absolutely - this is a great idea. What about where you're in an OTP
> application/library project and want to fetch your dependencies?
> Should we still work off a rel file out of the solver? I guess it
> doesn't matter that the rel/relup file is not needed, as we can still
> use it as an artefact in our process if we want to.
In this case the rel file just serves as a carrier of dependency
information. I think that works whether you have a single app or an
entire release. I dont think there is any need to invent something
ourselves that carries the exact same information.
>
> >
> > **builder**:
> > Inputs: A project that contains a rel+/relup file, a list of code paths for dependencies
> > Outputs: A built otp application(s) + updated list of codepaths
> >
> >
>
> Cool, I like this. As we've discussed before though, not every project
> is a release - sometimes it's just an application or a library. We
> need to support those use cases as well, so the rel/+relup can't be
> mandatory.
As I said above we need something to carry that information. Using a
dotrel makes more sense then inventing something ourselves. You can
have one without actually using it for anything else.
>
> >
> > **assembler**:
> > Inputs: A rel/+relup, A list of codepaths
> > Outputs: An OTP Release directory structure
> >
> > **packager**:
> > Inputs: An OTP Release directory structure
> > Outputs: A dist tarball (debian package, rpm etc)
>
> Ok I'm fine with these. Also the packager needs to support packaging
> plain applications/libraries as well as releases.
if by applications you mean OTP Applications of the active and
inactive variety (curse erlang's name overloading) then I am on board.
>
> **publisher**
> > Inputs: A dist tarball and choice of index to publish to + the users personal configuration (passwords, etc)
> > Outputs-1: an HTTP request to github/bitbucket/nexus/etc to upload/store the publication metadata in an index
> > Outputs-2: an HTTP request to github/bitbucket/nexus/etc to upload/store the binary artefact online
Absolutely.
There are probably tools for a repository maintainer as well. For
pulling merge requests, verifying signatures and the like. To come up
with something we forward to the erlang-questions list we dont need to
detail that. It is something we will want to keep in mind though.
Eric B Merritt
I just realized that we could turn our 'list of code paths' to a
viable source in and of itself and then use that is inputs and outputs
to everything even a rel/relup compiler. The reality is that we need a
bunch of stuff in addition to whats in the so having that in this spec
and making the rel a compile out put would work. Though we loose the
completely vanilla otpishness of some of the tools.
Tim Watson
I have some outstanding questions about configuration. Where do the dependency lists go?In the .app file? Doesn't adding the ">=" and version number screw other tools up? I know that my 'appstart' utility will puke, although I'll happily change it if it's the only thing that does.
Eric B Merritt
Tim Watson wrote:
>
> I'm not sure I fully understand what this means, some of the tools
> don't care about the list of code paths. I suppose each of the tools
> cares about a list of something (repositories, dependencies, code
> paths, etc).
I was thinking about just those tools that take code paths. That is
everything but the manager and packager really.
> I think we are going to end up passing around data that
> isn't pure OTP (app and rel files) anyway. If the OTP metadata was
> rich enough to describe all of what we're trying to achieve, we
> wouldn't have a job to do because it'd already be written.
I am not sure thats the case. The OTP guys never tackled and so simply
never addressed these issues. That doesn't mean the OTP metadata isn't
sufficient.
It also does not mean that we should either, but I would like to avoid
redundancies if possible.
>
> I have some outstanding questions about configuration. Where do the
> dependency lists go?In the .app file?
I suspect thats where they would go by default. Though I suspect we
need to be able to specify them directly as well.
> Doesn't adding the ">=" and
> version number screw other tools up?
Yup. It would screw up everything. However, we can add another tuple
there without a problem and that is the approach I would take. That
extra tuple would give us the information we would need.
> I know that my 'appstart'
> utility will puke, although I'll happily change it if it's the only
> thing that does.
Again none of this is set in stone. We could also stick it in a
separate file in priv or any number of things.
Tim Watson
> At Sat, 3 Mar 2012 18:29:48 +0000,
> Tim Watson wrote:
>>
>> I'm not sure I fully understand what this means, some of the tools
>> don't care about the list of code paths. I suppose each of the tools
>> cares about a list of something (repositories, dependencies, code
>> paths, etc).
>
> I was thinking about just those tools that take code paths. That is
> everything but the manager and packager really.
>
Ok that makes sense now.
>> I think we are going to end up passing around data that
>> isn't pure OTP (app and rel files) anyway. If the OTP metadata was
>> rich enough to describe all of what we're trying to achieve, we
>> wouldn't have a job to do because it'd already be written.
>
> I am not sure thats the case. The OTP guys never tackled and so simply
> never addressed these issues. That doesn't mean the OTP metadata isn't
> sufficient.
>
> It also does not mean that we should either, but I would like to avoid
> redundancies if possible.
>
>>
>> I have some outstanding questions about configuration. Where do the
>> dependency lists go?In the .app file?
>
> I suspect thats where they would go by default. Though I suspect we
> need to be able to specify them directly as well.
>
>> Doesn't adding the ">=" and
>> version number screw other tools up?
>
> Yup. It would screw up everything. However, we can add another tuple
> there without a problem and that is the approach I would take. That
> extra tuple would give us the information we would need.
>
And that extra tuple doesn't screw things up? Baring in mind that the design decision to break up the tool chain into lots of little chunks is as much about giving people choice as any aesthetic judgement, I'm assuming there may well be a class of users who continue to use reltool, opting only to utilise the resolver and one or two other bits. So we can't even break reltool compatibility, and the requirement to not break existing code is tough to meet given the amount of tools coverage we'd need to deal with.
>> I know that my 'appstart'
>> utility will puke, although I'll happily change it if it's the only
>> thing that does.
>
> Again none of this is set in stone. We could also stick it in a
> separate file in priv or any number of things.
>
For my reason above - not risking breaking any existing code in OTP or elsewhere - I prefer the option of a separate file, despite the obvious redundancy. It is a safe decision: we will know for sure we are not going to clash with anything else.
Eric B Merritt
Tim Watson wrote:
>
> >
> >> Doesn't adding the ">=" and
> >> version number screw other tools up?
> >
> > Yup. It would screw up everything. However, we can add another tuple
> > there without a problem and that is the approach I would take. That
> > extra tuple would give us the information we would need.
> >
>
> And that extra tuple doesn't screw things up?
I have been using it for a long time in sinan and it doesn't break
anything at all.
> Baring in mind that the design decision to break up the tool chain
> into lots of little chunks is as much about giving people choice as
> any aesthetic judgement, I'm assuming there may well be a class of
> users who continue to use reltool, opting only to utilise the
> resolver and one or two other bits. So we can't even break reltool
> compatibility, and the requirement to not break existing code is
> tough to meet given the amount of tools coverage we'd need to deal
> with.
I have never tested this with reltool. However, I suspect it will
work. The fact that it is a association list usually implies that
things only pay attention to the key value pairs they care about.
However, I agree 100% that we should not break compatibility and need
to check this.
>
> >> I know that my 'appstart'
> >> utility will puke, although I'll happily change it if it's the only
> >> thing that does.
> >
> > Again none of this is set in stone. We could also stick it in a
> > separate file in priv or any number of things.
> >
>
> For my reason above - not risking breaking any existing code in OTP
> or elsewhere - I prefer the option of a separate file, despite the
> obvious redundancy. It is a safe decision: we will know for sure we
> are not going to clash with anything else.
I am ok with that. We can do the same thing for releases. Lets
document it will though and make it trivially parsable in
erlang. Basically as an erlang term file.
The only downside is it will not be accessible from escript archives.
Tim Watson
it in priv/ would be a good idea, just so that it can be included in
escripts. IIRC Ulf has a patch for OTP (erl_prim_loader) that fixes
the accessibility of files in .ez archives. For escripts it's another
story though - but that's another thing we'll need to think about
whether we support or not - bundled executable escripts as a packaging
type!?
Eric Merritt
> Sorry about top posting - using someone else's machine. Maybe putting
> it in priv/ would be a good idea, just so that it can be included in
> escripts.
Would be or would *not* be? I am a bit confused.
> IIRC Ulf has a patch for OTP (erl_prim_loader) that fixes
> the accessibility of files in .ez archives. For escripts it's another
> story though - but that's another thing we'll need to think about
> whether we support or not - bundled executable escripts as a packaging
> type!?
To be fair *app files are not available in escript archives either.
Hmmm, I think we are probably going to have to support escripts and
come up with some way to spec dependencies there too.
So for apps, something in the priv dir. for releases something
somewhere else. same for rel files. we just have to define something
else.
Tim Watson
On 5 Mar 2012, at 20:39, Eric Merritt wrote:
> On Mon, Mar 5, 2012 at 2:26 PM, Tim Watson <watson....@gmail.com> wrote:
>> Sorry about top posting - using someone else's machine. Maybe putting
>> it in priv/ would be a good idea, just so that it can be included in
>> escripts.
>
> Would be or would *not* be? I am a bit confused.
>
I guess I'm thinking it might be a good thing - I'm not really sure. I'm torn between wanted to keep the dependency config separate and at the project top level for a cleaner looking build environment, and wanting to obey OTP habbits in the distribution by putting the config into either the app file or the priv directory.
>
>> IIRC Ulf has a patch for OTP (erl_prim_loader) that fixes
>> the accessibility of files in .ez archives. For escripts it's another
>> story though - but that's another thing we'll need to think about
>> whether we support or not - bundled executable escripts as a packaging
>> type!?
>
> To be fair *app files are not available in escript archives either.
>
Really? I didn't know that.
> Hmmm, I think we are probably going to have to support escripts and
> come up with some way to spec dependencies there too.
>
Yes.
> So for apps, something in the priv dir. for releases something
> somewhere else. same for rel files. we just have to define something
> else.
>
I can buy into that.
Eric Merritt
> Yes - really not sure what to do with binary escripts. Rebar handles fetching files out of ./priv in its escript by running escript:extract/2 and then calling zip:foldl/3, which works ok.
I suspect we shoud treat escripts as on output. That is they are
something the packager produces but are not stored in the system for
use as dependencies. That would solve the problem.
>
> On 5 Mar 2012, at 20:39, Eric Merritt wrote:
>
>> On Mon, Mar 5, 2012 at 2:26 PM, Tim Watson <watson....@gmail.com> wrote:
>>> Sorry about top posting - using someone else's machine. Maybe putting
>>> it in priv/ would be a good idea, just so that it can be included in
>>> escripts.
>>
>> Would be or would *not* be? I am a bit confused.
>>
>
> I guess I'm thinking it might be a good thing - I'm not really sure. I'm torn between wanted to keep the dependency config separate and at the project top level for a cleaner looking build environment, and wanting to obey OTP habbits in the distribution by putting the config into either the app file or the priv directory.
well the dependency for an app cant be at the project top level simply
because a project can mean different things to different tools. For me
a project is often several related OTP apps. If you mean at the OTP
app top level I probably wouldn't want it there simply from an
organizational standpoint.