Re: OTP Application Namespaces
Martin Logan
before describing the solution? I generally think I know what you are
talking about but not completely sure.
On Fri, Feb 24, 2012 at 11:03 AM, Eric Merritt <ericbm...@gmail.com> wrote:
> I am thinking about this problem and I have not come up with an
> elegant solution. However, I think I can expound a few invariants.
>
> 1. Every OTP application lives in a namespace.
> 2. For most applications the namespace is `'*global*'`.
>
>
> **NOTE:** all of this presupposes that once published an Namespace,
> AppName, Version does not change. I realize this is not always the
> case in realty. However, I believe it is something we must treat as an
> invariant to have any sane system.
>
> Multiple namespaces exist in the *cloud*. However, during constraint
> solving namespaces are *flattened*. So when the constraint solver
> solves a problem and comes up with a list of namespace, app, versions
> it is an error if there are two namespaces with the same app name and
> version in that solution. In this case an exception should be
> signaled.
>
> If a user does not specify a namespace in constraints then the
> namespace is assumed to be `'*global*'`. This is the case when the
> user is creating dependency constraints and the like. No need to type
> 'global' everywhere.
>
> # On Disk
>
> A directory on disk that contains OTP applications (a local repo) is a
> flattened namespace environment. Therefore, an otp application
> (<app-name>-<vsn>) should never be overwritten in a repo. If the
> resolver is asked to pull down that <app-name>-<vsn> it should signal
> an exception and tell the user how to resolve it.
>
> If the use puts more then one repo in his code path, he might be in
> for trouble. Thats on the user though and there is not a lot we can do
> about it. We should just make it obvious in the documentation.
>
> --
> You received this message because you are subscribed to the Google Groups "erlware-questions" group.
> To post to this group, send email to erlware-...@googlegroups.com.
> To unsubscribe from this group, send email to erlware-questi...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/erlware-questions?hl=en.
>
--
Martin Logan
Erlang & OTP in Action (Manning) http://manning.com/logan
http://twitter.com/martinjlogan
http://erlware.org
Eric Merritt
emails that have been going back and forth over the last few days.
This email just pulls that existing subtopic out for further
discussion.
However, to be succint. With all the forking going on you could easily
have basho/ec/1.0 and hyperthunk/ec/1.0 with different code bases.
Tim Watson
>
> **NOTE:** all of this presupposes that once published an Namespace,
> AppName, Version does not change. I realize this is not always the
> case in realty. However, I believe it is something we must treat as an
> invariant to have any sane system.
>
Agreed. If I have an app-vsn published in my personal namespace (e.g.,
hyperthunk/libconf-0.1.0) and I want to 'change the namespace' then
I'm effectively going to put libconf-0.1.0 into my other namespace and
create nebularis/libconf-0.1.0 but IMO the simplest solution to this
is that we consider these two completely different things and exist
independently.
> Multiple namespaces exist in the *cloud*. However, during constraint
> solving namespaces are *flattened*. So when the constraint solver
> solves a problem and comes up with a list of namespace, app, versions
> it is an error if there are two namespaces with the same app name and
> version in that solution. In this case an exception should be
> signaled.
I don't understand this. Why would you want to do this unless the
problem is missing the namespace information?
If I want to get hold of my own fork of apache commons-lang, I should
be able to do that without the solver crashing because both org.apache
and org.nebularis happen to contain an artefact named 'commons-lang'
with version 2.5.1. If you can't deal with that problem then you're
better off without namespaces/publishers at all. I think maybe I've
misunderstood what you're saying here.
Basho will publish their lager logging library to a repository. I want
to publish my nebularis fork to a repository. Both organisations want
to make their artefacts available in the central repository. If you
'flatten the namespaces' and 'the constraint solver comes up with a
list of (ns, app, vsn)' then it should not crash - this defeats the
whole object surely?
This is why you introduce the namespace to begin with - so that you
can get around the problem of duplicate application names (due to
forks). The key to solving this is to make the user specify the
publisher *as well as* the artefact and version they want.
>
> If a user does not specify a namespace in constraints then the
> namespace is assumed to be `'*global*'`. This is the case when the
> user is creating dependency constraints and the like. No need to type
> 'global' everywhere.
>
I don't agree with this at all. If there was a global namespace, then
it would be the one in which the OTP libraries reside and no other
artefacts would belong in there. I don't see the point in adding
support for publisher/namespace based resolution and then building in
a workaround so that people don't have to use it.
If we are to solve the problem of clashing application names, we must
have a container one level above application. Once we do this,
dependencies have to be stated in terms of the container as well as
the application and version.
> # On Disk
>
> A directory on disk that contains OTP applications (a local repo) is a
> flattened namespace environment. Therefore, an otp application
> (<app-name>-<vsn>) should never be overwritten in a repo. If the
> resolver is asked to pull down that <app-name>-<vsn> it should signal
> an exception and tell the user how to resolve it.
>
I don't see how this will work unless you get rid of namespace, which
puts us back to square one. I also don't understand how this could
possibly work if we're to support multiple installs of the same
dependency, built for different target environments (i.e., when the
application includes a driver or NIF or is compiled using HIPE).
Today, rebar can solve all of these problems, by simply downloading
the specific dependencies you need for the build environment you run
it in, into a project local folder. It isn't pretty and it has it's
fair share of ugly warts as approaches go, but it works in practise -
it is a practical solution.
If we can't handle these situations, nobody is going to benefit when
there are already solutions available. And these problems aren't
intractable. Maven works fine in practise, precisely because you have
to explicitly state the three parts for each dependency:
<org>-<artefact>-<version>.
> If the use puts more then one repo in his code path, he might be in
> for trouble. Thats on the user though and there is not a lot we can do
> about it. We should just make it obvious in the documentation.
There should be just one, configurable location for the local repo. It
should not be code:lib_dir or ERL_LIBS either, otherwise you get into
a mess when you have multiple OTP installations.
Eric Merritt
elegant solution. However, I think I can expound a few invariants.
1. Every OTP application lives in a namespace.
2. For most applications the namespace is `'*global*'`.
**NOTE:** all of this presupposes that once published an Namespace,
AppName, Version does not change. I realize this is not always the
case in realty. However, I believe it is something we must treat as an
invariant to have any sane system.
Multiple namespaces exist in the *cloud*. However, during constraint
solving namespaces are *flattened*. So when the constraint solver
solves a problem and comes up with a list of namespace, app, versions
it is an error if there are two namespaces with the same app name and
version in that solution. In this case an exception should be
signaled.
If a user does not specify a namespace in constraints then the
namespace is assumed to be `'*global*'`. This is the case when the
user is creating dependency constraints and the like. No need to type
'global' everywhere.
# On Disk
A directory on disk that contains OTP applications (a local repo) is a
flattened namespace environment. Therefore, an otp application
(<app-name>-<vsn>) should never be overwritten in a repo. If the
resolver is asked to pull down that <app-name>-<vsn> it should signal
an exception and tell the user how to resolve it.
If the use puts more then one repo in his code path, he might be in
Eric Merritt
>> Multiple namespaces exist in the *cloud*. However, during constraint
>> solving namespaces are *flattened*. So when the constraint solver
>> solves a problem and comes up with a list of namespace, app, versions
>> it is an error if there are two namespaces with the same app name and
>> version in that solution. In this case an exception should be
>> signaled.
>
> I don't understand this. Why would you want to do this unless the
> problem is missing the namespace information?
We have the problem that erlang, the vm can not handle namespaces.
There is no getting around that. So anything that is presented to the
vm must be resolved to the appname-vsn. The solver, which is solving
to produce a rel file for consumption by the VM must realize it to
appname-vsn. I dont see any way around this.
> If I want to get hold of my own fork of apache commons-lang, I should
> be able to do that without the solver crashing because both org.apache
> and org.nebularis happen to contain an artefact named 'commons-lang'
> with version 2.5.1. If you can't deal with that problem then you're
> better off without namespaces/publishers at all. I think maybe I've
> misunderstood what you're saying here.
Remember that the solver is solving to create a rel file. How would
you see commons-lang 2.5.1 for both nebularis and apache apearing in a
rel file? or in a single directory that is in a vms code path?
> Basho will publish their lager logging library to a repository. I want
> to publish my nebularis fork to a repository. Both organisations want
> to make their artefacts available in the central repository. If you
> 'flatten the namespaces' and 'the constraint solver comes up with a
> list of (ns, app, vsn)' then it should not crash - this defeats the
> whole object surely?
I think we are looking at two different things. We have what is out
there in repositories, and what is going to be local consumed one way
or the other. We need to support the namespaces out there in the
world, but we cant support them locally. At least, we can support them
locally a location that is going to be directly consumed. I am getting
redudant here.
> This is why you introduce the namespace to begin with - so that you
> can get around the problem of duplicate application names (due to
> forks). The key to solving this is to make the user specify the
> publisher *as well as* the artefact and version they want.
Exactly, but we cant get around that for realizing apps but not for
consuming them.
>>
>> If a user does not specify a namespace in constraints then the
>> namespace is assumed to be `'*global*'`. This is the case when the
>> user is creating dependency constraints and the like. No need to type
>> 'global' everywhere.
>>
>
> I don't agree with this at all. If there was a global namespace, then
> it would be the one in which the OTP libraries reside and no other
> artefacts would belong in there. I don't see the point in adding
> support for publisher/namespace based resolution and then building in
> a workaround so that people don't have to use it.
Ok, we have a different idea about this i think. Not that that cant be
realized. I have been thinking in a flat namespace world where having
a namespace that matters is an exception that must be supported
(unfortunately). I think you are thinking more along the lines that
there is always a namespace we just tend to ignore them. I am not sure
what I think about that at the moment. I will have to give it some
thought.
>
> If we are to solve the problem of clashing application names, we must
> have a container one level above application. Once we do this,
> dependencies have to be stated in terms of the container as well as
> the application and version.
>
>> # On Disk
>>
>> A directory on disk that contains OTP applications (a local repo) is a
>> flattened namespace environment. Therefore, an otp application
>> (<app-name>-<vsn>) should never be overwritten in a repo. If the
>> resolver is asked to pull down that <app-name>-<vsn> it should signal
>> an exception and tell the user how to resolve it.
>>
>
> I don't see how this will work unless you get rid of namespace, which
> puts us back to square one. I also don't understand how this could
> possibly work if we're to support multiple installs of the same
> dependency, built for different target environments (i.e., when the
> application includes a driver or NIF or is compiled using HIPE).
Ok. I think we are both making assumptions about what the other is
thinking that may not be valid and thats brining in a lot of
ambiguity. Let me step back and rearticulate a bit.
So we have at least two things (maybe three)
1. Repos out there in the ether that contain namespaces and all kinds
of good things.
2. OTP Applications realized on disk that are intended to be consumed
by the erlang vm.
There may be a 1.5 which is a local on disk cach of the OTP Apps out
there in the repo. That make contain a similar structure to the Repos
out in the ether along with the actual built .
There may be tons of number 2. I will probably have one of those for
every single project. Although others might have one per box.
>
> Today, rebar can solve all of these problems, by simply downloading
> the specific dependencies you need for the build environment you run
> it in, into a project local folder. It isn't pretty and it has it's
> fair share of ugly warts as approaches go, but it works in practise -
> it is a practical solution.
Ummm, this is basically exactly what I am saying.
> If we can't handle these situations, nobody is going to benefit when
> there are already solutions available. And these problems aren't
> intractable. Maven works fine in practise, precisely because you have
> to explicitly state the three parts for each dependency:
> <org>-<artefact>-<version>.
>
>> If the use puts more then one repo in his code path, he might be in
>> for trouble. Thats on the user though and there is not a lot we can do
>> about it. We should just make it obvious in the documentation.
>
> There should be just one, configurable location for the local repo. It
> should not be code:lib_dir or ERL_LIBS either, otherwise you get into
> a mess when you have multiple OTP installations.
Ah, this is where we are running into conflict I think. I dont
actually foresee needing or wanting an 'local repo' step 1.5 above. If
this local repo or local cache exists then you are right its going to
need to support namespaces and all of the above. I just don't see much
need for it and if there is I consider it an optimization more then
anything else.
Eric Merritt
thinking as I was writing the previous.
On having namespaces, you are probably right they need to be there.
There probably is a namespace for everything though in the vast
majority of cases there should only be one. That is going to make
writing specs and the like rather fugly though. At the very least lets
try to stay away from the javaish verbosity.
Let me make some terminology here before moving on.
Repo -> metadata repos out there in the ether
Index -> A local cache of the metadata we have access to
Local Repo -> a local cache of the source packages (contains namespace
and metadata)
App Dir -> A directory designed to be consumed by erlang with no namespaces
Our multi-part namespace semantics exist through, repo, index and
local repo. They can not exist at app dir. Anything that output
appname-version for consumption by the vm must realize a flattened
namespace. Basically, the flattened namespace is a compile artifact.
Anything down to that point must retain it.
Tim Watson
your post a little bit. I realise now that when you say 'flatten the
namespaces' you're talking about the directory in which the
applications reside which are going into a release. That's absolutely
a flat namespace because, as you say, that's how Erlang/OTP works.
On 24 February 2012 20:20, Eric Merritt <ericbm...@gmail.com> wrote:
> I am going to follow up here out of line, just because I have been
> thinking as I was writing the previous.
>
> On having namespaces, you are probably right they need to be there.
> There probably is a namespace for everything though in the vast
> majority of cases there should only be one. That is going to make
> writing specs and the like rather fugly though. At the very least lets
> try to stay away from the javaish verbosity.
>
Yes that sounds about right. And don't get me wrong - plenty of stuff
about java is hiddeous and horrible, but there are reasons beyond just
marketing why it is successful in the enterprise space. I need to
solve these problems in every day development.
> Let me make some terminology here before moving on.
>
> Repo -> metadata repos out there in the ether
> Index -> A local cache of the metadata we have access to
> Local Repo -> a local cache of the source packages (contains namespace
> and metadata)
> App Dir -> A directory designed to be consumed by erlang with no namespaces
>
Perfect. It is the distinction between the local repo and the app dir
that was missing.
> Our multi-part namespace semantics exist through, repo, index and
> local repo. They can not exist at app dir. Anything that output
> appname-version for consumption by the vm must realize a flattened
> namespace. Basically, the flattened namespace is a compile artifact.
> Anything down to that point must retain it.
>
Perfect. And the local repo can have an arbitrarily complex directory
structure, as long as the packager is able to either munge the code
path or copy/symlink the right folders into place when building the
release.
- repository
- index.meta
- nebularis
- eav/0.0.1
- basho
- lager/1.0.2/...
- riak/erts-5.7.2
- Darwin/x86/32bit/riak-1.5.2 % installed with --wordsize=32
- Darwin/x86/64bit % std install
uses erts/os wordsize
- 1.5.2
- 1.5.1
- riak/1.5.2
- erts-5.9
- webmachine/2.0.1
Or even something more complex if it's needed. As long as the packager
can say 'tell me where to find X' then we're ok.
Tim Watson
>
> Ah, this is where we are running into conflict I think. I dont
> actually foresee needing or wanting an 'local repo' step 1.5 above. If
> this local repo or local cache exists then you are right its going to
> need to support namespaces and all of the above. I just don't see much
> need for it and if there is I consider it an optimization more then
> anything else.
>
This is one of the things that I *really* want out of a dependency
management tool chain. If I have to go download all the dependencies
(including transitive ones) every time I build (or refresh) a project,
then I might as well carry on using rebar.
What I'm really after is this. Once a dependency is downloaded into my
local repo, it is isolated and doesn't interfere with any of my OTP
installations. When I want to pull this dependency into a project (at
compile time to add features such as parse_transforms, at runtime for
local development work, at test time for testing or at packaging time
for building a release) then the tool(s) should 'make it available to
me' in the right way. Once I move out of that project context (i.e.,
cd away from the directory or whatever) then the local copy of X
should just go back to being ignored.
Hope that makes sense.
Eric Merritt
>
> Yes that sounds about right. And don't get me wrong - plenty of stuff
> about java is hiddeous and horrible, but there are reasons beyond just
> marketing why it is successful in the enterprise space. I need to
> solve these problems in every day development.
I agree. lets just try to keep the verbosity down where possible.
>> Let me make some terminology here before moving on.
>>
>> Repo -> metadata repos out there in the ether
>> Index -> A local cache of the metadata we have access to
>> Local Repo -> a local cache of the source packages (contains namespace
>> and metadata)
>> App Dir -> A directory designed to be consumed by erlang with no namespaces
>>
>
> Perfect. It is the distinction between the local repo and the app dir
> that was missing.
>
>> Our multi-part namespace semantics exist through, repo, index and
>> local repo. They can not exist at app dir. Anything that output
>> appname-version for consumption by the vm must realize a flattened
>> namespace. Basically, the flattened namespace is a compile artifact.
>> Anything down to that point must retain it.
>>
>
> Perfect. And the local repo can have an arbitrarily complex directory
> structure, as long as the packager is able to either munge the code
> path or copy/symlink the right folders into place when building the
> release.
The erlr should do all this for them, its pretty trivial to do
algorithmicly. In general, we should be thinking about this as a
funnel that eventually resolves into some compiled result. One of the
finial results is a directory that contains the entire release.
>
> - repository
> - index.meta
> - nebularis
> - eav/0.0.1
> - basho
> - lager/1.0.2/...
> - riak/erts-5.7.2
> - Darwin/x86/32bit/riak-1.5.2 % installed with --wordsize=32
> - Darwin/x86/64bit % std install
> uses erts/os wordsize
> - 1.5.2
> - 1.5.1
> - riak/1.5.2
> - erts-5.9
> - webmachine/2.0.1
>
> Or even something more complex if it's needed. As long as the packager
> can say 'tell me where to find X' then we're ok.
>
Tim Watson
>
> The erlr should do all this for them, its pretty trivial to do
> algorithmicly. In general, we should be thinking about this as a
> funnel that eventually resolves into some compiled result. One of the
> finial results is a directory that contains the entire release.
>
Agreed.