Re: [erlang-questions] support for .ez archives

[Tim Watson]

Hello list,

I have a few questions about support for .ez archives, and one or two suggestions as well.

- is a timeline for fully supporting them?
- does the -include_lib(...) directive now work (in R15) when the headers are in an archive?
- does code:lib_dir/2 work properly with archives?

As a suggestion: I've noticed that the way most people build an escript binary is to glue everything together into a zip file with the appropriate header content and I wonder whether having 'executable (.ez) archives' would be a better approach? Perhaps this works today and I haven't figured it out? What are peoples thoughts about this?

Cheers,

Tim
_______________________________________________
erlang-questions mailing list
erlang-q...@erlang.org
http://erlang.org/mailman/listinfo/erlang-questions

[Håkan Mattsson]

On Mon, Mar 5, 2012 at 23:54, Tim Watson <watson....@gmail.com> wrote:

Hello list,

I have a few questions about support for .ez archives, and one or two suggestions as well.

- is a timeline for fully supporting them?
- does the -include_lib(...) directive now work (in R15) when the headers are in an archive?

Björn, is there a timeline?

 

- does code:lib_dir/2 work properly with archives?

Yes.
 

As a suggestion: I've noticed that the way most people build an escript binary is to glue everything together into a zip file with the appropriate header content and I wonder whether having 'executable (.ez) archives' would be a better approach? Perhaps this works today and I haven't figured it out? What are peoples thoughts about this?

The current trick with the shebang header works on many platforms.
It would however be nice with executable archives (without any
shebang), but these solutions tend to be more platform specific
(non-portable).

/Håkan

[Kenneth Lundin]

2012/3/6 Håkan Mattsson <h...@tail-f.com>

On Mon, Mar 5, 2012 at 23:54, Tim Watson <watson....@gmail.com> wrote:

Hello list,

I have a few questions about support for .ez archives, and one or two suggestions as well.

- is a timeline for fully supporting them?
- does the -include_lib(...) directive now work (in R15) when the headers are in an archive?

Björn, is there a timeline?

There is no timeline and it is even questionable if it should ever work. What is the argument

for having the include files inside the archive?

Include files are for off-line use during the build process. 

The ebin directory is for runtime use and the contents could be inside an archive.

The privdir is also for runtime use but must be possible to open from the OS and is therefore not suitable to have inside the archive.

/Kenneth Erlang/OTP, Ericsson 

 

- does code:lib_dir/2 work properly with archives?

Yes.
 

As a suggestion: I've noticed that the way most people build an escript binary is to glue everything together into a zip file with the appropriate header content and I wonder whether having 'executable (.ez) archives' would be a better approach? Perhaps this works today and I haven't figured it out? What are peoples thoughts about this?

The current trick with the shebang header works on many platforms.
It would however be nice with executable archives (without any
shebang), but these solutions tend to be more platform specific
(non-portable).

/Håkan

[Ulf Wiger]

On 6 Mar 2012, at 13:02, Kenneth Lundin wrote:

There is no timeline and it is even questionable if it should ever work. What is the argument

for having the include files inside the archive?

I ran into this problem, and made an "OTP patch" for it:

http://article.gmane.org/gmane.comp.lang.erlang.general/52060

(Don't think I ever submitted it as a patch, since Håkan and I disagreed on some of the implementation details).

I wrote: "The idea (see http://github.com/esl/run_eqc) was that it should be possible to deliver an application as a single archive, supporting command-line operations, interactive code loading, and compilation including hrl files, parse_transforms etc., without having to unpack the archive."

Philip Stanhope wrote: "Very cool. I can see how this will make deployments very easy."

http://article.gmane.org/gmane.comp.lang.erlang.general/52061

Dizzy writes: "+1 on this feature -- it opens some very interesting doors."

[Kenneth Lundin]

What I meant was that I am not convinced that it is necessary and very useful to have the include files

inside an archive (while using them).

But it might be possible to convince me :)

/Kenneth

[Ulf Wiger]

On 6 Mar 2012, at 15:03, Kenneth Lundin wrote:

What I meant was that I am not convinced that it is necessary and very useful to have the include files

inside an archive (while using them).

But it might be possible to convince me :)

In the case of run_eqc, the idea was to be able to have either PropEr or EQC Lite available as zipped archives. This worked beautifully, at least in terms of executing them from within the zip file. However, in order to compile the test code that is supposed to be executed by PropEr/EQC, you need the include file, and this became the _only_ reason why the archive had to be unpacked. And given that you _had_ to extract the .hrl, there wasn't much point in not also extracting the beam files. 

But this made the whole idea of run_eqc less attractive, so I put the project on ice.

In this setting, there is really not much difference between run-time and compile-time. Running EQC without being able to first compile with the help of eqc.hrl is fairly useless.

BR,

Ulf W

[Anders Nygren]

On Tue, Mar 6, 2012 at 8:03 AM, Kenneth Lundin <kenneth...@gmail.com> wrote:
> What I meant was that I am not convinced that it is necessary and very
> useful to have the include files
> inside an archive (while using them).
>
> But it might be possible to convince me :)

Yaws and gettext requires some .hrl files for compiling dynamic pages.
I think there may be other cases where code is compiled during runtime
where it is useful to have access to the .hrl files.

/Anders

>
> /Kenneth
>
>
> On Tue, Mar 6, 2012 at 1:20 PM, Ulf Wiger <u...@feuerlabs.com> wrote:
>>
>>
>> On 6 Mar 2012, at 13:02, Kenneth Lundin wrote:
>>
>> There is no timeline and it is even questionable if it should ever work.
>> What is the argument
>> for having the include files inside the archive?
>>
>>
>> I ran into this problem, and made an "OTP patch" for it:
>>
>> http://article.gmane.org/gmane.comp.lang.erlang.general/52060
>>
>> (Don't think I ever submitted it as a patch, since Håkan and I disagreed
>> on some of the implementation details).
>>
>> I wrote: "The idea (see http://github.com/esl/run_eqc) was that it should
>> be possible to deliver an application as a single archive, supporting
>> command-line operations, interactive code loading, and compilation including
>> hrl files, parse_transforms etc., without having to unpack the archive."
>>
>> Philip Stanhope wrote: "Very cool. I can see how this will make
>> deployments very easy."
>>
>> http://article.gmane.org/gmane.comp.lang.erlang.general/52061
>>
>> Dizzy writes: "+1 on this feature -- it opens some very interesting
>> doors."
>>
>>
>
>

[Tim Watson]

On 6 Mar 2012, at 14:42, Anders Nygren wrote:

On Tue, Mar 6, 2012 at 8:03 AM, Kenneth Lundin <kenneth...@gmail.com> wrote:

What I meant was that I am not convinced that it is necessary and very

useful to have the include files

inside an archive (while using them).

But it might be possible to convince me :)

Yaws and gettext requires some .hrl files for compiling dynamic pages.
I think there may be other cases where code is compiled during runtime
where it is useful to have access to the .hrl files.

/Anders

In general, I think .ez archives are an ideal unit of packaging and distribution (as dependencies) and therefore making them fully self contained is very helpful. If you specify 'lager >= 1.0.2' as a dependency and all I have to do is find and fetch the archive (and check the MD5 and whatnot) then stick it somewhere useful on the machine, the subsequent build (handling of dependent packages), release assembly and other things just work really nicely. Having to 'manually' pull the includes out before being able to utilise the archive as a compile time dependency is just a bit annoying for tools authors that's all. 

From what Ulf and Anders have mentioned, it sounds like there are other use cases too. Some other thoughts: If you want to be able to bundle an executable archive, are you at the point where you are going to want to bundle multiple applications in it (like it was a release) perhaps? This is certainly how bundled escripts seem to work (in terms of having lots of modules in them) although admittedly the 'application' concept doesn't really exist within those any more.

Also, there is a 'where_is_file' function in the code module, which is quite useful for things that are in 'known' places. What would be a lovely increment beyond that, would be a way to resolve a resource path within a 'known' place inside an archive (such as an application's priv dir and so on) to a binary without having to unpack the whole archive. For loading templates and config files and the like, not having to unpack is really useful and this feels like the kind of thing you'd want to have done properly in one place, rather than lots of people implementing different mechanisms to do the same thing. Sure this isn't completely essential, but it's the kind of thing that avoids 'fiddling around' with the packages post installation and makes the feature more useful and accessible to developers - much like Java's ClassLoader.getClassPathResource(String) or .NET's equivalent. 

I do realise that loading shared libraries from inside of archives isn't feasible though, so there are obviously some cases where you've got to extract at least part of the contents onto the file system.  

Cheers,

Tim

[Steve Davis]

Some .hrl files count as interfaces - a particular case being "public" record definitions made in header files. 

Usually, if a .hrl contains "application private" definitions, I'm inclined to leave it in src. If it includes "public" definitions then I'd drop it in include.

Probably the greatest case for having .ez files is libraries -- where the public api is the most important.

So I'd support +1 the idea that .hrl files should be accessible from within a zipped ez - even if constrained (or maybe should be so?) to just an include directory...

My 2c

/s

[Jack Moffitt]

Whoops, accidentally sent this only to Tim. Resending to the list:

> In general, I think .ez archives are an ideal unit of packaging and
> distribution (as dependencies) and therefore making them fully self
> contained is very helpful. If you specify 'lager >= 1.0.2' as a dependency
> and all I have to do is find and fetch the archive (and check the MD5 and
> whatnot) then stick it somewhere useful on the machine, the subsequent build
> (handling of dependent packages), release assembly and other things just
> work really nicely. Having to 'manually' pull the includes out before being
> able to utilise the archive as a compile time dependency is just a bit
> annoying for tools authors that's all.

+1

Clojure gets something like this for free since it uses Java's jar
system. Jars can contain configuration, source, images, or anything
else. Not only do all the distribution and build tools make use of
this, but the libraries contain things to make reading files from
inside the jar really trivial.

I would love to see something similar in Erlang.

jack.