Preliminary analysis of existing ASDF practices
Juanjo
systems can be integrated with a particular implementation, ECL, but I
hope you will be able to see beyond that.
http://tream.dreamhosters.com/tream/musings/49-lisp/76-analysis-of-existing-asdf-files
Comments better here at comp.lang.lisp
His kennyness
Looks good.
I did not see anything about the whacky choice by default not to load
anything until everything is compiled, forcing endless declarations of
dependencies 98% of which are achieved just by ordering the files a
certain way.
There are other bad choices all over ASDF.
Is it worth saving? Is mk:defsystem in better shape?
kt
Juanjo
> There are other bad choices all over ASDF.
> Is it worth saving? Is mk:defsystem in better shape?
If you have a look at the topmost table, I analyzed 291 distinct
packages from two popular distributions (libcl and clbuild), plus some
others I analyzed by hand. The question is whether we can and should
move all that software to an orthogonal model.
I believe we do not need something completely different. We may
progressively introduce back some rationality into ASDF, making it
closer to mk:defsystem but keeping the extensibility -- it is for
instant intereting to be able to introduce new kinds of components, in
particular for interfacing with other languages (fortran, C, etc).
I started this discussion at the ASDF mailing list, but only to learn
that is the wrong forum: just four people talking, some of them
actually do not caring about other implementations or build models.
What I would like to learn is what people think they have learnt that
ASDF does wrong, things it gets right and to what extent the Common
Lisp community is willing to accept any progress along one or another
direction.
Faré
wrote:
I am the current de facto ASDF maintainer.
I became so shortly after I blogged myself into it.
http://fare.livejournal.com/149264.html
My objective with ASDF 2 was merely to package all the progress that
happened since various versions of ASDF 1 previously spread
to various Common Lisp implementations and distributions,
clean up the current ASDF, make its interface more portable,
and make it upgradable, configurable and releasable so that further
development become possible. Another side objective was also to
introduce some configuration mechanism which I could share with XCVB
and which I hope would become universal amongst Common Lisp build
systems.
Amongst the strong constraints of ASDF 2 was that we should be 100%
backwards compatible, especially since ASDF 1 users have no means of
matching their system definitions to ASDF versions. The ASDF mailing-
list
is indeed NOT the place currently to seriously discuss changing model.
However, once ASDF 2 is released, and thanks to its better ASDF
versioning,
it is not unconceivable that a future ASDF 3 should introduce
incompatibilities allowing for a better build model.
That said, I am also the author of XCVB, and I believe that
it provides a better starting point for developing
a new, incompatible, build system for Common Lisp.
Unlike ASDF, it is not predicated around the
"load, load, load and dump image model", but around the
"load and compile, load and compile, and link model"
that should appeal to ECL hackers.
XCVB currently doesn't have a good extensibility story except
"hack XCVB sources themselves". Sorry about that.
I'd most welcome help designing an API.
But XCVB also doesn't have problems with backwards incompatibility,
it doesn't have the "all must fit in one file" limitation,
the image-based paradigm, the confusion between build-time and run-
time,
the ambiguity and non-determinism in what side effects have or haven't
happened before any given file is compiled or loaded, etc.
All problems that you'll have a hard time removing from ASDF.
[ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org
]
If all values are relative, then cannibalism is a matter of taste.
— Leo Strauss
His kennyness
> On Apr 13, 7:32 pm, His kennyness <kentil...@gmail.com> wrote:
>> There are other bad choices all over ASDF.
>> Is it worth saving? Is mk:defsystem in better shape?
>
> If you have a look at the topmost table, I analyzed 291 distinct
> packages from two popular distributions (libcl and clbuild), plus some
> others I analyzed by hand. The question is whether we can and should
> move all that software to an orthogonal model.
Lisp will live forever and soon rule the world, let's not feel we have
to stick with anything let alone something as shot through with problems
as ASDF.
The beauty is Dr McCarthy's main contrib: code as data. Fix ASDF any way
needed and then write one simple parser that can work off existing .asd
files (with more or less hinting, perhaps).
>
> I believe we do not need something completely different. We may
> progressively introduce back some rationality into ASDF, making it
> closer to mk:defsystem but keeping the extensibility -- it is for
> instant intereting to be able to introduce new kinds of components, in
> particular for interfacing with other languages (fortran, C, etc).
>
> I started this discussion at the ASDF mailing list, but only to learn
> that is the wrong forum: just four people talking, some of them
> actually do not caring about other implementations or build models.
>
> What I would like to learn is what people think they have learnt that
> ASDF does wrong, things it gets right and to what extent the Common
> Lisp community is willing to accept any progress along one or another
> direction.
Gets right? I hear it handles different lisp implementations effectively.
kt
Juanjo
> The beauty is Dr McCarthy's main contrib: code as data. Fix ASDF any way
> needed and then write one simple parser that can work off existing .asd
> files (with more or less hinting, perhaps).
Point is not even that can be done 100% safely for all ASDF files out
there and for others that people write. It is an illusion to try to
bring a new system out of the blue without some cooperation from
developers
As an example, if you have a look at the code people have added to
their ASDF files (*) you will find #. reader evaluated forms and a lot
of other toplevel forms that only work when some dependencies have
been loaded prior to that or when the value of the current directory
is the appropriate one (sic). In the end there is going to be so much
ad-hoc in the rules to process all that information that it ends up
being quite a lot of manual work.
Juanjo
(*) See the failures in the html file to see the errors that happen
when one tries to parse *.asd files with simple READ.
Juanjo
> I am the current de facto ASDF maintainer.
> I became so shortly after I blogged myself into it.http://fare.livejournal.com/149264.html
No intention to make this a campaing for the position. I just want to
take the fruitless discussions out of the ASDF developers list and
into the public. Decisions on the future of build systems should be
based on feedback from developers, not only on trying to get them move
to the next big thing.
> My objective with ASDF 2 was merely to package all the progress that
> happened since various versions of ASDF 1 [...]
> Amongst the strong constraints of ASDF 2 was that we should be 100%
> backwards compatible, [...]
> it is not unconceivable that a future ASDF 3 should introduce
> incompatibilities allowing for a better build model.
> That said, I am also the author of XCVB, and I believe that
> it provides a better starting point for developing
> a new, incompatible, build system for Common Lisp.
The only way to preserve 100% compatibility is to stay in the past.
This is a fact. Moreover when none of ASDF contributors actually
understands fully the main function that drives ASDF, TRAVERSE. What
matters right now is 1) getting things to work and 2) making the chain
as painless as possible.
Step 1) can be ensured by gathering the 1000 software components out
there and ensuring they build, at least with 99% fidelity. Step 2) is
achieved by carefully dropping unneeded features, bloat and
complexity.
You must know how hard it is to push a new build system through the CL
community, specially when you want to introduce different ways of
organizing and making dependencies explicit.
http://comments.gmane.org/gmane.lisp.cffi.devel/1781?set_blog_all=yes
That said, I have read XCVB's manual and I feel it introduces an
unneeded level of complexity in it all. Parallel compilation is nice,
but forcing the use of makefiles, permitting the developers to scatter
the dependencies of their software over different files, and relying
on external drivers, plus the size of the software itself -- ASDF
could and should be so lean! --, does not really convince me.
Right now ASDF contains four components
1- The notion of system definition, components and its implementation
using CLOS
2- The infrastracture for locating, parsing and connecting
dependencies in *.asd files
3- The logic for deciding how to chain operations based on the
dependencies.
4- An interface to the filesystem for storing and locating files.
What XCVB does is fixing 3 by restricting the family of operations to
2 and imposing certain restrictions in the dependencies, but it alters
1 and 2 by changing the syntax of dependencies and hiding the process
from the user. Furthermore it replaces the interface with Makefiles
and makes filesystem dependencies explicit (rules may contain paths).
Not having the extension model of ASDF it probably can not handle
other components, such as compiled files generated for a FFI, or other
stuff. Moreover, it has not faced the problems of deciding a good
design for an extension model, so it is not free from making the same
mistakes as ASDF.
XCVB faces a similar problem as the one I described when integrating
with ASDF systems. If I interpret the manual correctly, it can not
really handle them, but relies on ASDF
http://common-lisp.net/project/xcvb/doc/README.html#xcvb-builds-depending-on-asdf-systems
and expects people to gradually convert their ASDF systems to XCVB
format using tools XCVB also provides. In other words, it still needs
them to fix the problems I pointed out.
Now the question is whether it pays of to "fix" ASDF, we jump on the
XCVB wagon or we wait for some kind of illumination to write the next
big thing. What is wrong with being lazy and
* Give people tools to write better *.asd files, side-effect free.
* Restrict the set of operations that ASDF supports to a few: compile,
load & test
* Simplify the way components are written, perhaps adding some
syntactic sugar.
* Create an API to load and traverse the set of systems and their
dependencies.
* Replace the routine that drives ASDF, TRAVERSE, with something
everybody understands.
* Test it works 99% of the cases with the huge set of software we have
around.
* Write makefile generators in the XCVB fashion, if they are useful
for somebody.
* Write scripts that can be distributed with libraries so that one may
type
./asdf-configure
./asdf-install
and get a library built and installed the proper way.
The hardest part is changing the core, the logic which chooses what to
do and when. Maybe it makes sense to consider the benefits of the old
MK:DEFSYSTEM model where compilation was an optional step and thus
files relied not on compile-time side effects but on the order in
which files were loaded. How many existing systems would break if we
reinterpreted ASDF files that way?
Mario S. Mommer
Juanjo <juanjose.g...@googlemail.com> writes:
> On Apr 13, 7:32 pm, His kennyness <kentil...@gmail.com> wrote:
>> There are other bad choices all over ASDF.
>> Is it worth saving? Is mk:defsystem in better shape?
I think maybe that would be better.
> We may progressively introduce back some rationality into ASDF, making
> it closer to mk:defsystem but keeping the extensibility -- it is for
> instant interesting to be able to introduce new kinds of components, in
> particular for interfacing with other languages (fortran, C, etc).
It does not do that at all well, and with mk:defsystem you could do that
too. Maybe ASDF does it better than mk:defsystem, but I suspect it just
does so differently or even worse.
There is a .asd file for matlisp floating around that did take a lot of
work but does not really work well for obscure reasons.
> What I would like to learn is what people think they have learnt that
> ASDF does wrong,
What ASDF does wrong is to implement a crippled functional programming
language whose domain is loading software in a lazy-evaluation kind of
way. And then it presents a pretty baroque interface for "overloading"
the operators, so to speak, which you do by defining classes and methods
and keyword args and so on and so on. This is bad, because one can spend
a lot of time figuring out how to do things that would be trivial when
done in an ad-hoc fashion with a proper programming language.
ASDF is something of a Turing tar pit.
(As to the original authors, well, nobody complained about ASDF for a
long time, so no one should judge them.)
> things it gets right
What it gets right, IMO, is to come with an accepted and more or less
adhered-to way of specifying how to load code, and how to put that in
one file, and make that information available to the CL
implementation. There is a lot of irregularity in the details, but at
least that pattern is adhered to. This has made a lot of things possible
that were not possible before (asdf-install, etc).
You should be aware of the alternatives, which were having ad hoc
loaders that did whatever they liked. Like, files loading other files
that then loaded other files. There are examples of that around in old
repositories. ASDF and mk:defsystem might not be perfect, but they
brought some discipline into the issue.
> and to what extent the Common
> Lisp community is willing to accept any progress along one or another
> direction.
Right now it seems that progress will encounter little resistance.
Pascal J. Bourguignon
> What I would like to learn is what people think they have learnt that
> ASDF does wrong, things it gets right and to what extent the Common
> Lisp community is willing to accept any progress along one or another
> direction.
My use of ASDF is rather limited.
- One thing I find wrong as a developer is that I would have to write
the dependencies. I don't. I write programs to sort out the
dependencies automatically and to generate the ASD files
automatically. If the ASD file is a byproduct, is it really needed?
- As a user, ASDF is usable. But obviously, it should provide a
simplier API than OPERATE for users, everyone is writing his own
asdf-load function...
- Another point I had difficulties with, as a user, are the warning that
considered as bugs, and preventing compilation/loading of an ASDF
system. (It may be relevant only to older versions of ASDF and/or
SBCL, I've not tried the newest versions).
(For my own projects, I usually stay with the loader.lisp approach).
--
__Pascal Bourguignon__
Tim X
>
> What I would like to learn is what people think they have learnt that
> ASDF does wrong, things it gets right and to what extent the Common
> Lisp community is willing to accept any progress along one or another
> direction.
I think this is the ideal starting point. It may also be worthwhile
asking what people like and dislike about other approaches, though ther
is a danger of information overload.
From the many discussions I've read on this group regarding
package/library management, the needs of developers, end-users,
implementations and distributions (both CL and OS level), the one thing
that seems to be common is a general agreement that the current situation
needs improvement. However, the one thing we don't seem to have is any
consensus on exactly what form that improvement needs to have.
While I suspect this will be an extremely dificult thing to define, the
potential benefits from doing so are great. However, trying to
do so with reference to an existing approach, regardless of whether it
is ASDF or cl-build or mkdefsystem or whatever, is likely to be counter
productive. This is because
1. It tends to constrain our thinking and possible solutions to how ASDF
is structured and what fits in with its architecture.
2. Introduces potentially distracting emotional baggage. Some people
love ASDF while others hate it.
3. I suspect there is considerable variation in what people beleive ASDF
is supposed to do and for whom.
4. Can overlook alternative approaches used in different systems.
I have seen a number of attempts to produce alternative system
definition frameworks on this list. Nearly all of them have had some
significant benefits over existing solutions in at least one particular
feature area, but generally have failed to address requirements in many
others. As a consequence, they have failed to gain any real traction and
tend to disappear into obscurity. I think this is partially due to
different perspectives on exactly what problem is being addressed and
partially the natural consequence of individuals 'scratching their own
itch'.
I wonder if we could obtain a better outcome if we first tried to obtain
more consensus on exactly what functions/features we want - lets say the
top 10. This should be at a fairly high level of abstraction. Then it
should be possible to prioritise things and then investigate how other
systems have addressed the requirement, identify what is good and what
is bad and then design a better solution.
The main issue I can see with this approach is it does run the
danger of decision by committee and could either result in an overly
complex over designed mess that never gets realised or something that is
so heavy-weight nobody wants to use it or is too limited and nobody
finds it any more useful than what they already have.
What is really needed is a strong personallity who can
extract the main themes from all the noise, who is able to push through
an initial solid design (and probably initial implementation) and who
has the character to keep things on track and focused when dealing with
a community that is somewhat more opinionated and forceful than most.
They need to be very experienced with CL, have developed a good level of
respect within the community, recognise the dynamics that can be
involved in such a project, have the confidence to make some difficult
decisions and have the emotional maturity to deal with the crap that is
almost certain to occur from time to time.
Juanjo, from the little I can tell from your posts, your initial efforts
in this area and your track record with respect to work and
contributions, you may well be the one who could do this (either that or
you are an insane madman!). My only comments are
1. Your analysis of ASDF is a good starting point. However, I think you
need to be careful regarding some of your phrasing as it can come across
as an emotional rather than a valid technical criticism. It also runs a
risk of encouraging emotional rather than technical based responses.
Stating that a feature of ASDF is 'stupid' can result in responses that
are like 'No, your stupid' or worse yet, responses from people who feel
you have somehow stated they are stupid.
2. I would recommend being very careful about using examples. A poorly
chosen example often distracts people from the real argument and ends up
leading discussion up the wrong path. Using Debian as an
example of the difficulty of delivering pre-compiled packages is
possibly a bad choice as it seems to go against the underlying Debian
philosophy wrt CL libs and things like the clc-controller etc. I find a
concrete example, such as a sample asdf file with bad forms or dangerous
side effects or psudo code of how it could be done often works better as
it focuses on the specific issue and avoids distracting side issues,
such as Debian CL package policy.
In general, I agree with all your criticisms of ASDF. At a high level,
it promises a lot, but doesn't quite deliver. My experiences seem to
have been similar to yours in that when it has failed, generally, this
has been due to 'unusual' uses or definitions in the asdf files etc.
I have two main requirements. In order of priority, these are;
1. Ability to easily define dependencies in systems I am developing. In
general, ASDF does this pretty well provided I don't include other libs
with unusual asdf forms or complex dependency graphs. The problem is, I
tend to only find this out when everything blows up and then I've got to
clean it up by hand. Of course, the types of apps I've been developing
are for my own use (or a limited well defined workgroup) and I've not
been concerned with multi-implementation support etc, so my requirements
are simple.
2. I would love a system similar to CPAN. I'd love to be able to just
issue the command CLAN install <some-lib> and hav it identify any
dependencies not available locally, download all necessary sources,
build them, run tests and if all goes well install everything. If it
fails, I want it to clean up and remove everythig it did and not leave
half completed bits of crap scattered everywhere. I also like having the
ability to override some steps. For example, sometimes, I know some
tests will fail because they deal with components I'm not interested in
or because I have an atypical setup and the tests just won't cope with
that etc.
The other requirement I'd find useful wold be the ability to easily
define dependencies so that I could add my libs to an archive network
without having to jump through too many hoops. However, once we have 1
and 2 above, this one will probably jus be a natural consequence.
Tim
--
tcross (at) rapttech dot com dot au
Juanjo
> Juanjo <juanjose.garciarip...@googlemail.com> writes:
>
> > What I would like to learn is what people think they have learnt that
> > ASDF does wrong, things it gets right and to what extent the Common
> > Lisp community is willing to accept any progress along one or another
> > direction.
>
> I think this is the ideal starting point. It may also be worthwhile
> asking what people like and dislike about other approaches, though ther
> The main issue I can see with this approach is it does run the
> danger of decision by committee and could either result in an overly
> complex over designed mess that never gets realised or something that is
> so heavy-weight nobody wants to use it or is too limited and nobody
> finds it any more useful than what they already have.
It depends on how one sets up the path. If the goal is
1) Simplify ASDF till you can see the bare bones
2) Build tools on top of it
you can get something that is not too complex by design.
> Juanjo, from the little I can tell from your posts, your initial efforts
> in this area and your track record with respect to work and
> contributions, you may well be the one who could do this (either that or
> you are an insane madman!). My only comments are
>
> 1. Your analysis of ASDF is a good starting point. However, I think you
> need to be careful regarding some of your phrasing as it can come across
> as an emotional rather than a valid technical criticism.
Having experienced this myself in the last weeks. I am not a
diplomatic person, just a nerd who does Physics and programming, and I
can be rather emotional myself, specially when entering debates which
just lead to opinions and not to facts. I do not think I would qualify
for the person with leadership that you mentioned before and it was
not my intention to put myself forward as such, but rather stimulate a
more rational and more open discussion.
> 2. I would recommend being very careful about using examples.
Yes, I found this myself. If I start talking about ECL and how we
build software people just disregard the rest of the discussion. If I
start talking about delivering *.deb or *.rpm, just the like. If I say
anything about the load & dump development model, people quickly
disconnect and forget the rest of the debate.
It is very tough and discouraging, and one of the reasons I am
pursuing this is just because ECL's progress is impeded by the current
status -- hey, I can not even properly automate testing of existing
libraries because of it!
> 1. Ability to easily define dependencies in systems I am developing. In
> general, ASDF does this pretty well provided I don't include other libs
> with unusual asdf forms or complex dependency graphs.
It would be nice to have even artificially cooked examples, for any
effort that aims at fixing / replacing ASDF will encounter them.
> 2. I would love a system similar to CPAN. I'd love to be able to just
> issue the command CLAN install <some-lib> and hav it identify any
> dependencies not available locally, download all necessary sources,
> build them, run tests and if all goes well install everything.
This can be done if *.asd files at some point become really pure
descriptions of the software.
Mario S. Mommer
My experience with CPAN has always been negative. It just did not work
that way, and in fact for me it did not work.
> This can be done if *.asd files at some point become really pure
> descriptions of the software.
Maybe a good starting point would be to find out if such a thing is
possible. I mean, it certainly sounds like a good idea, but given the
many decissions that are conditional on context (load-load-load-dump
vs. alternatives, for example) I am not too optimisitc. Make & co
certainly do not do it that way, I think.
Tim X
> On Apr 14, 6:14 am, Tim X <t...@nospam.dev.null> wrote:
>> Juanjo <juanjose.garciarip...@googlemail.com> writes:
>>
>> > What I would like to learn is what people think they have learnt that
>> > ASDF does wrong, things it gets right and to what extent the Common
>> > Lisp community is willing to accept any progress along one or another
>> > direction.
>>
>> I think this is the ideal starting point. It may also be worthwhile
>> asking what people like and dislike about other approaches, though ther
>> is a danger of information overload. [...]
>> The main issue I can see with this approach is it does run the
>> danger of decision by committee and could either result in an overly
>> complex over designed mess that never gets realised or something that is
>> so heavy-weight nobody wants to use it or is too limited and nobody
>> finds it any more useful than what they already have.
>
> It depends on how one sets up the path. If the goal is
> 1) Simplify ASDF till you can see the bare bones
> 2) Build tools on top of it
> you can get something that is not too complex by design.
Yes, I can see merit in that approach and it probably would avoid some
of the issues I was concerned about. Of course, the risk is that the
bones have cancer and therefore are a bad base to try and build
something better on. I guess we won't really know until we have actualy
stripped away everything and can examine them closely. Perhaps that
would be the point at which a decision is made whether to continue in
that direction or some other one.
>
>> Juanjo, from the little I can tell from your posts, your initial efforts
>> in this area and your track record with respect to work and
>> contributions, you may well be the one who could do this (either that or
>> you are an insane madman!). My only comments are
>>
>> 1. Your analysis of ASDF is a good starting point. However, I think you
>> need to be careful regarding some of your phrasing as it can come across
>> as an emotional rather than a valid technical criticism.
>
> Having experienced this myself in the last weeks. I am not a
> diplomatic person, just a nerd who does Physics and programming, and I
> can be rather emotional myself, specially when entering debates which
> just lead to opinions and not to facts. I do not think I would qualify
> for the person with leadership that you mentioned before and it was
> not my intention to put myself forward as such, but rather stimulate a
> more rational and more open discussion.
>
My intention is to encourage you, but in a realistic light. We are
unlikely to find anyone with all the attributes I feel are important and
of course, this is just my opinion anyway. Unfortunately, too often the
person who is best able to do a task like this is frequently someone who
is very reluctant to take it on. Conversely the person who really wants
this sort of role is too often the wrong one for the task!
Of course, you do have a few of the most important ingredients -
enough interest and drive to put in an initial analysis, write it up and
put it out there for comment and it is something that represents a
'personal itch' due to its impact on ECL.
Maybe, with some support form others, you could kick things off and hope
at some point someone will be prepared to pick it up and run with it. I
would certainly be willing to try and assist where I can, but I don't
feel I have enough technical experience with CL yet to really cotribute
much at this early stage.
>> 2. I would recommend being very careful about using examples.
>
> Yes, I found this myself. If I start talking about ECL and how we
> build software people just disregard the rest of the discussion. If I
> start talking about delivering *.deb or *.rpm, just the like. If I say
> anything about the load & dump development model, people quickly
> disconnect and forget the rest of the debate.
>
Yes, its extremely difficult. For some reason, it also seems to be a
bigger problem within this community. I'm not sure why, but I am
frequently amazed by the number of threads that start off dealing with
something interesting and then rapidly degrade into opinions about
matters that were only of partial or passing relevance to the initial
topic and which usually cannot be resolved one way or the other.
> It is very tough and discouraging, and one of the reasons I am
> pursuing this is just because ECL's progress is impeded by the current
> status -- hey, I can not even properly automate testing of existing
> libraries because of it!
>
Thats the itch you just have to scratch :-)
>> 1. Ability to easily define dependencies in systems I am developing. In
>> general, ASDF does this pretty well provided I don't include other libs
>> with unusual asdf forms or complex dependency graphs.
>
> It would be nice to have even artificially cooked examples, for any
> effort that aims at fixing / replacing ASDF will encounter them.
>
Actually, thats a pretty good idea. A collection of difficult/unusual
asd files and custom loaders etc would provide some very valuable data.
It could help identify those difficult edge cases and could make up the
basis for a good set of tests etc.
I will keep an eye out for any 'unusual' asd files I coame across.
I would be careful about artificially constructed ones. We have some
very clever and deep thinkers in this community who could undoubtably
come up with some hair curling 'theoretical' asdf examples. However, a
complex mind melting asdf example is of little practicle use if it never
actually arises in reality or if it only occurs extremely rarely. Such
examples are likely to only cause distraction and discouragement.
>> 2. I would love a system similar to CPAN. I'd love to be able to just
>> issue the command CLAN install <some-lib> and hav it identify any
>> dependencies not available locally, download all necessary sources,
>> build them, run tests and if all goes well install everything.
>
> This can be done if *.asd files at some point become really pure
> descriptions of the software.
I agree. If 'lesser' languages are able to do it, then CL should be able
to show them how it could have been done better!
Tim X
> Juanjo <juanjose.g...@googlemail.com> writes:
>> On Apr 14, 6:14 am, Tim X <t...@nospam.dev.null> wrote:
>>> 2. I would love a system similar to CPAN. I'd love to be able to just
>>> issue the command CLAN install <some-lib> and hav it identify any
>>> dependencies not available locally, download all necessary sources,
>>> build them, run tests and if all goes well install everything.
>
> My experience with CPAN has always been negative. It just did not work
> that way, and in fact for me it did not work.
>
That is interesting. In the 10+ years that I have used it, I've only had
a couple of times where it has let me down and without exception, that
was due to bad decisions on my part. This is not to say I think its
perfect. There are a few issues I've learnt to watch out for, but in
an overwhelming majority of times, it has worked without a hitch. I
frequently find a search to identify possible modules, together with a
bit of background searching to see if anyohe has had issues with those
modules and then a simple cpan -i <module name> and a few minutes later
I'm coding with that module.
Contrast this with the current CL situation, where it has taken me much
much longer to even find a possible CL candidate, a lot more time
getting it and getting it installed and then finding it doesn't do what
I wanted in the first place or depends on other libs I cannot find or
just won't load/compile etc.
The result has been that if I'm doing something in CL, it will
frequently be faster for me to implement something which I know has
already been done millions of times by others. Unfortunately, while this
is possibly good for improving my CL skills, it means that I often have
to use another language if time is critical. Frequently, this will be
something like Perl as I will probably be able to find a module that
will do a large part of the work for me and allow me to get the job done
within the deadline.
The fact you have had problems with CPAN means you are likely to have
some really valuable input into issues to watch out for, features that
would be useful and things that may be best avoided.
>> This can be done if *.asd files at some point become really pure
>> descriptions of the software.
>
> Maybe a good starting point would be to find out if such a thing is
> possible. I mean, it certainly sounds like a good idea, but given the
> many decissions that are conditional on context (load-load-load-dump
> vs. alternatives, for example) I am not too optimisitc. Make & co
> certainly do not do it that way, I think.
Well, its pretty difficult to determine if something is possible without
trying to do it. Some concrete suggestons on how this could be done
would be useful.
I suspect that any solution will have some limitations and there will be
some situations where it just won't work. However, provided these are
small edge cases, I'm not sure that matters too much. If we have
somehting that will work for over 80% of cases, then we are going to be
a lot better off than we are now. The other 20% (maybe even less, with
luck!), are not going to be any worse off, they just won't benefit as
much. More likely, as our understanding improves, we will chip away at
that last 80% and reduce it over time (though a system that would work
with 100% of the possible cases is unlikely).
What we need is concrete examples, not theoretical or vague 'gut
feelings' regarding where problems may or may not exist. I think the
object should be to improve the current situation, not aim for the
ultimate perfect outcome. Even if we completely fail, we are likely to
add additional knowledge that will be extremely useful for the next
attempt!
His kennyness
> On Apr 13, 10:47 pm, His kennyness <kentil...@gmail.com> wrote:
>> The beauty is Dr McCarthy's main contrib: code as data. Fix ASDF any way
>> needed and then write one simple parser that can work off existing .asd
>> files (with more or less hinting, perhaps).
>
> Point is not even that can be done 100% safely for all ASDF files out
> there and for others that people write.
> It is an illusion to try to
> bring a new system out of the blue without some cooperation from
> developers
I think you are doing the right thing by first measuring community
sentiment. If I am the only one who loathes ASDF, we're stuck with it.
kt
Thomas M. Hermann
wrote:
> ... and some ideas about the future. Some of them affect how existing
> systems can be integrated with a particular implementation, ECL, but I
> hope you will be able to see beyond that.
>
>
> Comments better here at comp.lang.lisp
I use ASDF, but my use is very rudimentary, so I generally don't run
into the issues other people are encountering. Based on the number of
criticisms I've seen, though, apparently I'm just not sophisticated
enough to appreciate the failings of ASDF.
I think it would be useful to get copies of existing systems, contrive
examples, including pathological ones, apply each system to the
example and analyze. The systems that I am aware of are:
- mk-defsystem
- ASDF
- XCVB
- mudballs
The mudballs project has been discontinued and the website is down,
but there is a point of contact for the author in the final Google
groups message.
To put the issue in context, look at the myriad of build systems for
other languages. Personally, I like good ol' Makefiles, but, then
again, maybe I'm just unsophisticated. So, is the goal a lisp only
build system, or will it support defining rules and dependencies for
building other things? I think a solid core that is lisp focused with
'contribs' for other things would be ideal.
~ Tom
Thomas M. Hermann
o.ja...@gmail.com
History has lessons, yet, but it is also a sea of words and
information to drown in.. How about a more theoretical approach, let
each system is just a bunch of nodes in a bunch of different graphs.
Each graph corresponds to an action one can do on the system.
All the asdf replacer needs to do is:(Trying to more clear than i was
in lispforum.org
* Locate keep track of and read the asdforms. (defsystem should next
to the regular files also store *load-pathname*)
* Make conventions for the asdform, how it refers to files and other
systems, and some extras like :description.
* Allow action writers to specify dependencies based on this data.
Probably in many cases exactly the files/systems listed(it should be
the default)
* Allow writers to leave information on the state; for instance,
denoting that it has been compiled, with date/md4 checksums.
This is _completely_not_ a compilation, tool, it is _only_ a
dependency tool, but one that allows for compilation if such action is
defined! So whatever compilation and loading tool we will want can
then use this as a library for its purpose, the library itself not
containing any actions itself.
Advantages of viewing it this way:
* The dependency library can also be useful for other things.
* Multiple people can register the same sort of action, you could
have :load :compile to be the defaults of the implementation/user, or
you could have it :sbcl-compile, :ecl-compile(if the C code can then
be cffi-ed in), etcetera.
* It is simple and clear.
New actions might even add conventions to the asdform above. (But
preferably uses ones that)
Just to prove this is infact not hard, http://ojasper.nl/src/lisp/dep/dep.lisp,
i suggest something like that, except for better potential clearness
of dependencies between the files.(Like asdf provides) If you think it
can't do something, we'll have to look at that, of course. But i see
no real limitations. It is also not complete, no storing of the
systems or data submitted to them yet, and one would also like to
document the conventions used in the Def-a-system form. Maybe one
would also like to give the user some tools to indicate where he has
been already and such.
PS,BTW there is an automatic .asd file creation system in my project
http://lisp-editor.berlios.de/, but i am not satisfied about it, i
should try it on the project itself. And it sillilly writes directly
to file instead of making the defsystem form and then writing that to
file. It works by expression-scanning the files, and then making an
system for each of the packages based on :use and what other external
symbols it found within the package.
Juanjo
complex.
1) Dependencies should be different depending on the "action". That
complicates things quite a bit.
2) Some actions trigger other actions (:compiled -> :load or :load -
> :compile). That is even more complicated.
All this complexity, which does not exist in defsystem, plus the
sophisticated and too powerful CLOS API for building additional
operations and components, is what makes ASDF both powerful and
incomprehensible.
Nathan Froyd
wrote:
> Now the question is whether it pays of to "fix" ASDF, we jump on the
> XCVB wagon or we wait for some kind of illumination to write the next
> big thing. What is wrong with being lazy and
>
> * Give people tools to write better *.asd files, side-effect free.
This is the big sticking point in switching to something else. There
are
good reasons why side-effectful bits are useful in an .asd file.
Custom
component definitions and specializing operations on those components
are what come immediately to mind
> * Restrict the set of operations that ASDF supports to a few: compile,
> load & test
Please don't do this. There are very good reasons to support more
than
three operations on a system; I've written at least two custom
operations
for my own personal use (building tar files and autogenerating
documentation);
I'm sure you can think of more.
> * Simplify the way components are written, perhaps adding some
> syntactic sugar.
> * Create an API to load and traverse the set of systems and their
> dependencies.
This would be something useful standalone; coaxing a list of files out
of ASDF can be an exercise in frustration. Doubly so if you want
system-relative names.
> * Replace the routine that drives ASDF, TRAVERSE, with something
> everybody understands.
Method combination would be a good start here. TRAVERSE was written
the way it was because of poor CLOS support from some implementations.
-Nathan
Mario S. Mommer
Juanjo <juanjose.g...@googlemail.com> writes:
> It may just as simple, but some people believe it should be more
> complex.
>
> 1) Dependencies should be different depending on the "action". That
> complicates things quite a bit.
> 2) Some actions trigger other actions (:compiled -> :load or :load -
>> :compile). That is even more complicated.
...right.
IMO it is better to write an ad-hoc loader with a good library to asist,
and conforming to a realistic and practical set of best practice
guidelines. The complicated cases of system definition are best handled
programatically anyway. At least it is more natural.
>
> All this complexity, which does not exist in defsystem,
Are you sure? IIRC defsystem is fully extensible.
Juanjo
> Here's something on the lines of what I have in mind:
>
> (load-module "src"
> (load-file "file1"
> (load-file "file 2"))
> (load-file "file3"))
>
> where load module would append the directory "src" to some special, and
> the load-file would be implemented in a lazy fashion.
That does not look much simpler to me, specially when you look around
for the kind of dependencies that people are writing, where C, D and E
depend on A, D depends on A B and C, E depends on D an some other
module, and the graph gets rather complex. I just wonder, like you do,
to what extent this all makes sense, other than to enhance the
parallelizability of the build process -- one of the goals of XCVB and
something apparently valued at ITA.
The real problem, though is the other thing you pointed out: when some
of the operations needed to build the system depend on components of
the system itself. Say for instance that I implement a compiler and
the first three files A B and C implement a first simplified version
and the other ones are built with it, D E and F. How do you write such
a thing? We have to write rules, and that means extending ASDF in the
complicated way it provides us, and we find compilcations such as the
fact that those rules must use symbols which are defined in the files
A B and C.
From a conceptual (not yet implementation) point of view I am toying
with a declarative extension to ASDF, adding kind of makefile-type
rules. The idea would be something like what I show below. Instead of
tinkering with methods and so on, we provide some kind of pattern
matching that produces code (code is not evaluated, but concatenated)
which may rely on some macrolet forms, by which ASDF provides most of
the information: where to place files, etc.
By placing the package file in the :asdf-support we allow the rules to
depend on the rest of the system. By storing the rules as just S-expr,
not lisp code, we allow the code to work once the appropriate
dependencies have been loaded. In addition to this, rules may be
installed by other extensions (say a hypothetical cffi-asdf tiny
module) and the ASDF files remail purely declarative -- no more side
effects than package definitions.
mysystem.asd:
(defsystem :my-system
:asdf-dependency ((:file "package") (:file "mysystem.asd"))
:components ((:file "package") (:file "A") ... (:file "D" :depends-
on "D-dsl" "A" "B" "C") ...))
;;; ASDF would stop reading as soon as it reaches one defsystem form
;;; The following is loaded as part of the :asdf-dependency,
;;; since it is loaded _after_ "package" we have the package
;;; available here.
(def-asdf-rule (:my-system :file "D")
(my-system:generate-file (asdf:output-file) (first
(asdf:dependencies))))
Juanjo
> This is the big sticking point in switching to something else. There
> are good reasons why side-effectful bits are useful in an .asd file.
> Custom component definitions and specializing operations on those components
> are what come immediately to mind
I argue that this can be done in a "declarative" manner. That would be
ok because it would also impose a stricter syntax and we can enforce
further requirements.
> > * Restrict the set of operations that ASDF supports to a few: compile,
> > load & test
>
> Please don't do this. There are very good reasons to support more
> than three operations on a system; I've written at least two custom
> operations for my own personal use (building tar files and autogenerating
> documentation); I'm sure you can think of more.
Answer this: assume that ASDF becomes a library and can provide you
with an annotated graph with all the components of a system,
topologically sorted, each one futher annotated with the files it
produces.
Do you really need then the current system that ASDF has or wouldn't
it suffice to just traverse the list, a-la MAPC, with a function that
creates a tar file?
The problem is not extensibility, but that ASDF right now is so
confusing that it does not allow people to realize there are other
ways to do things than with methods. Simplifying the set of operations
would allow creating the previous graph in a simpler form: one graph
for LOAD and one graph for COMPILE, the former one being the one you
actually need because it lists what a running image needs -- compiled
files, documentation, etc.
> > * Simplify the way components are written, perhaps adding some
> > syntactic sugar.
> > * Create an API to load and traverse the set of systems and their
> > dependencies.
>
> This would be something useful standalone; coaxing a list of files out
> of ASDF can be an exercise in frustration. Doubly so if you want
> system-relative names.
This is precisely what I meant and what jasper so clearly exposed:
ASDF is just a hidden database of dependencies. We only need to
provide a clean mean to expose those dependencies and make them as
simple and well specified as possible
> > * Replace the routine that drives ASDF, TRAVERSE, with something
> > everybody understands.
>
> Method combination would be a good start here. TRAVERSE was written
> the way it was because of poor CLOS support from some implementations.
An even better version would not need methods at all! What makes
TRAVERSE so unpredictable is that class dependencies, method
specialization, :around, :after, and other apparent niceties can make
a developer blow everything up -- to name a few things, ASDF recently
changed the way it specializes some methods and it broke ECL's
extensions in various and really misterious ways.
OO is nice, but it is a distraction in this camp.
Juanjo
Jochen Schmidt
> That said, I have read XCVB's manual and I feel it introduces an
> unneeded level of complexity in it all. Parallel compilation is nice,
> but forcing the use of makefiles, permitting the developers to scatter
> the dependencies of their software over different files, and relying
> on external drivers, plus the size of the software itself -- ASDF
> could and should be so lean! --, does not really convince me.
That was also my feelings when I investigated it. I quickly came to the
conclusion "XCVB" must stand for "Xtra Complicated Variant of a
Buildsystem". Actually I think the idea behind XCVB is good, but I fear
that the majority of lisp projects do not face the complexities which
triggered it. More, I think a successor to what ASDF provides today
should really be as simple as possible. A separation of concerns would
certainly a step in the right direction.
A system in its minimal core doesn't need any inherent facilities to
build it or install it or package it - it should only be a
straightforward description which allows others to implement stuff like
building, installing and so on ON TOP of it. It should be readable by
lisp but NOT mandate a complete implementation of CL to interpret it.
It shouldn't even be mandatory to write system descriptions by hand -
there could easily be generators and IDE support to do this.
To me the most simple form of a system consists of a set of CL
components, their dependencies and the external dependencies to other
systems.
Jochen
--
Jochen Schmidt
CRISPYLOGICS
Uhlandstr. 9, 90408 Nuremberg
Fon +49 (0)911 517 999 82
Fax +49 (0)911 517 999 83
mailto:(format nil "~(~36r@~36r.~36r~)" 870180 1680085828711918828 16438)
http://www.crispylogics.com
o.ja...@gmail.com
system :t..) (that is why it doesn't load)
You can do anything with dependencies with the given libs, including
calling different actions back and forth. Compiling/loading would be
something like this: http://www.ojasper.nl/src/lisp/dep/compiler.lisp
I guess one could even make a package to make loader/compilers with
this, showing the files in the correct order to some loader/compiler.
Note that my approach is from 'first principles'; being that if you
need to build something that depends on something, you need to get
what it depends on first. And that you might need to store stuff about
it during, like where you put the compiled result, whether it is
compiled, whether it is loaded.
Apply experience onto first principles. That is how physics works, the
first principles being postulates, and experiments and measurements
being 'experience'. Why? History is a cludge, but it is all the data
you have, however, first principles are simple! So find a way to
combine them. Somewhat like in the horses joke like here
http://www.silvers.org/GenEngineering.html (just a random link, those
jokes are repeated all over.)
I don't mean to imply any sort of dismissiveness of the data, Juanjo,
this really is something that needs attention, and it is good to have
a list of projects that misuse asd files. Not sure that i'd consider
two Defsystems in an asd file an error.(Well, seems like asdf doesn't
read in both, but hardly much of an issue in-principle if it would.) I
don't think regular users should be doing anything else but fill some
kind of Defsystem form, no defmethod-ing or anything like that.
Nathan Froyd
wrote:
> On Apr 15, 10:06 pm, Nathan Froyd <froy...@gmail.com> wrote:
> > > * Restrict the set of operations that ASDF supports to a few: compile,
> > > load & test
>
> > Please don't do this. There are very good reasons to support more
> > than three operations on a system; I've written at least two custom
> > operations for my own personal use (building tar files and autogenerating
> > documentation); I'm sure you can think of more.
>
> Answer this: assume that ASDF becomes a library and can provide you
> with an annotated graph with all the components of a system,
> topologically sorted, each one futher annotated with the files it
> produces.
>
> Do you really need then the current system that ASDF has or wouldn't
> it suffice to just traverse the list, a-la MAPC, with a function that
> creates a tar file?
>
> The problem is not extensibility, but that ASDF right now is so
> confusing that it does not allow people to realize there are other
> ways to do things than with methods. Simplifying the set of operations
> would allow creating the previous graph in a simpler form: one graph
> for LOAD and one graph for COMPILE, the former one being the one you
> actually need because it lists what a running image needs -- compiled
> files, documentation, etc.
I don't understand what creating the graph and then traversing it buys
you versus writing methods that are calling by generic traversal
code. Maybe it's simpler to understand. Maybe. I personally find
ASDF straightforward...it's also possible that upstream ASDF is
significantly more complicated than the ASDF distributed by SBCL,
though.
The whole point here is that if you want to annotate the graph, a
given annotation depends on what operation you're doing. In C terms,
`make doc' is different is from `make program' or `make library'.
Writing the Meant-To-Rule-Them-All system definition facility and
saying, "oh, well, if you want to do anything more than LOAD or
COMPILE or test, you can grovel through a list of all the files in the
system"...that's not very useful. Perhaps you mean to have a way of
providing that sort of extensibility; it's not clear from your
descriptions thus far that you're aiming for that level of
flexibility, though.
I do concur that ASDF it's perfect. The relationship between ASDF and
ASDF-Install needs some help. Writing the main system and the test
system separately is a little strange, but I think it'd be a pain to
write all the methods that would unify them in a single system.
Extensions are a pain to write and dependencies likewise. I could
enumerate problems for a while. But ASDF seems to be Good Enough for
the moment.
-Nathan
RPG
wrote:
> Juanjo <juanjose.garciarip...@googlemail.com> writes:
> > What I would like to learn is what people think they have learnt that
> > ASDF does wrong, things it gets right and to what extent the Common
> > Lisp community is willing to accept any progress along one or another
> > direction.
>
> My use of ASDF is rather limited.
>
> - One thing I find wrong as a developer is that I would have to write
> the dependencies. I don't. I write programs to sort out the
> dependencies automatically and to generate the ASD files
> automatically. If the ASD file is a byproduct, is it really needed?
Should every user of every system do this dependency generation
system
him/herself? If not, perhaps the tool is the right thing, but it's
also correct to distribute the ASDF file.
I'm not convinced that we are yet at the stage of groveling
automatically,
though. Especially not for our systems that interact with external
functions. But I am certainly looking forward to getting this new
tool
and using it!
>
> - As a user, ASDF is usable. But obviously, it should provide a
> simplier API than OPERATE for users, everyone is writing his own
> asdf-load function...
This is being added in ASDF 2. I agree though --- the original ASDF
seemed to
take pride in forcing users to engage with its internals at what seems
an
unnecessary level of detail. Somewhat like making a user crank over
the car
engine, because it's important to know that cranking is part of what
makes
the car start.
These issues should largely go away in ASDF 2.
RPG
> On Apr 13, 6:29 pm, Juanjo <juanjose.garciarip...@googlemail.com>
> wrote:
>
> > * Restrict the set of operations that ASDF supports to a few: compile,
> > load & test
>
> Please don't do this. There are very good reasons to support more
> than
> three operations on a system; I've written at least two custom
> operations
> for my own personal use (building tar files and autogenerating
> documentation);
> I'm sure you can think of more.
I would like to second this. More and more I find that I write
systems that
combine Lisp with other tools. It is common for me to write new
operations in my ASDF files.
It might be reasonable to think harder about what the BUILDING BLOCKS
for
such new operations and components should be --- I think ASDF got a
number of things wrong here --- but not take this ability away.
...
> > * Replace the routine that drives ASDF, TRAVERSE, with something
> > everybody understands.
>
> Method combination would be a good start here. TRAVERSE was written
> the way it was because of poor CLOS support from some implementations.
Method combination, with all due respect, is a red herring here.
What
is needed is not a new software tool, or even a refactoring, but a
specification. The component that generates the build/load/operate
plan
(TRAVERSE) should be specified clearly and (I can almost hear the
howling)
formally using the tools of theoretical computer science, especially
graph
theory. If we can say what TRAVERSE /should do/, which I believe has
everything
to do with computing a topological ordering on a particular graph, it
will
be a lot easier to do it, and to check its correctness when it's
done.
If someone wants to make a successor for ASDF, let's have a RFP for
what
its successor /should/ do, first, instead of just banging together
more
code.
Tim X
I agree and think this exposes one of the reasons we have the situation
we are in. To some extent, ASDF has been so powerful that the individual
has been able to do things which on one had can make their life easier
when you are only looking at a system from a single perspective, but
once you move out into a wider world of possibilities, things get too
complex to manage.
I think there is merit in much of what Jasper has pointed out. the
discussions have also shown that there is a broad cross-section of
differing views as to what exactly ASDF or any system definition
framework is meant to do. It is this lack of consensus, combined with
the flexibility and loose specification that has created much of the
complexity that causes problems with the existing system.
Clearly articulating exactly what the core requirements are and then
going back to first principals as suggested by Jasper would seem to be a
good starting point. Once we have a good base, additional functionality
could be added and would likely be easier to add as it would have more
defined constraints to work within.
The hard part could be getting consensus on what are the core
requirements. My view would be to keep these as simple and basic as
possible. Concise and clear definition of dependencies would seem to be
the most critical base requirement.
Tim X
> Tim X <ti...@nospam.dev.null> writes:
>> m_mo...@yahoo.com (Mario S. Mommer) writes:
>>
>>> Juanjo <juanjose.g...@googlemail.com> writes:
>>>> On Apr 14, 6:14 am, Tim X <t...@nospam.dev.null> wrote:
>>>>> 2. I would love a system similar to CPAN. I'd love to be able to just
>>>>> issue the command CLAN install <some-lib> and hav it identify any
>>>>> dependencies not available locally, download all necessary sources,
>>>>> build them, run tests and if all goes well install everything.
>>>
>>> My experience with CPAN has always been negative. It just did not work
>>> that way, and in fact for me it did not work.
>>>
>>
>> That is interesting. In the 10+ years that I have used it, I've only had
>> a couple of times where it has let me down and without exception, that
>> was due to bad decisions on my part. This is not to say I think its
>> perfect. There are a few issues I've learnt to watch out for, but in
>> an overwhelming majority of times, it has worked without a hitch. I
>> frequently find a search to identify possible modules, together with a
>> bit of background searching to see if anyohe has had issues with those
>> modules and then a simple cpan -i <module name> and a few minutes later
>> I'm coding with that module.
>
> I know 0 perl, so I went by what the docs said on how to install
> things. IIRC, I was trying to install one of the mail.yahoo to mbox
> thingies. Neither the stuff packaged with debian or ubuntu, nor the
> straight-from-upstream thing worked. Lots of errors, many pertaining to
> hard-coded paths in whatever makefilelike facility was used, others
> about incompatibility of modules. It just bombed over and over until I
> just stopped. I repeated that exercise twice with a few years appart,
> the last time about two years ago.
>
Sounds like either that was a badly package module, which is unusual,
but not unknown or it was badly written documentation, which is more
common than we would like. In the end, it sounds like you were unlucky
and a victim of the balance between creating a system that
enables/facilitates develpers distributing their modules and provides a
reliable repository that others can use.
>> The fact you have had problems with CPAN means you are likely to have
>> some really valuable input into issues to watch out for, features that
>> would be useful and things that may be best avoided.
>
> Well, as I said, 0 perl here. So I am afraid I can't really say much
> beyond "it did not work for me".
>
How well do you think someone with zero CL would do installing ASDF
based libs/applications?
>>> Maybe a good starting point would be to find out if such a thing is
>>> possible. I mean, it certainly sounds like a good idea, but given the
>>> many decissions that are conditional on context (load-load-load-dump
>>> vs. alternatives, for example) I am not too optimisitc. Make & co
>>> certainly do not do it that way, I think.
>>
>> Well, its pretty difficult to determine if something is possible without
>> trying to do it. Some concrete suggestons on how this could be done
>> would be useful.
>
> ASDF went ahead and tried it, and did a reasonably good job to bring
> things to one standard, so that's what we are complaining about know,
> instead of a lack of standards. Without any sarcasm I'd say we have
> progressed.
I don't think anyone has suggested that ASDF hasn't been useful or
hasn't improved the situation. Rather, I would argue that its time to
look at progressing further. Using the experiences and knowledge
obtained through the use of ASDF and seeing if we can do it better.
One of the criticisms of ASDF is that it is very weak with respect to
any form of standardisation. There is one school of thought that would
argue this is a good thing and even a lispy way to do it. However, there
is another school of thought which would recognise the advantages of a
flexible and only loosely specified system, but also recognises that
this can result in a lot of complexity, particularly when you need to
work with libraries that have very different ways of using ASDF.
My view of this current discussion is to look at ASDF (and possibly
other solutions) to identify what it does well, what it doesn't do so
well and see if we can come up with something even better. This could be
as simple as just defining some basic conventions regarding how to use
it or it could be a total architectual re-engineering and maybe not even
called ASDF.
As yet, I don't think we even have a clear view of exactly what we want
in a system definition framework or even agreement on what we feel ASDF
does well and does not do well.
>
>> What we need is concrete examples, not theoretical or vague 'gut
>> feelings' regarding where problems may or may not exist. I think the
>> object should be to improve the current situation, not aim for the
>> ultimate perfect outcome. Even if we completely fail, we are likely to
>> add additional knowledge that will be extremely useful for the next
>> attempt!
>
> Writing an .asd to compile and link some fortran files and then load the
> resulting .so is IMO already too complicated.
>
> If you want to use a compiler that is written in Lisp, the .asd itself
> cannot be loaded without the compiler being loaded first, because the
> symbols the compiler exports does not exist. That leads to the
> asdf:operate forms at the beginning of .asd.
>
> I think the most important part is really about policy. If software
> loader/builder scripts adhered to some common and good guidelines, and
> implemented the relevant scripts using some common library that made
> things easy to write and read, then we would be done. The general case
> includes people programatically downloading patches and applying them,
> etc, which is simply too much for a defsystem form to express cleanly
> and easily.
>
I think these are exactly the types of issues that need to be looked at.
and discussed. Can we do this better? Is ASDF the right tool/place for
these concerns? Are these high priority or low priority requirements?
etc.
> Here's something on the lines of what I have in mind:
>
> (load-module "src"
> (load-file "file1"
> (load-file "file 2"))
> (load-file "file3"))
>
> where load module would append the directory "src" to some special, and
> the load-file would be implemented in a lazy fashion. This would cover
> the simple cases. It is possible, and IMO better, to do all the
> dependency tracking programatically and sanely, instead of doing it
> using some DSL (defsystem keyword maze) and then going through hoops to
> define and modify the dependencies and operators, etc.
This is the type of use case we need to collect. However, we also need
other use cases and identify exceptions, corner cases etc.
I don't htink we can answer the question as to whether a programatic
approach, a DSL approach or a hybrid approach is better until we
udnerstand the range of possibilities better and we probably won't be
able to define that until we have some consensus on what we expect the
system to do and what we don't expect it to do. A well designed DSL may
not be as difficult or painful as you may expect. Likewise, a clear
understanding of using a programatic approach would likely facilitate
the development of library utilities that would make such a process
easier and help ensure people adhere to a common standard or policy.
My feeling is that we are lagging behind in how we manage this. These
days, we have whole operating systems and other languages that handle
things like dependency management and build processes far more reliably
than is currently the case with CL. As yet, I don't know what the answer
is and I'm not even sure I udnerstand the full nature of the problem.
I don't even expect we will get a perfect or even near perfect solution.
However, I do believe we can improve matters.
Tim X
> On 2010-04-14 00:29:34 +0200, Juanjo said:
>
>> That said, I have read XCVB's manual and I feel it introduces an
>> unneeded level of complexity in it all. Parallel compilation is nice,
>> but forcing the use of makefiles, permitting the developers to scatter
>> the dependencies of their software over different files, and relying
>> on external drivers, plus the size of the software itself -- ASDF
>> could and should be so lean! --, does not really convince me.
>
> That was also my feelings when I investigated it. I quickly came to the
> conclusion "XCVB" must stand for "Xtra Complicated Variant of a Buildsystem".
> Actually I think the idea behind XCVB is good, but I fear that the majority of
> lisp projects do not face the complexities which triggered it. More, I think a
> successor to what ASDF provides today should really be as simple as possible.
> A separation of concerns would certainly a step in the right direction.
>
> A system in its minimal core doesn't need any inherent facilities to build it
> or install it or package it - it should only be a straightforward description
> which allows others to implement stuff like building, installing and so on ON
> TOP of it. It should be readable by lisp but NOT mandate a complete
> implementation of CL to interpret it. It shouldn't even be mandatory to write
> system descriptions by hand -
> there could easily be generators and IDE support to do this.
>
> To me the most simple form of a system consists of a set of CL components,
> their dependencies and the external dependencies to other systems.
>
Yes, I like this approach. Part of where I feel things have fallen down
as been due to trying to do too much all at once. I like the idea of
building up from a good base and that the most fundamental part of that
base is some way of expressing/representing dependencies. Once we crack
that nut, I think we can build more complex functionality. The hard part
is identifying/defining how to best do this and doing so in such a way
that the simple case is still simple and the complex case is no more
complex than it needs to be.
Tim
Faré
>>: Jochen Schmidt
>: Tim X
>>> That said, I have read XCVB's manual and I feel it introduces an
>>> unneeded level of complexity in it all.
OK. How would you simplify it?
One simplification I had in mind would have been that at a deep level,
XCVB, like Make and other build systems, should manage files,
and not abstract entities called Lisp modules.
>>> Parallel compilation is nice,
>>> but forcing the use of makefiles, permitting the developers to scatter
>>> the dependencies of their software over different files, and relying
>>> on external drivers, plus the size of the software itself -- ASDF
>>> could and should be so lean! --, does not really convince me.
Wait! Size of the software?
No, you should compare ASDF to XCVB-master.
XCVB-master is what would replace ASDF.
It is ten times smaller than ASDF
(though I plead guilty to making ASDF three times larger).
That's the only thing that lives in the Lisp image and allows
for interactive compilation and loading of stuff.
The rest of XCVB (not counting dependency libraries)
is only twice as big as ASDF, but does much much more.
It includes tool to migrate from ASDF to XCVB and vice versa,
enforcing and non-enforcing Makefile backends,
the beginning of a standalone parallel backend,
an extensive command-line interface,
support for cross-compilation,
plenty of documentation and error-checking, etc.
You should compare to a hypothetical set of ASDF extensions
that would do as much.
And if you don't use all these features --
they don't impact your Lisp image, unlike they do with ASDF.
XCVB does not force you to use Makefiles. It can use them for you.
In the future (when I don't have to maintain ASDF and can focus on
XCVB),
it will do without them altogether.
There are many ways in which XCVB does suck
but I don't think the points you raise are relevant.
XCVB suckage include:
* an architecture that is too specific to ASDF-like defsystems,
* difficulty to extend it into a general-purpose tool.
* CLOS is too rigid, notably; Sheeple might be better (or not).
* no standalone backend yet - it is waiting on IOlib support for
processes,
support for CFFI grovel and Ironclad readtable extensions to ASDF,
etc.
>> That was also my feelings when I investigated it. I quickly came to the
>> conclusion "XCVB" must stand for "Xtra Complicated Variant of a Buildsystem".
Backward compatibility makes XCVB complex.
Please explain to me
1- how you would do a build system if you could forget backward
compatibility.
2- how you would provide a backward compatibility bridge to the above.
>> Actually I think the idea behind XCVB is good, but I fear that the majority of
>> lisp projects do not face the complexities which triggered it. More, I think a
>> successor to what ASDF provides today should really be as simple as possible.
>> A separation of concerns would certainly a step in the right direction.
>
You want simple? Look at XCVB-Master. As simple as can be.
>> A system in its minimal core doesn't need any inherent facilities to build it
>> or install it or package it - it should only be a straightforward description
>> which allows others to implement stuff like building, installing and so on ON
>> TOP of it. It should be readable by lisp but NOT mandate a complete
>> implementation of CL to interpret it. It shouldn't even be mandatory to write
>> system descriptions by hand -
>> there could easily be generators and IDE support to do this.
>>
Please be more specific. Are you envisioning something like
Smalltalk's
Monticello or something?
For all the Lisp rant about code is data, Common Lisp code is
surprisingly
like files, and very little like data. Moreover, files with complex,
brittle dependencies with each other, rather than declarative pieces
that can be processed as independent units.
Building Common Lisp software these days has all the disadvantages of
building C++ software plus all the disadvantages of building Smalltalk
software, with none of the advantages.
>> To me the most simple form of a system consists of a set of CL components,
>> their dependencies and the external dependencies to other systems.
>
XCVB solves that simple case to perfection.
ASDF does it mostly well (except for how it mismanages system
dependencies).
What are you complaining about?
> Yes, I like this approach. Part of where I feel things have fallen down
> as been due to trying to do too much all at once. I like the idea of
> building up from a good base and that the most fundamental part of that
> base is some way of expressing/representing dependencies. Once we crack
> that nut, I think we can build more complex functionality. The hard part
> is identifying/defining how to best do this and doing so in such a way
> that the simple case is still simple and the complex case is no more
> complex than it needs to be.
>
I'm most open to suggestions on how I could rewrite XCVB in a better
way.
Please explain.
[ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org
]
To authorize all commercial relations between consenting adults.
Juanjo
> Note: in the linked lisp file, there are two mistakes in (def-a-
> system :t..) (that is why it doesn't load)
>
> You can do anything with dependencies with the given libs, including
> calling different actions back and forth. Compiling/loading would be
> something like this:http://www.ojasper.nl/src/lisp/dep/compiler.lisp
This is what ASDF is doing right now, but what ASDF does (and I think
it gets it right as for the goal, but probably not for the means) not
to execute things immediately, but to produce a "plan", a
topologically sorted lists of actions.
> I don't mean to imply any sort of dismissiveness of the data, Juanjo,
> this really is something that needs attention, and it is good to have
> a list of projects that misuse asd files. Not sure that i'd consider
> two Defsystems in an asd file an error.(Well, seems like asdf doesn't
> read in both, but hardly much of an issue in-principle if it would.)
Please understand that my study does not qualify them as "errors" from
the ASDF point of view, but as making the system unusable for
automated inspection, for it is actually hiding the dependencies: we
do not know all the systems that exist unless we read in all the
specifications. The study is about "practices" and knowing how people
use ASDF and how this would interfere with other ways of doing things.
Juanjo
> On Apr 15, 3:06 pm, Nathan Froyd <froy...@gmail.com> wrote:
>
> > On Apr 13, 6:29 pm, Juanjo <juanjose.garciarip...@googlemail.com>
> > wrote:
>
> > > * Restrict the set of operations that ASDF supports to a few: compile,
> > > load & test
>
>
> I would like to second this. More and more I find that I write
> systems that combine Lisp with other tools. It is common for me to write new
> operations in my ASDF files.
>
> It might be reasonable to think harder about what the BUILDING BLOCKS
> for such new operations and components should be --- I think ASDF got a
> number of things wrong here --- but not take this ability away.
Please understand that I do not want to remove extensibility, but I
would like it to be more "rational" in a sense that polluting the
space of methods with actions for which you do not declare any
relation between them (which ones do produce files, how is the result
of their action affecting the system, which actions do result in the
lisp loading things,...) prevent any kind of analysis of the resulting
dependencies.
In other words, by using operations or "classes" that are separate
from the ASDF specification you are preventing any kind of automated
analysis of what you are going to do.
Note that I am the most interested one in getting interoperation with
the outer world (C, C++, Fortran) right! What is ECL but a C library
in the end?
The problem is noticing that there are two different things to do
here: 1) providing means to make lisp systems depend and actually
process external software and data, and 2) making them integrate well
with other build systems (makefiles, other projects, etc).
Role 1) is what is making people right now write new kinds of
components, methods, and other tools that live in the *.asd file
itself and from which it is difficult to extract information such as
the actual set of lisp sources that make up a program. Furthermore
these tools actually have complicated dependencies on the software
they build, making for all sort of ugly workarounds such as creating
packages in the asd file itself, double loading, etc.
Role 2) is more related to exposing the information that ASDF has in a
programmable way, so that one can create makefiles, or allowing you
the developer to gather information about files that have to be
installed, where they live, etc.
o.ja...@gmail.com
weren't possible that way.
> it[asdf] gets it right as for the goal, but probably not for the means) not
> to execute things immediately, but to produce a "plan", a
> topologically sorted lists of actions.
It does seem silly to first gather a list of things to do and then do
them when you can just do it off/hand like this. Although one has to
think about wanting to use use threading later, however this should be
doable by calling the action-onning the lower systems in threads first
and the process of action-on will wait for those to finish as it
depends on it, and then after that, do it's own thing. Pretty much as
it is one now. (Infact ithink that if mapcar had threadding in it, it
would already be doing it like that) (Of course making such an plan of
action list is perfectly possible with an action.)
> Please understand that my study does not qualify them as "errors"
True, they're not exactly errors from the users pov, but they pretty
much seem like some sort of mistake to me if they hamper future
development and creation of other tools to asdf,
I just had a thought, by overloading the load/compile classes and
making appropriate methods with :around, one should be able to make it
download too, or even hook it into stuff like aptitude repos. However,
it needs to be clear where the downloaded code/binaries goes, and how
that is somehow equivalent to a way of downloading manually.(Perhaps
asking for a directory, with a default in user space, at least for
development-mode) Further _any_ sort of repo entry must be dutifully
updated, otherwise it is nothing more than a fancy download tool.
Btw, presuming people will update their asdf version, we could do
backwards compatibility by 1) modifying asdf to be able to load
systems of the new asdf system 2) making the new system load asdf
systems when needed, it can just look in the new first and then the
old: (or (when (get-system name) ..new way..) (when (find-system
name) ..call old way..), it would extensions would also only have to
implement for both old and new systems, or just focus on one of the
two. (One might have to use a special variable to transmit information
through while asdf is doing stuff.)
Extending might be slightly complicated, but i really don't think any
project that isn't specifically about extending has to extend it, and
i think one could even provide another little package which helps that
a little too, or a *system-hook* and a *file-hook* called upon the
systems and files in appropriate order.
> Furthermore these tools actually have complicated dependencies on the software they build, making for all sort of ugly workarounds such as creating packages in the asd file itself, double loading, etc.
I don't really get what could be complicated about the dependencies. I
mean, they're dependencies, it is gonna be a tree. Something like lack
of knowledge of the dependencies, or limitations is likely the
problem.
jos...@lisp.de
> I somehow got the impression that you were saying that load/compile
> weren't possible that way.
>
> > it[asdf] gets it right as for the goal, but probably not for the means) not
> > to execute things immediately, but to produce a "plan", a
> > topologically sorted lists of actions.
>
> It does seem silly to first gather a list of things to do and then do
> them when you can just do it off/hand like this.
Plans have several useful purposes:
* one can generate the plan and look at it while constructing/refining
a system description.
Is the sequence of operations the one I want? Are all necessary
files in the plan?
Are all systems/files that are mentioned in the plan available?
This also helps debugging a DEFSYSTEM implementation. It helps
to see what the DEFSYSTEM will do, without the need to execute the
actions.
* the plan is a more compact representation of what will be done -
than looking at
various system descriptions
* when porting software, I sometimes generate a compile plan and then
walk manually through
the files, fixing obvious problems and execute the plan steps by
hand for each file.
* sometimes during development I want to get rid of one or more steps
of a plan.
In LispWorks I can deselect these unwanted steps in the user
interface.
* if a plan is aborted for some reason, one might be able to execute
the
remaining steps later.
Much of that is for more advanced usage. Many users might just want to
start over and restart the Lisp compiler. It's fast nowadays. In an
IDE with lots of state, restarting is more costly and I want to avoid
that.
<...>
His kennyness
> On Apr 15, 3:06 pm, Nathan Froyd <froy...@gmail.com> wrote:
>> On Apr 13, 6:29 pm, Juanjo <juanjose.garciarip...@googlemail.com>
>> wrote:
>>
>
>>> * Restrict the set of operations that ASDF supports to a few: compile,
>>> load & test
>> Please don't do this. There are very good reasons to support more
>> than
>> three operations on a system; I've written at least two custom
>> operations
>> for my own personal use (building tar files and autogenerating
>> documentation);
>> I'm sure you can think of more.
>
> I would like to second this. More and more I find that I write
> systems that
> combine Lisp with other tools. It is common for me to write new
> operations in my ASDF files.
I smell "make"!
>
> It might be reasonable to think harder about what the BUILDING BLOCKS
> for
> such new operations and components should be --- I think ASDF got a
> number of things wrong here --- but not take this ability away.
The way past this is to stop talking in the abstract and Get On With
It(tm), with the benefit of everyone's experience at hand to guide
decisions.
So one guy does not want these make-ish side-effects firing and one guy
wants make-ish behavior -- sounds like a global build option is born,
and those of you who think defsystem is make need to accept that these
other deals will have to be done within a structure, not by slapping
arbitrary code in a def.
As long as the process is public and keeps in mind the edge cases on
hand in 2010 after fifty years of Lisp, we should do OK.
kt
Tim X
>>>>: Juanjo
>>>: Jochen Schmidt
>>: Tim X
>
>> Yes, I like this approach. Part of where I feel things have fallen down
>> as been due to trying to do too much all at once. I like the idea of
>> building up from a good base and that the most fundamental part of that
>> base is some way of expressing/representing dependencies. Once we crack
>> that nut, I think we can build more complex functionality. The hard part
>> is identifying/defining how to best do this and doing so in such a way
>> that the simple case is still simple and the complex case is no more
>> complex than it needs to be.
>>
> I'm most open to suggestions on how I could rewrite XCVB in a better
> way.
> Please explain.
>
Note that I made no reference to xcb, only on identifying the best way
to represent dependencies that would allow us to build more advanced
functionality. This could well be xcb or some other build system. My
point was that we have various different approaches and no consensus on
how to address the problem and have fragmented how we do things.
I don't think this should be a battle over how one way does x better
than another way. This should be about identifying exactly what is
expected/required and then taking the best ideas and seeing if we can
create something that is even better. The fact we have more than one
system indicates a lack of consensus. We probably need more consensus
before we start debating the merits and weaknesses of any specific
approach.
Juanjo
>
> You want simple? Look at XCVB-Master. As simple as can be.
Fare, this is cheating: xcvb-master does nothing, it just calls a
program that does the actual job, compiling and returning the list of
files to be loaded. It is like factoring all of ASDF in a separate
program and then writing a four lines driver of a complex C++ program
that processes the dependencies and does the job.
Faré
wrote:
Users of CL software shouldn't have to share their image with the
build system, and then either have big in-image versioning
incompatibility problems, or dirt poor build system feature-set
because their build system can't afford to use libraries.
In the future, I'd like to add a distributed system to XCVB,
cryptography, using rich efficient data-structure libraries, etc. Just
can't do that with ASDF.
It's not "cheating". It's "freeing yourself from the self-inflicted
pains of a needless constraint".
[ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org
]
What one person receives without working for, another person must work
for
without receiving. — Adrian Rogers
o.ja...@gmail.com
'send'.
@joswig, as i said, you could just add an action to list the
dependencies. Or one could go the special-var-hooks route instead of
the methods. With *depends-on*, and *file-hook* *system-hook*, let the
default of *depends-on* be as the method i gave, and then: using
http://www.lispforum.com/viewtopic.php?f=2&t=663&start=10#p4206
(defun get-plan (system)
(nconcing (:into collected)
(let ((*file-hook* #'nconc-it) (*system-hook* #'nconc-it))
(act-on system))
collected))
And i am sure there is a way to do this with the current methods too.
So there is no dire situation that the action to get the list may
differ from the plan list from the :get-plan action. So there is no
reason to wait for the entire plan. Even if you'd want to weigh some
kind of priorities in loading, that should be a matter of a threading
library.
I see no reason to think that executing a plan produced this way is a
problem either. Just iterating the list in sequence would load/compile
things in the correct order.(albeit impossible to thread it with such
basic way of outputting.)
o.ja...@gmail.com
a good idea.
Juanjo
> I did not see anything about the whacky choice by default not to load
> anything until everything is compiled, forcing endless declarations of
> dependencies 98% of which are achieved just by ordering the files a
> certain way.
You do not need to declare all compile-time dependencies -- just use
the serial option in which every file (loaded or compile) depends on
all that precede it. And furthermore, whether things are compiled or
not could be easily changed
Mario S. Mommer
Tim X <ti...@nospam.dev.null> writes:
> Juanjo <juanjose.g...@googlemail.com> writes:
>> An even better version would not need methods at all! What makes
>> TRAVERSE so unpredictable is that class dependencies, method
>> specialization, :around, :after, and other apparent niceties can make
>> a developer blow everything up -- to name a few things, ASDF recently
>> changed the way it specializes some methods and it broke ECL's
>> extensions in various and really misterious ways.
>>
>> OO is nice, but it is a distraction in this camp.
Oh oh, that's... heretical! :-)
Seriously though, the problem is not OO but trying to cram everything in
that dependency graph image which, it seems, simply is not a good
metaphor as soon as you are doing something more interesting, at least
not in its current version.
> I agree and think this exposes one of the reasons we have the situation
> we are in. To some extent, ASDF has been so powerful that the individual
> has been able to do things which on one had can make their life easier
> when you are only looking at a system from a single perspective, but
> once you move out into a wider world of possibilities, things get too
> complex to manage.
(let me just interrupt to say that I think that the words being used in
this discussion are signs of trouble: Control is to be imposed, policies
are to be enforced, the power of the individual curtailed, etc)
> Concise and clear definition of dependencies would seem to be
> the most critical base requirement.
I don't think that that is the problem. The problem is identifying what
kind of thing depend on what, and I'd like to propose viewing actions
with side-effects as the things, and the dependencies as being that what
everybody calls a dependency anyway.
For example, consider this very simple rule.
"the compilation of 'file1.lisp' depends on 'macros.lisp' and
'packages.lisp' being loaded".
ASDF and defsystem treat this as dependencies on files, and not
dependencies on the actions themselves.
Instead of saying 'file1.lisp' depends on 'macros.lisp' and
'packages.lisp', and then awkwardly saying how, it would be better to
say these things in a straight forward way. ASDF and defsystem tried the
former approach, and maybe it is time to try the latter.
Juanjo
> Tim X <t...@nospam.dev.null> writes:
> > Juanjo <juanjose.garciarip...@googlemail.com> writes:
> >> OO is nice, but it is a distraction in this camp.
>
> Oh oh, that's... heretical! :-)
Yes, and it feels good.
> Seriously though, the problem is not OO but trying to cram everything in
> that dependency graph image which, it seems, simply is not a good
> metaphor as soon as you are doing something more interesting, at least
> not in its current version.
There is no problem with the graph image. Makefile uses it internally
and a lot of people learned to live with that "primitive" environment.
ASDF can improve over it, though.
> (let me just interrupt to say that I think that the words being used in
> this discussion are signs of trouble: Control is to be imposed, policies
> are to be enforced, the power of the individual curtailed, etc)
Let's not get scared about any discipline. Libraries have to coexist
and just allowing _anything_ is not ok, specially because not
everybody is capable or has the capability of knowing how the
extensions and additions she / he creates will behave in the future...
I for example do not.
ASDF needs an specification. It needs contracts for the extensions to
obey: that they inform the system about the things they do and when
they do it, and what it leads to (creation of files, reading other
stuff, etc)... we will figure out along the way.
Please notice that the reason why we need some control is because
these libraries are going to be built in very different environments
and with very different purposes:
- Maybe in a Windows system where only an interpreter is available.
- Maybe in a Linux with SBCL
- Maybe using ECL to produce a shared library
- Maybe with ABCL to be ran in some server
If we have a formal, well defined specification of minimums, then this
can be implemented with various targets in mind. If on the other hand
one allows everybody to assume they are in any environment, packages
can be created at any time without informing ASDF, libraries are
loaded in the asd file without notifying the dependency, side effects
take place in the asd file which are not propagated to the compiled
files, or arbitrary methods are created which add dependencies on the
fly, then ASDF will only work in a very narrow environment: that of
always running lisp images that load all of it software, piece by
piece.
> ASDF and defsystem treat this as dependencies on files, and not
> dependencies on the actions themselves.
Yes and no: actions depend on files but actions may also depend on
other actions (internally in the ASDF graph). It is just that this
possibility is not exposed in the DEFSYSTEM grammar.
I am very well aware of this situation and if you peek my previous
emails that is why I introduced the notion of "rules" and makefile-
style programming.
> Instead of saying 'file1.lisp' depends on 'macros.lisp' and
> 'packages.lisp', and then awkwardly saying how, it would be better to
> say these things in a straight forward way. ASDF and defsystem tried the
> former approach, and maybe it is time to try the latter.
I do not agree with this. The current situation is ok for a library
with a set of existing lisp files for one simple reason: one may come
with a set of rules for "loading", "compiling" and the like, which may
be ok for a lisp where things are loaded and ready to be dumped, but
if I want to reuse your library for building a set of object files to
be linked together in ECL then it is not going to work just as nicely.
In this case, a simple declarative description with a fixed set of
rules and different backends (SBCL, ECL, CCL, ABCL...) is a powerful
thing.
The problem, though, begins when one wants to have things that are not
contemplated by the Common Lisp specification and by ASDF as it stands
now:
* Automatically generated lisp files
* Automatically generated documentation
* Binary files for other languages (C++, C, fortran)
* Shared libraries in those languages
* Dependencies of compiled files on the previous things
* ...
What we need among other things is the possibility to
* Add rules that express how to create some of the elements that
participate in a system
* Create rules that are executed for certain actions
(:install, :uninstall, ...)
* Implement reusable components that we know how to compile and
*LOAD*, but which use other languages
Note that all this can not be done with arbitary, hand written CLOS
methods: ASDF has to be informed about how to do those tasks
declaratively so that it can incorporate the knowledge in other
contexts, such as when building an executable instead of dumping an
image.
Nathan Froyd
wrote:
> The problem, though, begins when one wants to have things that are not
> contemplated by the Common Lisp specification and by ASDF as it stands
> now:
> * Automatically generated lisp files
ASDF already does this, or at least there exist systems using ASDF
that do this. See $SBCL/contrib/sb-grovel/
> * Automatically generated documentation
I don't see why this is particularly hard, could you explain?
> * Binary files for other languages (C++, C, fortran)
Very doable. I don't have an example offhand, but earlier
incarnations of $SBCL/contrib/sb-bsd-sockets/ were perfectly
comfortable compiling C files as part of a system
> * Shared libraries in those languages
Ditto.
> * Dependencies of compiled files on the previous things
Ditto.
> * ...
>
> What we need among other things is the possibility to
> [...]
> * Implement reusable components that we know how to compile and
> *LOAD*, but which use other languages
Again, sb-grovel.
> Note that all this can not be done with arbitary, hand written CLOS
> methods: ASDF has to be informed about how to do those tasks
> declaratively so that it can incorporate the knowledge in other
> contexts, such as when building an executable instead of dumping an
> image.
I must be missing something here. Could you explain what the
procedure for building an executable and/or shared library with ECL is
and why that doesn't work with existing ASDF systems? I'm not
familiar with ECL; the few times I have used it it worked just fine
with my ASDF files. I suppose the problem with shared libraries is
that you need a "link together object files" step that's not really
explicitly specified in most libraries and it's hard to wedge it in
with the current infrastructure--is that correct?
I agree that providing reusable components/extensiosn for ASDF is full
of problems (need extensions loaded to load .asd file, etc.). But
it's not clear to me how waving the "declarative!" flag about is
providing solutions; you still need to write code *somewhere* to
implement those declarative rules. And declarative systems can be
abused too--see glibc's Makefiles for numerous examples.
-Nathan
Tim X
> Tim X <ti...@nospam.dev.null> writes:
>> Juanjo <juanjose.g...@googlemail.com> writes:
>>> An even better version would not need methods at all! What makes
>>> TRAVERSE so unpredictable is that class dependencies, method
>>> specialization, :around, :after, and other apparent niceties can make
>>> a developer blow everything up -- to name a few things, ASDF recently
>>> changed the way it specializes some methods and it broke ECL's
>>> extensions in various and really misterious ways.
>>>
>>> OO is nice, but it is a distraction in this camp.
>
> Oh oh, that's... heretical! :-)
>
> Seriously though, the problem is not OO but trying to cram everything in
> that dependency graph image which, it seems, simply is not a good
> metaphor as soon as you are doing something more interesting, at least
> not in its current version.
I agree that OO is not the base issue. The same problems could as easily
exist with any paradigm.
>> I agree and think this exposes one of the reasons we have the situation
>> we are in. To some extent, ASDF has been so powerful that the individual
>> has been able to do things which on one had can make their life easier
>> when you are only looking at a system from a single perspective, but
>> once you move out into a wider world of possibilities, things get too
>> complex to manage.
>
> (let me just interrupt to say that I think that the words being used in
> this discussion are signs of trouble: Control is to be imposed, policies
> are to be enforced, the power of the individual curtailed, etc)
Let me interrupt to say I think your over stating what people have been
saying. I've not seen anywhere a suggestion that the power of the
individual is to be curtailed or even that control is to be imposed.
Putting everything else aside, there wold be no way to do this anyway.
What there has been is a sub-thread within this discussion that has
suggested the problem is not with the power that ASDF provides, but
rather with a lack of any standard application of that power. It ahs
been suggested, through various ways, that some of the issues could be
resolved if more conventions were established regarding how ASDF is
applied. This is a long way from the totalitarian interpretation you
appear to be implying.
It strikes me that there are two different camps emerging here. One camp
is concerned about anything that would restrict what they can do with
ASDF. This could well be a valid concern if the restrictions prevented
required or very useful functionality. I do get the impression that many
who come form this perspective are focusing on how they use ASDF for
their project.
The other camp appears to be people who are more focused on how to use
ASDF to install and use modules and libraries from multiple different
sources. For them, the problem is the weird and often unpredictable
interaction that occurs between different ASDF uses/definitions or
sometimes, things breaking because of differences in the CL
implementations being used and assumptions about an implementation which
may not hold for all implementations.
Any successful improvement really needs to address the concerns and
requirements of both camps (and more/others of course).
More consistent and standardised applicaiton of ASDF or more clearly
articulated conventions regarding its use does not necessarily mean loss
of power or flexibility, but it could reduce unnecessary conflicts and
reduce the frequency of inconsistent outcomes.
>> Concise and clear definition of dependencies would seem to be
>> the most critical base requirement.
>
> I don't think that that is the problem. The problem is identifying what
> kind of thing depend on what,
Huh? Isn't that just a concise definition of dependency? I don't gget
it. Are you possibly reading more into what I wrote than was there? Note
that I've not said anything on how dependencies should be
expressed/defined, only that a good starting point is to have a way to
clearly and concisely specify them.
and I'd like to propose viewing actions
> with side-effects as the things, and the dependencies as being that what
> everybody calls a dependency anyway.
>
> For example, consider this very simple rule.
>
> "the compilation of 'file1.lisp' depends on 'macros.lisp' and
> 'packages.lisp' being loaded".
>
> ASDF and defsystem treat this as dependencies on files, and not
> dependencies on the actions themselves.
>
> Instead of saying 'file1.lisp' depends on 'macros.lisp' and
> 'packages.lisp', and then awkwardly saying how, it would be better to
> say these things in a straight forward way. ASDF and defsystem tried the
> former approach, and maybe it is time to try the latter.
Maybe it is time to try an alternative, but I'm not sure we have even got
that far yet.
Please, using the same example you used above, give an example of
how you would express these dependencies using your model so that we can
better understand.
Mario S. Mommer
Tim X <ti...@nospam.dev.null> writes:
>>> Concise and clear definition of dependencies would seem to be
>>> the most critical base requirement.
>>
>> I don't think that that is the problem. The problem is identifying what
>> kind of thing depend on what,
>
> Huh? Isn't that just a concise definition of dependency? I don't gget
> it. Are you possibly reading more into what I wrote than was there? Note
> that I've not said anything on how dependencies should be
> expressed/defined, only that a good starting point is to have a way to
> clearly and concisely specify them.
Ok, ok, so you did not mean the arrows, you meant the nodes. Fair
enough. No offense intended.
What I am saying is that taking a graph whose nodes are "files" and then
manipulating the definition of the arrows to get what one wants is not
the only way to do things, and maybe not the best. It definitely has
some Turing tar pit quality to it; at least that's the way it feels when
I am writing an .asd to do something moderately complex. I can't say
defsystem felt better. I am just questioning the approach.
>> Instead of saying 'file1.lisp' depends on 'macros.lisp' and
>> 'packages.lisp', and then awkwardly saying how, it would be better to
>> say these things in a straight forward way. ASDF and defsystem tried the
>> former approach, and maybe it is time to try the latter.
>
> Maybe it is time to try an alternative, but I'm not sure we have even got
> that far yet.
>
> Please, using the same example you used above, give an example of
> how you would express these dependencies using your model so that we can
> better understand.
As an extension of the defsystem notation:
(requires (compile "file1.lisp")
(load "macros.lisp")
(load "packages.lisp"))
I agree that this is not too cool ;-). But how about this:
(let ((files
"packages" "macros"
"file1" "file2")) ;; etc. A directory listing would be
;; better, probably
(load-if-source-is-newer "packages.lisp")
(load-if-source-is-newer "macros.lisp")
;; You can even write a helpful comment here, explaining why you need
;; the macros.lisp file before compiling other stuff. It even feels
;; natural.
(mapcar #'compile-file-if-source-is-newer files)
(mapcar #'load-file-if-fasl-is-newer files))
Small, clean, and easy to understand. With a bit of policy, and a decent
way of handling paths (can be handled by a small lib) it would be
quite usable and explicit. In ASDF you have the :serial option for this,
but this is a hack, and does not really explain what depends on
what. And BTW this hack is needed because otherwise you'd need to write
("fileN" :depends-on ("packages" "macros"))
for each N, and that is obviously not acceptable.
I am not advocating the use of ad-hoc loaders, just trying to make a
more general point. I guess that what I am trying to say is that we have
this fantastic programming language that can solve so many problems, but
when it comes to specifying how to build/load software, we insist on
using a very strange DSL (the defsystem form) that is quite limiting and
makes us jump through redundant hoops to do even simple things.
Regards,
Mario.
Mario S. Mommer
Juanjo <juanjose.g...@googlemail.com> writes:
>> ASDF and defsystem treat this as dependencies on files, and not
>> dependencies on the actions themselves.
>
> Yes and no: actions depend on files but actions may also depend on
> other actions (internally in the ASDF graph). It is just that this
> possibility is not exposed in the DEFSYSTEM grammar.
Maybe it is better that way :-)
> I am very well aware of this situation and if you peek my previous
> emails that is why I introduced the notion of "rules" and makefile-
> style programming.
Ok.
Basically, IIUYC, what you have in mind is a rule based DSL where the
interpretations of the rules are parametrized over the backend
implementation. Sounds good. I'll not miss writing CLOS methods and
figuring out how they'll interact by basically trial and error.
OTOH, I fear that wether something depends on something else also is
dependent on the surrounding semantics, and there might lie some
potential for still having to remind people that not all lisps are
of the load-load-load-dump type.
Rainer Joswig
m_mo...@yahoo.com (Mario S. Mommer) wrote:
...
> > Please, using the same example you used above, give an example of
> > how you would express these dependencies using your model so that we can
> > better understand.
>
> As an extension of the defsystem notation:
>
> (requires (compile "file1.lisp")
> (load "macros.lisp")
> (load "packages.lisp"))
>
> I agree that this is not too cool ;-). But how about this:
>
> (let ((files
> "packages" "macros"
> "file1" "file2")) ;; etc. A directory listing would be
> ;; better, probably
>
> (load-if-source-is-newer "packages.lisp")
> (load-if-source-is-newer "macros.lisp")
>
> ;; You can even write a helpful comment here, explaining why you need
> ;; the macros.lisp file before compiling other stuff. It even feels
> ;; natural.
> (mapcar #'compile-file-if-source-is-newer files)
> (mapcar #'load-file-if-fasl-is-newer files))
Are you aware that this already exists
in the DEFSYSTEMs of Symbolics Genera, LispWorks
Allegro CL?
Mario S. Mommer
Rainer Joswig <jos...@lisp.de> writes:
> In article <87ochfw...@padme.localdomain>,
> m_mo...@yahoo.com (Mario S. Mommer) wrote:
> Are you aware that this already exists in the DEFSYSTEMs of Symbolics
> Genera, LispWorks Allegro CL?
No. What exactly, the imperative compiler/loader, or the other idea?
Rainer Joswig
m_mo...@yahoo.com (Mario S. Mommer) wrote:
> Rainer Joswig <jos...@lisp.de> writes:
> > In article <87ochfw...@padme.localdomain>,
> > m_mo...@yahoo.com (Mario S. Mommer) wrote:
> >> [...]
> > Are you aware that this already exists in the DEFSYSTEMs of Symbolics
> > Genera, LispWorks Allegro CL?
>
> No. What exactly, the imperative compiler/loader, or the other idea?
The description of dependencies for various actions.
Raymond Toy
Does that mean mk:defsystem has these features? The comments in
mk:defsystem says the syntax was inspired by the defsystem from
Symbolics 3600 and improves on it as well as the PCl defsystem.
Ray, who uses mk:defsystem because there was no asdf when he started
using defsystems.
Rainer Joswig
Raymond Toy <toy.r...@gmail.com> wrote:
> On 4/19/10 5:54 PM, Rainer Joswig wrote:
> > In article <87r5mbg...@padme.localdomain>,
> > m_mo...@yahoo.com (Mario S. Mommer) wrote:
> >
> >> Rainer Joswig <jos...@lisp.de> writes:
> >>> In article <87ochfw...@padme.localdomain>,
> >>> m_mo...@yahoo.com (Mario S. Mommer) wrote:
> >>>> [...]
> >>> Are you aware that this already exists in the DEFSYSTEMs of Symbolics
> >>> Genera, LispWorks Allegro CL?
> >>
> >> No. What exactly, the imperative compiler/loader, or the other idea?
> >
> > The description of dependencies for various actions.
> >
>
> Does that mean mk:defsystem has these features? The comments in
> mk:defsystem says the syntax was inspired by the defsystem from
> Symbolics 3600 and improves on it as well as the PCl defsystem.
The Symbolics DEFSYSTEM (a part of the System Construction Tool (SCT))
is in many ways quite different from mk:defsystem.
The dependency description of mk:defsystem is based on
the structure of the modules and systems. To answer
your question, it lacks these features.
The Symbolics defsystem dependencies are based on
operations. To define the operation dependencies the
Symbolics defsystem provides a short form
(:serial, :parallel, ...) and a long form, where these
dependencies between actions can be written in detail.
...
(:in-order-to (:load :compile) (:load defs))
(:in-order-to :compile (:load macros))
...
> Ray, who uses mk:defsystem because there was no asdf when he started
> using defsystems.
Tim X
> Tim X <ti...@nospam.dev.null> writes:
>>>> Concise and clear definition of dependencies would seem to be
>>>> the most critical base requirement.
>>>
>>> I don't think that that is the problem. The problem is identifying what
>>> kind of thing depend on what,
>>
>> Huh? Isn't that just a concise definition of dependency? I don't gget
>> it. Are you possibly reading more into what I wrote than was there? Note
>> that I've not said anything on how dependencies should be
>> expressed/defined, only that a good starting point is to have a way to
>> clearly and concisely specify them.
>
> Ok, ok, so you did not mean the arrows, you meant the nodes. Fair
> enough. No offense intended.
>
> What I am saying is that taking a graph whose nodes are "files" and then
> manipulating the definition of the arrows to get what one wants is not
> the only way to do things, and maybe not the best. It definitely has
> some Turing tar pit quality to it; at least that's the way it feels when
> I am writing an .asd to do something moderately complex. I can't say
> defsystem felt better. I am just questioning the approach.
>
No offense taken. My point is that all I was stating is that we need a
clear and concise way to represent dependencies. I have as yet no
preferred approach to how this is done - this is what I feel we still
need to identify and I believe it is the basic first step that needs to
be nailed down before anything else can be done. Whether this is a
declaritive, programatic, graph based or whatever is still I think an
open question.
Much of what you say I agree with. However, I'm not convinced that
because past attempts of a DSL have had weaknesses that all/any attempt
will suffer the same. Perhaps we can learn from the past mistakes and
make something better. As you say, this is supposed to be one of the
most powerful languages available, so maybe its the way its been applied
as a DSL that is the problem rather than the DSL approach per se. I
suspect the ultimate solution will be a hybrid that makes the majority
of cases very easy, but makes the complex very possible.
One thing I do think we need to make any real judgement is a selection
of use cases that we can use to evaluate different approaches. It is
very easy to fall into a trap where you have a solution that looks good,
but in reality only addresses a small sub-set of requirements.
Joshua Taylor
>
> Rainer Joswig<jos...@lisp.de> writes:
>> The dependency description of mk:defsystem is based on the structure
>> of the modules and systems. To answer your question, it lacks these
>> features.
>>
>> The Symbolics defsystem dependencies are based on operations. To
>> define the operation dependencies the Symbolics defsystem provides a
>> short form (:serial, :parallel, ...) and a long form, where these
>> dependencies between actions can be written in detail.
>>
>> ...
>> (:in-order-to (:load :compile) (:load defs))
>> (:in-order-to :compile (:load macros))
>> ...
>>
>
> current CL?
>
> Are there some documents about it, and maybe even some source??
LispWorks' still has a similar DEFSYSTEM. E.g., from
http://www.lispworks.com/documentation/lw445/LWRM/html/lwref-273.htm
there's an example:
(defsystem clos-sys
(:default-pathname "/usr/users/clc/defsys/"
:default-type :lsp-file
:package defsystem)
:members
(("defsys-macros" :type :system :root-module nil)
"class"
"time-methods"
("scm-pathname" :source-only t)
"execute-plan"
"file-types"
"make-system"
"conv-defsys")
:rules
((:in-order-to :compile ("class" "time-methods")
(:caused-by (:compile "defsys-macros"))
(:requires
(:load "defsys-macros")))
(:in-order-to :compile
("time-methods" "execute-plan")
(:requires (:load "class")))))
//JT
Mario S. Mommer
Rainer Joswig <jos...@lisp.de> writes:
> The dependency description of mk:defsystem is based on the structure
> of the modules and systems. To answer your question, it lacks these
> features.
>
> The Symbolics defsystem dependencies are based on operations. To
> define the operation dependencies the Symbolics defsystem provides a
> short form (:serial, :parallel, ...) and a long form, where these
> dependencies between actions can be written in detail.
>
> ...
> (:in-order-to (:load :compile) (:load defs))
> (:in-order-to :compile (:load macros))
> ...
>
Amazing. Why did that not live on? Is the mechanism incompatible with
Rainer Joswig
m_mo...@yahoo.com (Mario S. Mommer) wrote:
The implementation is extensive and Symbolics Genera specific.
> Are there some documents about it, and maybe even some source??
It is documented in the 'Program Development Utilities' manual.
A lower-quality PDF version of the manual can be found
at bitsavers.org .
The source is Symbolics specific and comes with Genera.
Rainer Joswig
Joshua Taylor <tay...@cs.rpi.edu> wrote:
> On 2010.04.20 3:13 PM, Mario S. Mommer wrote:
> >
> > Rainer Joswig<jos...@lisp.de> writes:
> >> The dependency description of mk:defsystem is based on the structure
> >> of the modules and systems. To answer your question, it lacks these
> >> features.
> >>
> >> The Symbolics defsystem dependencies are based on operations. To
> >> define the operation dependencies the Symbolics defsystem provides a
> >> short form (:serial, :parallel, ...) and a long form, where these
> >> dependencies between actions can be written in detail.
> >>
> >> ...
> >> (:in-order-to (:load :compile) (:load defs))
> >> (:in-order-to :compile (:load macros))
> >> ...
> >>
> >
> > Amazing. Why did that not live on? Is the mechanism incompatible with
> > current CL?
> >
> > Are there some documents about it, and maybe even some source??
>
> LispWorks' still has a similar DEFSYSTEM. E.g., from
More similar to the Symbolics DEFSYSTEM is the Franz Allegro CL
DEFSYSTEM:
http://www.franz.com/support/documentation/current/doc/defsystem.htm
Though I'd guess that the Allegro CL defsystem is a bit
simpler.
>
> http://www.lispworks.com/documentation/lw445/LWRM/html/lwref-273.htm
>
> there's an example:
>
> (defsystem clos-sys
> (:default-pathname "/usr/users/clc/defsys/"
> :default-type :lsp-file
> :package defsystem)
> :members
> (("defsys-macros" :type :system :root-module nil)
> "class"
> "time-methods"
> ("scm-pathname" :source-only t)
> "execute-plan"
> "file-types"
> "make-system"
> "conv-defsys")
> :rules
> ((:in-order-to :compile ("class" "time-methods")
> (:caused-by (:compile "defsys-macros"))
> (:requires
> (:load "defsys-macros")))
> (:in-order-to :compile
> ("time-methods" "execute-plan")
> (:requires (:load "class")))))
>
> //JT
A syntactic difference is that the rules in LispWorks
have their own main clause under :RULES.
o.ja...@gmail.com
replying.. Sorry for that.
Most of my opinion still stands, except i noticed an issue with how i
suggested to have backwards-compatibility; just starting asdf when
there is no way around it isn't enough.. one has to stop it, and
convince asdf it has the dependencies that were obtained from outside
it.