ASDF 2 struggle
Didier Verna
Hi,
I'm trying to upgrade my CL configuration to ASDF 2, and I have three
questions:
1/ why isn't "source-registry.conf" named "asdf-source-registry.conf"?
2/ Should we understand that the convention of having a "systems"
directory with .asd files symlinked there is now obsolete ?
I still prefer to keep that because I can select which of the
installed sources I do want to be seen. I guess you could also use
exclude patterns for doing so, but it seems way more complicated.
3/ The following contents for source-registry.conf fails:
(:source-registry
(:directory (merge-pathnames "share/common-lisp/systems/"
(user-homedir-pathname)))
(:directory "/usr/local/share/common-lisp/systems")
(:directory "/usr/share/common-lisp/systems")
:inherit-configuration)
Am I to understand that this file cannot really contain Lisp code (and
so I couldn't use #'merge-pathnames)? Because in that case, this
sucks. I have several machines on which $HOME is different (e.g.
/home/ or /Users/, and no, I don't want to create a /home/ -> /Users/
symlink on my Macs)
Thanks !
--
Resistance is futile. You will be jazzimilated.
Scientific site: http://www.lrde.epita.fr/~didier
Music (Jazz) site: http://www.didierverna.com
Didier Verna
And now, I'm struggling with the output translations.
I'm happy with the default settings (using .cache/blabla) except for one
case. For lisp files located under ~/Science/Source/Common Lisp/
(recursively), I want the compiled files to remain at the same place,
under an implementation-specific subdirectory, like
asdf-binary-locations did for me before.
For instance, ~/Science/Source/Common Lisp/foo/bar/baz.lisp would
produce somthing like:
~/Science/Source/Common Lisp/foo/bar/sbcl-1.0.32.30-linux-x86-64/baz.fasl
I tried this:
(:output-translations
("/Users/didier/Science/Source/Common Lisp/"
(:current-directory :implementation))
:inherit-configuration)
but this doesn't work. SBCL says:
This is SBCL 1.0.42.52, an implementation of ANSI Common Lisp.
More information about SBCL is available at <http://www.sbcl.org/>.
SBCL is free software, provided as is, with absolutely no warranty.
It is mostly in the public domain; some portions are provided under
BSD-style licenses. See the CREDITS and COPYING files in the
distribution for more information.
; loading system definition from
; /usr/local/lib/sbcl/sb-bsd-sockets/sb-bsd-sockets.asd into #<PACKAGE "ASDF0">
ASDF could not load sb-grovel because :CURRENT-DIRECTORY fell through ETYPECASE expression.
Wanted one of ((EQL :DEFAULT-DIRECTORY)
(EQL :SYSTEM-CACHE)
(EQL :USER-CACHE)
(EQL :HOME) (EQL :ROOT)
CONS STRING PATHNAME)..
ASDF could not load asdf-install because Error while trying to load definition
for system sb-bsd-sockets from
pathname
/usr/local/lib/sbcl/sb-bsd-sockets/sb-bsd-sockets.asd:
:CURRENT-DIRECTORY fell through ETYPECASE expression.
Wanted one of ((EQL
:DEFAULT-DIRECTORY)
(EQL :SYSTEM-CACHE)
(EQL :USER-CACHE)
(EQL :HOME) (EQL :ROOT)
CONS STRING PATHNAME)..
debugger invoked on a LOAD-SYSTEM-DEFINITION-ERROR:
Error while trying to load definition for system sb-bsd-sockets from pathname
/usr/local/lib/sbcl/sb-bsd-sockets/sb-bsd-sockets.asd:
:CURRENT-DIRECTORY fell through ETYPECASE expression.
Wanted one of ((EQL :DEFAULT-DIRECTORY) (EQL :SYSTEM-CACHE) (EQL :USER-CACHE)
(EQL :HOME) (EQL :ROOT) CONS STRING PATHNAME).
So what's the magick? Boy, do I wish there were an ASDF user manual :-(
Thanks!
Didier Verna
Another question: does ASDF 2 provide a built-in mechanism for
recompiling outdated fasl's, or do we still need the old black magick ?
Teemu Likonen
> Another question: does ASDF 2 provide a built-in mechanism for
> recompiling outdated fasl's [...] ?
Yes. It works automatically.
Zach Beane
> Hi,
>
> I'm trying to upgrade my CL configuration to ASDF 2, and I have three
> questions:
For what it's worth, I'm not sure anyone involved with the current
design and implementation of ASDF2 reads this newsgroup. The asdf-devel
list has active discussion. (The current discussion is about finding a
maintainer for the software.)
Zach
Tim Bradshaw
> Am I to understand that this file cannot really contain Lisp code (and
> so I couldn't use #'merge-pathnames)? Because in that case, this
> sucks. I have several machines on which $HOME is different (e.g.
> /home/ or /Users/, and no, I don't want to create a /home/ -> /Users/
> symlink on my Macs)
I'm not an expert, but I think that's right. What I do (now) is to
have something in my general Lisp init file which calls
asdf:initialize-source-repository with a backquoted form into which I
splice things of my own devising.
D Herring
said, the asdf-devel mailing list is the best place for these
questions. I mostly lurk there, trying to follow how to use ASDF,
occasionally nudging its development until we get something different.
On 09/23/2010 09:24 AM, Didier Verna wrote:
> I'm trying to upgrade my CL configuration to ASDF 2, and I have three
> questions:
>
> 1/ why isn't "source-registry.conf" named "asdf-source-registry.conf"?
I believe the intent was to make this generic across build systems
(i.e. both ASDF and XCVB support it).
> 2/ Should we understand that the convention of having a "systems"
> directory with .asd files symlinked there is now obsolete ?
> I still prefer to keep that because I can select which of the
> installed sources I do want to be seen. I guess you could also use
> exclude patterns for doing so, but it seems way more complicated.
I agree with you; but this is the minority position on the asdf-devel
mailing list. I even got Lichteblau's patch for MS-broken-windows
shortcuts committed to ASDF; but it appears to have broken in the
version 2 release cycle.
You could hide files by renaming their extension...
> 3/ The following contents for source-registry.conf fails:
>
> (:source-registry
> (:directory (merge-pathnames "share/common-lisp/systems/"
> (user-homedir-pathname)))
> (:directory "/usr/local/share/common-lisp/systems")
> (:directory "/usr/share/common-lisp/systems")
> :inherit-configuration)
>
> Am I to understand that this file cannot really contain Lisp code (and
> so I couldn't use #'merge-pathnames)? Because in that case, this
> sucks. I have several machines on which $HOME is different (e.g.
> /home/ or /Users/, and no, I don't want to create a /home/ -> /Users/
> symlink on my Macs)
In an attempt to sanitize things, there is general consensus that ASDF
files should be purely declarative forms. I think you are supposed to
play with CL's logical pathname translations for this.
Note that *central-registry* can be used to override the new mechanisms...
[splicing in other messages]
> And now, I'm struggling with the output translations.
Here's a snippet from "output-translations documentation" on
asdf-devel, dated to March 12, 2010. (I hope it is still accurate.)
> (asdf:initialize-output-translations
> `(:asdf-output-translations
> (:root (,(truename *default-pathname-defaults*) :implementation-type))))
>
> or if you want to only redirect output for what's in
> the current self-contained directory, and want it to
> be inside a predictable subdirectory thereof, you can use:
>
> (let ((cwd (truename *default-pathname-defaults*)))
> (asdf:initialize-output-translations
> `(:asdf-output-translations
> (,cwd (,cwd "asdf-output" :implementation-type)))))
I never gave this a full test. See also
asdf:enable-asdf-binary-locations-compatibility.
> So what's the magick? Boy, do I wish there were an ASDF user manual :-(
There is a fairly good one (thanks to Robert Goldman). See the
"Documentation" links right on the homepage.
http://common-lisp.net/project/asdf/
> Another question: does ASDF 2 provide a built-in mechanism for
> recompiling outdated fasl's, or do we still need the old black magick ?
AIUI, the dependency tracking code has been improved greatly, but
there are still a few corners where it fails. I would not trust it.
I do trust things like
# find $FASL_DIR -name \*.fasl -exec rm -f {} +
Later,
Daniel
Didier Verna
#### WARNING: I'm in a bad mood this morning.
D Herring <dher...@at.tentpost.dot.com> wrote:
> Fare, Robert Goldman, and Juanjo rank among the ASDF experts. As Zach said,
> the asdf-devel mailing list is the best place for these questions. I mostly
> lurk there, trying to follow how to use ASDF, occasionally nudging its
> development until we get something different.
Yup. I think Zach told me so yesterday as well, so I sent the
questions but got moderated because I'm not a subscriber. Needing to
subscribe to X-devel@ just because you have a question as an X-*user*
says a lot...
>> 1/ why isn't "source-registry.conf" named "asdf-source-registry.conf"?
>
> I believe the intent was to make this generic across build systems (i.e. both
> ASDF and XCVB support it).
OK.
>> 2/ Should we understand that the convention of having a "systems"
>> directory with .asd files symlinked there is now obsolete ?
>> I still prefer to keep that because I can select which of the
>> installed sources I do want to be seen. I guess you could also use
>> exclude patterns for doing so, but it seems way more complicated.
>
> I agree with you; but this is the minority position on the asdf-devel
> mailing list.
Yeah. There's no fun in making simple things simple right?
>> 3/ The following contents for source-registry.conf fails:
>>
>> (:source-registry
>> (:directory (merge-pathnames "share/common-lisp/systems/"
>> (user-homedir-pathname)))
>> (:directory "/usr/local/share/common-lisp/systems")
>> (:directory "/usr/share/common-lisp/systems")
>> :inherit-configuration)
>>
>> Am I to understand that this file cannot really contain Lisp code (and
>> so I couldn't use #'merge-pathnames)? Because in that case, this
>> sucks. I have several machines on which $HOME is different (e.g.
>> /home/ or /Users/, and no, I don't want to create a /home/ -> /Users/
>> symlink on my Macs)
>
> In an attempt to sanitize things, there is general consensus that ASDF files
> should be purely declarative forms. I think you are supposed to play with
> CL's logical pathname translations for this.
Somebody also gave me the hint of using the API directly yesterday,
which I did. Seems to work.
> Note that *central-registry* can be used to override the new mechanisms...
I know, but that's not the spirit, right?
> [splicing in other messages]
>> And now, I'm struggling with the output translations.
>
> Here's a snippet from "output-translations documentation" on asdf-devel, dated
> to March 12, 2010. (I hope it is still accurate.)
>
>> (asdf:initialize-output-translations
>> `(:asdf-output-translations
>> (:root (,(truename *default-pathname-defaults*) :implementation-type))))
>>
>> or if you want to only redirect output for what's in
>> the current self-contained directory, and want it to
>> be inside a predictable subdirectory thereof, you can use:
>>
>> (let ((cwd (truename *default-pathname-defaults*)))
>> (asdf:initialize-output-translations
>> `(:asdf-output-translations
>> (,cwd (,cwd "asdf-output" :implementation-type)))))
From this, I seem to understand that it works for one directory only,
whereas what I need is a special behavior for a whole tree...
> asdf:enable-asdf-binary-locations-compatibility.
I know, but that's not the spirit, right? :-)
OK, so now it's official. ASDF has gone beyond hope. It's taking the
path of C++.
>> So what's the magick? Boy, do I wish there were an ASDF user manual :-(
>
> There is a fairly good one (thanks to Robert Goldman). See the
> "Documentation" links right on the homepage.
> http://common-lisp.net/project/asdf/
Sorry, I'm talking about a *user* manual. Throwing a DSL's BNF
description at the face of people doesn't even start to make a manual.
Let alone "documenting" things that you realize don't even exist in the
code when you look at it (e.g. :current-directory or :function, no
wonder my code failed).
>> Another question: does ASDF 2 provide a built-in mechanism for
>> recompiling outdated fasl's, or do we still need the old black
>> magick?
>
> AIUI, the dependency tracking code has been improved greatly, but there are
> still a few corners where it fails. I would not trust it.
>
> I do trust things like
> # find $FASL_DIR -name \*.fasl -exec rm -f {} +
Yup. Thanks for your help!
RPG
> #### WARNING: I'm in a bad mood this morning.
>
> > Fare, Robert Goldman, and Juanjo rank among the ASDF experts. As Zach said,
> > the asdf-devel mailing list is the best place for these questions. I mostly
> > lurk there, trying to follow how to use ASDF, occasionally nudging its
> > development until we get something different.
>
> Yup. I think Zach told me so yesterday as well, so I sent the
> questions but got moderated because I'm not a subscriber. Needing to
> subscribe to X-devel@ just because you have a question as an X-*user*
> says a lot...
mailman is a bagbiting way to provide support. It would be great to
have something better. In my dreams this would be something where new
questions are served up as an RSS feed, and you can log in with OpenID
(so you don't need to jump through subscription hoops). Maybe someone
needs to set up a StackExchange site for Common Lisp.... I am not
enough of a webbie to have a solution for this problem.
https://bugs.launchpad.net/asdf
> >> 2/ Should we understand that the convention of having a "systems"
> >> directory with .asd files symlinked there is now obsolete ?
> >> I still prefer to keep that because I can select which of the
> >> installed sources I do want to be seen. I guess you could also use
> >> exclude patterns for doing so, but it seems way more complicated.
>
> > I agree with you; but this is the minority position on the asdf-devel
> > mailing list.
>
> Yeah. There's no fun in making simple things simple right?
The systems directory/link farm scales poorly. In particular, when
new .asd files are added to a library (say one that you pull from a
revision control system), there is no discipline for updating the link
farm. Hence the preference for something that allows you to specify a
source registry.
Similarly, if one works on > 1 lisp project concurrently, with
different mixes of libraries, said approach works poorly.
Finally, the symlink approach was working poorly for integration with
system-supplied libraries, especially when working with multiple CL
implementations. E.g., you use SBCL and clisp; the former has
libraries installed for you which you should be able to use, but
should not use with the latter.
Yes, it's painful to cut over (I still haven't done it and rely on a
different discipline I developed for ASDF 1). We don't have a
solution for that.
....
> >> So what's the magick? Boy, do I wish there were an ASDF user manual :-(
>
> > There is a fairly good one (thanks to Robert Goldman). See the
> > "Documentation" links right on the homepage.
> >http://common-lisp.net/project/asdf/
>
> Sorry, I'm talking about a *user* manual. Throwing a DSL's BNF
> description at the face of people doesn't even start to make a manual.
> Let alone "documenting" things that you realize don't even exist in the
> code when you look at it (e.g. :current-directory or :function, no
> wonder my code failed).
This is hardly a useful request for clarification, although I suspect
that there is some actual bug report hiding in here that I do not have
the time to extract. You don't like somethign about :current-
directory and :function, but I don't know what.
I understand that the existing manual is suboptimal. However, writing
and repairing ASDF documentation is not what my employer pays my
salary to do, nor does it help rear my children. I must squeeze such
improvements in during what I laughingly call my "spare time." If you
want this fixed, please provide SPECIFIC requests for clarification/
repair, ideally through launchpad. Just complaining that the manual
is not a user manual, does not help me, or anyone else, target his or
her efforts.
If you REALLY want a user manual, pay for an Allegro Common Lisp
license, and use their defsystem. They have people who are paid to
write user manuals, and their documentation is excellent. (LispWorks
may be excellent, too --- no deprecation intended; I don't use it
myself).
If anyone out there has good example configuration snippets, send them
to me, or post them as launchpad tickets.
>
> >> Another question: does ASDF 2 provide a built-in mechanism for
> >> recompiling outdated fasl's, or do we still need the old black
> >> magick?
>
> > AIUI, the dependency tracking code has been improved greatly, but there are
> > still a few corners where it fails. I would not trust it.
Dependency tracking across system boundaries still does not work. If
you change an upstream library, you should either use :force to force
recompliation downstream or, if you like , blow away the fasls per
below...
>
> > I do trust things like
> > # find $FASL_DIR -name \*.fasl -exec rm -f {} +
>
> Yup. Thanks for your help!
I have a CLEAN-OP patch that needs to be merged. Waiting for said
spare time, though.
[sigh --- would be nice if google groups had a preview button; hope
this is readable...]
Didier Verna
> This is hardly a useful request for clarification, although I suspect
> that there is some actual bug report hiding in here that I do not have
> the time to extract. You don't like somethign about :current-
> directory and :function, but I don't know what.
Wait a minute. Are you pretending that these actually exist? Because
then I'm lost. After trying to decipher the online doc, I tried to use
:current-directory, as exposed in my previous post, but got an error.
Then, I grabbed asdf.lisp and found out that there is no mention of this
key in resolve-*-location or anywhere else in fact. Then I thought maybe
current SBCL's asdf is outdated, then went to common-lisp.net and
grabbed http://common-lisp.net/project/asdf/asdf.lisp which is indeed
more recent, but does not have a :current-directory either.
I admit I lost my temper before trying the git repo... (I didn't try it
yet; should I ?) but since SBCL ships with its own ASDF, I'm not sure
this is going to solve anything.
> I understand that the existing manual is suboptimal. However, writing
> and repairing ASDF documentation is not what my employer pays my
> salary to do, nor does it help rear my children.
Don't worry. I understand all that. I'm not paid for writing my
libraries or their manuals either.
> If anyone out there has good example configuration snippets, send them
> to me, or post them as launchpad tickets.
On the other hand, if you can explain how to do the following, I would
be grateful:
| And now, I'm struggling with the output translations.
|
| I'm happy with the default settings (using .cache/blabla) except for one
| case. For lisp files located under ~/Science/Source/Common Lisp/
| (recursively), I want the compiled files to remain at the same place,
| under an implementation-specific subdirectory, like
| asdf-binary-locations did for me before.
|
| For instance, ~/Science/Source/Common Lisp/foo/bar/baz.lisp would
| produce somthing like:
| ~/Science/Source/Common Lisp/foo/bar/sbcl-1.0.32.30-linux-x86-64/baz.fasl
|
| I tried this:
|
| (:output-translations
| ("/Users/didier/Science/Source/Common Lisp/"
| (:current-directory :implementation))
| :inherit-configuration)
Thank you.
Faré
and was directed to this thread by a message on the asdf-devel mailing-
list,
which is where the action is taking place.
On Sep 23, 9:24 am, Didier Verna <did...@lrde.epita.fr> wrote:
> 1/ why isn't "source-registry.conf" named "asdf-source-registry.conf"?
>
Because it's also used by XCVB already, and
is intended to be potentially used by other future build systems.
> 2/ Should we understand that the convention of having a "systems"
> directory with .asd files symlinked there is now obsolete ?
> I still prefer to keep that because I can select which of the
> installed sources I do want to be seen. I guess you could also use
> exclude patterns for doing so, but it seems way more complicated.
>
This convention is still supported. It's not the new recommended way,
but will work for as long as the new way is not considered
satisfactory
by people like you.
> 3/ The following contents for source-registry.conf fails:
>
> (:source-registry
> (:directory (merge-pathnames "share/common-lisp/systems/"
> (user-homedir-pathname)))
> (:directory "/usr/local/share/common-lisp/systems")
> (:directory "/usr/share/common-lisp/systems")
> :inherit-configuration)
>
> Am I to understand that this file cannot really contain Lisp code (and
> so I couldn't use #'merge-pathnames)? Because in that case, this
> sucks. I have several machines on which $HOME is different (e.g.
> /home/ or /Users/, and no, I don't want to create a /home/ -> /Users/
> symlink on my Macs)
>
a- I suggest you use #.(merge-pathnames ...) for the time being.
b- I should probably extend the source-registry to use the same DSL as
asdf-output-translations for naming things relative to :home
Please file a bug on launchpad.net.
[ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org
]
Die Mathematiker sind eine Art Franzosen: redet man zu ihnen, so
übersetzen
sie es in ihre Sprache, und dann ist es alsobald ganz etwas anderes.
[Les mathématiciens sont comme les Français: quoiqu'on leur dise,
ils le traduisent dans leur propre langue, et cela devient alors
quelque chose de complètement différent.] — Johann Wolfgang von Goethe
Faré
> RPG <rpgold...@gmail.com> wrote:
> > This is hardly a useful request for clarification, although I suspect
> > that there is some actual bug report hiding in here that I do not have
> > the time to extract. You don't like somethign about :current-
> > directory and :function, but I don't know what.
>
> Wait a minute. Are you pretending that these actually exist? Because
> then I'm lost. After trying to decipher the online doc, I tried to use
> :current-directory, as exposed in my previous post, but got an error.
>
> Then, I grabbed asdf.lisp and found out that there is no mention of this
> key in resolve-*-location or anywhere else in fact. Then I thought maybe
> current SBCL's asdf is outdated, then went to common-lisp.net and
> more recent, but does not have a :current-directory either.
>
platforms (notably Windows) it didn't make sense, at which point I
dropped it, but forgot to update the documentation.
> I admit I lost my temper before trying the git repo... (I didn't try it
> yet; should I ?) but since SBCL ships with its own ASDF, I'm not sure
> this is going to solve anything.
>
Maybe you'd get more efficient help faster if you asked the right
people, i.e. the authors, rather than random people off the Interweb.
> > I understand that the existing manual is suboptimal. However, writing
> > and repairing ASDF documentation is not what my employer pays my
> > salary to do, nor does it help rear my children.
>
> Don't worry. I understand all that. I'm not paid for writing my
> libraries or their manuals either.
>
I apologize for the confusion.
> > If anyone out there has good example configuration snippets, send them
> > to me, or post them as launchpad tickets.
>
> On the other hand, if you can explain how to do the following, I would
> be grateful:
>
> | And now, I'm struggling with the output translations.
> |
> | I'm happy with the default settings (using .cache/blabla) except for one
> | case. For lisp files located under ~/Science/Source/Common Lisp/
> | (recursively), I want the compiled files to remain at the same place,
> | under an implementation-specific subdirectory, like
> | asdf-binary-locations did for me before.
>
> | For instance, ~/Science/Source/Common Lisp/foo/bar/baz.lisp would
> | produce somthing like:
> | ~/Science/Source/Common Lisp/foo/bar/sbcl-1.0.32.30-linux-x86-64/baz.fasl
> |
> | I tried this:
> |
> | (:output-translations
> | ("/Users/didier/Science/Source/Common Lisp/"
> | (:current-directory :implementation))
> | :inherit-configuration)
>
> Thank you.
>
Try this entry:
((:home "Science/Source/Common Lisp") (:home "Science/Source/Common
Lisp" :implementation))
[ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org
]
In a country where the sole employer is the State, opposition means
death by
slow starvation. The old principle: who does not work does not eat,
has been
replaced by a new one: who does not obey shall not eat.- Leon Trotsky
Scott L. Burson
> I don't want to create a /home/ -> /Users/ symlink on my Macs)
Why not? I do exactly that (for reasons having nothing to do with ASDF).
-- Scott
D Herring
> The systems directory/link farm scales poorly. In particular, when
> new .asd files are added to a library (say one that you pull from a
> revision control system), there is no discipline for updating the link
> farm. Hence the preference for something that allows you to specify a
> source registry.
>
> Similarly, if one works on> 1 lisp project concurrently, with
> different mixes of libraries, said approach works poorly.
This is due to deficiencies in ASDF itself, not due to deficiencies in
the approach. You are trading one set of complexities for another
(updating links vs maintaining sets of paths in lisp files). Your
approach is similar to static linking in C.
One pattern that works fairly well (though not perfectly):
# mkdir base-link-farm
# mkdir overlay-link-farm
(setf *central-registry* overlay base)
The Unix/C and Java mechanisms have it about right:
- a couple central places for libraries (easy to reuse)
- a mechanism that allows multiple API versions to coexist
- a specification for local overlays (LD_LIBRARY_PATH, RPATH, etc.)
- no automagic recompilation of installed libraries
ASDF has most, but not all of this.
> Finally, the symlink approach was working poorly for integration with
> system-supplied libraries, especially when working with multiple CL
> implementations. E.g., you use SBCL and clisp; the former has
> libraries installed for you which you should be able to use, but
> should not use with the latter.
That was fixed years ago by asdf-binary-locations. It is now baked in
to ASDF2.
Later,
Daniel