organization of lisp packages/modules, three questions

[ccc31807]

I've been writing some simple Common Lisp programs, and am now getting into more complicated ones. This is a very basic question about organizing a file structure.

I use both clisp and sbcl on windows. To date, I've had good success with quicklisp, but asdf and mk:system both remain mysteries. I'm happy with quicklisp, but would like to find some simple documentation about alternatives. I may be very dense, but after many months of trying, I have not been able to use or understand asdf.

My primary language is Perl, so I can explain my problem/confusion using Perl concepts. According to Peter Seibel, the Lisp package system is more like Perl's module system than any other language, so maybe this won't be too hard for people who don't know Perl to understand.

My Perl directories look like this:
.../perl
.../perl/lib
.../perl/lib/Text
.../perl/lib/Text/ParseWords.pm
.../perl/lib/PDF
.../perl/lib/PDF/API2.pm

So when I write a program and want to use functions from ParseWords.pm and API2.pm, I start my Perl program like this:

#! perl
use strict;
use warnings;
use PDF::API2;
use Text::ParseWords
# later on ...
my @line = parse_line(...) # in Text/ParseWords.pm
my $pdf = PDF::API2->new(...) # in PDF/API2.pm

I have downloaded and 'installed' (please note the scare quotes) a number of Common Lisp packages, but they remain mostly unusable for me. I have them scattered all over the place, and my Lisp session squawk badly if I try to use them.

So here are my three questions, obviously related to each other. If I simply have a wrong paradigm, please be gentle when you explain to me how dumb I am.

First Question: How do I arrange my file structure with respect to my Common Lisp packages so that I don't have to jump through hoops just to use the functions they contain? Can I have them in one place, or do I have to have duplicate sets in /clisp and /sbcl? Maybe I can but an instruction in my configuration files to tell the interpreters where the packages are located ... if so, I haven't found the documentation yet.

Second Question: How do I import/use/load/require packages so that I can call/use the functions the packages contain? Right now, I'm attempting to use cl-sqlite, cl-ppcre, cl-pdf, and cgi-utils. What would a program header look like to import the code so I can use the functions these packages contain, with or without a leading package qualifier?

Question 3: Is there anything in Common Lisp (perhaps quicklisp) that works similar to CPAN? For those of you who don't know CPAN, it's a functionality that allows me to download, compile, test, and install Perl modules so that they can be used quickly and easily in programs.

Note 1: As I said, I have found quicklisp easy to use without actually understanding fully how it works, but I'd rather have the source of the packages on my machine so I don't have to rely on an internet connection to use them.

Note 2: I'm at a point where I have a rather large amount of code in a number of packages, complete with asd files, so I might be asking a lot of tedious questions if I can't figure this out shortly.

Thanks, CC.

[Pascal J. Bourguignon]

ccc31807 <cart...@gmail.com> writes:

> I've been writing some simple Common Lisp programs, and am now getting
> into more complicated ones. This is a very basic question about
> organizing a file structure.
>
> I use both clisp and sbcl on windows. To date, I've had good success
> with quicklisp, but asdf and mk:system both remain mysteries. I'm
> happy with quicklisp, but would like to find some simple documentation
> about alternatives. I may be very dense, but after many months of
> trying, I have not been able to use or understand asdf.

Forget mk:defsystem, it's an old system superseded by asdf.
If you have to deal with an old library using mk:defsystem, my advice
would be to write an asd file to replace it.

> I have downloaded and 'installed' (please note the scare quotes) a
> number of Common Lisp packages, but they remain mostly unusable for
> me. I have them scattered all over the place, and my Lisp session
> squawk badly if I try to use them.

You may install your lisp libraries manually installed in
~/quicklisp/localprojects/
quicklisp will automatically pick them up, if they have an asd file.

> First Question: How do I arrange my file structure with respect to my
> Common Lisp packages so that I don't have to jump through hoops just
> to use the functions they contain?

Files and CL packages are totally unrelated.

> Can I have them in one place, or do
> I have to have duplicate sets in /clisp and /sbcl?

Unless you're writing implementation specific code, (and even in this
case) there's no reason to separate much less to duplicate source code.

You don't duplicate source files for gcc, icc, cling, etc!?!

> Maybe I can but an instruction in my configuration files to tell the
> interpreters where the packages are located ... if so, I haven't found
> the documentation yet.

The package is not the unit of loading, so there's little point of
mapping CL packages with file locations.


So, to answer your first question: mu.

You don't arrange your file structure with respect to your CL packages.
You arrange your file structure according to some logical scheme useful
to your code editing habits.



One option would be to store all the methods specialized on a class in a
file named for that class. Then when you're thinking about debugging or
adding a method for that class, you can go directly to that file.

Another option would be to store all the methods implementing some
feature in a file named for that feature. Then when you're thinking
about debugging or modifying a feature, you can go directly to that
file.

Another option would be to store your functions and methods
alphabetically, in files named a.lisp, b.lisp, c.lisp, etc. So that
when you're thinking about debugging or modifying a function named xyz,
you can go directly to the file x.lisp.

Another option if you have a big program, with modules, is to have
subdirectories for each modules and files in each subdirectories.

Another option is to put everything in a single file. With emacs on
64-bit systems, you can edit files bigger than 2 GB, so you have some
margin here…


Now, the real good modern way to do it, would be to leave emacs store
your toplevel forms in the files it wants, and you wouldn't have to deal
with them. Depending on what you want to debug or modify or add, it
would present you various lists of toplevel forms: alphabetical, by
class, by feature, by category, by module, etc, that you could edit.

> Second Question: How do I import/use/load/require packages so that I
> can call/use the functions the packages contain?

Again, you're confusing different things. You cannot import, load or
require a CL package.

You can import a SYMBOL: (import 'some-package:some-symbol)
You can use a PACKAGE: (use-package :some-package)
You can load a FILE: (load "some-file.lisp")
You can require a MODULE: (require :some-module)

With asdf or quickload,
you can also load systems: (ql:quickload :some-system)

> Right now, I'm
> attempting to use cl-sqlite, cl-ppcre, cl-pdf, and cgi-utils.

Those are systems. You cannot use them, you can only quickload or
asdf:load-op them.


(ql:quickload :cl-ppcre)

> What would a program header look like to import the code so I can use the
> functions these packages contain, with or without a leading package
> qualifier?

There' no notion of program header in lisp. Only a notion of
compilation unit (which may be mapped to source files, but not
necessarily) and a notion of toplevel form.

> Question 3: Is there anything in Common Lisp (perhaps quicklisp) that
> works similar to CPAN? For those of you who don't know CPAN, it's a
> functionality that allows me to download, compile, test, and install
> Perl modules so that they can be used quickly and easily in programs.

Yes, there's quicklisp.

> Note 1: As I said, I have found quicklisp easy to use without actually
> understanding fully how it works, but I'd rather have the source of
> the packages on my machine so I don't have to rely on an internet
> connection to use them.

Yes, you found quicklisp easy to use without understand ANYTHING about
it. Bravo.

> Note 2: I'm at a point where I have a rather large amount of code in a
> number of packages, complete with asd files, so I might be asking a
> lot of tedious questions if I can't figure this out shortly.

There's a lot of documentation, tutorials, example programs and
libraries, and blog entries, describing all this stuff. Use google!


Since you sound dumb enough not knowing how to use google, I'll present
here a minimal project. Let's write an application that will produce
the factorization of the factorial of a number:

- the user enters a positive integer,
- the system computes the factorial of this number,
- the system computes the prime factors of the factorial,
- the prime factors are printed.

To compute the prime factors we'll use the FUNCTIONS bound to the
com.informatimago.common-lisp.arithmetic.primes:factorize and
com.informatimago.common-lisp.arithmetic.primes:print-factorization
SYMBOLS from the com.informatimago.common-lisp.arithmetic.primes PACKAGE
which are loaded by the com.informatimago.common-lisp.arithmetic SYSTEM.


We'll divide the program in two components: user-interface and compute.
We'll spread the toplevel forms we will write into those files:

fafa/packages.lisp
fafa/compute/factorial.lisp
fafa/user-interface/main.lisp
fafa/user-interface/validation.lisp


We will have our program be read in a single Common Lisp package, since
it's simple enough: we're alone to program it, and it will have only a
small number of definitions. But for more complex projects, written by
several programmers, and having a lot of definitions, we would define
several packages to match for example the component structure, so that
there's no risk of name collision for auxiliary definitions in the
various components.

----(fafa/packages.lisp)------------------------------------------------

(in-package :common-lisp-user)

(defpackage :com.informatimago.fafa
(:use :cl
:com.informatimago.common-lisp.arithmetic.primes)
(:export :main)
(:documentation "
Implements the fafa program: print factorial factorizations.
Copyright Pascal Bourguignon 2012 - 2012
License: AGPL3
"))

----(fafa/compute/factorial.lisp)---------------------------------------

(in-package :com.informatimago.fafa)

(defun factorial (n)
(loop
:for i :from 1 :upto n
:for prod = i :then (* prod i)
:finally (return prod)))

(defun factorial/test ()
(assert (= (factorial 10) 3628800))
(assert (= (factorial 20) 2432902008176640000))
:success)

(factorial/test)

----(fafa/user-interface/main.lisp)-------------------------------------

(in-package :com.informatimago.fafa)

(defun main ()
(format *query-io* "~%Fafa~%")
(let* ((n (loop
:for v = (let ((*read-eval* nil))
(format *query-io* "~%Please enter an integer between 1 and 18:")
(finish-output *query-io*)
(read *query-io*))
:until (validate v "an integer between 1 and 18" '(integer 1 18))
:finally (return v)))
(n! (factorial n))
(factorization (rest (factorize n!))))
(format t "~% ~A ! = ~A " n (first factorization))
(dolist (term (rest factorization))
(if (listp term)
(format t "* ~A^~A " (second term) (third term))
(format t "* ~A " term)))
(format t "~2%"))
(values))


----(fafa/user-interface/validation.lisp)-------------------------------

(in-package :com.informatimago.fafa)

(defun validate (object type-desc type)
(if (typep object type)
t
(progn
(format *error-output* "~%~S is not ~A, but a ~A~%"
object type-desc (type-of object))
nil)))

------------------------------------------------------------------------


Now, we want to load those FILES so that their definitions be available
to the program. Since all the source files contain an in-package
:com.informatimago.fafa form, the package definition form must be loaded
and evaluated first! So fafa/packages.lisp must be loaded first. Since
main.lisp uses functions defined in validation.lisp and in
factorial.lisp, they should probably be loaded first too. [Actually,
lisp allows compiling a function calling another function before that
later being defined, but most compilers would signal a warning, and for
other definitions (special variables macros, etc, this wouldn't be
allowed).]

So we have the rules that:

to load or compile user-interface/main.lisp, we must load first packages.lisp
user-interface/validation.lisp
and compute/factorial.lisp

to load or compile user-interface/validation.lisp, we must first load packages.lisp
to load or compile compute.lisp, we must first load packages.lisp


This translates into asdf as:

:components ((:file "user-interface/main" :depends-on ("packages"
"user-interface/validation"
"compute/factorial"))
(:file "user-interface/validation" :depends-on ("packages"))
(:file "compute/factorial" :depends-on ("packages"))
(:file "packages" :depends-on ()))


But before loading this system, we must load the system
com.informatimago.common-lisp.arithmetic, which translates in asdf as:

:depends-on ("com.informatimago.common-lisp.arithmetic")

So we can wrap that in an asdf:defsystem form.

Since there's a single CL package in that system, I will use the same
name for the system as the package, but systems can have and often have
different names. To let quicklisp and asdf find systems easily, we put
them in files named like the system they contain (but one could also put
different systems in the same file, I don't advise to do that however,
or to put anything else than an asdf:defsystem form in those asd files).


----(fafa/com.informatimago.fafa.asd)-----------------------------------

(asdf:defsystem :com.informatimago.fafa
:name "Fafa"
:description "Factorize Factorial"
:author "Pascal Bourguignon"
:version "1.0.0"
:license "AGPL3"
:depends-on ("com.informatimago.common-lisp.arithmetic")
:components ((:file "user-interface/main" :depends-on ("packages"
"user-interface/validation"
"compute/factorial"))
(:file "user-interface/validation" :depends-on ("packages"))
(:file "compute/factorial" :depends-on
("packages"))))

------------------------------------------------------------------------

Now if you've put fafa/ in ~/quicklisp/local-projects/fafa/ then you
just quickload it. If you've put it elsewhere, you can add it to
asdf:*central-registry*:

cl-user> (pushnew #P"~/fafa/" asdf:*central-registry* :test (function equalp))
(#P"/home/pjb/fafa/" #P"/home/pjb/quicklisp/quicklisp/")

Or, if you want to gather your own CL projects in a different directory
than ~/quicklisp/local-projects, then you can add the super directory to
ql:*local-project-directories*:

cl-user> (pushnew #P"~/src/" ql:*local-project-directories* :test (function equalp))
(#P"/home/pjb/src/" #P"/home/pjb/quicklisp/local-projects/")

assuming you put fafa/ in ~/src/fafa/.


Then you can quickload and run your program:


cl-user> (ql:quickload :com.informatimago.fafa)
To load "com.informatimago.fafa":
Load 1 ASDF system:
com.informatimago.fafa
; Loading "com.informatimago.fafa"
[package com.informatimago.fafa] ; <-- the package being defined
(:com.informatimago.fafa) ; <-- the systems been loaded

cl-user> (com.informatimago.fafa:main)

Fafa

Please enter an integer between 1 and 18:33

33 is not an integer between 1 and 18, but a (integer 0 1152921504606846975)

Please enter an integer between 1 and 18:5

5 ! = 5 * 3 * 2^3

; No value
cl-user>


--
__Pascal Bourguignon__ http://www.informatimago.com/
A bad day in () is better than a good day in {}.

[Willem Rein Oudshoorn]

ccc31807 <cart...@gmail.com> writes:

> I've been writing some simple Common Lisp programs, and am now getting
> into more complicated ones. This is a very basic question about
> organizing a file structure.
>
> I use both clisp and sbcl on windows. To date, I've had good success
> with quicklisp, but asdf and mk:system both remain mysteries. I'm
> happy with quicklisp, but would like to find some simple documentation
> about alternatives. I may be very dense, but after many months of
> trying, I have not been able to use or understand asdf.

Let me try to help you here, but note that I do not have access to
windows right now, and I do not know anything about Perl, so take
my answers with a grain of salt.
In addition, it is not completely clear where your confusion comes from
and where you are stuck.

So first, roughly:

CPAN <---> QuickLisp [As far as it downloads packages/systems/modules]

Additionally, QuickLisp uses ASDF to load the packages/systems before
you want to use them. This is important to remember, for loading
packages in nonstandard locations or custom locations you have to
look into the documentation of ASDF.

> My primary language is Perl, so I can explain my problem/confusion
> using Perl concepts. According to Peter Seibel, the Lisp package
> system is more like Perl's module system than any other language, so
> maybe this won't be too hard for people who don't know Perl to
> understand.
>
> My Perl directories look like this:
> .../perl
> .../perl/lib
> .../perl/lib/Text
> .../perl/lib/Text/ParseWords.pm
> .../perl/lib/PDF
> .../perl/lib/PDF/API2.pm

[Snipped text]

> So when I write a program and want to use functions from ParseWords.pm and API2.pm, I start my Perl program like this:
>
> #! perl
> use strict;
> use warnings;
> use PDF::API2;
> use Text::ParseWords

From this I understand that in Perl the directory structure is related
ot the names you use to load them in perl, e.g. use PDF::API2 implies
loading PDF/API2.pm.
Now in Lisp this is not the case. The directory where the
package/system is located has nothing to do with the name of the package
nor the name of the system.

> I have downloaded and 'installed' (please note the scare quotes) a
> number of Common Lisp packages, but they remain mostly unusable for
> me. I have them scattered all over the place, and my Lisp session
> squawk badly if I try to use them.

Here you are losing me.
I understad that with installed you did

(ql:quickload "cl-ppcre")

right? Maybe it helps to think about it this way, (ql:quickload ...)
does basically the following:

1 - Tries to use ASDF to load the system "cl-ppcre"
2 - If it fails, download the system cl-ppcre and try step 1 again.

> Note 2: I'm at a point where I have a rather large amount of code in a
> number of packages, complete with asd files, so I might be asking a
> lot of tedious questions if I can't figure this out shortly.

Now this got me stumped, you have code in packages with asd files.
So what do you mean above with "squawk badly"?

If your code depends on the system lets say cl-ppcre, you should have in
your asd file the following line:

(asdf:defsystem #:my-great-system
....
:depends-on (#:cl-ppcre)
....)

This will trigger loading cl-ppcre when you load my-great-system
(when using ASDF of course.)

> So here are my three questions, obviously related to each other. If I
> simply have a wrong paradigm, please be gentle when you explain to me
> how dumb I am.
>
> First Question: How do I arrange my file structure with respect to my
> Common Lisp packages so that I don't have to jump through hoops just
> to use the functions they contain? Can I have them in one place, or do
> I have to have duplicate sets in /clisp and /sbcl?

You can do a lot of things. But basically you can configure ASDF to
search for certain directories for asd (system) files.

On MacOSX and linux systems this is easiest configured by putting a file

50-my-projects.conf

with content

(:tree "/Users/woudshoo/Development/Source/Lisp/")

in the directory

~/.config/common-lisp/source-registry.conf.d/

The same should work in windows except that the path above obviously is
different. (And for this you have to look into the ASDF manual.
If this is your stumbling block, I can look up what I use on my work
windows machine next week.)
(Also see notes at the end!)

Now after you have done this,
You can put all your projects in subdirectories of

"/Users/woudshoo/Development/Source/Lisp/"

e.g.

/Users/woudshoo/Development/Source/Lisp/cl-git/cl-git.asd
/Users/woudshoo/Development/Source/Lisp/cl-git/cl-git.lisp
...

But note that you can basically organize the source in whatever way you
like. Also not that the fact that cl-git.asd defines the package cl-git
and that the directory is also called cl-git are just conventions.
These names can be different if you want.

> Maybe I can but an
> instruction in my configuration files to tell the interpreters where
> the packages are located ... if so, I haven't found the documentation
> yet.

Chapter 3 in the asdf manual

> Second Question: How do I import/use/load/require packages so that I
> can call/use the functions the packages contain? Right now, I'm
> attempting to use cl-sqlite, cl-ppcre, cl-pdf, and cgi-utils. What
> would a program header look like to import the code so I can use the
> functions these packages contain, with or without a leading package
> qualifier?

See above, you specify them in your package asd file.

Note: If you do that for the cl-ppcre packege, you still
have to call the functions with the package prefix, like so:

(cl-ppcre:scan ...)

If you do not want to use the package prefix you have to 'use' the
package by putting in your

(defpackage #:my-great-system
...
(:use #:cl-ppcre)
...)

This use clause in the defpackage form will import all exported
(think of those as the published API) cl-ppcre symbols into
your my-great-system and allows you call the function above as:

(scan ...)

> Question 3: Is there anything in Common Lisp (perhaps quicklisp) that
> works similar to CPAN? For those of you who don't know CPAN, it's a
> functionality that allows me to download, compile, test, and install
> Perl modules so that they can be used quickly and easily in programs.

Yes, quicklisp does the downloading and easy installing.

>
> Note 1: As I said, I have found quicklisp easy to use without actually
> understanding fully how it works, but I'd rather have the source of
> the packages on my machine so I don't have to rely on an internet
> connection to use them.

Note that it does that. It downloads the source to your machine
into a subdirectory where quicklisp is installed.
(and in addition loads it)
So doing

(ql:quickload "cl-ppcre")

will work without an internet connection if you have downloaded it
before.



NOTE about ASDF and loading:

1 - I do not know if clisp ships with asdf.
If not you have to make sure that you load asdf
(or load quicklisp, which will in turn load asdf).

2 - If you are playing around with asd files and
installing your own packages, remember to do:

(asdf:clear-source-registry)

becuase asdf caches all names and locations of the systems it finds
and if you change the asdf condfiguration (by modifying or adding a
configuration file). Or add an asd file somewhere. It will fail to
find it, until you have issued the command above. Because quicklisp
uses ASDF it quicklisp will also fail for your own custom system
modifications.


I hope I understood your question correctly and that this helps.

Kind regards,
Wim Oudshoorn.

[Marco Eccettuato]

"Pascal J. Bourguignon" <p...@informatimago.com> writes:



> ccc31807 <cart...@gmail.com> writes:
>

... (snip)

Can you elaborate on this last option, which seems your preferred one?
Perhaps giving a short example of how you use Emacs to accomplish what
you describe, in particular the "leave emacs store your toplevel forms
in the files it wants, and you wouldn't have to deal with them" part?

Thanks for any answer to the above question, and for yet another
very good and useful article: the best I've seen so far summing
up what one needs to know about CL packages, modules, and systems.


(rest of original article snipped)



thanks,
Marco

[Pascal J. Bourguignon]

Marco Eccettuato

<m...@vanguard.i-did-not-set--mail-host-address--so-tickle-me> writes:

>> Now, the real good modern way to do it, would be to leave emacs store
>> your toplevel forms in the files it wants, and you wouldn't have to deal
>> with them. Depending on what you want to debug or modify or add, it
>> would present you various lists of toplevel forms: alphabetical, by
>> class, by feature, by category, by module, etc, that you could edit.
>>
>
> Can you elaborate on this last option, which seems your preferred one?

It is not implemented yet. :-(

[ccc31807]

Pascal,

First of all, thank you very much for your detailed and informative response. I have been reading and rereading it, and have found it more helpful than the other material I have looked at. I've been slow, but I certainly have not been ignoring it. At some point, I will spend a couple of days playing with it and that may (or may not) generate more questions.

With regard to the following, I just want to check my understanding. Don't reply unless I totally screw it up.

On Friday, August 31, 2012 3:38:16 PM UTC-4, informatimago wrote:
> You can import a SYMBOL: (import 'some-package:some-symbol)
> You can use a PACKAGE: (use-package :some-package)
> You can load a FILE: (load "some-file.lisp")
> You can require a MODULE: (require :some-module)

- A 'symbol' is simply a name that can refer to itself, a variable, a function, a property list, a package, and possibly some other things, and is a lookup key in a package..
- A 'package' is simply a lookup table that, for each package contains a list of symbols with the objects (e.g., variable name or function name) with which they are associated.
- A 'file' is an object composed of a source listing, and has no necessary correlation to a package, that is, a file can contain no packages, one package, or multiple packages, and a package (which must exist in some file if it exists as a persistent object) may be contained in one or more files.
- I don't know what a module is -- according to the Hyperspec *modules* is deprecated -- unless 'module' and 'package' are similar concepts.
- A 'system' is a series of variable declarations, function definitions, and other programmatic constructs, which is contained variously in packages and files, that comprises what we may informally call a 'computer program.'

> __Pascal Bourguignon__ http://www.informatimago.com/
> A bad day in () is better than a good day in {}.

Don't take this personally: I've studied a number of documents, by Ron Garrett, Peter Seible, Nick Levine, Zach Beane, the HyperSpec, the ASDF manual, etc., and these are inconsistent, confusing, and contradictory. For example, Nick Levine doesn't mention quicklisp in his chapter on systems, and I can't find anything that reconciles ASDF with XCVB (other than the self evident fact that the names consist of adjacent keys on the QWERTY keyboard). It might be helpful to the CL community, especially learners, to have a 2012 snapshot (or snapship, as the case might be) of the different ways to load, compile, and run whole programs.

Thanks again, your post is being helpful and is greatly appreciated, CC.

[Pascal J. Bourguignon]

ccc31807 <cart...@gmail.com> writes:

> Pascal,
>
> First of all, thank you very much for your detailed and informative
> response. I have been reading and rereading it, and have found it more
> helpful than the other material I have looked at. I've been slow, but
> I certainly have not been ignoring it. At some point, I will spend a
> couple of days playing with it and that may (or may not) generate more
> questions.
>
> With regard to the following, I just want to check my
> understanding. Don't reply unless I totally screw it up.
>
> On Friday, August 31, 2012 3:38:16 PM UTC-4, informatimago wrote:
>> You can import a SYMBOL: (import 'some-package:some-symbol)
>> You can use a PACKAGE: (use-package :some-package)
>> You can load a FILE: (load "some-file.lisp")
>> You can require a MODULE: (require :some-module)
>
> - A 'symbol' is simply a name that can refer to itself, a variable, a
> function, a property list, a package, and possibly some other things,
> and is a lookup key in a package..

Yes. It's a fundamental lisp data type. You can imagine there's a
definition such as:

(defstruct symbol
name
package
(plist '())
(value *unbound*)
(function *unbound*))

defstruct would generate the accessors symbol-name symbol-package
symbol-plist symbol-value and symbol-function.

It's a conceptual definition, an implementation can and probably will
implement symbols quite differently. A symbol can have a home package
or not. (Interned symbols have one usually, uninterned symbols (created
by gensym or make-symbol) haven't). symbol-name and symbol-package are
specified as functions, not as accessors (so you can't change the name
and home package of a symbol).


Symbols are used as names or designators.

When a symbol is declared SPECIAL, then it names a dynamic variable that
is its symbol-value. But symbols have their symbol-value even if not
declared SPECIAL. And in presence of threads, there may be one special
variable per thread for the same symbol declared special, plus the
global variable.
See http://groups.google.com/group/comp.lang.lisp/msg/58f1c7ed53d0c0d6


When a symbol is not declared SPECIAL, when it can name:

- a global constant variable (the value of the constant is stored in
symbol-value and it should not be changed), or

- a global symbol macro (the expansion is stored wherever the
implementation wants, you can still use symbol-value slot).

- a lexical variable. The value of the lexical variables are stored in
stack frames and those variables bear no relation with the symbol
naming them after compilation (unless debugging informations are
kept). Conceptually, the link between the symbol and the lexical
variables exist only in the compiler environment.



;; example is not declared special.

(setf (symbol-value 'example) 42)

(let ((example 0)) ; a lexical variable
(print example) ; prints 0
(print (symbol-value 'example)) ; prints 42
(let ((example 33)) ; a special variable
(declare (special example))
(print example) ; prints 33
(print (symbol-value 'example))) ; prints 33
(print example) ; prints 0
(print (symbol-value 'example)) ; prints 42
(terpri))

When/Where the symbol is declared special, the symbol names a variable
that is the symbol-value slot of the symbol.


Symbols are string designators. The designate a string that has the
same value as their name. So you can pass symbols to functions taking
string designators. Most string- functions take string designators.

(string-equal 'hello "Hello") --> T
(string= 'hello "Hello") --> NIL
(string= 'hello "HELLO") --> T


Packages are named by STRINGs, not by symbols, but package designators
are string designators or packages, so symbols can be used as package
designators.

(find-package "COMMON-LISP") --> #<PACKAGE "COMMON-LISP">
(find-package :common-lisp) --> #<PACKAGE "COMMON-LISP">
(find-package 'common-lisp) --> #<PACKAGE "COMMON-LISP">
(find-package '|common-lisp|) --> NIL

(defpackage "Example" (:use)) --> #<Package "Example">
(find-package 'example) --> NIL
(find-package "Example") --> #<Package "Example">
(find-package '|Example|) --> #<Package "Example">

Keywords are symbols in the KEYWORD package.
Symbol names are case sensitive. Package names too.
The lisp reader is configured by default to uppercase symbol names, so
it's practical to name your packages in all uppercase, but not required.



You can use symbols to name anything you want in your programs, as soon
as you have a table mapping them to something.

In Common Lisp, symbols are used to name:

- blocks,
- classes,
- conditions,
- declarations,
- functions,
- keywords of keyword parameters,
- macros,
- method combinations (method qualifiers),
- optimization qualities,
- restarts,
- slots,
- special operators,
- streams (but really, thru their symbol-value, cf. synonym-stream),
- symbol macros,
- tags in tagbody forms,
- types,
- variables (constant, special, lexical), including parameters,

and probably a few others I forgot, and you can add "name spaces" just
maintaining a symbol table in a macro.

> - A 'package' is simply a lookup table that, for each package contains
> a list of symbols with the objects (e.g., variable name or function
> name) with which they are associated.

Only a table of symbols. The packages don't care what's bound or fbound
to a symbol, what's on its plist, or where or how it's used.

> - A 'file' is an object composed of a source listing, and has no
> necessary correlation to a package, that is, a file can contain no
> packages, one package, or multiple packages, and a package (which must
> exist in some file if it exists as a persistent object) may be
> contained in one or more files.

A file can contain lisp forms, or other text or binary data. In the
context we're discussing, it'll contain lisp forms (form = S-expr
intended to be compiled or evaluated).

> - I don't know what a module is -- according to the Hyperspec
> *modules* is deprecated -- unless 'module' and 'package' are similar
> concepts.

It isn't standardized. It could be mapped to asdf systems, or some
other implementation dependant things, such as loadable modules in
clisp, etc. An implementation may provide a way to user code to map
module names to anything you want, so that you can load it with REQUIRE.

Note: uses of REQUIRE with two arguments is not implementation
dependant, so you can use it conformingly. But then, it's
basically equivalent to LOAD:

(defun require (module-name &optional pathname-list)
(if pathname-list
(unless (member module-name *modules* :test (function string=))
(mapcar (function load) (if (consp pathname-list)
pathname-list
(list pathname-list))))
(implementation-specific-require module-name)))

> - A 'system' is a series of variable declarations, function
> definitions, and other programmatic constructs, which is contained
> variously in packages and files, that comprises what we may informally
> call a 'computer program.'

Yes. Or a library.

> Don't take this personally: I've studied a number of documents, by Ron
> Garrett, Peter Seible, Nick Levine, Zach Beane, the HyperSpec, the
> ASDF manual, etc., and these are inconsistent, confusing, and
> contradictory. For example, Nick Levine doesn't mention quicklisp in
> his chapter on systems,

It didn't exist when Nick wrote his book.

> and I can't find anything that reconciles ASDF
> with XCVB (other than the self evident fact that the names consist of
> adjacent keys on the QWERTY keyboard).

XCVB is intended as a replacement to ASDF, but it hasn't caught yet.
You need to be 10 times better than the competition to replace it ;-)
XCVB is only twice better?

> It might be helpful to the CL community, especially learners, to have
> a 2012 snapshot (or snapship, as the case might be) of the different
> ways to load, compile, and run whole programs.

http://cliki.net is reasonably kept up to date.
Also: http://planet.lisp.org

> Thanks again, your post is being helpful and is greatly appreciated, CC.

--

[news.eternal-september.org]

The single most useful post I happened to read so far in the NG.
Also, the single best article I read on CL packaging. Thank you!

Salvatore