I am happy to announce a new version of my lexicons code. This version fixes a rather serious bug in the previous version, and adds new features, most notably semi-hygienic lexical macros and lexical classes and methods. The implementation is also significantly cleaned up (it consists of a single file of portable ANSI Common Lisp) and is more seamlessly integrated into Common Lisp (lexicons are now implemented using packages).
Lexicons are first-class lexical environments for Common Lisp, what in other languages would be called modules or namespaces. They are designed to be an adjunct, or potentially a replacement (depending on your needs) for packages. Packages map strings on to symbols at read-time. Lexicons map symbols onto bindings at compile-time. For writing modular code, lexicons can be significantly more convenient than packages. In particular, lexicons are unaffected by read-time side-effects, and there is no need to manually maintain lists of exported symbols.
This version has been tested only on Clozure Common Lisp, but it is written in portable ANSI Common Lisp so it should run under any conforming implementation.
Hi! I can not pay much attention for lexicons now. I think they introduce too many new constructs to lisp, and that's why I think I won't use it. I too tried to solve package problems. I developed a rather minimalistic approach. This approach produced some dispute here. It is the time to give a link here: http://norvig.com/python-lisp.html Piter Norvig says that package system is hard to use and says it is a disadvantage of CL. I agree completely to this. My solution is to add macro (in-packages package &rest more-packages) which acts as following: - when this macro is in effect, reader behaviour changes. When reading, unqualified symbol is sought for in all packages listed. It can be found as internal symbol in first package and as external symbol only in the rest of the packages. If it is found once, it is used. If it is found more than once, error is produced. If it is not found, it it interned into package. Maybe (I think) one more rule is needed: if this symbol is in shadowing symbols list of the "package", it can be used without qualifier even if it is external in some of other packages. - you may use qualified symbols to specify exactly what you mean. - optionally, package locks are imposed to some packages (syntax not shown). Additionally, read-macro of kind #with-packages (package &rest packages) form is added so that next form is read in scope of (with-packages) form.
This approach allows minimize both import and use-package operations and it is relatively safe. It does not remove the need to export some symbols manually when you want, say, have a custom + or - in your package. Idea is borrowed from sql syntax:
select t1.id, t2.id, t1.unique_field_of_t1 from t1, t2 where t1.id=t2.t1_id
Which proved in my practice to be fairly reliable and convinient. I say it just in case you didn't read the topic with keywords "symbol clashes - how to avoid them"
For this to work we need either to change implementation's "find- package" function (which is not portable), or use portable lisp reader written in CL (it can be done rather easily based on some of open- source implementations, there is such a reader already, written by Pascal J. Bourguignon, but it is under GPL and it has bugs). I managed to substitute lisp reader completely with other lisp reader, for all activity including compiler. So, for now all what is left to do is to take some open-source reader and replace implementation-specific reading primitives with CL ones. Don't know when I get time for this.
In article <54b8cf3e-3bf9-47e4-a46a-c307ba53c...@j35g2000yqh.googlegroups.com>,
budden <budde...@gmail.com> wrote: > Hi! > I can not pay much attention for lexicons now. I think they > introduce too many new constructs to lisp, and that's why I think I > won't use it.
I don't know what you consider a "new" construct, but lexicons only introduce lexical equivalents of existing CL defining forms (defun, defvar, defmacro, defclass and defmethod). The lexical equivalent of each of these has the same name but preceded by an L (ldefun, ldefvar, ldefmacro, ldefclass and ldefmethod). One could easily shadow the existing defining forms so that programming with lexicons is no different from programming in regular CL, but I wasn't going to do this until I had time to do some really thorough testing.
> I too tried to solve package problems. I developed a > rather minimalistic approach. This approach produced some dispute > here. It is the time to give a link here: http://norvig.com/python-lisp.html > Piter Norvig says that package system is hard to use and says it is a > disadvantage of CL. I agree completely to this.
Which is in fact the primary motivation for lexicons.
> My solution is to add macro > (in-packages package &rest more-packages) which acts as following:
[snip]
IMHO, the central problem with packages is that they try to resolve namespaces at read time, when this ought to be done at compile-time. The appropriate information for resolving namespaces is simply not available at read-time. So the problems with packages cannot be solved by changing the reader.
Note that this issue is in some sense the converse of the one investigated by Kent Pitman a long time ago:
Well, maybe I'll take a look at it later, but it is too much. Some packages overwrite defun too. One of them is ap5 which is very cool. Other is screamer which is cool to. I afraid there will be problems... And, say, I want to make a wrapper for defclass with standard names of accessors, initargs, etc. If I accept lexicons, I need to redefine ldefclass too.
Anyway, I think I'll take a closer look at them in a short just to find what ideas are there.
In article <cbe188e9-a2d4-4db3-81c7-0d6b6424b...@l42g2000yqe.googlegroups.com>,
budden <budde...@gmail.com> wrote: > Well, maybe I'll take a look at it later, but it is too much. > Some packages overwrite defun too. One of them is ap5 which is very > cool. > Other is screamer which is cool to. I afraid there will be > problems...
It depends on what the shadowing forms in these other packages expand into. If, for example, AP5:DEFUN expands into CL:DEFUN then you can just muck with the package system to have it expand into LEXICONS:DEFUN instead. It should Just Work. But this is the sort of thing I want to actually try before unleashing it on the world.
> And, say, I want to make a wrapper for defclass with standard names of > accessors, initargs, etc. > If I accept lexicons, I need to redefine ldefclass too.
Nope, you'll just need to change your wrapper to expand into ldefclass instead of defclass. All the LDEF* forms have the exact same syntax as their corresponding non-lexical forms. Alternatively you can shadow DEF* and just run your (unchanged) wrapper definition in the package with the shadowed symbols.
The ultimate goal is to be able to take existing code, remove (nearly) all the package-related stuff, move everything into the (not yet implemented) LEXICONS package, and have it all Just Work. This may be a pipe dream, but the fact that there is now a very close correspondence between lexicons and packages makes me optimistic.
Ron Garret wrote: > The implementation is also significantly cleaned up (it > consists of a single file of portable ANSI Common Lisp) and is more > seamlessly integrated into Common Lisp (lexicons are now implemented > using packages).
I'm not sure I like this, at least not the way a lexicon named "l1" now seems to "use up" a package named l1. Haven't really played with it enough yet to judge if it's all that much of an issue, but maybe packages used for the implementation should be called %lex-l1 or something.
> They are > designed to be an adjunct, or potentially a replacement (depending on > your needs) for packages. Packages map strings on to symbols at > read-time. Lexicons map symbols onto bindings at compile-time. For > writing modular code, lexicons can be significantly more convenient > than packages.
Yeah, I'm one of the people who uses packages for stuff other than writing modular code - they're independently useful for sorting symbols, so it'd be nice if both lexicons and packages would work with minimal interference with eachother...
budden wrote: > Hi! > I can not pay much attention for lexicons now. I think they > introduce too many new constructs to lisp, and that's why I think I > won't use it. I too tried to solve package problems. I developed a > rather minimalistic approach. This approach produced some dispute > here. It is the time to give a link here: http://norvig.com/python-lisp.html > Piter Norvig says that package system is hard to use and says it is a > disadvantage of CL. I agree completely to this.
Well then I do believe it is time to go shopping.
Only there is nothing hard about the package system unless your cup is full and you want it to be like some other thing but everything is like that.
> My solution is to add macro > (in-packages package &rest more-packages) which acts as following: > - when this macro is in effect, reader behaviour changes. When > reading, unqualified symbol is sought for in all > packages listed. It can be found as internal symbol in first package > and as external symbol only in the rest of the packages. If it is > found once, it is used. If it is found more than once, error is > produced. If it is not found, it it interned into package. Maybe (I > think) one more rule is needed: if this symbol is in shadowing symbols > list of the "package", it can be used without qualifier even if it is > external in some of other packages. > - you may use qualified symbols to specify exactly what you mean. > - optionally, package locks are imposed to some packages (syntax not > shown). > Additionally, read-macro of kind > #with-packages (package &rest packages) form > is added so that next form is read in scope of (with-packages) form.
> This approach allows minimize both import and use-package operations > and it is relatively safe. It does not remove the need to export some > symbols manually when you want, say, have a custom + or - in your > package. Idea is borrowed from sql syntax:
> select t1.id, t2.id, t1.unique_field_of_t1 from t1, t2 where > t1.id=t2.t1_id
> Which proved in my practice to be fairly reliable and convinient. > I say it just in case you didn't read the topic with keywords > "symbol clashes - how to avoid them"
> For this to work we need either to change implementation's "find- > package" function (which is not portable), or use portable lisp reader > written in CL (it can be done rather easily based on some of open- > source implementations, there is such a reader already, written by > Pascal J. Bourguignon, but it is under GPL and it has bugs). I managed > to substitute lisp reader completely with other lisp reader, for all > activity including compiler. So, for now all what is left to do is to > take some open-source reader and replace implementation-specific > reading primitives with CL ones. Don't know when I get time for this.
Good lord, don't you have any application programming to do?
In article <ghc6i2$v5...@aioe.org>, David Golden <david.gol...@oceanfree.net> wrote:
> Ron Garret wrote:
> > The implementation is also significantly cleaned up (it > > consists of a single file of portable ANSI Common Lisp) and is more > > seamlessly integrated into Common Lisp (lexicons are now implemented > > using packages).
> I'm not sure I like this, at least not the way a lexicon named "l1" now > seems to "use up" a package named l1. Haven't really played with it > enough yet to judge if it's all that much of an issue, but maybe > packages used for the implementation should be called %lex-l1 or > something.
That would be very easy to change.
> > They are > > designed to be an adjunct, or potentially a replacement (depending on > > your needs) for packages. Packages map strings on to symbols at > > read-time. Lexicons map symbols onto bindings at compile-time. For > > writing modular code, lexicons can be significantly more convenient > > than packages.
> Yeah, I'm one of the people who uses packages for stuff other than > writing modular code - they're independently useful for sorting > symbols, so it'd be nice if both lexicons and packages would work with > minimal interference with eachother...
Lexicons have always been orthogonal to (and therefore compatible with) packages from the beginning. It's just that now packages are used in the implementation whereas before they weren't.
Out of curiosity, what kind of "symbol sorting" are you using packages for?
Ron Garret wrote: > That would be very easy to change.
I assumed as much, but you did ask for feedback... :-)
> Out of curiosity, what kind of "symbol sorting" are you using packages > for?
If you're actually programming with symbols as pseudo-objects (hey, they are first-class in lisp), then it's just handy to be able to stash them into separate packages so that ones with the same name have separate identities but are easy to refer to.
You have symbols with the same symbol-name. You want them to be different objects still, and you don't want the emacs lisp "solution" (um, make them have different names...). I mean, not much more to it.
Personally, I've used it for an (unreleased and half-assed) glsl and c translators usable in one image, in combination, that used e.g. glsl:main and c:main. Probably more stuff.
Mostly convenience for humans writing/reading code, not for the computer - Nothing you _couldn't_ do with a different implementation strategy, without packages (or even first-class symbols), just a naming convention or a bunch of hashtables you look up different things in in different contexts. But you wind up implementing half a package system anyway, shrug. OTOH, people wouldn't go around calling your (non-performance-critical, simple, short, works) code "old fashioned"...
In article <c92308a2-7b39-4f65-bb8c-39a6556ff...@y1g2000pra.googlegroups.com>, "Leslie P. Polzer" <leslie.pol...@gmx.net> wrote:
> I have a minor gripe with the function MAKE-LEXICON:
> It modifies some global object (i.e. the list of lexicons).
> This is not consistent with the usual semantics of MAKE-* functions > and should probably be called DEFLEXICON instead.
> Leslie
I agree with you that this is a bit of a wart in the design, but there's a reason for it. The design of the lexicons API was intended to mirror the API for packages as much as possible while still achieving the design goals. Accordingly, MAKE-LEXICON is intended to be the counterpart to MAKE-PACKAGE, which also modifies a global. DEFLEXICON, if it were to exist, would be the counterpart to DEFPACKAGE. But the whole point of lexicons is to get rid of all the complexity that makes DEFPACKAGE necessary, so DEFLEXICON would in some sense be an oxymoron.
In an ideal world, MAKE-LEXICON would not modify a global. It would also allow you to make anonymous lexicons, and multiple lexicons with the same name. But it is not possible to make anonymous packages, nor is it possible to make multiple packages with the same name, since both of these would undermine the whole point of having packages. (Note that these features would not necessarily undermine the utility of lexicons, which is IMHO another indication that lexicons are the Right Way to achieve modularity.)
I did it this way because I thought that it would make package fans feel warmer and fuzzier about lexicons to know that the binding of a symbol S in lexicon L was invariably the symbol L::S. It would be easy enough to abandon this invariant and just use uninterned symbols for the bindings instead. But personally, although I am no fan of packages, the current design has kind of grown on me. It's far from clear that enforcing the global uniqueness of lexicon names is a bad idea, even if it may not be logically necessary.
Ron Garret wrote: > I did it this way because I thought that it would make package fans > feel warmer and fuzzier about lexicons to know that the binding of a > symbol S > in lexicon L was invariably the symbol L::S.
Hmm. I guess that does sound handy. Perhaps my clash complaint is adequately addressed by the user just adopting some explicit naming convention for lexicons, rather than lexicon code doing any name-munging. i.e. a lexicon named =lyekka= with an implementation package named =lyekka= isn't going to clash with a package named lyekka.
May somebody please explain me with simple words why cl packages are hard to use, 'couse I really don't get it? The best is probably to hear from somebody who just learned lisp or somebody who teaches lisp (do such people exist anymore?) and his/her students has problems with packages.
Slobodan Blazeski <slobodan.blaze...@gmail.com> writes: > May somebody please explain me with simple words why cl packages are > hard to use, 'couse I really don't get it?
Well, I'd say you're the best placed to explain with simple words why CL packages are hard to use, because for us they are not hard to use.
But if you cannot explaint it with simple words, you may also use complicated words, we've got good dictionaries.
> The best is probably to hear from somebody who just learned lisp or > somebody who teaches lisp (do such people exist anymore?) and his/her > students has problems with packages.
I don't find CL packages hard to use. On the contrary, they're a very simple concept, and at the same time they're very powerful.
Let's start without packages, and try to solve some simple practical problems.
Assume we create two symbols with the same name, "ABC", and "bind" them to different values:
Then we can find a symbol from a name, for example when we read the string "ABC" from the user, we wan to find what symbol it corresponds to, and retrieve its value:
But how could we find instead the other symbol named "ABC", with value 2? The solution proposed is to put the symbols in different lists, named "packages", having different names. So now, we can find a symbol named "ABC" in a current 'package' refered to by *pack*, or a symbol named "ABC" qualified by a package name such as "YOUR::ABC".
CL:IMPORT puts a symbol on the list of symbols of a package. CL:FIND-SYMBOL finds a symbol in the list of symbols of a package. CL:FIND-PACKAGE finds a package in the list of packages.
CL:INTERN does both CL:FIND-SYMBOL, and if not found, CL:MAKE-SYMBOL and CL:IMPORT.
With CL:USE-PACKAGE, there's a little technicality, in that the symbols from the used packages are not really "interned" or "imported" into the using package, but it works most of the time exactly the same.
Each package maintains also a list of "exported" symbols, and this is used only to signal an error when you try to read a qualified symbol that is not exported with only one colon.
SYMB is exported from PACK SYMB is not exported from PACK PACK:SYMB ok error PACK::SYMB ok ok
Slobodan Blazeski wrote: > May somebody please explain me with simple words why cl packages are > hard to use, 'couse I really don't get it? > The best is probably to hear from somebody who just learned lisp or > somebody who teaches lisp (do such people exist anymore?) and his/her > students has problems with packages.
In article <2c967dc5-a2f7-482d-9f45-b9ad85c8b...@j11g2000yqg.googlegroups.com>, Slobodan Blazeski <slobodan.blaze...@gmail.com> wrote:
> May somebody please explain me with simple words why cl packages are > hard to use, 'couse I really don't get it? > The best is probably to hear from somebody who just learned lisp or > somebody who teaches lisp (do such people exist anymore?) and his/her > students has problems with packages.
> bobi
Packages aren't "hard to use", they just do the Wrong Thing ;-) Packages operate at read-time, but most programmer's intuitions about modularity relate to global bindings, which is a compile-time concept.
The biggest practical problem with packages, and the central motivation for lexicons, is that exported symbol lists have to be maintained manually. This violates the DRY principle, and makes maintenance more difficult than it needs to be, rather like the burden imposed by C on the programmer to maintain the .c file and the .h files in parallel. Lexicons let you write modular code without manually maintaining an export list.
A secondary problem with packages is that because they do what they do at read time, merely parsing the program has potential side-effects (interning symbols). If you attempt to parse a program in an environment where your packages have not been set up properly, these side effects can end up messing up your environment to the point where recovery is very difficult because you suddenly have a zillion symbol interned in a package where they should not be interned. Lexicons avoid this problem because they operate at compile time, not read time. They also have a deferred binding mechanism that allows recovery from unresolved bindings cause by missing libraries to be done at run-time, so if you don't have your libraries set up properly you can recover by simply importing the right library. You don't have to recompile the client code.
Kenny <kentil...@gmail.com> writes: > Slobodan Blazeski wrote: >> May somebody please explain me with simple words why cl packages are >> hard to use, 'couse I really don't get it? >> The best is probably to hear from somebody who just learned lisp or >> somebody who teaches lisp (do such people exist anymore?) and his/her >> students has problems with packages.
With all due respect, this is not the kind of "practical problem" that most programmers deal with on a day-to-day basis, particularly not when they are starting out. Using MAKE-SYMBOL is a pretty advanced concept, and using it to make multiple symbols with the same name is a pretty esoteric technique. In fact, the whole concept of an uninterned symbol is pretty advanced. Peter Seibel's book, for example, doesn't even mention them until chapter 21, and then only in passing. MAKE-SYMBOL is not mentioned at all.
The vast majority of symbols that beginning and work-a-day Lispers deal with are interned by the reader, and most non-advanced Lisp programmers think about modular programming in terms of exporting functions, classes, macros, and methods, not symbols. If you doubt this, do a Google search in c.l.l. for "How do I export a function?" and see how many hits you get.
> > Slobodan Blazeski wrote: > >> May somebody please explain me with simple words why cl packages are > >> hard to use, 'couse I really don't get it? > >> The best is probably to hear from somebody who just learned lisp or > >> somebody who teaches lisp (do such people exist anymore?) and his/her > >> students has problems with packages.
Ron Garret wrote: > In article > <2c967dc5-a2f7-482d-9f45-b9ad85c8b...@j11g2000yqg.googlegroups.com>, > Slobodan Blazeski <slobodan.blaze...@gmail.com> wrote:
>> May somebody please explain me with simple words why cl packages are >> hard to use, 'couse I really don't get it? >> The best is probably to hear from somebody who just learned lisp or >> somebody who teaches lisp (do such people exist anymore?) and his/her >> students has problems with packages.
>> bobi
> Packages aren't "hard to use", they just do the Wrong Thing ;-) > Packages operate at read-time, but most programmer's intuitions about > modularity relate to global bindings, which is a compile-time concept.
And therein lies the answer: Only people who stick to their intuition and try to use packages as if they were about bindings will find it hard to use packages. You have to adjust your expectations, and then packages become very easy. (Some people don't need this kind of adjustment, because the view packages provide comes natural to them.)
IMHO, packages, i.e. managing the mapping from strings to symbols, solve the problem better than modules, i.e. managing the mapping from identifiers to concepts. So I wouldn't agree that they do the wrong thing, to the contrary, I think they do the right thing. But it's good that you put a smiley there... ;)
In article <6q2iu8Fai50...@mid.individual.net>, Pascal Costanza <p...@p-cos.net> wrote:
> IMHO, packages, i.e. managing the mapping from strings to symbols, solve > the problem better than modules, i.e. managing the mapping from > identifiers to concepts.
Ron Garret wrote: > In article <6q2iu8Fai50...@mid.individual.net>, > Pascal Costanza <p...@p-cos.net> wrote:
>> IMHO, packages, i.e. managing the mapping from strings to symbols, solve >> the problem better than modules, i.e. managing the mapping from >> identifiers to concepts.
> Why?
Because they can make distinctions that you cannot make with module systems. For example, if you had only a module system, a class would not be able to inherit from two different classes with slots of the same name and be able to keep them as two different slots (unless you come up with funky renaming mechanisms, like in the R6RS module system, for example).
With a CL-style package system, the choice is always unambiguous: If you use the same symbol as somebody else, you definitely mean the same concept. If you want to denote a different concept, you have to use a different symbol. It's much harder to make such distinctions with module systems.
[Maybe having both packages and modules, like your lexicons, in the same system is beneficial in that it provides the functionalities of both worlds - but I cannot tell because I haven't tried this.]
Ron Garret wrote: > I did it this way because I thought that it would make package fans > feel warmer and fuzzier about lexicons to know that the binding of a > symbol S > in lexicon L was invariably the symbol L::S.
a::x and b::x are different symbols supporting different bindings absent lexicons, IMO lexicons should just allow them to be lexical, not merge them, if lexicons are truly on another axis to packages.