Understanding packages

144 views
Skip to first unread message

Joel Shellman

unread,
Apr 5, 2012, 5:26:35 AM4/5/12
to qil...@googlegroups.com
I'm not clear on how packages are supposed to work.
http://shenlanguage.org/Documentation/shendoc.htm#Packages
Mentions internal symbols are "hidden", but I can still access them.

Ack... I obviously am just not understanding packages. I look at how
the maths lib is written and it "exports" most of the things (lists
them in L). But that means (sign) is defined. So if I also use some
other library that defines a function (sign)... there's a name
conflict which I thought package was supposed to help avoid. So... I'm
confused.

Here's some more evidence of my confusedness. I'm using the clojure
port. I put the following in a file:

(package mypackage- []
(define main -> (output "hello"))
(define test X Y -> (+ X (* 789 (/ 1 5))))
(define doit -> (time (test 5 5)))
)

At the REPL:

(69-) (load "Test.shen")
mypackage-main
mypackage-test
mypackage-doit

run time: 0.07400000000006912 secs
loaded

(70-) (mypackage-test 5 5)
162.8

(71-) (mypackage-doit)
get-time does not understand the parameter mypackage-run

So... I take it that time is actually a macro? and it results in the
use of the symbol run which doesn't exist here. So, I have to exclude
it from the prefixing so the first line should be:

(package mypackage- [run]

Is that right?

Mark Tarver

unread,
Apr 5, 2012, 6:28:32 AM4/5/12
to Qilang
It's a bug; yes time is a macro. The problem is that 'run' is a
keyword and has not been listed as such in the sources. In lieu of
the next upgrade you can write

(package mypackage- [run]
(define main -> (output "hello"))
(define test X Y -> (+ X (* 789 (/ 1 5))))
(define doit -> (time (test 5 5)))
)

which should work fine.

Mark

On Apr 5, 10:26 am, Joel Shellman <j...@mentics.com> wrote:
> I'm not clear on how packages are supposed to work.http://shenlanguage.org/Documentation/shendoc.htm#Packages

Joel Shellman

unread,
Apr 5, 2012, 9:52:26 AM4/5/12
to qil...@googlegroups.com
> It's a bug; yes time is a macro.  The problem is that 'run' is a

Ah, thanks. I still have questions about packages:

It mentions that internal symbols are "hidden", but I can still access
them by including the package prefix on them, so how are they
"hidden"?

Mark Tarver

unread,
Apr 5, 2012, 10:11:17 AM4/5/12
to Qilang
In Lisp, and in Shen, placing something in a package does not mean you
cannot access it. It just means you have to incorporate the package
name into your reference.

e.g.

Qi II 2008, Copyright (C) 2001-2008 Mark Tarver
www.lambdassociates.org
version 1.05


(0-) (FBOUNDP qi::implode) {this is a function in the qi package used
to define Qi}
T

(1-) (define qi::implode X -> X)
WARNING: DEFUN/DEFMACRO: redefining function qi::implode in C:
\Documents and
Settings\User\My Documents\Computer
Science\Languages\Qi\QiII1.05\Lisp\install.lsp, was defined
in
C:\Documents and Settings\User\My Documents\Computer
Science\Languages\Qi\QiII1.05\Lisp\toplevel_load8.fas
qi::implode

As long as your package name is unique you will not get a name clash.

Regarding the exported function symbols, you can, if you wish,
guarantee that they will not be overwritten by giving them the status
of system functions which will cause Shen to raise an error if any
future definition overwrites them.

Shen 2010, copyright (C) 2010 Mark Tarver
www.lambdassociates.org, version 3.1
running under Common Lisp, implementation: CLisp 2.49
port 1.1 ported by Mark Tarver


(0-) (define factorial
0 -> 1
N -> (* N (factorial (- N 1))))
factorial

(1-) (systemf factorial)
[factorial interror intoutput intmake-string abort absvector
absvector? address-> <-address adjoin and append apply arity assoc |
boolean boolean? bound? bind c
lose... etc]

(2-) (define factorial X -> X)
factorial is not a legitimate functor.

Mark

Joel Shellman

unread,
Apr 5, 2012, 10:50:54 AM4/5/12
to qil...@googlegroups.com
Here's an example. Let's say I have two libraries that I want to use.
So, I load them:

Test1.shen:
(package mypackage1- [commonName]
(define commonName -> (output "1"))
)
Test2.shen:
(package mypackage2- [commonName]
(define commonName -> (output "2"))
)

How can I access both versions of commonName? REPL session below
showing how loading the second overwrites the first.

Perhaps I need to take a step back and ask: if I'm writing a package
in a file and it needs to depend on various libraries, what is the
proper way of importing them?

I could see how one could possibly call (load filename), magically
figure out what package it just loaded, use (external pkgname) on that
and prepend a qualifier to all those symbols selectively maybe, but
that would need to run at compile time...


Shen 2010, copyright (C) 2010 Mark Tarver
www.lambdassociates.org, version 3.1

running under Clojure, implementation: Clojure 1.4.0-beta5 [jvm 1.6.0_26]
port 0.1.4 ported by Håkan Råberg


(0-) (load "Test.shen")
commonName

run time: 0.028999999999999915 secs
loaded

(1-) (commonName)
1"1"

(2-) (load "Test2.shen")
commonName

run time: 0.023999999999997357 secs
loaded

(3-) (commonName)
2"2"

Mark Tarver

unread,
Apr 5, 2012, 2:32:24 PM4/5/12
to Qilang
As said, if you want to guarantee your external symbols cannot be
overwritten, you need to systemf them.

In Shendoc it says

The philosophy of Shen is that once a programmer has decided a symbol
is internal to a package and is hidden from view that decision cannot
be overridden except by changing the definition of the package. Hence
the complexities of IMPORT and EXPORT found in Common Lisp are not
reproduced in Shen. You cannot be 'in' a package in Shen within the
REPL. It is possible to declare a package in a package.

Symbols that are declared external are supposed to be freely usable by
all packages.

Mark

On Apr 5, 3:50 pm, Joel Shellman <j...@mentics.com> wrote:
> Here's an example. Let's say I have two libraries that I want to use.
> So, I load them:
>
> Test1.shen:
> (package mypackage1- [commonName]
> (define commonName -> (output "1"))
> )
> Test2.shen:
> (package mypackage2- [commonName]
> (define commonName -> (output "2"))
> )
>
> How can I access both versions of commonName? REPL session below
> showing how loading the second overwrites the first.
>
> Perhaps I need to take a step back and ask: if I'm writing a package
> in a file and it needs to depend on various libraries, what is the
> proper way of importing them?
>
> I could see how one could possibly call (load filename), magically
> figure out what package it just loaded, use (external pkgname) on that
> and prepend a qualifier to all those symbols selectively maybe, but
> that would need to run at compile time...
>
> Shen 2010, copyright (C) 2010 Mark Tarverwww.lambdassociates.org, version 3.1

Mark Tarver

unread,
Apr 5, 2012, 3:02:26 PM4/5/12
to Qilang
> Symbols that are declared external are supposed to be freely usable by
> all packages.

I should have said 'Symbols that are declared external are supposed to
be freely usable by the REPL or in any unpackaged program'. If you
use external symbols S of package A inside package B you have to
specifically say in package B that you want these symbols exported in
the package B. The 'external' function is there to help you recover
those symbols i.e. S = (external A).

Mark

Joel Shellman

unread,
Apr 5, 2012, 3:02:57 PM4/5/12
to qil...@googlegroups.com
> As said, if you want to guarantee your external symbols cannot be
> overwritten, you need to systemf them.

But that's not what I asked. I want to know how I can access both at
the same time.


> Symbols that are declared external are supposed to be freely usable by
> all packages.

But what if they conflict? Thousands of people making various
libraries are going to export the same symbol name because they are
intended to be used. And when I need to use both libraries at the same
time, how do I do that?

Mark Tarver

unread,
Apr 5, 2012, 3:29:05 PM4/5/12
to Qilang
Well, if you make a function external to a package, meaning that you
are allowing people to refer to it without qualification; and somebody
else does the same then either you will get an overwrite or a systemf
error if you try to load both. There is no possibility of being able
to access two different definitions of the same symbol at the same
time. I'm not sure what this could mean actually.

Mark

Joel Shellman

unread,
Apr 5, 2012, 3:48:39 PM4/5/12
to qil...@googlegroups.com
On Thu, Apr 5, 2012 at 12:29 PM, Mark Tarver <dr.mt...@gmail.com> wrote:
> Well, if you make a function external to a package, meaning that you
> are allowing people to refer to it without qualification; and somebody
> else does the same then either you will get an overwrite or a systemf
> error if you try to load both.  There is no possibility of being able
> to access two different definitions of the same symbol at the same
> time.  I'm not sure what this could mean actually.

maths already exports all its functions. So, if that convention is
followed, there are going to be a lot of conflicts.

Is this only a problem in the REPL or unpackaged program? You
mentioned something about how a package could access other packages in
a qualified way and so a packaged program could access both at the
same time?

How does one package state that it depends on another package so that
other package will be loaded and it's symbols become available (when
we're NOT working at the REPL)?

In short, I guess my question is: how does one write and organize an
application in Shen, like when I'm writing an application that may
have 10's or 100's of packages that have various dependencies with
each other and other libraries.

For that matter, is there even a way to run Shen outside the REPL yet.
Sorry... I read every page under Learning Shen and at least most of
the Shen Standard doc, if there is something else I should read
through that answers these questions, let me know.

Mark Tarver

unread,
Apr 5, 2012, 4:37:10 PM4/5/12
to Qilang
Well, maths only exports basic stuff like tan, cos etc. Willi kept
all the auxiliary stuff internal.

If your package P requires other packages to function then you need to
place in your code and before the package itself; a list of files that
you expect the system to load. If you want to refer to the external
symbols of these packages in P, then you need to have something like

(load "P1.shen")
.......
(load "Pn.shen")
(package my-package- (append L (external P1) .... (external Pn))

<package code>)

You can't access two different symbol definitions of one symbol if
they have both been made external. If they are internal to each
package, no problem.

You cannot as yet run Shen outside the REPL though you can use the
dump function to dump native code to a file which can be run outside
the REPL if the native code allows it. Ramil's JS port actually does
not allow this because there is a JS restriction about writing to
files based (I guess) on client security. Hakan's Clojure port does
allow this - at least I seem to remember that his later version did,
which will be be only days old.

Mark

On Apr 5, 8:48 pm, Joel Shellman <j...@mentics.com> wrote:

Joel Shellman

unread,
Apr 5, 2012, 4:50:00 PM4/5/12
to qil...@googlegroups.com
> If your package P requires other packages to function then you need to
> place in your code and before the package itself; a list of files that
> you expect the system to load.  If you want to refer to the external
> symbols of these packages in P, then you need to have something like
>
> (load "P1.shen")

Let's say P1 and P2 both have external symbol S. At this point, could
we prepend all P1's external symbols with "p1-" so when we run:

(load "P2.shen")

it doesn't redefine the prepended symbols and then we prepend all P2's
symbols with "p2-" and then...


> .......
> (load "Pn.shen")
> (package my-package- (append L (external P1) .... (external Pn))

And then here we list the prepended symbol lists (instead of just
(external P#)) we created above? So, now in our package we can access
both p1-S and p2-S.

The above would give us the "qualified import" functionality available
in many languages.

Thanks for your patience in helping me understand.

Mark Tarver

unread,
Apr 5, 2012, 6:51:52 PM4/5/12
to Qilang
I think the easiest way to understand the Shen package system is to
understand how it works. The approach is very simple. Shendoc says

The polyadic function package has the form (package S L E1 ... En)
where

1. S is a symbol beginning in lowercase which is the name of a
package; (e.g mypackage).
2. A list L (possibly empty) of non-variable symbols.
3. E1 ... En are a series of Shen expressions.

The Shen reader prepends the package symbol before all the symbols
when evaluating E1 ... En apart from those symbols which are

(a) symbols listed as belonging to the system (such as ->, define,
cons, append etc) or ...
(b) symbols which are variables or ...
(c) ... symbols which are listed in L or .....
(d) ... symbols internal to the Shen package or ....
(e) ... symbols consisting entirely of underscores or entirely of
equals signs.

So ....

> Let's say P1 and P2 both have external symbol S. At this point, could
> we prepend all P1's external symbols with "p1-" so when we run:

The answer is 'no' because the external symbols of P are fixed by the
package P and not by anything outside of P. But you could use an
external symbol of P1 internally in P2 with a totally different
meaning without any clash. The idea is that when a programmer makes
a symbol external he intends to make that function globally accessible
*under the meaning he has given*, and to override his decision is
really to change the package and this should and does require a code
change in the package itself.

Mark

Joel Shellman

unread,
Apr 5, 2012, 7:36:51 PM4/5/12
to qil...@googlegroups.com
>> Let's say P1 and P2 both have external symbol S. At this point, could
>> we prepend all P1's external symbols with "p1-" so when we run:
>
> The answer is 'no' because the external symbols of P are fixed by the
> package P and not by anything outside of P.  But you could use an
> external symbol of P1 internally in P2 with a totally different
> meaning without any clash

I was suggesting that as a proposal of how we might be able to solve
the problem. I need to access a function with the same name in P1 and
P2 from within my package P3. If you're saying that's impossible... it
makes no sense.

For example, right now I have a workspace open in which there are 1977
methods named "read" across all libraries. The likelihood that at some
point I'll need to access two of those from the same package is very
high. You might suggest they should not be declared external? But then
it's the case of whatever library author "gets there first"? For
example, maths exports the sign symbol. So that means that all future
users of Shen should never declare sign to be external or else others
won't be able to use their package at the same time as the maths
package?

I'm sorry to keep pursuing this, but this is really not making sense
to me. All other languages I have used provide a way to import things
from other packages and deal with conflicting names. So, I'm really
confused by the suggestion that that would not be possible.

Mark Tarver

unread,
Apr 6, 2012, 7:04:30 AM4/6/12
to Qilang
I see your thinking. You say 'I have a workspace open in which there
are 1977 methods named "read" across all libraries.' You want to have
a tool that enables you to deal with a library like this.

The basic answer is that Shen does not have it because I would regard
it as extremely undesirable to have 1977 functions called 'read'
external across a library. So I guess for me the question is a bit
like 'How does your plane fly when the tanks are empty and a wing has
fallen off?' . The answer is it does not, because you are not
supposed to get into that awful state. Providing support tools for
people to do bad things was not my goal. Library management means
avoiding that mess.

> I was suggesting that as a proposal of how we might be able to solve
> the problem. I need to access a function with the same name in P1 and
> P2 from within my package P3. If you're saying that's impossible... it
> makes no sense.

What I'm saying is that it makes no sense to maintain a library where
this happens because the point of making a symbol external is that you
are saying 'This is unique and can be used freely in the REPL'. I
actually would incline to 'systemf' all external symbols of a package
so that any programmer who did overwrite one definition with another
in loading packages would raise an error and be required to actually
change the code he is loading.

Mark

Joel Shellman

unread,
Apr 6, 2012, 11:59:04 AM4/6/12
to qil...@googlegroups.com
> The basic answer is that Shen does not have it because I would regard
> it as extremely undesirable to have 1977 functions called 'read'
> external across a library.

It's many libraries--hundreds. And since they are written by different
authors, there's no way to coordinate who is "allowed" to externalize
"read" and who is not. As soon as one person has done so in Shen, he
makes it impossible for anyone else to do it.

> supposed to get into that awful state.   Providing support tools for
> people to do bad things was not my goal.  Library management means
> avoiding that mess.

So avoiding externalizing at all? Then why have that feature?

> What I'm saying is that it makes no sense to maintain a library where
> this happens because the point of making a symbol external is that you
> are saying 'This is unique and can be used freely in the REPL'.

But that is just one library author. There will be thousands of
library authors. And as soon as one of them externalizes a name, it
means that noone else can.

And I'm not talking about the REPL. The REPL is not really interesting
to me. It's useful during development, but you don't write
applications that way, do you? I understand Shen is kind of limited to
the REPL right now, but I'm looking forward to how I can write big
complex applications with it. And package/namespace management is a
crucial part of any such undertaking.


> I actually would incline to 'systemf' all external symbols of a package
> so that any programmer who did overwrite one definition with another
> in loading packages would raise an error and be required to actually
> change the code he is loading.

In what I'm discussing, you can't change the code you're loading. It's
written by someone
else, it's binary, it's not your code. That's like telling me I need
to go change the maths library every time I run into a conflict,
except I might not even have the source code to the library so it
would be impossible for me to change it even if I wanted to.

Mark Tarver

unread,
Apr 6, 2012, 1:07:33 PM4/6/12
to Qilang


On Apr 6, 4:59 pm, Joel Shellman <j...@mentics.com> wrote:
> > The basic answer is that Shen does not have it because I would regard
> > it as extremely undesirable to have 1977 functions called 'read'
> > external across a library.
>
> It's many libraries--hundreds. And since they are written by different
> authors, there's no way to coordinate who is "allowed" to externalize
> "read" and who is not. As soon as one person has done so in Shen, he
> makes it impossible for anyone else to do it.

What you need here is a library management tool, not a package
management tool, to ensure that externalised symbols remain unique
within the library. Not an impossible task but not one that we need be
too unduly worried about at this stage. Yes; if someone takes your
favourite word and externalises it, it can be mildly annoying, but not
generally incapacitating. Surely better than having 1977 meanings of
one symbol pushed to the front and people never being certain in any
application, exactly what one they are talking to.

> So avoiding externalizing at all? Then why have that feature?

Because basic bread-and-butter functions like 'tan' are more usefully
invoked in that form rather than 'maths-lib-tan'. We try to avoid
cluttering up the namespace with auxiliary functions like 'tan-help'
etc.

> In what I'm discussing, you can't change the code you're loading. It's
> written by someone
> else, it's binary, it's not your code. That's like telling me I need
> to go change the maths library every time I run into a conflict,
> except I might not even have the source code to the library so it
> would be impossible for me to change it even if I wanted to.

The library is open source BSD.

Mark

Mark Tarver

unread,
Apr 6, 2012, 2:29:46 PM4/6/12
to Qilang
If the thing that is bothering you is that you want to use 'sign' but
the maths library has used it; then you can get round this by using
macros. You simply define a macro that maps 'sign' to your internal
symbol. If you load the maths library first and then your macro, you
will have all the benefit of the maths library and control over the
'sign' symbol. However anybody using your work and expecting to find
'sign' with the original maths meaning will be surprised no doubt.

Mark

Thomas Bartscher

unread,
Apr 7, 2012, 7:14:20 AM4/7/12
to qil...@googlegroups.com

Am Freitag, 6. April 2012 19:07:33 UTC+2 schrieb Mark Tarver:


On Apr 6, 4:59 pm, Joel Shellman <j...@mentics.com> wrote:
> > The basic answer is that Shen does not have it because I would regard
> > it as extremely undesirable to have 1977 functions called 'read'
> > external across a library.
>
> It's many libraries--hundreds. And since they are written by different
> authors, there's no way to coordinate who is "allowed" to externalize
> "read" and who is not. As soon as one person has done so in Shen, he
> makes it impossible for anyone else to do it.

What you need here is a library management tool, not a package
management tool, to ensure that externalised symbols remain unique
within the library.
Why unique "in the library"? It's not about a symbol being unique in one library but across several (if I understood Joel Shellman right).
 
> So avoiding externalizing at all? Then why have that feature?

Because basic bread-and-butter functions like 'tan' are more usefully
invoked in that form rather than 'maths-lib-tan'.  We try to avoid
cluttering up the namespace with auxiliary functions like 'tan-help'
etc.
But one writing on a sunbathing library might need the math library while using his tan function more often.
 
> In what I'm discussing, you can't change the code you're loading. It's
> written by someone
> else, it's binary, it's not your code. That's like telling me I need
> to go change the maths library every time I run into a conflict,
> except I might not even have the source code to the library so it
> would be impossible for me to change it even if I wanted to.

The library is open source BSD.
The sunbathing library is proprietary and you cannot change the code.
 
I thought packaging was there to avoid naming conflicts and exporting was there for convenience. Where is the point when exporting leads to naming conflicts despite packaging and packaging only has any meaning when you don't export symbols?

Mark Tarver

unread,
Apr 7, 2012, 11:15:20 AM4/7/12
to Qilang
> Why unique "in the library"? It's not about a symbol being unique in one
> library but across several (if I understood Joel Shellman right).

There's several things being mixed together here. The number of
libraries is not important. And its not true that 'As soon as one
person has [made a symbol external] in Shen, he makes it impossible
for anyone else to do it.' You can, at the moment, overwrite external
symbols. What is important is that in a well organised library, which
is what a standard library is, you do not want this to happen. This
IMO is a question of civilised library management. What happens in
the Wild West is another matter altogether.

> > > So avoiding externalizing at all? Then why have that feature?

As explained; externalising makes sense when you have a symbol like
'tan' with a recognised meaning. So you make it external for ease of
use.

> But one writing on a sunbathing library might need the math library while
> using his tan function more often.

I don't understand this statement.

> The sunbathing library is proprietary and you cannot change the code.

Well arguably if somebody has made their code proprietary and
externalised a function with a recognised meaning perhaps one should
respect their choice and, if using their work, accept their decision.

>
> I thought packaging was there to avoid naming conflicts and exporting was
> there for convenience. Where is the point when exporting leads to naming
> conflicts despite packaging and packaging only has any meaning when you
> don't export symbols?-

Packaging does not eliminate naming conflicts; what it does do is
reduce their probability by hiding auxiliary functions and
(potentially) providing a structured way of dealing with them when
they do arise. I would, for example, favour 'systemf'ing external
symbols in the standard library to make it impossible to overwrite
external functions thus avoiding overwrites.

Mark
Reply all
Reply to author
Forward
0 new messages