Usually the operator of a compound form is evaluated for its
functional value at execution time [1].

What if this evaluation of functional value of operators was rather
done at load time? What would be the consequences?

One obvious consequence would be that some form of forward
reference to function definition would have to be provided by mean
of back-patching or something equivalent. But this is hardly new
and unusual since compilers and linkers in other languages have been
handling such things for many decades now.

Another one is that in a multi-threaded context the following
function has an intuitive and predictable behavior (which is
usually to return T or some other equivalent of true) as long as
function foo is stateless and returns a result that is palatable
to =, this even if foo is redefined during the execution of bar.

(defun bar (y) (= (foo y) (foo y)))

And finally, if parts of the machine code generation is done at
load time by the CL runtime system then some significant new
optimizations become available that were not previously possible.

If load-time evaluation of operators is made the default behavior
of then system then a new declaration, say "fspecial", can easily
be provided to allow access to the old behavior of execution-time
evaluation for the few occasions where this could be desirable
or required.

[1] See HyperSpec 3.1.2.1.2.3 at
http://www.lispworks.com/documentation/HyperSpec/Body/03_ababc.htm

Cheers,

Jean-Claude Beaudoin

Bien faire et laisser braire.

Check rather: http://www.lispworks.com/documentation/HyperSpec/Body/11_abab.htm
and:          http://www.lispworks.com/documentation/HyperSpec/Body/03_bbc.htm  

Inside a file, any reference to another function in the same file can
be compiled hard-wired (unless the called function is declared
notinline).

Any function in the CL package can be hard-wired, or open-coded (as if
it was declared inline).

What more do you want?  
If your implementation doesn't do what you want, just patch it to do it
(in conformance with ANSI CL).

--
__Pascal Bourguignon__                     http://www.informatimago.com/

File scope is not quite enough, it's too limiting. What one wants is more
at the scope of a whole library or sub-system. At the level of an ASDF
one might say.

But thanks for the references, it shows that they were struggling with
the issue even back then...

Yes, and for that there is of course WITH-COMPILATION-UNIT.

And solved it.

--
__Pascal Bourguignon__                     http://www.informatimago.com/

On 9/4/2010 5:50 AM, Pascal J. Bourguignon wrote:
Thanks. I did not know this.

I know in other (staticky :) ) languages, inline generally means no
function call. I see it's the same in CL too.
But in notinline we have two steps, lookup symbol function value cell
and call function?
I did not see any reference to the symbol function cell in the
inline/notline CLHS page. I guess it does not matter?
Interestingly here 'notinline' is not the exact opposite of inline.
So to skip just the symbol look up part , one should arrange the caller
and callee to be in the same compilation unit.

This is an interesting twist on function look ups for sure.
Pascal, how did you even know this. Did you research this, or you
already knew. Just curious.

I am wondering how to make use of this information as a CL user.
I suppose one lesson is , load the whole file if you intend to redefine
any functions?

Thanks,
-Antony

On 2010-09-05 01:15:17 +0100, Pascal J. Bourguignon said:

I don't think that this allows the system to treat multiple files as
one.  It looks to me that what it does is to cause the system not to
warn about things like undefined functions until the end of the unit,
but not to be able to make the same assumptions it can within a file
(which are much stronger).  That's a deficiency, I think - it would be
nice to be able to have an option which says "treat all these files as
one thing".  Obviously one could achieve this by concatenating them...

I think I mentioned earlier that it would be nice to have a
finer-grained way of declaring things than NOTINLINE and INLINE.  
NOTINLINE says pretty much "assume nothing, you need to look it up
every time", but INLINE says "not only assume the definition is the one
you can see now, but inline it if you can", which is too much.  I'd
like "STATIC" which says "the function you call will be the same one
you can see now, but I'm not encouraging you to inline it
particularly".  In particular, this declaration would enable the
compiler to assume as much knowledge about user-defined things as it
has about the symbols exported from the CL package.

I've been reading cll for more than ten years...  You could read
archives, with the advantage that you would be able to skip
uninteresting flames.

For this,  you would need to check the documentation of your specific
implementation.  What the standard specifies is "MAYs", not "MUSTs",
so the actual behavior depends on the implementation.  

clisp doesn't do any inlining and always go thru the symbol.

sbcl inlines functions declared inline with sufficient optimization
levels, but doesn't inline other file local functions.

I've not tried other implementations.

--
__Pascal Bourguignon__                     http://www.informatimago.com/

You're correct, it seems to restrict that to warnings, but the
discussion of the issue mentions that other things including sharing
the compilation environment could be added as extensions.

--
__Pascal Bourguignon__                     http://www.informatimago.com/

On 2010-09-05 13:55:51 +0100, Pascal J. Bourguignon said:

Yes, you're right - I either had not read properly or had forgotten,
but even the spec says explicitly that you are allowed to add
extensions.

I'd still like something like my approach of being able to declare
things once and for all about functions as well.  I'd guess that most
implementations have something like that, because they presumably want
to be able to allow the compiler to make assumptions about internals of
the implementation.

The flip side of this is: if you do go through the symbol things like
redefinition and tracing are probably easier, and is the cost actually
excessive?  My guess is that it might not be, but it would be
interesting to hear from implementors on that.

In most implementations, I believe, this evaluation consists of a single
indirect load instruction.  There's less to be saved here than you might
think.

-- Scott

Pascal J. Bourguignon <p...@informatimago.com> wrote:
+---------------
| clisp doesn't do any inlining and always go thru the symbol.
|
| sbcl inlines functions declared inline with sufficient optimization
| levels, but doesn't inline other file local functions.
|
| I've not tried other implementations.
+---------------

CMUCL will produce multiple entry points to functions with complex
arg lists (&OPTIONAL, &KEY, etc.): a "generic" entry point for calls
from outside the compilation unit, and possibly additional specialized
entry points for calling from locations in which the target function
is "known". In the latter case(s), the compiler can ensure that the
argument invariants are met at the call site and then call a special
"NO-ARG-PARSING" entry point that skips the normal tests for that
call signature.

Said another way: the compiler can skip the tests for &OPTIONAL, etc.,
if there is an alternate entry point that exactly matches the call site
signature. [And there often will be, if the callee & caller were compiled
together.]

It also supports the INLINE, NOINLINE, and EXTENSIONS:MAYBE-INLINE
declarations:

    The EXTENSIONS:MAYBE-INLINE declaration is a CMUCL extension. It is
    similar to INLINE, but indicates that inline expansion may sometimes
    be desirable, rather than saying that inline expansion should almost
    always be done. When used in a global declaration, EXTENSIONS:MAYBE-INLINE
    causes the expansion for the named functions to be recorded, but the
    functions aren't actually inline expanded unless space is 0 or the
    function is eventually (perhaps locally) declared INLINE.

See "Chapter 5: Advanced Compiler Use and Efficiency Hints" in the
"CMUCL User's Manual".

-Rob

-----
Rob Warnock                     <r...@rpw3.org>
627 26th Avenue                 <URL:http://rpw3.org/>
San Mateo, CA 94403             (650)572-2607

I'm not sure how to take the would quoted passage.  It is yet another
place where X3J13 punted on a difficult issue and drafted nonsense.

The problem is that the term "defined in" is not defined in the ANS.
It might be meaningful referring to a regular function, but is quite
ambiguous applied to a generic function which has a distributed
implementation that might or might not extend beyond a single file.

On 2010-09-07 02:00:09 +0100, smh said:

There seems to be a current trend of depressingly literalist readings
of the spec - a kind of peculiar fundamentalism with the lack of any
willingness (or perhaps ability) to interpret.  I'm not sure why it's
happening: probably it's a combination of no-one who was involved in
creating it still posting here and the declining levels of general
education of programmers as it becomes a more specialised skill.  It's
really rather sad.

Anyway, so in this case: absent sealing, the set of methods on a
generic function is unbounded.  So no, it is not safe to assume you
know the whole definition of the function.  It *is* safe to assume you
know that the GF you know is the one which will you will be calling (so
for instance you can avoid the symbol->function lookup, if that matters
to you, and you can optimise away signature checking).

There is one edge case which is: if when calling the GF you know the
exact type of its arguments, and the GF also has *all possible* methods
which might apply to those arguments defined in the same file (so a
full set of before, after, around, and main methods), can you assume
that you know everything?  For the sake of argument, I'll say that I
think you can't.

On Sep 7, 1:47 am, Tim Bradshaw <t...@tfeb.org> wrote:

I'm only going to comment on your metacomment here, which was a bit of
unfortunate timing on your part in unleashing your frustration.  The
poster you responded to, smh, who is Steve Haflich, is indeed one of
those originally involved with the standardization process; he also
was the chair of the committee which followed up on the spec for many
years afterward (no changes to the spec were made during that time,
due to extreme lack of will of the Common Lisp community to put their
money where their mouths were for committee memberships, but at least
there was enough of a committee left to re-certify the spec as
needed).  Steve knows the spec inside and out, and more importantly,
he knows the intentions of the spec, from many different points of
view - he was there, and can tell you where the spec was well-agreed,
and where the design was thin.  Sometimes he can even remember (and if
he doesn't he has notes from the X3J13 committee meetings above and
beyond the formally logged actions to remind him) who said what and
why.  I always end up consulting Steve when I need to know not only
the literal meaning of the spec, but the intentions behind it.

I don't know what is sad to you; you're hearing from one of the
originals here.  He just doesn't pull any punches, so just roll with
it and get up off the floor and try again.

Duane

Well, I think with either the INLINE or the STATIC, there is also the
implicit promise on the part of the user that he will NOT redefine the
function.  

I would expect, as usual with the spec, that if you were to declare a
function STATIC and then redefine it anyway, you couldn't really predict
if the old or new definition were to be used.  You can't even get that
guarantee for INLINE declarations, since the compiler is free to choose
not to inline the code.

So I don't think either of these is a real solution to the OP's problem
of multi-thrreading and handling function redefinition during run-time.

--
Thomas A. Russ,  USC/Information Sciences Institute

* Tim Bradshaw <i64u72$ej...@news.eternal-september.org> :
Wrote on Tue, 7 Sep 2010 09:47:31 +0100:

| On 2010-09-07 02:00:09 +0100, smh said:
|
| There seems to be a current trend of depressingly literalist readings
| of the spec - a kind of peculiar fundamentalism with the lack of any
| willingness (or perhaps ability) to interpret.  I'm not sure why it's
| happening: probably it's a combination of no-one who was involved in
| creating it still posting here and the declining levels of general
| education of programmers as it becomes a more specialised skill.  It's
| really rather sad.

lies, lies and more lies.  pathological.

--
Madhu

In article <ymifwxlqwea....@blackcat.isi.edu>,
 t...@sevak.isi.edu (Thomas A. Russ) wrote:

What we struggled with when writing the standard was that we didn't want
to require specific compiler strategies.  So we tried to specify
high-level semantics, but in a way that allowed (and strongly suggested)
these optimizations.

So the term "INLINE" suggests a particular method of optimization, but
the description of it merely *allows* that technique, by specifying that
a redefinition of the function need not have an effect on callers that
were already compiled.  We then realized that this same semantic issue
also works for controlling optimization of intra-file calls.

--
Barry Margolin, bar...@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***
*** PLEASE don't copy me on replies, I'll read them in the group ***

On 2010-09-07 16:39:41 +0100, Duane Rettig said:

I did not know it was Steve (though if I'd read the address more
carefully I could probably have inferred that).  Obviously I know he
was there.

This does not change my view on the stupidity of what he said
unfortunately: without some kind of sealing a generic function can not
ever be completely defined, let alone defined within a single file or
other compilation unit, so the meaning of the spec is quite clear: "is
defined" is not ambiguous at all.

On 2010-09-07 18:46:37 +0100, Thomas A. Russ said:

I think STATIC is exactly such a solution: if you redefine a function
you have declared STATIC then your program is non-conforming and all
bets are off.  If you declare it <whatever the opposite of static is,
which is not quite NOTINLINE> then you should not make any such
assumptions, and redefinition ought to work,

My concern in my OP was not optimization but thread-safety.
Optimization is optional in nature, thread-safety cannot be optional
or at the discretion of the runtime system.
In these days of multi-core CPUs, a language in which a reasonably
thread-safe application cannot be written using normal construct of
the language does not have much of a future.

 From experience I get the impression that the average size of
a software product has grown substantially during the last 20 years,
by a factor of 10 to 50 times. A 10K lines compiler is considered
a mere component these days, if it is not seen as a toy. In that
context the relative value of the file as a unit of scope has been
greatly diminished almost to the point of irrelevance.

Cheers,

Jean-Claude Beaudoin

I realize now that when I sent my OP I did not appreciate properly
the game changing nature of generic functions.

The behavior of the "bar" function of my OP cannot be predicted
at all if function "foo" happens to be a generic function in
a multi-threaded context.

As currently specified generic functions are functions forever
in flux. This is a powerful feature but also a very difficult one
to control. We need some way to tame it somewhat.

Something like (close-generic-function <gf-name>*) could do the
trick. Such a closed generic function would have a fixed set of
methods and would not be in flux anymore and thus would be much
closer to a normal function.

What to do if a method is added to a closed generic function?
One option would be to simply signal an error but that would
not be very useful.  Another one would be to consider this as
a request to create a whole new generic function, one that picks
up where the previous closed generic function left off but with
a new identity and then add the method to this newly open
generic function. (In this here there is a parallel with the
class redefinition semantics I mentioned in a previous post).
A warning could also be considered.

Closed generic function would be thread-safe in the sense of
the OP's "bar" function behavior being predictable again but
it would not be the case for open generic functions.

Cheers,

Jean-Claude Beaudoin

P.S.: Once in a while this newsgroup turns into a very useful forum.
It seems to me that this is one of those occasions. Some progress
could be made here.

On 2010-09-08 17:10:08 +0100, Jean-Claude Beaudoin said:

This is basically whas sealing (which I mentioned in an earlier thread)
does, although as specified in Dylan it is more general than this.

But surely no-one writes programs which redefine classes or generic
functions and *expects* that stuff to be thread safe?

Well, I suppose in a narrow sense this is true.  But for the OP's
problem, any function that is in fact static won't cause problems
regardless of whether it is declared as such or not.  If you don't
redefine it, ever, then you can't have thread issues with redefinition
(by definition).

Other than perhaps inlining, though, I don't see what use a compiler
might make of this information.  So I guess I don't see the practical
benefit.  Well, maybe there is a sort of "partial inlining" that could
happen where the reference becomes a fixed jump rather than an indirect
one, but without actually inserting the code in place.

--
Thomas A. Russ,  USC/Information Sciences Institute

Well, this can, of course, be accomplished by software engineering
practices, such that you never add additional methods to any of the
generic functions once you deploy your system.  That makes them
effectively closed.  The benefit of an explicit declaration would be
that it would license certain compiler optimizations.

Well, I would be a fan of the error answer.

But I'm not at all sure how this whole new generic function thing would
work.  Would it just apply to newly compiled code (similarly to what
happens if you redefine a function that has been in-lined?).  And if you
do that, there isn't really any benefit over just creating a real, new
generic function and then programming using that one instead.

Otherwise you still are left with an updating problem since you have to
decide when to substitute the new function into the old one.  I don't
see how you would be able to automatically find a safe time to do this.

Well, it would be predictable, but it I don't see how it would be
updatable.  Especially when you consider that in a real system, the
potential places you have to check would be rather large.  In addition
to

  (defun bar (x) (+ (foo x) (foo x)))

you also have to worry about

  (defun baz (x) (+ (foo x) (bar x)))

where lexical analysis doesn't solve the problem for you.

I suppose one answer might be to have any new definitions only apply in
newly created threads.  Then any executing thread maintains its
environment until it terminates, and changes only affect threads created
after the change takes place.  But it seems that could be applied
universally and wouldn't need any notion of closed or static functions
to operate in that manner.

--
Thomas A. Russ,  USC/Information Sciences Institute

A normal function.   Such as this one for example:

(defvar *commands* (make-hash-table))

(defun do-command (object)
   (funcall (gethash (class-name (class-of object)) *commands*
                     (lambda (object)
                        (error "There's no command for an ~A such as ~S"
                               (class-name (class-of object)) object)))
            object))

(defun add-command (class cmd)
    (setf (gethash class *commands*) cmd))

(defmacro define-command ((object class) &body body)
   `(add-command  ',class (lambda (,object)  ,@body)))

What to do if a function makes use of run-time data?

Yes.  And functions that would not modify data at run-time would be
thread-safe too.  You want a pure functional language.  Unfortunately:

1- state and state-modification is good (in measured ways),

2- even in a pure functional language, you can reintroduce state and
   state-modification (by way of monads).  I don't know enough about
   them, but I'd suspect you'd be back to step 1, thread-wise.

IMO, the newsgroups only fall down to flames and chitchat when there's
no interesting questions like yours passing by.  What we need, is more
interesting queries!

--
__Pascal Bourguignon__                     http://www.informatimago.com/