The next ballot for R7RS-large is here

[John Cowan]

tl;dr:  Which docket is next, Orange or Kronos?

For the first time since the Tangerine Edition was balloted in 2019, another ballot is ready to go.  My original plan once the Orange Docket was sufficiently in shape (all the 23 SRFIs in it except one are finalized, and I think the one is very close to being so), was to make Orange the next docket we voted on: it contains three sections: numeric and quasi-numeric types, some syntax-rules macros, and "other".  All the proposals are portable.

However, some people have asked me to do a different ballot first so we can settle what macro systems other than syntax-rules will become part of R7RS-large. Once that is decided, a good many non-portable proposals may become portable.  Because I have moved a great many proposals between ballots, I decided to abandon the color names with their roughly spectral order and name each ballot after a minor Greek god/dess, with no implications about order of voting.  

Therefore, unless I get strong feedback to the contrary in the next week or so, I will create a ballot form for the Kronos Docket, which contains mostly macro-related proposals:  9 SRFIs; three unmodified R6RS sections (syntax-case, identifier-syntax as an operand, identifier-syntax for `set!`); and explicit renaming, which currently links to the MIT explanation as most comprehensive.  (The explanation in SRFI 211 is more precise but also depends on understanding syntax-case concepts.)  PLEASE NOTE that syntax-case and explicit renaming will be voted on separately, not as alternatives to one another.  They are equivalent in expressive power, but neither can be implemented on top of the other.

See <https://github.com/johnwcowan/r7rs-work/blob/master/ColorDockets.md> for all the dockets.  After summaries of Red and Tangerine, Orange is next and then Kronos.  Note that whichever docket we vote on, there will also be a straw poll to assist me and the various SRFI implementers in deciding which of the proposals in the Eos and Leto Dockets should be implemented for SRFI purposes.

[Marc Nieper-Wißkirchen]

Am Mi., 4. Aug. 2021 um 22:13 Uhr schrieb John Cowan <co...@ccil.org>:

Therefore, unless I get strong feedback to the contrary in the next week or so, I will create a ballot form for the Kronos Docket, which contains mostly macro-related proposals:  9 SRFIs; three unmodified R6RS sections (syntax-case, identifier-syntax as an operand, identifier-syntax for `set!`); and explicit renaming, which currently links to the MIT explanation as most comprehensive.  (The explanation in SRFI 211 is more precise but also depends on understanding syntax-case concepts.)  PLEASE NOTE that syntax-case and explicit renaming will be voted on separately, not as alternatives to one another.  They are equivalent in expressive power, but neither can be implemented on top of the other.

The statement that ER and syntax-case are equivalent in expressive power is wrong. With ER alone, correct unhygienic macros aren't possible.

The examples of 'loop' and 'while' on MIT's page do not work in all circumstances (see SRFI 211).

If you want correct unhygienic macros with ER based on existing implementations, you need to borrow forms from syntax-case (which are usually not part of existing ER systems).  SRFI 211 strives to correct ER but is yet only implemented fully in Unsyntax.  (And once you have given ER the power, you can add more or less for free the full syntax-case system.)

I think it is better if people are fully and correctly informed before the voting process starts, especially in the area of macro systems where, unfortunately, a lot of FUD was spread.

[John Cowan]

On Wed, Aug 4, 2021 at 5:29 PM Marc Nieper-Wißkirchen <marc....@gmail.com> wrote:

The examples of 'loop' and 'while' on MIT's page do not work in all circumstances (see SRFI 211).

If you want correct unhygienic macros with ER based on existing implementations, you need to borrow forms from syntax-case (which are usually not part of existing ER systems).  SRFI 211 strives to correct ER but is yet only implemented fully in Unsyntax.  (And once you have given ER the power, you can add more or less for free the full syntax-case system.)

Can you take a look at <https://www.mail-archive.com/larcen...@lists.ccs.neu.edu/msg00097.html> and see if this is equivalent to what you are describing?  It depends on an underlying syntax-case implementation in which the input to a transformer is unwrapped, such as the one in SRFI 72.  Both are by Andre van Tonder.

 

I think it is better if people are fully and correctly informed before the voting process starts, especially in the area of macro systems where, unfortunately, a lot of FUD was spread.

So do I, absolutely.

[Marc Nieper-Wißkirchen]

Am Do., 5. Aug. 2021 um 01:33 Uhr schrieb John Cowan <co...@ccil.org>:

On Wed, Aug 4, 2021 at 5:29 PM Marc Nieper-Wißkirchen <marc....@gmail.com> wrote:

The examples of 'loop' and 'while' on MIT's page do not work in all circumstances (see SRFI 211).

If you want correct unhygienic macros with ER based on existing implementations, you need to borrow forms from syntax-case (which are usually not part of existing ER systems).  SRFI 211 strives to correct ER but is yet only implemented fully in Unsyntax.  (And once you have given ER the power, you can add more or less for free the full syntax-case system.)

Can you take a look at <https://www.mail-archive.com/larcen...@lists.ccs.neu.edu/msg00097.html> and see if this is equivalent to what you are describing?  It depends on an underlying syntax-case implementation in which the input to a transformer is unwrapped, such as the one in SRFI 72.  Both are by Andre van Tonder.

Thanks for pointing this out.  With respect to that `bound-identifier=?' instead of `eqv?' must be used to compare the identity of identifiers, Van Tonder's implementation uses the same approach as SRFI 211.  To support traditional ER macros, SRFI 211, however, also allows raw symbols in the output of the transformer (which are then closed in the macro use environment).  It should be noted that this changes the algorithmic complexity of ER macros into that of IR macros.  (But this is already the case of Van Tonder's approach when used in a syntax-case implementation that has non-fully unwrapped syntax objects.)  Another difference is that Van Tonder's er-transformer is syntax, while SRFI 211's er-macro-transformer is a procedure (which is correct, I think).

From a semantic point of view, I think SRFI 211 gets it completely right (and if not, I should amend it) but it's slower than traditional ER systems (that do not behave well when hygiene is broken as also observed by Van Tonder).

So before voting on whether to include ER or not, we should settle which version and see that this version gets also the corner cases right.

It should also be noted that syntax-case has none of these problems.

 

 I think it is better if people are fully and correctly informed before the voting process starts, especially in the area of macro systems where, unfortunately, a lot of FUD was spread.

So do I, absolutely.

Thanks.

Would you mind postponing discussing the ballot and the actually ballotting by a few weeks.  At least in many places in the northern hemisphere, there are currently holidays so a part of our community wouldn't be able to discuss these important matters.

As for what's on the ballot, independently of what I think would be the best choice, just asking for ER and syntax-case independently is not enough, I think. People may have a strong opinion to include only one system, no matter what. Or an opinion of, say, ER at all events and maybe syntax-case, but not vice versa. A yet missing option is on the R4RS low-level macro system (see the appendix of R4RS and SRFI 211). It combines the soundness of the syntax-case system with the no-bells-and-whistles approach of ER (meaning no pattern matcher in the form of syntax-case). Together with SRFI 213 (orthogonal to all considerations so far), it would even allow providing syntax-case as a library.  And it does allow Van Tonder's approach to ER macros.

As for the other items on the Kronos list:

- Please postpone SRFI 147 to a later ballot.  It is for systems not necessarily offering procedural macros and is not compatible with syntax-case because syntax-case says that the <transformer> in a syntax definition is an ordinary expression.  (When available, the latter is much more powerful and definitely better.)

- Similarly, please postpone SRFI 149.  Syntax-case already includes it but does one thing differently (see the PFN of SRFI 149). I think this was observed by Will Clinger first.

- Before voting for the identifier-syntax form in R6RS (which is just library syntax in terms of syntax-case), it is important to vote for what is a macro use as per section 9.2 of the R6RS.  In R7RS, it is only the first form.  In R6RS, there are four forms.

- What is meant by "SRFI 17 or identifier-syntax"? How are they related? (See also my latest private email on this topic.)

But I feel that a meta-question should be answered first.  Does R7RS-large strive for compatibility with R6RS and existing R6RS implementations or is this considered irrelevant?  A high-quality implementation like Chez Scheme may be able to support some limited forms of ER, but it will probably never support the "raw" MIT/GNU form.  The same can be said about GNU Guile and Racket.  When R7RS was split into two languages, it was said that R7RS-small would not replace R6RS but only R5RS, while R7RS-large would be the language to replace R6RS.  For this to be possible, it should be at least as powerful and expressive and should IMO, if possible, retain backward compatibility with R6RS where it cannot be resolved through the library system (e.g. that "error" from (scheme base) is different than "error" from (rnrs base (6))).  While the latter is my opinion, the community may think differently but I believe that this sort of meta-question should be answered first.

[Dr. Arne Babenhauserheide]

Marc Nieper-Wißkirchen <marc....@gmail.com> writes:
> But I feel that a meta-question should be answered first. Does R7RS-large
> strive for compatibility with R6RS and existing R6RS implementations or is
> this considered irrelevant?

This question has been my thought, too, since I read the message, but I
didn’t see the approach to ask it.

I have some very non-trivial syntax-case macros in my hobby-projects
that I wrote when I had more free time. I rely on them and need them,
but I don‘t know whether I’d ever be able to port them to a new
macro-system. So if compatibility with R6RS is not a goal, these
projects — some of my most exciting projects — could forever be barred
from porting to R7RS.

Best wishes,
Arne
--
Unpolitisch sein
heißt politisch sein
ohne es zu merken

[Marc Nieper-Wißkirchen]

Am Do., 5. Aug. 2021 um 09:23 Uhr schrieb Dr. Arne Babenhauserheide <arne...@web.de>:

[...]

I have some very non-trivial syntax-case macros in my hobby-projects
that I wrote when I had more free time. I rely on them and need them,
but I don‘t know whether I’d ever be able to port them to a new
macro-system. So if compatibility with R6RS is not a goal, these
projects — some of my most exciting projects — could forever be barred
from porting to R7RS.

Would you mind sharing them? I'm asking because due to their non-triviality, they could become a kind of real-world test case on the expressiveness of the systems under debate and on how hard it would actually be to change code to accommodate one or the other option.

[Dr. Arne Babenhauserheide]

Marc Nieper-Wißkirchen <marc....@gmail.com> writes:

> Am Do., 5. Aug. 2021 um 09:23 Uhr schrieb Dr. Arne Babenhauserheide <
> arne...@web.de>:
>
> [...]
>
> I have some very non-trivial syntax-case macros in my hobby-projects
>> that I wrote when I had more free time. I rely on them and need them,
>> but I don‘t know whether I’d ever be able to port them to a new
>> macro-system. So if compatibility with R6RS is not a goal, these
>> projects — some of my most exciting projects — could forever be barred
>> from porting to R7RS.
>>
>
> Would you mind sharing them?

Gladly. They are already online.

They start with say-words that prints symbols character by character,
executing inline scheme when it is printed:

(define-syntax say-words
(lambda (x)
(syntax-case x ()
((_ word word2 words ...)
#`(begin
(let ((w `word))
(cond
((equal? w #f)
#f)
((equal? w '..)
(show ".")
(show " "))
(else
(show (->string w))
(show " "))))
(say-words word2 words ...)))
((_ last-word words ...)
#`(begin
(let ((w `last-word))
(cond
((equal? w #f)
#f)
((equal? w '..)
(show "."))
(else
(show (->string w)))))
(say-words words ...)))
((_)
#`(begin "")))))

https://hg.sr.ht/~arnebab/dryads-wake/browse/enter.scm?rev=0d065c9ddcc4#L68

And progress to Enter which creates macros that allow writing a stage
play in scheme:

(define-syntax Enter
(lambda (x)
(syntax-case x ()
((_ (name more ...) b ...)
; new binding: only create it if the binding is not already a macro
(not (eq? 'macro (syntax-local-binding (syntax name))))
#'(begin
;; process the name: define special syntax for this name (only
;; for the first word of the name, the correctness of the rest
;; of the words is checked at runtime in the say procedure)
(define-syntax name
(lambda (y)
(with-ellipsis :::
(syntax-case y (more ...)
; just forward matching rules to Speak
((_ more ... symbol :::)
#'(Speak (((name more ...))) symbol :::))
((_ symbols :::)
; TODO: this does not correctly make the second name
; part of the name, preventing differentiation between
; name and modifier
#`(Speak-indirect (((name symbols :::)))))))))
;; process the rest of the names
(Enter b ...)
;; record that the name was introduced. I do not see a way to do
;; this directly in the compiler, therefore it is checked later
;; during runtime.
(introduce! '(name more ...))))
;; add debug output, must be added it here, not in front
; write
; quote : list Enter (name more ...) b ...
; newline
((_ (name more ...) b ...)
; existing binding: Just allow using this.
#'(begin
(Enter b ...)
(introduce! '(name more ...))))
((_ b ...)
#'(begin #t)))))

https://hg.sr.ht/~arnebab/dryads-wake/browse/enter.scm?rev=0d065c9ddcc4#L319


They are originally written in wisp¹ and enable me to write full
Scheme-code like this:

define : first-encounter
Enter : Juli Fin :profile juli
Melter Lark :profile melter
Rooted Breeze :profile dryad
Old One
;; …
Juli Fin
Finally we have our own home!

Melter Lark
It took long enough.

Juli Fin
And it is moist for sure.

Melter Lark
I will dry it out.

Rooted Breeze :eerie
My slumber breaks
my mind awakes
who are you strangers
in my home?

Old One
How do you answer?
Juli is ,(score->description (profile-ability-score (game-state) 'juli 'explain))
. at explaining
and ,(score->description (profile-ability-score (game-state) 'juli 'fast-talk))
. at fast-talk.
Choose
: explain your situation to appease the dryad
,(explain-your-home)
: fast-talk the dryad to get her to leave and risk her anger
,(fast-talk-the-dryad)

https://hg.sr.ht/~arnebab/dryads-wake/browse/dryads-wake.w?rev=0d065c9ddcc4#L332

> I'm asking because due to their
> non-triviality, they could become a kind of real-world test case on the
> expressiveness of the systems under debate and on how hard it would
> actually be to change code to accommodate one or the other option.

That would be pretty cool! Looking forward to it!

Best wishes,
Arne

¹: https://www.draketo.de/software/wisp

[Taylan Kammer]

On 05.08.2021 09:49, Marc Nieper-Wißkirchen wrote:

I can also offer a sophisticated real-world use of syntax-case, from my
Bytestructures library.

Bytestructures lets you define the layout of bytes within a bytevector,
with structs and vectors etc. like C types, so you can access bytes in
bytevectors in a structured way.

A bytestructure descriptor (the object type that actually defines the
layout of bytes) is a first-class object and normally the calculation
of the offset of a byte is done at run-time by querying the descriptor.

However, since such descriptors will usually be static in nature and
don't need to be constructed from data that only exists at run-time,
it makes sense to offer an API that hoists the offset calculations to
expand-time.

An optional/auxiliary part of the library does just that, by offering
a macro-defining macro, define-bytestructure-accessors. It evaluates
your bytestructure descriptors at expand-time (or you could import them
at expand-time) and defines a set of macros which can thus execute the
offset-calculation logic at expand-time.


Here's the file that defines define-bytestructure-accessors at the end:

https://github.com/TaylanUB/scheme-bytestructures/blob/master/bytestructures/body/base.syntactic.scm


It implicitly uses some definitions from this file:

https://github.com/TaylanUB/scheme-bytestructures/blob/master/bytestructures/body/base.scm

(The library magic happens in other files that include these sources.)


Here is a brief example use from the README:

(define bv (make-bytevector 1000))

(define-bytestructure-accessors
(bs:vector 5 (bs:vector 5 (bs:struct `((x ,uint8)
(y ,uint8)
(z ,uint8)))))
bs-unwrap bs-ref bs-set!)

(define (ref x) (bs-ref bv 4 4 z))

;; (bs-ref 4 4 z) should expand to (bytevector-u8-ref bv 74)
;; if I calculated that offset correctly just now


The strategy makes extensive use of syntax objects as first-class values.
Without them, the whole thing would have to be redesigned I think.


For those who don't want to visit GitHub, the contents of the linked
files are pasted below.

--
Taylan


base.syntactic.scm:

;;; Commentary:

;; This is an extension to the base of the module which allows using the API
;; purely in the macro-expand phase, which puts some limitations on its use but
;; reduces run-time overhead to zero or nearly zero.

(define-syntax-rule (syntax-case-lambda <pattern> <body>)
(lambda (stx)
(syntax-case stx ()
(<pattern> <body>))))

(define syntax-car (syntax-case-lambda (car . cdr) #'car))
(define syntax-cdr (syntax-case-lambda (car . cdr) #'cdr))
(define syntax-null? (syntax-case-lambda stx (null? (syntax->datum #'stx))))

(define (syntactic-unwrap bytevector offset descriptor indices)
(let loop ((bytevector bytevector)
(offset offset)
(descriptor descriptor)
(indices indices))
(if (not (syntax-null? indices))
(let ((unwrapper (bd-unwrapper descriptor)))
(when (not unwrapper)
(error "Cannot index through this descriptor." descriptor))
(let-values (((bytevector* offset* descriptor*)
(unwrapper #t bytevector offset (syntax-car indices))))
(loop bytevector* offset* descriptor* (syntax-cdr indices))))
(let ((getter (bd-getter descriptor))
(setter (bd-setter descriptor)))
(values bytevector offset descriptor getter setter)))))

(define (bytestructure-unwrap/syntax bytevector offset descriptor indices)
(let-values (((bytevector* offset* _descriptor _getter _setter)
(syntactic-unwrap bytevector offset descriptor indices)))
#`(values #,bytevector* #,offset*)))

(define (bytestructure-ref/syntax bytevector offset descriptor indices)
(let-values (((bytevector* offset* descriptor* getter _setter)
(syntactic-unwrap bytevector offset descriptor indices)))
(if getter
(getter #t bytevector* offset*)
(error "The indices given to bytestructure-ref/syntax do not lead to a
bytestructure descriptor that can decode values. You must have used the wrong
getter macro, forgot to provide some of the indices, or meant to use the
unwrapper instead of the getter. The given indices follow." indices))))

(define (bytestructure-set!/syntax bytevector offset descriptor indices value)
(let-values (((bytevector* offset* descriptor* _getter setter)
(syntactic-unwrap bytevector offset descriptor indices)))
(if setter
(setter #t bytevector* offset* value)
(error "The indices given to bytestructure-set!/syntax do not lead to a
bytestructure descriptor that can encode values. You must have used the wrong
setter macro, or forgot to provide some of the indices. The given indices
follow." indices))))

(define-syntax-rule (define-bytestructure-unwrapper <name> <descriptor>)
(define-syntax <name>
(let ((descriptor <descriptor>))
(syntax-case-lambda (_ <bytevector> <offset> . <indices>)
(bytestructure-unwrap/syntax
#'<bytevector> #'<offset> descriptor #'<indices>)))))

(define-syntax-rule (define-bytestructure-getter <name> <descriptor>)
(define-syntax <name>
(let ((descriptor <descriptor>))
(syntax-case-lambda (_ <bytevector> . <indices>)
(bytestructure-ref/syntax #'<bytevector> 0 descriptor #'<indices>)))))

(define-syntax-rule (define-bytestructure-setter <name> <descriptor>)
(define-syntax <name>
(let ((descriptor <descriptor>))
(syntax-case-lambda (_ <bytevector> <index> (... ...) <value>)
(bytestructure-set!/syntax
#'<bytevector> 0 descriptor #'(<index> (... ...)) #'<value>)))))

(define-syntax-rule (define-bytestructure-accessors <descriptor>
<unwrapper> <getter> <setter>)
(begin
(define-bytestructure-unwrapper <unwrapper> <descriptor>)
(define-bytestructure-getter <getter> <descriptor>)
(define-bytestructure-setter <setter> <descriptor>)))


base.scm:

;;; Commentary:

;; This is the base of the module, defining the data types and procedures that
;; make up the bytestructures framework.

;;; Code:

;;; Descriptors

(define-record-type <bytestructure-descriptor>
(%make-bytestructure-descriptor size alignment unwrapper getter setter meta)
bytestructure-descriptor?
(size bd-size)
(alignment bd-alignment)
(unwrapper bd-unwrapper)
(getter bd-getter)
(setter bd-setter)
(meta bd-meta))

(define make-bytestructure-descriptor
(case-lambda
((size alignment unwrapper getter setter)
(%make-bytestructure-descriptor
size alignment unwrapper getter setter #f))
((size alignment unwrapper getter setter meta)
(%make-bytestructure-descriptor
size alignment unwrapper getter setter meta))))

(define bytestructure-descriptor-size
(case-lambda
((descriptor) (bytestructure-descriptor-size descriptor #f #f))
((descriptor bytevector offset)
(let ((size (bd-size descriptor)))
(if (procedure? size)
(size #f bytevector offset)
size)))))

(define (bytestructure-descriptor-size/syntax bytevector offset descriptor)
(let ((size (bd-size descriptor)))
(if (procedure? size)
(size #t bytevector offset)
size)))

(define bytestructure-descriptor-alignment bd-alignment)
(define bytestructure-descriptor-unwrapper bd-unwrapper)
(define bytestructure-descriptor-getter bd-getter)
(define bytestructure-descriptor-setter bd-setter)
(define bytestructure-descriptor-metadata bd-meta)

;;; Bytestructures

(define-record-type <bytestructure>
(make-bytestructure bytevector offset descriptor)
bytestructure?
(bytevector bytestructure-bytevector)
(offset bytestructure-offset)
(descriptor bytestructure-descriptor))

(define bytestructure
(case-lambda ((descriptor) (%bytestructure descriptor #f #f))
((descriptor values) (%bytestructure descriptor #t values))))

(define (%bytestructure descriptor init? values)
(let ((bytevector (make-bytevector
(bytestructure-descriptor-size descriptor))))
(when init?
(bytestructure-primitive-set! bytevector 0 descriptor values))
(make-bytestructure bytevector 0 descriptor)))

(define (bytestructure-size bytestructure)
(bytestructure-descriptor-size (bytestructure-descriptor bytestructure)
(bytestructure-bytevector bytestructure)
(bytestructure-offset bytestructure)))

(define-syntax-rule (bytestructure-unwrap <bytestructure> <index> ...)
(let ((bytestructure <bytestructure>))
(let ((bytevector (bytestructure-bytevector bytestructure))
(offset (bytestructure-offset bytestructure))
(descriptor (bytestructure-descriptor bytestructure)))
(bytestructure-unwrap* bytevector offset descriptor <index> ...))))

(define-syntax bytestructure-unwrap*
(syntax-rules ()
((_ <bytevector> <offset> <descriptor>)
(values <bytevector> <offset> <descriptor>))
((_ <bytevector> <offset> <descriptor> <index> <indices> ...)
(let ((bytevector <bytevector>)
(offset <offset>)
(descriptor <descriptor>))
(let ((unwrapper (bd-unwrapper descriptor)))
(when (not unwrapper)
(error "Cannot index through this descriptor." descriptor))
(let-values (((bytevector* offset* descriptor*)
(unwrapper #f bytevector offset <index>)))
(bytestructure-unwrap*
bytevector* offset* descriptor* <indices> ...)))))))

(define-syntax-rule (bytestructure-ref <bytestructure> <index> ...)
(let-values (((bytevector offset descriptor)
(bytestructure-unwrap <bytestructure> <index> ...)))
(bytestructure-primitive-ref bytevector offset descriptor)))

(define-syntax-rule (bytestructure-ref*
<bytevector> <offset> <descriptor> <index> ...)
(let-values (((bytevector offset descriptor)
(bytestructure-unwrap*
<bytevector> <offset> <descriptor> <index> ...)))
(bytestructure-primitive-ref bytevector offset descriptor)))

(define (bytestructure-primitive-ref bytevector offset descriptor)
(let ((getter (bd-getter descriptor)))
(if getter
(getter #f bytevector offset)
(make-bytestructure bytevector offset descriptor))))

(define-syntax-rule (bytestructure-set! <bytestructure> <index> ... <value>)
(let-values (((bytevector offset descriptor)
(bytestructure-unwrap <bytestructure> <index> ...)))
(bytestructure-primitive-set! bytevector offset descriptor <value>)))

(define-syntax-rule (bytestructure-set!*
<bytevector> <offset> <descriptor> <index> ... <value>)
(let-values (((bytevector offset descriptor)
(bytestructure-unwrap*
<bytevector> <offset> <descriptor> <index> ...)))
(bytestructure-primitive-set! bytevector offset descriptor <value>)))

(define (bytestructure-primitive-set! bytevector offset descriptor value)
(let ((setter (bd-setter descriptor)))
(if setter
(setter #f bytevector offset value)
(if (bytevector? value)
(bytevector-copy! bytevector offset value 0
(bytestructure-descriptor-size
descriptor bytevector offset))
(error "Cannot write value with this bytestructure descriptor."
value descriptor)))))

(define (bytestructure-ref/dynamic bytestructure . indices)
(let-values (((bytevector offset descriptor)
(bytestructure-unwrap bytestructure)))
(let loop ((bytevector bytevector)
(offset offset)
(descriptor descriptor)
(indices indices))
(if (null? indices)
(bytestructure-primitive-ref bytevector offset descriptor)
(let-values (((bytevector* offset* descriptor*)
(bytestructure-unwrap*
bytevector offset descriptor (car indices))))
(loop bytevector*
offset*
descriptor*
(cdr indices)))))))

(define (bytestructure-set!/dynamic bytestructure . args)
(let-values (((bytevector offset descriptor)
(bytestructure-unwrap bytestructure)))
(let loop ((bytevector bytevector)
(offset offset)
(descriptor descriptor)
(args args))
(if (null? (cdr args))
(bytestructure-primitive-set! bytevector offset descriptor (car args))
(let-values (((bytevector* offset* descriptor*)
(bytestructure-unwrap*
bytevector offset descriptor (car args))))
(loop bytevector*
offset*
descriptor*
(cdr args)))))))

(define-syntax-case-stubs
bytestructure-unwrap/syntax
bytestructure-ref/syntax
bytestructure-set!/syntax
define-bytestructure-accessors)

(cond-expand
(guile (include-from-path "bytestructures/body/base.syntactic.scm"))
(syntax-case (include "base.syntactic.scm"))
(else))

[John Cowan]

On Thu, Aug 5, 2021 at 2:43 AM Marc Nieper-Wißkirchen <marc....@gmail.com> wrote:

 

So before voting on whether to include ER or not, we should settle which version and see that this version gets also the corner cases right.

 

R7RS votes are not plebiscites: there may be more than one candidate (other than the usual No Vote, None, and Other).  I have no trouble adding more than one version of ER to the ballot.

 

Would you mind postponing discussing the ballot and the actually ballotting by a few weeks.  At least in many places in the northern hemisphere, there are currently holidays so a part of our community wouldn't be able to discuss these important matters.

After running two ballots, I have observed that trying to take this into account cancels out.  Some people don't read non-essential email on vacation, others read that kind of email *only* on vacation because they are too busy otherwise.

As for what's on the ballot, independently of what I think would be the best choice, just asking for ER and syntax-case independently is not enough, I think. People may have a strong opinion to include only one system, no matter what. Or an opinion of, say, ER at all events and maybe syntax-case, but not vice versa.

That's the purpose of the "Other" choice, which allows you to say what you want.

A yet missing option is on the R4RS low-level macro system (see the appendix of R4RS and SRFI 211). It combines the soundness of the syntax-case system with the no-bells-and-whistles approach of ER (meaning no pattern matcher in the form of syntax-case).

I would put that on the ballot if it was widely implemented, but it is not.  That is also why syntactic closures and implicit renaming are not on the ballot.  Again, "Other" remains an option.

- Please postpone SRFI 147 to a later ballot.  It is for systems not necessarily offering procedural macros and is not compatible with syntax-case because syntax-case says that the <transformer> in a syntax definition is an ordinary expression.  (When available, the latter is much more powerful and definitely better.)

Moved to Morpheus.

- Before voting for the identifier-syntax form in R6RS (which is just library syntax in terms of syntax-case), it is important to vote for what is a macro use as per section 9.2 of the R6RS.  In R7RS, it is only the first form.  In R6RS, there are four forms.

Voting for identifier-syntax (excluding the `set!` case) implies that identifier-syntax macro names in operand position are macro uses.

- What is meant by "SRFI 17 or identifier-syntax"? How are they related? (See also my latest private email on this topic.)

You were right to say that SRFI-17  ycan't be defined on top of the two-argument form of identifier-syntax calls.  But they provide essentially the same function, generalized `set!`, and so can appear as alternative proposals.  (I note that Chez, Guile, Larceny, and Sagittarius support both.)

But I feel that a meta-question should be answered first.  Does R7RS-large strive for compatibility with R6RS and existing R6RS implementations or is this considered irrelevant?

To quote the WG2 charter: "When deciding which features to include in the language, working group 2 should consider all features provided by R6RS Scheme, and all criticisms of those features. Insofar as practical, the language should be backwards compatible with an appropriate subset of the R6RS standard."

 

R7RS-large would be the language to replace R6RS.

I can't find any reference to such a claim, certainly not in the official documents at scheme-reports.org.  It's possible that such a statement was made informally.  If R6.1RS goes anywhere, that would be the natural successor to R6RS.

[Marc Nieper-Wißkirchen]

Am Fr., 6. Aug. 2021 um 03:22 Uhr schrieb John Cowan <co...@ccil.org>:

On Thu, Aug 5, 2021 at 2:43 AM Marc Nieper-Wißkirchen <marc....@gmail.com> wrote:

 

So before voting on whether to include ER or not, we should settle which version and see that this version gets also the corner cases right.

 

R7RS votes are not plebiscites: there may be more than one candidate (other than the usual No Vote, None, and Other).  I have no trouble adding more than one version of ER to the ballot.

Good, but we need to know which versions and what their semantics are in detail.

 

Would you mind postponing discussing the ballot and the actually ballotting by a few weeks.  At least in many places in the northern hemisphere, there are currently holidays so a part of our community wouldn't be able to discuss these important matters.

After running two ballots, I have observed that trying to take this into account cancels out.  Some people don't read non-essential email on vacation, others read that kind of email *only* on vacation because they are too busy otherwise.

I'm wondering whether those who have only their vacation to participate should count as much as those who are active more or less the rest of the year.

 

As for what's on the ballot, independently of what I think would be the best choice, just asking for ER and syntax-case independently is not enough, I think. People may have a strong opinion to include only one system, no matter what. Or an opinion of, say, ER at all events and maybe syntax-case, but not vice versa.

That's the purpose of the "Other" choice, which allows you to say what you want.

That sounds good in theory, but does it work in practice? People voting are lazy. Moreover, there would be a very strong bias towards the explicit options.

 

A yet missing option is on the R4RS low-level macro system (see the appendix of R4RS and SRFI 211). It combines the soundness of the syntax-case system with the no-bells-and-whistles approach of ER (meaning no pattern matcher in the form of syntax-case).

I would put that on the ballot if it was widely implemented, but it is not.  That is also why syntactic closures and implicit renaming are not on the ballot.  Again, "Other" remains an option.

Implementing the R4RS low-level system is less than trivial in a syntax-case system. It consists mostly of repackaging some already existing syntax keywords and procedures in a library. So you can effectively consider it widely implemented.

Adding it to an existing ER system is doable when the ER system has a mechanism to get unhygienic macros right.

 

The R4RS low-level system would be an option for those who want the soundness of the syntax-case system but dislike that a pattern matcher is built-in.  While that's not my opinion, I know that this was one of the points of criticism in Alex's 2008 "pamphlet" against syntax-case.

- Please postpone SRFI 147 to a later ballot.  It is for systems not necessarily offering procedural macros and is not compatible with syntax-case because syntax-case says that the <transformer> in a syntax definition is an ordinary expression.  (When available, the latter is much more powerful and definitely better.)

Moved to Morpheus.

Thank you.

 

- Before voting for the identifier-syntax form in R6RS (which is just library syntax in terms of syntax-case), it is important to vote for what is a macro use as per section 9.2 of the R6RS.  In R7RS, it is only the first form.  In R6RS, there are four forms.

Voting for identifier-syntax (excluding the `set!` case) implies that identifier-syntax macro names in operand position are macro uses.

You probably mean that "macro names in operand position are macro uses".  The notion an "identifier-syntax macro name" is not very interesting in itself.

And what about (foo . bar) when foo is a macro keyword? Would it be a macro use? It is in R6RS but not in R7RS.

 

- What is meant by "SRFI 17 or identifier-syntax"? How are they related? (See also my latest private email on this topic.)

You were right to say that SRFI-17  ycan't be defined on top of the two-argument form of identifier-syntax calls.  But they provide essentially the same function, generalized `set!`, and so can appear as alternative proposals.  (I note that Chez, Guile, Larceny, and Sagittarius support both.)

It seems to be that they are more orthogonal to each other than alternatives.

This is one reason why I think that SRFI 17 should be moved to a different, later ballot. It does not fit into the scope of this ballot and the efficiency concerns associated with it should be addressed first. I think it should be moved to a docket where different types and ways of generic methods are discussed.

 

But I feel that a meta-question should be answered first.  Does R7RS-large strive for compatibility with R6RS and existing R6RS implementations or is this considered irrelevant?

To quote the WG2 charter: "When deciding which features to include in the language, working group 2 should consider all features provided by R6RS Scheme, and all criticisms of those features. Insofar as practical, the language should be backwards compatible with an appropriate subset of the R6RS standard."

Can you explain what this is supposed to mean in practice? "Appropriate subset" can mean a lot, for example, the intersection of R6RS with R5RS.

 

 R7RS-large would be the language to replace R6RS.

I can't find any reference to such a claim, certainly not in the official documents at scheme-reports.org.  It's possible that such a statement was made informally.  If R6.1RS goes anywhere, that would be the natural successor to R6RS.

It was you in the early years of R7RS who made such a statement, but obviously informally. As the size and scope of R7RS-large will be much bigger than the size and scope of any R6.1RS, there would not necessarily be a contradiction of both languages extending (or replacing) the 2007 version of R6RS. Much like that R7RS-small and R7RS-large both replace R5RS.

[Marc Nieper-Wißkirchen]

PS: Here's the context of the claim made: https://small.r7rs.org/ticket/486/.

 

 R7RS-large would be the language to replace R6RS.

I can't find any reference to such a claim, certainly not in the official documents at scheme-reports.org.  It's possible that such a statement was made informally.  If R6.1RS goes anywhere, that would be the natural successor to R6RS.

It was you in the early years of R7RS who made such a statement, but obviously informally. As the size and scope of

[...]

[Linas Vepstas]

I would like to ask a question, about macros; but ...

* I understand it probably does not belong on this particular email chain,

* The question may sound nutty, as it's not actually about scheme, per se.

I work with a system that stores s-expressions in a database. Like all good databases, these can be added and removed at any time, and It has a pattern-matching system, so that users can search for what they want. Very loosely, very roughly, it is syntax-case-like, but, of course, at run-time, because the collection of s-expressions is constantly changing.  Over the years, it's gotten quite fancy, being driven by user demands.

The question is: would it be possible to build such a system in pure scheme, using one of the proposed macro systems to perform the term-rewriting? That is, given a collection of s-expressions, find the ones that match, per pattern-matcher, apply the transformations, and plop the result back into the database?

(My system has a bunch of non-scheme oddities, e.g. all s-expressions are typed, and each one is globally unique: it is impossible to store two identical s-expressions (even alpha-convertible ones). One very very important feature is the ability to ask and answer the question: given any s-expression, find all of those that contain it. The current code-base is in C++, with guile and python wrappers.  I'm asking, because this has proven to be useful enough, that I think the general idea should be more wide-spread. I have heard of people implementing datalog (the data subset of prolog) in scheme, but they did not seem to have any pattern matcher, and it wasn't scalable/fast. BTW, some, but not all of my s-expressions are executable/evaluatable. Thus, after pattern matching, not only is there a term-rewriting step, but it is followed by an execute/evaluate step. Apparently, graphQL for javascript is a related concept. Note that graphQL has become very popular. However, in javascript, all structures are always named; whereas all of my structures are always anonymous (and you have several choices for naming, if you really need to name something.). Thus, my system is far far more like scheme, than it is like javascript.)

-- linas

--
You received this message because you are subscribed to the Google Groups "scheme-reports-wg2" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scheme-reports-...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/scheme-reports-wg2/CAEYrNrS%2B5Pghu%3DX2_Hbn1iTsM2kvarpFc4WHyj2vJpnCDGKR5A%40mail.gmail.com.

--

Patrick: Are they laughing at us?

Sponge Bob: No, Patrick, they are laughing next to us.

 

[Lassi Kortela]

Scheme has traditionally had (match ...) for pattern-matching at
runtime. If you want to pattern-match using a macro, and the patterns
vary at runtime, you'd have to use (eval ...).

This discussion shows that we probably have too many pattern matchers in
Scheme. Furthermore, in Common Lisp there's destructuring-bind and its
many variations for parsing different kinds of lambda lists. Racket and
Clojure have also added their own pattern matchers, and then there's
type-based pattern matching in the ML family.

It would be nice if we could decouple macro hygiene from pattern
matching somehow, turning the two into orthogonal concepts, but it may
be too late to do that in standard Scheme.

[Marc Nieper-Wißkirchen]

Am Fr., 6. Aug. 2021 um 13:16 Uhr schrieb Lassi Kortela <la...@lassi.io>:

On 06.08.2021 14.07, Linas Vepstas wrote:

> I work with a system that stores s-expressions in a database. Like all
> good databases, these can be added and removed at any time, and It has
> a pattern-matching system, so that users can search for what they
> want. Very loosely, very roughly, it is syntax-case-like, but, of
> course, at run-time, because the collection of s-expressions is
> constantly changing.  Over the years, it's gotten quite fancy, being
> driven by user demands.
>
> The question is: would it be possible to build such a system in pure
> scheme, using one of the proposed macro systems to perform the
> term-rewriting? That is, given a collection of s-expressions, find the
> ones that match, per pattern-matcher, apply the transformations, and
> plop the result back into the database?

The syntax-case pattern matcher, which is the same as that of syntax-rules, can, of course, be used at runtime without the use of `eval`.  It destructures syntax objects, which, likewise, also exist at runtime.  Using `datum->syntax`, one can always turn an s-expression into a syntax object.

For example,

(define foo

  (lambda (stx)

    (syntax-case stx ()

      [(a b ...) #'(a foo b ...)]

      [_

       (syntax-violation 'foo "pattern not recognized" stx)])))

takes a syntax object representing a list of at least one element and inserts the identifier `foo` after the first element.

If you want to apply `foo` to ordinary datums, you can write a wrapper

(define foo*

  (lambda (x)

    (syntax->datum (foo (datum->syntax #'* x)))))

Now one can ask why one should take a detour through syntax objects.  Well, the advantage of syntax objects is that they can contain source location information.  So for erroneous input that is presented as a syntax object, the syntax violation raised can point to the source code location of the erroneous input.  (For example, it would have made a lot of sense for SRFI's regexp to take a syntax object instead of an s-expression.)

Scheme has traditionally had (match ...) for pattern-matching at
runtime. If you want to pattern-match using a macro, and the patterns
vary at runtime, you'd have to use (eval ...)

There is no `match` in the R[567]RS.  In R[57]RS, a pattern matcher is part of syntax-rules.

The `match` forms known in the Scheme community also differ; see SRFI 200.

 .

It would be nice if we could decouple macro hygiene from pattern
matching somehow, turning the two into orthogonal concepts, but it may
be too late to do that in standard Scheme.

This is exactly what the syntax-case system achieves: It decouples the two parts of the syntax-rules:  Pattern matching and templating.

[Lassi Kortela]

> The syntax-case pattern matcher, which is the same as that of
> syntax-rules, can, of course, be used at runtime without the use of
> `eval`.  It destructures syntax objects, which, likewise, also exist at
> runtime.

This is true if changing input is fed to fixed patterns. But if the
patterns themselves change at runtime, one needs `eval`, right? It's
like a fixed regexp vs a regexp that changes at runtime.

> Using `datum->syntax`, one can always turn an s-expression
> into a syntax object.
>

> If you want to apply `foo` to ordinary datums, you can write a wrapper
>
> (define foo*
>   (lambda (x)
>     (syntax->datum (foo (datum->syntax #'* x)))))

Doing this every time one wants to de-structure something is a bit of an
ordeal; it would be nice if there was an alternative that doesn't have
to go through syntax objects. It doesn't like much of a technical
problem, mainly a matter of agreement.

> Now one can ask why one should take a detour through syntax objects.
> Well, the advantage of syntax objects is that they can contain source
> location information.  So for erroneous input that is presented as a
> syntax object, the syntax violation raised can point to the source code
> location of the erroneous input.  (For example, it would have made a lot
> of sense for SRFI's regexp to take a syntax object instead of an
> s-expression.)

Good point.

> This is exactly what the syntax-case system achieves: It decouples the
> two parts of the syntax-rules:  Pattern matching and templating.

I have a good impression of syntax-case, but now feel there's an extra
layer of genericity that would still be nice to extract for wider use.
The round trip through syntax objects would be nice to avoid in many cases.

Would it make sense to have a framework to define custom pattern
matchers, irrespective of whether syntax objects or unwrapped objects
are being matched against? The syntax-rules and syntax-case pattern
language could then be defined using this meta-language, as could the
various versions of `match`.

[Aaron Hsu]

You talked about de-coupling hygiene from pattern matching, and that's what the syntax-case system (not the syntax-case syntax) does. Syntax objects are the encapsulation of hygiene. Take your pick of any of the numerous pattern matchers out there. They can all be used with syntax objects if you want, or unhygienic objects.

You don't need to use hygiene if you just want to destructure something, but you need syntax objects if you want hygiene. The two topics are orthogonal already.

A framework for defining new pattern matchers seems rather less than ideal to me. It seems like it would be better to attempt to increase the level of convergence on pattern matching as a whole, rather than to encourage more schismatic implementations of different implementations of pattern matchers. Now, the ability to compose different pattern matching functionality into a single pattern matching system might be interesting..

--
Aaron W. Hsu | arc...@sacrideo.us | http://www.sacrideo.us

On Fri, Aug 6, 2021, at 6:56 AM, Lassi Kortela wrote:
> This is true if changing input is fed to fixed patterns. But if the
> patterns themselves change at runtime, one needs `eval`, right? It's
> like a fixed regexp vs a regexp that changes at runtime.

[...]

> Doing this every time one wants to de-structure something is a bit of an
> ordeal; it would be nice if there was an alternative that doesn't have
> to go through syntax objects. It doesn't like much of a technical
> problem, mainly a matter of agreement.

[...]

[Marc Nieper-Wißkirchen]

Am Fr., 6. Aug. 2021 um 13:56 Uhr schrieb Lassi Kortela <la...@lassi.io>:

> The syntax-case pattern matcher, which is the same as that of
> syntax-rules, can, of course, be used at runtime without the use of
> `eval`.  It destructures syntax objects, which, likewise, also exist at
> runtime.

This is true if changing input is fed to fixed patterns. But if the
patterns themselves change at runtime, one needs `eval`, right? It's
like a fixed regexp vs a regexp that changes at runtime.

If the patterns of the matcher are only known at runtime, you need `eval`, yes.  But this is true for `syntax-case` as for the various implementations of `match`.  But such a device should probably better be called a unifier so that confusion with the application areas of `syntax-case` and the various `match` forms does not arise.

 

> Using `datum->syntax`, one can always turn an s-expression
> into a syntax object.
>
> If you want to apply `foo` to ordinary datums, you can write a wrapper
>
> (define foo*
>    (lambda (x)
>      (syntax->datum (foo (datum->syntax #'* x)))))

Doing this every time one wants to de-structure something is a bit of an
ordeal; it would be nice if there was an alternative that doesn't have
to go through syntax objects. It doesn't like much of a technical
problem, mainly a matter of agreement.

If you know that your input will never naturally be a syntax object, `syntax-case` will probably be the wrong tool.  But if, on the other hand, it can rightfully be a syntax object (as in the case of regexp patterns, for example, or other DSLs), `syntax-case` is your tool.

That said, the operations that underlie syntax-case/syntax conceptually are the procedure `unwrap-syntax` and the syntax `syntax` (ignoring pattern variables) of the appendix to R4RS.  One could add a construct like Guile's (and SRFI 211's) `with-ellipsis` but changes the meaning of `unwrap-syntax` and `syntax` in its lexical scope.  This way, syntax-case/syntax can be parameterized to different types of input.

Alternatively, add SRFI 213 and implement syntax-case/syntax portably in terms of the R4RS primitives and generalize this implementation to other use cases.

 

> Now one can ask why one should take a detour through syntax objects. 
> Well, the advantage of syntax objects is that they can contain source
> location information.  So for erroneous input that is presented as a
> syntax object, the syntax violation raised can point to the source code
> location of the erroneous input.  (For example, it would have made a lot
> of sense for SRFI's regexp to take a syntax object instead of an
> s-expression.)

Good point.
> This is exactly what the syntax-case system achieves: It decouples the
> two parts of the syntax-rules:  Pattern matching and templating.

I have a good impression of syntax-case, but now feel there's an extra
layer of genericity that would still be nice to extract for wider use.
The round trip through syntax objects would be nice to avoid in many cases.

You can always delay stripping the syntactic information until the very last step.  This definitely helps to debug because you will know where the syntax object (fragments) in the intermediate steps have come from.

 

Would it make sense to have a framework to define custom pattern
matchers, irrespective of whether syntax objects or unwrapped objects
are being matched against? The syntax-rules and syntax-case pattern
language could then be defined using this meta-language, as could the
various versions of `match`.

I'd say it makes a lot of sense to have different pattern matchers/pattern matcher generators for different kinds of data.  There is `syntax-case` for syntax objects, the various `match` forms for unstructured s-expressions, and there are the matchers created for languages in the Nanopass framework.  Given a fixed inductive data type, one shouldn't use `match` or `syntax-case` to destructure it but use a matcher generated especially generated for this data type, improving type safety.  The Nanopass framework shows how it works.

[Alaric Snell-Pym]

On 06/08/2021 12:36, Marc Nieper-Wißkirchen wrote:

> Now one can ask why one should take a detour through syntax objects. Well,
> the advantage of syntax objects is that they can contain source location
> information. So for erroneous input that is presented as a syntax object,
> the syntax violation raised can point to the source code location of the
> erroneous input. (For example, it would have made a lot of sense for
> SRFI's regexp to take a syntax object instead of an s-expression.)

As an aside, this notion of "source code location" and the fact it's not
there for s-expressions made using map, cons, etc has always faintly
bothered me; on my "to think about more deeply when retired" list is a
standardised notion of paths through s-expressions that is useful for
source location reporting, so rather than something like "column 50 of
line 123 of foo.scm" one can say "caddadadadadadaddadadadar of the sexpr
read from foo.scm", and have emacs understand that for me for source
code, and my own code easily able to handle it for other things :-)

--
Alaric Snell-Pym (M0KTN neé M7KIT)
http://www.snell-pym.org.uk/alaric/

[Marc Nieper-Wißkirchen]

Am Fr., 6. Aug. 2021 um 13:36 Uhr schrieb Marc Nieper-Wißkirchen <marc....@gmail.com>:

[...]

Now one can ask why one should take a detour through syntax objects.  Well, the advantage of syntax objects is that they can contain source location information.  So for erroneous input that is presented as a syntax object, the syntax violation raised can point to the source code location of the erroneous input.  (For example, it would have made a lot of sense for SRFI's regexp to take a syntax object instead of an s-expression.)

Read: "SRFI 115's regexp".

[...]

[Dr. Arne Babenhauserheide]

Marc Nieper-Wißkirchen <marc....@gmail.com> writes:
> I'm wondering whether those who have only their vacation to participate
> should count as much as those who are active more or less the rest of the
> year.

I think that this question leads nowhere, because you could ask it the
other way round: Whether those who contribute only outside of vacation
should count as much as those who are only active during holiday-weeks.

I can find arguments for both positions, but looking at the set of
arguments together, none are really so compelling that they would
overshadow all the others.

So I think we should avoid pitting work-week contributions against
holiday-week contributions. A contribution is a contribution.

Best wishes,
Arne

[Marc Nieper-Wißkirchen]

Am Fr., 6. Aug. 2021 um 16:45 Uhr schrieb Dr. Arne Babenhauserheide <arne...@web.de>:

Marc Nieper-Wißkirchen <marc....@gmail.com> writes:
> I'm wondering whether those who have only their vacation to participate
> should count as much as those who are active more or less the rest of the
> year.

I think that this question leads nowhere, because you could ask it the
other way round: Whether those who contribute only outside of vacation
should count as much as those who are only active during holiday-weeks.

Because most people have at most 1 month of vacation and 11 months outside vacation. :)

[John Cowan]

On Fri, Aug 6, 2021 at 8:57 AM Marc Nieper-Wißkirchen <marc....@gmail.com> wrote:

Am Fr., 6. Aug. 2021 um 13:36 Uhr schrieb Marc Nieper-Wißkirchen <marc....@gmail.com>:

Now one can ask why one should take a detour through syntax objects.  Well, the advantage of syntax objects is that they can contain source location information.

They can, but it's not guaranteed by R6RS.

Read: "SRFI 115's regexp".

Note also Serex, which is a SRFI-115-style set of combinators for expressing regexes that apply to S-expressions rather than strings.  See <http://inamidst.com/lisp/serex>.  No implementation yet, alas.

[Dr. Arne Babenhauserheide]

Lassi Kortela <la...@lassi.io> writes:

>> The syntax-case pattern matcher, which is the same as that of
>> syntax-rules, can, of course, be used at runtime without the use of
>> `eval`. It destructures syntax objects, which, likewise, also exist
>> at runtime.
>
> This is true if changing input is fed to fixed patterns. But if the
> patterns themselves change at runtime, one needs `eval`, right? It's
> like a fixed regexp vs a regexp that changes at runtime.

You can still create new macros during runtime, because you can have
a syntax-case macro that uses "inner" syntax-case to create a new
syntax-case macro from its input.

I’m doing that in dryads-wake/enter to introduce new speakers.

(dryad (welcome)) ;; error
(Enter (dryad))
(dryad (welcome)) ;; writes "welcome"

But that’s where madness lies :-)

And I’m not sure whether it is really applicable to your usecase.

[Marc Nieper-Wißkirchen]

Am Fr., 6. Aug. 2021 um 17:09 Uhr schrieb Dr. Arne Babenhauserheide <arne...@web.de>:

Lassi Kortela <la...@lassi.io> writes:

>> The syntax-case pattern matcher, which is the same as that of
>> syntax-rules, can, of course, be used at runtime without the use of
>> `eval`.  It destructures syntax objects, which, likewise, also exist
>> at runtime.
>
> This is true if changing input is fed to fixed patterns. But if the
> patterns themselves change at runtime, one needs `eval`, right? It's
> like a fixed regexp vs a regexp that changes at runtime.

You can still create new macros during runtime, because you can have
a syntax-case macro that uses "inner" syntax-case to create a new
syntax-case macro from its input.

If you mean the creation of transformers and the possibility of invoking them, that's true (because transformers can be mere procedures). But if you want to evaluate the code represented by the syntax object your transformer creates at runtime, you have to use `eval`.

 

I’m doing that in dryads-wake/enter to introduce new speakers.

(dryad (welcome)) ;; error
(Enter (dryad))
(dryad (welcome)) ;; writes "welcome"

As far as I understand your code, the macro use of `Enter` defines a new macro keyword named `dryad`.  But this happens at expansion and not at runtime.

(If you use the REPL, this doesn't count because it implicitly invokes `eval`.)

Or am I misunderstanding something?

[Marc Nieper-Wißkirchen]

Am Fr., 6. Aug. 2021 um 17:09 Uhr schrieb John Cowan <co...@ccil.org>:

On Fri, Aug 6, 2021 at 8:57 AM Marc Nieper-Wißkirchen <marc....@gmail.com> wrote:

Am Fr., 6. Aug. 2021 um 13:36 Uhr schrieb Marc Nieper-Wißkirchen <marc....@gmail.com>:

Now one can ask why one should take a detour through syntax objects.  Well, the advantage of syntax objects is that they can contain source location information.

They can, but it's not guaranteed by R6RS.

And R6RS does not guarantee that its flonum objects cover at least the numeric range of the IEEE singles.

;)

[Per Bothner]

On 8/6/21 5:50 AM, Alaric Snell-Pym wrote:
> As an aside, this notion of "source code location" and the fact it's not
> there for s-expressions made using map, cons, etc has always faintly
> bothered me; on my "to think about more deeply when retired" list is a
> standardised notion of paths through s-expressions that is useful for
> source location reporting,

In Kawa, a syntax object implements the SyntaxForm interface.
A Pair that is read by a reader is a PairWithPosition, which extends plain Pair.
A PairWithPositionSyntaxForm object is both a PairWithPosition and a SyntaxForm.
Calling cdr on a PairWithPositionSyntaxForm will typically return null or another
PairWithPositionSyntaxForm (created lazily).

So you can map, cons, car, cdr etc on syntax object lists.
--
--Per Bothner
p...@bothner.com http://per.bothner.com/

[John Cowan]

On Fri, Aug 6, 2021 at 1:54 AM Marc Nieper-Wißkirchen <marc....@gmail.com> wrote:

Good, but we need to know which versions and what their semantics are in detail.

I take it that you and others want SRFI 211, some will want Chicken (no syntactic closures), and some will want MIT/Chibi (with syntactic closures.)  I don't know if the last two are the same.  The more items, the more choice; the fewer items, the easier it is to get a resolution.

I'm wondering whether those who have only their vacation to participate should count as much as those who are active more or less the rest of the year.

Quite apart from me not being able to weigh votes rather than counting them, I find that the people under the most work pressure, at least pre-pandemic, are the professors.  And their weights, if we *were* to weigh votes, should certainly weigh more than, say, mine.  Anyone who wishes to cast a fractional vote or divide their vote is free to do so (again, use "Other").

 

That's the purpose of the "Other" choice, which allows you to say what you want.

That sounds good in theory, but does it work in practice? People voting are lazy.

Then they don't care that much about the result, most likely.

Implementing the R4RS low-level system is less than trivial in a syntax-case system. It consists mostly of repackaging some already existing syntax keywords and procedures in a library. So you can effectively consider it widely implemented.

Fair.

Adding it to an existing ER system is doable when the ER system has a mechanism to get unhygienic macros right.

Not so fair, since very few ER macro systems do so.

How hard is it to add to a system with only define-macro, like S9 from Empty Space, or only define-macro and syntax-rules, like Bigloo?

Voting for identifier-syntax (excluding the `set!` case) implies that identifier-syntax macro names in operand position are macro uses.

You probably mean that "macro names in operand position are macro uses".  The notion an "identifier-syntax macro name" is not very interesting in itself.

Identifier-syntax macros are necessarily visible in operand position, and are understood as visible in operator position too.  This is an upward compatible extension, since a macro name in operand position in R[57]RS is an error, as it is not an expression.

And what about (foo . bar) when foo is a macro keyword? Would it be a macro use? It is in R6RS but not in R7RS.

What use is there for such a thing?  (foo . bar) is not a function call either.

- What is meant by "SRFI 17 or identifier-syntax"? How are they related? (See also my latest private email on this topic.)

You were right to say that SRFI-17  ycan't be defined on top of the two-argument form of identifier-syntax calls.  But they provide essentially the same function, generalized `set!`, and so can appear as alternative proposals.  (I note that Chez, Guile, Larceny, and Sagittarius support both.)

It seems to be that they are more orthogonal to each other than alternatives.

Can you explain further?  What I meant by "essentially the same function" is that you can add (set! (leg dog) 'new-leg) to Scheme either by defining `leg` and `set-leg` as a SRFI 17 getter-setter pair of functions, or by defining an identifier-syntax macro `leg` that expands to calls on functions `leg-ref` and `set-leg`. 

I have for the moment moved this to Morpheus too, leaving only single-clause identifier-syntax like (define-syntax leg (identifier-syntax (leg (leg-ref dog)))).  (By the way, this is equivalent to CL's (define-symbol-macro leg `(leg-ref dog)), which of course is not hygienic.)

To quote the WG2 charter: "When deciding which features to include in the language, working group 2 should consider all features provided by R6RS Scheme, and all criticisms of those features. Insofar as practical, the language should be backwards compatible with an appropriate subset of the R6RS standard."

Can you explain what this is supposed to mean in practice?

 

Like everything in the charter, it means what I say it means, unless someone wants to appeal from my decision to the WG, in which case my decision is affirmed or overridden by majority vote.  If someone wants to appeal further, the Steering Committee has the final word.  (Of course, the SC can also fire me, in which case all bets are off.)

As the size and scope of R7RS-large will be much bigger than the size and scope of any R6.1RS,

I'd think that would be up to the R6.1RS WG.  There is no reason why some of the R7RS libraries, portable and otherwise, should not be considered for inclusion.  I don't know this for sure, but I think there might have been more if not for the artificial 24-month limit set by the SC for the first draft (which R7RS has carefully ignored).

I now see what I said, thanks to your link.  I was young and foolish....

[Linas Vepstas]

Thank you for the discussion. I read and pondered.  I realize now that my original email was misleading (sent early morning before waking up), based on a naive misapperception.  What I was really asking about was a full-scale query language for s-expressions. As Marc noted: this necessarily requires unification, which is outside of the bounds of simple syntax matching. As John noted, "serex" (http://inamidst.com/lisp/serex) is a necessary but not sufficient subset of a query language.

I'm now torn as to how to proceed. I can drop the subject, as it's perhaps too futuristic and too complex to merit debate. Or I can .. say more things, and see what happens. Let me say more things.

What do I mean by "query language"? It's something that will ground multiple variables, and "unify" them (this is an abuse of the word "unification", but will perhaps be OK for informal use).  Easiest to explain by example:

Suppose my dataset contains

(list 'a 'b 'c)

(list 'd 'e 'a)

(list 'd 'e (list 'a 'b))

(list (list 'a 'b) 'b 'c)

The above is a *set* -- an unordered collection of four expressions.

A query takes the form (this is pseudo-code, not real code):

(query

   (and

      (list $var 'b 'c)

      (list 'd 'e $var))

  (list 'p 'q $var 's 't))

I've used the $ as a syntactic device to say "this is a named variable". Hopefully that's clear. Hopefully, the intent of the query is clear: find groundings for the variable. The result of running the query should be clear, too, I hope: it is the unordered set

(list 'p 'q 'a 's 't)

(list 'p 'q  (list 'a 'b) 's 't)

The first expression is the result of binding (grounding) $var to 'a  and the second is the result of grounding $var to (list 'a 'b).

The above is more than just a regex -- it is a unification of regexes. That is, (list $var 'b 'c) can be understood to be a regex, and so can (list 'd 'e $var) and the (and ...) is  the requirement that both regexes must be fulfilled, with the same grounding for $var in both.

Typed variables are required to limit the search space (for performance) and in some opaque cases, to prevent infinite recursion (in case the dataset itself contains query statements) Typing is "real easy":

(query

   (and

      (is-literal? $var)

      (list $var 'b 'c)

      (list 'd 'e $var))

  (list 'p 'q $var 's 't))

which results in (list 'p 'q 'a 's 't) only, because of the two possible groundings for $var, (list 'a 'b) is not a literal.  To avoid a mis-reading, mis-interpretation of the above, it is convenient to write it in the equivalent form:

(query

   (and

      (is-literal? $var)

      (is-present? (list $var 'b 'c))

      (is-present? (list 'd 'e $var)))

  (list 'p 'q $var 's 't))

where is-present? evaluates to #t when the expression is present in the dataset.  To emphasize again: the variable $var must have the same grounding in all three statements: thus, this is a form of unification.

So far, so good.  The fun and games start with assorted "obvious" extensions:

-- multiple variables.

-- is-absent? - this cannot be evaluated until the needed variables have groundings.

-- always? - this cannot be evaluated until *all* groundings have been found. (for example, find all baskets which contain only red balls)

-- globs - In the above, $var matched one and only one thing.  Globs can match zero, one or more things in a list. They can be greedy, or not. so, just like regexes.

-- searches over unordered sets. - this requires exploring all possible permutations of a set. This is particularly challenging when the sets are nested, and there are multiple variables.

-- Allowing boolean expressions in the pattern.  In the above examples, the (and ...) is "the pattern". Some people want to nest "or" and "not" into there. This gets dicey when considering complex constructions like (or (not (is-present? ..)) (not (always? ...))) and this kind of stuff: (or (list 'a 'b $var1) (list $var2 'b 'c)) which leaves some variables ungrounded, but is perfectly valid in specific cases. (e.g. the robot arm picked up $var1 or $var2 is not on the table)

-- Allowing disconnected graphs. A quickie example:

(query

   (and

       (present? (list  $var-job1 $var-salary1))

       (present? (list  $var-job2 $var-salary2))

       ( > $var-salary1 $var-salary2))

   (string-concatenate (list "Occupation" $var-job1 "pays more than" $var-job2)))

applied to the dataset

(list "janitor" 14)

(list "welder" 50)

(list "executive" 500)

the point here being that the variables have to be grounded, before the greater-than can be evaluated.  The algorithmic problem is that, implemented incorrectly, this results in a combinatorial explosion: it is a cartesian product of disjoint graph components.

That this is a graph-query language  is hopefully obvious: A naive way of representing a graph is as a set of edges, each having two endpoints. This, the dataset (the unordered set)

(list 'vertex-1 'vertex-43)

(list 'vertex-43 'vertex-2)

(list 'vertex-43 'vertex-3)

is obviously a "graph." But since the query language is over s-expressions, and not two-element lists, the query language is actually a query over hypergraphs (aka metagraphs).

BTW, the true power of the system shows up when you use it with srfi-213 -- identifiers. (My system isn't scheme, and doesn't have srfi-213, but it has something similar/equivalent) This allows every s-expression in the dataset to have  a key-value database riding on it. Accessing these values can be handy, during search.  Particularly nifty are when the values are references to GPU nodes doing neural-net tensor-type stuff, or audio processing, or video processing.  The s-expressions then encode the structure of the processing pipeline, but the actual processing is done on the GPUs.

(Skimming srfi-213, I find it odd that json is never mentioned: a key-value system is effectively the same thing as JSON. Naming things with keys is exactly the difference between javascript and scheme. The transformers appear to be a sneaky way of laying groundwork for javascript-inside-of-scheme. But perhaps I skimmed too quickly.)

The hard part of implementing the above is making it fast.  Typical datasets have millions of s-expressions in them (e.g. genomics, proteomics datasets)   and it would be absurd to try to match something like (present? (list 'upregulates 'BCRA $var)) against all of the millions: you can trivially narrow the search by not even looking at anything that doesn't have 'BCRA in it. 

BTW: Please notice that the syntax I used above is ... almost childishly simple. Part of the power of the system is that you don't have to beat your brains to understand how to use it.

I guess that's about it, except I want to comment on Arne Babenhauserheide chatbots. It turns out that conventional chatbots, such as AIML and Chatscript are reverse-query engines. Best demonstrated by example:

Suppose my dataset consists of the (unordered) collection of queries:

(query (list "I" "like" $var)  (list " "I" "like" $var "too"))

(query (list "I" "like" $var)  (list " "I" "just" "love" $var))

(query (list $glob $verb "baseball")  (list "Do" "you" $verb "basketball?"))

Chatbots have tens of thousands of scripted forms like the above.  Chatbots proceed by performing the reverse query:

(reverse-query (List "I" "like" "baseball"))

and then returning all suitable matches. If there's more than one, assorted ad hoc strategies narrow them down to just one, which is then the chat-bots reply.  As it happens, the reverse-query can use exactly the same query engine as the earlier examples; there's only a minor alteration as to which direction the grounding is happening.

Oh .... actually, that is not quite it. Recently, after reviewing things like srfi-158 generators and srfi-155 promises and srfi-41 streams I've realized that I would like search capabilities compatible with these kinds of things.  My thoughts are too disorganized to say more, other than that in practice, some queries run for hours and generate hundreds of matches (e.g. find all genes A,B that express proteins C,D that occur on a common reactome pathway E) and users want to see the search results trickle out instead of waiting for a big batch of them at the end.  This in itself is not hard, unless you plan to use the outputs for additional querying. Something stream or generator-ish seems needed. It's confusing.

(There are many other details I'm glossing: large-datasets need to be marked read-only, but allow read-write deltas on top of them. This allows just one copy of large genomics datasets to be accessed by multiple users tinkering with it. Another point: what I described above was an in-RAM system. It's handy to be able to run most of it to disk. This is not hard, because there are whizzy systems like RocksDB that handle disk storage. Just that -- well, some minor advance planning is needed.)

Sorry for the long email. I'm just ... contemplating future paths.

-- Linas

--
You received this message because you are subscribed to the Google Groups "scheme-reports-wg2" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scheme-reports-...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/scheme-reports-wg2/CAEYrNrTmnnu27A18Kq9gqnPTX50dGVGPKRwWSYgCiWEyN%3DRQ%3DA%40mail.gmail.com.

[John Cowan]

On Fri, Aug 6, 2021 at 8:10 PM Linas Vepstas <linasv...@gmail.com> wrote:

 

So far, so good.

Up to here, you basically have Prolog.  There are multiple open-source Prologs out there, many with C interfaces (and/or Java).

The fun and games start with assorted "obvious" extensions:

-- multiple variables.

Prolog does that.

-- searches over unordered sets. - this requires exploring all possible permutations of a set. This is particularly challenging when the sets are nested, and there are multiple variables.

Prolog databases are typically indexed on the first element of a relation.

-- Allowing boolean expressions in the pattern.  In the above examples, the (and ...) is "the pattern". Some people want to nest "or" and "not" into there. This gets dicey when considering complex constructions like (or (not (is-present? ..)) (not (always? ...))) and this kind of stuff: (or (list 'a 'b $var1) (list $var2 'b 'c)) which leaves some variables ungrounded, but is perfectly valid in specific cases. (e.g. the robot arm picked up $var1 or $var2 is not on the table)

Prolog.

-- Allowing disconnected graphs. A quickie example:

Prolog.

(Skimming srfi-213, I find it odd that json is never mentioned: a key-value system is effectively the same thing as JSON.

There's a standard mapping between JSON and S-expressions, laid out in SRFI 180:

JSON <=> S-expression

null <=> the symbol named "null"

numbers, strings, booleans <=> the same

arrays <=> vectors

objects <=> alists whose keys are symbols

Example:

[32, "foo", true, null, [1, 2], {"key1": "value1", "key2": 2.5}] =>

#(32, "foo", #t, null, #(1 2),

   ((key1 . "value2) (key2 . 2.5)))

This in itself is not hard, unless you plan to use the outputs for additional querying. Something stream or generator-ish seems needed. It's confusing.

Generators are the most lightweight, as long as you don't worry about backtracking; they are (external) iterators.  Prolog C interfaces typically have the equivalent.

[Linas Vepstas]

Oh well. Perhaps I explained it badly. Prolog does exactly zero of what I describe. Mostly because prolog is not a database; its a programming language. It does not have any kind of query system. Yes, you can try to play with datalog, which is the knowledge-representation subset of prolog. But only if you want to work with something stunningly slow, non-scalable, immature, incomplete and nearly unusable.

Prolog itself is at best a slow and olde-fashioned tree walker.  If you have the urge to experiment with prolog, don't -- use ASP (Answer-Set Programming) instead. It uses almost exactly the same syntax as prolog. It's far more powerful. The U Potsdam solver is recommended. It is literally 3 ***orders of magnitude***  faster, and, depending on your problem, six or ten or more orders of magnitude faster than prolog. All thanks to the magic of SAT solver technology.  They may use almost the same syntax for "representing knowledge", but are otherwise incomparable.  People underestimated SAT in the 1980's. It's criminal to do that now.

Anyway, prolog has trouble with expressivity. It's got the fatal flaw that it only supports two possible valuations: true, and false. Nothing else, whatsoever. The whole point of things like srfi-213, and (cough cough) ideas like "dictionaries" is that they allow you to wedge arbitrary valuations onto expressions, instead of having to settle for true/false as the only allowed values.

I notice that you used the word "pipeline" in describing the latest srfi-221 and srfi-158 implying that you think of generators as composable things that process data. The generators are bits of plumbing, the data that they act on is the "water" flowing through that plumbing.  This idea of plumbing and things that flow shows up in multiple domains. Back in the 2000's, you had the concept of the "graphics pipeline" - line drawing, shading, coloring, rasterization - and the GPU, onto which you could download your pipeline, and the GPU processes pixels at gigapixel-per-second rates.

These days, if you look at google's tensorflow, you will see exactly the same idea, but this time it's for neural nets. You have a collection of composable parts, you assemble them, and press go - and bingo, it compiles down to GPU code that does deep-learning neural-net stuff.

I'm trying to say: look at that pipeline as a generic graph. it need not be specifically a tensorflow graph, or specifically a graphics pipeline, or specifically a srfi-158 collection of generators. Its a generic graph. The values that flow through it are not prologian true-false values, they can be any kind of values. I mean -- really -- srfi-158 generators are not limited to true/false. They'll generate any kind of data. Why, in god's name, do I want to go back to the bad old world where true and false are the only two values allowed to flow?

And, if there's something that we've learned about graphs, it's that they need a good query language to go with them.  Prolog ain't that. If prolog was enough, then things like SparQL and GraphQL would not exist.  There's a reason why new stuff gets invented.

Look, no skin off of my back - I have a working system. There's a dozen different users of it; it's been deployed in some large corporations. You've probably seen it in action on late-night TV. I'm trying to communicate an idea: here's something neat, something spiffy, something cool and useful that people want and like and use, and scheme should have something like that. If for no other reason than that the scheme folks i.e. you all, can probably do a better job of it, at least conceptually, theoretically, than what I see coming out of places like the Apache Foundation.  You folks seem to think more clearly than they do.

--linas

--

You received this message because you are subscribed to the Google Groups "scheme-reports-wg2" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scheme-reports-...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/scheme-reports-wg2/CAD2gp_ScVtms0C2UEyDG2U40tAbj5QRWWMFyPk_F-0t6hSUZ3A%40mail.gmail.com.

[Amirouche Boubekki]

Hello Linas et al.


As far as I know, except Linas, I am the most knowledgeable about the
intersection of Scheme, database engineering, opencog, and the
atomspace. I want to add "sadly" to the previous sentence because I
would be glad to be challenged (on matters that do not relate to
commercial or marketing success or even social credibility).

I understand this thread is about the AtomSpace, and the rework of the
AtomSpace. Here is quote from the opencog wiki about the AtomSpace:

> The OpenCog AtomSpace is a knowledge representation (KR) database and the associated query/reasoning engine to fetch and manipulate that data, and perform reasoning on it. Data is represented in the form of graphs, and more generally, as hypergraphs; thus the AtomSpace is a kind of graph database, the query engine is a general graph re-writing system, and the rule-engine is a generalized rule-driven inferencing system. The vertices and edges of a graph, known as Atoms, are used to represent not only "data", but also "procedures"; thus, many graphs are executable programs as well as data structures. These Atoms, which are permanent and immutable, can be assigned fleeting, changing Values to indicate the truth or likelihood of that atom, or to hold other kinds of transient data. The AtomSpace enables flow-based programming, where Atoms represent the pipes, and Values are what flows through the pipes.
>
> https://wiki.opencog.org/w/AtomSpace

Le ven. 6 août 2021 à 13:16, Lassi Kortela <la...@lassi.io> a écrit :
>
> On 06.08.2021 14.07, Linas Vepstas wrote:
>
> > I work with a system that stores s-expressions in a database.

That is a mistake. The data stored in the atomspace database can have
a primary representation that can be serialized to an s-expr. That is
different from "everything is an s-expr".

> > Like all
> > good databases, these can be added and removed at any time, and It has
> > a pattern-matching system, so that users can search for what they
> > want. Very loosely, very roughly, it is syntax-case-like, but, of

It is still unclear to me whether the current AtomSpace supports
recursive possibly cyclic pattern matching. What is clear is that
syntax-case does not support that.

> > course, at run-time, because the collection of s-expressions is
> > constantly changing. Over the years, it's gotten quite fancy, being
> > driven by user demands.

What the user demands is unclear to me.

> >
> > The question is: would it be possible to build such a system in pure
> > scheme, using one of the proposed macro systems to perform the
> > term-rewriting? That is, given a collection of s-expressions, find the
> > ones that match, per pattern-matcher, apply the transformations, and
> > plop the result back into the database?

I assume plop means "record" or "store" or "save" in the database.
Query, Transformation and Plop (QTP) really sounds like
Extract-Transform-Load (ETL). As far as I know, recording every QTP in
the atomspace, in the schema of the current atomspace is a mistake.

> >
> > (My system has a bunch of non-scheme oddities, e.g. all s-expressions
> > are typed, and each one is globally unique: it is impossible to store
> > two identical s-expressions (even alpha-convertible ones).

It is unclear to me how alpha conversion is applied to the data or the
code. Or even the data of the code stored in the AtomSpace.

> > One very
> > very important feature is the ability to ask and answer the question:
> > given any s-expression, find all of those that contain it.

That really depends on the query logic. The s-expression
representation of domain data and the representation in the database
can be much different. The s-expr representation of the query (aka. a
DSL) is the very last question that should be asked / answered.

> > The current
> > code-base is in C++, with guile and python wrappers.

I already asked to support my work to rewrite the AtomSpace with Chez
with a specific set of problems that the database should solve. C++ is
difficult. Guile and Python are slow. Make it work, make it
maintainable, make it fast.

> > I'm asking,
> > because this has proven to be useful enough, that I think the general
> > idea should be more wide-spread.

After much work around opencog and the atomspace I still do not
understand what is that "general idea" outside the Artificial General
Intelligence system.

> > I have heard of people implementing
> > datalog (the data subset of prolog) in scheme, but they did not seem
> > to have any pattern matcher,

datalog is a pattern matcher. Somewhat unrelated but possibly
interesting https://research.google/pubs/pub48190/

> > and it wasn't scalable/fast.

See the above google paper link.

> > BTW, some,
> > but not all of my s-expressions are executable/evaluatable. Thus,
> > after pattern matching, not only is there a term-rewriting step, but
> > it is followed by an execute/evaluate step.

Again bundling extraction, and transformation (and possibly loading
the result) is a mistake, it should be opt-in at user demand.

> > Apparently, graphQL for
> > javascript is a related concept. Note that graphQL has become very
> > popular.

Forget about GraphQL. GraphQL is not an algorithm, it is merely a
surface syntax for a neighbor query. Like I wrote before, the query
DSL is the last problem that should be solved. Getting together
example queries can help narrow / explain what problem must be solved,
but once you settle on a primary and secondary representation in the
database, a more powerful / fine-tuned DSL can emerge.

> > However, in javascript, all structures are always named;

GraphQL is usable and used in many programming languages. It is a
stringified query language like SQL. It is possible I am missing your
point.

> > whereas all of my structures are always anonymous (and you have
> > several choices for naming, if you really need to name something.).
> > Thus, my system is far far more like scheme, than it is like javascript.)

Yes, data in the atomspace can be anonymous, but you still can query
for "what is under the table" even if the results were added without
human supervision, and retrieve the uid of the object(s) under the
table. Hence, you could use graphql for that particular query.

>
> Scheme has traditionally had (match ...) for pattern-matching at
> runtime. If you want to pattern-match using a macro, and the patterns
> vary at runtime, you'd have to use (eval ...).
>

You can implement `match` logic with pattern combinators. So far, I
did not manage to make it work with cyclic data, and as far as I
understand from a performance point of view, the general case is a
cursed problem.

On the subject of choosing a database:

- What guarantees do you need. Question all the properties of ACID:
Atomic, Consistent, Isolation, Durability. Look into BASE. You do not
necessarily need ACID or BASE, but it is a good basis that is well
documented to know where you want / need to go.

- What is the size of the data?

- What is the shape of the data? Are they well defined types? Are they
polymorphic types or heterogeneous shapes?

- Workload: Write-once then Read-only, mostly reads, mostly writes, a
mix of both. Answer also the question how fast or slow data can be
written or read.

- Querying: What queries look like: recursive / deep, columns or rows,
or neighborhood queries (like graphql and SQL without recursive
queries). Again what is the expected time to respond.

ref: https://stackoverflow.com/a/68608928/140837

--
Amirouche ~ https://hyper.dev

[Marc Nieper-Wißkirchen]

Am Sa., 7. Aug. 2021 um 01:04 Uhr schrieb John Cowan <co...@ccil.org>:

On Fri, Aug 6, 2021 at 1:54 AM Marc Nieper-Wißkirchen <marc....@gmail.com> wrote:

Good, but we need to know which versions and what their semantics are in detail.

I take it that you and others want SRFI 211, some will want Chicken (no syntactic closures), and some will want MIT/Chibi (with syntactic closures.)  I don't know if the last two are the same.  The more items, the more choice; the fewer items, the easier it is to get a resolution.

The latter is certainly true. But what would it mean if the majority voted for "ER macro transformer" without any further qualification?

To make some hopefully helpful proposal, what about the following questions (that are mostly not alternatives) together with some notes that are hopefully not biased:

1. Do you want to have a basic version of er-macro-transformer that allows one to write procedural hygienic macros in R7RS-large? (Note: This will include basically all versions of er-macro-transformer found in existing implementations.)

2. If an er-macro-transformer system is included in R7RS-large, do you want that it supports writing unhygienic macros? (Note: This will rule out a number of existing implementations.)

3. If an er-macro-transformer system is included in R7RS-large that supports writing unhygienic macros, do you want raw symbols in the macro output to be closed in the use environment instead of causing undefined behavior?

4. If an er-macro-transformer system is included in R7RS-large, do you want that the sameness of identifiers can be tested with `eqv?`? (Note: This basically rules out interoperability with syntax-case.)

5. Do you want the low-level macro system of R4RS to be included in R7RS-large? (Note: This is basically the syntax-case system without syntax-case and pattern variables.)

6. Do you want the full syntax-case system as described in R6RS to be included in R7RS-large? (Note: This question does not touch what constitutes a macro use.)

7. Do you want to have a version of with-ellipsis (see SRFI 211) included in R7RS-large so that R7RS's syntax-rules can be easily expressed in syntax-case?

8. Do you want that a single keyword (not in operator position) can constitute a macro use? (Note: Informally, this is known as "identifier syntax".)

9. Do you want that a keyword as the first argument to `set!` can constitute a macro use? (Note: Informally, this is known as "variable transformers".)

10. Do you want to have no limits on what constitutes a macro use as in section 9.2 of the R6RS? (Note: This implies a positive answer to questions 8. and 9.)

11. Do you want `identifier-syntax` as a convenient form to implement macros that can be used outside operator position? (Note: A positive answer here implies at least a positive answer to question 8.)

12. Do you want R7RS-large to support both er-macro-transformer and syntax-case? (Note: There are currently few implementations that support this.  A positive answer here rules out question 4.)

13. Do you want R7RS-large to make only one of the two systems mandatory? (Note: A positive answer here obviously rules out a positive answer to question 12, and vice versa.)

14. Do you want R7RS-large to support unhygienic macros? (Note: A positive answer here rules out a negative answer to question 2. unless questions 5. and 6. are answered positively.)

15. If the syntax-case system is included in R7RS-large, do you want to impose that all syntax objects are always fully unwrapped? (Note: While this makes syntax-case closer to er-macro-transformer, this will rule out most existing implementations and some otherwise pleasant properties of syntax-case are lost.)

I know that this is a long list of questions but I don't think for the worse as these questions have to be considered and answered by every voter while making their decision anyway.

 

I'm wondering whether those who have only their vacation to participate should count as much as those who are active more or less the rest of the year.

Quite apart from me not being able to weigh votes rather than counting them, I find that the people under the most work pressure, at least pre-pandemic, are the professors.  And their weights, if we *were* to weigh votes, should certainly weigh more than, say, mine.  Anyone who wishes to cast a fractional vote or divide their vote is free to do so (again, use "Other").

You certainly didn't understand it this way but others who just read this en passant might, so please let me get it straight: I never would and never mentioned that the votes themselves should be weighted differently. It was just a question of consideration for those constantly actively participating in the process. Anyway, it was just a suggestion, of course, and I am fine with any decision, of course.

 

 

That's the purpose of the "Other" choice, which allows you to say what you want.

That sounds good in theory, but does it work in practice? People voting are lazy.

Then they don't care that much about the result, most likely.

I would have probably given the same answer. :)

But if even people don't care that much about the result, it doesn't mean that would have been wrong or that their input wouldn't have helped.

 

Implementing the R4RS low-level system is less than trivial in a syntax-case system. It consists mostly of repackaging some already existing syntax keywords and procedures in a library. So you can effectively consider it widely implemented.

Fair.

Adding it to an existing ER system is doable when the ER system has a mechanism to get unhygienic macros right.

Not so fair, since very few ER macro systems do so.

The question was whether it is, in principle, widely implemented. If you count syntax-case in this regard, you can count R4RS as well.

 

How hard is it to add to a system with only define-macro, like S9 from Empty Space, or only define-macro and syntax-rules, like Bigloo?

I don't know Bigloo enough to write anything substantial here.  If a Scheme system does not offer any kind of hygiene (not even syntax-rules), it is probably the easiest to replace the expander layer altogether with a sufficiently portable one (like Unsyntax).

 

Voting for identifier-syntax (excluding the `set!` case) implies that identifier-syntax macro names in operand position are macro uses.

You probably mean that "macro names in operand position are macro uses".  The notion an "identifier-syntax macro name" is not very interesting in itself.

Identifier-syntax macros are necessarily visible in operand position, and are understood as visible in operator position too.  This is an upward compatible extension, since a macro name in operand position in R[57]RS is an error, as it is not an expression.

"Identifier-syntax" is a keyword in R6RS that allows you to create such macros, but in general such a macro will have to be written with syntax-case or er-macro-transformer (for the same reason that syntax-rules, in general, do not suffice).  Daphne already mentioned in some other thread a possible source of confusion: "Identifier-syntax" is a tool to create certain kinds of macros, while the actual question should first be what kind of macro uses we want.

 

And what about (foo . bar) when foo is a macro keyword? Would it be a macro use? It is in R6RS but not in R7RS.

What use is there for such a thing?  (foo . bar) is not a function call either.

The latter is irrelevant.  (let ((x 2)) (+ x 1)) is neither a function call.

Actually, not allowing (foo . bar) but the rest would be an unnatural restriction.

 

- What is meant by "SRFI 17 or identifier-syntax"? How are they related? (See also my latest private email on this topic.)

You were right to say that SRFI-17  ycan't be defined on top of the two-argument form of identifier-syntax calls.  But they provide essentially the same function, generalized `set!`, and so can appear as alternative proposals.  (I note that Chez, Guile, Larceny, and Sagittarius support both.)

It seems to be that they are more orthogonal to each other than alternatives.

Can you explain further?  What I meant by "essentially the same function" is that you can add (set! (leg dog) 'new-leg) to Scheme either by defining `leg` and `set-leg` as a SRFI 17 getter-setter pair of functions, or by defining an identifier-syntax macro `leg` that expands to calls on functions `leg-ref` and `set-leg`.

In R6RS, `(set! (leg dog) 'new-leg)` is a syntax violation and never a macro use.  The first argument to `set!` has to be an identifier.  In this sense, SRFI 17 and R6RS macro uses don't touch each other. In any case, SRFI 17 should be discussed in the context of generics (after we know what macro facilities there will be).

 

 I have for the moment moved this to Morpheus too, leaving only single-clause identifier-syntax like (define-syntax leg (identifier-syntax (leg (leg-ref dog)))).  (By the way, this is equivalent to CL's (define-symbol-macro leg `(leg-ref dog)), which of course is not hygienic.)

As `(set! <id> <expr>)` doesn't touch SRFI 17, it should not be moved but discussed with the rest of the macro system, IMO.

 

To quote the WG2 charter: "When deciding which features to include in the language, working group 2 should consider all features provided by R6RS Scheme, and all criticisms of those features. Insofar as practical, the language should be backwards compatible with an appropriate subset of the R6RS standard."

Can you explain what this is supposed to mean in practice?

 

Like everything in the charter, it means what I say it means, unless someone wants to appeal from my decision to the WG, in which case my decision is affirmed or overridden by majority vote.  If someone wants to appeal further, the Steering Committee has the final word.  (Of course, the SC can also fire me, in which case all bets are off.)

As the size and scope of R7RS-large will be much bigger than the size and scope of any R6.1RS,

I'd think that would be up to the R6.1RS WG.  There is no reason why some of the R7RS libraries, portable and otherwise, should not be considered for inclusion.  I don't know this for sure, but I think there might have been more if not for the artificial 24-month limit set by the SC for the first draft (which R7RS has carefully ignored).

Can you remind us of what 24-month limit you are talking about?

 

I now see what I said, thanks to your link.  I was young and foolish....

Oh, please just don't start searching the web for what I said yesterday!

-- Marc

[Amirouche Boubekki]

Le sam. 7 août 2021 à 02:10, Linas Vepstas <linasv...@gmail.com> a écrit :
>
> Thank you for the discussion. I read and pondered. I realize now that my original email was misleading (sent early morning before waking up), based on a naive misapperception. What I was really asking about was a full-scale query language for s-expressions. As Marc noted: this necessarily requires unification, which is outside of the bounds of simple syntax matching. As John noted, "serex" (http://inamidst.com/lisp/serex) is a necessary but not sufficient subset of a query language.

On that subject those links can be interesting:

- https://swtch.com/~rsc/regexp/
- https://www.gnu.org/software/guile/manual/html_node/SXPath.html

>
> I'm now torn as to how to proceed. I can drop the subject, as it's perhaps too futuristic and too complex to merit debate. Or I can .. say more things, and see what happens. Let me say more things.
>
> What do I mean by "query language"? It's something that will ground multiple variables, and "unify" them (this is an abuse of the word "unification", but will perhaps be OK for informal use). Easiest to explain by example:
>
> Suppose my dataset contains
>
> (list 'a 'b 'c)
> (list 'd 'e 'a)
> (list 'd 'e (list 'a 'b))
> (list (list 'a 'b) 'b 'c)
>
> The above is a *set* -- an unordered collection of four expressions.

If they are unordered they are not lists. So, the above s-expr is a
partial / approximate view of the intended representation in the
database.

>
> A query takes the form (this is pseudo-code, not real code):
>
> (query
> (and
> (list $var 'b 'c)
> (list 'd 'e $var))
> (list 'p 'q $var 's 't))
>
> I've used the $ as a syntactic device to say "this is a named variable". Hopefully that's clear. Hopefully, the intent of the query is clear: find groundings for the variable. The result of running the query should be clear, too, I hope: it is the unordered set

Unordered vs. ordered can matter. It really depends on the end-user
intent and the intended future queries. Secondary representation in
the database can make unordered query faster than ordered indeed so it
can matter.

>
> (list 'p 'q 'a 's 't)
> (list 'p 'q (list 'a 'b) 's 't)
>
> The first expression is the result of binding (grounding) $var to 'a and the second is the result of grounding $var to (list 'a 'b).
>
> The above is more than just a regex -- it is a unification of regexes. That is, (list $var 'b 'c) can be understood to be a regex, and so can (list 'd 'e $var) and the (and ...) is the requirement that both regexes must be fulfilled, with the same grounding for $var in both.
>
> Typed variables are required to limit the search space (for performance) and in some opaque cases, to prevent infinite recursion (in case the dataset itself contains query statements) Typing is "real easy":
>

Starting with performance tuning is not the way to go, one needs to
understand what is represented, how it is queried (unlike what will be
the representation of the query).

[...]

> (There are many other details I'm glossing: large-datasets need to be marked read-only,

That is getting started to be a more precise specification.

> but allow read-write deltas on top of them.

I solved that problem with the vnstore

> This allows just one copy of large genomics datasets to be accessed by multiple users tinkering with it.

Each user has its own "blackboard" ?

> Another point: what I described above was an in-RAM system. It's handy to be able to run most of it to disk. This is not hard, because there are whizzy systems like RocksDB that handle disk storage. Just that -- well, some minor advance planning is needed.)

Given what *I understand and know* about opencog, you will shoot
yourself in the foot with rocksdb.

> To view this discussion on the web visit https://groups.google.com/d/msgid/scheme-reports-wg2/CAHrUA37Fp2hsoprzyOyO5HgtsX4dzaVGWpiCPtf3z9xLi2BBzQ%40mail.gmail.com.

[Amirouche Boubekki]

Le sam. 7 août 2021 à 04:34, Linas Vepstas <linasv...@gmail.com> a écrit :
>
> Oh well. Perhaps I explained it badly. Prolog does exactly zero of what I describe. Mostly because prolog is not a database; its a programming language. It does not have any kind of query system. Yes, you can try to play with datalog, which is the knowledge-representation subset of prolog. But only if you want to work with something stunningly slow, non-scalable, immature, incomplete and nearly unusable.

It is better to ask logicmoo people about that.

>
> Prolog itself is at best a slow and olde-fashioned tree walker. If you have the urge to experiment with prolog, don't -- use ASP (Answer-Set Programming) instead. It uses almost exactly the same syntax as prolog. It's far more powerful. The U Potsdam solver is recommended. It is literally 3 ***orders of magnitude*** faster, and, depending on your problem, six or ten or more orders of magnitude faster than prolog. All thanks to the magic of SAT solver technology. They may use almost the same syntax for "representing knowledge", but are otherwise incomparable. People underestimated SAT in the 1980's. It's criminal to do that now.
>
> Anyway, prolog has trouble with expressivity. It's got the fatal flaw that it only supports two possible valuations: true, and false. Nothing else, whatsoever. The whole point of things like srfi-213, and (cough cough) ideas like "dictionaries" is that they allow you to wedge arbitrary valuations onto expressions, instead of having to settle for true/false as the only allowed values.

I am far from a prolog expert, but some demos can emit structured
data, things like feature structures.

[...]

> Look, no skin off of my back - I have a working system. There's a dozen different users of it; it's been deployed in some large corporations. You've probably seen it in action on late-night TV. I'm trying to communicate an idea: here's something neat, something spiffy, something cool and useful that people want and like and use, and scheme should have something like that. If for no other reason than that the scheme folks i.e. you all, can probably do a better job of it, at least conceptually, theoretically, than what I see coming out of places like the Apache Foundation. You folks seem to think more clearly than they do.

Then you are in the best position ever to provide a list of schemas
and queries. From where I stand, fitting a graph or pattern matching
to the problem(s), is not the way to go, at least not anymore.

[John Cowan]

On Sat, Aug 7, 2021 at 4:11 AM Marc Nieper-Wißkirchen <marc....@gmail.com> wrote:

The latter is certainly true. But what would it mean if the majority voted for "ER macro transformer" without any further qualification?

That will not be a ballot option.  "Other" is still a possibility.

To make some hopefully helpful proposal, what about the following questions (that are mostly not alternatives) together with some notes that are hopefully not biased:

I'll respond to this in another post.

The question was whether it is, in principle, widely implemented. If you count syntax-case in this regard, you can count R4RS as well.

Distinguo.   The question is whether it is *actually* widely implemented.  (This applies to non-portable proposals that are not explicitly written as a spec for a bridging implementation, like SRFI 170.  Portable features are not held to the same standard, since they do not need to be implemented, only packaged.)

Granted, if we have either syntax-rules or SRFI 211 ER in the standard, it will be portable, so I'm putting it in Morpheus, with the understanding that if either of those pass it will be moved to a portable docket (as will some other things).

"Identifier-syntax" is a keyword in R6RS that allows you to create such macros, but in general such a macro will have to be written with syntax-case or er-macro-transformer (for the same reason that syntax-rules, in general, do not suffice).

Portably, certainly.  But after all, `syntax-rules` is implemented by systems that don't provide either low-level module system.  You have more experience with this than I, but I suppose the same is true of identifier-syntax.

In R6RS, `(set! (leg dog) 'new-leg)` is a syntax violation and never a macro use.

Concedo; I misunderstood two-clause identifier-syntax.  (R6RS can be irritatingly dense.)

As `(set! <id> <expr>)` doesn't touch SRFI 17, it should not be moved but discussed with the rest of the macro system, IMO.

Done.

Can you remind us of what 24-month limit you are talking about?

The R6RS charter, <http://www.r6rs.org/charter/charter-mar-2006.txt>, whose first version was dated in 2004-01, says: "The Editors should produce a draft standard core Scheme, a draft module system, and a draft set of initial libraries within 24 months of the Editors' establishment."   (I'm actually quoting the final draft of 2006-02, as the first draft is not on r6rs.org.)  The first draft of R6RS was published in 2006-09, 30 months later.  In the WG's emails, there are also many references to "we can't do that, we don't have enough time"

The R7RS WG1 charter sets the limit at 18 months for the first draft and 24 months for the final draft: it took approximately three years to the final draft.  The WG2 charter said 24 months for the first draft, and we have been working on it since 2013 when WG1 wrapped up.

[Marc Nieper-Wißkirchen]

Am Sa., 7. Aug. 2021 um 17:31 Uhr schrieb John Cowan <co...@ccil.org>:

The question was whether it is, in principle, widely implemented. If you count syntax-case in this regard, you can count R4RS as well.

Distinguo.   The question is whether it is *actually* widely implemented.  (This applies to non-portable proposals that are not explicitly written as a spec for a bridging implementation, like SRFI 170.  Portable features are not held to the same standard, since they do not need to be implemented, only packaged.)

Granted, if we have either syntax-rules or SRFI 211 ER in the standard, it will be portable, so I'm putting it in Morpheus, with the understanding that if either of those pass it will be moved to a portable docket (as will some other things).

You mean "syntax-case", not "syntax-rules", don't you?

Voting for R4RS low-level would only make sense if syntax-case (or some equally expressive system) hasn't made it into the standard, I think.

 

"Identifier-syntax" is a keyword in R6RS that allows you to create such macros, but in general such a macro will have to be written with syntax-case or er-macro-transformer (for the same reason that syntax-rules, in general, do not suffice).

Portably, certainly.  But after all, `syntax-rules` is implemented by systems that don't provide either low-level module system.  You have more experience with this than I, but I suppose the same is true of identifier-syntax.

Indeed, it could be that systems implement the "identifier-syntax" macro, but otherwise only syntax-rules.  I don't know of any, though. (But this doesn't necessarily mean anything.)

[...]

As `(set! <id> <expr>)` doesn't touch SRFI 17, it should not be moved but discussed with the rest of the macro system, IMO.

Done.

Thanks.

 

Can you remind us of what 24-month limit you are talking about?

The R6RS charter, <http://www.r6rs.org/charter/charter-mar-2006.txt>, whose first version was dated in 2004-01, says: "The Editors should produce a draft standard core Scheme, a draft module system, and a draft set of initial libraries within 24 months of the Editors' establishment."   (I'm actually quoting the final draft of 2006-02, as the first draft is not on r6rs.org.)  The first draft of R6RS was published in 2006-09, 30 months later.  In the WG's emails, there are also many references to "we can't do that, we don't have enough time"

The R7RS WG1 charter sets the limit at 18 months for the first draft and 24 months for the final draft: it took approximately three years to the final draft.  The WG2 charter said 24 months for the first draft, and we have been working on it since 2013 when WG1 wrapped up.

Thank you!

Marc

[Dr. Arne Babenhauserheide]

I won’t give the arguments for the other side, because then we lead the
discussion into nowhere.

But an aside: I have 30 *days* which means roughly 6 weeks of vacation.
If you have less, you should enter a union to fight for more.

[Dr. Arne Babenhauserheide]

Marc Nieper-Wißkirchen <marc....@gmail.com> writes:

>> I’m doing that in dryads-wake/enter to introduce new speakers.
>>
>> (dryad (welcome)) ;; error
>> (Enter (dryad))
>> (dryad (welcome)) ;; writes "welcome"
>>
>
> As far as I understand your code, the macro use of `Enter` defines a new
> macro keyword named `dryad`. But this happens at expansion and not at
> runtime.
>
> (If you use the REPL, this doesn't count because it implicitly invokes
> `eval`.)
> Or am I misunderstanding something?

I’m using the REPL, so you’re right in that.

[Taylan Kammer]

On 07.08.2021 18:25, Dr. Arne Babenhauserheide wrote:
>
> But an aside: I have 30 *days* which means roughly 6 weeks of vacation.
> If you have less, you should enter a union to fight for more.
>

I think we Germans are somewhat privileged in this regard. Apparently
it's not unusual to have only 10 paid vacation days per year in the US.

--
Taylan

[John Cowan]

On Fri, Aug 6, 2021 at 10:34 PM Linas Vepstas <linasv...@gmail.com> wrote:

 

Anyway, prolog has trouble with expressivity. It's got the fatal flaw that it only supports two possible valuations: true, and false. Nothing else, whatsoever. The whole point of things like srfi-213, and (cough cough) ideas like "dictionaries" is that they allow you to wedge arbitrary valuations onto expressions, instead of having to settle for true/false as the only allowed values.

Prolog returns a stream of bindings, which may be empty.  If there are no bound variables to return, the stream is true if it returns one result and false if it returns no results.  This corresponds in the relational algebra to a relation with zero attributes, which can have one tuple and be {{}}, or no tuples and be {}, informally called DEE and DUM respectively.  (Annoyingly, SQL doesn't let you have zero attributes.)

I notice that you used the word "pipeline" in describing the latest srfi-221 and srfi-158 implying that you think of generators as composable things that process data. The generators are bits of plumbing, the data that they act on is the "water" flowing through that plumbing.

Generators aren't composable as such, but there are generator operations, which are generators that take a generator as an argument.  You can create a pipeline "generator | operator | operator ... | accumulator."

Note that Datalog with stratified negation is isomorphic to the relational algebra.

And, if there's something that we've learned about graphs, it's that they need a good query language to go with them.  Prolog ain't that. If prolog was enough, then things like SparQL and GraphQL would not exist.  There's a reason why new stuff gets invented.

"If there wasn't something wrong with Lisp, no computer language (except Fortran) would exist."

[Lassi Kortela]

> "If there wasn't something wrong with Lisp, no computer language (except
> Fortran) would exist."

http://www.paulgraham.com/fix.html (This should be updated to cover more
and newer languages. Anyone want to work on it, email me.)

[Per Bothner]

On 8/7/21 10:51 AM, Taylan Kammer wrote:
> I think we Germans are somewhat privileged in this regard. Apparently
> it's not unusual to have only 10 paid vacation days per year in the US.

You have it backwards: It is unusual to have more than 10 paid vacation days per year in the US,
and it is not unusual to have zero paid vacation days per year.

[Linas Vepstas]

Hi Amirouche!

On Sat, Aug 7, 2021 at 2:52 AM Amirouche Boubekki <amirouche...@gmail.com> wrote:

Hello Linas et al.


As far as I know, except Linas, I am the most knowledgeable about the
intersection of Scheme, database engineering, opencog, and the
atomspace. I want to add "sadly" to the previous sentence because I
would be glad to be challenged (on matters that do not relate to
commercial or marketing success or even social  credibility).

I understand this thread is about the AtomSpace, and the rework of the
AtomSpace. Here is quote from the opencog wiki about the AtomSpace:

I was trying not to say "AtomSpace" because advertising seemed counter-productive for this mailing list.

> The OpenCog AtomSpace is a knowledge representation (KR) database and the associated query/reasoning engine to fetch and manipulate that data, and perform reasoning on it. Data is represented in the form of graphs, and more generally, as hypergraphs; thus the AtomSpace is a kind of graph database, the query engine is a general graph re-writing system, and the rule-engine is a generalized rule-driven inferencing system. The vertices and edges of a graph, known as Atoms, are used to represent not only "data", but also "procedures"; thus, many graphs are executable programs as well as data structures. These Atoms, which are permanent and immutable, can be assigned fleeting, changing Values to indicate the truth or likelihood of that atom, or to hold other kinds of transient data. The AtomSpace enables flow-based programming, where Atoms represent the pipes, and Values are what flows through the pipes.
>
> https://wiki.opencog.org/w/AtomSpace

Le ven. 6 août 2021 à 13:16, Lassi Kortela <la...@lassi.io> a écrit :
>
> On 06.08.2021 14.07, Linas Vepstas wrote:
>
> > I work with a system that stores s-expressions in a database.

That is a mistake. The data stored in the atomspace database can have
a primary representation that can be serialized to an s-expr. That is
different from "everything is an s-expr".

For the purposes of encouraging  the scheme community, I was glossing, in order to convey the salient properties.  In this case, the salient property is "a database of s-expressions".  The pertinent claim is roughly, that "once you have that, you can layer any other representational forms on top", including dictionaries, graphs, lambda expressions, first-order logic, a type system, and much more.

The AtomSpace has a number of historical quirks that have to be preserved for backwards compatibility. I don't see a need to get into those, other than to say that we've never really standardized on any representation other than s-expressions. Rearranging parenthesis and whitespace to get python doesn't count. Some people imagine that representing it as json is a viable idea. They're either young or inexperienced or naive.  So -- s-expressions it is.

> > Like all
> > good databases, these can be added and removed at any time, and It has
> > a pattern-matching system, so that users can search for what they
> > want. Very loosely, very roughly, it is syntax-case-like, but, of

It is still unclear to me whether the current AtomSpace supports
recursive possibly cyclic pattern matching.

It's not officially defined. People occasionally create infinite loops by accident. It does have an explicit tail-recursive construct, but it is not used during pattern matching.

 

What is clear is that
syntax-case does not support that.

There are also many other differences, and if I was fully awake when I wrote that first email, I would never have written it.  One of the bigger differences is that all patterns in the atomspace are anonymous. There's no define-anything. Well, there is, but more or less no one uses define. It's fairly pointless and unneeded (unless you are a human writing human-readable code. However, the atomspace (mostly) does not cater to humans, it's designed to be easy-to-use by machines / algorithms.  So it's assembly-language-like.)

> > course, at run-time, because the collection of s-expressions is
> > constantly changing.  Over the years, it's gotten quite fancy, being
> > driven by user demands.

What the user demands is unclear to me.

It's 15+ years in the making. New batches of users show up annually. They always have issues and demands and requirements. The system mutates in response. I'm the bad guy, because sometimes I put my foot down and say "no". This really pisses people off. I'm not loved, or much liked.  Insisting on clean, maintainable code is not a good way to make friends and influence people.

I can detail the historical evolution, but that's not relevant for the current discussion.

> >
> > The question is: would it be possible to build such a system in pure
> > scheme, using one of the proposed macro systems to perform the
> > term-rewriting? That is, given a collection of s-expressions, find the
> > ones that match, per pattern-matcher, apply the transformations, and
> > plop the result back into the database?

I assume plop means "record" or "store" or "save" in the database.
Query, Transformation and Plop (QTP) really sounds like
Extract-Transform-Load (ETL). As far as I know, recording every QTP in
the atomspace, in the schema of the current atomspace is a mistake.

Uhh, if you mean "does the atomspace maintain a blockchain of every transaction?" then you know the answer is "no". I've vaguely contemplated doing this, but never saw an efficient way to do so. If you're running thousands of transactions a second, it's not really practical to record them, even if they are relatively tiny.

But apparently, this is a sexy thing to do. No doubt, there are users ready and willing to use and abuse this idea. Here's one: LemonGraph - Log-based transactional graph engine.  Yes. That NSA.

When I said "plop", I was using colloquial language. I mean simply that, if the pattern engine searches for all occurrences of pattern P and then rewrites into Q, then Q is placed in the AtomSpace as well. On the one hand, this seems like a historical design quirk. In practice, there has never been a situation where one would *not* want to do this.

The pattern matcher is a graph rewrite system.  It implements the P->Q ("P implies Q") primitive of graph rewriting. Pretty much just that and only that. External systems build on this primitive to add more. There are no Kripke frames. This is probably a design flaw, since the URE people struggled mightily to add Kripke frames in a super ad-hoc fashion.  I don't like what they did. It would be complicated to figure this out correctly.

 

> >
> > (My system has a bunch of non-scheme oddities, e.g. all s-expressions
> > are typed, and each one is globally unique: it is impossible to store
> > two identical s-expressions (even alpha-convertible ones).

It is unclear to me how alpha conversion is applied to the data or the
code. Or even the data of the code stored in the AtomSpace.

I don't understand what you mean by "data" and "code".

The idea is very simple. You can store the expression (Lambda (Variable x) (stuff with x)) in the atomspace, and then later try to store (Lambda (Variable y) (stuff with y)) -- these two will be determined to be equivalent, and only the first "copy" will be retained.

In the atomspace, Lambdas are not executable or evaluateable; they do not define "code". They merely bind (hide) local variable names. Now, the body (stuff with x) might be executable, in which case applying the lambda to something will result in beta reduction and execution, but executability of the body of the lambda is not a requirement.

That is, atomese lambdas are declarative: they're for "representing knowledge", and if the knowledge that you're representing just happens to have some kind of meaningful semantics for execution, well, that's what you must have wanted. But first and foremost, the atomspace is a database, and not a functional programming language.

> > One very
> > very important feature is the ability to ask and answer the question:
> > given any s-expression, find all of those that contain it.

That  really depends on the query logic.

Yes.

The s-expression
representation of domain data and the representation in the database
can be much different. The s-expr representation of the query (aka. a
DSL) is the very last question that should be asked / answered.

I don't understand what you're saying. Performance is King. If you can't make it go fast, then nothing else matters. I was shocked, shocked, when it was discovered that  Neo4J ran at the rate of a few hundred transactions per second. I was being crucified for being an incompetent moron for delivering a pathetic 20K transactions/second. None of the crucifiers ever apologized. No one ever came back and said "hey, you were right, I'm sorry". People really don't like the way I express myself. I'm sharply aware of this.  It's OK, I get the impression that most programmers endure this kind of criticism.

> > The current
> > code-base is in C++, with guile and python wrappers.

I already asked to support my work to rewrite the AtomSpace with Chez
with a specific set of problems that the database should solve. C++ is
difficult. Guile and Python are slow. Make it  work, make it
maintainable, make it fast.

OK, well, this is where the rubber hits the road. Here's where we stand:

*) the current atomspace, let's call it atomspace1, has to stay what it is to preserve backwards compat.

*) Ben is talking about atomspace2 aka hyperon as some kind of "improvement" but he is being very cagey about exactly what that is. When I skim the assorted drafts, it looks like stuff that could be easily implemented on top of the existing system. So, for me, there's a "wtf are you doing?" factor. I'm not convinced he's working with the sharpest knives.

*) Let's call your Chez variant atomspace3 (or invent a new name) - I'm generally supportive of such an effort. The emails that I have written have been an attempt to communicate the salient aspects, the important parts.  I cannot be active in coding it, as I'm in the middle of another (very important) project (that is strongly related to all this, ... but I digress.) 

I have doubts, the primary one being performance. Algorithm-for-algorithm, you can't beat C++ in performance, so the only way you can hope to win is to find a better algorithm. As a general rule, any performance gains here result in performance losses there, and so you have to aim at some real or imagined user who has some real (or imagined) problem they're trying to solve.  The current Atomspace has 15+ years of such real and imaginary users, and seems fairly well-rounded. It's no longer obvious what could be done differently. I know of some things, but none of them are smaller than 6-person-month projects.

My second concern is that you might not appreciate just how much stuff is in there, under the covers. A "database for s-expressions" - well, that's a central part. But there's more; for example, one of which is a srfi-213 per s-expression (recorded in the database).

FWIW, I'm contemplating some rather radical alternatives, specifically, a database for the "jigsaw-puzzle pieces" and sheaf theoretical tools in general. This has not left the stage of ideation, yet. The current atomspace is adequate, but not optimal.

> > I'm asking,
> > because this has proven to be useful enough, that I think the general
> > idea should be more wide-spread.

After much work around opencog and the atomspace I still do not
understand what is that "general idea" outside the Artificial General
Intelligence system.

The "general idea" of the atomspace is "a database of s-expressions". Plus srfi-213 identifier-properties stored in that database.

The "general idea" of opencog is indeed "AGI", but what this really means is that it's a large collection of ad-hoc ideas, implemented in poorly-written code that never worked well, if at all, and large parts of which are bit-rotting.  All that, plus the ecstacy and the agony of arguing about almost everything.

The historical foundation is very naive: let's just staple together some natural language processing with a reasoning engine and it will be automatically smart (pun). To be fair, pretty much everyone believed this 15-20 years ago, and from what I can tell, the (vast?) majority still believes this. Famous academics still debate this in popular media. I'll happily tell you how they're all wrong. People do not like to hear what I say.  I'm supposedly a keynote speaker for an upcoming conference; I have not prepared my notes yet.

Anyway, the few things that work in opencog have been the result of long, hard-fought battles and the hard knocks of experience. For me, the atomspace (the database of s-expressions) is the crown-jewel.  There's some runners-up that deserve more users and more attention. The stuff that should bit-rot in hell has been mostly piped to /dev/null .. but .. see, you use this stuff, you find out what's wrong, you get a glimmer of the right way of doing it, but without programmers, without professional management, without $$$, it doesn't get built. But that's actually a good thing, because if it was built, all you'd have is a Siri or an Alexa or a Watson, and that is not my goal. And frustratingly, not Ben's either. Because he's been in a position to build Alexa/Siri/Watson before those existed, but rejected that goal. If only. He's curiously self-sabotaging. But then, aren't we all?

> > I have heard of people implementing
> > datalog (the data subset of prolog) in scheme, but they did not seem
> > to have any pattern matcher,

datalog is a pattern matcher.

There's nothing that even comes close to what the Atomspace can do.  I suppose I should be humble and say "as far as I know, there is nothing..." because anything is possible. But I've watched the industry for a decade, and I feel we've consistently been a decade ahead of everyone else.  Although this is now a race that we (opencog, Ben) are about to lose, simply because of the vast quantities of industry excitement and investment. A rag-tag team of visionaries can only go so far, when pitted against (for example) the focus of the industrial benefactors of the Apache Foundation.

I'm working on new things (the sheaf theory) but obviously, graph databases are now mainstream, if still underwhelming. The days of what the atomspace can do, and others cannot, those days are numbered. I can read the writing on the wall.

 

Somewhat unrelated but possibly
interesting https://research.google/pubs/pub48190/

> > and it wasn't scalable/fast.

See the above google paper link.

It's for access control lists. That has nothing to do with anything!?

> > BTW, some,
> > but not all of my s-expressions are executable/evaluatable. Thus,
> > after pattern matching, not only is there a term-rewriting step, but
> > it is followed by an execute/evaluate step.

Again bundling extraction, and transformation (and possibly loading
the result) is a mistake, it should be opt-in at user demand.

I have no clue what you just said.  Are you saying "there should be well-defined kripke frames?" Yes, maybe there should be. It's not that easy. If wishes were horses....

The "users" of the atomspace are algorithms, not human beings.

Yes, there are design mistakes in there, but that's not what this email chain is about.

> > Apparently, graphQL for
> > javascript is a related concept. Note that graphQL has become very
> > popular.

Forget about GraphQL. GraphQL is not an algorithm, it is merely a
surface syntax for a neighbor query. Like I wrote before, the query
DSL is the last problem that should be solved. Getting together
example queries can help narrow / explain what problem must be solved,
but once you settle on a primary and secondary representation in the
database, a more powerful / fine-tuned DSL can emerge.

Not sure what you're saying. What's a DSL?

There are a few dozen demo's, and a few hundred unit tests, half of which target the pattern matcher.  Applications have included natural language, reasoning, robotics, rule application, genomics/proteomics,  fintech financial market technology, mental health and medical treatment outcomes, military radar surveillance, cybersecurity and my personal obsession: solving the symbol grounding problem.

The atomspace is what has emerged, after being applied to the above areas. It's imperfect, it's got problems, it is what it is. And what it is is mostly a database for s-expressions.

> > However, in javascript, all structures are always named;

GraphQL is usable and used in many programming languages. It is a
stringified query language like SQL. It is possible I am missing your
point.

JSON consists of a hierarchical arrangement of key-value pairs.  The key is the name.  For example: {name: "Joe", address: "123 main street", phone: "212-555-1212"}   S-expressions are anonymous: ("Joe" "123 main street" "212-555-1212") and the "meaning" of this triple is out-of-band. These differences have real-world implications for RAM usage and disk-storage efficiency.

These differences also have very strong and powerful effects on the kinds of queries that can be run, and how one constructs a query. These differences appear to have very strong and powerful effects on the imagination of programmers, and it appears to limit what they can conceive of as being possible.  Sapir-Whorf: language limits thought.  From what I can tell, pretty much all *QL people, graphQL, sparQL, etc etc. folks are laboring in self-imposed jail-cells because they are thinking about labelled, non-anonymous data.

When you change your mindset to creating a query language for s-expressions, the scales fall off your eyes. You can say "oh".  Queries can be expressed as s-expressions. Queries are just lambdas. Queries are just rewrites. Queries are just logical implications.  Speaking roughly and imprecisely, any lambda can be a query, any lambda can be a pattern.  The *QL people have not figured this out yet.

 

> > whereas all of my structures are always anonymous (and you have
> > several choices for naming, if you really need to name something.).
> > Thus, my system is far far more like scheme, than it is like javascript.)

Yes, data in the atomspace can be anonymous, but you still can query
for "what is under the table" even if the results were added without
human supervision, and retrieve the uid of the object(s) under the
table.

UID's have been shit-canned about 4-5 years ago. They were not just a bad idea, they were a supremely bad idea.

 

Hence, you could use graphql for that particular query.

Only if you have a schema, right? How do you propose doing that?  Do you have a workable example?

This isn't sarcasm -- I've never really tried, and perhaps it would be a good homework exercise. I find it hard to believe it could ever be fast. To me, graphql is fugly, so I've not been motivated to do more with it than to skim the docs.  Maybe there's some charm to it, but honestly, I don't see it. It's just ugly and lame.

 

>
> Scheme has traditionally had (match ...) for pattern-matching at
> runtime. If you want to pattern-match using a macro, and the patterns
> vary at runtime, you'd have to use (eval ...).
>

You can implement `match` logic with pattern combinators. So far, I
did not manage to make it work with cyclic data, and as far as I
understand from a performance point of view, the general case is a
cursed problem.

On the subject of choosing a database:

- What guarantees do you need. Question all the properties of ACID:
Atomic, Consistent, Isolation, Durability. Look into BASE. You do not
necessarily need ACID or BASE, but it is a good basis that is well
documented to know where you want / need to go.

Me personally? The atomspace? Neither acid nor base.  Those are not even the right concepts. The only thing that matters is that s-expressions must be globally unique, where "globally" is within whatever horizon you care to specify.

- What is the size of the data?

The genomics databases have (tens of ??) millions of s-expressions.  The natural language datasets max out at about 100 million s-expressions. But this is more of a statement about inadequate algorithms, rather than about natural language.   The robotics datasets have at most hundreds of thousands, but are constrained to run at near-real-time.  The other applications are proprietary; I can vaguely guess but I don't know.

In practice, the average size of an s-expression on disk is 75 bytes. This is with RocksDB, which uses LZMA compression when writing to disk.  The average size of an s-expression in RAM is about 1K to 1.5K, this includes all indexes, all lookup tables, all caches, everything. That is, saying "ps aux" shows about 1 GB of RAM usage (RSS size) per million s-expressions.

Since affordable machines come with 128 or 256 GB RAM, this means a hard upper limit of about 150 to 200 million s-expressions, before you have to get clever and design your algos to fetch from disk on-demand.

- What is the shape of the data?

All kinds. Some are wide and flat. Some are very tall.  The genomics data is square-root-of-Zipf. The natural language is  Zipf plus square-root-of-zipf plus gaussians. 

Are they well defined types?

The current atomspace has about 130 types. For example, List is a list and Set is an unordered set and Lambda is a lambda. There's And Or Not. Variable is kind-of-ish vaguely like a scheme symbol, but not really. Then stuff not in scheme: Inheritance, Subset, Glob, Concept, Predicate, Choice, Meet, Join, Query, State, Present , Absent, Always, Quote, Unquote, Type, TypeSet, TypeChoice, Arrow, Signature .. and 130 more. It's extensible with loadable modules, users can define their own types: for example, Gene, Protein, etc. There are about 30 different loadable modules. Half or more are bit-rotted.

 

Are they
polymorphic types

Yes. There is a type system. You can specify polymorphic types, signatures, arrows.  Currently not supported are dependent types, because they're kind of hard, and no one asked for them.

Absolutely NONE OF THIS MATTERS. All of this is just a bunch of s-expressions, from the point of view of the database itself. The database doesn't care (mostly).  The only types the database cares about are:

Lambda: these are alpha convertible, and so the database looks at them.

State: Only one state is allowed, ever, and state updates must be atomic.

Unique: Only one is allowed, ever, and it cannot be updated.

Delete: Exactly zero are allowed. Because, duh.

When I say "exactly one" I mean "exactly one for that key, having zero free variables". If it has free variables, then you can have as many as you want. Obviously, duh, as otherwise you would not be able to query it.

When I say "exactly zero", that means it can only contain free variables. It's impossible to store a Delete with zero free variables. Duh.

 

or heterogeneous shapes?

Shapes are currently layered on top of s-expressions. I'm toying with turning this upside-down, and layering s-expressions on top of shapes.  This seems counter-intuitive, but it looks like it might decrease RAM usage and improve performance and expressibility.  It seems it will be less clunky that way.  But the jury is out.

 

- Workload: Write-once then Read-only, mostly reads, mostly writes, a
mix of both. Answer also the question how fast or slow data can be
written or read.

Current Atomspace with the C++ interface does about 1 million s-expression reads/second on a 2015-era AMD server. Writes run at about 70K/second. Deletes run at about 100K/second.

Both python and guile are comparable, dropping to about 20K/second for any/all operations. Obviously, almost all of this is due to the expense of entering and exiting guile/python. This includes tricks, like not re-entering guile if the thread is already in guile.

 

- Querying: What queries look like: recursive / deep, columns or rows,
or neighborhood queries (like graphql and SQL without recursive
queries). Again what is the expected time to respond.

It's not a *QL. It's an s-expression query. There are two dozen or more demos. Read them.

We've got benchmarks. it's maybe (ten??)-thousand per second for simple queries, and hours  for complex ones e.g. the two-gene, two-protein, one-pathway query.

There's some very primitive query-compilation stuff, the static parts are static-analyzed and run only once, so only the dynamic part of the pattern runs at run-time.  Static analysis was added maybe 8 years ago.

There's several caches, so subqueries that have already been found are not searched for again. Caching improved the genomics queries by a factor of 10x or 20x. Now, they only take hours. Woot! :-)  

TODO item is to compile queries to byte-code. Like gnu-lightning for example. This would be a major project.

-- Linas

 

ref: https://stackoverflow.com/a/68608928/140837

--
Amirouche ~ https://hyper.dev

--
You received this message because you are subscribed to the Google Groups "scheme-reports-wg2" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scheme-reports-...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/scheme-reports-wg2/CAL7_Mo_gHVxfpkG%3Ddav9XDo-J9SVeauzkHBXrNq6XabN-66dBA%40mail.gmail.com.

[Amirouche Boubekki]

Linas, thanks to the awesome answer. I think it is better we move the
conversation in private or the opencog google group. Let me know.

[Daphne Preston-Kendal]

On 7 Aug 2021, at 10:11, Marc Nieper-Wißkirchen <marc....@gmail.com> wrote:

> To make some hopefully helpful proposal, what about the following questions (that are mostly not alternatives) together with some notes that are hopefully not biased:
>
> 1. Do you want to have a basic version of er-macro-transformer that allows one to write procedural hygienic macros in R7RS-large? (Note: This will include basically all versions of er-macro-transformer found in existing implementations.)
> 2. If an er-macro-transformer system is included in R7RS-large, do you want that it supports writing unhygienic macros? (Note: This will rule out a number of existing implementations.)
> 3. If an er-macro-transformer system is included in R7RS-large that supports writing unhygienic macros, do you want raw symbols in the macro output to be closed in the use environment instead of causing undefined behavior?
> 4. If an er-macro-transformer system is included in R7RS-large, do you want that the sameness of identifiers can be tested with `eqv?`? (Note: This basically rules out interoperability with syntax-case.)
> 5. Do you want the low-level macro system of R4RS to be included in R7RS-large? (Note: This is basically the syntax-case system without syntax-case and pattern variables.)
> 6. Do you want the full syntax-case system as described in R6RS to be included in R7RS-large? (Note: This question does not touch what constitutes a macro use.)
> 7. Do you want to have a version of with-ellipsis (see SRFI 211) included in R7RS-large so that R7RS's syntax-rules can be easily expressed in syntax-case?
> 8. Do you want that a single keyword (not in operator position) can constitute a macro use? (Note: Informally, this is known as "identifier syntax".)
> 9. Do you want that a keyword as the first argument to `set!` can constitute a macro use? (Note: Informally, this is known as "variable transformers".)
> 10. Do you want to have no limits on what constitutes a macro use as in section 9.2 of the R6RS? (Note: This implies a positive answer to questions 8. and 9.)
> 11. Do you want `identifier-syntax` as a convenient form to implement macros that can be used outside operator position? (Note: A positive answer here implies at least a positive answer to question 8.)
> 12. Do you want R7RS-large to support both er-macro-transformer and syntax-case? (Note: There are currently few implementations that support this. A positive answer here rules out question 4.)
> 13. Do you want R7RS-large to make only one of the two systems mandatory? (Note: A positive answer here obviously rules out a positive answer to question 12, and vice versa.)
> 14. Do you want R7RS-large to support unhygienic macros? (Note: A positive answer here rules out a negative answer to question 2. unless questions 5. and 6. are answered positively.)
> 15. If the syntax-case system is included in R7RS-large, do you want to impose that all syntax objects are always fully unwrapped? (Note: While this makes syntax-case closer to er-macro-transformer, this will rule out most existing implementations and some otherwise pleasant properties of syntax-case are lost.)
>
> I know that this is a long list of questions but I don't think for the worse as these questions have to be considered and answered by every voter while making their decision anyway.

While this is certainly thorough, and may in fact be close to the best possible form of the ballot, I feel many of my own preferences, at least, are highly conditional on what the consensus of the WG on other questions is. I should note I feel that implementor voices are crucial in all of the questions about macro systems, since as John notes it is essentially *the* question about the one non-portable feature that opens the door to making lots of other things possible, and thus may ultimately decide the credibility of the R7 process in the eyes of both R5 and R6 implementation maintainers. Also, unlike non-portable features which are non-portable e.g. just because they need an FFI, or OS interface access, which can be given bridged implementations, macro systems are also the one non-portable feature which potentially require serious compiler hacking to implement. Therefore it is doubly important to try to get implementation *maintainers* involved and not merely *users* attached to different implementations — these voices have thus far largely been missing from this discussion.

For example, considering what the basis of the system is, I (as a Scheme programmer, not implementor) would rank the options syntax-case > R4RS > explicit renaming. But if explicit renaming is defined as in SRFI 211 such that it has proper hygiene and can correctly implement syntax-case, or if we adopt syntax-case outright, I think we can toss out R4RS altogether as a redundant proposal. But the SRFI 211 form of explicit renaming is also completely unimplemented. (Marc, can implementations which have both explicit renaming and syntactic closures implement your ‘correct’ form of explicit renaming by using the latter to hack the extra hygiene features into the ‘simple’/incorrect form? I know you did this for syntax-case in Chibi …) And if it’s possible to implement syntax-case in terms of the SRFI 211 form of explicit renaming, presumably syntax-case can also be implemented in terms of the R4RS system? Am I correct?

Another question: Should it be possible to implement identifier syntax using syntax-rules patterns, as you implemented for Chicken, but (as I recently learned) R6RS forbade? If so, I think `identifier-syntax` itself is redundant.
And an issue: I would like to vote against any and all forms of generalized set!, but since SRFI 17 and the identifier-syntax forms are split out here, I feel I can’t do that.


Daphne

[Marc Nieper-Wißkirchen]

Am So., 8. Aug. 2021 um 12:10 Uhr schrieb Daphne Preston-Kendal <d...@nonceword.org>:

On 7 Aug 2021, at 10:11, Marc Nieper-Wißkirchen <marc....@gmail.com> wrote:

> To make some hopefully helpful proposal, what about the following questions (that are mostly not alternatives) together with some notes that are hopefully not biased:
>
> 1. Do you want to have a basic version of er-macro-transformer that allows one to write procedural hygienic macros in R7RS-large? (Note: This will include basically all versions of er-macro-transformer found in existing implementations.)
> 2. If an er-macro-transformer system is included in R7RS-large, do you want that it supports writing unhygienic macros? (Note: This will rule out a number of existing implementations.)
> 3. If an er-macro-transformer system is included in R7RS-large that supports writing unhygienic macros, do you want raw symbols in the macro output to be closed in the use environment instead of causing undefined behavior?
> 4. If an er-macro-transformer system is included in R7RS-large, do you want that the sameness of identifiers can be tested with `eqv?`? (Note: This basically rules out interoperability with syntax-case.)
> 5. Do you want the low-level macro system of R4RS to be included in R7RS-large? (Note: This is basically the syntax-case system without syntax-case and pattern variables.)
> 6. Do you want the full syntax-case system as described in R6RS to be included in R7RS-large? (Note: This question does not touch what constitutes a macro use.)
> 7. Do you want to have a version of with-ellipsis (see SRFI 211) included in R7RS-large so that R7RS's syntax-rules can be easily expressed in syntax-case?
> 8. Do you want that a single keyword (not in operator position) can constitute a macro use? (Note: Informally, this is known as "identifier syntax".)
> 9. Do you want that a keyword as the first argument to `set!` can constitute a macro use? (Note: Informally, this is known as "variable transformers".)
> 10. Do you want to have no limits on what constitutes a macro use as in section 9.2 of the R6RS? (Note: This implies a positive answer to questions 8. and 9.)
> 11. Do you want `identifier-syntax` as a convenient form to implement macros that can be used outside operator position? (Note: A positive answer here implies at least a positive answer to question 8.)
> 12. Do you want R7RS-large to support both er-macro-transformer and syntax-case? (Note: There are currently few implementations that support this.  A positive answer here rules out question 4.)
> 13. Do you want R7RS-large to make only one of the two systems mandatory? (Note: A positive answer here obviously rules out a positive answer to question 12, and vice versa.)
> 14. Do you want R7RS-large to support unhygienic macros? (Note: A positive answer here rules out a negative answer to question 2. unless questions 5. and 6. are answered positively.)
> 15. If the syntax-case system is included in R7RS-large, do you want to impose that all syntax objects are always fully unwrapped? (Note: While this makes syntax-case closer to er-macro-transformer, this will rule out most existing implementations and some otherwise pleasant properties of syntax-case are lost.)
>
> I know that this is a long list of questions but I don't think for the worse as these questions have to be considered and answered by every voter while making their decision anyway.

While this is certainly thorough, and may in fact be close to the best possible form of the ballot, I feel many of my own preferences, at least, are highly conditional on what the consensus of the WG on other questions is. I should note I feel that implementor voices are crucial in all of the questions about macro systems, since as John notes it is essentially *the* question about the one non-portable feature that opens the door to making lots of other things possible, and thus may ultimately decide the credibility of the R7 process in the eyes of both R5 and R6 implementation maintainers. Also, unlike non-portable features which are non-portable e.g. just because they need an FFI, or OS interface access, which can be given bridged implementations, macro systems are also the one non-portable feature which potentially require serious compiler hacking to implement. Therefore it is doubly important to try to get implementation *maintainers* involved and not merely *users* attached to different implementations — these voices have thus far largely been missing from this discussion.

Unfortunately, a lot of representatives of major R6RS systems have remained silent in discussions on R7RS-large, so we may have to do some guesswork. Alternatively, one could ignore these systems but I don't think this would lead to the best outcome for (future) users of R7RS-large. So, IMO, one should keep them in mind.

I would also like to add that the issue of "serious compiler hacking", which may or not may be needed thanks to existing pluggable implementations, is probably not the major issue. After all, whatever macro system has to be implemented only once, so even if it took, say, four weeks of developer time, it would pay off. What the more serious issue is, I think, is that it may make it impossible for implementations to provide an R7RS-large-conformant mode that still supports all proprietary legacy extensions of R7RS-small of that particular implementation. Personally, I think that adding such a requirement would be opposed to the success of R7RS-large in the long run, but some implementers may see it differently.

We should also keep in mind that new implementations will be developed.

For example, considering what the basis of the system is, I (as a Scheme programmer, not implementor) would rank the options syntax-case > R4RS > explicit renaming. But if explicit renaming is defined as in SRFI 211 such that it has proper hygiene and can correctly implement syntax-case, or if we adopt syntax-case outright, I think we can toss out R4RS altogether as a redundant proposal.

Yes, indeed. This is what I meant in a previous reply to John. R4RS also makes sense as an option if the full syntax-case system doesn't win. The R4RS system would be a compromise to satisfy those who believe in Alex's arguments in his old post against syntax-case in the context of Chicken (IMO, the arguments are based on some misunderstandings, but that's not the point here).

 

But the SRFI 211 form of explicit renaming is also completely unimplemented.

It is implemented in Unsyntax, at least. But note that this correct form is as expensive as implicit renaming.

(Marc, can implementations which have both explicit renaming and syntactic closures implement your ‘correct’ form of explicit renaming by using the latter to hack the extra hygiene features into the ‘simple’/incorrect form? I know you did this for syntax-case in Chibi …)

The underlying ER implementation has to be modified in general to support something like `datum->syntax` or `construct-identifier`. I did this for Chibi. In general, an implementation where `eqv?` can be used throughout to compare identifiers cannot become fully compliant as the following code snippet shows:

(let ((x 'foo))

  (let ((x 'bar))

    ...)

Both instances of `x` are the same identifier (in the sense of bound-identifier=?) because binding one would bind the other, but they cannot be the same in the sense of `eqv?` because they have different syntactic environments (that can be observed through `datum->syntax`.

 

And if it’s possible to implement syntax-case in terms of the SRFI 211 form of explicit renaming, presumably syntax-case can also be implemented in terms of the R4RS system? Am I correct?

The difference between syntax-case and the R4RS system is the syntax-case pattern matcher. This is already essentially part of any R7RS-small system because syntax-rules needs the same matcher. However, syntax-case adds the concept of pattern variables.  In order to be able to implement syntax-case as macro in the R4RS system (or in some form of ER), one needs something like SRFI 213.  (The question of adding that is, in principle, completely orthogonal to the question of the macro system, but it is certainly nice to have the possibility of implementing syntax-case as a library form.)

Implementing syntax-case on top of only SRFI 211 ER even with SRFI 213 is still not possible in the strict sense because a syntax-case transformer is just a transformation procedure.  For example,

(define-syntax foo (lambda (x) 42))

does not even invoke any procedure that could invoke `er-macro-transformer`.

 

Another question: Should it be possible to implement identifier syntax using syntax-rules patterns, as you implemented for Chicken, but (as I recently learned) R6RS forbade?

To be precise, R6RS forbids extensions to syntax-rules in its (rnrs (6)) library in order to enforce writing portable programs. But it does not forbid to add any extensions in, say, (rnrs (6 1)) or in (fantastic-scheme). For example, Chez's syntax-rules supports fenders when imported from (chezscheme), but does not when imported from (rnrs (6)). (IMO, this approach of R6RS is not as bad as it is often understood; for example, when I run my C compiler I also want it to reject non-portable code as much as possible unless I explicitly opt-in.)

It's another story whether syntax-rules actually should support clauses of the form <identifier>.  I don't think it fits very well because syntax-rules treats top-level list patterns special in that the first pattern variable is ignored and it is unclear how to coherently extend this rule to more general patterns.

Furthermore, with ignoring the first pattern variable, syntax-rules still wouldn't support the `(set! <identifier> <expr>)` form.

With offering both a (restricted) syntax-rules and identifier-syntax (both as simple wrappers around the more general syntax-case), R6RS probably got it right.

 

If so, I think `identifier-syntax` itself is redundant.
And an issue: I would like to vote against any and all forms of generalized set!, but since SRFI 17 and the identifier-syntax forms are split out here, I feel I can’t do that.

What do you mean by generalized `set!`? SRFI 17 or macro uses with `set!` in the keyword position (aka variable transformers) or both?

When we allow keywords in operand position as macro uses, leaving out variable transformers would be a stupid mistake.  For example, if we have keywords in operand position as macro uses, we may want to use them to define fast versions of procedures with optionals and keywords:

(define/optionals (foo bla : blubb ...)

  ...)

But to make this transparent, we need to support a subsequent `(set! foo 42)`, and for this we need variable transformers.

A side remark on SRFI 17: Before anyone votes on this, don't forget to read the arguments of Matthias Felleisen on the SRFI 17 mailing list carefully. While the tone with which the discussions there were held may take some getting used to, the truth value of his words that SRFI 17 may be based on and promotes some deep misunderstanding of the role of `set!` is independent of that.

Marc

[Daphne Preston-Kendal]

On 8 Aug 2021, at 15:33, Marc Nieper-Wißkirchen <marc....@gmail.com> wrote:

> Unfortunately, a lot of representatives of major R6RS systems have remained silent in discussions on R7RS-large, so we may have to do some guesswork. Alternatively, one could ignore these systems but I don't think this would lead to the best outcome for (future) users of R7RS-large. So, IMO, one should keep them in mind.
>
> I would also like to add that the issue of "serious compiler hacking", which may or not may be needed thanks to existing pluggable implementations, is probably not the major issue. After all, whatever macro system has to be implemented only once, so even if it took, say, four weeks of developer time, it would pay off.

Nonetheless, our goal in designing R7RS is surely to maintain the respect and interest of the implementers who were alienated by R6RS because of the high bar it set in terms of adopting support for it, in terms of what they already had. We can assume any implementation or implementer of R6RS will want syntax-case to be the one macro system adopted by R7RS (at least if we do decide to have only one and only one low-level macro system in the standard). What people think who have thus far only implemented R5RS and/or R7RS small is the real question I’d like to know the answer to.

> What the more serious issue is, I think, is that it may make it impossible for implementations to provide an R7RS-large-conformant mode that still supports all proprietary legacy extensions of R7RS-small of that particular implementation. Personally, I think that adding such a requirement would be opposed to the success of R7RS-large in the long run, but some implementers may see it differently.

This is more or less exactly my point in wanting to get implementers involved.

> We should also keep in mind that new implementations will be developed.

Certainly, but I don’t think it’s so helpful to consider their needs. Whatever we choose can be adopted by a new implementation which explicitly sets out to follow R7RS (as long as we don't require them to solve the halting problem!).

>> And if it’s possible to implement syntax-case in terms of the SRFI 211 form of explicit renaming, presumably syntax-case can also be implemented in terms of the R4RS system? Am I correct?
>
> The difference between syntax-case and the R4RS system is the syntax-case pattern matcher. This is already essentially part of any R7RS-small system because syntax-rules needs the same matcher. However, syntax-case adds the concept of pattern variables. In order to be able to implement syntax-case as macro in the R4RS system (or in some form of ER), one needs something like SRFI 213. (The question of adding that is, in principle, completely orthogonal to the question of the macro system, but it is certainly nice to have the possibility of implementing syntax-case as a library form.)

If one added with-syntax to R4RS (which syntax-case typically defines in terms of the pattern matcher, thus tightly integrating pattern variable binding with pattern matching, which I guess is the core of Alex's objection), SRFI 213 isn’t needed though, is it? In other words, instead of considering the big complicated pattern matcher the primitive pattern variable binder and implementing with-syntax on top of it, could one not consider the simple with-syntax a primitive and implement the pattern matcher on top of it and the other features in R4RS’s system?

> Implementing syntax-case on top of only SRFI 211 ER even with SRFI 213 is still not possible in the strict sense because a syntax-case transformer is just a transformation procedure. For example,
>
> (define-syntax foo (lambda (x) 42))
>
> does not even invoke any procedure that could invoke `er-macro-transformer`.

Ah, yes. The only way to fix this I guess would be to add another analogue to {er,ir,sc,rsc}-macro-transformer around the definition procedure for macros using the syntax-case (or R4RS) system, so your example would be formally an error and have to be written:

(define-syntax foo (syntax-transformer (lambda (x) 42)))

or the like. Reading Alex’s post, the lack of an analogous form such as this was another one of his problems with the syntax-case system (as by having no such wrapper it essentially defines itself as the ultimate macro system to rule all others), so perhaps this would not be a bad idea even though it even further increases the verbosity of any syntax-case macro.

> With offering both a (restricted) syntax-rules and identifier-syntax (both as simple wrappers around the more general syntax-case), R6RS probably got it right.

Okay, fine.

>> And an issue: I would like to vote against any and all forms of generalized set!, but since SRFI 17 and the identifier-syntax forms are split out here, I feel I can’t do that.
>
> What do you mean by generalized `set!`? SRFI 17 or macro uses with `set!` in the keyword position (aka variable transformers) or both?

Both. Specifically, I think:
• the first argument to set! should always have to be an identifier
• if that identifier happens to be the name of a macro keyword, set! wipes the macro definition and makes it a regular variable

I believe this is consistent with R7RS small, even if not with R6RS. (Although I just tried it in Chibi and it segfaulted, and no other implementation I tried seems to actually behave like this either — they think it’s an error to set! any macro keyword. Reading the definition of set! in R7RS small, I don’t really understand why.)

> When we allow keywords in operand position as macro uses, leaving out variable transformers would be a stupid mistake. For example, if we have keywords in operand position as macro uses, we may want to use them to define fast versions of procedures with optionals and keywords:
>
> (define/optionals (foo bla : blubb ...)
> ...)
>
> But to make this transparent, we need to support a subsequent `(set! foo 42)`, and for this we need variable transformers.

I guess it depends what you what the `(set! foo 42)` to do ;-)

> Marc


Daphne

[Marc Nieper-Wißkirchen]

Am So., 8. Aug. 2021 um 16:23 Uhr schrieb Daphne Preston-Kendal <d...@nonceword.org>:

On 8 Aug 2021, at 15:33, Marc Nieper-Wißkirchen <marc....@gmail.com> wrote:

> Unfortunately, a lot of representatives of major R6RS systems have remained silent in discussions on R7RS-large, so we may have to do some guesswork. Alternatively, one could ignore these systems but I don't think this would lead to the best outcome for (future) users of R7RS-large. So, IMO, one should keep them in mind.
>
> I would also like to add that the issue of "serious compiler hacking", which may or not may be needed thanks to existing pluggable implementations, is probably not the major issue. After all, whatever macro system has to be implemented only once, so even if it took, say, four weeks of developer time, it would pay off.

Nonetheless, our goal in designing R7RS is surely to maintain the respect and interest of the implementers who were alienated by R6RS because of the high bar it set in terms of adopting support for it, in terms of what they already had. We can assume any implementation or implementer of R6RS will want syntax-case to be the one macro system adopted by R7RS (at least if we do decide to have only one and only one low-level macro system in the standard). What people think who have thus far only implemented R5RS and/or R7RS small is the real question I’d like to know the answer to.

> What the more serious issue is, I think, is that it may make it impossible for implementations to provide an R7RS-large-conformant mode that still supports all proprietary legacy extensions of R7RS-small of that particular implementation. Personally, I think that adding such a requirement would be opposed to the success of R7RS-large in the long run, but some implementers may see it differently.

This is more or less exactly my point in wanting to get implementers involved.

All these are very good points. Just let me add as a remark that a standard that is just the intersection of what is possible with existing implementations may not be very much more than R7RS-small together with a collection of portable libraries.  But for the latter, a package manager/resource is more important than just listing them in a standard.

 

> We should also keep in mind that new implementations will be developed.

Certainly, but I don’t think it’s so helpful to consider their needs. Whatever we choose can be adopted by a new implementation which explicitly sets out to follow R7RS (as long as we don't require them to solve the halting problem!).

>> And if it’s possible to implement syntax-case in terms of the SRFI 211 form of explicit renaming, presumably syntax-case can also be implemented in terms of the R4RS system? Am I correct?
>
> The difference between syntax-case and the R4RS system is the syntax-case pattern matcher. This is already essentially part of any R7RS-small system because syntax-rules needs the same matcher. However, syntax-case adds the concept of pattern variables.  In order to be able to implement syntax-case as macro in the R4RS system (or in some form of ER), one needs something like SRFI 213.  (The question of adding that is, in principle, completely orthogonal to the question of the macro system, but it is certainly nice to have the possibility of implementing syntax-case as a library form.)

If one added with-syntax to R4RS (which syntax-case typically defines in terms of the pattern matcher, thus tightly integrating pattern variable binding with pattern matching, which I guess is the core of Alex's objection), SRFI 213 isn’t needed though, is it? In other words, instead of considering the big complicated pattern matcher the primitive pattern variable binder and implementing with-syntax on top of it, could one not consider the simple with-syntax a primitive and implement the pattern matcher on top of it and the other features in R4RS’s system?

Yes, you could do that.  But I would say it would be more in the spirit of Scheme to provide the general tools to write something like a restricted `with-syntax` (namely SRFI 213) than to needlessly restrict to a single use-case.  That said, SRFI 93 makes a lot of points why the exact choice of abstractions of the syntax-case system benefits the user and promotes robust macro definitions, and experience proves Kent Dybvig right.

 

> Implementing syntax-case on top of only SRFI 211 ER even with SRFI 213 is still not possible in the strict sense because a syntax-case transformer is just a transformation procedure.  For example,
>
> (define-syntax foo (lambda (x) 42))
>
> does not even invoke any procedure that could invoke `er-macro-transformer`.

Ah, yes. The only way to fix this I guess would be to add another analogue to {er,ir,sc,rsc}-macro-transformer around the definition procedure for macros using the syntax-case (or R4RS) system, so your example would be formally an error and have to be written:

(define-syntax foo (syntax-transformer (lambda (x) 42)))

or the like. Reading Alex’s post, the lack of an analogous form such as this was another one of his problems with the syntax-case system (as by having no such wrapper it essentially defines itself as the ultimate macro system to rule all others), so perhaps this would not be a bad idea even though it even further increases the verbosity of any syntax-case macro.

The claim of the "ultimate macro system ruling all others" is just plain wrong. First of all, the idea that procedures can be macro transformers is orthogonal to syntax-case/syntax (as my above example shows, neither appears).  This only says that a transformer can be given by a procedure mapping an input syntax object into an output syntax object (which is, IMO, pedagogically a very good thing).  Secondly, already R6RS knows a different type of transformer, namely variable transformers, and adding more transformer types like ER macro transformers is straightforward and doesn't need to cause any incompatibilities.  By the way, this has already been discussed at length between Alex and some people supporting R6RS when they were still on board at the beginning of the R7RS-large process.  Adding something like `syntax-transformer` would help no user but only alienate some.

 

> With offering both a (restricted) syntax-rules and identifier-syntax (both as simple wrappers around the more general syntax-case), R6RS probably got it right.

Okay, fine.

>> And an issue: I would like to vote against any and all forms of generalized set!, but since SRFI 17 and the identifier-syntax forms are split out here, I feel I can’t do that.
>
> What do you mean by generalized `set!`? SRFI 17 or macro uses with `set!` in the keyword position (aka variable transformers) or both?

Both. Specifically, I think:
• the first argument to set! should always have to be an identifier
• if that identifier happens to be the name of a macro keyword, set! wipes the macro definition and makes it a regular variable

This would be wrong without any doubt. It would not even be clear what the lexical scope of such a wipeout would be.  It would also rule out a lot of very sensible macros.  The following is an example:

(define-syntax volatile-var

  (identifier-syntax

    (_ (get-i/o-port))

    ((set! _ e) (put-i/o-port e))))

I believe this is consistent with R7RS small, even if not with R6RS. (Although I just tried it in Chibi and it segfaulted, and no other implementation I tried seems to actually behave like this either — they think it’s an error to set! any macro keyword. Reading the definition of set! in R7RS small, I don’t really understand why.)

The first argument to `set!` in R7RS-small is a <variable>.  A <variable> is not a <keyword>.

It would make no sense anyway.

 

> When we allow keywords in operand position as macro uses, leaving out variable transformers would be a stupid mistake.  For example, if we have keywords in operand position as macro uses, we may want to use them to define fast versions of procedures with optionals and keywords:
>
> (define/optionals (foo bla : blubb ...)
>   ...)
>
> But to make this transparent, we need to support a subsequent `(set! foo 42)`, and for this we need variable transformers.

I guess it depends what you what the `(set! foo 42)` to do ;-)

In this case, `foo` would remain bound to the same keyword but that would expand into `42` from then on.

Scheme has lexical scope that comes structured in blocks.  In each lexical scope, an identifier is either a variable, or a keyword, or unbound.  (Or of some other type, like a pattern variable.)  Your `set!` semantics would only make sense if the type of Scheme identifiers was dynamically and not lexically determined.

[Linas Vepstas]

On Sun, Aug 8, 2021 at 1:05 AM Amirouche Boubekki <amirouche...@gmail.com> wrote:

Linas, thanks to the awesome answer. I think it is better we move the
conversation in private or the opencog google group. Let me know.

You're welcome. Without further prompting, I've nothing more to add.

-- Linas

[John Cowan]

On Sun, Aug 8, 2021 at 10:23 AM Daphne Preston-Kendal <d...@nonceword.org> wrote:

 

What people think who have thus far only implemented R5RS and/or R7RS small is the real question I’d like to know the answer to.

Well, we can judge what they think (to some extent) based on what they have done.   Of the non-R6RS implementations, Gambit, Kawa, SISC, SXM support syntax-case; Gauche, MIT, Chicken, Scheme48/scsh, Picrin, Chibi support ER.  (Chibi has a third-party syntax-case library now.)  Of these, all predate R7RS except Picrin and Chibi.

 

Both. Specifically, I think:
• the first argument to set! should always have to be an identifier

I think the CL / SRFI 17 format (set! (car x) y) as equivalent to (set-car! x y) is actually very nice, and more important than (set! car-x y), which must be defined for each x.

[Marc Nieper-Wißkirchen]

As for the former, read Felleisen. As for the latter: Why would one want to do it?

[Dr. Arne Babenhauserheide]

Marc Nieper-Wißkirchen <marc....@gmail.com> writes:

> All these are very good points. Just let me add as a remark that a standard
> that is just the intersection of what is possible with existing
> implementations may not be very much more than R7RS-small together with a
> collection of portable libraries. But for the latter, a package
> manager/resource is more important than just listing them in a standard.

The advantage of a standard is that all its parts can fit together and
it can be documented and understood in one go.

It’s the "batteries included" approach.

A package manager is great, but looking at the horror of tiny packages
of Javascript, I see the strength of having a strong set of always
available libraries in uniform style.

You can more easily understand them all together, when your assumptions
from other libraries just work in the other libraries.

[Daphne Preston-Kendal]

On 8 Aug 2021, at 17:02, Marc Nieper-Wißkirchen <marc....@gmail.com> wrote:

> In this case, `foo` would remain bound to the same keyword but that would expand into `42` from then on.
>
> Scheme has lexical scope that comes structured in blocks. In each lexical scope, an identifier is either a variable, or a keyword, or unbound. (Or of some other type, like a pattern variable.) Your `set!` semantics would only make sense if the type of Scheme identifiers was dynamically and not lexically determined.

Yes, indeed, you’re correct. D’oh. I now see the argument that the identifier-syntax variety of generalized set! (where the first argument to set! is still restricted to be an identifier) is likely necessary. (A very ugly way to avoid this would be to require identifier macros to expand to another identifier in all contexts if they’re supposed to be mutable bindings, and *that* identifier then gets rebound if it’s the first argument to set!. This turns identifier syntax, at least without the possibility of context-sensitive expansion at least, into little more than glorified aliasing.)

On 8 Aug 2021, at 15:33, Marc Nieper-Wißkirchen <marc....@gmail.com> wrote:

> A side remark on SRFI 17: Before anyone votes on this, don't forget to read the arguments of Matthias Felleisen on the SRFI 17 mailing list carefully. While the tone with which the discussions there were held may take some getting used to, the truth value of his words that SRFI 17 may be based on and promotes some deep misunderstanding of the role of `set!` is independent of that.

I have taken a look at his posts, and his arguments seem to mostly be that SRFI 17 (in an early draft) was unclear about the mechanism, which apparently was resolved in later drafts and the final spec? Or can you point to another specific thread?

Nonetheless my feeling is that SRFI 17 setters don’t compose well (though the worst of this problem is mitigated by the fact that it does already specify that there are already setters for cXr up to as many X as Scheme provides — but these are ‘fake’, because of course there is no set-cadr! for example, and also no setters for SRFI 1 first, second, third, etc, nor for any other number of procedures which might *appear* settable but aren’t. I guess this is what ‘promotes misunderstanding about set!’).


Daphne

[Lassi Kortela]

> The advantage of a standard is that all its parts can fit together and
> it can be documented and understood in one go.
>
> It’s the "batteries included" approach.
>
> A package manager is great, but looking at the horror of tiny packages
> of Javascript, I see the strength of having a strong set of always
> available libraries in uniform style.
>
> You can more easily understand them all together, when your assumptions
> from other libraries just work in the other libraries.

Strongly agreed, but RnRS is too rigorous a process for creating such
libraries. We should start another design process for non-core stuff,
more lightweight than RnRS and SRFI.

(The same problem exists in other languages communities; human ingenuity
doesn't scale up to this level of rigor and scope at once. The slogan
"batteries included" originated in Python, but some Python experts look
at the standard library as the place where modules go to die, meaning
useful development stops and better alternatives struggle to take over.
<https://leancrew.com/all-this/2012/04/where-modules-go-to-die/>)

[Marc Nieper-Wißkirchen]

Am Mo., 9. Aug. 2021 um 09:43 Uhr schrieb Daphne Preston-Kendal <d...@nonceword.org>:

On 8 Aug 2021, at 17:02, Marc Nieper-Wißkirchen <marc....@gmail.com> wrote:

> In this case, `foo` would remain bound to the same keyword but that would expand into `42` from then on.
>
> Scheme has lexical scope that comes structured in blocks.  In each lexical scope, an identifier is either a variable, or a keyword, or unbound.  (Or of some other type, like a pattern variable.)  Your `set!` semantics would only make sense if the type of Scheme identifiers was dynamically and not lexically determined.

Yes, indeed, you’re correct. D’oh. I now see the argument that the identifier-syntax variety of generalized set! (where the first argument to set! is still restricted to be an identifier) is likely necessary. (A very ugly way to avoid this would be to require identifier macros to expand to another identifier in all contexts if they’re supposed to be mutable bindings, and *that* identifier then gets rebound if it’s the first argument to set!. This turns identifier syntax, at least without the possibility of context-sensitive expansion at least, into little more than glorified aliasing.)

Would your "very ugly way" work with, say, the idea of using "identifier syntax" to generate fast procedure calls when keywords are involved?

That said, it is probably not a good idea to invent something new and create new incompatibilities when the R6RS approach just works.

 

On 8 Aug 2021, at 15:33, Marc Nieper-Wißkirchen <marc....@gmail.com> wrote:

> A side remark on SRFI 17: Before anyone votes on this, don't forget to read the arguments of Matthias Felleisen on the SRFI 17 mailing list carefully. While the tone with which the discussions there were held may take some getting used to, the truth value of his words that SRFI 17 may be based on and promotes some deep misunderstanding of the role of `set!` is independent of that.


I have taken a look at his posts, and his arguments seem to mostly be that SRFI 17 (in an early draft) was unclear about the mechanism, which apparently was resolved in later drafts and the final spec? Or can you point to another specific thread?

This was a sideshow.

The actual arguments why generalizing `set!` in a way SRFI 17 does promotes a deep misunderstanding of some basic semantic concepts of Scheme start around here: https://srfi-email.schemers.org/srfi-17/msg/2778583/.

Further down is the following quote by Matthias Felleisen:

**

So if you were with 100 other people and they all jumped off the cliff, you
would jump, too?

I have always hoped that Schemers developed things because they were right
not because the majority did it. Here "right" means there is supportive
pragmatic evidence and an underlying theory. [...]
**

We should always keep this quote under our pillows when we dream of new features.

 

Nonetheless my feeling is that SRFI 17 setters don’t compose well (though the worst of this problem is mitigated by the fact that it does already specify that there are already setters for cXr up to as many X as Scheme provides — but these are ‘fake’, because of course there is no set-cadr! for example, and also no setters for SRFI 1 first, second, third, etc, nor for any other number of procedures which might *appear* settable but aren’t. I guess this is what ‘promotes misunderstanding about set!’).

Full ack.

Moreover, what is the use case of SRFI 17 setters anyway? (Apart from saving a few keystrokes and helping people to misunderstand the semantics?)

[Daphne Preston-Kendal]

On 9 Aug 2021, at 10:14, Marc Nieper-Wißkirchen <marc....@gmail.com> wrote:

> Would your "very ugly way" work with, say, the idea of using "identifier syntax" to generate fast procedure calls when keywords are involved?

If a definition for a keyword procedure kf would expand to something like:

(begin
(define (kf-positional a b c d) … actual definition of kf with all keyword args replaced by positional args …)
(define (kf-dynamic a b . kws) … map keyword arguments in kws to the c and d from kf-positional at runtime, calling kf-positional with them …)
(define-syntax kf
(lambda (stx)
(syntax-case stx ()
((_ a b . kws) … map keyword arguments to c and d at compile time and expand into a call to kf-positional (as much as possible) …)
(_ #'kf-dynamic)))))

then you could set! kf, but that would only affect uses of it in operand position.

So I guess that’s a no.

> That said, it is probably not a good idea to invent something new and create new incompatibilities when the R6RS approach just works.

Yes, okay, agreed.

On 8 Aug 2021, at 15:33, Marc Nieper-Wißkirchen <marc....@gmail.com> wrote:

>> Nonetheless my feeling is that SRFI 17 setters don’t compose well (though the worst of this problem is mitigated by the fact that it does already specify that there are already setters for cXr up to as many X as Scheme provides — but these are ‘fake’, because of course there is no set-cadr! for example, and also no setters for SRFI 1 first, second, third, etc, nor for any other number of procedures which might *appear* settable but aren’t. I guess this is what ‘promotes misunderstanding about set!’).
>
> Full ack.
>
> Moreover, what is the use case of SRFI 17 setters anyway? (Apart from saving a few keystrokes and helping people to misunderstand the semantics?)

Yes, agreed. The convenience of the feature doesn’t justify the complexity, edge-cases, and confusion caused by the mechanism.


Daphne

[Marc Nieper-Wißkirchen]

Am Mo., 9. Aug. 2021 um 09:58 Uhr schrieb Lassi Kortela <la...@lassi.io>:

> The advantage of a standard is that all its parts can fit together and
> it can be documented and understood in one go.
>
> It’s the "batteries included" approach.
>
> A package manager is great, but looking at the horror of tiny packages
> of Javascript, I see the strength of having a strong set of always
> available libraries in uniform style.
>
> You can more easily understand them all together, when your assumptions
> from other libraries just work in the other libraries.

Strongly agreed, but RnRS is too rigorous a process for creating such
libraries. We should start another design process for non-core stuff,
more lightweight than RnRS and SRFI.

I agree with Lassi.  First and foremost, it is the foundations that matter.  If the foundations are chosen well, we get a beautiful language that supports writing everything else as portable libraries, at least in theory.  On top of the foundations, people can then craft coherent, well-documented sets of libraries.  It's important to have an idea of what these libraries would be to understand what the foundations have to provide.

If we look at the C standard, it is a very slowly evolving standard that only specifies an extremely limited set of libraries.  Yet, the C language is capable of supporting almost the whole existing world of computing. Admittedly, C has one big advantage over Scheme:  It is the only language where FFI means NFI on virtually all existing systems.

Marc

[Alex Shinn]

On Mon, Aug 9, 2021 at 9:37 AM John Cowan <co...@ccil.org> wrote:
>
> On Sun, Aug 8, 2021 at 10:23 AM Daphne Preston-Kendal <d...@nonceword.org> wrote:
>
>> What people think who have thus far only implemented R5RS and/or R7RS small is the real question I’d like to know the answer to.
>
> Well, we can judge what they think (to some extent) based on what they have done. Of the non-R6RS implementations, Gambit, Kawa, SISC, SXM support syntax-case; Gauche, MIT, Chicken, Scheme48/scsh, Picrin, Chibi support ER. (Chibi has a third-party syntax-case library now.) Of these, all predate R7RS except Picrin and Chibi.

Chibi predates R7RS. The first released version was in April 2009
(https://groups.google.com/g/comp.lang.scheme/c/9FGBFVE7X8U).
The Steering Committee announced its R7RS effort in August of that
year (http://www.scheme-reports.org/2009/announcement.html).
WG1 started work in February 2010.

The raison d'être of Chibi was a better TinyScheme with full R5RS
support in roughly the same size.
Gradually it became the unofficial R7RS-small reference implementation.

>> Both. Specifically, I think:
>> • the first argument to set! should always have to be an identifier
>
> I think the CL / SRFI 17 format (set! (car x) y) as equivalent to (set-car! x y) is actually very nice, and more important than (set! car-x y), which must be defined for each x.

I think it would be nice to consider removing restrictions to allow
more general macros (both identifier syntax and SRFI 17), than to add
each as separate features. Arbitrary pattern matching rules applied
to every form is one such option. This is experimental, though not
difficult (modulo compile-time efficiency).

This thread is too long for me to keep track of, but regarding
syntax-case: my "pamphlet" from way back was sent to the chicken-users
mailing list and specifically talking about the syntax-case egg, at a
time when Chicken allowed swapping in custom macro systems. I still
dislike syntax-case, but would make different complaints if it came to
ballot, and may do so. I would nonetheless probably vote in favor of
syntax-case for R7RS-large, though, in line with that language's
charter.

--
Alex

[Alex Shinn]

On Mon, Aug 9, 2021 at 12:02 AM Marc Nieper-Wißkirchen
<marc....@gmail.com> wrote:
>
> The claim of the "ultimate macro system ruling all others" is just plain wrong. First of all, the idea that procedures can be macro transformers is orthogonal to syntax-case/syntax (as my above example shows, neither appears). This only says that a transformer can be given by a procedure mapping an input syntax object into an output syntax object (which is, IMO, pedagogically a very good thing). Secondly, already R6RS knows a different type of transformer, namely variable transformers, and adding more transformer types like ER macro transformers is straightforward and doesn't need to cause any incompatibilities. By the way, this has already been discussed at length between Alex and some people supporting R6RS when they were still on board at the beginning of the R7RS-large process. Adding something like `syntax-transformer` would help no user but only alienate some.

Discussed, yes, but I never agreed with the other side.

It would help Chibi remove one of it's biggest warts, in the hack that
allows syntax-case to co-exist with ER macros.
And it would help implementations that wouldn't consider adding such a
hack in the first place.

The alienation for R6RS users is minimal. If you want you can even
define a define-syntax that implicitly wraps its argument with
syntax-transformer. No such trivial user-space workaround exists in
the other direction.

--
Alex

[Marc Nieper-Wißkirchen]

Am Mo., 9. Aug. 2021 um 13:10 Uhr schrieb Alex Shinn <alex...@gmail.com>:

On Mon, Aug 9, 2021 at 12:02 AM Marc Nieper-Wißkirchen
<marc....@gmail.com> wrote:
>
> The claim of the "ultimate macro system ruling all others" is just plain wrong. First of all, the idea that procedures can be macro transformers is orthogonal to syntax-case/syntax (as my above example shows, neither appears).  This only says that a transformer can be given by a procedure mapping an input syntax object into an output syntax object (which is, IMO, pedagogically a very good thing).  Secondly, already R6RS knows a different type of transformer, namely variable transformers, and adding more transformer types like ER macro transformers is straightforward and doesn't need to cause any incompatibilities.  By the way, this has already been discussed at length between Alex and some people supporting R6RS when they were still on board at the beginning of the R7RS-large process.  Adding something like `syntax-transformer` would help no user but only alienate some.

Discussed, yes, but I never agreed with the other side.

Yeah, I just didn't want to repeat all arguments from both sides. :)

 

It would help Chibi remove one of it's biggest warts, in the hack that
allows syntax-case to co-exist with ER macros.
And it would help implementations that wouldn't consider adding such a
hack in the first place.

It's not true that any hack is needed for the coexistence of ER macros with transformer procedures.  It's enough to make sure that evaluating the er-macro-transformer procedure returns an object of a type that is disjoint to (ordinary) procedures.  This is what Unsyntax does, for example.  I believe that that's a straightforward and very clean solution.

The case with Chibi is special because you chose to use procedure values also for the underlying implementation.  It's that choice (that wouldn't have been necessary and that is not exposed to the user through er/sc/rsc-macro-transformer) that has forced me to hack around it to add syntax-case.

It would be different if Chibi's (lambda (form use-env mac-env) ...) were part of just another macro facility to be standardized.  If that were the case, then we would have some incompatibility with R6RS.

 

The alienation for R6RS users is minimal.  If you want you can even
define a define-syntax that implicitly wraps its argument with
syntax-transformer.  No such trivial user-space workaround exists in
the other direction.

Can you explain what you exactly mean by your last sentence?  That no (user-space) workaround exists for implementations for which (define-syntax foo (lambda (arg) ...)) means something else than what binding a transformer procedure to foo in R6RS means?  But that would be very similar to the situation of implementations whose, say, native error procedure follows R6RS and not R7RS (or SRFI 23).  Such an implementation cannot offer its native error procedure in (scheme base) but has to offer a thin wrapper around it.  I mean, these are the benefits of the library system of R7RS.  An implementation can always offer non-standard stuff through non-standard libraries.  The only thing that has to be ensured is that implementation of the standard features is possible (so that they can be exported through the standard libraries).  And this is clearly the case.

It's my personal opinion that making both syntax-case and er-macro-transformer mandatory wouldn't do the standard and current and future implementers a big favor. Making one mandatory but not saying which would similarly not do the standard and its users a big favor.  The best outcome IMO is that only one would be mandatory and that it would be specified which.  In that case, the above discussion is probably moot because we would either have (er-macro-transformer (lambda (expr rename compare) ...)) or (lambda (stx) ...) in the standard.  (I left out the fourth possible outcome, namely no procedural macro system in R7RS-large, in which case the discussion is totally irrelevant as far as the standard goes.)

[Lassi Kortela]

> If we look at the C standard, it is a very slowly evolving standard that
> only specifies an extremely limited set of libraries.  Yet, the C
> language is capable of supporting almost the whole existing world of
> computing. Admittedly, C has one big advantage over Scheme:  It is the
> only language where FFI means NFI on virtually all existing systems.

R5RS was conservative like C, and most of the esteemed implementations
(MIT Scheme, Chez Scheme, Gambit, Chicken, Bigloo) are conservative.

Guile and Kawa are big, long-lived implementations that take more
liberties with the language, as does Racket obviously, but each of these
three has taken the language in quite a different direction.

[Per Bothner]

On 8/9/21 12:43 AM, Daphne Preston-Kendal wrote:
> On 8 Aug 2021, at 15:33, Marc Nieper-Wißkirchen <marc....@gmail.com> wrote:
>
>> A side remark on SRFI 17: Before anyone votes on this, don't forget to read the arguments of Matthias Felleisen on the SRFI 17 mailing list carefully. While the tone with which the discussions there were held may take some getting used to, the truth value of his words that SRFI 17 may be based on and promotes some deep misunderstanding of the role of `set!` is independent of that.
>
>
> I have taken a look at his posts, and his arguments seem to mostly be that SRFI 17 (in an early draft) was unclear about the mechanism, which apparently was resolved in later drafts and the final spec? Or can you point to another specific thread?

If I recall (and haven't re-read the thread) the argument was that 'set!' is about changing
bindings (variables) and should not be overloaded to change properties of objects.
(I disagree, parly because you can just as validly view a variable as a property of a
frame object.)

Claiming this is a "deep misunderstanding" is elevating a preference to a fundamental
principle. I think it is a distraction. The question is: does using SRFI-17 set! make
the code easier to read? (I believe it does.) Does it lead more or fewer errors or
misunderstandings of the code? (I see no reason to believe so.) Does the fact that most
other languages similarly overload assignment make it easier (or at least more familiar)
for new Scheme users? (I don't know, but it seems plausible to me.)

It might be preferable if set! bindings were lexical bindings that could be imported.
Common Lisp supports that, in the sense that (setf FOO) is a pseudo-symbol in certain contexts.
One could define
(set! (foo x y z) value)
as
($setter$-foo value x y z)

and then one can import and lexically bind $setter$-foo.

Alternative, one could use compound symbols, as in Common Lisp and Kawa: setter:foo

(There are some complications relating to hygiene that might require datum->syntax and syntax-datum.)

[Marc Nieper-Wißkirchen]

Am Mo., 9. Aug. 2021 um 17:31 Uhr schrieb Per Bothner <p...@bothner.com>:

On 8/9/21 12:43 AM, Daphne Preston-Kendal wrote:
> On 8 Aug 2021, at 15:33, Marc Nieper-Wißkirchen <marc....@gmail.com> wrote:
>
>> A side remark on SRFI 17: Before anyone votes on this, don't forget to read the arguments of Matthias Felleisen on the SRFI 17 mailing list carefully. While the tone with which the discussions there were held may take some getting used to, the truth value of his words that SRFI 17 may be based on and promotes some deep misunderstanding of the role of `set!` is independent of that.
>
>
> I have taken a look at his posts, and his arguments seem to mostly be that SRFI 17 (in an early draft) was unclear about the mechanism, which apparently was resolved in later drafts and the final spec? Or can you point to another specific thread?

If I recall (and haven't re-read the thread) the argument was that 'set!' is about changing
bindings (variables) and should not be overloaded to change properties of objects.
(I disagree, parly because you can just as validly view a variable as a property of a
frame object.)

The frame object (if exposed) would still be a different type of entity than, say, the vector argument to vector-ref.  This argument just moves up the incompatibility between the two concepts one level.

 

Claiming this is a "deep misunderstanding" is elevating a preference to a fundamental
principle.  I think it is a distraction. The question is: does using SRFI-17 set! make
the code easier to read?  (I believe it does.) Does it lead more or fewer errors or
misunderstandings of the code? (I see no reason to believe so.)   Does the fact that most
other languages similarly overload assignment make it easier (or at least more familiar)
for new Scheme users?  (I don't know, but it seems plausible to me.)

Of course, one should give preference to fundamental principles.  If Scheme hadn't been designed with this idea in mind, it wouldn't be the language we know now and we like because it does a number of things better than the mainstream languages.

I haven't taught enough programming courses to be able to tell but at least the people on the SRFI 17 mailing list who had done so claimed that it is exactly this ad-hoc overloading of two different concepts that caused regularly misunderstandings.

If we want to improve the Scheme language in both regards, we should deprecate the ordinary `set!` and promote the use of boxes instead (à la ML).  Then it makes sense to reuse `set!` for SRFI 17's idea.  Everything would be uniform.  Setting a variable would become (set! (unbox x) 42).

It might be preferable if set! bindings were lexical bindings that could be imported.
Common Lisp supports that, in the sense that (setf FOO) is a pseudo-symbol in certain contexts.
One could define
     (set! (foo x y z) value)
as
     ($setter$-foo value x y z)

and then one can import and lexically bind $setter$-foo.

Alternative, one could use compound symbols, as in Common Lisp and Kawa: setter:foo

(There are some complications relating to hygiene that might require datum->syntax and syntax-datum.)

That seems to solve at least the composability problem of the SRFI 17 proposal, Daphne pointed out.  But it cannot work this way because `foo` could be imported under an alternative name and so the constructed name `$setter$-...` would be wrong.

You can get something satisfactory (modulo Felleisen's objections) with identifier properties (SRFI 213):

set! would then be redefined as:

(define-syntax set!

  (lambda (stx)

    (syntax-case stx ()

      ((_ x e) (identifier? #'x) #'(scheme:set! x e))

      ((_ (proc arg ...) e)

       (identifier? #'proc)

       (lambda (lookup)

         (let ([setter-id (lookup #'secret-key #'proc)])

           (unless setter-id

             (syntax-violation 'set! "no setter associated with operator" stx #'proc))

           #'(setter-id arg ... e)))))))

Setters can then be defined with define-property and the secret key suitable packaged into a convenience macro.

The semantic difference between this and the SRFI 17 proposal is that the setter is looked up at expansion time, which is probably a good idea anyway.  And things like (set! (((lambda () vector-ref)) x i) v) wouldn't be supported, but this is probably also a good thing.

[Alex Shinn]

On Mon, Aug 9, 2021 at 8:57 PM Marc Nieper-Wißkirchen
<marc....@gmail.com> wrote:
>
>> It would help Chibi remove one of it's biggest warts, in the hack that
>> allows syntax-case to co-exist with ER macros.
>> And it would help implementations that wouldn't consider adding such a
>> hack in the first place.
>
> It's not true that any hack is needed for the coexistence of ER macros with transformer procedures. It's enough to make sure that evaluating the er-macro-transformer procedure returns an object of a type that is disjoint to (ordinary) procedures. This is what Unsyntax does, for example. I believe that that's a straightforward and very clean solution.
>
> The case with Chibi is special because you chose to use procedure values also for the underlying implementation. It's that choice (that wouldn't have been necessary and that is not exposed to the user through er/sc/rsc-macro-transformer) that has forced me to hack around it to add syntax-case.

All native ER (and synclo) implementations represent macro
transformers as procedures. Unsyntax is the first I've heard of a
system which does otherwise.

> It would be different if Chibi's (lambda (form use-env mac-env) ...) were part of just another macro facility to be standardized. If that were the case, then we would have some incompatibility with R6RS.
>
>> The alienation for R6RS users is minimal. If you want you can even
>> define a define-syntax that implicitly wraps its argument with
>> syntax-transformer. No such trivial user-space workaround exists in
>> the other direction.
>
> Can you explain what you exactly mean by your last sentence?

Sorry, the incompatibility exists at the implementation level, not the
user level. The user has no workaround only in the sense that the
implementation is unlikely to hack their internals to support
syntax-case to begin with.

Note also in most implementations the underlying macro transformer
signature is leaked anyway, and we can't stop some people from taking
advantage of this (e.g. in third-party implementations of
ir-macro-transformer for Chibi). So there would be breakage there as
well, but that's OK - I don't consider (lambda (form use-env mac-env)
...) a guarantee anyway, and would consider adding more parameters for
more context (e.g. expansion history, expected number of returned
values, etc).

--
Alex

[Marc Nieper-Wißkirchen]

Am Di., 10. Aug. 2021 um 07:38 Uhr schrieb Alex Shinn <alex...@gmail.com>:

On Mon, Aug 9, 2021 at 8:57 PM Marc Nieper-Wißkirchen
<marc....@gmail.com> wrote:
>
>> It would help Chibi remove one of it's biggest warts, in the hack that
>> allows syntax-case to co-exist with ER macros.
>> And it would help implementations that wouldn't consider adding such a
>> hack in the first place.
>
> It's not true that any hack is needed for the coexistence of ER macros with transformer procedures.  It's enough to make sure that evaluating the er-macro-transformer procedure returns an object of a type that is disjoint to (ordinary) procedures.  This is what Unsyntax does, for example.  I believe that that's a straightforward and very clean solution.
>
> The case with Chibi is special because you chose to use procedure values also for the underlying implementation.  It's that choice (that wouldn't have been necessary and that is not exposed to the user through er/sc/rsc-macro-transformer) that has forced me to hack around it to add syntax-case.

All native ER (and synclo) implementations represent macro
transformers as procedures.  Unsyntax is the first I've heard of a
system which does otherwise.

It would be trivial to swap the internal representation so this seems to be more like an "agent's war".

And it should be noted that procedures as transformers were already in the 1991 appendix of the R4RS.  The definitions of the ER and SC macro facilities of the same year (Clinger, Hanson) are fully compatible with R4RS because they do not mandate nor expose any implementation as procedures.

It was a conscious choice by later implementations to add an incompatibility with R4RS and the later syntax-case system.  Anyway, it should be simple for every implementation to revert that unfortunate decision.

As for Unsyntax's implementation strategy: To support variable transformers, one needs more than one kind of transformer expression anyway.

 

> It would be different if Chibi's (lambda (form use-env mac-env) ...) were part of just another macro facility to be standardized.  If that were the case, then we would have some incompatibility with R6RS.
>
>> The alienation for R6RS users is minimal.  If you want you can even
>> define a define-syntax that implicitly wraps its argument with
>> syntax-transformer.  No such trivial user-space workaround exists in
>> the other direction.
>
> Can you explain what you exactly mean by your last sentence?

Sorry, the incompatibility exists at the implementation level, not the
user level.  The user has no workaround only in the sense that the
implementation is unlikely to hack their internals to support
syntax-case to begin with.

If we ever want to move forward with Scheme, some implementations (if they want to support a newer standard) will always have to adapt their internals.  Actually, I don't think that the issue we are debating here is very relevant in this regard because this "hack" would be not more than an afternoon's work.  The real dealbreakers are things like properly implemented ephemerons, some particular version of keyword arguments, multi-threading, truly immutable pairs and vectors, etc.  (I am not saying that these will necessarily all be part of a future Scheme standard but these are things that are hard to implement in existing implementations.)

 

Note also in most implementations the underlying macro transformer
signature is leaked anyway, and we can't stop some people from taking
advantage of this (e.g. in third-party implementations of
ir-macro-transformer for Chibi).  So there would be breakage there as
well, but that's OK - I don't consider (lambda (form use-env mac-env)
...) a guarantee anyway, and would consider adding more parameters for
more context (e.g. expansion history, expected number of returned
values, etc).

That's why I pointed out the library system.  The (chibi) library does not have to export the same bindings as the (scheme base) library under equal names (it already doesn't).

Anyway, the above is IMO bikeshedding as long as the following questions haven't been resolved satisfactorily for ER macros:

- How to implement unhygienic macros properly?

- How to interpret raw symbols in the ER macro output? (This question is related to the first.)

- How to implement modularity for transformer code when the RENAME argument always closes over the transformer environment and not where it is used?

- How to allow for writing wrappers to er-macro-transformer that get rid of all the invocations of RENAME while not destroying the efficiency?

Marc

[Alex Shinn]

On Tue, Aug 10, 2021 at 3:46 PM Marc Nieper-Wißkirchen
<marc....@gmail.com> wrote:
>
> And it should be noted that procedures as transformers were already in the 1991 appendix of the R4RS. The definitions of the ER and SC macro facilities of the same year (Clinger, Hanson) are fully compatible with R4RS because they do not mandate nor expose any implementation as procedures.

Syntactic-closures were introduced by Bawden and Rees in 1988,
predating the R4RS, and exposed the transformers as procedures
directly.

The later revision by Hanson presumably introduced the wrapper layer
to distinguish between normal and reverse syntactic-closures, though
the internal implementation was and still is incompatible with R4RS.

> It was a conscious choice by later implementations to add an incompatibility with R4RS and the later syntax-case system. Anyway, it should be simple for every implementation to revert that unfortunate decision.

Yes, we can just tell the implementors to change their system
internals, but we want to tread lightly there. Showing too much
disregard for existing implementations is a good way to ensure they
don't even try to adopt syntax-case. On the other hand, there may be
a slim chance of adoption in those cases anyway.

--
Alex

[Marc Nieper-Wißkirchen]

Am Di., 10. Aug. 2021 um 11:16 Uhr schrieb Alex Shinn <alex...@gmail.com>:

On Tue, Aug 10, 2021 at 3:46 PM Marc Nieper-Wißkirchen
<marc....@gmail.com> wrote:
>
> And it should be noted that procedures as transformers were already in the 1991 appendix of the R4RS.  The definitions of the ER and SC macro facilities of the same year (Clinger, Hanson) are fully compatible with R4RS because they do not mandate nor expose any implementation as procedures.

Syntactic-closures were introduced by Bawden and Rees in 1988,
predating the R4RS, and exposed the transformers as procedures
directly.

There's no mention of define-syntax in that paper so the relationship of their expanders with respect to R4RS+ define-syntax is not even defined.  That's why I didn't cite this paper as it seems irrelevant (not for the development of hygienic macro facilities but for the question at hand).

 

The later revision by Hanson presumably introduced the wrapper layer
to distinguish between normal and reverse syntactic-closures, though
the internal implementation was and still is incompatible with R4RS.

As syntactic-closures, as described in that paper, are not powerful enough to implement R4RS's low-level system, it is irrelevant that the *implementation* is not compatible.

 

> It was a conscious choice by later implementations to add an incompatibility with R4RS and the later syntax-case system.  Anyway, it should be simple for every implementation to revert that unfortunate decision.

Yes, we can just tell the implementors to change their system
internals, but we want to tread lightly there.  Showing too much
disregard for existing implementations is a good way to ensure they
don't even try to adopt syntax-case.  On the other hand, there may be
a slim chance of adoption in those cases anyway.

The hard things to add (if not already present) are things like `syntax`, `datum->syntax` and a proper implementation of `bound-identifiers=?`.  These I would see as dealbreakers if implementers are unwilling to adapt (no pun intended) their expanders.  Getting the external interface to R4RS/syntax-case right is flyspeck compared to it.

Introducing a cludge like `syntax-transformer` would effectively not make life easier for implementers, but would just force portable programs to make use of this cludge at least until a new standard is created.  That would be bad language design IMO.

We should also keep in mind that we are talking about the large language here.  We cannot expect all R7RS-small implementations to adopt R7RS-large.  Nor would it make sense.

Anyway, as long as the four much more fundamental issues with ER aren't resolved satisfactorily, it isn't an alternative IMO, at least not for a language standard that is intended to survive.

Basically, we are asking two different questions and the discussion in this thread is complicated because different questions are discussed at different times:

(1) What do implementers want?

(2) What do users want?

The obvious answer to question (1) is that an implementer wants to see the system that is already implemented in their implementation.  Whatever we decide, we will necessarily disappoint one-half of the implementer's camp.  And when we decide to include a version of ER that resolves the issues I posted in my previous email, we will disappoint implementers of the ER camp who will have a hard time getting it right with their implementations.  But when we add features to resolve the issues, the resulting system is already syntax-case under the hood, so implementers going this would have no problem with syntax-case anymore.  So the realistic alternative for (1) is: A simple but broken version of ER or a R4RS/syntax-case system.

As for (2):  If users could choose, they would very likely choose IR over ER if both were provided.  But for the user, IR looks very much the same as syntax-case (modulo some "#'"), so for the casual macro writer, it wouldn't make much of a difference.  But syntax-case doesn't have the O(n^2) problem of IR, so if all I had to take care of was pleasing my user, I would deliver syntax-case to them.  (In fact, syntax-case is even either to use than IR because one doesn't have to use unquote thanks to pattern variables, but that is orthogonal because it is related to the matcher frontend.)

Marc

[Marc Nieper-Wißkirchen]

Am Di., 10. Aug. 2021 um 11:55 Uhr schrieb Marc Nieper-Wißkirchen <marc....@gmail.com>:

 

The hard things to add (if not already present) are things like `syntax`, `datum->syntax` and a proper implementation of `bound-identifiers=?`.  These I would see as dealbreakers if implementers are unwilling to adapt (no pun intended) their expanders.  Getting the external interface to R4RS/syntax-case right is flyspeck compared to it.

PS: Ignore the "no pun intended".  This was a left-over after I had edited my text before posting it.

[John Cowan]

On Mon, Aug 9, 2021 at 2:04 AM Marc Nieper-Wißkirchen <marc....@gmail.com> wrote:

I think the CL / SRFI 17 format (set! (car x) y) as equivalent to (set-car! x y) is actually very nice, and more important than (set! car-x y), which must be defined for each x.

As for the former, read Felleisen.

I have now read the whole email archive for SRFI 17 and found it extremely unedifying (as did Bothner by the end), but I have to agree with <https://srfi-email.schemers.org/srfi-17/msg/2778585/>,  in which he asks ironically "Do they teach condescension at Rice?"  I have observed that there is a tendency for professors to divide the world into other professors and students, even outside the classroom.

At any rate, when the debate gets focused, it is about "Shall set! refer to both variable binding and data structure mutation, on the grounds that variable binding *is* the mutation of a behind-the-scenes data structure, or shall set! be reserved for variable binding and something else be used for generic data structure mutation?"  Each basically appeals to intuition, and since their intuitions contradict one another, that is no help.

As for the latter: Why would one want to do it?

I have no clue.  Why does two-clause identifier-syntax support it?

[Marc Nieper-Wißkirchen]

Am Di., 10. Aug. 2021 um 20:43 Uhr schrieb John Cowan <co...@ccil.org>:

On Mon, Aug 9, 2021 at 2:04 AM Marc Nieper-Wißkirchen <marc....@gmail.com> wrote:

I think the CL / SRFI 17 format (set! (car x) y) as equivalent to (set-car! x y) is actually very nice, and more important than (set! car-x y), which must be defined for each x.

As for the former, read Felleisen.

I have now read the whole email archive for SRFI 17 and found it extremely unedifying (as did Bothner by the end), but I have to agree with <https://srfi-email.schemers.org/srfi-17/msg/2778585/>,  in which he asks ironically "Do they teach condescension at Rice?"  I have observed that there is a tendency for professors to divide the world into other professors and students, even outside the classroom.

As I said, the tone of that discussion takes some getting used to, but the validity of arguments (by either side) is independent of the tone.

In any case, I hope I don't divide the world in that way.

 

At any rate, when the debate gets focused, it is about "Shall set! refer to both variable binding and data structure mutation, on the grounds that variable binding *is* the mutation of a behind-the-scenes data structure, or shall set! be reserved for variable binding and something else be used for generic data structure mutation?"  Each basically appeals to intuition, and since their intuitions contradict one another, that is no help.

But the latter also appeals to the theory, so there is at least help from that dimension.  (Besides, the argument that variable binding *is* mutation of a behind-the-scenes data structure is a killer argument.  Every evaluation step when realized on a CPU mutates the (hardware) store.)

As for the latter: Why would one want to do it?

I have no clue. 

Unless a clear advantage of SRFI 17-style setters surfaces (more than that it looks neat or that it mimics other language's features or misfeatures), I don't see a reason to ignore the theory and the protest of many Schemers on that mailing list.

Why does two-clause identifier-syntax support it?

It doesn't support the setters of SRFI 17.

What it does support is something like the following (simplified to one clause of just one form):

(define-syntax & (syntax-rules ()))

(define-syntax let
  (lambda (stx)
    (syntax-case stx (vector-ref &)
      [(_ ((p & (vector-ref x i))) e1 e2 ...)
       (identifier? #'p)
       #'(scheme:let ([y x] [j i])
           (let-syntax ([p (identifier-syntax
                            [_ (vector-ref y j)]
                            [(set! _ v) (vector-set! y j v)])])
             e1 e2 ...))])))

;; Example:

(let ([p & (vector-ref x i)])
  p           ; => retrieve the vector element
  (set! p 3)) ; => set the vector element
 

And this is a behavior that follows the theory again.

[John Cowan]

On Tue, Aug 10, 2021 at 3:17 PM Marc Nieper-Wißkirchen each x.

 

It doesn't support the setters of SRFI 17.

Right.  The getter-setter relationship is between procedure objects; the `setter` function extracts the setter of a getter, both procedures.

 

What it does support is something like the following (simplified to one clause of just one form):

I now understand what a two-clause identifier-syntax macro is: it's a macro (and therefore second-class) emulation of Zetalisp locatives (see <https://hanshuebner.github.io/lmman/fd-loc.xml>).  This is not as simple to implement as Zetalisp pointer locatives, but it means the GC doesn't have to deal with pointers into the interior of a data structure.  A locative is just a pointer, and a pointer can always be emulated using a getter-setter pair.

That is exactly what SRFI 17 does.  (set! (car x) 32) does not expand to (set-car! x 32), although it is equivalent to it; it expands to ((setter car) x 32), where the procedure `setter` does a run-time lookup on the car procedure object and returns the set-car! procedure object.  This has a perfectly good theory, of course, since it is just function evaluation with a little syntactic sugar.

But the meat of the debate is whether two different macro keywords are to be provided or just one.  To which I say, "Who cares?"

[Per Bothner]

On 8/10/21 3:05 PM, John Cowan wrote:
> That is exactly what SRFI 17 does.  (set! (car x) 32) does not expand to (set-car! x 32), although it is equivalent to it; it expands to ((setter car) x 32), where the procedure `setter` does a run-time lookup on the car procedure object and returns the set-car! procedure object.

Conceptually. Of course Kawa optimizes this to a compile-time lookup,
and then set-car! is optimized.

So (when the argument to car is known to be a pair):

(define (f xx::pair y)
(set! (car xx) y))

becomes:

Method name:"f" public static Signature: (gnu.lists.Pair,java.lang.Object)void
Attribute "Code", length:62, max_stack:2, max_locals:2, code_length:6
0: aload_0
1: aload_1
2: invokestatic <Method kawa.lib.lists.setCar$Ex (gnu.lists.Pair,java.lang.Object)void>
5: return

[Alex Shinn]

On Tue, Aug 10, 2021 at 6:55 PM Marc Nieper-Wißkirchen

<marc....@gmail.com> wrote:
>
> Am Di., 10. Aug. 2021 um 11:16 Uhr schrieb Alex Shinn <alex...@gmail.com>:
>>
>> On Tue, Aug 10, 2021 at 3:46 PM Marc Nieper-Wißkirchen
>> <marc....@gmail.com> wrote:
>> >
>> > And it should be noted that procedures as transformers were already in the 1991 appendix of the R4RS. The definitions of the ER and SC macro facilities of the same year (Clinger, Hanson) are fully compatible with R4RS because they do not mandate nor expose any implementation as procedures.
>>
>> Syntactic-closures were introduced by Bawden and Rees in 1988,
>> predating the R4RS, and exposed the transformers as procedures
>> directly.
>
> There's no mention of define-syntax in that paper so the relationship of their expanders with respect to R4RS+ define-syntax is not even defined. That's why I didn't cite this paper as it seems irrelevant (not for the development of hygienic macro facilities but for the question at hand).

[We're getting off-topic so should probably carry this to a separate
thread, but I want to address this misinformation.]

You seem to be making the argument that the R6RS macro transformer
signature was the first and only signature, and that any alternative
approaches were willfully different and now is the time for them to
get back in line with R6RS. You're basing this on the fact that the
low-level macros described in the appendix to the R4RS shared the same
signature. However, that was just used as an example in R4RS, which
was very careful to note that it "is but one of several low-level
facilities that have been designed," explicitly cites Bawden and Rees'
work on syntactic-closures as an earlier influence, and states "The
macro committee has not endorsed any particular low-level facility."
That example low-level system was notably removed from the R5RS.

Even if the R6RS-style signature were "first," the disregard for
existing alternatives and unwillingness to even consider a compromise
is the sort of "my way or the highway" attitude that was so
off-putting about R6RS in the first place.

Agreed, not every impl will support all of R7RS-large, and maybe they
are unlikely to support syntax-case anyway. Note though, that some
impls may accept small patches to make third-party syntax-case support
possible. In this case, adding things like preserving the renamer in
wrapped syntax (necessary for `datum->syntax`) might be accepted in
the core, but changing the macro signature is much less likely (and
would be dangerous, touch a lot of code, and possibly break things for
users). So from a practical perspective, you have the difficulties
reversed: changing the signature is the hard part.

--
Alex

[Marc Nieper-Wißkirchen]

Am Mi., 11. Aug. 2021 um 06:53 Uhr schrieb Alex Shinn <alex...@gmail.com>:

On Tue, Aug 10, 2021 at 6:55 PM Marc Nieper-Wißkirchen
<marc....@gmail.com> wrote:
>
> Am Di., 10. Aug. 2021 um 11:16 Uhr schrieb Alex Shinn <alex...@gmail.com>:
>>
>> On Tue, Aug 10, 2021 at 3:46 PM Marc Nieper-Wißkirchen
>> <marc....@gmail.com> wrote:
>> >
>> > And it should be noted that procedures as transformers were already in the 1991 appendix of the R4RS.  The definitions of the ER and SC macro facilities of the same year (Clinger, Hanson) are fully compatible with R4RS because they do not mandate nor expose any implementation as procedures.
>>
>> Syntactic-closures were introduced by Bawden and Rees in 1988,
>> predating the R4RS, and exposed the transformers as procedures
>> directly.
>
> There's no mention of define-syntax in that paper so the relationship of their expanders with respect to R4RS+ define-syntax is not even defined.  That's why I didn't cite this paper as it seems irrelevant (not for the development of hygienic macro facilities but for the question at hand).

[We're getting off-topic so should probably carry this to a separate
thread, but I want to address this misinformation.]

You seem to be making the argument that the R6RS macro transformer
signature was the first and only signature, and that any alternative
approaches were willfully different and now is the time for them to
get back in line with R6RS.  You're basing this on the fact that the
low-level macros described in the appendix to the R4RS shared the same
signature.  However, that was just used as an example in R4RS, which
was very careful to note that it "is but one of several low-level
facilities that have been designed," explicitly cites Bawden and Rees'
work on syntactic-closures as an earlier influence, and states "The
macro committee has not endorsed any particular low-level facility."
That example low-level system was notably removed from the R5RS.

The only argument I am making is that

(define-sytnax foo

  (lambda (stx)

    ...))

has always been used for the R4RS low-level and for the later R6RS system *and* that there have always been public specifications of this particular idiom.  Already 1993, Dybvig/Hieb/Bruggeman published the full syntax-case system, building on the low-level system R4RS.

For the other macro facilities like SC or ER, there have also been publications, but the published APIs don't use the above idiom, so peaceful coexistence is possible.

And that's why I said that implementations who had chosen an internal representation incompatible with the syntax-case idiom had made a conscious choice of ignoring that system.

 

Even if the R6RS-style signature were "first," the disregard for
existing alternatives and unwillingness to even consider a compromise
is the sort of "my way or the highway" attitude that was so
off-putting about R6RS in the first place.

One could just turn around the argument: Why insist on the

(define-syntax foo

  (XXX-transformer

    ...))

for all macros?

I also believe that this argument is irrelevant as the surface syntax of ER/SC/IR/RSC and syntax-case is already disjoint and has always been, so there is no need to bend any of these systems, especially when it comes to what the user faces.

 

Agreed, not every impl will support all of R7RS-large, and maybe they
are unlikely to support syntax-case anyway.  Note though, that some
impls may accept small patches to make third-party syntax-case support
possible.  In this case, adding things like preserving the renamer in
wrapped syntax (necessary for `datum->syntax`) might be accepted in
the core, but changing the macro signature is much less likely (and
would be dangerous, touch a lot of code, and possibly break things for
users).  So from a practical perspective, you have the difficulties
reversed: changing the signature is the hard part.

I challenge these claims.  Changing define-syntax so that it works with transformers of the form (lambda (stx) ...) was IMO by far the simplest part when I added these things to Chibi.  In fact, it doesn't even have to be done in the core (Chibi's implementation is again an example of this).  I feel that it is more a political than a technical thing.  And the underlying question really is:  Should R7RS-large try to accommodate as many existing native extensions as possible or would it expect from implementations that have native features not compatible with the standard that they provide a means to the user to choose between the native and the standard behavior.  The GCC is an example of the latter.  R6RS is an example of the latter.  R7RS-small's development has been guided (correct if I am wrong) by the former idea.  Personally, I think that native and standard features, even if incompatible on the surface, can peacefully coexist (and can even be used together) because of the library system of R7RS (and possibly also through reader directives).  I mean, the whole of the R6RS standard could be implemented in Chibi without touching any program that does not import (rnrs (6)) or uses #!r6rs.

This unanswered question can also be illustrated with the question of keyword arguments.  Given that R7RS-large wants to maintain full compatibility with R7RS-small, we cannot declare "foo:" or ":foo" being keywords (supposing that these should be different from identifiers).  But we could say that "#:key" is a keyword.  Now this notation is already be used by some systems in a non-standard way.  One can either expect that these implementations provide a standard mode where "#:key" becomes a keyword in the R7RS-large sense (everything under the assumption that it gets keywords), or one can remove the option of "#:key" from what's on the table for R7RS-large.  But then it is becoming increasingly hard to provide new features that have already been implemented but in different ways in the standard.

In any case, the question of the exact macro signature is probably bikeshedding at this moment.  (And I think it only came up because of claims that syntax-case and ER cannot/can exist peacefully together.)  The real question is what procedural macro systems if any are going into R7RS-large.

And *if* a decision has been made of having both ER and syntax-case in R7RS-large, *then* it can make sense to discuss streamlining the surface API.  If we just get ER, the question above is irrelevant.  If we just get syntax-case, it should be as well.

Marc

PS When considering the question of ER vs syntax-case in R7RS-large, we should also consider the question of what we want users to be able to do with the macro system.  Take a look at the code of the Nanopass framework that uses procedural macros extensively (and which is a good example of syntactic abstraction): https://github.com/nanopass/nanopass-framework-scheme/tree/main/nanopass.  ER is simply not designed for writing macros on this scale.

[Dr. Arne Babenhauserheide]

Lassi Kortela <la...@lassi.io> writes:

>> The advantage of a standard is that all its parts can fit together and
>> it can be documented and understood in one go.

…

>> I see the strength of having a strong set of always
>> available libraries in uniform style.
>> You can more easily understand them all together, when your
>> assumptions
>> from other libraries just work in the other libraries.
>
> Strongly agreed, but RnRS is too rigorous a process for creating such
> libraries. We should start another design process for non-core stuff,
> more lightweight than RnRS and SRFI.

How do you want to get even more lightweight than SRFIs for a
design-process? The only thing I see that would be more lightweight is a
melpa or even emacswiki for Scheme. But that’s no design process.

> (The same problem exists in other languages communities; human
> ingenuity doesn't scale up to this level of rigor and scope at once.
> The slogan "batteries included" originated in Python,

Python has something that Scheme lacks: The Zen of Python aka

import this

It is a powerful unifier that helps people graft libraries that feel
pythonic.

I’ve tried to write one a few times, but I’m not yet happy with it.

One of my tries was:

Zen for Scheme

RR: Remove limitations to Reduce the number of features you need.
HA: Hygiene reduces Anxiety …
PP: but Practicality beats Purity.
RW: Recursion Wins.
FI: Freedom for the Implementation and from the Implementation.
WM: Use the Weakest Method that gets the job done.
Move to stronger methods as needed.
OM: Optimizability Matters.
Mind the small Systems! And the Large Systems!
CM: Community Matters: Join the Scheme you choose.
3P: 3 Pillars of improvement:
Experimentation, Implementation, Standardization.

> but some Python experts look at the standard library as the place
> where modules go to die, meaning useful development stops and better
> alternatives struggle to take over.
> <https://leancrew.com/all-this/2012/04/where-modules-go-to-die/>)

And for many modules, this is a good thing. It means that I can come
back after 3 years in other languages and most of my assumptions still
work.

All the infrastructure arround Python shuffled and changed so much that
it’s unclear what’s still working for me, but the standard library is a
reliable friend.

[Alex Shinn]

On Wed, Aug 11, 2021 at 3:35 PM Marc Nieper-Wißkirchen
<marc....@gmail.com> wrote:
>
>> Agreed, not every impl will support all of R7RS-large, and maybe they
>> are unlikely to support syntax-case anyway. Note though, that some
>> impls may accept small patches to make third-party syntax-case support
>> possible. In this case, adding things like preserving the renamer in
>> wrapped syntax (necessary for `datum->syntax`) might be accepted in
>> the core, but changing the macro signature is much less likely (and
>> would be dangerous, touch a lot of code, and possibly break things for
>> users). So from a practical perspective, you have the difficulties
>> reversed: changing the signature is the hard part.
>
> I challenge these claims. Changing define-syntax so that it works with transformers of the form (lambda (stx) ...) was IMO by far the simplest part when I added these things to Chibi.

[Actually, your change broke some third-party code, I fixed this
yesterday. Note the core define-syntax is exposed again so it won't
work directly with syntax-case.]

Challenge accepted. Consider Chicken, for example. It supports
R7RS-small, but is unlikely to adopt non-portable changes made by
R7RS-large. However, it would be possible to implement syntax-case as
a third-party egg. The only thing you could not do is make
(define-syntax (lambda (x) ...)) work - you would need a wrapper
and/or a redefinition of define-syntax as done in Chibi.

--
Alex

[Marc Nieper-Wißkirchen]

Am Mi., 11. Aug. 2021 um 00:05 Uhr schrieb John Cowan <co...@ccil.org>:

On Tue, Aug 10, 2021 at 3:17 PM Marc Nieper-Wißkirchen each x.

 

It doesn't support the setters of SRFI 17.

Right.  The getter-setter relationship is between procedure objects; the `setter` function extracts the setter of a getter, both procedures.

I was imprecise.  I meant that it doesn't support syntax like (set! (proc arg ...) ...).

 

 

What it does support is something like the following (simplified to one clause of just one form):

I now understand what a two-clause identifier-syntax macro is: it's a macro (and therefore second-class) emulation of Zetalisp locatives (see <https://hanshuebner.github.io/lmman/fd-loc.xml>).  This is not as simple to implement as Zetalisp pointer locatives, but it means the GC doesn't have to deal with pointers into the interior of a data structure.  A locative is just a pointer, and a pointer can always be emulated using a getter-setter pair.

Just recall that a variable reference is a particular type of syntax (as is a procedure call, etc.).  In this sense, a variable is a keyword that, when expanded, expands into something that, when evaluated, returns the value of the variable.  R6RS offers to create variable-like keywords.  A keyword evaluating into the current time would be a good example (for the feature, not for standardization).  (Of course, this feature can also be misused by creating variable-like keywords that do not behave like variables; but this is no different from the myriads of ways to misuse R5RS macros to make code incomprehensible.)

 

That is exactly what SRFI 17 does.  (set! (car x) 32) does not expand to (set-car! x 32), although it is equivalent to it; it expands to ((setter car) x 32), where the procedure `setter` does a run-time lookup on the car procedure object and returns the set-car! procedure object.  This has a perfectly good theory, of course, since it is just function evaluation with a little syntactic sugar.

Of course, this has a perfectly good theory.  It's just not the same theory that is underlying Scheme's `set!`.  So it should not be using the same syntax.

Michael Sperber's example in the SRFI 17 highlights the differences between `set!` and `vector-set!`, say, best IMO:  https://srfi-email.schemers.org/srfi-17/msg/2778611/.  Now if you take my example code of (let ((p & (vector-ref x i))) ...), you see that the set! in my code behaves as the set! we know from Scheme (and as demonstrated by Mike Sperber).

While Matthias Felleisen's tone could have been seen as detrimental if the idea was to get an argument across, with respect to the value of the theory aspect of CS, he was very right IMO.  While an argumentum ad verecundiam is vacuous in itself, one should think twice before ignoring the arguments of the many people coming from CS (and none of them are day flies).  Of course, their arguments may be wrong, but to prove this one has to indulge in the theory themselves.

But the meat of the debate is whether two different macro keywords are to be provided or just one.  To which I say, "Who cares?"

That question must have been meant ironically as you have read the posts on the SRFI 17 mailing list... ;)

A compromise on that question has also been made on the SRFI 17 mailing list.  Call it "set-location!" or whatever, just not "set!".

Anyway, the other relevant question about composability, for example, hasn't even be touched.  SRFI 17's interface has the same problems as the comparator registry we discussed at length.

[Marc Nieper-Wißkirchen]

Am Mi., 11. Aug. 2021 um 08:57 Uhr schrieb Alex Shinn <alex...@gmail.com>:

On Wed, Aug 11, 2021 at 3:35 PM Marc Nieper-Wißkirchen
<marc....@gmail.com> wrote:
>
>> Agreed, not every impl will support all of R7RS-large, and maybe they
>> are unlikely to support syntax-case anyway.  Note though, that some
>> impls may accept small patches to make third-party syntax-case support
>> possible.  In this case, adding things like preserving the renamer in
>> wrapped syntax (necessary for `datum->syntax`) might be accepted in
>> the core, but changing the macro signature is much less likely (and
>> would be dangerous, touch a lot of code, and possibly break things for
>> users).  So from a practical perspective, you have the difficulties
>> reversed: changing the signature is the hard part.
>
> I challenge these claims.  Changing define-syntax so that it works with transformers of the form (lambda (stx) ...) was IMO by far the simplest part when I added these things to Chibi.

[Actually, your change broke some third-party code, I fixed this
yesterday.  Note the core define-syntax is exposed again so it won't
work directly with syntax-case.]

But that code must have relied on the non-standard extensions of the macro facility.  I don't think that these should be exposed in (scheme base), because it is causing exactly these kinds of later incompatibilities.

IMHO, this is a step back for Chibi.  Should Chibi later wants to get R6RS syntax-case into its core (under the assumption that it gets voted into R7RS-large and under the assumption that Chibi wants to support a substantial part of it), nothing would have been won.  It is IMHO much better to separate the non-standard stuff into (chibi), for otherwise, the third-party code will just break again.

 

Challenge accepted.  Consider Chicken, for example.  It supports
R7RS-small, but is unlikely to adopt non-portable changes made by
R7RS-large.  However, it would be possible to implement syntax-case as
a third-party egg.  The only thing you could not do is make
(define-syntax (lambda (x) ...)) work - you would need a wrapper
and/or a redefinition of define-syntax as done in Chibi.

How would you implement syntax-case (including pattern variables and datum->syntax and the syntax form and bound-identifier=? and so that it works seamlessly with the module system) portably in Chicken?  That's a genuine question.  When I just look at the (chicken syntax) module, I don't see enough primitives to do so.

The third-party egg could, of course, just export the define-syntax wrapper you mentioned just as (chibi syntax-case) does it now.

[Lassi Kortela]

> How do you want to get even more lightweight than SRFIs for a
> design-process? The only thing I see that would be more lightweight is a
> melpa or even emacswiki for Scheme. But that’s no design process.

My proposal is:

- Start with a small group of schemers who mostly know each other.

- Gather one collection of libraries (implementation + texinfo spec).

- All libraries are collectively owned. Anyone can change anything at
any time.

- The group resolves to settle disagreements by talking, instead of
forking the libraries into Yet Another Collection (TM).

- Since the people know each other, there is social pressure against
changing too many things too quickly, or changing controversial things
without asking others.

- Once or twice a year, the group releases a stable version of the spec.
The releases happen on the same fixed date(s) every year. A release
contains whatever happens to be good enough for release at that point in
time. (Development would naturally slow down a month before release to
make it easier to do.) This is the OpenBSD release model, and it works
very well.

- Scheme programmers are free to keep using old stable versions of the
spec as long as they like. Volunteers can maintain the implementation of
those versions to fix bugs, but not to add features or change APIs. API
changes are only done to the latest in-development spec.

- Once the process is running well, add more schemers. The key is to add
people slowly enough that everyone still knows each other. That enables
social pressure to work as a conflict resolution mechanism. In groups
where people don't know each other, some kind of formal protocol is
needed to resolve disputes, and that would make the process too
heavyweight to cover as much ground as these libraries need to cover.

> Python has something that Scheme lacks: The Zen of Python aka
>
> import this
>
> It is a powerful unifier that helps people graft libraries that feel
> pythonic.
>
> I’ve tried to write one a few times, but I’m not yet happy with it.
>
> One of my tries was:
>
> Zen for Scheme
>
> RR: Remove limitations to Reduce the number of features you need.
> HA: Hygiene reduces Anxiety …
> PP: but Practicality beats Purity.
> RW: Recursion Wins.
> FI: Freedom for the Implementation and from the Implementation.
> WM: Use the Weakest Method that gets the job done.
> Move to stronger methods as needed.
> OM: Optimizability Matters.
> Mind the small Systems! And the Large Systems!
> CM: Community Matters: Join the Scheme you choose.
> 3P: 3 Pillars of improvement:
> Experimentation, Implementation, Standardization.

John has got you covered!
https://github.com/johnwcowan/this/blob/master/this.scm

However, Scheme has a more creative community than Python (I won't prove
this :p) so it's doubtful that schemers can codify one aesthetic as
easily as Python. Disagreement is an inherent part of the community
because almost everyone has some kind of personal vision.

> And for many modules, this is a good thing. It means that I can come
> back after 3 years in other languages and most of my assumptions still
> work.
>
> All the infrastructure arround Python shuffled and changed so much that
> it’s unclear what’s still working for me, but the standard library is a
> reliable friend.

Libraries can be versioned, as in the above proposal. For one reason or
another, Python is not doing that. R6RS has standard support for library
versions, and R7RS library names can contain numerical parts. We could:

(import (scheme libraries (2021))
(import (scheme libraries (2022))
etc.

This lets the community keep the old APIs around as long as there is
interest.

[Alex Shinn]

What's needed is a way to associate auxiliary data (the renamer) with
renamed identifiers. Storing these directly in an identifier object
is easiest, of course, but you can also use global (ideally key-weak)
hash-tables.

> The third-party egg could, of course, just export the define-syntax wrapper you mentioned just as (chibi syntax-case) does it now.

Yes, that's what I said.

--
Alex

[Marc Nieper-Wißkirchen]

Am Mi., 11. Aug. 2021 um 15:44 Uhr schrieb Alex Shinn <alex...@gmail.com>:

On Wed, Aug 11, 2021 at 4:53 PM Marc Nieper-Wißkirchen
<marc....@gmail.com> wrote:
>
>> Challenge accepted.  Consider Chicken, for example.  It supports
>> R7RS-small, but is unlikely to adopt non-portable changes made by
>> R7RS-large.  However, it would be possible to implement syntax-case as
>> a third-party egg.  The only thing you could not do is make
>> (define-syntax (lambda (x) ...)) work - you would need a wrapper
>> and/or a redefinition of define-syntax as done in Chibi.
>
> How would you implement syntax-case (including pattern variables and datum->syntax and the syntax form and bound-identifier=? and so that it works seamlessly with the module system) portably in Chicken?  That's a genuine question.  When I just look at the (chicken syntax) module, I don't see enough primitives to do so.

What's needed is a way to associate auxiliary data (the renamer) with
renamed identifiers.  Storing these directly in an identifier object
is easiest, of course, but you can also use global (ideally key-weak)
hash-tables.

For a truly faithful implementation of syntax-case, you need quite a bit more.  (And the code in Chibi is, for the same reasons a working approximation for a lot of use cases, but not a faithful implementation of SRFI 93/R6RS.)

A strict implementation has to handle the following case, for example:

(define-library (foo) (export foo) (import (scheme base)) (begin (define foo 'foo)))

(define-library (bar) (export bar) (import (scheme base)) (import (scheme syntax-case)) (import (foo))

  (define-syntax bar

    (lambda (stx)

      (datum->syntax #'* 'foo)))) ; (*)

;; Main program

(import (bar))

(bar) ; => 'foo

If one looks at the line marked with (*), an identifier has to be constructed whose binding is the binding of an identifier named foo as if it were introduced in the lexical scope of the expression #'*.  Now, in Chibi #'* is just a symbol (the value of syntax-quote in this example), so it doesn't know its lexical environment.  Thus the result of (datum->syntax #'* 'foo) cannot become something else than just the bare symbol foo, which then becomes part of the macro output.  Chibi's expander cannot know in which environment to close this bare symbol.  And the main program will fail because an identifier named foo is not bound in the lexical scope of the macro use.

My point here is not that someone shouldn't try to write such an approximative egg (as I did for Chibi), but that for standard compliance the core would have to be touched anyway, and much more seriously than just getting the surface define-syntax protocol right.

I hope that it now becomes clearer why I consider the talk of having syntax-transformer vs lambda (mostly) bikeshedding.  If syntax-case becomes part of R7RS-large, existing ER implementations will have to do some hard work with their expanders anyway so that it works flawlessly with the module system, etc.  IMHO, such work would pay off in any case.  (And it becomes much less work if an existing frontend like Unsyntax is used.)

By the way, the example shows that ER is less expressive than even the R4RS low-level system.

> The third-party egg could, of course, just export the define-syntax wrapper you mentioned just as (chibi syntax-case) does it now.

Yes, that's what I said.

Sorry, my formulation was a bit given to misunderstanding.  I meant that this would be a doable way under the assumption that Chicken does not want to change its (scheme base) for whatever reasons.

[Dr. Arne Babenhauserheide]

Lassi Kortela <la...@lassi.io> writes:

>> How do you want to get even more lightweight than SRFIs for a
>> design-process? The only thing I see that would be more lightweight is a
>> melpa or even emacswiki for Scheme. But that’s no design process.
>
> My proposal is:
> - Start with a small group of schemers who mostly know each other.
>
> - Gather one collection of libraries (implementation + texinfo spec).

You mean like SLIB?
http://people.csail.mit.edu/jaffer/SLIB.html

This process sounds like it could work, but it needs people who do the
work. The typical size of free software communities is 1, getting a
group of active people together is already quite a feat.

So while the process sounds like a good way forward, it misses the
hardest part: Having a group of active people work together on the
project.

If you see a reason why you’re not collaborating on SLIB, you know one
of the challenges.

>> Python has something that Scheme lacks: The Zen of Python aka
>> import this
>> It is a powerful unifier that helps people graft libraries that feel
>> pythonic.
>> I’ve tried to write one a few times, but I’m not yet happy with it.
>> One of my tries was:
>> Zen for Scheme
>> RR: Remove limitations to Reduce the number of features you need.
>> HA: Hygiene reduces Anxiety …
>> PP: but Practicality beats Purity.
>> RW: Recursion Wins.
>> FI: Freedom for the Implementation and from the Implementation.
>> WM: Use the Weakest Method that gets the job done.
>> Move to stronger methods as needed.
>> OM: Optimizability Matters.
>> Mind the small Systems! And the Large Systems!
>> CM: Community Matters: Join the Scheme you choose.
>> 3P: 3 Pillars of improvement:
>> Experimentation, Implementation, Standardization.
>
> John has got you covered!
> https://github.com/johnwcowan/this/blob/master/this.scm

This looks more like satire than like trying to represent the culture
within Scheme communities.

> However, Scheme has a more creative community than Python (I won't
> prove this :p) so it's doubtful that schemers can codify one aesthetic
> as easily as Python. Disagreement is an inherent part of the community
> because almost everyone has some kind of personal vision.

This is part of the aesthetic :-)

>> And for many modules, this is a good thing. It means that I can come
>> back after 3 years in other languages and most of my assumptions still
>> work.
>> All the infrastructure arround Python shuffled and changed so much
>> that
>> it’s unclear what’s still working for me, but the standard library is a
>> reliable friend.
>
> Libraries can be versioned, as in the above proposal. For one reason
> or another, Python is not doing that. R6RS has standard support for
> library versions, and R7RS library names can contain numerical parts.
> We could:
>
> (import (scheme libraries (2021))
> (import (scheme libraries (2022))
> etc.
>
> This lets the community keep the old APIs around as long as there is
> interest.

I do not think that this actually fills the need. You need to be able to
come back and still have everything you know work the same — but
interoperate with modern standards.

This requires careful API design so the APIs age well. They won’t be
perfect 10 years later, but they should still be ok. And Python is
legendary at that. Part of the reason for that (and also why Python will
be struggling with it much more in the future) is given in “Software
developers should avoid traumatic changes”:

“It requires everyone in the community, or nearly everyone, to overhaul
their code to get it idiomatic again”
- https://drewdevault.com/2019/11/26/Avoid-traumatic-changes

Python added changes to the core language which make a lot of existing
code un-idiomatic. That is a problem. It would be an even bigger
problem with versioned libraries.

[Lassi Kortela]

Thanks for the feedback -- thought-provoking comments!

> You mean like SLIB?
> http://people.csail.mit.edu/jaffer/SLIB.html

I asked Aubrey Jaffer whether he'd like to merge the idea with SLIB, but
our goals are so different that it didn't work out. SLIB is founded on
backward compatibility and follows a more typical release process. It's
also a GNU project, which presents its own obligations.

> This process sounds like it could work, but it needs people who do the
> work. The typical size of free software communities is 1, getting a
> group of active people together is already quite a feat.
>
> So while the process sounds like a good way forward, it misses the
> hardest part: Having a group of active people work together on the
> project.

Agreed. Like RnRS, it's all about agreement and trust.

>> https://github.com/johnwcowan/this/blob/master/this.scm
>
> This looks more like satire than like trying to represent the culture
> within Scheme communities.

Possibly; it's phrased directly after the Zen of Python, whereas your
version is more original.

>> Disagreement is an inherent part of the community
>> because almost everyone has some kind of personal vision.
>
> This is part of the aesthetic :-)

:)

>> Libraries can be versioned,

>
> I do not think that this actually fills the need. You need to be able to
> come back and still have everything you know work the same — but
> interoperate with modern standards.
>
> This requires careful API design so the APIs age well. They won’t be
> perfect 10 years later, but they should still be ok. And Python is
> legendary at that. Part of the reason for that (and also why Python will
> be struggling with it much more in the future) is given in “Software
> developers should avoid traumatic changes”:
>
> “It requires everyone in the community, or nearly everyone, to overhaul
> their code to get it idiomatic again”
> - https://drewdevault.com/2019/11/26/Avoid-traumatic-changes
>
> Python added changes to the core language which make a lot of existing
> code un-idiomatic. That is a problem. It would be an even bigger
> problem with versioned libraries.

That blog post is a good one; I hadn't read it before.

The blogger writes about Py 2 v3, as well as sync vs async I/O. Both of
those transitions are close to the core, as you say. In Scheme, similar
changes would go into RnRS, and indeed R6 produced a rift.

I think stuff like that should stay in RnRS, and RnRS should aim at the
kind of stability you praise. There should be a pretty clear division
between what is the core language, and what is just library stuff.

The libraries developed by a group, should not touch any core stuff.
They should be portable to all actively used RnRS editions, as far as
porting them is feasible. Here we can take more liberties with
refactoring, for example, an HTTP library, though the caution you advise
still pays. But upsetting users of an HTTP library (while it still
should be avoided) is far from being as catastrophic as something like a
fundamental renovation of the way I/O, macros, or continuations are done.

[Dr. Arne Babenhauserheide]

Lassi Kortela <la...@lassi.io> writes:

> Thanks for the feedback -- thought-provoking comments!
>
>> You mean like SLIB?
>> http://people.csail.mit.edu/jaffer/SLIB.html
>
> I asked Aubrey Jaffer whether he'd like to merge the idea with SLIB,
> but our goals are so different that it didn't work out. SLIB is
> founded on backward compatibility and follows a more typical release
> process. It's also a GNU project, which presents its own obligations.

In my opinion both are strengths :-)

>> This process sounds like it could work, but it needs people who do the
>> work. The typical size of free software communities is 1, getting a
>> group of active people together is already quite a feat.
>> So while the process sounds like a good way forward, it misses the
>> hardest part: Having a group of active people work together on the
>> project.
>
> Agreed. Like RnRS, it's all about agreement and trust.

That’s not the most essential part. The most essential part is to find
at least 3 active people who want to work together on that vision.

Requirements:

- active.
- work together.

And activity as challenge is not to be underestimated. I’ve seen someone
in the Freenet project singlehandedly realize Freenet on Smartphones
within less that 4 weeks. Just the amount of work someone puts in can
make a difference of an order of magnitude (and more) for
hobby-projects (or at least for not full-time paid projects).

> The blogger writes about Py 2 v3, as well as sync vs async I/O. Both
> of those transitions are close to the core, as you say. In Scheme,
> similar changes would go into RnRS, and indeed R6 produced a rift.

The only requirement to create such a traumatic change is affecting lots
of code. In Scheme every library has the power to do such deep changes.

Used SRFI-42 and it’s no longer en-vogue? Soft trauma.

That’s the flipside of power: We must be extremely careful how, whether,
and when to wield it.

> I think stuff like that should stay in RnRS, and RnRS should aim at
> the kind of stability you praise. There should be a pretty clear
> division between what is the core language, and what is just library
> stuff.
>
> The libraries developed by a group, should not touch any core stuff.
> They should be portable to all actively used RnRS editions, as far as
> porting them is feasible. Here we can take more liberties with
> refactoring, for example, an HTTP library, though the caution you
> advise still pays. But upsetting users of an HTTP library (while it
> still should be avoided) is far from being as catastrophic as
> something like a fundamental renovation of the way I/O, macros, or
> continuations are done.

It could even be more catastrophic, because many fundamental renovations
can be shimmed behind compatibility macros, but the library might be in
a heavily optimized critical path of applications. And if it is
successful (let’s assume that, otherwise this discussion is moot) it
will affect people on every Scheme and every edition.

I already abandoned several otherwise fully working projects, because I
do not have the time needed to port them to Python 3. Basically I cannot
maintain them anymore due to those changes. But most of the cost is not
in some fundamental language feature. Most of the cost is that library
functions now return bytes instead of strings, so all interfaces between
datatypes have to change.

Computing is full of barely maintained but fully functional tools and
libraries. As long as those are the tools people really use, they will
gain new required capabilities, because people will add what they really
need and send patches.

You’re active while you create them, but if you need to change them
later, you’re already active on some other project and expect to be able
to maintain what works at 1% of the effort.

But if you rely on some old version of a library and that version does
not support http2, then you lost. You suddenly have to invest much more
than expected just to keep your tool working with the changing
environment.

[Alex Shinn]

On Wed, Aug 11, 2021 at 11:48 PM Marc Nieper-Wißkirchen
<marc....@gmail.com> wrote:
>
> For a truly faithful implementation of syntax-case, you need quite a bit more. (And the code in Chibi is, for the same reasons a working approximation for a lot of use cases, but not a faithful implementation of SRFI 93/R6RS.)

My point was that one could implement Chibi-style syntax-case for
Chicken (and MIT, etc.).
If you argue that Chibi is not conformant, then we should be careful
to exclude mention of the existence of that from discussions of
syntax-case support.

But I'm not convinced of your example. I think it suffices to assume
if the syntax argument of datum->syntax is a bare symbol, then the
current lexical environment should be used (which is the normal
interpretation of ER macros to begin with).

Even if it were not possible without some internal changes, it can be
easier to get such changes accepted than to get the whole syntax-case
expander into the core. So this is _not_ just a matter of
bikeshedding.

What absolutely can't be implemented without intrusive internal
changes is identifier syntax, so we should be sure to address that
separately.

--
Alex

[Marc Nieper-Wißkirchen]

Am Do., 12. Aug. 2021 um 01:28 Uhr schrieb Alex Shinn <alex...@gmail.com>:

On Wed, Aug 11, 2021 at 11:48 PM Marc Nieper-Wißkirchen
<marc....@gmail.com> wrote:
>
> For a truly faithful implementation of syntax-case, you need quite a bit more.  (And the code in Chibi is, for the same reasons a working approximation for a lot of use cases, but not a faithful implementation of SRFI 93/R6RS.)

My point was that one could implement Chibi-style syntax-case for
Chicken (and MIT, etc.).

Well, that's fair enough.  But when it comes to R7RS-large, we would want the full syntax-case (if any) because would want to get the interaction with modules right as well.

 

If you argue that Chibi is not conformant, then we should be careful
to exclude mention of the existence of that from discussions of
syntax-case support.

When we want to talk about systems that would need no further action at all to become syntax-case compliant, then, yes, we have to exclude Chibi.

(We can talk about strategies how to get the missing part into Chibi somewhere else in case you are interested.)

 

But I'm not convinced of your example.  I think it suffices to assume
if the syntax argument of datum->syntax is a bare symbol, then the
current lexical environment should be used (which is the normal
interpretation of ER macros to begin with).

The syntax argument of `datum->syntax` is an identifier.  In my example, it was the value of #'*.  It just happens that Chibi *represents* this identifier as the symbol * (through its built-in syntax-quote).  What is the notion of a current lexical environment with respect to `datum->syntax` supposed to mean?  `datum->syntax` is a procedure that cannot grab it.

And if you mean by "current lexical environment" the environment where the macro is eventually used, then `datum->syntax` wouldn't do what it is supposed to do because it is the lexical environment of the value of #'* that it would have to use.  So unless there is a built-in form capable of catching its lexical environment, you cannot get a faithful implementation.

Another example, a faithful implementation has to get right is the following:

(let ((x 0))

  (let ((id1 (datum->syntax #'* 'name)))

    (let ((x 1))

      (let ((id2 (datum->syntax #'* 'name)))

        (list (eq? id1 id2) (bound-identifier=? id1 id2)))))) ; => (#f #t)

The two identifiers id1 and id2 are bound-identifier=? because binding one would bind the other.  But they can't be equivalent because (datum->syntax id1 'x) has a different binding than (datum->syntax id2 'x).

Even if it were not possible without some internal changes, it can be
easier to get such changes accepted than to get the whole syntax-case
expander into the core.

It is enough if these changes are visible in (scheme base).  What would be in (chibi), (chicken), (chezscheme), ... is for the implementation to decide.

But if implementations are opposed to syntax-case on any other grounds, then any technical arguments are lost anyway.

 

  So this is _not_ just a matter of
bikeshedding.

We should resume this point once there are actual cases of implementations that have amended their internals so that the syntax-case system is supported, in principle, but then have difficulties to support the define-syntax API of syntax-case.

What absolutely can't be implemented without intrusive internal
changes is identifier syntax, so we should be sure to address that
separately.

What do you mean by "intrusive"?  I am not a native speaker, but it sounds negatively connoted to me.

I agree with you that it has to be addressed separately because it is an orthogonal question to ER vs syntax-case.

It should be noted, though, that the changes necessary to get macro uses of the form "keyword" supported are a magnitude smaller than to get `syntax` and `datum->syntax` (and thus unhygienic macros) in an existing implementation right.  It took me literally 2 hours to implement "identifier syntax" in Chicken, and I hadn't even looked at its source code before.

This question is more about whether one wants it to be part of the language or not, independently of implementations.

I know that you have argued a lot against "identifier syntax", but your assertions have also been refuted, so I am not sure where you stand today.

[Alex Shinn]

On Thu, Aug 12, 2021 at 3:45 PM Marc Nieper-Wißkirchen
<marc....@gmail.com> wrote:
>
> I know that you have argued a lot against "identifier syntax", but your assertions have also been refuted, so I am not sure where you stand today.

This is the second time you've stated that something has been argued
before and I have been shown to be wrong, when I never agreed.

I don't think there's any point in me discussing further. You will
simply assert that I was wrong and the discussion is over.

--
Alex

[Lassi Kortela]

Arne, thank you for this discussion. You are making the most
thought-provoking comments on the library proposal so far.

>> Like RnRS, it's all about agreement and trust.

> That’s not the most essential part. The most essential part is to find
> at least 3 active people who want to work together on that vision.
>
> Requirements:
>
> - active.
> - work together.

People who agree and trust are motivated to work together actively.

> And activity as challenge is not to be underestimated. I’ve seen someone
> in the Freenet project singlehandedly realize Freenet on Smartphones
> within less that 4 weeks. Just the amount of work someone puts in can
> make a difference of an order of magnitude (and more) for
> hobby-projects (or at least for not full-time paid projects).

We can take technical excellence for granted. The average schemer is
very good. I surveyed 5-10 existing library collections, all impressive
work (as expected for Scheme) by one person (also expected for Scheme).

What we need to consciously work on is agreement.

> The only requirement to create such a traumatic change is affecting lots
> of code. In Scheme every library has the power to do such deep changes.
>
> Used SRFI-42 and it’s no longer en-vogue? Soft trauma.
>
> That’s the flipside of power: We must be extremely careful how, whether,
> and when to wield it.

The major imperative for applying this care is in RnRS. It wasn't
applied in R6 and it's not being applied in R7. Someone has to say it.

>> But upsetting users of an HTTP library (while it
>> still should be avoided) is far from being as catastrophic as
>> something like a fundamental renovation of the way I/O, macros, or
>> continuations are done.

> It could even be more catastrophic, because many fundamental renovations
> can be shimmed behind compatibility macros, but the library might be in
> a heavily optimized critical path of applications. And if it is
> successful (let’s assume that, otherwise this discussion is moot) it
> will affect people on every Scheme and every edition.

There's no basis for assuming that a language core inherently makes no
fundamental changes whereas libraries do.

It's also important to distinguish between interface and implementation.
We should make stable versions of the interface spec. If people want to
go back later and optimize an implementation, so much the better.

> I already abandoned several otherwise fully working projects, because I
> do not have the time needed to port them to Python 3. Basically I cannot
> maintain them anymore due to those changes. But most of the cost is not
> in some fundamental language feature. Most of the cost is that library
> functions now return bytes instead of strings, so all interfaces between
> datatypes have to change.

String vs bytevector interoperability is mainly RnRS territory.

> Computing is full of barely maintained but fully functional tools and
> libraries. As long as those are the tools people really use, they will
> gain new required capabilities, because people will add what they really
> need and send patches.

True, but your line of argumentation is putting us in an impossible
position:

- If we make breaking changes, we'll cause trouble.

- If we add features yet never make breaking changes, we'll accumulate a
pile of cruft that no-one dares clean up.

- If we do nothing, people have a scattered pile of old libraries tied
to a single implementation. These libraries also suffer from bit rot
(e.g. Chicken 4) and lack of maintenance.

Which proposal do you stand behind?

- Non-core libraries in SLIB

- Non-core libraries in R7RS-large

- My proposal

- Each schemer does their personal collection of non-core libs

- Each Scheme implementation has its own collection of non-core libs

All of these have been tried for years except R7-large and my proposal.
They give us the results we currently get.

> You’re active while you create them, but if you need to change them
> later, you’re already active on some other project and expect to be able
> to maintain what works at 1% of the effort.

Agreed.

> But if you rely on some old version of a library and that version does
> not support http2, then you lost. You suddenly have to invest much more
> than expected just to keep your tool working with the changing
> environment.

I don't understand that. If a new protocol supports the same API as the
old protocol, then support can be added without changing the API. The
old API can be kept working with the new protocol.

OTOH if the new protocol requires a new API then the application needs
to be upgraded to use a new API anyway. It's probably harder to switch
to a whole new library than the newest version of the old library.

[Dr. Arne Babenhauserheide]

Lassi Kortela <la...@lassi.io> writes:

> Arne, thank you for this discussion. You are making the most
> thought-provoking comments on the library proposal so far.
>
>>> Like RnRS, it's all about agreement and trust.
>
>> That’s not the most essential part. The most essential part is to find
>> at least 3 active people who want to work together on that vision.
>> Requirements:
>> - active.
>> - work together.
>
> People who agree and trust are motivated to work together actively.

I’ve seen too many projects fail due to lack of time to believe the
"actively" part of this — you either need to secure longterm funding to
people can do this as work, or you need to be really good at motivating.

>> And activity as challenge is not to be underestimated. I’ve seen someone
>> in the Freenet project singlehandedly realize Freenet on Smartphones
>> within less that 4 weeks. Just the amount of work someone puts in can
>> make a difference of an order of magnitude (and more) for
>> hobby-projects (or at least for not full-time paid projects).
>
> We can take technical excellence for granted. The average schemer is
> very good. I surveyed 5-10 existing library collections, all
> impressive work (as expected for Scheme) by one person (also expected
> for Scheme).

It’s not technical excellence that’s the most limiting aspect, but raw free time.

> What we need to consciously work on is agreement.



>> The only requirement to create such a traumatic change is affecting lots
>> of code. In Scheme every library has the power to do such deep changes.
>> Used SRFI-42 and it’s no longer en-vogue? Soft trauma.
>> That’s the flipside of power: We must be extremely careful how,
>> whether,
>> and when to wield it.
>
> The major imperative for applying this care is in RnRS. It wasn't
> applied in R6 and it's not being applied in R7. Someone has to say it.
>
>>> But upsetting users of an HTTP library (while it
>>> still should be avoided) is far from being as catastrophic as
>>> something like a fundamental renovation of the way I/O, macros, or
>>> continuations are done.
>
>> It could even be more catastrophic, because many fundamental renovations
>> can be shimmed behind compatibility macros, but the library might be in
>> a heavily optimized critical path of applications. And if it is
>> successful (let’s assume that, otherwise this discussion is moot) it
>> will affect people on every Scheme and every edition.
>
> There's no basis for assuming that a language core inherently makes no
> fundamental changes whereas libraries do.

note the "might" :-)

I agree with you here. It’s the same the other way roud. And both hurt.

> It's also important to distinguish between interface and
> implementation. We should make stable versions of the interface spec.
> If people want to go back later and optimize an implementation, so
> much the better.

If you can manage to separate the interface sufficiently well from the
implementation, that an old library can support modern features both in
an old interface and in a new interface, then you might be able to get
the best of both worlds.

But that makes the challenge harder.

>> I already abandoned several otherwise fully working projects, because I
>> do not have the time needed to port them to Python 3. Basically I cannot
>> maintain them anymore due to those changes. But most of the cost is not
>> in some fundamental language feature. Most of the cost is that library
>> functions now return bytes instead of strings, so all interfaces between
>> datatypes have to change.
>
> String vs bytevector interoperability is mainly RnRS territory.

Changed return value of library functions is what really hurts.

>> Computing is full of barely maintained but fully functional tools and
>> libraries. As long as those are the tools people really use, they will
>> gain new required capabilities, because people will add what they really
>> need and send patches.
>
> True, but your line of argumentation is putting us in an impossible
> position:
>
> - If we make breaking changes, we'll cause trouble.
>
> - If we add features yet never make breaking changes, we'll accumulate
> a pile of cruft that no-one dares clean up.
>
> - If we do nothing, people have a scattered pile of old libraries tied
> to a single implementation. These libraries also suffer from bit rot
> (e.g. Chicken 4) and lack of maintenance.

Yes. That is the position. But all three answers are extreme viewpoints.

The article Volatile Software
<http://stevelosh.com/blog/2012/04/volatile-software/>
points to a solution:

- Always provide the shim that makes sure that people do not suffer from
the change.
- You’re allowed breaking changes before 1.0.

⇒ have as stable part and a non-stable part. Libraries that never
stabilize can be discarded after some time.

> Which proposal do you stand behind?
>
> - Non-core libraries in SLIB
>
> - Non-core libraries in R7RS-large
>
> - My proposal

Firstoff: The one where people actually do it.

That’s the most important requirement. And the hardest.

After that there are two answers:

- What I would use: One with stability guarantees. I need my code to
still work in 10 years.
- What I might contribute to: One that tackles problems I’m deeply
interested in.

> - Each schemer does their personal collection of non-core libs

This would be worst, because it prevents newcomers from starting up quickly.

> - Each Scheme implementation has its own collection of non-core libs

That’s the "choose your tribe" approach.

> All of these have been tried for years except R7-large and my
> proposal. They give us the results we currently get.

There’s also SRFIs which actually work pretty well.

>> But if you rely on some old version of a library and that version does
>> not support http2, then you lost. You suddenly have to invest much more
>> than expected just to keep your tool working with the changing
>> environment.
>
> I don't understand that. If a new protocol supports the same API as
> the old protocol, then support can be added without changing the API.
> The old API can be kept working with the new protocol.

Yes, but someone has to do the work to keep it working in the API.

> OTOH if the new protocol requires a new API then the application needs
> to be upgraded to use a new API anyway.

Here’s the question then, whether this requires touching any other part.
If it does, then you have a problem again.

This means you cannot just change other APIs to be uniform with the new API.

> It's probably harder to switch
> to a whole new library than the newest version of the old library.

If you only need a few features from your existing tool, it can actually
be easier to just recreate that part. And push the problem down on all
your users who needed another part, so the problem multiplies.

Keep the 1% in mind — and add to that significant time delay ("not now,
I’m in this other project right now, I’ll do this when I’m free" — and
this is how porting from python 2 to python 3 can take more than a
decade and result in discarding working tools).

[Lassi Kortela]

Great discussion. Things are looking sunnier!

>> People who agree and trust are motivated to work together actively.
>
> I’ve seen too many projects fail due to lack of time to believe the
> "actively" part of this — you either need to secure longterm funding to
> people can do this as work, or you need to be really good at motivating.

> It’s not technical excellence that’s the most limiting aspect, but raw free time.

I concede you're right. We just need to try something and go on faith.

Funding for Scheme is 5 years away even in ideal circumstances.

> If you can manage to separate the interface sufficiently well from the
> implementation, that an old library can support modern features both in
> an old interface and in a new interface, then you might be able to get
> the best of both worlds.
>
> But that makes the challenge harder.

Now we're talking! Enthusiastically agreed :)

IMHO the best prescription is:

- Avoid change in RnRS. Have an extreme preference for keeping old
interfaces around.

- Occasionally change old interfaces in non-core libraries, but try to
avoid it as much as possible, and use versioning so the old interfaces
keep working. Try to allow people to mix current year's version of
Library A with prior year's version of Library B as much as possible, by
keeping data types compatible even where procedures change.

>> String vs bytevector interoperability is mainly RnRS territory.
>
> Changed return value of library functions is what really hurts.

Good point. Strings have been one of the pain points in R7-large design,
and Python 2 vs 3 demonstrates that we struggle for good reason.

Thanks to your comments, I now realize that changing data is much more
disruptive than changing procedures. It's easy to pass new data to old
APIs if the types stay compatible.

>> line of argumentation is putting us in an impossible position:
>

> Yes. That is the position. But all three answers are extreme viewpoints.

Yes. That was the point I was trying to make :)

We arrive at an API evolution maxim like the CAP theorem:

"Coverage, cleanliness, stability: pick any two."

> The article Volatile Software
> <http://stevelosh.com/blog/2012/04/volatile-software/>
> points to a solution:
>
> - Always provide the shim that makes sure that people do not suffer from
> the change.
> - You’re allowed breaking changes before 1.0.
>
> ⇒ have as stable part and a non-stable part. Libraries that never
> stabilize can be discarded after some time.

I've read that post before, and agree that it makes a good case.

Those solutions are exactly the right ones. Let's do them!

In particular, Python 3 brought some changes where there is no way to
write code that loads in both 2 and 3. This was a great disservice to
library authors, and should never be done. RnRS 6 vs 7 is the same
thing. E.g. (library ...) vs (define-library ...) are not compatible.

A non-core Sceme library collection should be designed as far feasible
such that people can mix different years' versions of libraries if they
so choose. This happens automatically if data types stay compatible.

>> Which proposal do you stand behind?
>

> Firstoff: The one where people actually do it.
>
> That’s the most important requirement. And the hardest.

SRFI and implementation-specific libraries are the leader here.
R7RS-large has on average 3 active contributors at any given time which
sincerely is very good for Scheme.

SLIB and the other library collections are one-person projects.

Single-person or single-implementation libs can never grow Scheme's user
base very much, which means they also don't bring in new maintainers for
implementation-specific stuff.

> After that there are two answers:
>
> - What I would use: One with stability guarantees. I need my code to
> still work in 10 years.
> - What I might contribute to: One that tackles problems I’m deeply
> interested in.
>
>> - Each schemer does their personal collection of non-core libs
>
> This would be worst, because it prevents newcomers from starting up quickly.

Agreed.

In particularly, 100% agreed on the stability guarantee for published
interfaces.

>> - Each Scheme implementation has its own collection of non-core libs
>
> That’s the "choose your tribe" approach.

My (controversial) opinion is the bigger the tribe, the better. "Scheme"
is a size of tribe that we can realistically work toward. "Lisp" or
"FP", not yet; a lot more groundwork would have to be laid.

> There’s also SRFIs which actually work pretty well.

Agreed, but it's proven too hard for most things.

> Yes, but someone has to do the work to keep it working in the API.

> Here’s the question then, whether this requires touching any other part.
> If it does, then you have a problem again.

> This means you cannot just change other APIs to be uniform with the new API.

Striving to keep data types compatible ought to go most of the way.

If data types stay compatible, then prior years' exported procedures can
be kept as internal procedures in the newest implementation, making it
possible for that implementation to also support the prior year's API.

>> It's probably harder to switch
>> to a whole new library than the newest version of the old library.
>
> If you only need a few features from your existing tool, it can actually
> be easier to just recreate that part. And push the problem down on all
> your users who needed another part, so the problem multiplies.
>
> Keep the 1% in mind — and add to that significant time delay ("not now,
> I’m in this other project right now, I’ll do this when I’m free" — and
> this is how porting from python 2 to python 3 can take more than a
> decade and result in discarding working tools).

The only alternatives are:

1. Make a new version of the old library with API changes.

2. Start a new library from scratch whenever the old one is too crufty.

3. Be clairvoyant so we never have to change everything.

I vote for #1.

[Marc Nieper-Wißkirchen]

I feel sorry for the apparent misunderstanding.  We are communicating over a medium that is quite limited.  I'm pretty sure that should we meet personally, we would both find the guy on the other side quite likable.

We are not in mathematics where there is an absolute notion of the validity of a proof (or argument), so when I write "you have been refuted" (I should have better written: "your arguments have been refuted"), it doesn't mean that there are no reasons to refute the refutation.  Maybe you have already argued against that refutation, in which case I haven't yet come across it. (And if "refute" was too strong a word, it would be because of my limited vocabulary and not because I wanted to alienate anyone - wanting to alienate one's discussion partner wouldn't make any sense anyway.)

It's a bit sad that you seemingly have got the impression that I would simply assert that some argument was wrong without giving any argument for such a claim of wrongness.  I think we have never discussed on such a low level; quite the contrary.  It would be a pity if fruitful discussion stopped because of personal offense and/or perception of such.  When I say the sky is red and you finally prove me wrong, it doesn't make me as a person worth more or less.  The only object that would have been touched is the claim the sky is red, a claim that stands on its own and is detached from any person.

While I believe that your former arguments against "identifier syntax" are based on some misconceptions, and I may even believe that it would be hard to refute the refutation, I would be foolish to ignore any further arguments for or against it.  Firstly, whoever claims they already know the absolute truth is most likely wrong already with such a statement, and, secondly, this is not a game that is won if the claim with which I entered the discussion still stands at the end of it.

I have given examples for cases where the ER system cannot handle an unhygienic macro and where ER identifiers simply do carry not enough lexical information (for example in conjunction with a module system).  I believe that the syntax-case system, on the other hand, is sound in this regard.  But if someone shows me that it is not, I will certainly change my belief.

It's up to you whether we continue the discussion; it's your free choice, of course, but I would certainly like to. There are still a number of points that seem beneficial to discuss further.

Marc

[Robert Klein]

What about version numbers for API's, so a program using it can check?

Best regards
Robert

[Lassi Kortela]

> What about version numbers for API's, so a program using it can check?

Version numbers are necessary part of a good design. They are already
used in R6RS, in fact: e.g. (import (rnrs bytevectors (6))).

R7RS lets us do things like this:

(cond-expand ((library (some library v1))
(import (some library v1)))
((library (another library v2))
(import (another library v2))))

[John Cowan]

On Thu, Aug 12, 2021 at 3:09 AM Lassi Kortela <la...@lassi.io> wrote:

 

> Used SRFI-42 and it’s no longer en-vogue? Soft trauma.

That's programming-by-fashion, and based on the maxim "Grow or die" (which is usually proclaimed as the first principle of running a business, and has been many times debunked).  I have a counter-maxim: "When something hasn't changed for a while, it may be abandoned.  Then again, it may be just finished."

For Python, there's supposed to be just one way to do something, so Python programmers are constantly subject to the whims of fashion.  Lisp has never held with that: we usually try to provide *every* way to do something.

The CL community hasn't changed its standard for 28 years.  Scheme isn't quite that stable, but the number of backwards incompatible changes since R4RS (31 years ago) is ~10 in R7 and ~15 in R6: things like case sensitivity, transcript-on and transcript-off, "a\n" being two characters instead of three, string<? and friends aren't required to be lexicographic extensions of char<? and friends, etc.

The major imperative for applying this care is in RnRS. It wasn't
applied in R6 and it's not being applied in R7. Someone has to say it.

Well, you've said it: where's your concrete examples?

It's also important to distinguish between interface and implementation.
We should make stable versions of the interface spec. If people want to
go back later and optimize an implementation, so much the better.

That's what we have always done, and why SRFI implementations are just sample implementations: if they are inefficient, write better ones; if they disagree with the spec, fix the spec.

- Non-core libraries in SLIB

- Non-core libraries in R7RS-large

What is "core"?

- "Core is what I need, non-core is what I don't need."

- "Core is what I don't need, non-core is what I need."

- "Core is what the WG says it is."

> But if you rely on some old version of a library and that version does
> not support http2, then you lost. You suddenly have to invest much more
> than expected just to keep your tool working with the changing
> environment.

I don't understand that. If a new protocol supports the same API as the
old protocol, then support can be added without changing the API. The
old API can be kept working with the new protocol.

Yes, but supporting HTTP/2 is a Very Big Deal.  Fortunately, few if any clients and servers communicate only in HTTP/2, and I expect it never to become prevalent on the server side, especially for offbeat servers.

[Marc Nieper-Wißkirchen]

Just one meta-comment about the regular comparisons with Python:  Not everything from there applies necessarily to Scheme because Python is effectively defined by one implementation.  

--
You received this message because you are subscribed to the Google Groups "scheme-reports-wg2" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scheme-reports-...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/scheme-reports-wg2/CAD2gp_Sz7gkw%3DMyWmAGhq12b-Kq90LvHk3BkwaRY%2BpBaU_zDjg%40mail.gmail.com.

[John Cowan]

On Thu, Aug 12, 2021 at 6:37 AM Lassi Kortela <la...@lassi.io> wrote:

 

Version numbers are necessary part of a good design. They are already
used in R6RS, in fact: e.g. (import (rnrs bytevectors (6))).

It turns out that 100% of existing R6 implementations ignore the version number, which is one reason why they don't exist (with the special comparison algorithm and all) in R7.

[Marc Nieper-Wißkirchen]

The assertion that 100% of existing R6 implementations ignore the version number is untrue.

Chez Scheme, by many measures a leading R6RS implementation, does not ignore version numbers.

Marc

PS Unsyntax, although not being an R6RS implementation, does implement the R6RS library syntax and semantics parallel to the R7RS library syntax and like Chez Scheme does not ignore R6RS library version numbers.

PPS For R7RS+, version numbers may become interesting for the standard libraries unless the R7RS-defined contents of, say, (scheme base) are supposed to be frozen forever. 

[Taylan Kammer]

On 13.08.2021 08:19, Marc Nieper-Wißkirchen wrote:

> Am Do., 12. Aug. 2021 um 23:22 Uhr schrieb John Cowan <co...@ccil.org <mailto:co...@ccil.org>>:
>
> It turns out that 100% of existing R6 implementations ignore the version number, which is one reason why they don't exist (with the special comparison algorithm and all) in R7.
>
>
> The assertion that 100% of existing R6 implementations ignore the version number is untrue.
>
> Chez Scheme, by many measures a leading R6RS implementation, does not ignore version numbers.
>
> Marc
>
> PS Unsyntax, although not being an R6RS implementation, does implement the R6RS library syntax and semantics parallel to the R7RS library syntax and like Chez Scheme does not ignore R6RS library version numbers.
> PPS For R7RS+, version numbers may become interesting for the standard libraries unless the R7RS-defined contents of, say, (scheme base) are supposed to be frozen forever. 

It's probably also worth considering that more R6 implementations might
implement version number checking once there are non-R6 version of the
libraries defined originally in the R6RS.

--
Taylan

[Lassi Kortela]

>>> Used SRFI-42 and it’s no longer en-vogue? Soft trauma.

> That's programming-by-fashion, and based on the maxim "Grow or die"
> (which is usually proclaimed as the first principle of running a
> business, and has been many times debunked).  I have a counter-maxim:
> "When something hasn't changed for a while, it may be abandoned.  Then
> again, it may be just finished."

I agree with "then again, it may be just finished" but would add that "a
project is finished when you stop working on it and decide to call it
finished". The things finalized in an RnRS have to be finished in this
sense, and changing them around would cause the kind of "trauma"
detailed in the blog post linked by Arne. We all seem to agree on that,
modulo R6RS.

> For Python, there's supposed to be just one way to do something, so
> Python programmers are constantly subject to the whims of fashion.  Lisp
> has never held with that: we usually try to provide *every* way to do
> something.

I don't know Python well enough to evaluate this claim.

> The CL community hasn't changed its standard for 28 years.  Scheme isn't
> quite that stable, but the number of backwards incompatible changes
> since R4RS (31 years ago) is ~10 in R7 and ~15 in R6: things like case
> sensitivity, transcript-on and transcript-off, "a\n" being two
> characters instead of three, string<? and friends aren't required to be
> lexicographic extensions of char<? and friends, etc.

In Scheme, (library ...) and (syntax-case ...) don't work in R7. Those
are the major backward-incompatible changes within the RnRS lineage.
Additionally, when R6 came out, (library ...) and (syntax-case ...)
weren't adopted by some prominent Scheme implementations, and people
kept using non-standard module and macro systems.

As a result, Scheme still doesn't have de facto standard modules and
procedural macros. The R6/R7 module systems' feature set and semantics
are compatible and now de facto, which is great, but surface syntax is
incompatible so people can't take advantage of the semantic similarity.

Details like transcripts and case sensitivity are not a problem.

>>> That’s the flipside of power: We must be extremely careful how, whether,
>>> and when to wield it.

>> The major imperative for applying this care is in RnRS. It wasn't
>> applied in R6 and it's not being applied in R7. Someone has to say it.

> Well, you've said it: where's your concrete examples?

The major ones are (library ...) and (syntax-case ...).

It'd be nice if things like the R6/R7 port procedures and condition
hierarchy were compatible, but it looks like it's simple enough to work
around these things.

There are different opinions of what obligations RnRS has, ordered from
weaker to stronger:

(1) RnRS should be self-consistent.

(2) RnRS should be consistent with some prior versions of the standard.

(3) RnRS should be consistent with all prior versions of the standard
still in use.

(4) RnRS should track the things that Scheme implementations agree on.
An RnRS that is consistent within the RnRS series, yet doesn't represent
implementer consensus, is not a successful standard.

The reason I keep being a bother is that I feel I'm one of the only ones
here who are in camp 4. It's hard to find any place to voice these
concerns, and it's hard to express them without coming off as rude, but
to me the issue is far more important than anything else pertaining to RnRS.

When outsiders and newbies evaluate how much (if at all) to use Scheme,
they care about point 4. They only care about the RnRS series to the
extent that implementers do, and they don't want to think about
aesthetic disputes between implementers, wishing those were settled.
RnRS would be the natural nexus for implementers' dispute-settling
activities, but R6 started to be lax about that, and in R7-large we're
doing something completely different.

Apart from R6 compat, my major concern with "wielding power" in R7-large
(again, being firmly in camp 4) is that any RnRS that adds a feature
puts a moral obligation on future RnRS editions to also support that
feature. R6 put an obligation on R7 to carry all the new stuff, which R7
didn't want to do, causing the 6/7 split.

Currently it looks like we're not fixing the 6/7 split, and are
re-creating the conditions in which the 6/7 split arose, so that an
additional 7/8 split will arise in the future. I can't find a way around
that conclusion.

> What is "core"?
>
> - "Core is what I need, non-core is what I don't need."
>
> - "Core is what I don't need, non-core is what I need."
>
> - "Core is what the WG says it is."

Core is:

- Syntax

- Control (procedures, continuations, conditions)

- Library and macro systems

- Essential or ubiquitous data types (lists, vectors, strings, hash tables)

Everyone needs both core and non-core stuff to write useful code. If
people don't agree on core stuff, chaos ensues. If people don't agree on
non-core stuff, there's some difficulty but not that much.

[Arthur A. Gleckler]

On Fri, Aug 13, 2021 at 11:51 PM Lassi Kortela <la...@lassi.io> wrote:

 

Apart from R6 compat, my major concern with "wielding power" in R7-large
(again, being firmly in camp 4) is that any RnRS that adds a feature
puts a moral obligation on future RnRS editions to also support that
feature. R6 put an obligation on R7 to carry all the new stuff, which R7
didn't want to do, causing the 6/7 split. 

This isn't a fair characterization of the history of R6RS and R7RS.  While I participated in the R6RS process at the time, I knew a number of people involved in prominent implementations who were unhappy with the way that the R6RS process had been established, and in particular with the switch away from the consensus-based approach that had been in place through R5RS.  Those folks had never agreed to the new R6RS process, and R6RS therefore didn't create any obligation for them.  That was apparent long before R6RS was ratified.  With R7RS, the Scheme Steering Committee tried hard to find a delicate compromise that might make use of the best ideas of R6RS (in R7RS Large) while following a process that everyone could accept.  It remains to be seen whether they achieved this balance.

For my part, I just want to be able to contribute to writing substantial, useful programs in Scheme, and for people using different high-quality Scheme implementations to be able to use those programs without doing a lot of porting work.