doc strings

35 views
Skip to first unread message

Arthur A. Gleckler

unread,
Apr 5, 2020, 12:49:30 AM4/5/20
to scheme-re...@googlegroups.com, John Cowan
John, do you think that doc strings might be considered for R7RS Large? I've been including them in my code for years in anticipation of the day that they would actually be supported, and it occurs to me that now might be the time to speak up:

(define (find-previous-tail predicate list)
  "If the car of any pair of `list' after the first one satisfies
`predicate', return the previous pair.  Otherwise, return #f."
  (if (null? list)
      #f
      (let next ((previous list)
                 (tail (cdr list)))
        (cond ((null? tail) #f)
              ((predicate (car tail)) previous)
              (else (next (cdr previous)
                          (cdr tail)))))))

Marc Nieper-Wißkirchen

unread,
Apr 5, 2020, 3:34:21 AM4/5/20
to scheme-re...@googlegroups.com, John Cowan
May I chime in and ask about the semantics of the doc strings?

The following questions have come to my mind so far:

Are they supposed to be associated to objects at runtime or only at expand time? Or will they have no special meaning to the Scheme expander und runtime system but interpreted by a different tool as it is done with Scribble? What is the advantage of doc strings versus Scribble and vice versa?

How to associate doc strings to non-procedure definitions? How to syntax definitions? Do they work well with case-lambda? Shall doc strings only be associated to top-level definitions or to individual objects?

Scribble and some other documentation systems have some TeX-like markup language that allows to generate rich documentation and to extract meta-information. How are doc strings positioned in this regard?

Marc

--
You received this message because you are subscribed to the Google Groups "scheme-reports-wg2" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scheme-reports-...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/scheme-reports-wg2/CALnw4L%2Bh1JN7kdi45z_ULgcFwe0VorPUEQosvQqGvLX-nBBwFQ%40mail.gmail.com.

Arthur A. Gleckler

unread,
Apr 5, 2020, 3:45:37 AM4/5/20
to scheme-re...@googlegroups.com, John Cowan
On Sun, Apr 5, 2020 at 12:34 AM Marc Nieper-Wißkirchen <marc....@gmail.com> wrote:
 
Are they supposed to be associated to objects at runtime or only at expand time? Or will they have no special meaning to the Scheme expander und runtime system but interpreted by a different tool as it is done with Scribble? What is the advantage of doc strings versus Scribble and vice versa?

Ideally, they would be associated with objects at runtime.  That doesn't mean that they would have to be kept in memory; only that some key for retrieving them would be present.  Ideally, a simple API could retrieve a doc string from any object that was defined with one.

Sorry, I don't yet know anything about Scribble.  I'll try to read up on it.
 
How to associate doc strings to non-procedure definitions?

I've seen systems that have a comment form that can be wrapped around any definition, attaching a string to the binding.
 
How to syntax definitions? Do they work well with case-lambda? Shall doc strings only be associated to top-level definitions or to individual objects?
 
Scribble and some other documentation systems have some TeX-like markup language that allows to generate rich documentation and to extract meta-information. How are doc strings positioned in this regard?

There's no reason that doc strings couldn't use markup as well, e.g. Markdown, but I'd be okay if that were left unspecified.

I would just like some simple way to attach a runtime-accessible string to each definition so that the REPL and other interactive tools can display it when requested.

Marc Nieper-Wißkirchen

unread,
Apr 5, 2020, 4:04:01 AM4/5/20
to scheme-re...@googlegroups.com, John Cowan
Am So., 5. Apr. 2020 um 09:45 Uhr schrieb Arthur A. Gleckler <a...@speechcode.com>:
On Sun, Apr 5, 2020 at 12:34 AM Marc Nieper-Wißkirchen <marc....@gmail.com> wrote:
 
Are they supposed to be associated to objects at runtime or only at expand time? Or will they have no special meaning to the Scheme expander und runtime system but interpreted by a different tool as it is done with Scribble? What is the advantage of doc strings versus Scribble and vice versa?

Ideally, they would be associated with objects at runtime.  That doesn't mean that they would have to be kept in memory; only that some key for retrieving them would be present.  Ideally, a simple API could retrieve a doc string from any object that was defined with one.

Sorry, I don't yet know anything about Scribble.  I'll try to read up on it.

Chibi and Racket both use Scribble.  As far as I understand, Racket's (excellent) library documentation has been generated from it.

 
How to associate doc strings to non-procedure definitions?

I've seen systems that have a comment form that can be wrapped around any definition, attaching a string to the binding.

I would say such a comment form should rather wrap a value, not a definition as Scheme is very flexible when it comes to defining things:

(define-values (x y)
  (let ((foo (foobricator)))
    (values (comment "Twizzle foos one at a time" (make-proc foo 1)) (comment "Twizzle foos two at a time" (make-proc foo 2)))))

Doc strings in define forms would then just be syntactic sugar (Warning: Not implementable with syntax-rules!):

(define (name . formals) doc body) => (define name (comment doc (lambda formals body)))
(define name doc val) => (define name (comment doc val))
 
 
How to syntax definitions? Do they work well with case-lambda? Shall doc strings only be associated to top-level definitions or to individual objects?
 
Scribble and some other documentation systems have some TeX-like markup language that allows to generate rich documentation and to extract meta-information. How are doc strings positioned in this regard?

There's no reason that doc strings couldn't use markup as well, e.g. Markdown, but I'd be okay if that were left unspecified.

I would just like some simple way to attach a runtime-accessible string to each definition so that the REPL and other interactive tools can display it when requested.

In the case of syntactic bindings, we would attach this string to the actual transformer (which is, or can be seen, as a runtime object in a phase shifted by one)?

(define-syntax foo doc transformer) => (define-syntax foo (comment doc transformer))

Marc

--
You received this message because you are subscribed to the Google Groups "scheme-reports-wg2" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scheme-reports-...@googlegroups.com.

Arne Babenhauserheide

unread,
Apr 5, 2020, 5:38:55 AM4/5/20
to scheme-re...@googlegroups.com, John Cowan

Marc Nieper-Wißkirchen <marc....@gmail.com> writes:

> Am So., 5. Apr. 2020 um 09:45 Uhr schrieb Arthur A. Gleckler <
> a...@speechcode.com>:
>
>> On Sun, Apr 5, 2020 at 12:34 AM Marc Nieper-Wißkirchen <
>> marc....@gmail.com> wrote:
>>
>>
>>> Are they supposed to be associated to objects at runtime or only at
>>> expand time? Or will they have no special meaning to the Scheme expander
>>> und runtime system but interpreted by a different tool as it is done with
>>> Scribble? What is the advantage of doc strings versus Scribble and vice
>>> versa?
>>>
>>
>> Ideally, they would be associated with objects at runtime. That doesn't
>> mean that they would have to be kept in memory; only that some key for
>> retrieving them would be present. Ideally, a simple API could retrieve a
>> doc string from any object that was defined with one.
>>
>> Sorry, I don't yet know anything about Scribble. I'll try to read up on
>> it.
>
> Chibi and Racket both use Scribble. As far as I understand, Racket's
> (excellent) library documentation has been generated from it.

Guile provides doc-strings as part of procedure-properties:

https://www.gnu.org/software/guile/docs/docs-2.0/guile-ref/Procedure-Properties.html#index-procedure_002ddocumentation


Minimal documented function:

(define (foo)
"doc"
#f)

(procedure-documentation foo) ; => "doc"
(foo) ; => #f


They are available at runtime.


To preserve Scheme-semantics,
(define (foo) "not-a-docstring")
has the return value "not-a-docstring", but no docstring.


In addition, procedures can have properties which can be used to
implement more natural doctests than for example seen in Python:

https://hg.sr.ht/~arnebab/wisp/browse/33b4fcdd8a17aa19d57971e4f6db5fcb7758843c/examples/doctests.scm#L10

Their advantage over Python-doctests is that test and implementation are
uniform: The test uses regular s-exp syntax instead of being
string-parsed as it is done in Python (with all the quoting hassles
string-parsing brings).

Best wishes,
Arne
--
Unpolitisch sein
heißt politisch sein
ohne es zu merken

Marc Nieper-Wißkirchen

unread,
Apr 5, 2020, 5:43:12 AM4/5/20
to scheme-re...@googlegroups.com, John Cowan
Is this feature of Guile well-thought out?

(1) There are more values than just procedures; we want to associate doc strings to, say, global constants or macro transformers as well.

(2) The convention that the first expression in a procedures body is a docstring if it is a string and not the only expression doesn't work with case-lambdas, because we don't want each individual clause to be documented but the whole procedure.

Marc



They are available at runtime.


To preserve Scheme-semantics,
(define (foo) "not-a-docstring")
has the return value "not-a-docstring", but no docstring.


In addition, procedures can have properties which can be used to
implement more natural doctests than for example seen in Python:

https://hg.sr.ht/~arnebab/wisp/browse/33b4fcdd8a17aa19d57971e4f6db5fcb7758843c/examples/doctests.scm#L10

Their advantage over Python-doctests is that test and implementation are
uniform: The test uses regular s-exp syntax instead of being
string-parsed as it is done in Python (with all the quoting hassles
string-parsing brings).

Best wishes,
Arne
--
Unpolitisch sein
heißt politisch sein
ohne es zu merken

--
You received this message because you are subscribed to the Google Groups "scheme-reports-wg2" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scheme-reports-...@googlegroups.com.

Arthur A. Gleckler

unread,
Apr 5, 2020, 3:44:19 PM4/5/20
to scheme-re...@googlegroups.com, John Cowan

On Sun, Apr 5, 2020 at 12:45 AM Arthur A. Gleckler <a...@speechcode.com> wrote:

Sorry, I don't yet know anything about Scribble. I'll try to read up on it.

Okay, I found 4.5 In-Source Documentation and the In-Source Documentation section of Submodules. The latter gives examples. It is geared to support Racket's contract system, but it does provide a way to add documentation strings accessible at runtime, although in forms separate from the defining forms.

I still prefer documentation strings a la Common Lisp and Emacs Lisp and the example I gave. They're right inside the defining form, which means that the identifier doesn't have to be repeated, and they're less verbose.

It should be easy to add them to lambda, case-lambda, define-syntax, etc. without affecting existing code. Just off the top of my head:

(define section-element
  (let ((associate (association-procedure string=? car)))
    (lambda (section element-name)
      "Return element named `element-name' from `section'."
      (let ((association (associate element-name section)))
        (and association
             (cdr association))))))

(define convui-regexp
  (case-lambda
   "Match regular expression `pattern', obeying `case-fold?' (default #f), and
use optional `lift' to lift the result."
   ((pattern) (convui-regexp pattern #f))
   ((pattern case-fold?)
    (convui-regexp pattern case-fold? (lambda (r s) (re-match-extract s r 0))))
   ((pattern case-fold? lift)
    ...)))

(define-syntax assert
  (syntax-rules ()
   "Return the symbol `passed' when the test passes.  The unit test
framework..."
    ((_ (operator argument ...))
     ...)
    ((_ (operator argument ...) error-values ...)
     ...)
    ((_ expression)
     ...)
    ((_ expression error-values ...)
     ...)))

And for defining forms that haven't been modified to accept documentation strings, e.g. when making a top-level binding to a list, a comment form like this might work:

or even:

The goals I'd like to achieve:

  • documentation strings accessible at runtime, e.g. through the REPL or editor tools
  • terse definitions
  • existing code runs unchanged
Thanks for considering the idea.

Marc Nieper-Wißkirchen

unread,
Apr 5, 2020, 4:55:20 PM4/5/20
to scheme-re...@googlegroups.com, John Cowan
Am So., 5. Apr. 2020 um 21:44 Uhr schrieb Arthur A. Gleckler <a...@speechcode.com>:

On Sun, Apr 5, 2020 at 12:45 AM Arthur A. Gleckler <a...@speechcode.com> wrote:

Sorry, I don't yet know anything about Scribble. I'll try to read up on it.

Okay, I found 4.5 In-Source Documentation and the In-Source Documentation section of Submodules. The latter gives examples. It is geared to support Racket's contract system, but it does provide a way to add documentation strings accessible at runtime, although in forms separate from the defining forms.

One advantage of Racket's model is that it is extensible with Scheme code and that it is expanded and run at different phases than the library itself is expanded and run.

Maybe Alex Shinn can explain Chibi's version of Scribble, which is also used to generate package documentation for snow-fort.org.


For case-lambda, this would break code that does not understand documentation strings.
Likewise.

And for defining forms that haven't been modified to accept documentation strings, e.g. when making a top-level binding to a list, a comment form like this might work:

or even:


The latter form allows "comment" to be implemented as a macro; the former version doesn't. Thus, the latter form is preferable.

A new "define" syntax with three arguments could also be defined.

Marc

The goals I'd like to achieve:

  • documentation strings accessible at runtime, e.g. through the REPL or editor tools
  • terse definitions
  • existing code runs unchanged
Thanks for considering the idea.

--
You received this message because you are subscribed to the Google Groups "scheme-reports-wg2" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scheme-reports-...@googlegroups.com.

Arthur A. Gleckler

unread,
Apr 5, 2020, 4:59:11 PM4/5/20
to scheme-re...@googlegroups.com, John Cowan
On Sun, Apr 5, 2020 at 1:55 PM Marc Nieper-Wißkirchen <marc....@gmail.com> wrote:

For case-lambda, this would break code that does not understand documentation strings. 

Of course, but existing code wouldn't break.  People who wanted to use this form of case-lambda would make sure to import it.
 
The latter form allows "comment" to be implemented as a macro; the former version doesn't. Thus, the latter form is preferable.

Yes, I prefer the latter form, anyway.
 
A new "define" syntax with three arguments could also be defined.

Yes, that's a good point. 

Alex Shinn

unread,
Apr 6, 2020, 4:01:58 AM4/6/20
to scheme-re...@googlegroups.com, John Cowan
On Mon, Apr 6, 2020 at 5:55 AM Marc Nieper-Wißkirchen <marc....@gmail.com> wrote:
Am So., 5. Apr. 2020 um 21:44 Uhr schrieb Arthur A. Gleckler <a...@speechcode.com>:

On Sun, Apr 5, 2020 at 12:45 AM Arthur A. Gleckler <a...@speechcode.com> wrote:

Sorry, I don't yet know anything about Scribble. I'll try to read up on it.

Okay, I found 4.5 In-Source Documentation and the In-Source Documentation section of Submodules. The latter gives examples. It is geared to support Racket's contract system, but it does provide a way to add documentation strings accessible at runtime, although in forms separate from the defining forms.

One advantage of Racket's model is that it is extensible with Scheme code and that it is expanded and run at different phases than the library itself is expanded and run.

Maybe Alex Shinn can explain Chibi's version of Scribble, which is also used to generate package documentation for snow-fort.org.

The basic idea is that comment lines immediately starting with > (and no space) are used for documentation of the _following_ definition:

;;> Returns true iff str is equal to the empty string "".
(define (string-null? str) (equal? str ""))

;;> The ratio of a circle's circumference to its diameter.
(define pi 3.14159)

The docs can use scribble markup, and can optionally start with a signature (otherwise inferred from source):

;;> \procedure{(string-null? str)}.
;;> Returns true iff \var{str} is equal to the empty string \scheme{""}.
(define (string-null? str) (equal? str ""))

If you want to summarize a group of definitions, instead put the documentation after all definitions and start the comment with / on the final line.  It then applies to all _preceding_ definitions up to any previous comment:

(define black-escape (make-simple-escape-procedure 30))
(define red-escape (make-simple-escape-procedure 31))
...
(define white-escape (make-simple-escape-procedure 37))

;;> Return a string consisting of an ANSI escape code to select the
;;> specified text color.
;;/

You can use \section{name} for section grouping and \example{expr} to include an expression along with it's evaluated result in the docs.

The (chibi doc) module provides the (print-procedure-docs proc) to output the documentation for a procedure at runtime, and (print-module-docs module-name) to print the docs for a whole module, with options to display as HTML (used by the tooling on snow-fort.org).  This parses the source at runtime so there is zero overhead if not using (chibi doc).

--
Alex

Arne Babenhauserheide

unread,
Apr 6, 2020, 7:22:56 AM4/6/20
to scheme-re...@googlegroups.com, John Cowan

Alex Shinn <alex...@gmail.com> writes:
> The basic idea is that comment lines immediately starting with > (and no
> space) are used for documentation of the _following_ definition:
>
> ;;> Returns true iff str is equal to the empty string "".
> (define (string-null? str) (equal? str ""))
>
> ;;> The ratio of a circle's circumference to its diameter.
> (define pi 3.14159)

When you parse comments, you lose the property that code is data. If you
(read) your code and process it, the documentation is gone.

When your documentation is part of the code tree, you can process the
code and still have the documentation.

Therefore I strongly prefer a model which includes documentation in the
code instead of adding something on top that’s parsed by a separate
tool.

I’ve experienced both styles first hand — docstrings as part of the
definition in Python and Guile and documentation in form of comments in
Java and C, and having docstrings as first class elements in your code
is much cleaner and makes it easier for programmers to build on them.

(define (string-null? str)
"Returns true iff str is equal to the empty string"
(equal? str ""))

elf

unread,
Apr 6, 2020, 7:53:51 AM4/6/20
to scheme-re...@googlegroups.com
How, then, does one distinguish between a docstring and a string return value? This gets especially hairy if one allows multiline docstrings.

There is also size overhead for non-parsed comments, as they must necessarily be included in the code proper. This can lead to some debugging issues, especially in a case like that above.

Anything that I can think of to solve these involves making any reader system so much more complicated as to negate any benefit. Please tell me if I'm wrong here, of course.

(This is not a theoretical set of problems, either. When hacking on scwm 20 years ago, this came up where the guile docstring was being incorrectly added to the string return value, iirc. Something like that, unfortunately I do not remember exactly atm but I can dig through my archive for the concrete case if desired.)

I would strongly prefer either a separate read syntax (as in the ;;> above), a macro definition (docstring-lambda that will parse out the relevant bits), or a separate documentation lib of some sort to avoid these sorts of issues.

-elf

Marc Nieper-Wißkirchen

unread,
Apr 6, 2020, 8:13:51 AM4/6/20
to scheme-re...@googlegroups.com
Addressing Arne's and elf's concerns, I think we can learn a lot from Racket's model, which has documentation (in some form of Scheme code) and the documented code itself in the same file while not producing any runtime overhead.

In the R6RS library model as being advertised by Ghuloum and Dybvig (I would talk about the R7RS library model but it isn't yet as sophisticated as it doesn't have to be yet because the only macro transformer we have is syntax-rules), a library can be *expanded*, *visited* and *invoked*.  When a library appears in an import declaration (or in a clause of cond-expand), it will be located and expanded (if not already expanded).  The expansion will reveal all top-level definitions and whether they are variables or keywords.  The expansion will produce two procedures, a visit procedure and an invoke procedure.  When any keyword binding is needed from the library, the visit procedure will be called (if not already called) to produce the actual transformer of the keyword.  When any variable binding is needed from the same library, the invoke procedure will be called (if not already called) to produce the actual value of the variable.  This model, in particular, ensures that only code that is needed is to be evaluated and, say, macro transformers in a library do not add to the runtime of the final program.

In-file documentation can be added to this model by adding a procedure to any expanded version of a library that actually registers the documentation with the Scheme system.  A repl would call this procedure, a compiled program wouldn't.

Marc

--
You received this message because you are subscribed to the Google Groups "scheme-reports-wg2" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scheme-reports-...@googlegroups.com.

elf

unread,
Apr 6, 2020, 9:07:28 AM4/6/20
to scheme-re...@googlegroups.com
If I understand this correctly, this only is relevant for code with docstrings as part of a library. Not all documented code is necessarily part of a library, I think.

This appears to me, on the surface, as a very complex way to get around fairly simple reader syntax or a lightweight macro or lambda properties solution... I do not see the need for so much extra gunk. (One of my objections to R6 at the time.)

-elf

Marc Nieper-Wißkirchen

unread,
Apr 6, 2020, 9:13:37 AM4/6/20
to scheme-re...@googlegroups.com
Am Mo., 6. Apr. 2020 um 15:07 Uhr schrieb elf <e...@ephemeral.net>:
If I understand this correctly, this only is relevant for code with docstrings as part of a library. Not all documented code is necessarily part of a library, I think.

I thought this as well, but then I began to wonder how much code you want to document that is not part of a library. Moreover, how do you access such documentation?
 
This appears to me, on the surface, as a very complex way to get around fairly simple reader syntax or a lightweight macro or lambda properties solution... I do not see the need for so much extra gunk. (One of my objections to R6 at the time.)

I didn't want to advertise to copy Racket's approach verbatim. But we should learn from the lessons they have learned. A large system wants so much extra "gunk", a small system wants to live without, so we have to find a solution that is easy to start with and whose model is sophisticated enough for a large system that separates phases, etc.

Note that a lambda properties solution isn't enough; non-procedure globals and defaults also want to be documented.

Marc


Arne Babenhauserheide

unread,
Apr 6, 2020, 9:47:20 AM4/6/20
to scheme-re...@googlegroups.com

elf <e...@ephemeral.net> writes:
> How, then, does one distinguish between a docstring and a string return value? This gets especially hairy if one allows multiline docstrings.

What Guile does: if the string is the only part of the body, then there
is no docstring: it is not a docstring but a return value.

> Anything that I can think of to solve these involves making any reader system so much more complicated as to negate any benefit. Please tell me if I'm wrong here, of course.

The above ensures that a string is only a docstring if using it as a
docstring has no side-effect.

Parsing a docstring is the same as parsing any other string.

It is already allowed in r7rs to have a string as first expression in
the body, so nothing changes in the syntax (different from the
comment-version where there’s new syntax that’s hidden in comments).

What changes is that if the first form in define is a string, and it is
not the last form, then this string becomes the docstring.

The long form is
(set-procedure-property! fun 'documentation "the docstring")

But since there is obviously no consensus about this (strong preferences
both ways), first creating SRFIs might be a better way forward.

Best wishes,
Arne

Marc Nieper-Wißkirchen

unread,
Apr 6, 2020, 9:58:03 AM4/6/20
to scheme-re...@googlegroups.com
Am Mo., 6. Apr. 2020 um 15:47 Uhr schrieb Arne Babenhauserheide <arne...@web.de>:

elf <e...@ephemeral.net> writes:
> How, then, does one distinguish between a docstring and a string return value? This gets especially hairy if one allows multiline docstrings.

What Guile does: if the string is the only part of the body, then there
is no docstring: it is not a docstring but a return value.

> Anything that I can think of to solve these involves making any reader system so much more complicated as to negate any benefit. Please tell me if I'm wrong here, of course.

The above ensures that a string is only a docstring if using it as a
docstring has no side-effect.

Parsing a docstring is the same as parsing any other string.

It is already allowed in r7rs to have a string as first expression in
the body, so nothing changes in the syntax (different from the
comment-version where there’s new syntax that’s hidden in comments).

A string as the first expression is not allowed if you want to have internal definitions in the body.

 

What changes is that if the first form in define is a string, and it is
not the last form, then this string becomes the docstring.

The long form is
(set-procedure-property! fun 'documentation "the docstring")

But since there is obviously no consensus about this (strong preferences
both ways), first creating SRFIs might be a better way forward.

Isn't it more effective to seek consensus first? Otherwise, we have several competing SRFIs and still have no consensus.

At least, we can try to agree on the technical advantages and disadvantages and make up our minds accordingly.

Marc

Best wishes,
Arne


--
You received this message because you are subscribed to the Google Groups "scheme-reports-wg2" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scheme-reports-...@googlegroups.com.

Arne Babenhauserheide

unread,
Apr 6, 2020, 10:16:27 AM4/6/20
to scheme-re...@googlegroups.com

Arne Babenhauserheide <arne...@web.de> writes:

> elf <e...@ephemeral.net> writes:
>> How, then, does one distinguish between a docstring and a string return value? This gets especially hairy if one allows multiline docstrings.
>
> What Guile does: if the string is the only part of the body, then there
> is no docstring: it is not a docstring but a return value.

Addition: Parsing comments is harder than this: Any comment parser must
include a full string-parser, because it must distinguish between
comments and multiline-strings. Example:

"foo \\\"
;;> foo "
(define (foo) #f)

>> Anything that I can think of to solve these involves making any reader system so much more complicated as to negate any benefit. Please tell me if I'm wrong here, of course.
>
> The above ensures that a string is only a docstring if using it as a
> docstring has no side-effect.

Addition: I do not know a clean way to document objects that are defined
in a way which does not allow empty expressions.

(define foo "doc" #f) ;; <- syntax error.
(define (foo) "doc" #f) ;; <- valid and nice.

Marc Nieper-Wißkirchen

unread,
Apr 6, 2020, 10:28:48 AM4/6/20
to scheme-re...@googlegroups.com
Am Mo., 6. Apr. 2020 um 16:16 Uhr schrieb Arne Babenhauserheide <arne...@web.de>:

Arne Babenhauserheide <arne...@web.de> writes:

> elf <e...@ephemeral.net> writes:
>> How, then, does one distinguish between a docstring and a string return value? This gets especially hairy if one allows multiline docstrings.
>
> What Guile does: if the string is the only part of the body, then there
> is no docstring: it is not a docstring but a return value.

Addition: Parsing comments is harder than this: Any comment parser must
include a full string-parser, because it must distinguish between
comments and multiline-strings. Example:

"foo \\\"
;;> foo "
(define (foo) #f)

The Scheme reader has to cope with comments and strings anyway, so the necessary code should already be there.

What is probably not much work is the following: Change Scheme's "read" into a, say "read*" procedure that reads Scheme datums but Scheme datums wrapped into two special records, namely, say, #<source-position <datum> <pos>> and #<comment <datum> <comment>>.  The #<source-position ...> wrap can appear around every datum (and subdatum) whenever the reader can communicate the source position of the datum; the #<comment ...> wrap will appear around every datum that is preceded by some special comment syntax.

This solves the problem of reading and having code and comments together. It doesn't answer the question of how to associate the comment with the expanded code. This is not independent of the expander as this snippet

;;> foo
(baz (foo) #f)

may expand into a definition.
 
>> Anything that I can think of to solve these involves making any reader system so much more complicated as to negate any benefit. Please tell me if I'm wrong here, of course.
>
> The above ensures that a string is only a docstring if using it as a
> docstring has no side-effect.

Addition: I do not know a clean way to document objects that are defined
in a way which does not allow empty expressions.

(define foo "doc" #f) ;; <- syntax error.
(define (foo) "doc" #f) ;; <- valid and nice.

If I understand Arthur correctly, he suggests extending the `define' form to take more than two arguments when not defining a procedure.
 

Best wishes,
Arne
--
Unpolitisch sein
heißt politisch sein
ohne es zu merken

--
You received this message because you are subscribed to the Google Groups "scheme-reports-wg2" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scheme-reports-...@googlegroups.com.

Arne Babenhauserheide

unread,
Apr 6, 2020, 1:07:03 PM4/6/20
to scheme-re...@googlegroups.com

Marc Nieper-Wißkirchen <marc....@gmail.com> writes:

> Am Mo., 6. Apr. 2020 um 15:47 Uhr schrieb Arne Babenhauserheide <
> A string as the first expression is not allowed if you want to have
> internal definitions in the body.

Ah, you’re right. This is a limitation Guile just lifted in 3.0. It was
one of the most jarring things when I learned Scheme.

>> What changes is that if the first form in define is a string, and it is
>> not the last form, then this string becomes the docstring.
>>
>> The long form is
>> (set-procedure-property! fun 'documentation "the docstring")
>>
>> But since there is obviously no consensus about this (strong preferences
>> both ways), first creating SRFIs might be a better way forward.
>>
>
> Isn't it more effective to seek consensus first? Otherwise, we have several
> competing SRFIs and still have no consensus.

Can we reach consensus when there are such strong opinions?

If we can, you’re right.

> At least, we can try to agree on the technical advantages and disadvantages
> and make up our minds accordingly.

That sounds good.

elf

unread,
Apr 6, 2020, 1:51:11 PM4/6/20
to scheme-re...@googlegroups.com
FWIW, i don't have a strong opinion; if everyone agrees to something else, i'm good with that. That said, i would prefer a solution that involves something simpler (like an extra reader syntax symbol, that can be handled in implementation dependent ways, if desired, but won't cause problems for any non-implementing implementations) to something that overhauls everything about how the reader itself works.

Regarding the non-library documentation use cases, top-level applications are not necessarily libraries (are often not libraries) but could require internal documentation; likewise, prototyping and testing code may not be packaged as a library but some form of docstring may be useful. I think it is a mistake to tightly couple the idea of docstrings with the idea of libraries.

(i'm sorry i'm responding out of order but i'm on very limited access at the moment with extremely flaky internet and even flakier hardware, so i'm trying to condense things.)

Is there an inherent problem with a form something like:
(define (foo x)
(docstring "x get fooed")
...
)

with docstring as either a macro or a procedure? (macro is probably better, although i think, on reflection, that this will boil down to a macro redefinition of define in order to work properly without incurring per call overhead, which may not be ideal...)

-elf

Edwin Watkeys

unread,
Apr 6, 2020, 2:07:09 PM4/6/20
to scheme-re...@googlegroups.com
Hi all,

As a general comment, I’d like to mention that it may be worth thinking about object metadata more broadly and considering whether the larger issue is worth pondering, and if so, how documentation could be implemented within a more general metadata framework.

I menton this for two reasons: Clojure uses its metadata system, which has been put to many valuable uses, to implement doc strings IIRC, and I think that, if doc strings are to be supported, names bound to non-procedures (and/or those non-procedure objects themselves) should be supported.

Regards,
Edwin

> On Apr 6, 2020, at 13:51, elf <e...@ephemeral.net> wrote:
>
> FWIW, i don't have a strong opinion; if everyone agrees to something else, i'm good with that. That said, i would prefer a solution that involves something simpler (like an extra reader syntax symbol, that can be handled in implementation dependent ways, if desired, but won't cause problems for any non-implementing implementations) to something that overhauls everything about how the reader itself works.
> --
> You received this message because you are subscribed to the Google Groups "scheme-reports-wg2" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to scheme-reports-...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/scheme-reports-wg2/755B2E26-8EC5-4E2C-BCAE-0CAF738F20E9%40ephemeral.net.

Per Bothner

unread,
Apr 6, 2020, 3:08:19 PM4/6/20
to scheme-re...@googlegroups.com
On 4/6/20 7:28 AM, Marc Nieper-Wißkirchen wrote:
> What is probably not much work is the following: Change Scheme's "read" into a, say "read*" procedure that reads Scheme datums but Scheme datums wrapped into two special records, namely, say, #<source-position <datum> <pos>> and #<comment <datum> <comment>>.  The #<source-position ...> wrap can appear around every datum (and subdatum) whenever the reader can communicate the source position of the datum; the #<comment ...> wrap will appear around every datum that is preceded by some special comment syntax.

FWIW: The Kawa reader will read each pair (cons) as a PairWithPosition, which is a subclass of Pair
augmented with position information. This makes it somewhat transparent to code that doesn't
care/know about positions.

How do you get the position of a string literal? Well, (almost) every string literal is the
car of some list so the position of the string is the position of the car.
Exceptions are dotted-lists (which are rarely used in Scheme code, except for lambda rest
parameters), and top-level forms (which Kawa deals with by wrapping them in a pseudo-begin form).

--
--Per Bothner
p...@bothner.com http://per.bothner.com/

Arne Babenhauserheide

unread,
Apr 6, 2020, 4:11:03 PM4/6/20
to scheme-re...@googlegroups.com

Edwin Watkeys <e...@poseur.com> writes:
> As a general comment, I’d like to mention that it may be worth thinking about object metadata more broadly and considering whether the larger issue is worth pondering, and if so, how documentation could be implemented within a more general metadata framework.
>
> I menton this for two reasons: Clojure uses its metadata system, which has been put to many valuable uses, to implement doc strings IIRC, and I think that, if doc strings are to be supported, names bound to non-procedures (and/or those non-procedure objects themselves) should be supported.

Regarding this, Guile implements docstrings as a special case of object
properties:


(define my-property (make-object-property))
(set! (PROPERTY OBJ) VAL)
;; A single object property created by ‘make-object-property’ can
;; associate distinct property values with all Scheme values that are
;; distinguishable by ‘eq?’ (ruling out numeric values).
;;
;; Internally, object properties are implemented using a weak key hash
;; table. This means that, as long as a Scheme value with property values
;; is protected from garbage collection, its property values are also
;; protected. When the Scheme value is collected, its entry in the
;; property table is removed and so the (ex-) property values are no longer
;; protected by the table.

(procedure-documentation proc) ; => docstring
(procedure-property proc 'documentation); => docstring
(procedure-properties proc) ; Return the properties associated with PROC, as an association list.
;; and
(set-procedure-properties! proc alist)
(set-procedure-property! proc key value)
;; for other objects
(object-properties obj) ; Return OBJ’s property list.
(object-property obj key)
(set-object-properties! obj alist)
(set-object-property! obj key value)


If you define with docstring
(define (foo) "bar" #t)
you just set the 'documentation property.
(procedure-properties foo)
$1 ((name . foo) (documentation . "bar"))

And you can set the properties directly:
(define (foo) #((foo . bar)) #t)
(procedure-properties foo)
$2 = ((name . foo) (foo . bar))

which can be combined with docstrings:

(define (foo) "myfoo" #((foo . bar)) #t)
(procedure-properties foo)
$3 = ((name . foo) (documentation . "myfoo") (foo . bar))

But in case of conflict, the properties win over the docstring:
(define (foo) "myfoo" #((foo . bar) (documentation . "yourfoo")) #t)
(procedure-properties foo)
$4 = ((name . foo) (foo . bar) (documentation . "yourfoo"))

Alaric Snell-Pym

unread,
Apr 7, 2020, 5:39:31 AM4/7/20
to scheme-re...@googlegroups.com
On 06/04/2020 19:07, Edwin Watkeys wrote:
> Hi all,
>
> As a general comment, I’d like to mention that it may be worth thinking about object metadata more broadly and considering whether the larger issue is worth pondering, and if so, how documentation could be implemented within a more general metadata framework.
>
> I menton this for two reasons: Clojure uses its metadata system, which has been put to many valuable uses, to implement doc strings IIRC, and I think that, if doc strings are to be supported, names bound to non-procedures (and/or those non-procedure objects themselves) should be supported.

I'm interested in this, too.

One of my back-burner thought experiments is a language design whereby
modules aren't just maps from names to things, but something more
general: lists of Prolog-style predicate-logic claims, which can include
"name X is bound to value Y" or "name X is bound to macro-expander Y" as
well as "value Y is a function with these argument/return types" or
"value Y has these pre/post conditions" or "value Y has intended usage
Z, a string of English text".

Note that I propose attaching all this metadata to the values, not the
names - because a name might be bound to a list, and then documentation
attached to the elements of the list, etc.

(I'm hoping I can use that to implement some kind of generic function
system, which is an example of a means of binding values to names that's
more complicated than just "name = value", but I've not figured out the
details of that yet!)

On another note: In the syntax that generates these things, I'm also
toying with having deeper metadata than just a bland "comment" form.
Examples would include:

- Intended usage (useful for exported bindings)

- Example usage (not just a string like "(+ 2 1) => 3", more like an
executable testcase that can be rendered in documentation as well)

- Implementation explanation (useful to wrap around ANY sub-expression,
not just top-level definitions) - let's give this a nice short name like
'?':

(define (total-power-used)
(+ power-used-by-motor
power-used-by-lights
(? "The power used by the light on the motor counts in both "

"the above totals, so we'd count it twice otherwise"
(- power-used-by-motor-light))))

- A FIXME, documenting work that can be done to improve the code, but
which is not required to release it:

(define (find-things-to-do-in-priority-order)
(FIXME "Use a heap to store the task list in order rather than
sorting it every time"
(sort *current-tasks*)))

- A marker for placeholders/stubs for not yet implemented things, which
needs to be done before the code can be released:

(define (check-access-is-permitted user action object)
(PLACEHOLDER "Interface into corporate LDAP server to check group
membership; for now, anything goes"
#t))

...a CI system can issue warnings for placeholders in branch code and
refuse to accept a merge to master with any placeholders in, for instance.

On another note, to avoid too much nesting caused by all these comment
blocks that wrap a value, my syntax also includes a syntactic sugar for
them - "(A . X): Y" -> (A . (X . (Y)))", so you can write this:

(define (some-func)
(? "foo"):
...body...)

to mean:

(define (some-func)
(? "foo"
...body...))

...without needing an extra set of parens around the body that cause it
to be indented further. A reader extension along those lines might be a
reasonable compromise between the (comment STRING BODY) and "STRING"
BODY camps?

> Regards,
> Edwin

Thanks,

--
Alaric Snell-Pym (M7KIT)
http://www.snell-pym.org.uk/alaric/

signature.asc

elf

unread,
Apr 7, 2020, 6:48:10 AM4/7/20
to scheme-re...@googlegroups.com
um, alaric, i think this runs into the halting paradox, or something very much like it, in the general case :)

-elf

Alaric Snell-Pym

unread,
Apr 7, 2020, 8:15:49 AM4/7/20
to scheme-re...@googlegroups.com
On 07/04/2020 11:47, elf wrote:
> um, alaric, i think this runs into the halting paradox, or something very much like it, in the general case :)

If you allow *rules* as well as *assertions* in your modules, then
extracting the semantics / metadata of a module might be
non-terminating, is that what you mean?

Well, don't do that then ;-)

>
> -elf
signature.asc

elf

unread,
Apr 7, 2020, 8:53:58 AM4/7/20
to scheme-re...@googlegroups.com
define the difference between deriving assertions and rules? in some of the cases you mentioned, this would be a problem, i think. i could, of course, be wrong. can any of the formal grammarians put in 2c?

-elf

Alaric Snell-Pym

unread,
Apr 7, 2020, 10:55:34 AM4/7/20
to scheme-re...@googlegroups.com
On 07/04/2020 13:53, elf wrote:
> define the difference between deriving assertions and rules? in some of the cases you mentioned, this would be a problem, i think. i could, of course, be wrong. can any of the formal grammarians put in 2c?

Sure! If you have an input syntax like, for example:

(define-library (example addition)
(import (scheme base))
(export add)
(begin
(define (add x y) "Return the sum of x and y" (+ x y))
(expect (add 1 2) 3)))

...then the "first-class module value" that produces when separately
compiled or whatever might, assuming a fancy type-infering compiler,
consist of these assertions:

(binding 'add <closure value X>)
(type <closure value X> (-> (number number) (number)))
(argument-names <closure value X> '(x y))
(documentation <closure value X> "Return the sum of x and y")
(example <closure value X> (1 2) (3))

These are just "statements of fact" rendered as tuples. All of them can
be derived from the module definition above using existing techniques,
and extracting them from the assertion list is easy - we can ask "What
names are bound in this module?" by finding matches for `(binding
<something> <something>)` in O(N) time, etc.

As opposed to a more conventional implementation of a module which might
just return a kind of alist of bindings, from names to either values or
macro transformers. Instead, we return a list of these assertions about
things.

However, languages like Prolog that operate in this kind of model extend
those simple lists of assertions with the ability to also specify
"rules", such as:

(rule (parent X Y) (son Y X))
(rule (parent X Y) (daughter Y X))
(rule (ancestor X Y) (parent X Y))
(rule (ancestor X Y)
(parent X Z)
(ancestor Z Y))

...which, when combined with some facts like:

(son adam steve)
(son bob steve)
(daughter carol adam)
(son bob carol)

...then lets the system answer questions such as:

(ancestor bob <something>)

...with a list of possible somethings: carol, adam, and steve (but not
bob), by repeatedly applying the rules and assertions until it can find
all the answers. However, it's possible to encode arbitrary computations
in that, and so answering questions in the presence of arbitrary rules
such as the above might, indeed, not terminate.

Although I *am* interested in the possibility of allowing rules as part
of a type system in this manner, because it does let you do some fun
stuff as long as you can find ways to deal with compilation possibly not
terminating, that's outside the scope of using assertions alone as an
example of a more general metadata system that might be used in the
context of doc strings, the original intent of this thread :-)

>
> -elf
>

TTFN,
signature.asc

elf

unread,
Apr 7, 2020, 11:19:07 AM4/7/20
to scheme-re...@googlegroups.com
so, related to previous discussion in a diff thread here... what happens when a procedure can return #f, eol, or a string? even the statements of fact can be arbitrarily complex and (possibly) misleading without further assertions for explanation, which themselves require further assertions, etc.

this becomes even hairier if we have, say, map, or even better, fold... i do not believe any generic iterators or generic generators (if im using the terminology correctly) could be reasonably asserted in such a fashion....

how would macros be handled? macro introspection can even get weirder...

it's a fun idea, i think, but runs into too many problems in a system like scheme. again, just my 2c. in a strongly typed scheme variant it would be interesting to see.

-elf

elf

unread,
Apr 7, 2020, 11:19:14 AM4/7/20
to scheme-re...@googlegroups.com
so, related to previous discussion in a diff thread here... what happens when a procedure can return #f, eol, or a string? even the statements of fact can be arbitrarily complex and (possibly) misleading without further assertions for explanation, which themselves require further assertions, etc.

this becomes even hairier if we have, say, map, or even better, fold... i do not believe any generic iterators or generic generators (if im using the terminology correctly) could be reasonably asserted in such a fashion....

how would macros be handled? macro introspection can even get weirder...

it's a fun idea, i think, but runs into too many problems in a system like scheme. again, just my 2c. in a strongly typed scheme variant it would be interesting to see.

-elf

Alaric Snell-Pym

unread,
Apr 7, 2020, 1:51:07 PM4/7/20
to scheme-re...@googlegroups.com
On 07/04/2020 13:53, elf wrote:
> define the difference between deriving assertions and rules? in some of the cases you mentioned, this would be a problem, i think. i could, of course, be wrong. can any of the formal grammarians put in 2c?

Marc Nieper-Wißkirchen

unread,
Apr 7, 2020, 2:01:37 PM4/7/20
to scheme-re...@googlegroups.com
Am Mo., 6. Apr. 2020 um 19:07 Uhr schrieb Arne Babenhauserheide <arne...@web.de>:

Marc Nieper-Wißkirchen <marc....@gmail.com> writes:

> Am Mo., 6. Apr. 2020 um 15:47 Uhr schrieb Arne Babenhauserheide <
> A string as the first expression is not allowed if you want to have
> internal definitions in the body.

Ah, you’re right. This is a limitation Guile just lifted in 3.0. It was
one of the most jarring things when I learned Scheme.

Neither in the R6RS, nor in the R7RS this restriction was lifted.

I think every modern Scheme could cope with an extension that allows expressions before definitions (as they are allowed in programs).  A procedure body like `(<expr> <definition><expr>)' would be equivalent to `((define-values <dummy-var> <expr>) <definition> <expr>)'.  (Cf the semantics of a program in the R6RS.)


>> What changes is that if the first form in define is a string, and it is
>> not the last form, then this string becomes the docstring.
>>
>> The long form is
>> (set-procedure-property! fun 'documentation "the docstring")
>>
>> But since there is obviously no consensus about this (strong preferences
>> both ways), first creating SRFIs might be a better way forward.
>>
>
> Isn't it more effective to seek consensus first? Otherwise, we have several
> competing SRFIs and still have no consensus.

Can we reach consensus when there are such strong opinions?

If we can, you’re right.

> At least, we can try to agree on the technical advantages and disadvantages
> and make up our minds accordingly.

That sounds good.

Before we think of any syntax, we should answer the following questions:

(1) Is it enough to document top-level definitions or do we want to be able to attach documentation to any value?
(2) Should we make sure that there is absolutely no runtime overhead when documentation is added?
(3) Do we just want to document procedures or any value?
(4) Do we want to be able to document syntactic keywords?
(5) How is the documentation related to the library system?
(6) Do we want structured documentation or just plain strings?
(7) Shall the documentation follow the code-is-data principle of Scheme?
(8) ...

Please note that, say, a yes to question (6) or (7) does not preclude simple documentation strings as long as they can be mapped homomorphically to a more sophisticated system.

-- Marc

 

Best wishes,
Arne
--
Unpolitisch sein
heißt politisch sein
ohne es zu merken

--
You received this message because you are subscribed to the Google Groups "scheme-reports-wg2" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scheme-reports-...@googlegroups.com.

Arne Babenhauserheide

unread,
Apr 7, 2020, 4:26:22 PM4/7/20
to scheme-re...@googlegroups.com

Marc Nieper-Wißkirchen <marc....@gmail.com> writes:
> Before we think of any syntax, we should answer the following questions:
>
> (1) Is it enough to document top-level definitions or do we want to be able
> to attach documentation to any value?
> (2) Should we make sure that there is absolutely no runtime overhead when
> documentation is added?
> (3) Do we just want to document procedures or any value?
> (4) Do we want to be able to document syntactic keywords?
> (5) How is the documentation related to the library system?
> (6) Do we want structured documentation or just plain strings?
> (7) Shall the documentation follow the code-is-data principle of Scheme?
(8) Do we want to be able to access the documentation at runtime?
(9) ?

Arne Babenhauserheide

unread,
Apr 7, 2020, 5:03:05 PM4/7/20
to scheme-re...@googlegroups.com

Arne Babenhauserheide <arne...@web.de> writes:

> Marc Nieper-Wißkirchen <marc....@gmail.com> writes:
>> Before we think of any syntax, we should answer the following questions:

My answers — mind that I’m mostly a user of Scheme, not an implementor.

>> (1) Is it enough to document top-level definitions or do we want to be able
>> to attach documentation to any value?

I think it should be possible to attach documentation to any *variable*.
This can include nested definitions.

Attaching documentation to anything might have too high costs elsewhere.

>> (2) Should we make sure that there is absolutely no runtime overhead when
>> documentation is added?

I think "absolutely no" is too strong a statement. Even the increase in
filesize of the sources can add minimal runtime overhead.

I think it’s important that the system also works, if I only have
bytecode around and not the actual sources (because I don’t want to
force the system to switch to slower source-parsing when I access
docstrings: That would disincentivize using that information).

However keep in mind that we’re talking about r7rs large here. There
could easily be specialized schemes with harsher resource limitations
that don’t implement this.

>> (3) Do we just want to document procedures or any value?

I think any value, but I’m currently happy with the Guile approach of
providing a setter for properties.

I’m pretty sure, however, that I would not want to go back after having
used a define form for variables that allows me to attach documentation
to variables that don’t point to procedures :-)

(define wants-strange-behavior
"irregular and wonderful if set to #t"
#f)

>> (4) Do we want to be able to document syntactic keywords?

Do you mean the results of define-syntax?

>> (5) How is the documentation related to the library system?

I’m not sure. What do you think?

>> (6) Do we want structured documentation or just plain strings?

Both: structured documentation proved to be awesome for doctests, and
I’m sure more uses will be found if we add it.

>> (7) Shall the documentation follow the code-is-data principle of Scheme?

Yes. After having experienced that I can safely say that that’s awesome.

I started my doctests with python-like string-parsing, but the
property-based code-is-data doctests I have now (after people suggested
the properties) are simply much nicer.

> (8) Do we want to be able to access the documentation at runtime?

Yes, definitely. This is great, for example, to build self-testing
procedures, and for docstrings used in --help argument parsing.

(define (check-help args)
"--help shows this message and exits"
(when (member "--help" args)
(string-append "usage: "
(first args) " "
(procedure-documentation parse-args))
(exit 0)))

John Cowan

unread,
Apr 7, 2020, 5:55:50 PM4/7/20
to scheme-re...@googlegroups.com
Here's a little taxonomy that might help:

Should we be able to document:

Objects?
Variables?  Global only?
Variables bound to procedures? Global only?
Macros? Global only?
Record types?  Global only?


--
You received this message because you are subscribed to the Google Groups "scheme-reports-wg2" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scheme-reports-...@googlegroups.com.

Arne Babenhauserheide

unread,
Apr 7, 2020, 6:22:29 PM4/7/20
to scheme-re...@googlegroups.com

Alaric Snell-Pym <ala...@snell-pym.org.uk> writes:
> On another note: In the syntax that generates these things, I'm also
> toying with having deeper metadata than just a bland "comment" form.
> Examples would include:
>
> - Intended usage (useful for exported bindings)
>
> - Example usage (not just a string like "(+ 2 1) => 3", more like an
> executable testcase that can be rendered in documentation as well)
>
> - Implementation explanation (useful to wrap around ANY sub-expression,
> not just top-level definitions) - let's give this a nice short name like
> '?':

You might be interested in the doctest stuff I did:

(define (A)
"returns 'A"
#((tests (test-eqv 'A (A))
(test-assert #t)))
'A)

This already works in Guile and is quite similar to your ? idea, though
not as general.

'?' as name sounds dangerous to me, though: This is already used in
other places (it’s far too common). Basically every special character
for not-absolutely-core-operations like λ and '`, is dangerous, because
it increases complexity for human readers and newcomers.

The path to Haskell-type syntax complexity is short and we should think
thrice before adding anything.

This is why I like the approach in Guile to avoid increasing the number
of chars with special handling by having long-name functions and
re-using existing literal forms.

But that can only be used in define-level granularity right now, so your
form can do more.

> (define (total-power-used)
> (+ power-used-by-motor
> power-used-by-lights
> (? "The power used by the light on the motor counts in both "
>
> "the above totals, so we'd count it twice otherwise"
> (- power-used-by-motor-light))))

(why the multi-strings above, given that there are multi-line strings in
Scheme?)

> - A FIXME, documenting work that can be done to improve the code, but
> which is not required to release it:
>
> (define (find-things-to-do-in-priority-order)
> (FIXME "Use a heap to store the task list in order rather than
> sorting it every time"
> (sort *current-tasks*)))

FIXME looks like something I might want to use as a procedure- or
variable-name; why not fold it into the general metadata form?.

John Cowan

unread,
Apr 7, 2020, 6:23:42 PM4/7/20
to scheme-re...@googlegroups.com
Question mark is not a name, it's a question mark!   In other words:  Should we be able to document any object?  Should we be able to document any variable? etc.


--
You received this message because you are subscribed to the Google Groups "scheme-reports-wg2" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scheme-reports-...@googlegroups.com.

Arthur A. Gleckler

unread,
Apr 7, 2020, 6:55:25 PM4/7/20
to scheme-re...@googlegroups.com

On Tue, Apr 7, 2020 at 11:01 AM Marc Nieper-Wißkirchen <marc....@gmail.com> wrote:

Before we think of any syntax, we should answer the following questions:

(1) Is it enough to document top-level definitions or do we want to be able to attach documentation to any value?

Ideally, I'd like to be able to document all definitions, regardless of level.

I'd also like to be able to document values like procedures and syntax transformers independent of whether they have bindings or where they are bound. It's easy to add an optional slot to both that can hold a documentation string or a key that can be used to fetch one at runtime. For other types, keeping a hash table from values to documentation strings would not be expensive.

(2) Should we make sure that there is absolutely no runtime overhead when documentation is added?

"Absolutely no overhead" is extreme. There are ways to minimize the overhead.

(3) Do we just want to document procedures or any value?

All bindings as well as the values of procedures and syntax transformers.

(4) Do we want to be able to document syntactic keywords?

Yes.

(5) How is the documentation related to the library system?

Unrelated.

(6) Do we want structured documentation or just plain strings?

I'd rather not impose a specific syntax on everyone, but it would be nice to allow implementations to support it. Perhaps reader syntax like the following might work, for example to indicate the use of Markdown markup:

I'm not advocating this specific reader syntax, but just the idea that a string could be labeled concisely to identify what syntax was being used. Alternatively:

In this example, the language would have to treat documentation specially in defining forms in addition to or instead of bare strings.

(7) Shall the documentation follow the code-is-data principle of Scheme?

Absolutely. I would like to avoid magic comments, for example.

(8) …

Arthur A. Gleckler

unread,
Apr 7, 2020, 6:55:56 PM4/7/20
to scheme-re...@googlegroups.com

On Tue, Apr 7, 2020 at 2:55 PM John Cowan <co...@ccil.org> wrote:

Should we be able to document:

Objects?

Yes.

Variables? Global only?

Yes. Once the mechanism is in place, there doesn't seem to be a reason to restrict it to top-level definitions.

Variables bound to procedures? Global only?

Yes. No.

Macros? Global only?

Yes. No.

Record types? Global only?

Good idea, and yes. No.

John Cowan

unread,
Apr 7, 2020, 6:58:42 PM4/7/20
to scheme-re...@googlegroups.com
On Tue, Apr 7, 2020 at 6:55 PM Arthur A. Gleckler <a...@speechcode.com> wrote:

Variables? Global only?

Yes. Once the mechanism is in place, there doesn't seem to be a reason to restrict it to top-level definitions.


Ah, that leads to another question:  Let-bindings, or just definitions?



John Cowan          http://vrici.lojban.org/~cowan        co...@ccil.org
I could dance with you till the cows come home.  On second thought,
I'd rather dance with the cows when you come home.
        --Rufus T. Firefly

Arthur A. Gleckler

unread,
Apr 7, 2020, 7:06:09 PM4/7/20
to scheme-re...@googlegroups.com
On Tue, Apr 7, 2020 at 3:58 PM John Cowan <co...@ccil.org> wrote:
 
Ah, that leads to another question:  Let-bindings, or just definitions? 

I don't see a way to make adding doc strings to let bindings unobtrusive — except bindings of procedures, where there's a natural place to put the documentation string.

John Cowan

unread,
Apr 7, 2020, 7:55:18 PM4/7/20
to scheme-re...@googlegroups.com
That would be a documentation of the object rather than the variable; those ideas are disjoint, and you could have both.

 

--
You received this message because you are subscribed to the Google Groups "scheme-reports-wg2" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scheme-reports-...@googlegroups.com.

Arthur A. Gleckler

unread,
Apr 7, 2020, 8:42:42 PM4/7/20
to scheme-re...@googlegroups.com
On Tue, Apr 7, 2020 at 4:55 PM John Cowan <co...@ccil.org> wrote:

On Tue, Apr 7, 2020 at 7:06 PM Arthur A. Gleckler <a...@speechcode.com> wrote:
 
I don't see a way to make adding doc strings to let bindings unobtrusive — except bindings of procedures, where there's a natural place to put the documentation string.

That would be a documentation of the object rather than the variable; those ideas are disjoint, and you could have both.
 
Yes, of course, but I don't see how to add documentation to a let binding itself and not make it obtrusive.

elf

unread,
Apr 8, 2020, 2:03:09 AM4/8/20
to scheme-re...@googlegroups.com
Am Mo., 6. Apr. 2020 um 19:07 Uhr schrieb Arne Babenhauserheide <
arne...@web.de>:

>
> Marc Nieper-Wißkirchen <marc....@gmail.com> writes:
>
> > Am Mo., 6. Apr. 2020 um 15:47 Uhr schrieb Arne Babenhauserheide <
> > A string as the first expression is not allowed if you want to have
> > internal definitions in the body.
>
> Ah, you’re right. This is a limitation Guile just lifted in 3.0. It was
> one of the most jarring things when I learned Scheme.
>

Neither in the R6RS, nor in the R7RS this restriction was lifted.

I think every modern Scheme could cope with an extension that allows
expressions before definitions (as they are allowed in programs). A
procedure body like `(<expr> <definition><expr>)' would be equivalent to
`((define-values <dummy-var> <expr>) <definition> <expr>)'. (Cf the
semantics of a program in the R6RS.)


> >> What changes is that if the first form in define is a string, and it is
> >> not the last form, then this string becomes the docstring.
> >>
> >> The long form is
> >> (set-procedure-property! fun 'documentation "the docstring")
> >>
> >> But since there is obviously no consensus about this (strong preferences
> >> both ways), first creating SRFIs might be a better way forward.
> >>
> >
> > Isn't it more effective to seek consensus first? Otherwise, we have
> several
> > competing SRFIs and still have no consensus.
>
> Can we reach consensus when there are such strong opinions?
>
> If we can, you’re right.
>
> > At least, we can try to agree on the technical advantages and
> disadvantages
> > and make up our minds accordingly.
>
> That sounds good.
>

Before we think of any syntax, we should answer the following questions:

(1) Is it enough to document top-level definitions or do we want to be able
to attach documentation to any value?

Documenting internal definitions seems to me to break one of the fundamental ideas of internal definitions. Documenting procedure parameters is part of documenting the procedure itself; documenting how everything works internally, externally exposed, removes the line between documentation and commenting.

(2) Should we make sure that there is absolutely no runtime overhead when
documentation is added?

I would say that documentation that incurs runtime overhead is problematic. Try debugging or optimising a tight loop with other people's documentation in small memory.

(3) Do we just want to document procedures or any value?

Exposed values are generally either immutable (constants or special symbols/types) or through what chicken (and others) calls parameters, ie procedures. In the first case, internal docstring-type documentation seems generally unnecessary; in the second, it's already covered.

(4) Do we want to be able to document syntactic keywords?

If they're externally exposed, yes.

(5) How is the documentation related to the library system?

It shouldn't be, unless either a) duplication of the same documentation system is desirable for top-level applications and libraries or b) all code should be in the form of a library. Both of these seem not so good to me.

(6) Do we want structured documentation or just plain strings?

Why does this matter? That is to say, a simple implementation should be able to treat the documentation as a simple string and still be understandable. Any structure should be something that can evolve as a set of keywords or the like, decided through srfis. Trying to define the total future behaviour seems to me to be a) limiting b) a design flaw c) something that will bite us later d) very unschemelike.

(7) Shall the documentation follow the code-is-data principle of Scheme?

Clarify this point, please? Documentation is a string. That a string can be evalable data is inherent in the concept of a scheme string, last I checked.

(8) ...

Please note that, say, a yes to question (6) or (7) does not preclude
simple documentation strings as long as they can be mapped homomorphically
to a more sophisticated system.

-- Marc




If i may add my (probably unwanted at this point) 2c...
there seems to me to be a worrisome trend towards overdefinition in many of the proposals and srfis as of late. This may be a misperception on my part; I have not been as active as i used to be. That said, my understanding of the concept of scheme (and one of the things that makes it so fun to work with and play with and implement and .... etc) is that there's an inherent simplicity, and much of the expressiveness and power stems from the freedom that simplicity brings. A simple system, requiring as few changes as possible, that will serve the purpose of internal run-time documentation, seems to me to be far superior to any over-engineered behemoth that requires changes to every part of the repl and compiler. Making something _extremely_ simple and then seeing how (or if) it evolves in community-useful ways may be a better path than trying to find all the ways that all of us could ever theoretically want to use it ... and it will be less limiting to the next generation who may come up with things that we never thought of.

I am not saying that a complex system is necessary bad or a simple one is necessarily good, just that it might be a good idea to follow a route that requires the least amount of future correction if it turns out not to be what we had hoped.

(I will be unavailable for the next few days due to Passover, so I'm sorry in advance for putting this point out at this time. I will try to catch up on the discussion on Saturday night my time. Only mentioning so you're all free to flame or disagree and not be worried that I got offended and left in a huff. :) )

-elf

>
> Best wishes,
> Arne
> --
> Unpolitisch sein
> heißt politisch sein
> ohne es zu merken
>
> --
> You received this message because you are subscribed to the Google Groups
> "scheme-reports-wg2" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to scheme-reports-...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/scheme-reports-wg2/87ftdgr1ai.fsf%40web.de
> .
>

--
You received this message because you are subscribed to the Google Groups "scheme-reports-wg2" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scheme-reports-...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/scheme-reports-wg2/CAEYrNrRZ0%3D8teAG%2BN3Xz7Hm3qyzNGzmsK8imWhGiS7ruemapkQ%40mail.gmail.com.

Arne Babenhauserheide

unread,
Apr 8, 2020, 2:55:00 AM4/8/20
to scheme-re...@googlegroups.com

John Cowan <co...@ccil.org> writes:

> Question mark is not a name, it's a question mark!

It can be used as the name of a variable or procedure:

(define (? arg) arg)
(? '?-is-the-name)

That is what I mean by name.

And there are many domains in which ? is a natural choice for a name. If
we have a standard procedure take it, all those domains are prevented
from using it.

imaginary example that first came to my head: (solve (> 5 ?))

Arne Babenhauserheide

unread,
Apr 8, 2020, 3:33:04 AM4/8/20
to scheme-re...@googlegroups.com

elf <e...@ephemeral.net> writes:
> Before we think of any syntax, we should answer the following questions:
>
> (1) Is it enough to document top-level definitions or do we want to be able
> to attach documentation to any value?
>
> Documenting internal definitions seems to me to break one of the fundamental ideas of internal definitions. Documenting procedure parameters is part of documenting the procedure itself; documenting how everything works internally, externally exposed, removes the line between documentation and commenting.

Whether the documentation of the inner definition is accessible outside
the outer definition is a second question.

You’re right there: I would not want this to be visible from outside the
outer definition.

(define (foo)
(define (bar) "keep the bar for visibility"
(display (procedure-documentation bar)) ;; should work
#t)
(bar))

(display (procedure-documentation bar)) ;; should not

> (2) Should we make sure that there is absolutely no runtime overhead when
> documentation is added?
>
> I would say that documentation that incurs runtime overhead is problematic. Try debugging or optimising a tight loop with other people's documentation in small memory.

To me it depends on the amount of runtime overhead. It does not have to
be overhead when calling the procedure. It could just be overhead when
loading the program (due to additional operations when the procedure is
defined).

Something that makes procedure-calls slower would be a no-go for me.

> (3) Do we just want to document procedures or any value?
>
> Exposed values are generally either immutable (constants or special symbols/types) or through what chicken (and others) calls parameters, ie procedures. In the first case, internal docstring-type documentation seems generally unnecessary; in the second, it's already covered.

There are mutable values. For example:

(define foo (list 'foo))
(list-set! foo 0 'bar)

> (7) Shall the documentation follow the code-is-data principle of Scheme?
>
> Clarify this point, please? Documentation is a string. That a string can be evalable data is inherent in the concept of a scheme string, last I checked.

A string needs to be parsed for evaluation. code-is-data would mean that
you for example have an alist.

> A simple system, requiring as few changes as possible, that will serve
> the purpose of internal run-time documentation, seems to me to be far
> superior to any over-engineered behemoth that requires changes to
> every part of the repl and compiler.

I think so, too.

Simplicity is the reason why I prefer Scheme over Python.

> I am not saying that a complex system is necessary bad or a simple one is necessarily good, just that it might be a good idea to follow a route that requires the least amount of future correction if it turns out not to be what we had hoped.
>
> (I will be unavailable for the next few days due to Passover, so I'm sorry in advance for putting this point out at this time. I will try to catch up on the discussion on Saturday night my time. Only mentioning so you're all free to flame or disagree and not be worried that I got offended and left in a huff. :) )

What’s passover?

Best wishes,
Arne

Alaric Snell-Pym

unread,
Apr 8, 2020, 4:50:13 AM4/8/20
to scheme-re...@googlegroups.com
On 07/04/2020 16:18, elf wrote:
> so, related to previous discussion in a diff thread here... what happens when a procedure can return #f, eol, or a string? even the statements of fact can be arbitrarily complex and (possibly) misleading without further assertions for explanation, which themselves require further assertions, etc.
>
> this becomes even hairier if we have, say, map, or even better, fold... i do not believe any generic iterators or generic generators (if im using the terminology correctly) could be reasonably asserted in such a fashion....

Those are all issues for the type system / inference algorithms of the
language... all I'm discussing here is a way of representing metadata
such as those type declarations, wherever they come from and whatever
form they may take!

Per Bothner

unread,
Apr 8, 2020, 6:18:15 AM4/8/20
to scheme-re...@googlegroups.com
On 4/7/20 11:54 PM, Arne Babenhauserheide wrote:
> And there are many domains in which ? is a natural choice for a name. If
> we have a standard procedure take it, all those domains are prevented
> from using it.

? is used for the Kawa conditional pattern-matching operator:

https://www.gnu.org/software/kawa/Conditionals.html#idm45230723555216

Example:

(if (? x::integer expression)
(+ x 1) ;; if pattern matches, x is bound
'invalid) ;; if pattern does not match, x is unbound
--
--Per Bothner
p...@bothner.com http://per.bothner.com/

Marc Nieper-Wißkirchen

unread,
Apr 8, 2020, 8:20:43 AM4/8/20
to scheme-re...@googlegroups.com
Am Mi., 8. Apr. 2020 um 01:55 Uhr schrieb John Cowan <co...@ccil.org>:


On Tue, Apr 7, 2020 at 7:06 PM Arthur A. Gleckler <a...@speechcode.com> wrote:
On Tue, Apr 7, 2020 at 3:58 PM John Cowan <co...@ccil.org> wrote:
 
Ah, that leads to another question:  Let-bindings, or just definitions? 

I don't see a way to make adding doc strings to let bindings unobtrusive — except bindings of procedures, where there's a natural place to put the documentation string.

That would be a documentation of the object rather than the variable; those ideas are disjoint, and you could have both.

I think this is one of the main points and one that hasn't mentioned explicitly a lot:

Do we want to document bindings or do we want to document values?

As far as simplicity goes, a single documentation system should either attach documentation to the former or to the latter.  In this regard, both options are orthogonal.

Attaching documentation to values seems to be the simpler approach at first, but it has a number of disadvantages:

(1) Documentation cannot be easily attached to non-structured values.  It has been suggested to use a weak hash-table, whose equality predicate would most likely be `eq?' or `eqv?'.  But what would happen if we documented SRFI 143 with its `fx-width' value, which may in some implementation be 24.  We do not want attach documentation to the number 24.

(2) In Scheme, a procedure object is in fact a closure.  Usually, we don't want to attach the documentation to the closure but to the underlying `lambda', which doesn't appear as a value, though.

(3) While documentation attached to values is easy to use at the REPL, code has to be run to get access to the values. In general, I am thinking of an intelligent IDE, we don't want code to be run before we retrieve documentation (and other meta-information).

Thus, I think what we really want is to attach documentation to bindings, that is to variables and keywords.

For define/define-syntax forms, we have already seen a few ways how extend their syntax so that documentation can be attached.  We can think of syntax how to attach documentation to formal arguments as well, which will not only cover the case of documenting procedure arguments but also of let(rec)(*)(-syntax)-bindings.

Marc



 

--
You received this message because you are subscribed to the Google Groups "scheme-reports-wg2" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scheme-reports-...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/scheme-reports-wg2/CALnw4LKC2mHCo8P93wC%2BOCsY%2B%2B7qMTJ5bDhJ%3Dr8AuAe0xTCe9g%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "scheme-reports-wg2" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scheme-reports-...@googlegroups.com.

Arthur A. Gleckler

unread,
Apr 8, 2020, 1:53:44 PM4/8/20
to scheme-re...@googlegroups.com
On Wed, Apr 8, 2020 at 5:20 AM Marc Nieper-Wißkirchen <marc....@gmail.com> wrote:
 
(1) Documentation cannot be easily attached to non-structured values.  It has been suggested to use a weak hash-table, whose equality predicate would most likely be `eq?' or `eqv?'.  But what would happen if we documented SRFI 143 with its `fx-width' value, which may in some implementation be 24.  We do not want attach documentation to the number 24.

That's why you want to be able to attach documentation to variables as well as values.  While the implementation of documentation for bindings and for values is orthogonal, they are conceptually strongly related, which is why it's good to discuss them both at the same time.
 
(2) In Scheme, a procedure object is in fact a closure.  Usually, we don't want to attach the documentation to the closure but to the underlying `lambda', which doesn't appear as a value, though.

I disagree.  I want the documentation to be attached to the closure itself so that my IDE and REPL can ask for the documentation no matter what variable or data structure it is found in.
 
(3) While documentation attached to values is easy to use at the REPL, code has to be run to get access to the values. In general, I am thinking of an intelligent IDE, we don't want code to be run before we retrieve documentation (and other meta-information).

Actually, I've always thought that the runtime heap is the ultimate data structure for keeping meta-information, and that all the approximations that modern editors do with regular expressions, elaborate partial parsing systems, etc. are inferior to read — or to a version of read that records line numbers, source file pathnames, etc.  One of the wonderful things about Scheme is that I can load code even when it is unfinished, e.g. even when some variables are unbound and the code doesn't type check.  I can do experiments, run parts of the code, find out what identifiers there are, etc.  I would like to add the ability to extract documentation strings to that list of capabilities.

Marc Nieper-Wißkirchen

unread,
Apr 9, 2020, 2:40:18 AM4/9/20
to scheme-re...@googlegroups.com
Am Mi., 8. Apr. 2020 um 19:53 Uhr schrieb Arthur A. Gleckler <a...@speechcode.com>:
On Wed, Apr 8, 2020 at 5:20 AM Marc Nieper-Wißkirchen <marc....@gmail.com> wrote:
 
(1) Documentation cannot be easily attached to non-structured values.  It has been suggested to use a weak hash-table, whose equality predicate would most likely be `eq?' or `eqv?'.  But what would happen if we documented SRFI 143 with its `fx-width' value, which may in some implementation be 24.  We do not want attach documentation to the number 24.

That's why you want to be able to attach documentation to variables as well as values.  While the implementation of documentation for bindings and for values is orthogonal, they are conceptually strongly related, which is why it's good to discuss them both at the same time.

While both are about documentation, I am not sure how much the concepts are related. In any case, it would be good if whatever solution we find for either concept it does not step on the toes of the other.
 
 
(2) In Scheme, a procedure object is in fact a closure.  Usually, we don't want to attach the documentation to the closure but to the underlying `lambda', which doesn't appear as a value, though.

I disagree.  I want the documentation to be attached to the closure itself so that my IDE and REPL can ask for the documentation no matter what variable or data structure it is found in.

Through the closure value, the Scheme system could access the lambda form and thus any documentation attached to the code. Or do you really want to have different documentation on different closures of the same procedure?
 
 
(3) While documentation attached to values is easy to use at the REPL, code has to be run to get access to the values. In general, I am thinking of an intelligent IDE, we don't want code to be run before we retrieve documentation (and other meta-information).

Actually, I've always thought that the runtime heap is the ultimate data structure for keeping meta-information, and that all the approximations that modern editors do with regular expressions, elaborate partial parsing systems, etc. are inferior to read — or to a version of read that records line numbers, source file pathnames, etc.  One of the wonderful things about Scheme is that I can load code even when it is unfinished, e.g. even when some variables are unbound and the code doesn't type check.  I can do experiments, run parts of the code, find out what identifiers there are, etc.  I would like to add the ability to extract documentation strings to that list of capabilities.

The Racket editor's understanding of the source is very elaborate, I think.  It doesn't have to run the code, though. While at the REPL, this may be true, I don't think that you always want to run code to retrieve documentation.  Running general code has usually side effects.  Moreover, one run may differ from the next one.  We should also note that there are Scheme systems that are not REPL-centered but use the usual compile-and-run cycle.

Anyway, having the ability to attach information to values in running code (through weak hash tables, for example) is good to have, but does not render unnecessary documentation attached to bindings.  As said above, these are really orthogonal concepts.

Marc

--
You received this message because you are subscribed to the Google Groups "scheme-reports-wg2" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scheme-reports-...@googlegroups.com.

Marc Nieper-Wißkirchen

unread,
Apr 9, 2020, 2:52:59 AM4/9/20