Production Usage of Shen

[Greg Spurrier]

I've been working on a small application in Shen. It's been a good experience and has pointed to a few things that I can do in ShenRuby to make the process easier.

There are also a few things that I've noticed about Shen itself that would be nice to address for using it in stand-alone applications. For lack of a better term, let's call this "production" usage:

• Suppress output when loading files, e.g., what is accomplished by setting *hush* to true.
• When a partial function is encountered, raise an exception rather than prompting the user for whether to track the function and waiting for input. This is particularly problematic when *hush* is true, because it appears to the application user that the program is stuck.
• Provide access to the command line arguments with which the application was launched. In ShenRuby I can use native calls to access this, but it would be nice if they were available in a platform-agnostic way. E.g., perhaps adding a *argv* variable containing a vector of the arguments and having the Kl port populate it.
• Have a standard way to exit a Shen program, e.g., (exit ReturnCode). Again, I can do this with native calls, but it is non-portable.

There are probably a few other things as well. Consider the above as a starting point for discussion. What other things have people found as lacking for developing general purpose applications in pure Shen?

Greg

[Ramil Farkhshatov]

Greg Spurrier <greg.s...@gmail.com> wrote:

> I've been working on a small application in Shen. It's been a good
> experience and has pointed to a few things that I can do in ShenRuby to
> make the process easier.
>
> There are also a few things that I've noticed about Shen itself that would
> be nice to address for using it in stand-alone applications. For lack of a
> better term, let's call this "production" usage:
>

> - Suppress output when loading files, e.g., what is accomplished by
> setting *hush* to true.

There is a workaround for hiding initial REPL prompt as well (see shen_run[1]).

> - When a partial function is encountered, raise an exception rather than

> prompting the user for whether to track the function and waiting for input.
> This is particularly problematic when *hush* is true, because it appears to
> the application user that the program is stuck.

> - Provide access to the command line arguments with which the

> application was launched. In ShenRuby I can use native calls to access
> this, but it would be nice if they were available in a platform-agnostic
> way. E.g., perhaps adding a *argv* variable containing a vector of the
> arguments and having the Kl port populate it.

shen_run[1] provides command line arguments access but the whole thing
is a workaround for Shen.

> - Have a standard way to exit a Shen program, e.g., (exit ReturnCode).

> Again, I can do this with native calls, but it is non-portable.

Also is handled by shen_run[1], but currently it handles only
success/fail exit statuses.

> There are probably a few other things as well. Consider the above as a
> starting point for discussion. What other things have people found as
> lacking for developing general purpose applications in pure Shen?

* Lack of error output stream. So I am unable to print warning messages
without spoiling output data when using Shen in such a manner:

./do-something.shen | some-processing

or

./do-something.shen >some-data

* 'compile' that works on generic streams instead of lists. That would
allow parsing some arbitrary sized input like log files and other
generally infinite data streams but consisting of toplevel lexems of
limited size.

I was going to write about those two issues separatedly but after I
finish with my current Shen-related tasks. But could not leave this mail
untouched. (-;

1. https://github.com/gravicappa/shen_run

[Greg Spurrier]

Good point about not having a standard error stream. That is definitely something I would like to see added.

Greg

--
You received this message because you are subscribed to the Google Groups "Shen" group.
To unsubscribe from this group and stop receiving emails from it, send an email to qilang+un...@googlegroups.com.
To post to this group, send an email to qil...@googlegroups.com.

Visit this group at http://groups.google.com/group/qilang.
For more options, visit https://groups.google.com/d/optout.

[Mark Tarver]

I think that with a partial function error, the first thing I want to know is what argument caused the partial error.   Hence I immediately track the function and having a facility which gives me the power to do this without typing (track <function name>) is a boon.     If you are using *hush* at all, it should only be when you have got your app running smoothly w.o. errors; so there should not be a conflict here.  

Of course if you are wanting to redirect all your errors to a file this is not a help (of this more below); though you can simply make your erring partial function total and have your own 'fall through' message which is easy to do.  

The idea of having a standard error stream can be implemented but it presents certain issues.  Presumably the idea is to send the error messages to a file.  This is easy to do but the question then arises as to what the process is supposed to do afterwards.  In particular does it return something or does it abort?

If it aborts then all you have got is a file with a single error in it which IMO is less useful than a REPL error message.   If it is allowed to continue then you must return some value and this value will be either type invariant or not.  If it is type invariant (e.g. you always return skip) then the type integrity will be broken.  If it is not then the user will have to arrange to supply the default object and write the original message to a file.

This is what trap-error already does and it is not hard to use trap-error to write the original message to a file and to supply a default object.

Regarding command line stuff - I take it your mean the series of commands that actually start up Shen - I think this is too language/OS specific for me to put this in a standard.  Does Ramil's work operate outside CL?  

Mark

To unsubscribe from this group and stop receiving emails from it, send an email to qilang+unsubscribe@googlegroups.com.

[Bruno Deferrari]

Unix-like OSes and Windows offer three standard streams, STDIN (where
input comes from), STDOUT (where output goes to), and STDERR (where
errors are sent out, by default it is the same as STDOUT). Those
streams can be redirected when calling the program (so for example,
STDERR can be redirected to a null output so that error messages don't
mess with the program's output), but this is something that is done by
whoever is invoking the program, and the program doesn't need to know
about this. The error stream is pretty universal, and adding it to the
spec shouldn't be an issue IMO (all ports should be able to implement
it trivially).

Right now Shen doesn't know about STDERR, but thats something that
should be easy to add, and even if Shen itself doesn't use it
internally, it is something that is very useful to have available when
writing programs, and it would be good to be able to do so portably
across Shen implementations.

Regarding other issues, I think it all boils down to Shen only having
REPL semantics defined right now, but not "standalone program"
semantics (how things should behave when a program is run but not
through the REPL).

Some of the issues raised by Greg would have to be solved internally
in Shen (what happens when a partial function is found, load printing
stuff, etc). Having those functions be customisable through variables
would be a good start (for example, have *partial-function-handler*
point to a lambda that will be called handle partial functions, and
that defaults to a handler that implements the current behaviour, same
for when the REPL has to print some information like in `load`,
*information-printer* or something like that).

Other raised issues are more a mater of port maintainers to decide on
a common interface and implement it (a mechanism to access command
line arguments, be it either a `main` function that gets called with a
list like with shen_run, a global variable that stores those
arguments, or a function that returns that list). For the later we
could have something like Scheme's SRFIs[1], and have anything that
gets universally accepted and that makes sense to be part of the core,
be integrated into the spec in the future. Either the wiki or a
repository on GitHub with markdown files could be used for these
proposals.

Just an idea.

[1] http://srfi.schemers.org

>>> email to qilang+un...@googlegroups.com.

>>> To post to this group, send an email to qil...@googlegroups.com.
>>> Visit this group at http://groups.google.com/group/qilang.
>>> For more options, visit https://groups.google.com/d/optout.
>>
>>

> --
> You received this message because you are subscribed to the Google Groups
> "Shen" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to qilang+un...@googlegroups.com.
> To post to this group, send email to qil...@googlegroups.com.

> Visit this group at http://groups.google.com/group/qilang.
> For more options, visit https://groups.google.com/d/optout.

--
BD

[Bruno Deferrari]

Here is Common Lisp's STDERR handle (the *error-output* variable):

http://www.lispworks.com/documentation/HyperSpec/Body/v_debug_.htm#STtrace-outputST

I see Common Lisp also defines other special streams, *trace-output*,
and the *debug-io* bidirectional stream for debugging purposes:

http://www.lispworks.com/documentation/HyperSpec/Body/26_glo_d.htm#debug_iSLo

Those are Common Lisp specific, but I would say that an error output
stream is universal.

--
BD

[Mark Tarver]

The issue is one of logic and not what CL or anything else does - CL is full of stuff I wouldn't want to import.  The only type secure way override error handling is for the user to arrange to redirect errors himself and supply the defaults - and the code is very easy to write, but needs human intelligence to apply. 

Mark

[Ramil Farkhshatov]

Mark Tarver <dr.mt...@gmail.com> wrote:

> I think that with a partial function error, the first thing I want to
> know is what argument caused the partial error. Hence I immediately
> track the function and having a facility which gives me the power to
> do this without typing (track <function name>) is a boon.

Workflow of another programmers may be different. In case of 'partial
error' I first look into the code and often find a typo or a mistake
without running it. Also the program couldn't know that there is a REPL
in standard input but not a data stream (see example below).

> If you are using *hush* at all, it should only be when you have got
> your app running smoothly w.o. errors; so there should not be a
> conflict here.

1. It is hard to write a program that runs 100% smoothly on a first
*production* runs especially when it's processing external input.

2. I would have no idea on what's happening if a complex application
like

generate-or-get-some-output | ./process-data.shen | store-or-send-results

doesn't store/send me results because it silently waits for 'y' in
standard input. And that's another issue: there might be actual data
on standard input at the moment.

> Of course if you are wanting to redirect all your errors to a file this is
> not a help (of this more below); though you can simply make your erring
> partial function total and have your own 'fall through' message which is
> easy to do.

> The idea of having a standard error stream can be implemented but it
> presents certain issues. Presumably the idea is to send the error
> messages to a file. This is easy to do but the question then arises
> as to what the process is supposed to do afterwards. In particular
> does it return something or does it abort?

It just goes on. Writing to standard error stream is no different than
writing to any other streams for the writer's point of view (there is
buffering difference but it is handled by host language/runtime).

> If it aborts then all you have got is a file with a single error in it
> which IMO is less useful than a REPL error message. If it is allowed to
> continue then you must return some value and this value will be either type
> invariant or not. If it is type invariant (e.g. you always return skip)
> then the type integrity will be broken. If it is not then the user will
> have to arrange to supply the default object and write the original message
> to a file.
>
> This is what trap-error already does and it is not hard to use trap-error to
> write the original message to a file and to supply a default object.
>
> Regarding command line stuff - I take it your mean the series of commands
> that actually start up Shen - I think this is too language/OS specific for
> me to put this in a standard.

This is host specific indeed. And I find it rather organizational
issue. We can have a series of 'implementation standards' like SRFI[2]
(mentioned earlier in other message) for Scheme language. So any porter
would use similar interface for such facilities that would help whole
'run everywhere' business.

> Does Ramil's work operate outside CL?

It is implementation agnostic, but alas there is no Windows version of
shen_run.

2. http://srfi.schemers.org/

[Mark Tarver]

Well 'it just goes on' doesn't really address the issue of what is returned.  The writing part is trivial.  

[Mark Tarver]

I've repeated myself twice now which I generally don't do; so I'll bow out of this debate before going round in cricles.   Willi may have something to add when he returns from his short break.

Mark

--
You received this message because you are subscribed to the Google Groups "Shen" group.

To unsubscribe from this group and stop receiving emails from it, send an email to qilang+un...@googlegroups.com.

To post to this group, send email to qil...@googlegroups.com.

[Raoul Duke]

> If you are using *hush* at all, it should only be when you have got your app running smoothly w.o. errors; so there should not be a conflict here.

I wonder if that is one assumption that is not valid cf. Ramil's
comments vis-a-vie external inputs.

[Greg Spurrier]

On Tue, Apr 21, 2015 at 6:06 AM, Bruno Deferrari <uti...@gmail.com> wrote:

Unix-like OSes and Windows offer three standard streams, STDIN (where
input comes from), STDOUT (where output goes to), and STDERR (where
errors are sent out, by default it is the same as STDOUT). Those
streams can be redirected when calling the program (so for example,
STDERR can be redirected to a null output so that error messages don't
mess with the program's output), but this is something that is done by
whoever is invoking the program, and the program doesn't need to know
about this. The error stream is pretty universal, and adding it to the
spec shouldn't be an issue IMO (all ports should be able to implement
it trivially).

Right now Shen doesn't know about STDERR, but thats something that
should be easy to add, and even if Shen itself doesn't use it
internally, it is something that is very useful to have available when
writing programs, and it would be good to be able to do so portably
across Shen implementations.

Agreed. I would not expect that adding *sterror* to the spec alongside *stinput* and *stoutput* to be controversial. Having it available in a standard way for application developers gives them the option of sending output to the error stream in a portable way.

Regarding other issues, I think it all boils down to Shen only having
REPL semantics defined right now, but not "standalone program"
semantics (how things should behave when a program is run but not
through the REPL).

Exactly. For Shen to reach wide spread use by programmers, it must be possible to use it to create programs to be used by non-programmers. The end user of the application should not need to even know that the program was written in Shen.

The question is: do we want this goal to be achievable in pure, portable Shen?

 

For me, the answer is yes, but we've got work to do to make this a reality.

Some of the issues raised by Greg would have to be solved internally
in Shen (what happens when a partial function is found, load printing
stuff, etc). Having those functions be customisable through variables
would be a good start (for example, have *partial-function-handler*
point to a lambda that will be called handle partial functions, and
that defaults to a handler that implements the current behaviour, same
for when the REPL has to print some information like in `load`,
*information-printer* or something like that).

This is a good suggestion and would be a good start.

 

Other raised issues are more a mater of port maintainers to decide on
a common interface and implement it (a mechanism to access command
line arguments, be it either a `main` function that gets called with a
list like with shen_run, a global variable that stores those
arguments, or a function that returns that list). For the later we
could have something like Scheme's SRFIs[1], and have anything that
gets universally accepted and that makes sense to be part of the core,
be integrated into the spec in the future. Either the wiki or a
repository on GitHub with markdown files could be used for these
proposals.

Just an idea.

[1] http://srfi.schemers.org

This is an interesting idea and I expect that, if the number of actively-maintained ports increases in the future, it will become necessary. For now, I'd hope that we can address the basics that are required for stand-alone, non-REPL use here and make the necessary modifications and additions to the spec directly.

If we discover anything that truly has to depend on the host language, then the SRFI idea is a good one to maintain as much cross-port commonality as possible for those features.

Greg

[Bruno Deferrari]

Agree. Note though that SRFIs are not mean for changes to Scheme
itself (even though they may be considered for future versions). They
exist as a reference for Scheme implementors (that in the case of
Scheme, are too many). Lets say you come up with an API for using
sockets, you like it and think it is very Shen-like and other ports
could implement it, so you share it with others so that we can discuss
it and agree on a common API. Each port can then expose this API to
users (even if they also provide a different API), but it is up to
each port, and Shen itself does not change.

What is important here is that except for some particular cases, it is
not even necessary for the port maintainer to have to take care of
implementing the SRFI, a third party can do it (at least, as long as
the port provides access to the underlying platform). If I don't feel
like implementing the sockets API because I'm not interested in
network programming or don't have time to do so, someone else who is
interested can do it and provide it as a library.

It is also common for SRFIs to (in addition of defining and
documenting the expected API) provide a portable implementation that
will not require any port to change anything for the functionality to
be available. But Scheme implementations are free (and in some cases
encouraged) to implement the API natively (for performance or other
reasons) as long as they conform to the defined API (SRFI-4[1] and
SRFI-[9] are two good examples of SRFIs with portable implementations
that are very commonly implemented natively).

For almost everything that has been discussed here, it is just up to
us to decide on a common interface and provide it without needing to
change the Shen spec.

So:

- Error output stream: could be named *erroutput*, easy (1 line) in
chibi-shen and I'm sure it is easy in other ports

- Access to command-line arguments: could be done like shen_run
(function named main that receives a list), easy in chibi-shen too,
and probably easy in other ports too (and for ports that don't
implement it natively, there is shen_run). Could also be a variable as
you suggest. Haven't thought about it much but I don't have any strong
preference for now.

- Suppressing output, ports could set *hush* to true when running in
"standalone program" mode

- Standard way to exist: providing (exit N) sounds good to me, and I
don't think it is a hard thing to do in any port.

The one that I think would ideally be done as an internal change in
Shen but can still be changed from the outside:

- Partial function handling: if running in "standalone program" mode,
you could forcibly overwrite `shen.f_error`[3] so that it raises an
error, but here you are depending on Shen's internals that may change
in the future. Alternatively, when compiling your programs have
`shen.err-condition`[4] return code to trigger an error as the last
condition instead of a call to `shen.f_error`.

Then there is `compile` working with arbitrary streams (I would also
like that btw, but then also maybe a different library should be
written), but that requires a direct change to Shen-YACC, because the
code it generates uses `hd` / `tl` /`cons?`. This generated Kl code
could be post-processed to change those calls, but sounds too
complicated and fragile to me, so, not really a solution.

[1] http://srfi.schemers.org/srfi-4/srfi-4.html
[2] http://srfi.schemers.org/srfi-9/srfi-9.html
[3] https://github.com/Shen-Language/shen-sources/blob/shen-18.1/sources/track.shen#L33-L39
[4] https://github.com/Shen-Language/shen-sources/blob/shen-18.1/sources/core.shen#L388-L389

> Greg

[Bruno Deferrari]

I think you two are discussing different issues here.

One is having a standard error stream as part of Shen just like standard input and output, without having the implementations do it just because they want to. Having that is useful when writing programs, independently of if Shen itself makes use of it or not. I think this is what Ramil was talking about.

The other is how Shen itself will behave when an error occurs (in particular when it finds a partial function), and I think here is where it is useful to distinguish between REPL semantics and standalone program semantics.

Right now Shen implements REPL semantics, which are interactive (for example, notifying the user about a function being partial, and asking him if he wants to track it). This is useful, and what you want when you are developing your program.

In standalone program semantics, this would be just a runtime error (a "Partial function <name> with argument <arg>" error or something like that, it is up to the program what to do with that). This is what OCaml and Haskell do when this is catched at runtime. In any situation where Shen would normally ask the programmer for input, in this mode an error would be raised.

This is what I get in OCaml if I compile a program with a partial function and tell the compiler to compile it anyway (the last line is the important one, an exception was raised at runtime:

faust /tmp > ./prog
Fatal error: exception Match_failure("p.ml", 1, 8)

> --
> You received this message because you are subscribed to the Google Groups
> "Shen" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to qilang+un...@googlegroups.com.
> To post to this group, send email to qil...@googlegroups.com.

>
> Visit this group at http://groups.google.com/group/qilang.
> For more options, visit https://groups.google.com/d/optout.

--
BD

[Mark Tarver]

The interactive part is not the issue;  nor is the writing.  Type theoretically, errors are not just like any other output and cannot be piped to a file and then 'the process runs on'.   This is meaningless in a functional context.  The user has to determine what is done in the individual case and no computer can make that choice for you unless you just want a file with one error message in it.   In fact it is easy to devise a package that redirects errors.

(package errstream [errout init-errs]

(defmacro redirect-error-macro

  [errout X Default] -> (protect [trap-error X [/. E [let Err [error-to-string E]

                                                          Record [errstream Err]

                                                          Default]]]))

                                                          

(define init-errs

  -> (set *errs* ""))

                                                      

(define errstream

  Err -> (let Errs (set *errs* (@s (value *errs*) "c#13;" Err))

              Open (open "errs.shen" out)

              Write (pr Errs Open)

              Close (close Open)

              Err))

\\ initialise error reports              

(init-errs)              

              

(declare errstream [string --> string])

(declare init-errs [--> string]))

______________________________________________________________________

(17+) (errout (/ 1 0) 42)

42 : number

(18+) (errout (/ 1 0) 42)

42 : number

(19+) (errout (/ 1 0) 42)

42 : number

              

gives in errs.shen

/: division by zero

/: division by zero

/: division by zero

As regards partial function;  IMO you should not be running production code with partial function errors.  And disabling diagnostics before debugging your program is asking for trouble. But if the default error handling is not what you want, the better solution is to fill out the partial functions with your own meaningful default code which again will depend on what you want to do and is likely to be highly dependent on individual cases.

Mark

> email to qilang+unsubscribe@googlegroups.com.

[Mark Tarver]

2. I would have no idea on what's happening if a complex application 
like 
    
    generate-or-get-some-output | ./process-data.shen | store-or-send-results 

doesn't store/send me results because it silently waits for 'y' in 
standard input.  And that's another issue:  there might be actual data 
on standard input at the moment. 

I'll place a flag to disable interaction set by default to interaction.

Mark

On Tuesday, April 21, 2015 at 6:23:59 PM UTC+1, Ramil Farkhshatov wrote:

Mark Tarver <dr.mt...@gmail.com> wrote: 2. I would have no idea on what's happening if a complex application 

[Mark Tarver]

It is implementation agnostic, but alas there is no Windows version of
shen_run.

2. http://srfi.schemers.org/

If you can add some doc,  I'll put this in standard lib.

Mark