Pyret Command Line REPL

[Benjamin Attal]

Hey all,

I've been working on a command line repl for Pyret over the past 3-4 weeks, and I finally got the go-ahead to send it out to the discuss group. Woohoo!

You can clone my forked copy here, and checkout my work on the "repl" branch.  After building pyret normally, you can run it with `node src/scripts/repl-cli.js` from the main folder. Here's some information that might be handy:

- Enter or return will either add a newline to the current command or run the command.  The repl detects automatically if you intend to add a newline. For example, pressing enter on the following command will run it --

>> x(1)

     ^

But if you press enter on a line like:

>> fun x(a):

                  ^

Then a newline will be added.  To run multiline commands, add a newline at the end of the command by pressing enter, and then press enter again on the blank line like so --

>> fun x(a):

>>     a + 1

>> end

>> 

    ^

- Left and right arrow keys navigate through commands as would be expected normally.

- Up and down arrow keys navigate forward and backward through history, or through the lines of a multiline command.

- Double tapping the up key while at the top of a multiline command will go backwards in history.  

- Double tapping the down key while at the bottom of a multiline command will go forward in history.  

- Pressing tab on a line within a multiline command will indent that line correctly (most likely).

- Double tapping the tab key will indent an entire command, multiline or not. 

- Ctrl-c exits out of a command if you are currently typing one, and exits the program on empty prompts.

- You can define multiple functions in a single command like so:

>> fun times-eight(a):

>>    x(a) * 2

>> end

>> fun x(b):

>>     y(b) * 2

>> end

>> fun y(c):

>>    c * 2

>> end

>> 

    ^

Windows / linux folk -- there are most likely some compatibility issues that I have yet to discover, but it would terrific if you could give the repl whirl and tell me what happens, :D 

Feedback, suggestions, and bug reports (on everything from colors to smoothness of the experience) are welcome and encouraged! Feel free to create an issue on github, or shoot me or joe.p...@gmail.com an email. 

Enjoy! Or don't -- in which case let me know that you didn't.

~Ben Attal

[Benjamin Attal]

I should mention that you can't yet start up the repl with another Pyret file yet like `node src/scripts/repl-cli.js hello.arr` but we're planning on adding that feature soon.

[Benjamin Attal]

Another piece of information:

- You can add new lines manually by double tapping the enter / return key

[Joe Gibbs Politz]

Thanks, Ben!

One of the things Ben and I have been trying to figure out is a good interface
for block-editing with multi-line history at the command line.

There are a number of use cases that aren't immediately obvious that have
constrained the design.

1. Supporting paste. Most terminals treat paste events as a sequence of
keypresses, so it's useful if typing "normally" using the Enter key for
newlines plays nicely (e.g. no special modifier key for inserting a
newline rather than causing "run" to happen when pressing Enter).

2. Supporting multi-line entries. Python's REPL does (IMO) a sort of terrible
job at this; if you enter something that spans multiple lines, and press "Up",
you get one line of the multi-line entry at a time. The double-tap of up and
down to navigate was an attempt at this that seemed pretty natural. Single up
and single down can then be used to navigate within a multi-line block.

Racket's REPL can be smart about this because it allows only a single
complete s-expression per entry. We could do something similar with
detecting a single Pyret expression or statement, but...

3. Allowing mutually recursive definitions at a single prompt. In order to
keep scope tidy, the REPL disallows forward definitions; this is disallowed,
for example:

fun f():
g() # g is not defined, plan on defining it later
end

In CPO, you can get around this by including f and g in the same REPL prompt:

fun f():
g() # g is not defined
end
fun g():
5
end

And that works just fine, because it's treated like a normal Pyret block. To
type that in the CPO repl, Shift-Enter can be used to insert the newlines. At
the command-line (and to support pasting) this needs to by typable with just
regular Enter for the newlines, and the REPL also cannot auto-submit after the
definition of f() is completed. So having a rule like "run after a complete
expression/statement is detected to have finished" is also a little
unsatisfying.

Python's REPL _does_ have a pretty nice rule here, which is that a newline on a
blank line submits the entry to be run. That gives the user lots of control
over multi-line blocks, though it may require a special rule for the _first_
time Enter is pressed (on the first line), because it can be annoying if you
type

> 5 + 5 <Enter>
! nothing happens on the first enter
<Enter>
! now 5 + 5 is run

which adds an extra keystroke and newline to every entry. That's where things
stand now.


These constraints aren't all hard constraints, and some amount of compromise
might be required to get to the best system, but I wanted to outline the knobs I
feel we have to turn.

Node and various eventing libraries for it give pretty good control
over listening
for e.g. double-taps and similar things, so creative ideas are welcome.

[Shriram Krishnamurthi]

Did something change in the CPO REPL too? I used to keep running into
a problem when I hit down-arrow after up-arrow (I think) and lost the
entire line I was working on, but I can't reproduce it now.

[Joe Gibbs Politz]

Nope, no changes have been pushed to CPO. This is strictly
command-line/terminal based, and all on Ben Attal's fork.

[Martin DeMello]

Have you considered using ipython for the frontend? I started writing
a ruby repl with better multiline support, but stopped when I realised
I'd be reinventing a lot of the work ipython has done, and I'd be
better off just contributing to the ruby/ipython kernel instead. (Full
confession - I have yet to actually do so, but I like the idea of
supporting their efforts to be language-agnostic.)

martin

> --
> You received this message because you are subscribed to the Google Groups "Pyret Discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to pyret-discus...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Benjamin Attal]

To be honest, I hadn't this known was an option. Now that you mention it, it may be a good idea for Pyret, in the long run, to integrate one of its services with such a well known, and well-used framework.  I still think there's something to be said for having a lightweight repl baked into Pyret itself (though I am biased), but once this version of the repl is production ready, making a Pyret / IPython kernel is a possible next step.

Anyone else have opinions on the topic?

> To unsubscribe from this group and stop receiving emails from it, send an email to pyret-discuss+unsubscribe@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Pyret Discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to pyret-discuss+unsubscribe@googlegroups.com.

[Benjamin Greenman]

I think a lightweight repl is worthwhile. I mean, what's the purpose/goals of the Pyret repl -- unless I'm on trapped on a boat without internet access, why would I use it over the editor at https://code.pyret.org/editor?

If I had a local version of that editor, I don't think I'd use a repl except as a calculator (for single-line statements). That's the only thing I use iPython for, unless I'm %paste-ing from a file. Multi-line statements belong in a text editor.

To unsubscribe from this group and stop receiving emails from it, send an email to pyret-discus...@googlegroups.com.

[Joe Gibbs Politz]

On Mon, Jan 12, 2015 at 7:20 PM, Benjamin Greenman <bl...@cornell.edu> wrote:
> I think a lightweight repl is worthwhile. I mean, what's the purpose/goals
> of the Pyret repl -- unless I'm on trapped on a boat without internet
> access, why would I use it over the editor at https://code.pyret.org/editor?

Thanks for asking, I should have included that with my list of
constraints :-) Aside from purely offline editing:

- A course or project that's teaching or using command-line scripting
can use it. Certain interfaces like file redirection just don't make
sense on CPO. Right now the lack of a command-line REPL is a major
thing holding back that kind of offline Pyret experience, which is
worth having.
- A lightweight offline editor built on something like Emacs can use
it, since it's not much more than a stdin/stdout interface
(potentially with terminal coloring)
- For hacking on Pyret itself, a nice terminal REPL can be faster than
spinning up a new instance of the web service to test a new feature
interactively and get a feel for it

> If I had a local version of that editor, I don't think I'd use a repl except
> as a calculator (for single-line statements). That's the only thing I use
> iPython for, unless I'm %paste-ing from a file. Multi-line statements belong
> in a text editor.

I disagree. We have real uses for multi-line entries at REPL prompts
in CPO, especially when, say, trying out a check block at the REPL (or
copy/pasting a check block at the REPL to debug it without all the
other tests/program). And live-coding activities in front of a class
can involve trying out a (potentially multi-line) function definition
at the REPL. We used that ability a lot in the fall in Shriram's
courses.

I think it's worth trying to keep the command-line REPL in sync with
the behavior of the CPO REPL, which is pretty useful. Maybe we can't
because of terminal UI weaknesses, but it seems worth a shot to not
have two extremely divergent models.

[Shriram Krishnamurthi]

I'm not sure I would want this for the simple reason it makes the
install even more complicated. As it is people have to install
JavaScript, but having the REPL require nothing more than that would,
I think, be a feature. Especially given the hassle that it is, even to
this date, to get pathnames to hook up across platforms (which will
have to happen for these things to invoke one another).

Shriram

PS: Racket has brilliant REPL support, too. (-:

[mven...@yahoo.co.uk]

When are we going to have a desktop version Pyret,if at all?

[Shriram Krishnamurthi]

Hi — thanks for asking. 

Ben Attal has made a bunch more progress on his REPL library this winter. We're right now in transition between module systems (separate message coming up), which is the main obstacle to integration. Once that is done, we can merge in his code and there'll be something ready for people to play with!

Shriram

On Mon, Jan 25, 2016 at 10:00 AM mvengayi via Pyret Discuss <pyret-...@googlegroups.com> wrote:

When are we going to have a desktop version Pyret,if at all?

[samuel_ainsworth]

What's the status of the command line REPL? Has this been merged into master yet? If not, how can I go about pulling up a REPL?

Samuel

[Attal, Benjamin]

Hey Samuel,

Unfortunately, I got caught up in schoolwork early this semester and was unable to integrate the repl code with the current module system. One of it's key features, which is including other files from within the repl, also seems to be a bit buggy / unusable at the moment. 

You can still checkout the repl on my forked copy of pyret at https://github.com/breuckelen/pyret-lang.git. The code is on the "repl" branch, and you should be able to run it with the command "node src/scripts/repl-cli.js." Once school is finished, I should hopefully have more time to sort this out so if you find any bugs / have any suggestions, let me know!

Best,

Ben Attal

[doug....@gmail.com]

Was wondering about the status of a command-line Pyret REPL. Is this now part of the main pyret git repo? Or did it never make it?

-Doug

[Shriram Krishnamurthi]

Sorry for not replying sooner. We've been out doing a bunch of teacher training for our data science and physics courses, with most of the Pyret team engaged in teaching or support.

Even though this hasn't been our top priority, Joe has actually made some really good progress on this — roughly mimicking the javac/java pipeline with a pyret/node pipeline. I don't want to steal his thunder, though — Joe, why don't you weigh in?

Shriram

To unsubscribe from this group and stop receiving emails from it, send an email to pyret-discuss+unsubscribe@googlegroups.com.

[Joe Gibbs Politz]

Yeah this is very much a current issue for me. I want Pyret to be a lot more
pleasant and the command line, and a REPL is just one part of that.

The short answer for the REPL is that it's not on the main branch – there was a
lot of code churn around when it was written and reviving that version is now
pretty nontrivial.

That said, I'm going to use this as an opportunity to talk about some of the
things we've made a bunch of progress on in the last 9 months or so, because
they are related to getting a good CLI repl going.


The main reason Pyret is a nuisance to use at the CLI is that it has all kinds
of startup time issues. In CPO this is paid once each time the editor loads,
and lots of intelligent in-memory caching of modules happens to give a snappy
experience after that. We don't have the same abstractions available at the
command line where the in-memory state of the compiler is constructed at
each startup.

This is mostly because Pyret causes some significant code blowup to get those
good error messages, and because of the stack management story. The upshot is
that it takes a few _seconds_ for the compiler to start up, and hundreds of
milliseconds to a second for the simplest compiled files to run.

This isn't a new problem, just an engineering problem, and we're attacking this
from a few angles:

- Many languages support _compile servers_ or _language servers_ (see the Flow
type checker and the Scala compile server for good examples) that cache lots
of useful information. These typically start up semi-transparently in the
background, and client executables connect to them for much snappier
performance. There's an experimental compile server in the master branch,
with a test repo for it at https://github.com/jpolitz/pyret-starter-repo,
which will let you install Pyret via `npm install`

- Much of the fancy compilation Pyret does for the browser is simply irrelevant
at the command line, because the fancy compilation is all about managing
control flow by simulating pre-emptive concurrency atop a single
JavaScript thread. For lots of reasons, we've historically just had one
version of the compiler that generates this code, but we're working on
compile options to turn off a lot of this support, which significantly speeds
up a lot of operations at the command line. (See
https://github.com/brownplt/pyret-lang/pull/1054)

- Since the Pyret compiler has historically relied on a lot of npm packages,
and we can customize the build process for code.pyret.org as much as we
wanted, command-line Pyret had lots of bad assumptions about the existence of
`node_modules/` and other implicit node dependencies. This made using the
compiler, and the files it generated, overly frustrating outside of contexts
that we set up in the pyret-lang and code.pyret.org repos. We're working to
fix that with a combination of `browserify` and more information about raw JS
dependencies. (See https://github.com/brownplt/pyret-lang/pull/1053)

(I note the irony that a tool called `browserify` helps the offline
experience.)

The overall goal here is to give Pyret a command-line experience where users
get snappy feedback, can run Pyret code without the overhead of CPO, and don't
need to think too hard about installation or nonsense about tracking lots of
relative paths. This is "just" engineering that we need to get through, and
we'll be sure to let folks know when we have things worth trying out.

All of these things are aligned with getting the CLI _REPL_ to work well, too,
because it faces a lot of the same startup and dependency issues as general
standalone Pyret files.


I also want to make a note that a huge amount of this progress is thanks to Ian
Boros, a terrific student from Brown we were lucky enough to work with this
past academic year and some of this summer (the two pull requests above were
created by my account, but mostly consist of Ian's commits). You can see his
Github commits as the user puppyofkosh.

[Joe Gibbs Politz]

Hi all, I wanted to follow up on this, because there's some real
movement on points 1 and 3 from my last message (language server and
dealing with npm dependencies).

It's experimental (with lots of caveats below), as the name suggests,
but I've put out this npm package:

https://www.npmjs.com/package/pyret-experimental-cli

It's (the start of) a reasonable command-line interface for
users/students to run Pyret; I've been using it for a few weeks and it
seems to do a decent job of letting me run Pyret offline.

The compile + runtime for a simple program is still something like a
second, but it's infinitely better for programming offline. Given what
I'm used to dealing with for compiling and running, say, Java programs
with JUnit at the command line in some intro courses, this is close to
being an acceptable command line experience.

It's useful for programming on planes or wherever you might not have
WiFi, if you want to manage directories of Pyret files outside of
GDrive, or if you want to install Pyret on something like Cloud9 IDE
and write Pyret files using their online environment.

This is something I'm happy to push on more, and would love to hear if
folks are interested in seeing more of this.

Major missing features:

- The image and reactor libraries aren't available offline yet;
suitable Electron-style wrappers are possible, but not in what's built
right now
- I haven't tried it on windows, and suspect it doesn't work because
there's some hardcoded stuff that assumes the creation of file sockets
in /tmp.

This work is on the server-dev branch of pyret-lang, which will likely
get PR'd to horizon as it stabilizes.

[Tom Hoffman]

Thanks Joe!

Does this assume I've already done something to install Pyret before doing npm i pyret-experimental-cli?

Just running that command doesn't put pyret in my path.  

--Tom

[Joe Gibbs Politz]

Try:

npm -g i pyret-experimental-cli

The "-g" tells npm to install a global link to it, so it should just
end up on your path.

>> email to pyret-discus...@googlegroups.com.

>> For more options, visit https://groups.google.com/d/optout.
>
>

> --
> You received this message because you are subscribed to the Google Groups
> "Pyret Discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to pyret-discus...@googlegroups.com.

[Tom Hoffman]

Yep, that did it.  Thanks!

>> email to pyret-discuss+unsubscribe@googlegroups.com.

>> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Pyret Discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to pyret-discuss+unsubscribe@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Pyret Discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to pyret-discuss+unsubscribe@googlegroups.com.

[Joe Gibbs Politz]

As another update to this, I added a new option to the CLI that's useful for speeding up command-line programs. You can run with the option --perilous:

  --perilous
                              Compromises error semantics for speed. If the program has no errors, it will
                              produce the same outputs and answers. Currently, this means eliding most
                              annotation checks in compiled code and in libraries. Fine-grained control is
                              intentionally not provided since the specifics of how this works may
                              change, but in general it will do less error checking in exchange for faster
                              execution, and not change the meaning of programs with no errors.

Use

$ npm update -g pyret-experimental-cli

if you've previously installed it to see the update. (Use install rather than update if you want to install for the first time).

On some benchmark-y programs I've seen close to 2x improvements in time taken. It's pretty helpful for list-heavy programs, because this will omit the checks that are performed each time a link is created, for example.

We've started experimenting with some options like this to explore balances between Pyret's extremely detailed error reporting, and running at speeds that can tackle larger inputs and datasets. It also hints at what it might look like to run fully-type-checked Pyret, where many of these checks can be _safely_ elided because they've been discharged statically. For now, omitting them is, well, --perilous.

There's a strong analogy here to, for example, Dart-to-JS's "production" vs "checked" mode (https://www.dartlang.org/resources/dart-tips/dart-tips-ep-2), which omits dynamic type tests, and programs that violate types may let underlying JavaScript errors surface.

As an example, the first error below points out that the error is on the first line of the program (when `link(1, 3)` is called).

With --perilous, the ill-formed List can be created, and the program fails more inscrutably in the internals of .append.

Other operations, like `.first`, would succeed on `not-really-a-list` in perilous mode.

$ cat perilous.arr

not-really-a-list = link(1, link(2, 3))

not-really-a-list.append(link(5, empty))

$ pyret perilous.arr

1/1 modules compiled (perilous.arr)

Cleaning up and generating standalone...

The run ended in error:

Expected to get `List` because of the annotation at builtin://lists:112:29-112:36 but got: 3

Pyret stack:

  file:///Users/joe/src/pyret-starter-repo/perilous.arr: line 1, column 28

$ pyret --perilous perilous.arr

1/1 modules compiled (perilous.arr)

Cleaning up and generating standalone...

The run ended in error:

Evaluating the object lookup at builtin://lists:177:27-177:50 errored. The left side was not an object: 3

Pyret stack:

  builtin://lists: line 177, column 27

  builtin://lists: line 177, column 27

  file:///Users/joe/src/pyret-starter-repo/perilous.arr: line 3, column 0

[pouls...@gmail.com]

Hey All, 

I've recently discovered that Node exposes its own REPL interface to the user, so you can instantiate it with your own custom evaluation function if you want (docs here, source here). 

This seems like it should be a good way to get a simple, up to date Pyret REPL almost for free. 

The REPL library will call the evaluation function you give it every time that it receives input. I've tried for a while and I can't figure out a simple way to have it happily call a Pyret function repeatedly as a call back. 

I know there is a simple way to work with Pyret and callbacks when you just need the callback to be called once--by using runtime.pauseStack(). Is there currently an easy way to work Pyret into JS code with a callback that needs to be called repeatedly? 

[Joe Gibbs Politz]

Wow, this does sound like a good option. I sketched something out at
https://github.com/brownplt/pyret-lang/pull/1322

Getting all the history tracking, etc, for free would be great. I see
there's even a detector for whether a newline should run the program
or insert a newline, which we may be able to adapt the codemirror mode
for.

I sketched out a way to do this: write a JS native Pyret module that
imports the "repl" node module, and provides a Pyret interface to it.
Then we could make the "main" be a Pyret program, or an option to the
Pyret compiler (e.g. node pyret.jarr --repl).

What I sketched out has no error handling to speak of, some really
silly code to drill down to the right value to print, and has lots of
open questions about how to start up a main file that you then
interact with, how to do imports, and more. But it gets the basic idea
across, and should do the right thing with using our existing repl
abstraction to fold over the context of defined identifiers.

Thanks for pointing this out!

> --
> You received this message because you are subscribed to the Google Groups
> "Pyret Discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to pyret-discus...@googlegroups.com.