Proposal for a new core expression

[Joe Gibbs Politz]

I'm making progress on serializing stores/environments/mvals to
s-expressions. I want to add a snapshotting facility to the
interpreter to allow us to write things like:

___take_interp_snapshot()

And get back the current store and environment at that point of the
computation. We might want other facilities like this in the future
to do meta-kinds of operations while the interpreter is running.
Here's what I propose:

1. Add a new CExpr: ESpecial (string, CExpr)
2. Add a new macro ___hint(str, CExpr)
3. Extend the interpreter to understand some special hints

I want these to be distinct from primitives because they aren't for
manipulating values; they are strictly for us to build things on top
of the existing interpreter. For example, for
___take_interp_snapshot, I want ESpecial ("snapshot", e) to raise a
Racket-land exception containing the store and environment that I can
catch in python-main.rkt to write those values out to disk (and start
new computations w.r.t. those). We might have other uses for ESpecial
later, like annotating expressions with types or other information, or
performing other instrumentation.

Other ideas/comments on the proposal? Better names than ESpecial?

[Joe Gibbs Politz]

(The overall goal here is to make running tests much, much faster by
skipping the time it takes to run the initial libraries).

[Joe Gibbs Politz]

OK so I went ahead and did this on my fork:

https://github.com/jpolitz/lambda-py/compare/718b0f41685336133cd8c14484b0d8d02ea7dd88...a2d1be7e017380414429a498435fb45975f546dd

Saving stores and environments to disk naively (by converting to s-expression) causes huge blowup and runs me out of memory.  I assume this is because the environments that functions close over are shared in memory, but get copied when serialized; we need serializable structs, which maybe we can get or define, but that's another issue.

As an experiment, I tried in-memory caching of the starting store and environment for tests.  By paying an up-front cost of a couple seconds, we seem to be able to save about 3 seconds per test.  That's about 9 minutes total off the running time of the tests; not the huge speedup I was hoping for, but at least something.

It works by having the interpreter throw a special Snapshot exception when it sees a snapshot special form.  This exception is caught by run-python/snapshot in python-tools, which stores it in a global.  run-python checks for this variable and, if its present, doesn't do any library wrapping and calls interp-env directly with the snapshot's environment and store.

Any thoughts or comments?

[Alejandro Martinez]

This should not increment notably the memory requirements, should it?

I ask because I had to partition one test in python-reference in two pieces due to Racket virtual machine running out of virtual memory in Windows.

2013/4/16 Joe Gibbs Politz <joe.p...@gmail.com>

--
You received this message because you are subscribed to the Google Groups "lambda-py" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lambda-py+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
Alejandro.

[Joe Gibbs Politz]

No noticeable memory increase for me.  It now uses the same (meaning eq?-equivalent) initial env and store for every test, so I think if anything it helps avoid any lingering references that would be duplicated due to not being garbage collected between running tests.  Running tests still takes up a lot of memory during some of the larger tests (nearly .4G for me), but I think no worse.