Rhino-based continuations engine example (was: "Is Rhino the only interpreter with support for sandboxing and serializable continuations?")

24 views
Skip to first unread message

Ian Sollars

unread,
Oct 7, 2011, 3:08:03 AM10/7/11
to collabora...@googlegroups.com
Hello everyone,

Nathanael suggested I share this with you guys - he & I have had an intermittent conversation over the past few months about long-running processes and continuations, see below. Since our original mails I've wrapped a Twitter interaction API and UNIX-style process model around Rhino, and I'd be grateful for any feedback or critiques on the project:

http://ianso.github.com/scriptus/

It's a different problem domain and a slightly different approach, but you might find it interesting.

Best regards,
 - Ian

---------- Forwarded message ----------
From: Ian Sollars <ian.s...@gmail.com>
Date: 4 October 2011 13:10
Subject: Re: "Is Rhino the only interpreter with support for sandboxing and serializable continuations?"
To: "Nathanael D. Jones" <nathana...@gmail.com>


I can't believe I didn't think about an AMI. That's a great idea, I'm definitely going to do that :-) My next post will be to the list, I just need to debug an example script ...


On 4 October 2011 12:34, Nathanael D. Jones <nathana...@gmail.com> wrote:
Let's copy the list on the conversation - it could be useful for some.

I agree with the small pieces loosely joined; more or less the smallest piece is a function, and by allowing function calls to be 'upgrade points' (if the user code uses 'switchto('func')' instead of func()), it seems to work pretty well. 

I'm referring to the mainframes, when continuations and serialization weren't required due to process longevity and manageable user base. I guess it's an unfair complaint - scaling is easy when there are 1000 people on the internet.

Just tossing out another idea - but you could create an AMI, and allow people to enter their twitter/aws keys into the user data box, and just have a small shell script set the server up. Amazon has that free server for a year deal, so it would let (aws-knowledgable) people set up their server with a significant time or monetary investment. 

Best regards,
Nathanael



On Tue, Oct 4, 2011 at 10:31 AM, Ian Sollars <ian.s...@gmail.com> wrote:
I can see you've thought a lot about the problems involved - hot-replacement of code and collaborative editing isn't something I've thought about much to be honest. My approach has been to do the smallest thing that could possibly work, and I'm consistently surprised at how big that small thing is :-/ Since I've stolen most of my process model from UNIX my first stab at the problem would be a "small pieces loosely joined" approach, although I would need to implement the exec() system call in Scriptus to make this possible.

How do you mean that things were easier 30 years ago? Are you referring to the era of mainframes and minicomputers?

I will make a post on your mailing list and see what happens after I've made my next commit. Do you think it would be a good idea to copy the list on this conversation or to start a new one?

Finally, thanks for your suggestion of getting a server up and running. I've considered this but I have the problem that this would effectively allow people use of my Twitter & AWS accounts which wouldn't be a problem on a small scale, but on a large scale would be open to abuse. Alternately, if I allowed each user to enter their own Twitter & AWS API keys and so use their own accounts, I would then effectively be responsible for a keystore database that would present a very high-value target for intrusion attempts. I know enough about security to realise that this is not an easy problem and that designing an ultra-secure web application infrastructure is a challenge on its own, and one I don't want to deal, at least not just yet :-)

Best regards,
 - Ian


On 3 October 2011 17:18, Nathanael D. Jones <nathana...@gmail.com> wrote:
I only actually deal with a call stack - Pluto persists the coroutines which contain the variables. It's just an array of paused coroutines. I only have to do the 'outer' call stack outside pluto because you can't have two functions with different sandbox settings directly calling each other. By performing function calls through the messaging system (and therefore the outer stack), the sandbox can be replaced for each file. The additional benefit of message-based function calls is that it allows the latest revision of code to be used instead - hot code swapping can't happen in Pluto. I'm assuming Rhino has the same restriction. I've seen coroutine persistence engines with support for it, but they require manual annotation of every function with a unique and persistent ID and can't survive major changes (and also have unexpected behavior sometimes).

Frankly, I've forgotten most of my problems with Rhino. It's been almost a year since I worked on the js branch.

Your immediate goals for Scriptus are very similar to my long-term goals for the Weaver engine. While the practical planning and coordination application is my 'end-game', I had a very hard time coming up with intense/useful test samples, and realized that a real-time  MMORPG would exercise the system in the same way (and possibly in a more thorough fashion) , but be easier (and more enjoyable) to create  content for. The few people on my mailing list aren't particularly interested in games - they are more interested in the computer sciences and the general application. 

One of my original motivations for the weaver engine was not being able to create 'smart' to-do lists. It's frustrating that no reminder software seems to take the 'forgetting curve' into effect, but it's easy to see how the human aspect gets overlooked. It's also an example of how difficult long-running logic is to create today. It's strange, but the situation was better 30 years ago for this kind of application. 

Considering the tiny size of my community, I wouldn't worry about the possible mismatch - I think fragmentation would be more of an issue. These people would definitely be interested in your work.

I haven't yet settled on a license for the game content (which is being developed by another coder), but consider it MIT licensed for now. Suggestions are welcome. 

Anyhow, I think it's good that you're going directly for the target application - it's important to get people to see the possibilities early so you can build a community faster. I'd suggest getting a server running as soon as you are confident of the sandboxing, and let people set up and share scripts freely. 

Right now Weaver engine only has web and console interaction, but I'd like to add SMS and e-mail abilities eventually - it should be pretty easy. 

I'm probably going to make the weaver engine play games for a few years before I present it as a business framework, so I can get a really thorough understanding  of both long-running logic, opportunistic code upgrading, and collaboratively edited software systems. It's kind of taking dynamic software and extreme programming to the limit - it may be a bad idea, or it may bear fruits that can affect the software development field. We'll see :)

Best regards,
Nathanael





On Mon, Oct 3, 2011 at 4:34 PM, Ian Sollars <ian.s...@gmail.com> wrote:
Hi Nathanael,

I've taken a quick look and it's very interesting - if I read it correctly you used Pluto to essentially create your own persistence engine for the stack, variable & function, am I understanding it correctly?

I'd love to know the problems you had with Rhino - I suspect I've already found some :-/ and in terms of cross-pollination I'm also interested in keeping track of your project, hence my follow - I'm always interested in different approaches to similar problems.

I don't have a wiki or mailing list yet but if I create one I suspect the audience would be quite different to Weaver, depending on what your end goal is. Vaguely put, mine is to help to encourage people to create long-term programs that act as 'social armature', to augment the long-term planning & coordination capabilities that (one could argue) have been diminished or fragmented by modern technologies instead of reinforced. Do you think that's a good match?

And finally, your suggestion of cross-porting tests is a great idea - I've checked out the lua under code/world and I like the structure (and the writing) of the adventure - I think if it's OK with you I'll try and replicate the adventure in Scriptus and see how far I get. Likewise if you want, my working examples are going to go under scripts/, although there are only two simple ones right now, and I'd be interested in how you deal with them in Weaver.

I'll follow your advice and make a post to the mailing list with the link.

Best regards,

 - Ian


On 3 October 2011 15:38, Nathanael D. Jones <nathana...@gmail.com> wrote:
That's great news!

Feel free to post on the mailing list about your project. I've updated the wiki with a link to it. Free free to edit the wiki to your heart's content :)


As serialized continuations are a bit of an experimental feature in most languages, I've been exploring (in parallel) an implementation in both Rhino and Lua. I kept running into issues with Rhino, but I recently found a patched version of Pluto that allowed the Lua branch to continue very rapidly. 

In three days of mad coding I got a decent framework built in Lua. Performance seems extremely good. Lua and javascript aren't very different, but Lua has extremely nice sandboxing abilities buit-in. Feel free to take a look at the (rough) code at https://github.com/nathanaeljones/weaver-lua

Right now I'm trying to get content into it so I can hava a good understanding of how to optimize the framework architecture. Without extensive scripts, it's hard to make the framework very good. I'm currently using a message passing system, with subprocess execution built on top. I have an 'artificial' call stack of continuations so that the functional paradigm can persist through sandbox level changes. I'm also working on having named continuations, so that a function can load/pause a named 'subprogram' that exists outside the normal call stack.

The shared-world data model is currently my challenge... For latency reasons I'd like to have the game distributed, using a git-like-db for synchronization... haven't found that kind of db engine, yet, however. I'm thinking of building it on libgit2 or jgit, haven't decided which.

My end goal with the weaver project isn't a game, but I think a game does the most for exercising this kind of framework and locating the weakness in a design. 

Originally I designated a 'plan A' and 'plan B' for the weaver engine, with the assumption that one of the plans would hit a dead end. However, now that you've had success with your project and weaver-lua is also doing well, it seems both are possible. 

What do you think we should do? I think it would be a good idea to share a mailing list (or suggest users to subscribe to both), and either cross-link sites or share a wiki. I'd be happy to name your project part of the weaver engine project if you're interested.

I think both a Java/js and a C/Lua engine have a place - Java and C are mutually exclusive 'parent' languages, yet together they reach over 80% of the 'other code'.  We can explore different approaches to problems, and share the results, hopefully finding the ideal design twice as fast :)

I definitely would like to keep in touch, and I'd like to share our results with our approaches. (And perhaps we can cross-port our tests/exercise scripts to evaluate differences in syntax complexity). 

Anyhow, feel free to edit http://weaverengine.com with anything you'd like.

Thanks,
Nathanael

On Mon, Oct 3, 2011 at 2:52 PM, Ian Sollars <ian.s...@gmail.com> wrote:
Hi Nathanael,

If you're still interested in Rhino, sandboxing & continuations, then you may like to know that Scriptus is now on GitHub, and with much more functionality than the code I sent you: http://ianso.github.com/scriptus

Best regards,
 - Ian

On 15 March 2011 17:52, Nathanael D. Jones <nathana...@gmail.com> wrote:
Sure, I'd love to take a look at your work. 

Since the core of the weaver engine is just that, continuation management, perhaps I can learn something from your code.

The most difficult part of the weaver engine design is how to have multiple levels of sandboxing and allow communication between them.

If you want to keep tabs on future developments in the weaver project, subscribe to the mailing list. Very little activity, not going to spam you. weaverengine.com

Best regards,
Nathanael Jones


On Tue, Mar 15, 2011 at 5:43 PM, Ian Sollars <ian.s...@gmail.com> wrote:
Hi there (Nathanael?),

I saw your post on SO, which I found by googling "javascript engines supporting continuations". I was searching for this because I'm interested in implementing something very similar to what you're proposing, as I've sketched out here.

I'm generally very surprised that more people aren't excited about the possibilities of serialisable continuations in vastly expanding what we currently conceive of as being "a program", or even of what is "programmable".

I have a small project hacked together during a train journey that you might be interested in looking at. It's in Java, and uses the Rhino JS engine in a very simple "client-server"-style application that uses continuations. I'd be happy to send you the source if you want. In my exceedingly rare spare time, I'm working on turning it into an engine such as might easily sit in the Weaver engine you've mentioned on the Wiki you created.

Best regards,

 - Ian


--
Blog: http://ianso.blogspot.com/
Profile:http://www.google.com/profiles/ian.sollars

Reply all
Reply to author
Forward
0 new messages