In-game softcode and scripting

[Kelketek Rritaa]

Hello there!

I come from the MUCK crowd, where user scripting and building is a staple of many of the communities therein, not unlike it is for many MUSHes. Evennia has some awesome extensibility, but I haven't found anything yet on in-game scripting that can be done by players. Is this currently possible?

A few enthusiast coders, including myself, began work on the MuStar project, which, as we found out, is very similar to what we've seen Evennia do. In fact the biggest difference between our core needs and the Evennia core (As we see it) is that there doesn't seem to be an API through which one can plug in their own programming language to interface with the game's object model and database (though perhaps we're missing something?). Normally, we'd want to use Python as the language player softcode is written in. But this appears to have major sandboxing issues. We also have old code that we'd like to 'port' initially to get things off the ground. This lead us to set up an API for MuStar, but I'm thinking that if this is the only core difference between our project and yours (there are other things we'd like to do, but they should be easy enough to place on top of the existing Evennia core), why not work from a project that is already established with almost all of the same engineering constraints and goals? Ours isn't anywhere near as far along.

Let me know what you folks think and if you have any ideas on how this might be implemented.

[gsb]

Well, I know for a fact that evennia is meant to be purely python, meaning anything you write for your game is written using just normal python modules that then leverage the underlying classes/object models.  This works quite well, and as I can tell is intended to keep everything purely python.

I don't know that this necessarily makes it impossible to do what you are asking, I am SURE you could add this functionality into whatever you end up designing and it would just be local to your game environment instead of baked into the server code.

--geoff

--
You received this message because you are subscribed to the "Evennia" MU* server mailing list.
To post to this group, send email to eve...@googlegroups.com (or use the web interface).
To unsubscribe from this group, send email to evennia+u...@googlegroups.com
For more options, visit this group at http://groups.google.com/group/evennia?hl=en

[Griatch Art]

Hi and welcome to Evennia! Interesting to hear about your project!

It is correct that Evennia does not have an online scripting language per se. One of the motivations for starting it was indeed specifically to get away from softcode. The original motivations can be found here: http://code.google.com/p/evennia/wiki/SoftCode  In short, an online scripting language is great for adding small pizzazz and interactivity to your objects, but never to build complex systems or the game itself. There is really no excuse for a small development team to not use proper text editors and a good version control system for actual game development (we think).

That said, we are not explicitly against supporting some sort of online scripting language for small-scale command line development. It's not a priority at the moment but that doesn't mean it cannot be discussed! :) 

As you say, Python is notoriously difficult to sandbox safely, and a full Python sandbox is not something we are likely to ever implement. I'm also reluctant to look into something like Lua - mingling yet another scripting language inside Python doesn't feel very good. However, Evennia allows for extremely powerful customization as it is. Since you code everything using normal Python modules and classes, designing a very flexible online building system is something which shouldn't be too hard to do. Heck, the lock system can be seen as a very powerful (and safe) online scripting system out of the box if you want. So you could probably offer most useful scripting possibilities through working with Evennia's command classes. 

That's not actually a "language" though, admittedly. If one looked to support something like this in the core (or maybe as a contribution), one could consider extending the lock concept. This way one could offer "safe" functions for people to use and combine without opening the full sandbox safety mess. That is, you store "safe" python functions in a module, and can store calls to such functions in a string on your objects - along with whichever arguments the online coder wants to give. 

The point is that these functions are never run through Python's eval/exec mechanism, but the string is custom-parsed, allowing only safe functions and operations. You can put whatever complex things you want inside the functions - i.e. the part you control and can make safe. So from the user's perspective it'd not be "real" Python, but a subset working mostly like Python (well, as much like Python as you care to make it, really). As said, the lock handler already works like this. Here's an example of a more advanced lock string:

  "background_check: attr(is_agent) or attr_gt(intelligence, 45) and not attr(is_undercover_fed)"

Here background_check is the name of the lock to check, attr and attr_gt are "safe" lock functions. The arguments can be safely supplied by the user to be handled and validated properly behind the scenes by each safe function. Supply enough safe functions and hey presto- you have a language! All the lock can do with the result is to get a boolean out of it however (pass/not pass the lock). A theoretical online coding system should be able to do more things with the result, such as store it in an attribute, send it to people etc. 

So this could be one way to implement a "safe" online "language" Python subset in Evennia. Should one want to go that route I would need to ponder it more before I could come up with an actual design though.

.

Griatch

[Jorge Vargas]

Check this answer at stackoverflow: http://stackoverflow.com/a/6625703/13390

Is this a safe way to sandbox some scripts? By adding to the scope only the modules related to the online language. I'm currently checking out mud code bases and that's one of the features I'd love to have. 

[Griatch Art]

@Jorge Vargas

That is a sort of blacklisting scheme. As far as I've read this is a trickier notion than one might think - there are a host of various ways to examine Python objects and get to their inner workings (which in turn leads you through __class__ up to type and object etc (even your normal integers are objects after all)). I don't know an example off-hand, but I'd think it'd be a well known thing if the solution was this simple ... the old module in Python that was intended for safe Python operation used stuff like this (including turning off __builtins__ I think) and was still scrapped for not being able to guarantee safety.

That said, this is not really a "sandbox" I guess - you are crippling the language quite substantially. Supposing you could continue with the crippling until you decided you felt "safe" using this, you need to also make sure to check for scripters supplying things like "while True: pass" which are perfectly normal python structures with components harmless on their own, but which will lock the server up quite nicely. One needs to either parse the structure to detect stuff like this (hard to do), or launch the code in a separate process that you can watch and kill if it times out. In experiments I have done one such thing in normal Python (which blocks), one can also do one in Twisted that doesn't block the reactor, but it's potentially a lot of overhead, especially since such scripts are probably very small and easy most of the time. But it's doable.

If one uses Linux which a scheme checking for too much processor usage, it's probably possible to safeguard also normal Python from "outside" by using the OS safety features. Linux/Unix is very good at restricting access for individual users. This way one could at least stop a nasty script from accessing things on your computer. For example, making sure to run the script as "nobody" and inside a chroot jail or something. Whereas this might be fine for individual solutions, It's not portable or usable for a one-fits-all solution of course. :)
.
Griatch

[Jorge Vargas]

Thanks for answering with so much detail. You make some excellent
points that I never took into consideration. It looks like the best
way to do this is by expanding on the lock system to create a more
powerful DSL. That's fine by me, thanks.

> >> normal Python modules and classes, designing a *very* flexible online

> >> building system is something which shouldn't be too hard to do. Heck, the
> >> lock system can be seen as a very powerful (and safe) online scripting
> >> system out of the box if you want. So you could probably offer most useful
> >> scripting possibilities through working with Evennia's command classes.
>
> >> That's not actually a "language" though, admittedly. If one looked to
> >> support something like this in the core (or maybe as a contribution), one
> >> could consider extending the lock concept. This way one could offer "safe"
> >> functions for people to use and combine without opening the full sandbox
> >> safety mess. That is, you store "safe" python functions in a module, and
> >> can store calls to such functions in a string on your objects - along with
> >> whichever arguments the online coder wants to give.
>
> >> The point is that these functions are never run through Python's
> >> eval/exec mechanism, but the string is custom-parsed, allowing only safe
> >> functions and operations. You can put whatever complex things you want
> >> inside the functions - i.e. the part you control and can make safe. So from
> >> the user's perspective it'd not be "real" Python, but a subset working
> >> mostly like Python (well, as much like Python as you care to make it,
> >> really). As said, the lock handler already works like this. Here's an
> >> example of a more advanced lock string:
>
> >>   "background_check: attr(is_agent) or attr_gt(intelligence, 45) and not
> >> attr(is_undercover_fed)"
>

> >> Here *background_check* is the name of the lock to check, *attr* and *
> >> attr_g*t are "safe" lock functions. The arguments can be safely supplied

[Griatch Art]

Update concerning "softcode" ...

Revision 60f2b118fa66  holds a experimental restricted Python environment I've tentatively dubbed "Evlang" (mainly to be able to avoid confusion between "evlang scripts" and Evennia's normal "Scripts").  It's added as a "contrib", i.e. an optional addition to Evennia.

Now, I went through a few iterations on developing this. I first implemented a full custom parser similar to that used in the lockhandler as I originally suggested in this thread. It worked, but this is a considerably more complex problem than the simple syntax allowed for lock strings. In short, with the possibility of nested structures it becomes very error prone and probably also very expensive to make sure all individual tokens are accepted. And then one still hasn't started to deal with control structures (I thought about supplying a "safe" if-like function, but the function-argument parsing for that would have to have been a special case - and when a code becomes full of special cases one knows things are going awry).

So in the end I abandoned the custom-tokenization (aka lock-handler-similar) approach and looked around for something else. Of course Python's ast module handles full tokenization (batteries included and all that), so that can in principle be used to sidestep much of the parsing trouble. Once you use the AST however, you are no longer reading a string from left to right but are interpreting real Python syntax structures (The AST can after all be compiled directly to Python code). It's like approaching the problem from the other direction - restricting what is already there rather than creating what is there from scratch.

I tried a few recipes off the web with various results until I eventually settled on a hybrid approach - this combines a blacklist+whitelist (to restrict dangerous builtins and other stuff), but also uses an AST-tree traversal check on the source code before the code is executed. This is a lot more complex than simply executing in a limited context - the AST walker will catch and kill entire families of Python functionality in one broad swath.

Now, as mentioned earlier, blacklisting is a dangerous thing. The internet is full of warnings against relying on such techniques to "sandbox" Python. For an Evennia "softcode" language however, I believe we have an advantage in that we have a very small and specific purpose with our scripting. We are not looking for a sandbox - in fact we can impose severe restrictions to the language, hopefully beyond the level of easy exploits. We also have the further advantage of being able to supply "safe" methods into the language to befit our current, limited use-case.

So Evlang is a severely stunted subset of Python. While loops are forbidden, so are Attribute access, function definitions and many other things. But it works, and it's pretty nifty.

One problem is the aspect of DOS attacks - a script locking up the server because it's too processor heavy. The evlang script is run in a separate thread, so it won't slow down the server while it runs. But it will also not be killed. It turns out to be very, very hard to kill subthreads safely in Python, and to do so from a position being embedded inside Twisted is, at least so far, not obvious. I looked into subprocesses, but doing that while having access to the current states of Evennia objects is an even bigger pain, so I dropped that route for now.  Since Evlang simply deny the most common ways to create infinite run-times (while loops, huge numbers etc), I hope it shouldn't be very easy to DOS the server this way. But it's something to keep in mind.

Evlang comes with a huge "WARNING" and "EXPERIMENTAL" flag for now. Test it, try to break it and report what you find. In the commit I also include example Commands for coding an object as well as custom evlang-scriptable Typeclasses along with lots of documentation.

Anyway, to the original poster of this thread: Evennia now does support a "softcode" language of sorts, enjoy!

.
Griatch



On Tuesday, May 8, 2012 6:59:18 PM UTC+2, Kelketek Rritaa wrote: