Asynchronous multithreading?

[Strika]

It's completely unconventional (for Evennia) and there are many well-defined ways of solving this problem using other tools, but where's the fun in that?

I'm trying to do data-processing within Evennia. A Script initiates asynchronous task A, which needs to do three things (B, C and D) that will each take time to complete, so they must be run in parallel. When it's done A will do something with the result of all three operations and the caller is sent a notification.

What's the safest/sanest way to do this with native constraints? Is there one?

    * Can async tasks kick off other async tasks (or will this create an explosion of zombie processes/memory leakage over time)?

    * Can I use libraries like multithreading and Queue, or would that clash with things like the SharedMemoryModel and Twisted itself?

    * Do third-party libraries like Celery play nice with Evennia models?

    * Any other ways you could see this being successful?

Thanks :)

[Vincent Le Goff]

Hi,

What you describe is asynchronous programming.  It has nothing to do with multithreading or multiprocessing per-se, that's just two possibilities.  Evennia has chosen a third approach: asynchronous tasks.  Basically when you execute a single command it creates a task and does wait for it to process, but others can enter commands.  Or to take another example: going on the Evennia webserver and logging at the same time on telnet does use asynchronous operations (but not multithreading or multiprocessing)?  How dies it work?  It's actually a bit complicated to explain in a single post, but if you want to research the topic, Evennia uses twisted, and twisted's handle of tasks through callbacks is probably what you want to do.  There are times when multithreading or multiprocessing is indeed necessary.  You'll have to decide whether overcoming the obstacles these two techniques throw at you are worth using them instead of the integrated callback systems.

Now that Evennia has migrated toward Python 3.7 though, notice that asynchronous handling has been integrated, not in Evennia, but in Python itself, though the await and async keywords.  In the situation you described, you would have a function calling await on a "long" task, executing it won't block the rest of Evennia (assuming the task operates on asynchronous principles) and when the task is ready, the lines after your "await" will be called.  Python does this through a subtle (and a bit surprising) use of generators.  Using this syntax in Evennia is still a bit tricky, though Twisted does offer a way to "transform" coroutines into callbacks.  Before rushing toward multithreading and multiprocessing, I'll highly encourage you look further into that, it makes code so much easier to write, read and bugs harder to get past, in contrast to multithreaded or multiprocessed applications.  If you are willing to read a long tutorial/introduction, I'll warmly recommend this one: https://realpython.com/async-io-python/ .  It's long, but it's thorough on the subject.

Otherwise, yeah, if you plan well and feel you have to, I think you can generate either threads or processes from Evennia, nothing prevents you from doing that.  I tend to avoid this alternative because, although it looks incredibly simple, it actually leads to a lot of errors and hard-to-debug situations.  I much prefer async to that!  Personal preference of mine.

HTH,

Vincent-lg

--
You received this message because you are subscribed to the Google Groups "Evennia" group.
To unsubscribe from this group and stop receiving emails from it, send an email to evennia+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/evennia/0d8e8c08-110b-4501-8240-1692a2f04be9%40googlegroups.com.

[Griatch Art]

Belatedly,

As vincent mentions, don't confuse parallelism with asynchronous operation. Async does *not* run in parallel (that is, it does not make use of the multiple cores and threads of your CPUs). Async tasks run serially, it's just that execution gets chopped up so that it *feels* like everything happens at the same time. This is very powerful and is what allows multiple users to run Evennia next to each other without one blocking the other. A huge advantage of this is that you don't need to worry about race conditions - everything happens just in the order you expect (at least for each player, if not necessarily between players), making coding simple and more bug-free. But heavy CPU-bound processes should be avoided and if you put `time.sleep()` in there somehwere, you *will* stop the server for everyone. In Evennia's case, the async operation is provided by the Twisted library.

Furthermore, threading will not behave quite as you expect. This has nothing to do with Evennia but is rather a thing with CPython and the Global Interpreter Lock (GIL) it uses. There are a lot of resources about this online, but in short - if your code is I/O bound, multithreading will work just fine. If it's CPU-bound though, multithreading *will not help you at all*. In Python in general (and CPytthon in particular), you must use *multi-processing* to get true parallelism.

With this out of the way, to your questions:

1. Yes, async tasks can kick off other async tasks. It will not be any more strange than having functions calling other functions. Unless you create an infinite loop of funcs calling one another you should be fine. There should also be no process/memory leakage at all since this is all run in the main Python execution thread.

2. Yes, you can use both multithreading and (which is probably more relevant for you), multi-processing from within Evennia. To link external processes into the async code you need to use some special Twisted functions, https://stackoverflow.com/questions/5715217/mix-python-twisted-with-multiprocessing is a good answer to this. Like glyph, I also recommend the ampoule library if you want to do more sophisticated multiprocessing from within Twisted/Evennia. Any multiprocessing solution *will* clash with the SharedMemoryModel and the async process overall though - Typeclasses work so as to store their on-object state in memory. There is a lot of caching going on, and if you are, for example, editing the database from another process you will end up making the main thread's cache out of sync. It's the normal race-condition risk - with true parallelism comes the dragon if very, very hard to track bugs.

3. I don't see why Celery would not work, but it's not a common combination and I don't know if there are any ready-made linkings between Twisted and Celery.

4. For truly making use of multi-processing in a sane way, I would make sure to make it highly specialized - that is, use it for heavy computations like AI etc but don't use it to run game loops. Make an API so that the main code can request/start some computation and get the result back asynchronously when it's done (using multi-process), but don't let the other processes themselves change the game state or the database. That's what I'd recommend.

Hope that helps,

Griatch