New to Pulsar - a few questions

[Michael S]

I've been researching the actor model as much as I can as it's an interesting topic. I'm mostly attempting to understand the differences between various architectures and the best way I can go about implementing a concurrent, local (not networking) application.

1) it's mentioned in various places that Pulsar can be configured to use threads or processes, for instance here: https://news.ycombinator.com/item?id=10018050

"

scribu 593 days ago [-] Great link - thanks for posting! Pulsar can be configured to use either processes or threads. If you use processes, you pay the IPC penalty, just like NodeJS does. If you use threads, you pay the GIL penalty. So yeah, it won't be as fast as BEAM or the JVM either way.

"

So from my understanding, if you're using processes, then pulsar will connect event loops to ports (with their mailboxes) for sending and receiving messages. If you're using threads, it will basically just be using the python Threading module (with all the GIL issues). How accurate/inaccurate is my understanding? Also - what is the "IPC" penalty the comment mentions? Is it really going to be that much slower than erlang or java?

If Threading does get locked by the GIL, is there a way to get around that for multiple cores, maybe with Cython? 

Finally, how do you configure pulsar to use one or the other in the first place?

2) In the documentation (https://quantmind.github.io/pulsar/design.html) it mentions using executor() for intense processes. I can't find this function anywhere or in the code - what is this referring to, and what are its use cases? Can you show an example?

3) In the example remote.py (https://github.com/quantmind/pulsar/blob/master/examples/snippets/remote.py), what exactly is the metaclass doing? I understand what a metaclass is and does, but the code confuses me. It has the same for loop twice. Is it just creating a class attribute that has the names of all the remote functions? 

Apologies for so many questions. Pulsar looks really great. I am just new to this area of programming and have a lot of things to learn :).

[lsbardel]

Hi Michael,

So from my understanding, if you're using processes, then pulsar will connect event loops to ports (with their mailboxes) for sending and receiving messages. If you're using threads, it will basically just be using the python Threading module (with all the GIL issues). How accurate/inaccurate is my understanding?

It is correct, threads actors are just for development really, you should not take them seriously. But not because of the GIL!

Pulsar is an asynchronous framework, therefore it is based on the assumption that no piece of code block for too long (i.e. more than few milliseconds).

Threads don't make much sense in pulsar unless you need to use blocking IO third party code, but than you can use the executor (see below) in a process-based actor.

 

Also - what is the "IPC" penalty the comment mentions? Is it really going to be that much slower than erlang or java?

 

Those are just comments, benchmarks can prove that wrong, or right, depending on the implementation of your code.

 

If Threading does get locked by the GIL, is there a way to get around that for multiple cores, maybe with Cython? 

Pulsar should not do any blocking operation if concurrency is what you are looking for. That applies to threads, processes or greenlets.

If you are looking to perform CPU bound operations, than concurrency is not what you need.

 

Finally, how do you configure pulsar to use one or the other in the first place?

Use pulsar with default config, process concurrency.

 

2) In the documentation (https://quantmind.github.io/pulsar/design.html) it mentions using executor() for intense processes. I can't find this function anywhere or in the code - what is this referring to, and what are its use cases? Can you show an example?

Docs needs some love ;-), the executor is an asyncio feature https://docs.python.org/3/library/asyncio-eventloop.html#executor 

The executor can be used to perform IO blocking operations, i.e. IO operations which don't have an asynchronous implementation (third party libraries, such as database querying and so forth). But not CPU bound operations!

 

3) In the example remote.py (https://github.com/quantmind/pulsar/blob/master/examples/snippets/remote.py), what exactly is the metaclass doing? I understand what a metaclass is and does, but the code confuses me. It has the same for loop twice. Is it just creating a class attribute that has the names of all the remote functions? 

The metaclass collects all methods prefixed with remote_ into a list of remote functions 

Apologies for so many questions. Pulsar looks really great. I am just new to this area of programming and have a lot of things to learn :).

No problem, your questions are welcome!