One interpreter-per-process limit
1,994 views
Skip to first unread message
Jason McCampbell
Dec 6, 2018, 1:58:17 AM12/6/18
to cython-users
Hi all,
I am experimenting w/ Cython in our system and was getting fairly good performance results until I ran into the error "Interpreter change detected - this module can only be loaded into one interpreter per process". The code is clear that it limits loading to one interpreter. Is this a fundamental limitation like mutable data stored in static C variables? Or is just something no one has asked for before? We use a lot of scratch interpreters to keep independent tasks isolated from one another which is when I ran into the error.
Best regards,
Jason
Stefan Behnel
Dec 6, 2018, 3:24:59 AM12/6/18
to Cython-users
Hi,
Jason McCampbell schrieb am 05.12.18 um 23:14:
global constants (both C and Python objects), various optimisations here
and there, …
> Or is just something no one has asked for before? We use a lot of scratch
> interpreters to keep independent tasks isolated from one another which is
> when I ran into the error.
In theory, PEP-489 would allow this.
https://www.python.org/dev/peps/pep-0489/
In practice, it's not that easy, because avoiding global state requires a
lot of work and makes some things slower, especially access to module
globals. It also cannot be done in normal C in all cases, because global
cdef functions simply do not have access to non-static module globals,
since you cannot pass an additional context into them without changing
their signature. Imagine a (statically defined global) C callback function
that tries to do a type check against a (module/runtime instance specific)
extension type. Which is the right type to check against in that case?
Depending on how such a function gets called, there might not even be a
thread-local to recover its global module context from.
These things could still be done by generating module specific C functions
at runtime, but then you're really leaving the platform independent sector
of C.
These problems are not specific to Cython. Most CPython extension modules
are not prepared to work with multiple interpreters, mostly for the same
reasons. These issues can be worked around in some cases by carefully
crafting the global state in a way that allows it to be shared across
interpreters, but this is such a special case that Cython rather assumes
that the module init code is not safe to be re-executed.
And the code that Cython generates internally is also far from PEP-489
clean. I started doing some work towards getting rid of globals here:
https://github.com/cython/cython/pull/1919
And, what a surprise, it's not easy. It turned out that PEP-489 isn't
really enough here, so PEP-573 was written to investigate the details and
resolve the remaining issues.
https://www.python.org/dev/peps/pep-0573/
Very long story short: Cython detects reloading now and prevents it, rather
than crashing in arbitrary places or leaking resources. The situation will
probably improve over time (and help is always appreciated), but that's how
things are now.
Stefan
Jason McCampbell schrieb am 05.12.18 um 23:14:
> I am experimenting w/ Cython in our system and was getting fairly good
> performance results until I ran into the error "Interpreter change detected
> - this module can only be loaded into one interpreter per process". The
> code is clear that it limits loading to one interpreter. Is this a
> fundamental limitation like mutable data stored in static C variables?
Yes, it's mostly about global C state. That involves extension types,
> performance results until I ran into the error "Interpreter change detected
> - this module can only be loaded into one interpreter per process". The
> code is clear that it limits loading to one interpreter. Is this a
> fundamental limitation like mutable data stored in static C variables?
global constants (both C and Python objects), various optimisations here
and there, …
> Or is just something no one has asked for before? We use a lot of scratch
> interpreters to keep independent tasks isolated from one another which is
> when I ran into the error.
https://www.python.org/dev/peps/pep-0489/
In practice, it's not that easy, because avoiding global state requires a
lot of work and makes some things slower, especially access to module
globals. It also cannot be done in normal C in all cases, because global
cdef functions simply do not have access to non-static module globals,
since you cannot pass an additional context into them without changing
their signature. Imagine a (statically defined global) C callback function
that tries to do a type check against a (module/runtime instance specific)
extension type. Which is the right type to check against in that case?
Depending on how such a function gets called, there might not even be a
thread-local to recover its global module context from.
These things could still be done by generating module specific C functions
at runtime, but then you're really leaving the platform independent sector
of C.
These problems are not specific to Cython. Most CPython extension modules
are not prepared to work with multiple interpreters, mostly for the same
reasons. These issues can be worked around in some cases by carefully
crafting the global state in a way that allows it to be shared across
interpreters, but this is such a special case that Cython rather assumes
that the module init code is not safe to be re-executed.
And the code that Cython generates internally is also far from PEP-489
clean. I started doing some work towards getting rid of globals here:
https://github.com/cython/cython/pull/1919
And, what a surprise, it's not easy. It turned out that PEP-489 isn't
really enough here, so PEP-573 was written to investigate the details and
resolve the remaining issues.
https://www.python.org/dev/peps/pep-0573/
Very long story short: Cython detects reloading now and prevents it, rather
than crashing in arbitrary places or leaking resources. The situation will
probably improve over time (and help is always appreciated), but that's how
things are now.
Stefan
Jason McCampbell
Dec 6, 2018, 1:25:22 PM12/6/18
to cython-users
Hi Stefan,
Thank you very much for the detailed response, understanding what's involved helps a lot.
Regards,
Jason
Reply all
Reply to author
Forward
0 new messages