Threaded-perl & C++XS future.

[Олег Пронин]

Currently i'm having a lot of troubles with developing XS frameworks
while keeping support for threaded perls

1) annoying pTHX/aTHX
2) annoying class members 'my_perl' needed for destructors (or dTHX
instead) when PERL_NO_GET_CONTEXT (because C++ desctructors has no
args, you cant pass aTHX, so you either save my_perl as an object
member in constructor, or use less efficient dTHX in desctructor).
The same applies for callbacks which are called from C libs
(libevent/libuv) and works with perl variables.
3) C++ XS frameworks not working after threads->create, because you
either get core dump on double C++ object delete, or (if CLONE_SKIP)
get a ref to undef which is not better anyway.

I didn't find any callback/hook that perl calls on every blessed
object when perl_clone().
Is there any?

If not it seems impossible to properly implement XS objects with C
underlying data.
What is the status of threaded perl? does anybody use it? will it be
okay if XS cpan modules will no longer support it?

--
Oleg Pronin,
CTO, Co-Founder,
Crazy Panda LTD
CP Decision LTD

[Leon Timmermans]

On Thu, Feb 5, 2015 at 9:15 PM, Олег Пронин <sy...@crazypanda.ru> wrote:

Currently i'm having a lot of troubles with developing XS frameworks
while keeping support for threaded perls

1) annoying pTHX/aTHX
2) annoying class members 'my_perl' needed for destructors (or dTHX
instead) when PERL_NO_GET_CONTEXT (because C++ desctructors has no
args, you cant pass aTHX, so you either save my_perl as an object
member in constructor, or use less efficient dTHX in desctructor).
The same applies for callbacks which are called from C libs
(libevent/libuv) and works with perl variables.

Yes, this is certainly annoying, but I'm not really seeing a clean solution to this issue. The macro-madness that mostly hides these complications in C just doesn't port over well to C++.

 

3) C++ XS frameworks not working after threads->create, because you
either get core dump on double C++ object delete, or (if CLONE_SKIP)
get a ref to undef which is not better anyway.

I didn't find any callback/hook that perl calls on every blessed
object when perl_clone().
Is there any?

There is: svt_dup magic. File::Map and SysV::SharedMem are two modules of mine that use it, there must be more of them. In those particular cases I'm refcounting my resource, in other cases it may make sense to copy them instead.

 

If not it seems impossible to properly implement XS objects with C
underlying data.

It is possible, but more complicated than it should be.
 

What is the status of threaded perl? does anybody use it? will it be
okay if XS cpan modules will no longer support it?

Everyone hates threads on perl, and XS authors more so. You wouldn't be the first not to support it, but personally I do prefer to support it: there are still uses for them.

Leon

[bulk88]

Олег Пронин wrote:
> Currently i'm having a lot of troubles with developing XS frameworks
> while keeping support for threaded perls
>
> 1) annoying pTHX/aTHX

typedef struct{
#ifdef PERL_IMPLICIT_CONTEXT
pTHX;
#endif
MORE_TYPE elements;
} some_struct;

// later on

#ifdef PERL_IMPLICIT_CONTEXT
# define dMYTHX tTHX aTHX = var->aTHX
#else
# define dMYTHX dNOOP
#endif


void
some_function(OBJECT var) {
dMYTHX;
.....

}

> 2) annoying class members 'my_perl' needed for destructors (or dTHX
> instead) when PERL_NO_GET_CONTEXT (because C++ desctructors has no
> args, you cant pass aTHX, so you either save my_perl as an object
> member in constructor, or use less efficient dTHX in desctructor).
> The same applies for callbacks which are called from C libs
> (libevent/libuv) and works with perl variables.

C callbacks almost always have void pointers you supply. Simple remember
to record the aTHX/my_perl in your void * struct. Remember, that "dTHX"
and passing my_perl on C stack must always be syncronized. If your C
library switches OS threads or uses a thread pool, you must call must
call PERL_SET_CONTEXT to move a perl thread between OS threads. If you
move a perl thread between OS threads you must have your own
mutexes/locks to make sure 1 perl thread is never ever running on 2 OS
threads simultaneously. See http://perlmonks.org/?node_id=870109 and
http://perlmonks.org/?node_id=870516 and example isage
http://grep.cpan.me/?q=PERL_SET_THX|PERL_SET_CONTEXT

> 3) C++ XS frameworks not working after threads->create, because you
> either get core dump on double C++ object delete, or (if CLONE_SKIP)
> get a ref to undef which is not better anyway.
> I didn't find any callback/hook that perl calls on every blessed
> object when perl_clone().
> Is there any?

svt_dup see http://perldoc.perl.org/perlguts.html#Magic-Virtual-Tables
you have to decide if your C++ object is "duplicatable" in concept or
not, which can only be determined by design of that C++ library.

See my implementation of using atomic CPU instructions to create a
non-perl refcount ontop of Perl's SV refcounts to handle multiple
ithreads owning and using the same C pointer, and last ithread owner of
the pointer frees the pointer. The C pointer that is shared between
ithreads in Win32-API came from a special "executable" malloc that
allocates memory blocks to place machine code to execute as C functions
later.

https://metacpan.org/source/BULKDD/Win32-API-0.79/Callback/Callback.xs#L273
https://metacpan.org/source/BULKDD/Win32-API-0.79/Callback/Callback.xs#L666
https://metacpan.org/source/BULKDD/Win32-API-0.79/Callback/Callback.xs#L692

> If not it seems impossible to properly implement XS objects with C
> underlying data.

Not true. All the resources are provided to you by Perl core.

> What is the status of threaded perl?

99% of Win32 Perls are threaded. ithreads are the only way to use more
than 1 CPU core without fork, or maintain a "thread pool" or workers.
Other languages such as

> does anybody use it? will it be
> okay if XS cpan modules will no longer support it?

That is your choice. There are modules on CPAN that hard code author's
/home directory with his computer's username to look up their
resources/permanent data tables. Of course, they will never work on
anyone elses computer.

[Steffen Mueller]

On 02/06/2015 06:49 AM, bulk88 wrote:
>> If not it seems impossible to properly implement XS objects with C
>> underlying data.
>
> Not true. All the resources are provided to you by Perl core.

Correct. And I think the specific tool for Oleg is "CLONE" (not
CLONE_SKIP). Using CLONE is incredibly onerous, though, because you
basically have to keep a registry of all objects in the class. Perl
ithread creation already clones all SVs. If there was an inverted way of
doing CLONE that would register "call this per SV that's blessed into my
class" with the interpreter cloning logic, then that would make people's
lives much easier. It might also make thread creation slower ("is this
blessed?"), but I'm not even sure it would compared to using CLONE.

--Steffen

[bulk88]

bulk88 wrote:
>
> 99% of Win32 Perls are threaded. ithreads are the only way to use more
> than 1 CPU core without fork, or maintain a "thread pool" or workers.
> Other languages such as
>
>> does anybody use it? will it be
>> okay if XS cpan modules will no longer support it?

I hit send too soon.

Other languages, such as SpiderMonkey JS, use "my_perl" identically to
Perl (
https://developer.mozilla.org/en-US/docs/Mozilla/Projects/SpiderMonkey/JSAPI_reference/JS_NewContext
) , or if you don't like Perl's dragging around a pointer in a C auto to
every single function call design, there is Python's GIL (
http://en.wikipedia.org/wiki/Global_Interpreter_Lock ) and more
generally, green threads that will never use more CPU than 1 core. Green
thread's only big advantage is using async I/O behind the scenes, but it
looks like much easier to program, sync I/O on language level. If a
faux-sync I/O call is done, the VM engine just sets up the async I/O and
runs another green thread on the 1 and only OS thread.