[perl #133021] Removed the word "discouraged" from threads' documentation

[E. Choroba]

# New Ticket Created by "E. Choroba"
# Please include the string: [perl #133021]
# in the subject line of all future correspondence about this issue.
# <URL: https://rt.perl.org/Ticket/Display.html?id=133021 >


-----------------------------------------------------------------
[Please describe your issue here]

The documentation of threads.pm says that perl threads are
"discouraged". This causes lots of confusion, as the term is defined
in perlpolicy in the following way:

discouraged
From time to time, we may mark language constructs and
features which we consider to have been mistakes as
discouraged. Discouraged features aren't currently candidates
for removal, but we may later deprecate them if they're found
to stand in the way of a significant improvement to the Perl
core.

See also #125106, which seems to lead nowhere, maybe because it was
too ambitious. This patch provides just the minimal change.

See also http://www.perlmonks.org/?node_id=1107534 .

[Please do not change anything below this line]
-----------------------------------------------------------------
---
Flags:
category=library
severity=medium
module=threads
---
Site configuration information for perl 5.27.11:

Configured by choroba at Sat Mar 24 15:15:34 CET 2018.

Summary of my perl5 (revision 5 version 27 subversion 11) configuration:
Commit id: e77195360c1a2b081a806df017630b3165e2aeac
Platform:
osname=linux
osvers=4.4.120-45-default
archname=x86_64-linux-thread-multi
uname='linux still 4.4.120-45-default #1 smp wed mar 14 20:51:49 utc 2018 (623211f) x86_64 x86_64 x86_64 gnulinux '
config_args='-Dusedevel -de -Dusethreads -Dprefix=/home/choroba/localperl'
hint=recommended
useposix=true
d_sigaction=define
useithreads=define
usemultiplicity=define
use64bitint=define
use64bitall=define
uselongdouble=undef
usemymalloc=n
default_inc_excludes_dot=define
bincompat5005=undef
Compiler:
cc='cc'
ccflags ='-D_REENTRANT -D_GNU_SOURCE -fwrapv -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -D_FORTIFY_SOURCE=2'
optimize='-O2'
cppflags='-D_REENTRANT -D_GNU_SOURCE -fwrapv -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include'
ccversion=''
gccversion='4.8.5'
gccosandvers=''
intsize=4
longsize=8
ptrsize=8
doublesize=8
byteorder=12345678
doublekind=3
d_longlong=define
longlongsize=8
d_longdbl=define
longdblsize=16
longdblkind=3
ivtype='long'
ivsize=8
nvtype='double'
nvsize=8
Off_t='off_t'
lseeksize=8
alignbytes=8
prototype=define
Linker and Libraries:
ld='cc'
ldflags =' -fstack-protector -L/usr/local/lib'
libpth=/usr/local/lib /usr/lib64/gcc/x86_64-suse-linux/4.8/include-fixed /usr/lib64/gcc/x86_64-suse-linux/4.8/../../../../x86_64-suse-linux/lib /usr/lib /lib/../lib64 /usr/lib/../lib64 /lib /lib64 /usr/lib64 /usr/local/lib64
libs=-lpthread -lnsl -ldl -lm -lcrypt -lutil -lc
perllibs=-lpthread -lnsl -ldl -lm -lcrypt -lutil -lc
libc=libc-2.22.so
so=so
useshrplib=false
libperl=libperl.a
gnulibc_version='2.22'
Dynamic Linking:
dlsrc=dl_dlopen.xs
dlext=so
d_dlsymun=undef
ccdlflags='-Wl,-E'
cccdlflags='-fPIC'
lddlflags='-shared -O2 -L/usr/local/lib -fstack-protector'


---
@INC for perl 5.27.11:
/home/choroba/localperl/lib/site_perl/5.27.11/x86_64-linux-thread-multi
/home/choroba/localperl/lib/site_perl/5.27.11
/home/choroba/localperl/lib/5.27.11/x86_64-linux-thread-multi
/home/choroba/localperl/lib/5.27.11

---
Environment for perl 5.27.11:
HOME=/home/choroba
LANG=en_US.UTF-8
LANGUAGE (unset)
LD_LIBRARY_PATH (unset)
LOGDIR (unset)
PATH=/home/choroba/perl5/bin:/home/choroba/bin:/bin:/sbin:/usr/bin:/usr/local/bin:/usr/games:/usr/X11R6/bin:/opt/gnome/bin:.
PERL_BADLANG (unset)
SHELL=/bin/bash

[Sergey Aleynikov via RT]

On Sat, 24 Mar 2018 09:31:06 -0700, cho...@matfyz.cz wrote:
> From time to time, we may mark language constructs and
> features which we consider to have been mistakes as
> discouraged.

Threads in their current implementation are arguable a mistake. You loose ~30% of performance and who-measured-how-much memory by just using perl build with threads. And most of XS CPAN is up to some degree threads-unsafe (pure-perl CPAN is safe, so the goals for threads is reached, but still).

> Discouraged features aren't currently candidates
> for removal, but we may later deprecate them if they're found
> to stand in the way of a significant improvement to the Perl
> core.

Threads are not going anywhere by this definition. Just discouraged.

---
via perlbug: queue: perl5 status: open
https://rt.perl.org/Ticket/Display.html?id=133021

[Dave Mitchell]

On Sun, Mar 25, 2018 at 04:20:13PM -0700, Sergey Aleynikov via RT wrote:
> Threads in their current implementation are arguable a mistake. You
> loose ~30% of performance and who-measured-how-much memory by just using
> perl build with threads.

The test suite on blead on my system runs 17% slower under ithreads, but
that includes running a bunch of ithreads-only tests too. So I think 30%
is overstating it.

I'm not aware that a normal single-threaded process uses significantly
more memory under ithreads.


--
Hofstadter's Law: It always takes longer than you expect, even when you
take into account Hofstadter's Law.

[Sergey Aleynikov via RT]

On Mon, 26 Mar 2018 01:36:54 -0700, davem wrote:
> The test suite on blead on my system runs 17% slower under ithreads, but
> that includes running a bunch of ithreads-only tests too. So I think 30%
> is overstating it.

It depends on the operation type. I've measured hash accesses and got ~10% loss on threaded build for 5.26.1.

[Dave Mitchell]

On Mon, Mar 26, 2018 at 04:52:54AM -0700, Sergey Aleynikov via RT wrote:
> On Mon, 26 Mar 2018 01:36:54 -0700, davem wrote:
> > The test suite on blead on my system runs 17% slower under ithreads, but
> > that includes running a bunch of ithreads-only tests too. So I think 30%
> > is overstating it.
>
> It depends on the operation type. I've measured hash accesses and got
> ~10% loss on threaded build for 5.26.1.

10% isn't 30%.


--
A walk of a thousand miles begins with a single step...
then continues for another 1,999,999 or so.

[Jerry D. Hedden via RT]

As one of several people who have maintained the various threads modules over the years, I regret that I missed being part of the original discussion that lead to the inclusion of the 'discouraged' message.

The fact is that threads work, they are maintained, and they currently do not have any bugs preventing their use. I acknowledge that not all Perl modules are thread-safe, but there is sufficient documentation to that affect in the POD.

I also acknowledge that the threads implementation is not ideal nor optimal. Nonetheless, threads are useful, and are being used in the wild. (I, for one, have even used them to good effect in production code. <Gasp!>) Yes, if you don't know what you're doing, threads can be problematic. ("There be dragons...," and all that.) However, the same can be argued to greater or lesser degrees of any programming language feature in the hands of unsophisticated users.

I feel that, while the wording of the POD notice is reasonable, the WARNING heading is alarmist. I feel strongly that the heading should be changed to NOTICE, and the word 'discouraged' be changed to 'not recommended' (as per the original poster's patch).

Since the addition of the 'discouraged' message, I have received several emails from professional Perl developers from around the world expressing concern about it. I expressed to them the same opinions I have given above, namely that threads work (but you have to know what you're doing), and that threads have not been deprecated.

I'm not trying to convert anyone who doesn't like "interpreter-based threads". Don't use them as you so choose. After all, Perl has always been about options. However, there ARE Perl developers that do feel they are a VERY useful feature.

Just my two cents.

[Karl Williamson]

I agree with the above. And I really don't like 'not recommended' as I
think it is too strong. Maybe just list the problems

Even a 30% slowdown will be fully acceptable if you can divide the work
up into 8 or 16 parallel pieces. The gain far outweighs the cost.

[Dave Mitchell]

On Mon, Mar 26, 2018 at 07:20:29PM -0600, Karl Williamson wrote:
> On 03/26/2018 06:54 PM, Jerry D. Hedden via RT wrote:
> > As one of several people who have maintained the various threads modules over the years, I regret that I missed being part of the original discussion that lead to the inclusion of the 'discouraged' message.
> >
> > The fact is that threads work, they are maintained, and they currently do not have any bugs preventing their use. I acknowledge that not all Perl modules are thread-safe, but there is sufficient documentation to that affect in the POD.
> >
> > I also acknowledge that the threads implementation is not ideal nor optimal. Nonetheless, threads are useful, and are being used in the wild. (I, for one, have even used them to good effect in production code. <Gasp!>) Yes, if you don't know what you're doing, threads can be problematic. ("There be dragons...," and all that.) However, the same can be argued to greater or lesser degrees of any programming language feature in the hands of unsophisticated users.
> >
> > I feel that, while the wording of the POD notice is reasonable, the WARNING heading is alarmist. I feel strongly that the heading should be changed to NOTICE, and the word 'discouraged' be changed to 'not recommended' (as per the original poster's patch).
> >
> > Since the addition of the 'discouraged' message, I have received several emails from professional Perl developers from around the world expressing concern about it. I expressed to them the same opinions I have given above, namely that threads work (but you have to know what you're doing), and that threads have not been deprecated.
> >
> > I'm not trying to convert anyone who doesn't like "interpreter-based threads". Don't use them as you so choose. After all, Perl has always been about options. However, there ARE Perl developers that do feel they are a VERY useful feature.
> >
> > Just my two cents.

+1

> I agree with the above. And I really don't like 'not recommended' as I
> think it is too strong. Maybe just list the problems

+1

--
The Enterprise's efficient long-range scanners detect a temporal vortex
distortion in good time, allowing it to be safely avoided via a minor
course correction.
-- Things That Never Happen in "Star Trek" #21

[Rocco Caputo]

On Mar 27, 2018, at 07:15, Dave Mitchell <da...@iabyn.com> wrote:

On Mon, Mar 26, 2018 at 07:20:29PM -0600, Karl Williamson wrote:

On 03/26/2018 06:54 PM, Jerry D. Hedden via RT wrote:

I feel that, while the wording of the POD notice is reasonable, the WARNING heading is alarmist.  I feel strongly that the heading should be changed to NOTICE, and the word 'discouraged' be changed to 'not recommended' (as per the original poster's patch).

I started the discussion that led to the addition of the discouragement warning.  Its original purpose was to manage users' expectations of the kind of help they'd get from the larger Perl community for trying to use this feature.

Toning down the notice doesn't break its original purpose.

I agree with the above.  And I really don't like 'not recommended' as I

think it is too strong.  Maybe just list the problems

Getting rid of the notice entirely would resume failing to manage users' expectations.

In case someone on-list isn't aware of the advice given off-list, I'm closing with a small list of quotes from freenode #perl.  These are pieces of real advice given to Perl users looking for help using threads.  Some of you may recognize things you've said.

Some of you may be tempted to say that IRC isn't representative of a larger perl community, but many of the vocal opponents on IRC also participate in other areas.  They use and advocate Perl professionally.  They attend or speak at Perl conferences.  They are perl5 porters.

This list includes both old and recent statements:

don't use threads

When someone says "I need to use threads" in perl, they (99.9% of the time) mean "I don't know what I'm talking about, please (re-)educate me"

But at least you didn't start out trying to use threads.

Yes.. Don't use threads. At all.

The simple answer is "don't use threads." Just fork if you must.

no, because threads are usually a stupid answer.

do not use threads.

'use threads' is "please let me use the windows fork hack code"

also, don't use threads in Perl; if your problem requires threads, don't use Perl

you seem to have missed the point earlier that was made about "dont use threads"

the thread lecture is don't use threads in Perl, or don't use Perl

before anything else, notice that people here will start telling you about how you probably shouldn't use threads in Perl.

Hi. Please don't use threads.

I wonder how many times we've told qubit not to use threads and how mayn times they've ignored us.

Because threads are not what you want. Threads in Perl, doubly so

(Almost) Everyone who wants to use threads in perl understands them incorrectly.

very few professional perl devs will use threads for real code

because threads are a stupid way to do concurrency in perl.

I really wish there was this little genie like the little green dude in The Flintstones that would pop up and educate people when they think they want to use threads in perl.

Don't use threads, would be our suggestion

If you want to wrap your head in crazy concurrency bugs and random failures, then use threads.

Suggestion: don't use threads

don't use threads in Perl

perlbot: threads is Friends don't let friends use threads ; Elizabeth on Threads - http://www.perlmonks.org/index.pl?node_id=288022

basically: stop trying to use threads.

and really you don't want to use threads at all

or you can use threads, and we will ignore you when it all breaks.

step 1: don't use threads

The response is don't use threads

never use threads in perl

So then they're always asking "how can I do threads" and we say "don't use threads" and they complain and say "waaaaa I need them for doing lots of things at once" and then we say "no you dno't, you can do other things".. and they give a look of complete utter confusion.

yes but in your case it is because threads are known to be broken and so will cause you problems

First convince your department not to use threads.

I don't imagine you'll get much help.. Our advice is "don't use threads"... hardly anyone here does much if anything with threads.. So likely nobody here really knows

This may be the definitive idea of why allowing people to use threads is a bad idea.

you should not use threads.

don't use threads. problem solved.

don't use threads is the answer

You don't want to use threads.

ah, the truth "I use threads because I'm too dumb to understand select"

But if you're wanting to use threads, odds are you are doing something wrong :)

the answer is invariably "don't use threads"    :)

ah, well, your first mistake was to actually seriously use threads in Perl

look, we can't actually advise you on a concurrency solution unless you tell us what the program's going to do - although the odds are pretty good that we'll say "don't use threads" since that's almost always the worst choice

code containing 'use Switch' or 'use threads' can be assumed to be broken

the best advice I can give is to not use threads in Perl

some idiot did 'use threads;' and now you have a broken program that needs replacing

DON'T USE THREADS, DON'T TELL OTHERS TO USE THREADS.

also, #perl will almost always refuse to help with threads. because threads in perl are shit.

-- 

Rocco Caputo <rca...@pobox.com>

[Sergey Aleynikov via RT]

On Mon, 26 Mar 2018 06:06:47 -0700, davem wrote:
> 10% isn't 30%.

My idea was to show that you can't attribute all the test run slowdown to additional threading tests. Probably, I should've made this clearer in that message. As for the total slowdown figures, it depends on the application, but it's still "free" - you get it even if you don't start any threads.

[Sergey Aleynikov via RT]

On Mon, 26 Mar 2018 18:21:02 -0700, pub...@khwilliamson.com wrote:
> I agree with the above. And I really don't like 'not recommended' as
> I
> think it is too strong. Maybe just list the problems
>
> Even a 30% slowdown will be fully acceptable if you can divide the
> work
> up into 8 or 16 parallel pieces. The gain far outweighs the cost.

But if you use processes for parallelism and not threads, you don't get this slowdown at all and get the same multicore boost. And you don't have to guess which XS module will segfault next. The only case when I find using threads reasonable is writing GUI, where you _have to_ be in the same address space.

[Petr Pisar]

On 2018-03-27, Elizabeth Mattijsen <l...@dijkmat.nl> wrote:
> Perhaps a reference to the “forks” module on CPAN could be added?
>
Which does not work either
<https://rt.cpan.org/Public/Bug/Display.html?id=123248> (because of
thread-safety fixes in File::Glob).

-- Petr

[Aristotle Pagaltzis]

* E. Choroba <perlbug-...@perl.org> [2018-03-25 21:31]:

> See also #125106, which seems to lead nowhere, maybe because it was
> too ambitious. This patch provides just the minimal change.

The minimal change would be to simply remove the sentence, and that is
what I would favour to begin with.


* Christian Walde <walde.c...@gmail.com> [2018-03-26 12:49]:
> Since this change doesn't have a ticket attached to it, i figured it
> would be useful to provide the context of what led to its creation.

Thank you. I vaguely remembered having been a part of that discussion
and am glad to have the context back; I am also relieved to see that my
current position is close to my previous stance.

> This discouragement was introduced by rjbs in commit
> 10a4597703f4b044d3f968bf3923644ea2387f5c. I've extracted some of the
> chat log that led to it.

I regret not catching this at the time, but the reason given for calling
the feature discouraged does not meet the technical definition of the
term in perlpolicy. So the statement in the threads.pm POD is simply not
true.

I must admit my own anti-ithreads bias here, and that this is probably
what led me to skip over that issue. I am not unhappy to see threads
discouraged-in-the-colloquial-sense and have always avoided them myself
(with success).

However, for some people they are the only option.

And they are not ever being removed from Perl, even as a far-future
possibility. They are thus not discouraged by the definition of that
term in perlpolicy. However contentious everything else may be, that
much is unambiguously true.


* Jerry D. Hedden via RT <perlbug...@perl.org> [2018-03-27 02:59]:

> As one of several people who have maintained the various threads
> modules over the years, I regret that I missed being part of the
> original discussion that lead to the inclusion of the 'discouraged'
> message.

Ditto.

> The fact is that threads work, they are maintained, and they currently
> do not have any bugs preventing their use.

Ditto.

> I acknowledge that not all Perl modules are thread-safe, but there is
> sufficient documentation to that affect in the POD.

Ditto.

> I also acknowledge that the threads implementation is not ideal nor
> optimal. Nonetheless, threads are useful, and are being used in the
> wild. (I, for one, have even used them to good effect in production
> code. <Gasp!>) Yes, if you don't know what you're doing, threads can
> be problematic. ("There be dragons...," and all that.) However, the
> same can be argued to greater or lesser degrees of any programming
> language feature in the hands of unsophisticated users.

Here we somewhat depart. To me the main point is that some people some
of the time have no alternative to ithreads. And so if you need to use
them, well then you need to use them.

> I feel that, while the wording of the POD notice is reasonable, the
> WARNING heading is alarmist. I feel strongly that the heading should
> be changed to NOTICE, and the word 'discouraged' be changed to 'not
> recommended' (as per the original poster's patch).

The discouragement notice just belongs deleted. I’m not sure that the
heading would be too strong without it so I’m -0 on changing that.

> Since the addition of the 'discouraged' message, I have received
> several emails from professional Perl developers from around the world
> expressing concern about it. I expressed to them the same opinions
> I have given above, namely that threads work (but you have to know
> what you're doing), and that threads have not been deprecated.

+1

> I'm not trying to convert anyone who doesn't like "interpreter-based
> threads". Don't use them as you so choose. After all, Perl has always
> been about options. However, there ARE Perl developers that do feel
> they are a VERY useful feature.

This may depend on perspective. If I were in a situation without the
choice of something else, I could imagine finding them VERY useful. :-)


* Leon Timmermans <faw...@gmail.com> [2018-03-28 02:30]:
> I think the main problem with the current wording is that it
> discourages without explaining why.

There is a long list of appropriately neutral explanation of footguns,
but only miles down the page under BUGS AND LIMITATIONS. There has to
be a reference to that.

> In particular it doesn't explain that threads.pm does something
> different than what many people expect it to do.
>
> People expect it to be good for "share all the data" scenarios
> (because pretty much anything else called threads is), even though it
> is terrible at that. This confusion isn't helped by threads
> implementations in commonly used implementations of languages
> occupying the same niches as we do (e.g. cpython, Ruby MRI) sucking in
> exactly the opposite way (they're good at sharing but useless at
> actually being parallel).
>
> It's not so much threads.pm that needs a big fat warnings, it's
> threads::shared that does.

+1

I would be in favour of a second patch which adds this up top, which can
be bikeshedded if people feel strongly enough about it. I am willing to
propose new language for that.

First up I’d want to remove the untrue statement, however.


* Rocco Caputo <rca...@pobox.com> [2018-03-27 16:35]:

> I started the discussion that led to the addition of the
> discouragement warning. Its original purpose was to manage users'
> expectations of the kind of help they'd get from the larger Perl

> community for trying to use this feature. […] Getting rid of the

> notice entirely would resume failing to manage users' expectations.

The notice was simply incorrect and therefore could not possibly manage
users’ expectations correctly. If the fact that you’re talking about is
that using threads will get you yelled at by some, then how is any user
expectation managed correctly by *not* telling the user that and instead
just telling them to not use threads?

> In case someone on-list isn't aware of the advice given off-list, I'm
> closing with a small list of quotes from freenode #perl. These are
> pieces of real advice given to Perl users looking for help using
> threads. Some of you may recognize things you've said.

How heavy is the tail on the long tail graph of who these quotes came
from? How many counterexamples of users getting real help with threads
are there?

> Some of you may be tempted to say that IRC isn't representative of
> a larger perl community, but many of the vocal opponents on IRC also
> participate in other areas. They use and advocate Perl professionally.
> They attend or speak at Perl conferences. They are perl5 porters.

Are you aware that what comes after your “but” doesn’t rebut what comes
before it? :-)


* Sergey Aleynikov via RT <perlbug-...@perl.org> [2018-03-26 01:29]:

> On Sat, 24 Mar 2018 09:31:06 -0700, cho...@matfyz.cz wrote:

> > From time to time, we may mark language constructs and
> > features which we consider to have been mistakes as
> > discouraged.
>

> Threads in their current implementation are arguable a mistake. You
> loose ~30% of performance and who-measured-how-much memory by just
> using perl build with threads. And most of XS CPAN is up to some
> degree threads-unsafe (pure-perl CPAN is safe, so the goals for
> threads is reached, but still).

Any feasible implementation would be a mistake of a different kind. So
faced with the problem of needing some kind of threads but having only
bad choices for their implementation, Perl made a different bad choice
than other similar languages. (Cf. Leon’s reply.)

And no random scary numbers pulled out of a hat please. I don’t mind
mentioning that thread-supporting perls are slower at an above-noise
level, but if you’re going to quote specific numbers, they must come
with actual benchmarks to contextualise them. Just “30% slower!!1!” is
FUD. (I myself compile my perls without threads, because if I never ever
use threads then why pay for them? That is an argument that holds even
in absence of any exact figures.)

> > Discouraged features aren't currently candidates
> > for removal, but we may later deprecate them if they're found
> > to stand in the way of a significant improvement to the Perl
> > core.
>

> Threads are not going anywhere by this definition.

Indeed.

> Just discouraged.

In the colloquial sense maybe, but as you found, not in the perlpolicy
sense.


* Sergey Aleynikov via RT <perlbug-...@perl.org> [2018-03-28 03:04]:

> On Mon, 26 Mar 2018 18:21:02 -0700, pub...@khwilliamson.com wrote:

> > I agree with the above. And I really don't like 'not recommended' as
> > I think it is too strong. Maybe just list the problems
> >

> > Even a 30% slowdown will be fully acceptable if you can divide the
> > work up into 8 or 16 parallel pieces. The gain far outweighs the
> > cost.
>
> But if you use processes for parallelism and not threads, you don't
> get this slowdown at all and get the same multicore boost. And you
> don't have to guess which XS module will segfault next. The only case
> when I find using threads reasonable is writing GUI, where you _have
> to_ be in the same address space.

Except some people do not have the option.


Regards,
--
Aristotle Pagaltzis // <http://plasmasturm.org/>

[Karl Williamson]

It turns out that #125106 also is about this topic, and I have merged
the two tickets. You might want to look at the discussion there.

[Rocco Caputo]

On Apr 5, 2018, at 04:39, Sawyer X <xsaw...@gmail.com> wrote:

Did I miss anything?

I think so.  The only part of the existing paragraph that tries to warn the reader about the difference between "supported" and "you will receive support":

"Few people ... will be able to provide help."

Even that is an understatement.  Maybe it should say "willing or able".

In the context of the discussions leading up to the change, telling people that they won't get much help is the most important part of that paragraph.  The rest of it merely attempts to justify why it has to be said.

-- 

Rocco Caputo <rca...@pobox.com>

[Dave Mitchell]

On Mon, Apr 09, 2018 at 12:18:10PM +0300, Sawyer X wrote:
> On 04/09/2018 10:51 AM, Dave Mitchell wrote:
> > On Sun, Apr 08, 2018 at 10:23:20PM +0000, Christian Walde wrote:
> >> - perl threads can work fine under a certain load, but crash randomly once
> >> the load becomes too big
> >> - perl threads can work fine on one computer, but crash randomly on another
> >> due to slight module differences
> > Is this based on the idea that the ithreads implementation itself is
> > inherently buggy, or that some modules may not be thread-safe.
> >
> > I'd be surprised if the former, and if the latter, surely that applies
> > to any "real" threading implementation in any programming language?
>
> Could you please expand on this, Dave?

1) I think the perl ithreads implementation is fairly bug-free and stable
these days.

2) If one's threaded code randomly craps out due to a 3rd party module
being non-thread-safe, or supposedly thread-safe but buggy, then that's an
issue with the third-party module. In principle any programming language
will struggle with threads if used with a 3rd-party library that isn't
thread-safe.

Perl's threads are supposedly safe at the perl language level. For
example, if two threads simultaneously try to push a value to the same
array, then
a) the internal representation of the array (AV etc) won't be
corrupted;
b) both elements will get pushed;
c) perl won't guarantee which order they're pushed in.

This is a lot better guarantee than threads in C for example.

The main danger area is with XS modules; pure perl modules should
at least not corrupt the interpreter state; whether they work correctly
(e.g. two threads both trying to update the same data file) is down to
whether the module's author does the correct locking etc.

--
No matter how many dust sheets you use, you will get paint on the carpet.

[Dave Mitchell]

(missed out Ccing p5p)

----- Forwarded message from Dave Mitchell <da...@iabyn.com> -----

Date: Mon, 9 Apr 2018 15:59:14 +0100
From: Dave Mitchell <da...@iabyn.com>
To: Elizabeth Mattijsen <l...@dijkmat.nl>
Subject: Re: [perl #133021] Removed the word "discouraged" from threads'
documentation
Message-ID: <20180409145...@iabyn.com>

On Mon, Apr 09, 2018 at 02:34:21PM +0200, Elizabeth Mattijsen wrote:
> Is that something that was fixed in the past 10 years or so? I
> distinctly remember this only being true for *shared* arrays, which
> involve a lot of overhead.

Well, non-shared arrays can only be accessed by a single thread, so they
are not an issue.

Running the following code, which has 100 threads pushing and popping things
off a shared array shows that it ends with the same number of elements
as when it started.

use threads;
use threads::shared;

my @a : shared = (1..100);

sub inc {
for (1..10_000) {
push @a, $_;
pop @a;
}
}

my @t;
push @t, threads->new(\&inc) for 1..100;
$_->join for @t;
printf "size=%d\n", scalar(@a);


I've run this code on 5.10.1 and 5.27.10 and in both cases it finishes
with 100 elements.

> OOC, is this also true for hashes to which keys are being added by
> several threads?

(I don't know what OOC stands for).

If the different threads are using disjoint sets of keys when accessing
the same shared hash, they won't interfere with each other. If multiple
threads are adding, modifying and deleting the same key in a shared hash,
then of course the final result for that key in the hash will depend on
the timing; but it won't be corrupt.

> Also OOC, if multiple threads increase / decrease the reference count of
> something, is that also threadsafe by default? AKA, will you never get
> double frees (increment missed) or DESTROY never getting called
> (decrement missed)?

The whole point of ithreads is that each thread gets its own interpreter,
with its own copy of every SV. So it's not possible for multiple threads
to access, let alone modify, the reference count of an SV. threads::shared
uses locking behind the scenes to mediate access to shared variables, so
again, ref-counting isn't an issue.

--
"You're so sadly neglected, and often ignored.
A poor second to Belgium, When going abroad."
-- Monty Python, "Finland"

----- End forwarded message -----

--
Please note that ash-trays are provided for the use of smokers,
whereas the floor is provided for the use of all patrons.
-- Bill Royston

[Father Chrysostomos via RT]

On Mon, 09 Apr 2018 08:02:10 -0700, davem wrote:
> (I don't know what OOC stands for).

Probably ‘out of curiosity’. Or maybe it’s a Dutch word.

--

Father Chrysostomos

---
via perlbug: queue: perl5 status: open

https://rt.perl.org/Ticket/Display.html?id=125106

[Dave Mitchell]

On Mon, Apr 09, 2018 at 07:26:30PM +0200, Elizabeth Mattijsen wrote:
> Ok, so if I understand you correctly: if you run some pthreads outside
> of a Perl interpreter thread, you will never be able to run any Perl
> code in that pthreads thread that could potentially access variables
> from a Perl interpreter running in another thread?

I'm not sure I understand that sentence, but the basic rule of doing any
sort of threading in perl is that two OS-level threads must not
simultaneously access/modify the same perl interpreter struct (and all the
data hanging off it, e.g. SVs etc).

This was why the 5005THREADS model was abandoned. Nobody could think of a
way of avoiding all the side-effects that any trivial usage of an SV might
give rise to (e.g. changing the ref count, converting from an IV to PVIV),
without having a lock in about every second line of code in the perl
core.

I vaguely understand that the approach in Python is to have one big
central lock so only one thread can execute at any one time.

--
"I used to be with it, but then they changed what ‘it’ was, and now what
I’m with isn’t it. And what’s ‘it’ seems weird and scary to me."
-- Grandpa Simpson
(It will happen to you too.)

[Dave Mitchell]

On Mon, Apr 09, 2018 at 11:01:22PM +0200, Christian Walde wrote:
> > 2) If one's threaded code randomly craps out due to a 3rd party module
> > being non-thread-safe, or supposedly thread-safe but buggy, then that's an
> > issue with the third-party module. In principle any programming language
> > will struggle with threads if used with a 3rd-party library that isn't
> > thread-safe.
>

> This has dangerous assumptions in it:
>
> - Thread safety is not a binary on/off switch. Modules can appear to be
> thread-safe under some conditions and not be so under others. That's why
> i called it a heisenbug in my previous email.
>
> - Putting the blame on 3rd party modules is fine for most languages
> where it's easier to switch to other implementations or roll your own.
> However this is Perl. The language lives on CPAN, not in the
> interpreter. Asking someone who just spent hours implementing a parallel
> web downloader to gradually replace any of the bits they used will not
> be taken well.
>
> - Very few cpan maintainers know how to do thread-safety. Heck, most
> don't even practice string/binary safety for filehandles. Most of them
> are Perl developers first and you're asking them to move at least 2
> degrees of separation outside of their zone of comfort. For a lot of
> modules there's simply a good chance this is not going to happen.

But everything you've said above applies just as much to any 3rd-party
library/module in any programming language which supports concurrency.
I really fail to see how CPAN is different in this regard.

Most CPAN authors don't need to worry about thread-safety, as long as it's
a perl-level module. Perl is thread-safe at the language level, and you
don't need to do anything special to make Perl code thread-safe. Or at
least, nothing more than you would have to do to make your module safe
across forks (which is what ithreads is emulating after all). For example,
if a thread is created, the module's variables will be cloned, and a
destructor might be called for a variable in each of the threads. And of
course if the module is updating something external like a file, then it
may need to exclusively lock that file.

XS code is of course a lot harder, because now you're doing concurrency
programming in C. But even here, its easier than general C concurrency,
because of perl's ITHREADS "not shared by default" model. You mainly have
to avoid global/static mutable data, and be prepared for any of your
data stored in SVs (e.g. DB handles) being cloned when a new thread is
created.

--
All wight. I will give you one more chance. This time, I want to hear
no Wubens. No Weginalds. No Wudolf the wed-nosed weindeers.
-- Life of Brian

[Sergey Aleynikov via RT]

On Tue, 10 Apr 2018 04:04:06 -0700, davem wrote:

The problem is in people's expectations. When most part of something is thread-safe, they'll just assume "everything is thread-safe" and then'll be hit very hard. One of the examples I like is Mouse, which was thread-unsafe until the year 2015. While it's not as popular as Moose, it's still a module with large a large user base. And your chances to encounter bugs in not so popular XS modules are much, much higher. What's even worse, you're likely to hit them only under some unlucky circumstances (see example below).

As for the core, I agree that it's now much more thread-stable than before. Somewhere around 5.12 or 5.14 I've prepared a talk named "no threads" with some nice crash examples in it - they crash no more. But still, locale handling was thread-unsafe until 5.24 (or 5.26?) - just because of no one has discovered that. And the following still fails loudly (though not dumps a core);

while (1) {
push @foo, threads->create(sub {
require IO::Handle;
});
$_->detach for(splice @foo);
}

So while yes, perl is much more thread-safe inherently as it used to be, I won't recommend using threads in it to anyone.

[Dave Mitchell]

On Tue, Apr 10, 2018 at 02:59:50PM +0200, Christian Walde wrote:

> On Tue, 10 Apr 2018 13:03:45 +0200, Dave Mitchell <da...@iabyn.com> wrote:
>
> > I really fail to see how CPAN is different in this regard.
>

> Because this is Perl where not using CPAN is not an option and CPAN
> library consumers rely on CPAN libraries being almost entirely
> rock-solid, or at least the author having easy and quick ways to fix
> bugs.

You still haven't differentiated perl+CPAN from some_other_language +
3rd_party_libraries_needed_to_get_the_job_done.

If using a programming language other than perl, then it is likely that:

1) not using 3rd-party libraries is not an option;
2) consumers of 3rd-party libraries rely on those libraries being almost
entirely rock-solid, or at least the author having easy and quick ways
to fix bugs.

> If you really look at all 3 points i made, in aggregate, and don't see
> how this is a problem and a danger, then i don't think i can come up
> with other word combinations to make you see it.

Can you come up with a hypothetical scenario, e.g. a multi-threaded
program that uses libraries to connect to a database and retrieve and
parse some XML data. Then got through it step by step so that I can see
why using perl and CPAN is dangerous, but using (e.g.) java and a DB and
XML library is safe? What is the crucial difference between the two that
flips it from being safe to unsafe?

--
If life gives you lemons, you'll probably develop a citric acid allergy.

[Dave Mitchell]

On Tue, Apr 10, 2018 at 06:42:44AM -0700, Sergey Aleynikov via RT wrote:
> As for the core, I agree that it's now much more thread-stable than
> before. Somewhere around 5.12 or 5.14 I've prepared a talk named "no
> threads" with some nice crash examples in it - they crash no more. But
> still, locale handling was thread-unsafe until 5.24 (or 5.26?) - just
> because of no one has discovered that. And the following still fails
> loudly (though not dumps a core);
>
> while (1) {
> push @foo, threads->create(sub {
> require IO::Handle;
> });
> $_->detach for(splice @foo);
> }

Oh, that's fun, Looks like _create_getline_subs in IO.xs is directly
modifying the global PL_check[] rather than via the official API which
does the necessary locking.

> So while yes, perl is much more thread-safe inherently as it used to be,
> I won't recommend using threads in it to anyone.

Perl has bugs. Perl's threading has bugs. I haven't seen any particular
evidence yet that the number of threaded bugs is disproportionately large
on recent perls.

--
The Enterprise successfully ferries an alien VIP from one place to another
without serious incident.
-- Things That Never Happen in "Star Trek" #7

[Karl Williamson]

On 04/10/2018 07:42 AM, Sergey Aleynikov via RT wrote:
> On Tue, 10 Apr 2018 04:04:06 -0700, davem wrote:
>
> The problem is in people's expectations. When most part of something is thread-safe, they'll just assume "everything is thread-safe" and then'll be hit very hard. One of the examples I like is Mouse, which was thread-unsafe until the year 2015. While it's not as popular as Moose, it's still a module with large a large user base. And your chances to encounter bugs in not so popular XS modules are much, much higher. What's even worse, you're likely to hit them only under some unlucky circumstances (see example below).
>
> As for the core, I agree that it's now much more thread-stable than before. Somewhere around 5.12 or 5.14 I've prepared a talk named "no threads" with some nice crash examples in it - they crash no more. But still, locale handling was thread-unsafe until 5.24 (or 5.26?) - just because of no one has discovered that.

Actually threaded pure perl programs are still unsafe in 5.26. perl
switches the locale behind your back, even if you follow our admonitions
to not explicitly use locales. I'm to blame for some of these, in my
earlier naivete, and some have been there for a long time. Sergey found
one case that I added in 5.24 I believe, and contributed a test file to
verify it's still fixed

5.28 uses thread safe locales if available on the system. On other
systems, I avoid switching locales, and added a mutex for those cases
where switching is still done.

When perl starts up, it reads the environment to see what each locale
category should be set to. If not all categories are set to the same
thing, this would cause perl to potentially switch locales to gather
information about the outliers. This is potentially problematic on
unsafe threaded builds. I solved this by gathering the information at
start up, and caching it. This is one of the ways 5.28 avoids switching
locales.

In researching this, I looked in the POSIX standard for functions it
allows to be non-thread safe. I have a WIP to add cautions about these
to XS writers. I also noticed that the Linux man pages indicated that
they have failed to implement correctly some that are supposed to be
thread-safe. Other systems may implement these safely, but not others.

I searched the perl source code for instances of these calls. And then
manually started to examine them to see if there was a problem. I have
not finished (and of course I may make mistakes in my analysis). The
glaring case where there is a problem is in accessing the environment
(getenv() et.al.) These need to be protected by a mutex, but it's only
a problem if another thread is changing the environment at the same
time, a much less common occurrence. My guess is that these aren't
crashing things because most environment changes would tend to be done
at start-up, even before thread creation. Things you might not expect
to, without thinking about it, like tzset(), do access the environment,
and there is a potential race if the environment is changed by another
thread during tzset's execution. And tzset is called from places that
at first glance you wouldn't expect.

And the following still fails loudly (though nott dumps a core);

[Rocco Caputo]

On Apr 10, 2018, at 10:27, Dave Mitchell <da...@iabyn.com> wrote:

I am all in favour of having, at the same location as the current
'discouraged' text, but instead of it, a big flashing neon sign saying
that perl threads are a bit different from what you might expect and only
use them if you understand that (e.g. each thread is a non-shared clone
of the parent, with memory and start-up-cost implications).

Also that you will mostly be on your own navigating the differences,

as Perl's users on average are reluctant at best to help you with them

and will often "actively encourage" you to seek other implementations.

-- 

Rocco Caputo <rca...@pobox.com>

[bulk88 via RT]

On Tue, 10 Apr 2018 15:25:00 -0700, LeonT wrote:
> If that's not what one wants, it's probably possible to implement GIL
> threading instead (though I'm not sure if we'd really want that
> either). I think that's unexplored territory though.

Wouldnt one of the existing fibers/future/promises/Coro/async-but-not-async-with-explicit-yields modules that swap Perl stacks inside the same interp/same perl thread be perl's already-implemented GIL concept?

--
bulk88 ~ bulk88 at hotmail.com