[rt.cpan.org #85336] Fails often when tested in parallel

[Sisyphus via RT]

Sat May 18 02:20:38 2013: Request 85336 was acted upon.
Transaction: Correspondence added by SISYPHUS
Queue: Inline
Subject: Fails often when tested in parallel
Broken in: 0.53
Severity: (no value)
Owner: Nobody
Requestors: AN...@cpan.org
Status: new
Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=85336 >


On Wed May 15 23:11:13 2013, ANDK wrote:

> There seems to be a race condition somewhere.

Both of the affected test scripts (t/21read_DATA.t and t/22read_DATA.t) use the same Inline build directory (.Inline) - and they therefore need to access the same configuration file.

If you think that could be the source of the problem, I could work around that by designating 2 different (separate) build directories for those 2 test scripts - thereby assuring that each script has its own copy of the configuration details.
Is that worth trying ? (If so, let me know, and I'll upload a new developer version of Inline that does exactly that and we can see how that goes.)

Cheers,
Rob

[(Andreas J. Koenig) via RT]

Sat May 18 10:40:47 2013: Request 85336 was acted upon.
Transaction: Correspondence added by andreas.koe...@franz.ak.mind.de
Queue: Inline
Subject: Re: [rt.cpan.org #85336] Fails often when tested in parallel

Broken in: 0.53
Severity: (no value)
Owner: Nobody
Requestors: AN...@cpan.org

Status: open
Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=85336 >

"Sisyphus via RT" <bug-I...@rt.cpan.org> writes:

> Is that worth trying ?

This, or maybe apply some locking? Depends on how much work it is, and
on how relevant for later real world behaviour it is.

Except for this, I'd leave the judgement to the implementor;)

--
andreas

[David Oswald via RT]

Sat May 18 11:23:47 2013: Request 85336 was acted upon.
Transaction: Correspondence added by daoswald

Queue: Inline
Subject: Re: [rt.cpan.org #85336] Fails often when tested in parallel
Broken in: 0.53
Severity: (no value)
Owner: Nobody
Requestors: AN...@cpan.org
Status: open
Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=85336 >

I get test failures in Inline::CPP when running tests in parallel.
I've gone through Inline::CPP and implemented file locking to
eliminate race conditions, but the issue persists. That leads me to
believe that there needs to be a similar fix in Inline. Changing
Inline's tests to use separate build directories wouldn't fix the
underlying issue of Inline not supporting concurrency. Proper file
locking probably would resolve the issue, not only for Inline, but
also for plugins such as Inline::CPP.

--

David Oswald
daos...@gmail.com

[Sisyphus via RT]

Sun May 19 01:37:37 2013: Request 85336 was acted upon.

Transaction: Correspondence added by SISYPHUS
Queue: Inline
Subject: Fails often when tested in parallel
Broken in: 0.53
Severity: (no value)
Owner: Nobody
Requestors: AN...@cpan.org

Status: open
Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=85336 >

On Sat May 18 10:40:47 2013, andreas.koe...@franz.ak.mind.de wrote:
> "Sisyphus via RT" <bug-I...@rt.cpan.org> writes:
>

> > Is that worth trying ?
>

> This, or maybe apply some locking? Depends on how much work it is, and
> on how relevant for later real world behaviour it is.
>
> Except for this, I'd leave the judgement to the implementor;)

I don't think it's very relevant to real world usage - and hence my (personal) interest in this is nowhere near as keen as it would otherwise be.

However, David Oswald wants to write some patches that will put in place file locking. If there's no problem with those patches, I'll apply them (when I receive them) and release a new devel version of Inline.

Hopefully, that will keep everyone happy :-)

Cheers,
Rob

[Andreas Koenig]

"Sisyphus via RT" <bug-I...@rt.cpan.org> writes:

> Is that worth trying ?

This, or maybe apply some locking? Depends on how much work it is, and
on how relevant for later real world behaviour it is.

Except for this, I'd leave the judgement to the implementor;)

--
andreas

[Reini Urban via RT]

Fri Apr 04 11:51:31 2014: Request 85336 was acted upon.
Transaction: Correspondence added by rur...@x-ray.at

Queue: Inline
Subject: Fails often when tested in parallel
Broken in: 0.53
Severity: (no value)
Owner: Nobody

Requestors: AN...@cpan.org, KEN...@cpan.org

Status: open
Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=85336 >

On Thu Feb 20 03:51:10 2014, KENTNL wrote:
> You'll note quite a few fail reports on cpan testers from me.

Confirmed and repro even standalone.

The pure Inline/C tests fail on SMP systems with make (tested with gmake 4.0)
in t/25proto.t and t/08taint_1.p
On 5.6 and 5.14 perls (linux)

$@: make[1]: Entering directory `/home/rurban/Perl/Inline/C/_Inline_test/build/PROTO4_7cc8'
make[1]: *** write jobserver: Bad file descriptor. Stop.
make[1]: *** Waiting for unfinished jobs....
make[1]: *** write jobserver: Bad file descriptor. Stop.

I'll try to fix it at https://github.com/rurban/Inline

[Reini Urban via RT]

Fri Apr 04 13:56:12 2014: Request 85336 was acted upon.

Transaction: Correspondence added by rur...@x-ray.at
Queue: Inline
Subject: Fails often when tested in parallel
Broken in: 0.53
Severity: (no value)
Owner: Nobody
Requestors: AN...@cpan.org, KEN...@cpan.org
Status: open
Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=85336 >

On Fri Apr 04 11:51:31 2014, rur...@x-ray.at wrote:
> On Thu Feb 20 03:51:10 2014, KENTNL wrote:
> > You'll note quite a few fail reports on cpan testers from me.
>
> Confirmed and repro even standalone.
>
> The pure Inline/C tests fail on SMP systems with make (tested with
> gmake 4.0)
> in t/25proto.t and t/08taint_1.p
> On 5.6 and 5.14 perls (linux)
>
> $@: make[1]: Entering directory
> `/home/rurban/Perl/Inline/C/_Inline_test/build/PROTO4_7cc8'
> make[1]: *** write jobserver: Bad file descriptor. Stop.
> make[1]: *** Waiting for unfinished jobs....
> make[1]: *** write jobserver: Bad file descriptor. Stop.
>
> I'll try to fix it at https://github.com/rurban/Inline

The problem is obvious.
make inherits these ENV values from the master make test:
MAKEFLAGS = w --jobserver-fds=3,4 -j -- PREFIX=/usr/local LINKTYPE=dynamic LIBPERL_A=libperl.a
MAKELEVEL = 2
MAKEOVERRIDES = ${-*-command-variables-*-}

The jobserver-fds are wrong, need to be stripped.

All tests with make -j4 pass now. See https://github.com/rurban/Inline

[Reini Urban via RT]

Fri Apr 04 14:02:09 2014: Request 85336 was acted upon.

Transaction: Correspondence added by rur...@x-ray.at
Queue: Inline
Subject: Fails often when tested in parallel
Broken in: 0.53
Severity: (no value)
Owner: Nobody
Requestors: AN...@cpan.org, KEN...@cpan.org
Status: open
Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=85336 >

Patches for the next release attached

[Sisyphus via RT]

Sun Apr 06 03:55:40 2014: Request 85336 was acted upon.
Transaction: Correspondence added by SISYPHUS

Queue: Inline
Subject: Fails often when tested in parallel
Broken in: 0.53
Severity: (no value)
Owner: Nobody
Requestors: AN...@cpan.org, KEN...@cpan.org
Status: open
Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=85336 >

On Fri Apr 04 14:02:09 2014, rur...@x-ray.at wrote:
> Patches for the next release attached

Thanks Reini.
These patches have been applied (by hand) .... hope I got 'em right. Inline-0.54_01 (which contains these patches) has now been uploaded to CPAN.
All being well, it will be re-released as Inline-0.55 in a week or so.

For the Changes and C/Changes files I generally structure it so that Changes lists alterations to Inline and C/Changes alterations to Inline::C. (All this really achieves is to make it difficult for myself - especially in those cases where both packages are affected by the one set of changes.)

One will find that Changes and C/Changes have not been altered in strict accordance with the provided patches.

The only other alteration made to the provided patches was to rewrite the 2 occurrences of:

local $ENV{MAKEFLAGS} = $ENV{MAKEFLAGS} =~ s/(--jobserver-fds=[\d,]+)//;

as:

if($ENV{MAKEFLAGS}) {
local $ENV{MAKEFLAGS} = $ENV{MAKEFLAGS} =~ s/(--jobserver-fds=[\d,]+)//;
}

This was done to avoid 'uninitialized' warnings on systems where $ENV{MAKEFLAGS} was unset.

Cheers,
Rob

[Shawn Laffan via RT]

Sat Apr 12 18:46:34 2014: Request 85336 was acted upon.
Transaction: Correspondence added by SLAFFAN

Queue: Inline
Subject: Fails often when tested in parallel
Broken in: 0.53
Severity: (no value)
Owner: Nobody
Requestors: AN...@cpan.org, KEN...@cpan.org
Status: open
Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=85336 >

The local assignment to $ENV{MAKEFLAGS} won't apply outside the if block, will it?

This should, though:

local $ENV{MAKEFLAGS} = $ENV{MAKEFLAGS} =~ s/(--jobserver-fds=[\d,]+)//

if $ENV{MAKEFLAGS};


Or it could be localised before that condition if postfix conditions are not preferred (albeit postfix conditions are used elsewhere in the package):


local $ENV{MAKEFLAGS} = $ENV{MAKEFLAGS};
if($ENV{MAKEFLAGS}) {

$ENV{MAKEFLAGS} = $ENV{MAKEFLAGS} =~ s/(--jobserver-fds=[\d,]+)//;
}

Regards,
Shawn.

[sisy...@optusnet.com.au]

-----Original Message-----
From: Shawn Laffan via RT

> The local assignment to $ENV{MAKEFLAGS} won't apply outside the if block,
> will it?

Duh !!

> This should, though:
>
> local $ENV{MAKEFLAGS} = $ENV{MAKEFLAGS} =~ s/(--jobserver-fds=[\d,]+)//
> if $ENV{MAKEFLAGS};

Thanks - fixed in git.

Cheers,
Rob

[sisyphus1@optusnet.com.au via RT]

Sun Apr 13 07:17:59 2014: Request 85336 was acted upon.
Transaction: Correspondence added by sisy...@optusnet.com.au
Queue: Inline
Subject: Re: [rt.cpan.org #85336] Fails often when tested in parallel

Broken in: 0.53
Severity: (no value)
Owner: Nobody
Requestors: AN...@cpan.org, KEN...@cpan.org
Status: open
Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=85336 >

[Ed J via RT]

<URL: https://rt.cpan.org/Ticket/Display.html?id=85336 >

I'd be very interested to know whether the change proposed in https://github.com/mohawk2/inline-pm/commit/9fef7cfbd731249579deb3510d96a318115a0928 fixes this issue.

[Ed J via RT]

Thu Jun 12 04:37:14 2014: Request 85336 was acted upon.
Transaction: Correspondence added by ETJ

Queue: Inline
Subject: Fails often when tested in parallel
Broken in: 0.53
Severity: (no value)
Owner: Nobody
Requestors: AN...@cpan.org, KEN...@cpan.org
Status: open
Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=85336 >

[Sisyphus via RT]

Tue Jun 24 05:06:27 2014: Request 85336 was acted upon.
Transaction: Correspondence added by SISYPHUS

Queue: Inline
Subject: Fails often when tested in parallel
Broken in: 0.53
Severity: (no value)
Owner: Nobody
Requestors: AN...@cpan.org, KEN...@cpan.org
Status: open
Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=85336 >

Fixed as of 0.55_01

[Sisyphus via RT]

Tue Jul 01 04:27:36 2014: Request 85336 was acted upon.

Transaction: Correspondence added by SISYPHUS
Queue: Inline
Subject: Fails often when tested in parallel
Broken in: 0.53
Severity: (no value)
Owner: Nobody
Requestors: AN...@cpan.org, KEN...@cpan.org

Status: resolved
Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=85336 >


Re-opening because of the following patch that was applied to Inline-0.55 as part of this ticket:

####################################
Subject: [PATCH 4/5] MSWin32: disable BUILD_NOISY redirects on MSWin32 with
cmd.exe

also print exitcode with failed commands

diff --git C/C.pm C/C.pm
index f76e34b..21f7dfe 100644
--- C/C.pm
+++ C/C.pm
@@ -804,6 +804,7 @@ sub makefile_pl {
-f ($perl = $Config::Config{perlpath})
or ($perl = $^X)
or croak "Can't locate your perl binary";
+ $perl = qq{"$perl"} if $perl =~ m/\s/;
$o->system_call("$perl Makefile.PL", 'out.Makefile_PL');
$o->fix_make;
}
@@ -841,6 +842,7 @@ sub system_call {
defined $ENV{PERL_INLINE_BUILD_NOISY}
? $ENV{PERL_INLINE_BUILD_NOISY}
: $o->{CONFIG}{BUILD_NOISY};
+ $build_noisy = undef if $build_noisy and $^O eq 'MSWin32' and $Config::Config{sh} =~ /^cmd/;
if (not $build_noisy) {
$cmd = "$cmd > $output_file 2>&1";
}
@@ -861,11 +863,12 @@ sub build_error_message {
close OUTPUT;
}

+ my $errcode = $? >> 8;
return $output . <<END;

A problem was encountered while attempting to compile and install your Inline
$o->{API}{language} code. The command that failed was:
- $cmd
+ \"$cmd\" with error code $errcode

The build directory was:
$build_dir
####################################

I wondered at the time (and still wonder) what that was about - but I applied it anyway, as it didn't break any tests.

However, it does break BUILD_NOISY on Win32 - to the extent that the compiler/linker commands/warnings of a successful build are not seen.

simply removing the line:

$build_noisy = undef if $build_noisy and $^O eq 'MSWin32' and $Config::Config{sh} =~ /^cmd/;

from the patched (0.55) C.pm is sufficient to regain correct functioning of BUILD_NOISY on Windows.
However, doing that probably also destroys whatever it was that the patch was designed to fix.

This episode exposes a need for a test script that examines the output of a BUILD_NOISY build to detect that this output is present.
It would be hard to check that the entire output is as it should be, but we should at least be able to check for the presence of certain key elements like - eg that the output matches the string "perl", that it matches the (interpolated)"$Config{LD}" and that it matches the name of any Inline-C function whose compilation is expected to emit a warning.

I think this breakage of BUILD_NOISY needs to be fixed for the next stable release.

Attached is a try.pl that demonstrates the problem. It shows the expected output of the script, and the actual output I get on one of my Win32 perls using Inline-0.55.

Cheers,
Rob

[Ed J via RT]

Wed Jul 02 00:55:23 2014: Request 85336 was acted upon.
Transaction: Correspondence added by ETJ

Queue: Inline
Subject: Fails often when tested in parallel
Broken in: 0.53
Severity: (no value)
Owner: Nobody
Requestors: AN...@cpan.org, KEN...@cpan.org

Status: open
Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=85336 >

FYI, the code to show the correct operation of BUILD_NOISY can be one-linered like so:

perl -MInline=C,Config,BUILD_NOISY,1,FORCE_BUILD,1 -e "use Inline C => q[void inline_warner() { int *x = 2; }]"

My reading of the patch in question is that it turns off BUILD_NOISY when it's Windows and the shell is cmd. If BUILD_NOISY does the right thing with Win32 and CMD, let's undo that change?

[Sisyphus]

----- Original Message -----
From: "Ed J via RT" <bug-I...@rt.cpan.org>

> FYI, the code to show the correct operation of BUILD_NOISY can be
> one-linered like so:
>
> perl -MInline=C,Config,BUILD_NOISY,1,FORCE_BUILD,1 -e "use Inline C =>
> q[void inline_warner() { int *x = 2; }]"

Yes, for a test I'm thinking just have the test script run something like
that as a system command with output redirected to a file - and then check
that file (to an extent that allows us to be confident that BUILD_NOISY is
behaving as expected).

> My reading of the patch in question is that it turns off BUILD_NOISY when
> it's Windows and the shell is cmd.

That's about the extent of it. But I'm damned if I can think of any reason
that ought to be done.

> If BUILD_NOISY does the right thing with Win32 and CMD, let's undo that
> change?

BUILD_NOISY has always done the right thing for me on Win32 in the cmd.exe
shell - that is, until 0.55 ;-)
So yes - we definitely need to revert to pre-0.55 behaviour.

(This Windows laptop I'm using while I'm not at home doesn't have a git
client, and I can't be bothered installing one on it. I'll be back home
tomorrow night and will attend to this BUILD_NOISY issue then, if no-one
else has.)

Cheers,
Rob

[sisyphus1@optusnet.com.au via RT]

Wed Jul 02 05:52:27 2014: Request 85336 was acted upon.

Transaction: Correspondence added by sisy...@optusnet.com.au
Queue: Inline

Subject: Re: [rt.cpan.org #85336] Fails often when tested in parallel

Broken in: 0.53
Severity: (no value)
Owner: Nobody
Requestors: AN...@cpan.org, KEN...@cpan.org
Status: open
Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=85336 >

[Ed J via RT]

Wed Jul 02 18:01:36 2014: Request 85336 was acted upon.

Transaction: Correspondence added by ETJ
Queue: Inline
Subject: Fails often when tested in parallel
Broken in: 0.53
Severity: (no value)
Owner: Nobody
Requestors: AN...@cpan.org, KEN...@cpan.org
Status: open
Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=85336 >

commit 0bcdb0f7dfa065ff5bf68f2f3033ec7c549e38c3
Author: ...
Date: Wed Jul 2 22:43:44 2014 +0100

Undo change disabling BUILD_NOISY for Win32 when shell eq "cmd".

In new 0.55_03:

diff --git a/C/C.pm b/C/C.pm
index 0b8073e..cc2f4a0 100644
--- a/C/C.pm
+++ b/C/C.pm
@@ -852,7 +852,8 @@ sub system_call {

defined $ENV{PERL_INLINE_BUILD_NOISY}
? $ENV{PERL_INLINE_BUILD_NOISY}
: $o->{CONFIG}{BUILD_NOISY};

- $build_noisy = undef if $build_noisy and $^O eq 'MSWin32' and $Config::Conf
+ # test this functionality with:
+ #perl -MInline=C,Config,BUILD_NOISY,1,FORCE_BUILD,1 -e "use Inline C => q[v