Re: perl 5.18.0 status update

[Peter Rabbitson]

On Mon, Mar 11, 2013 at 09:46:55PM -0400, Ricardo Signes wrote:
>
> ### 5. add warnings for things that may be endangered
> EBCDIC - https://rt.perl.org/rt3/Ticket/Display.html?id=116915
> WinCE - https://rt.perl.org/rt3/Ticket/Display.html?id=115118
>
> The EBCDIC and WinCE warnings can, if needed, live entirely in perldelta.

I still do not entirely understand the rationale for this. There are 2
people actively working on the WinCE port currently. EBCDIC - *an actual
5.17.9 binary* that does IO properly was produced days ago.

Even if after an eventual blead merge these volunteers drop off the face
of the earth entirely, their legacy will be a 5.18 building and working
on both of these platforms.

But yes to maintain it - you will need extra volunteers later on. Which
brings us back to the problem - guess how many volunteers are you going
to hear from if *the official docs* list EBCDIC as "we will throw it
away unless there is a multi-year suppot commitment from someone". Such
text would be exceedingly motivating.

To rephrase - such text being included will imho serve no other purpose
but to *ensure* the discontinuation/death of the EBCDIC and WinCE ports.

My 2c

Cheers

[demerphq]

On 12 March 2013 02:46, Ricardo Signes <perl...@rjbs.manxome.org> wrote:
> ### 1. Hash Randomization
> JSON::PP - https://rt.perl.org/rt3/Ticket/Display.html?id=116825
> EU::MM - https://rt.perl.org/rt3/Ticket/Display.html?id=116857
> the rest - https://rt.perl.org/rt3/Ticket/Display.html?id=115908
>
> Yves has a final bout of changes to hash randomization to land. I haven't
> spoken with him in a few days, but my understanding is that the only thing
> blocking this landing is (a) some rebasing and squashing and (b) applying some
> patches to two upstream-CPAN dists that haven't made new releases.
>
> As far am I am concerned, this should land as soon as it's ready, so we can get
> it even more tested.

I will try to do a merge tonite. I have been very busy recently, sorry
for the lag.


> ### 2. split "\x20" does not work as documented
> problem - https://rt.perl.org/rt3/Ticket/Display.html?id=116086
> cure - https://rt.perl.org/rt3/Ticket/Display.html?id=116911
>
> I would love to see this fix in, because I think it makes things a lot more
> sensible. Unfortunately, the branch with the cure seems to have hung Tony's
> smokers a few times. Needs eyes on.

Id love a second set of eyes on this one. I thought i had fixed it properly.

Ill try to find time for this in the next few days.

Yves

--
perl -Mre=debug -e "/just|another|perl|hacker/"

[Johan Vromans]

Ricardo Signes <perl...@rjbs.manxome.org> writes:

> Also, guess what? It's time for us to start taking this code freeze
> seriously. Please slow down on changing blead unless it is a clear,
> simple bug fix.

I've released Getopt::Long 2.39 to CPAN.
It contains a number of small fixes (and doc fixes) that have been in
beta for much to long already.

I hope it's in time.

Thanks,
Johan

[Dave Mitchell]

On Mon, Mar 11, 2013 at 09:46:55PM -0400, Ricardo Signes wrote:

> ### 3. can't use @var in regexp, only $var
> https://rt.perl.org/rt3/Ticket/Display.html?id=115004
> ### 4. Regexp::Grammars broken since 5.17.1
> https://rt.perl.org/rt3/Ticket/Display.html?id=116823
> ### 10. __SUB__ should be undef in regex code block
> https://rt.perl.org/rt3/Ticket/Display.html?id=113928

These three are on my TODO list. I might add other items on your list of
things later.


--
In economics, the exam questions are the same every year.
They just change the answers.

[bulk88]

https://rt.perl.org/rt3/Ticket/Display.html?id=116601 is still waiting
for a response

[Nicholas Clark]

On Tue, Mar 12, 2013 at 01:28:06PM +0100, demerphq wrote:
> On 12 March 2013 02:46, Ricardo Signes <perl...@rjbs.manxome.org> wrote:
> > ### 1. Hash Randomization

> > As far am I am concerned, this should land as soon as it's ready, so we can get
> > it even more tested.
>
> I will try to do a merge tonite. I have been very busy recently, sorry
> for the lag.

Which branch is this (now?)

I'm also horribly laggy (And likely to be fairly distracted by the German
Perl Workshop for the next 3 days), but I'm curious which bits of my
feedback you liked.

It won't affect the basic testing of stuff on CPAN. And I think that yes,
sooner rather than later on getting an idea of what CPAN code "enjoys" this.

> > I would love to see this fix in, because I think it makes things a lot more
> > sensible. Unfortunately, the branch with the cure seems to have hung Tony's
> > smokers a few times. Needs eyes on.
>
> Id love a second set of eyes on this one. I thought i had fixed it properly.

I don't think that I'm the best person to look at this. It's not anything
I'm familiar with.

Nicholas Clark

[Karl Williamson]

Thanks for your persistence. I have created a branch smoke-me/khw-wince
that is smoking these patches with the intent to apply them before 5.18
code freeze.

My having a commit bit is fairly recent, so I remember well the
frustrations of submitting patches that sat for half a year or more
unlooked at and unapplied.

I generally apply patches only in areas I've expertise in, which isn't
many. These, however, affect only the win32 and more specifically wince
builds, which are currently broken, so it seems to me better to apply
even if I don't fully understand them.

[Steve Hay]

My apologies for forgetting about this. I had been watching the thread
titled "Re: status of WinCE Perl port in 2012 (part 2)" in early to
mid-February, in which Ricardo was seriously questioning whether WinCE
should indeed have a notice added in 5.18 that it would be dropped in
5.20 (as per RT#115118). It never reached a firm conclusion, and I
confess I kind of forgot about this and the other ticket (RT#83154)
which were outstanding, having mentally deferred looking at them until
some decision was reached...

[Karl Williamson]

On 03/12/2013 10:59 AM, Steve Hay wrote:
> and the other ticket (RT#83154)

I tried to look up that ticket, and it wasn't found

[bulk88]

It is a CPAN RT ticket. Not Perl RT.

[Konovalov, Vadim (Vadim)** CTR **]

> From: Karl Williamson [mailto:pub...@khwilliamson.com]

> On 03/12/2013 09:33 AM, bulk88 wrote:
> > Peter Rabbitson wrote:
> >> To rephrase - such text being included will imho serve no other
> >> purpose but to *ensure* the discontinuation/death of the EBCDIC and
> >> WinCE ports.
> >>

> > https://rt.perl.org/rt3/Ticket/Display.html?id=116601 is still waiting
> > for a response
> >
>
>
> Thanks for your persistence. I have created a branch
> smoke-me/khw-wince
> that is smoking these patches with the intent to apply them
> before 5.18
> code freeze.

/me too,
while not being able to add meaningfull help WRT wince, I still consider
the branch useful,

thanks for saving it for futher possibilities on being improved, which is
hopefully to follow.

Regards,
Vadim

[Nicholas Clark]

On Tue, Mar 12, 2013 at 03:42:32PM +0000, Nicholas Clark wrote:
> On Tue, Mar 12, 2013 at 01:28:06PM +0100, demerphq wrote:
> > On 12 March 2013 02:46, Ricardo Signes <perl...@rjbs.manxome.org> wrote:
> > > ### 1. Hash Randomization
>
> > > As far am I am concerned, this should land as soon as it's ready, so we can get
> > > it even more tested.
> >
> > I will try to do a merge tonite. I have been very busy recently, sorry
> > for the lag.
>
> Which branch is this (now?)
>
> I'm also horribly laggy (And likely to be fairly distracted by the German
> Perl Workshop for the next 3 days), but I'm curious which bits of my
> feedback you liked.

Bother. Your private message suggests that these might not have got pushed
anywhere. My fault, I suspect.

Attached.

The key thing was that my assumption was that the most important part is to
store the random state for iteration. Iteration already allocates an hv_aux
structure to store iteration state. Hence put the new U32 member in hv_aux,
so that we don't increase the overhead of every hash (ie pretty much every
object) by 1 pointer.

Nicholas Clark

[h...@crypt.org]

Nicholas Clark <ni...@ccl4.org> wrote:
:+ if ((((PTRV)hv ^ (PTRV)oentry) >> (sizeof(*entry) / 3)) && 1) {

I have no idea what this is doing - the comment above is insufficient help.
But shifting right with a count based on a sizeof() without a lg() seems
unlikely to do anything desirable.

Hugo

[Karl Williamson]

I don't know what lg() is. Also, that '&& 1' seems useless at first glance

[demerphq]

On 12 March 2013 20:21, Nicholas Clark <ni...@ccl4.org> wrote:
> On Tue, Mar 12, 2013 at 03:42:32PM +0000, Nicholas Clark wrote:
>> On Tue, Mar 12, 2013 at 01:28:06PM +0100, demerphq wrote:
>> > On 12 March 2013 02:46, Ricardo Signes <perl...@rjbs.manxome.org> wrote:
>> > > ### 1. Hash Randomization
>>
>> > > As far am I am concerned, this should land as soon as it's ready, so we can get
>> > > it even more tested.
>> >
>> > I will try to do a merge tonite. I have been very busy recently, sorry
>> > for the lag.
>>
>> Which branch is this (now?)
>>
>> I'm also horribly laggy (And likely to be fairly distracted by the German
>> Perl Workshop for the next 3 days), but I'm curious which bits of my
>> feedback you liked.
>
> Bother. Your private message

Oh, that was an accident, sorry.

> suggests that these might not have got pushed
> anywhere. My fault, I suspect.
>
> Attached.
>
> The key thing was that my assumption was that the most important part is to
> store the random state for iteration. Iteration already allocates an hv_aux
> structure to store iteration state. Hence put the new U32 member in hv_aux,
> so that we don't increase the overhead of every hash (ie pretty much every
> object) by 1 pointer.

The argument that we should keep hashes small is compelling. On the
other hand the simplicity of the approach I took is nice, and comes
with a few benefits, such as insertion more or less *always* changing
iteration order (I found a few bugs that way), and the basic operation
being fast. But it doesn't really matter, all we need is a cheap
source of semi-random bits that come from a source outside of the
control and visibility of a user.

Anyway, ill look at merging your patches and tweak them a bit based on
feedback in this thread.

[demerphq]

The idea is to use the address of things as a source of "random
coinflips" for perturbing the bucket chain during hash insertion of
colliding keys. The intention is to make it harder to "see" a
collision by viewing the keys of a hash.

[h...@crypt.org]

Karl Williamson <pub...@khwilliamson.com> wrote:

:On 03/12/2013 01:07 PM, h...@crypt.org wrote:
:> Nicholas Clark <ni...@ccl4.org> wrote:
:> :+ if ((((PTRV)hv ^ (PTRV)oentry) >> (sizeof(*entry) / 3)) && 1) {
:>
:> I have no idea what this is doing - the comment above is insufficient help.
:> But shifting right with a count based on a sizeof() without a lg() seems
:> unlikely to do anything desirable.
:>
:> Hugo
:>
:
:I don't know what lg() is.

Sorry, it's "base 2 log". (I now see from [1] that "lb" is apparently
an ISO standard, but I've never seen that.)

:Also, that '&& 1' seems useless at first glance

True.

Hugo

[1] http://en.wikipedia.org/wiki/Binary_logarithm

[h...@crypt.org]

demerphq <deme...@gmail.com> wrote:

The intention I'd mostly got from the comment, it's the implementation
I'm confused about.

Hugo

[John Peacock]

On 03/11/2013 09:46 PM, Ricardo Signes wrote:
> Also, guess what? It's time for us to start taking this code freeze seriously.
> Please slow down on changing blead unless it is a clear, simple bug fix.

Any chance my recent patch to update blead to version-0.9902 can get
applied. It fixes a fairly serious leak (reported by sprout) in qv()
handling error conditions. I wouldn't cry if it got missed (since you
can always install the CPAN release), but it would be nice to not ship
known bad code...

Thanks

John

[Craig A. Berry]

Does any of this:

http://perl5.git.perl.org/perl.git/commitdiff/b2a8d771f2f5721aa711c6ecdb42fdc198bfd244

look familiar? :-)

[h...@crypt.org]

I wrote:

Thinking further on this last night, it's not lg() I was missing but
BITS_PER_BYTE. I think it's taking a pointer-sized quantity and shifting
out one eighth of its bits; I think further the logical-and '&& 1' was
intended to be a bitwise-and, so that it was probably intended to be
equivalent to something like:

* out there gets bigger. The bottom few bits of these pointers will tend
* to be zero, but ignoring 1/8 of the bits seems a reasonable heuristic.
*/
if (((PTRV)hv ^ (PTRV)oentry) & (1 << sizeof(PTRV))) {

Nick, is there a reason it wants to use C<sizeof(*entry) / 3> instead, or
sizeof some particular pointer?

Generally, I don't think it's wise to embed indirect struct details
either in the code as magic numbers or in the comments, except directly
around the struct definition - they are unlikely to be updated when
the definition changes. Better to define a POINTERS_PER_HE or somesuch
at the same place the struct is defined, if that's needed.

Hugo

[demerphq]

FWIW, I think this discussion deserves its own thread.

[Dr.Ruud]

On 2013-03-12 21:51, demerphq wrote:

> all we need is a cheap
> source of semi-random bits that come from a source outside of the
> control and visibility of a user.

It should probably also be different per interpreter.

Maybe something similar to md5( `hostname` . ".$$." . time ).

--
Ruud

[John Peacock]

On 03/12/2013 10:37 PM, Craig A. Berry wrote:
> Does any of this:
>
> http://perl5.git.perl.org/perl.git/commitdiff/b2a8d771f2f5721aa711c6ecdb42fdc198bfd244
>
> look familiar? :-)

Doh! Sorry. When I didn't get a "Thanks, applied" reply and all of the
churn about 5.16.3 et al was going on, I just assumed it got lost in the
noise. I'll try and keep my clone up to date more often...

John

[Nicholas Clark]

Yes, that's a bug. Thanks. And superceded by what Yves has now pushed.

All these bugs are there to be seen on the commit list (usually).

http://lists.perl.org/list/perl5-changes.html

But few people read it.

Nicholas Clark

[Nicholas Clark]

On Thu, Mar 14, 2013 at 07:06:00AM +0000, h...@crypt.org wrote:
> I wrote:
> :Nicholas Clark <ni...@ccl4.org> wrote:
> ::+ if ((((PTRV)hv ^ (PTRV)oentry) >> (sizeof(*entry) / 3)) && 1) {
> :
> :I have no idea what this is doing - the comment above is insufficient help.
> :But shifting right with a count based on a sizeof() without a lg() seems
> :unlikely to do anything desirable.
>
> Thinking further on this last night, it's not lg() I was missing but
> BITS_PER_BYTE. I think it's taking a pointer-sized quantity and shifting
> out one eighth of its bits; I think further the logical-and '&& 1' was
> intended to be a bitwise-and, so that it was probably intended to be
> equivalent to something like:
>
> * out there gets bigger. The bottom few bits of these pointers will tend
> * to be zero, but ignoring 1/8 of the bits seems a reasonable heuristic.
> */
> if (((PTRV)hv ^ (PTRV)oentry) & (1 << sizeof(PTRV))) {
>
> Nick, is there a reason it wants to use C<sizeof(*entry) / 3> instead, or
> sizeof some particular pointer?

Not sure if the question is now irrelevant, but the comment was meant to
explain this (and clearly failed). Or I got something wrong.

HE structures are 12 (LP32) or 24 bytes (LP64). So (if I have my mental
arithmetic correct) pointers to them will always have the bottom 2 (or 3)
bits 0, and the next two bits are are not independent, either 11 or 00.

So the bottom few bits are definitely predictable.

> Generally, I don't think it's wise to embed indirect struct details
> either in the code as magic numbers or in the comments, except directly
> around the struct definition - they are unlikely to be updated when
> the definition changes. Better to define a POINTERS_PER_HE or somesuch
> at the same place the struct is defined, if that's needed.

It probably was. But I think that Yves built a better solution.

Nicholas Clark

[Dave Mitchell]

On Tue, Mar 19, 2013 at 10:45:19PM -0400, Ricardo Signes wrote:
> ### 2. can't use @var in regexp, only $var
> https://rt.perl.org/rt3/Ticket/Display.html?id=115004
>

> I would love to see this fix in, because I think it makes things a lot more
> sensible. Unfortunately, the branch with the cure seems to have hung Tony's

> smokers a few times. Needs eyes on. Still. (Yves would love your help,
> anybody!)

I don't think that comment applies to that ticket. As far as I'm aware,
there isn't a branch with a cure for it. It's on my list of things to do.

> ### 3. Regexp::Grammars broken since 5.17.1
> https://rt.perl.org/rt3/Ticket/Display.html?id=116823

I'm in the middle of working on this one.

--
"You're so sadly neglected, and often ignored.
A poor second to Belgium, When going abroad."
-- Monty Python, "Finland"

[demerphq]

On 20 March 2013 12:24, Dave Mitchell <da...@iabyn.com> wrote:
> On Tue, Mar 19, 2013 at 10:45:19PM -0400, Ricardo Signes wrote:
>> ### 2. can't use @var in regexp, only $var
>> https://rt.perl.org/rt3/Ticket/Display.html?id=115004
>>
>> I would love to see this fix in, because I think it makes things a lot more
>> sensible. Unfortunately, the branch with the cure seems to have hung Tony's
>> smokers a few times. Needs eyes on. Still. (Yves would love your help,
>> anybody!)

It applies to ticket #1. I started looking into it the other night but
other priorities (sleep) intervened.

[Brian Fraser]

On Tue, Mar 19, 2013 at 11:45 PM, Ricardo Signes
<perl...@rjbs.manxome.org> wrote:
> * Ricardo Signes <perl...@rjbs.manxome.org> [2013-03-11T21:46:55]
>> We're chugging along toward Perl 5.18.0. There are still ten blockers.
>
> Remember when I said that? Still true. Let's review.
>
> (Yes, expect me to do this weekly.)
>
> Here are blockers, which I list in something like a rough priority order:
>
> ### 1. split "\x20" does not work as documented
> problem - https://rt.perl.org/rt3/Ticket/Display.html?id=116086
> cure - https://rt.perl.org/rt3/Ticket/Display.html?id=116911

>
> ### 2. can't use @var in regexp, only $var
> https://rt.perl.org/rt3/Ticket/Display.html?id=115004
>
> I would love to see this fix in, because I think it makes things a lot more
> sensible. Unfortunately, the branch with the cure seems to have hung Tony's
> smokers a few times. Needs eyes on. Still. (Yves would love your help,
> anybody!)
>

> ### 3. Regexp::Grammars broken since 5.17.1
> https://rt.perl.org/rt3/Ticket/Display.html?id=116823
>

> ### 4. add warnings for things that may be endangered
>

Is there a list of these somewhere?

[Nicholas Clark]

On Wed, Mar 20, 2013 at 11:45:18AM -0400, Ricardo Signes wrote:
> * Brian Fraser <fras...@gmail.com> [2013-03-20T11:38:05]

> > Is there a list of these somewhere?
>

> https://rt.perl.org/rt3/Public/Search/Simple.html?Query=Queue%20=%20'perl5'%20AND%20(Status%20=%20'open'%20OR%20Status%20=%20'new'%20OR%20Status%20=%20'stalled'%29%20AND%20DependedOnBy%20=%20114502

Having @*, &*, ** and %* deprecated isn't in that list, but I think that you
agreed that it would be good to have, as getting rid of those (and $*) frees
up syntax for something better. (I don't see it as a *blocker*)

I've just got it working and pushed it to smoke-me/nicholas/doublestarungood

I forget the RT ticket for this, but it has one.

Nicholas Clark

[Brad Gilbert]

https://rt.perl.org/rt3/Public/Bug/Display.html?id=116943

[Nicholas Clark]

On Wed, Mar 20, 2013 at 11:22:56PM -0400, Ricardo Signes wrote:
> * Nicholas Clark <ni...@ccl4.org> [2013-03-20T11:51:41]

> > I've just got it working and pushed it to smoke-me/nicholas/doublestarungood
>

> As we discussed on IRC, I agree that this should go in.

It's now merged in.

Nicholas Clark

[Nicholas Clark]

Thanks.

I've now marked it as resolved.

Nicholas Clark

[Salvador Fandino]

On 03/12/2013 02:46 AM, Ricardo Signes wrote:
>
> We're chugging along toward Perl 5.18.0. There are still ten blockers.
>

> Also, guess what? It's time for us to start taking this code freeze seriously.

> Please slow down on changing blead unless it is a clear, simple bug fix. That
> doesn't mean stop working on cool stuff... just keep it in another branch for
> now. The next task is to get blead smoothed out and released.
>
> Perl 5.17.10 is due on March 21st. (That's a day later than the usual 20th of
> the month. It is not a sigifnicant or interesting change, just a matter of
> convenience for Max.) After that, I'd like to see the rate of change slow
> down.

>
> Here are blockers, which I list in something like a rough priority order:

Could you also consider #116082: print builtin is broken

I am not completely sure it has been solved in trunk, and IMO it is a
critical bug.

[Karl Williamson]

On 03/12/2013 02:03 AM, Peter Rabbitson wrote:
> On Mon, Mar 11, 2013 at 09:46:55PM -0400, Ricardo Signes wrote:
>>
>> ### 5. add warnings for things that may be endangered
>> EBCDIC - https://rt.perl.org/rt3/Ticket/Display.html?id=116915
>> WinCE - https://rt.perl.org/rt3/Ticket/Display.html?id=115118
>>
>> The EBCDIC and WinCE warnings can, if needed, live entirely in perldelta.
>
> I still do not entirely understand the rationale for this. There are 2
> people actively working on the WinCE port currently. EBCDIC - *an actual
> 5.17.9 binary* that does IO properly was produced days ago.
>
> Even if after an eventual blead merge these volunteers drop off the face
> of the earth entirely, their legacy will be a 5.18 building and working
> on both of these platforms.
>
> But yes to maintain it - you will need extra volunteers later on. Which
> brings us back to the problem - guess how many volunteers are you going
> to hear from if *the official docs* list EBCDIC as "we will throw it
> away unless there is a multi-year suppot commitment from someone". Such
> text would be exceedingly motivating.

I think you meant de-motivating

>
> To rephrase - such text being included will imho serve no other purpose
> but to *ensure* the discontinuation/death of the EBCDIC and WinCE ports.
>

> My 2c
>
> Cheers
>


I essentially agree with this. And even if we are wrong, I don't
believe the EBCDIC ticket should be a 5.18 blocker. The reason is that
no one will be able to compile 5.18 on EBCDIC, as we have discovered. A
bunch of patches are required to get it to compile; these aren't going
into 5.18; ergo no warning about using it is necessary.

[Karl Williamson]

Re-reading the thread on this bug, I don't see a way to reproduce it
now. Without an easy way to reproduce a bug, no one is going to bother
to investigate it further. If you think it is critical, you'll need to
post a test case that people can try, since the case you posted no
longer exhibits the problem.