[gentoo-dev] GCC USE flag changes

[Ryan Hill]

Since people like to start whinging threads every time we have to change flags
on gcc this is a friendly notice of some upcoming changes.

I'm adding an "lto" flag to work around some issues we have with stabilization
of 4.6 on alpha (though the flag will be added to all versions).

I'm also going to rename the "test" flag to "regression-test" or something
similar to get it out of FEATURES="test" control. The testsuite is a huge
time-suck and only useful to developers IMO (always expected to fail and
primarily meant to be used to check for regressions between patchsets). I'm a
big supporter of FEATURES="test" by default and I think this is a small step
towards that.

I'll make these changes at the same time as 4.7.3 is bumped to minimize
rebuilding but of course stable versions will be affected as well. That's just
waiting for me to go through the open bugs so probably end of weekish.

I'd also like to drop graphite support for versions 4.4 and 4.5. We use a
patch to dlopen graphite libs but older versions were written for ppl-0.10/0.11
(0.12 is the current version) so I doubt they still work properly (missing
symbols and whatnot). Some archs still have 4.5 as the latest stable but
graphite isn't a "supported" feature and the number of users on those archs is
likely tiny.

Any thoughts?


--
gcc-porting
toolchain, wxwidgets
@ gentoo.org

[Patrick Lauer]

On 05/01/2013 11:25 AM, Ryan Hill wrote:
> Since people like to start whinging threads every time we have to change flags
> on gcc this is a friendly notice of some upcoming changes.
>

[snip lots of good ideas]
>
> Any thoughts?

I'm in favour of unleashing 4.8 in ~arch soon - I've been building
"everything" with it for a while, I've reached >50% coverage of all
packages on amd64, and the amount of new bugs is about the same as still
open bugs for 4.7.

All in all it's a very benign release that (imo) doesn't need to be held
back.

[Ryan Hill]

Yeah it's been amazingly quiet. The loop transformation improvements may
only bite us at runtime however (another vote for FEATURES=test by default and
fix your testsuites people!). In any case I'll be waiting until 4.8.1 until
unmasking it, if only for superstitious reasons.

[Rick "Zero_Chaos" Farina]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 04/30/2013 11:49 PM, Ryan Hill wrote:
> On Wed, 01 May 2013 11:33:00 +0800
> Patrick Lauer <pat...@gentoo.org> wrote:
>
>> On 05/01/2013 11:25 AM, Ryan Hill wrote:
>>> Since people like to start whinging threads every time we have to change
>>> flags on gcc this is a friendly notice of some upcoming changes.
>>>
>> [snip lots of good ideas]
>>>
>>> Any thoughts?
>>
>> I'm in favour of unleashing 4.8 in ~arch soon - I've been building
>> "everything" with it for a while, I've reached >50% coverage of all
>> packages on amd64, and the amount of new bugs is about the same as still
>> open bugs for 4.7.
>>
>> All in all it's a very benign release that (imo) doesn't need to be held
>> back.
>
> Yeah it's been amazingly quiet. The loop transformation improvements may
> only bite us at runtime however (another vote for FEATURES=test by default and

I don't mean to start a flamewar here but the test suite situation is so
bad with circular deps (I'm looking at you ruby herd) and random
failures that I only enable tests for my own packages. Sadly it is so
bad that we have a FEATURES=test-fail-continue.... I can't really say
anything negative, that fact really says it all...

> fix your testsuites people!).

Please, pretty please.

- -Zero

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBAgAGBQJRgKggAAoJEKXdFCfdEflKTYoP/Rm80i7Fca3Ao+H0WQriTrgS
1tYHMVQ7IN05tnq8f/JKSItE+23VPFH4lyOVJRNW+SBghy0wEA5/kl2GSTY4GrxE
lTMmnUUPWy7/wGvR+oYc0E2+XJdWwG4smZBenlARM912Oq+i24D00eicVPchGL9g
PNFB8mSTPO3LUkSmra3mzKbnkezpNiJx77bd/D7EK9ojXKq4rA/lt85G8DlgWuav
LbfWtPCjHSZ+4faxSsae/aF31ptxX0qFdLCS8RVeT1y4fx5qehvi5rSSMCV84Tl4
SLJv0p+52SrGXQgcaWxW/mhgAcfVmMx7TTZwnqYp1+T1Ixl5aQ6nG1l3hTC0bm5Y
vqdJBur4NuqWT/BStSkD5vaasPug+9f8NhuIFla6U2MnGEVHOUAb4zr+xt++C7Nr
jO0uMNCP21pzs8mdTvrUT9xiwH62/Wj9Q1/u52yCNpn7gRgT4v6Cq8AeiPcvLQSu
A+X9b86gHe3ArAUkCJZ4OvPbAjFfU/aM+HV9BG5e/DHvjtYx4c2Z1hRiGB7IOpHq
VhR46owcQ+8Dn4LXsTTBbVzLFTsim8AFbyErBw0vAi+W5lo8i49QxkDNqBz1xLES
6NtRl7GOcIU9ZSQSrxwAG/ZCC9YveJz/joghZJxpdgRCIZi6vJKgc8dLktVxSYPf
YlcuAI6zCAyFNa4LiO9K
=/kLa
-----END PGP SIGNATURE-----

[Diego Elio Pettenò]

On 01/05/2013 06:29, Rick "Zero_Chaos" Farina wrote:
> I don't mean to start a flamewar here but the test suite situation is so
> bad with circular deps (I'm looking at you ruby herd) and random
> failures that I only enable tests for my own packages. Sadly it is so
> bad that we have a FEATURES=test-fail-continue.... I can't really say
> anything negative, that fact really says it all...

You might not mean to but honestly, do you really think that we have
circular deps because we like them?

FFS I've been the biggest user of FEATURES=test, and the one who poured
in more time to get stuff to work, so please don't effing go around
blaming people without even having a clue about what's going on, you
just piss me off.

--
Diego Elio Pettenò — Flameeyes
flam...@flameeyes.eu — http://blog.flameeyes.eu/

[Ralph Sennhauser]

On Tue, 30 Apr 2013 21:25:40 -0600
Ryan Hill <dirt...@gentoo.org> wrote:

> I'm also going to rename the "test" flag to "regression-test" or
> something similar to get it out of FEATURES="test" control. The
> testsuite is a huge time-suck and only useful to developers IMO
> (always expected to fail and primarily meant to be used to check for
> regressions between patchsets). I'm a big supporter of
> FEATURES="test" by default and I think this is a small step towards
> that.

This step is so tiny that we wont ever reach the goal like this. Let's
start to properly classify test into categories, like for instance

- expected to be run (cheap, no silly deps)
- good thing if run (still reasonable wrt resources) (current src_test)
- if you are the maintainer or simply curious. (boost, jtreg and
friends)

... and improve on how to configure Portage whether to run tests of any
given category.

[Ralph Sennhauser]

On Wed, 01 May 2013 01:29:05 -0400
"Rick \"Zero_Chaos\" Farina" <zero...@gentoo.org> wrote:

> Sadly it is so
> bad that we have a FEATURES=test-fail-continue.... I can't really say
> anything negative, that fact really says it all...

My beef is not with the existence of this FEATURE but that it's enabled
in the dev profile. I was lucky once to not commit a broken testsuite.

["Paweł Hajdan, Jr."]

On 4/30/13 8:25 PM, Ryan Hill wrote:
> I'm also going to rename the "test" flag to "regression-test" or something
> similar to get it out of FEATURES="test" control. The testsuite is a huge
> time-suck and only useful to developers IMO (always expected to fail and
> primarily meant to be used to check for regressions between patchsets). I'm a
> big supporter of FEATURES="test" by default and I think this is a small step
> towards that.

+1

> I'll make these changes at the same time as 4.7.3 is bumped to minimize
> rebuilding but of course stable versions will be affected as well. That's just
> waiting for me to go through the open bugs so probably end of weekish.

Sounds good.

> I'd also like to drop graphite support for versions 4.4 and 4.5. We use a
> patch to dlopen graphite libs but older versions were written for ppl-0.10/0.11
> (0.12 is the current version) so I doubt they still work properly (missing
> symbols and whatnot). Some archs still have 4.5 as the latest stable but
> graphite isn't a "supported" feature and the number of users on those archs is
> likely tiny.

Yeah, +1 as well.

Paweł

[Ryan Hill]

On Wed, 01 May 2013 08:00:29 +0100
Diego Elio Pettenò <flam...@flameeyes.eu> wrote:

> On 01/05/2013 06:29, Rick "Zero_Chaos" Farina wrote:
> > I don't mean to start a flamewar here but the test suite situation is so
> > bad with circular deps (I'm looking at you ruby herd) and random
> > failures that I only enable tests for my own packages. Sadly it is so
> > bad that we have a FEATURES=test-fail-continue.... I can't really say
> > anything negative, that fact really says it all...
>
> You might not mean to but honestly, do you really think that we have
> circular deps because we like them?
>
> FFS I've been the biggest user of FEATURES=test, and the one who poured
> in more time to get stuff to work, so please don't effing go around
> blaming people without even having a clue about what's going on, you
> just piss me off.

He wasn't singling you out, just naming a part of the tree that has a lot of
circular deps. He could've said python just as easy.

Relax, have a Guinness.

[Ryan Hill]

On Wed, 1 May 2013 10:14:02 +0200
Ralph Sennhauser <se...@gentoo.org> wrote:

> On Tue, 30 Apr 2013 21:25:40 -0600
> Ryan Hill <dirt...@gentoo.org> wrote:
>
> > I'm also going to rename the "test" flag to "regression-test" or
> > something similar to get it out of FEATURES="test" control. The
> > testsuite is a huge time-suck and only useful to developers IMO
> > (always expected to fail and primarily meant to be used to check for
> > regressions between patchsets). I'm a big supporter of
> > FEATURES="test" by default and I think this is a small step towards
> > that.
>
> This step is so tiny that we wont ever reach the goal like this.

I was hoping it would set a precedent and then people would start thinking of
splitting up test into categories, maybe even start a thread about it ;).

> Let's
> start to properly classify test into categories, like for instance
>
> - expected to be run (cheap, no silly deps)
> - good thing if run (still reasonable wrt resources) (current src_test)
> - if you are the maintainer or simply curious. (boost, jtreg and
> friends)

Something like "dev-test" or "qa-test"? I can think of a couple packages..

> ... and improve on how to configure Portage whether to run tests of any
> given category.

Yeah I'd love to be able to do something like emerge TESTS="dev qa
system -extradeps -expensive" @world.

[Kent Fredric]

2 other classes of tests you may want to consider :

- network/internet accessibility required tests

- markers for tests that are known/expected to fail under many conditions and are not worth end-user-testing, but end-users can force-running any way if they really want to see the individual failures.

--
Kent

perl -e  "print substr( \"edrgmaM  SPA NOcomil.ic\\@tfrken\", \$_ * 3, 3 ) for ( 9,8,0,7,1,6,5,4,3,2 );"

http://kent-fredric.fox.geek.nz

[Matt Turner]

He may not have meant to, but he said ruby herd specifically... on
which Diego does a lot of work.

[Mike Frysinger]

On Wednesday 01 May 2013 21:24:07 Kent Fredric wrote:

please do not top post
-mike

[Kent Fredric]

My apologies, Gmail has forced upon us this new message composer, and it sucks, it actively discourages bottom posting, and I'm stuck with it.

I even complained to them about it during their transition/experimentation/beta period, and they made it adequately clear to me they don't give a shit about technical users, especially not technical users who participate in lengthy mailing lists :(

I do try, but basically, I have to mentally remember to work around its banal annoying changes for each message :(

--
Kent

[Rick "Zero_Chaos" Farina]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 05/01/2013 03:00 AM, Diego Elio Pettenò wrote:
> On 01/05/2013 06:29, Rick "Zero_Chaos" Farina wrote:
>> I don't mean to start a flamewar here but the test suite situation is so
>> bad with circular deps (I'm looking at you ruby herd) and random
>> failures that I only enable tests for my own packages. Sadly it is so
>> bad that we have a FEATURES=test-fail-continue.... I can't really say
>> anything negative, that fact really says it all...
>
> You might not mean to but honestly, do you really think that we have
> circular deps because we like them?
>
> FFS I've been the biggest user of FEATURES=test, and the one who poured
> in more time to get stuff to work, so please don't effing go around
> blaming people without even having a clue about what's going on, you
> just piss me off.
>

I am sorry that I have offended you, but unfortunately this is what
happens when I try to enable tests:

http://dev.gentoo.org/~zerochaos/distfiles/ruby_test_deps.txt

As you can see, enabling this by default in any profile would be rather
crippling to gentoo as a whole. I am not blaming anyone, nor do I
understand what has caused the current situation. What I do understand
is that I am unable to run with FEATURES=test enabled by default despite
having the desire to do so. I have the hardware to run automated test,
I have the desire to run automated tests, the only thing I don't have is
the time to sort out hundreds of circular deps in ruby.

If it helps lower the tempers, I don't bitch about things I'm not
willing to help fix. If someone from the ruby team wants to discuss
this with me (on or off this list) I will be happy to listen to the
problem and try to find a sane solution. Likely this will require
changes to the dep structure or portage, but these subjects interest me
very much and I feel it is well worth my time to help with this.

You have two choices:

1.) Continue to curse me and we stay where we are
2.) accept the help offered

Your call.

- -Zero
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBAgAGBQJRgfJFAAoJEKXdFCfdEflKyYUP/R+GNFv9SC4NASSFAh8nSNBH
MUG3+5k5UpI1AtbSyt1VdnQztgK5b4x9RmSard0PptlOgB0g/Nf24b6qbaT/SYFR
DI8FmhQaZLmf4icw2/LjK6mO0r9gPfiB5Rm4LFmkHMJVDxd2HmkV4bMV2Y+zfoLZ
A0SwK2Y3jMGjTmL4Eg/j3ExRVYDQP2Uo1uF56XU6+5pcn+gI4gA+dCB3Q7RDEqtM
L8sds5WijA10Jd5q6QKSNnDQmv7jLioRqFuVupJ5QlieIdQCY0jaYEwlasTfGKKh
FlxmSLF3Uw08tLDgMWXpjrRsnyzup9V/zNwQiMD6SOukTvqoQPJ1p+/5qrur+OVD
/NjvdqXBk8kYjmhXVGdyOhVDgeEyknG/hqxgT/z7abNqIaYPufel8tV5+DQL28H+
2FLU8DorMfXBzpYqPW/g55gSalOytR3jpVGPEthTt91b8YbbvOgdWkdIl0c8qmeV
r7xrPhMQ6wZUX0hapcX3670sUYXeLjactYvJLj6WQyT+pIfnSTaI0QYjsDpnmOfy
0KPuw3MMjuSYdp0iIq8IxVRbUJ94sM9PqzoFVp8hXHFf80IWV7RaZBzABwzBY9fW
eKlWTylRM7rw08SddHaU4lSiOhN3y9AtPV0NQSnxq5klT5IH8MSTwgB4t3oo5M7h
xlkncCfm1vHt9kS3A5Qi
=SO5/
-----END PGP SIGNATURE-----

[Rick "Zero_Chaos" Farina]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Honestly, IMHO, I think breaking it down to more than "test" and "qa" is
excessive. I certainly wouldn't block anyone that wishes to do that
work, but I think we all realize that enabling tests (and especially
some of the really awesome QA tests Diego does in the tinderbox) are
really expensive time wise. The average user should never have this
enabled, and honestly, most devs shouldn't have it enabled either as
they simply won't have the hardware to run the tests every time (I know
I've commited things from my chromebook without excessive testing...).

I have a little cron script that builds all the packages I'm marked
maintainer of every night with FEATURES=test and would be easy enough to
adapt to looking at herd instead of maintainer.

I don't claim to be the expert on testing, but in case anyone finds
something like this useful... here:

#!/bin/sh

emerge --sync -q

#maintainer check (excluding madwifi)
MAINTAINER_OF="$(fgrep 'zero...@gentoo.org'
/usr/portage/*/*/metadata.xml | cut -d/ -f4-5 | grep -v madwifi)"

#build test
FEATURES="test" emerge --deep --oneshot -q ${MAINTAINER_OF}
if [ $? -ne 0 ]; then
echo "build failed, see above"
fi

#repoman check
for atom in ${MAINTAINER_OF}; do
cd /usr/portage/${atom}
repoman -qd full > /dev/null
if [ $? -ne 0 ]; then
repoman -qd full
fi
done

Thanks,
Zero


PS> I welcome suggestions for this script, new functionality is always
welcome

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBAgAGBQJRgfRKAAoJEKXdFCfdEflKJPYP/2qUMB5T3KZ1jT2dusdgg3KA
yuPAc6xL9VNg25pnVYl14JoS66z1hKpisSCCZeeGT7uqRDEkh+jseEaZIOhrektb
Zgn3WXmekmU5pc9wicUW4aYVzNyc9PBbw2NcBWq1HwgfMcwBoX87FxVECw0L71dO
hNOWQbJ1amGGoLXiMmAI/S4CpkZcfX3WZv0R2oosKK/Iu34gZeHGk2w5ui4tuY6g
sZmKawjXTFzbluIcaZlUPyajBjyZEXx0Vsp3uYUY1lStUNG45q5jaz3/83V+NkF2
Lu0FHrXrSW97f19k2HkcTdl4icF48k6bmIbw22xZ1lII+rnV/3uqsok/UayRyF1c
w6EIFnS37M/MhDknJ9R65P18v/SWAP2MLnfhcIqFDWs6jXZqq8vpBOfW+waEWsKK
8qflxpU08joUBsUSBcz+Y8s6JBxbWVdlY+f+jLsP6kPUH5otHKym7vB2P0nqCGQQ
SayT8fpK5izX4UraNhpuOiDV8nH7cvD1h3t6MRlSzRshj7UWyw4nYIkhqFZJhFzW
g6hfmLUGqHeHxRKMyImVLY4+PST820s+5mTTG57YuXmzD/nYU9N2Yor61FtgLR26
975mApTT4cunHdiUudyGbeaE0j0f/wOe1CAivzTCmDi+tM4dYR78XscPRZCsrk2+
JeNw6XtA5A6jvVk12jYN
=+Zzv
-----END PGP SIGNATURE-----

[Hans de Graaff]

On Thu, 2013-05-02 at 00:57 -0400, Rick "Zero_Chaos" Farina wrote:
> On 05/01/2013 03:00 AM, Diego Elio Pettenò wrote:
> > On 01/05/2013 06:29, Rick "Zero_Chaos" Farina wrote:
> >> I don't mean to start a flamewar here but the test suite situation is so
> >> bad with circular deps (I'm looking at you ruby herd) and random
> >> failures that I only enable tests for my own packages. Sadly it is so
> >> bad that we have a FEATURES=test-fail-continue.... I can't really say
> >> anything negative, that fact really says it all...
> >
> > You might not mean to but honestly, do you really think that we have
> > circular deps because we like them?
> >
> > FFS I've been the biggest user of FEATURES=test, and the one who poured
> > in more time to get stuff to work, so please don't effing go around
> > blaming people without even having a clue about what's going on, you
> > just piss me off.
> >
> I am sorry that I have offended you, but unfortunately this is what
> happens when I try to enable tests:
>
> http://dev.gentoo.org/~zerochaos/distfiles/ruby_test_deps.txt

Note that these are not circular dependencies.

dev-ruby/mocha should really be slotted now. This has been creeping up
on us with versions initially looking to be upward compatible. I've put
it on my todo list. I'll have a look at ruby_parser and sexp-processor
as well.

> You have two choices:
>
> 1.) Continue to curse me and we stay where we are
> 2.) accept the help offered

Fixing https://bugs.gentoo.org/show_bug.cgi?id=175808 would be useful
(though it wouldn't help with this particular problem).

Hans

[Ralph Sennhauser]

On Wed, 1 May 2013 19:18:52 -0600

Ryan Hill <dirt...@gentoo.org> wrote:

> On Wed, 1 May 2013 10:14:02 +0200
> Ralph Sennhauser <se...@gentoo.org> wrote:
>
> > On Tue, 30 Apr 2013 21:25:40 -0600
> > Ryan Hill <dirt...@gentoo.org> wrote:
> >
> > > I'm also going to rename the "test" flag to "regression-test" or
> > > something similar to get it out of FEATURES="test" control. The
> > > testsuite is a huge time-suck and only useful to developers IMO
> > > (always expected to fail and primarily meant to be used to check
> > > for regressions between patchsets). I'm a big supporter of
> > > FEATURES="test" by default and I think this is a small step
> > > towards that.
> >
> > This step is so tiny that we wont ever reach the goal like this.
>
> I was hoping it would set a precedent and then people would start
> thinking of splitting up test into categories, maybe even start a
> thread about it ;).

Hehe, yes, you forced me to speak up with trying to set a precedent I
think will get in the way of solving it in a more complete way.

Though for that we have to agree on

- split is desirable
- which categories and how to classify tests
- how to implement the splitting (EAPI support?)

[...]

> > ... and improve on how to configure Portage whether to run tests of
> > any given category.
>
> Yeah I'd love to be able to do something like emerge TESTS="dev qa
> system -extradeps -expensive" @world.
>
>

I was thinking about a /etc/portage/package.test that works more like
package.use. So most users will want to have something like:

# package.test
*/* cheap

Others might use:

# package.test
*/* cheap normal
*/*::sunrise expensive

my-pkg/foo *

# broken test suite, bug XXX
=dev-foo/bar-1.1 -*

This would also be pretty similar to what your regression-tests useflag
for gcc would have been. Even though Portage allows you to configure
FEATURES=test on a per package basis since a couple years it doesn't
seem to have become a common practice. While Portages mechanism is
powerful it might be just to complex or tedious for the average user.


As for your example command line 'emerge TESTS="dev qa system
-extradeps -expensive" @world', could you elaborate on what the
categories dev qa system are about?

[Ryan Hill]

On Tue, 30 Apr 2013 21:25:40 -0600
Ryan Hill <dirt...@gentoo.org> wrote:

> [...]

This is pushed out now. I also ended up dropping LTO support for 4.5. If
you're using LTO as shipped in 4.5 you are braver than anyone deserves to be.