[PROPOSAL] To retire autoconf for 4.1

[Andrew Bartlett]

G'day,

Michael Adam tells me he raised this in brief at SambaXP, but I wanted
to bring it up here:

As we consider dates for Samba 4.1, I think it's finally time to have
a discussion about removing autoconf. I know this is controversial in some circles, so
I have avoided starting this discussion for some time.

Delaying this proposal has been useful, because in the almost 6 months since we
released Samba 4.0, I've seen vanishingly few complaints about our waf
build system. Indeed, the users who seem to admire it most are those on
the old Solaris systems, because they say it handles the use of
non-system OpenLDAP libs better, and provides a modern kerberos
bundled-in!

I've personally worked with a Solaris 8 site with the (sensible, for a
system of that age) requirement that it not be directly connected to the
internet, and it is from working with that user that the
build_with_python.sh script was improved.

I'm not going to say that waf is perfect - it certainly isn't, but for
Samba 4.1 and beyond, we should focus our energies on a single build
system, and making it better, rather than continuing to build this large
and complex code base with two independent build systems that we have to
keep in sync.

The autoconf build system has supported us for a long time, but just as
it is time that we retire SWAT, Samba 4.1 is the appropriate time to
retire autoconf, and move to a single way to build a unified Samba
project.

That we are making a Samba 4.1 release without major feature upgrades
makes this a particularly good opportunity, as users who might be
adversely impacted (and not have noticed any of our previous list
announcements, or BUILD_SYSTEMS.txt) will almost certainly be able to
use Samba 4.0 as a fall-back.

I'm sure many of you will have comments, and I fully acknowledge that
the way waf was brought to Samba was not a pleasant process for any of
us. I hope that despite that pain we can still make a decision here.

Some background on the long slog to ensure config.h files match can be found in:
https://bugzilla.samba.org/show_bug.cgi?id=8969

Thanks,

Andrew Bartlett
--
Andrew Bartlett http://samba.org/~abartlet/
Authentication Developer, Samba Team http://samba.org

[Michael Adam]

Hi Andrew,

I agree that 4.1 would be a good opportunity to remove the
support for the autoconf build, as I am personally confident
enough with the waf build as the major build system.

But I think before taking the step, we should verify
the current state of the config results - the bug #8969
you mentioned has not been updated in a long time.

And maybe we can get some voices here of people using
Samba file servers on various Unixes and of the distributors
(of Linux, but also BSD flavours maybe).

Cheers - Michael

[Volker Lendecke]

On Tue, May 21, 2013 at 11:31:20AM +0200, Michael Adam wrote:
> Hi Andrew,
>
> I agree that 4.1 would be a good opportunity to remove the
> support for the autoconf build, as I am personally confident
> enough with the waf build as the major build system.
>
> But I think before taking the step, we should verify
> the current state of the config results - the bug #8969
> you mentioned has not been updated in a long time.
>
> And maybe we can get some voices here of people using
> Samba file servers on various Unixes and of the distributors
> (of Linux, but also BSD flavours maybe).

Can we make it easier to compile just one binary, for
example smbtorture or so? I am right now messing with talloc
for example, and if I change talloc.h, it will recompile
almost everything before it even thinks about linking
smbstatus. Also, the startup time even on a beefy laptop is
several seconds before it even starts *something*. This
makes coding central components much more tedious than with
the autoconf build.

I know that from an ivory tower perspective changing
talloc.h requires rebuilding everything, but in my real
world I know what I'm doing and I don't need 100% precision.
What I need is quick turnaround with a possible fallback to
git clean -dxf.

Thanks,

Volker

--
SerNet GmbH, Bahnhofsallee 1b, 37081 Göttingen
phone: +49-551-370000-0, fax: +49-551-370000-9
AG Göttingen, HRB 2816, GF: Dr. Johannes Loxen
http://www.sernet.de, mailto:kon...@sernet.de

[Jelmer Vernooij]

IIRC

./buildtools/bin/waf build --targets=smbtorture

should rebuild just smbtorture.

Cheers,

Jelmer

[Volker Lendecke]

Thanks!

That does save linking of all non-involved libs and
binaries, but it will still compile everything first. At
least it did, when I last tried.

[Volker Lendecke]

On Tue, May 21, 2013 at 12:28:53PM +0200, Jelmer Vernooij wrote:

> IIRC
>
> ./buildtools/bin/waf build --targets=smbtorture
>
> should rebuild just smbtorture.

Ah, another one: When coding larger changes, in autoconf I
like the ability to just compile one .o file. For example, I
frequently do a "make smbd/open.o" for example if I mess
with that. So far I have not found a way to do that in waf
quickly. Any hint for that?

[Andrew Bartlett]

On Tue, 2013-05-21 at 12:51 +0200, Volker Lendecke wrote:
> On Tue, May 21, 2013 at 12:28:53PM +0200, Jelmer Vernooij wrote:
> > IIRC
> >
> > ./buildtools/bin/waf build --targets=smbtorture
> >
> > should rebuild just smbtorture.
>
> Ah, another one: When coding larger changes, in autoconf I
> like the ability to just compile one .o file. For example, I
> frequently do a "make smbd/open.o" for example if I mess
> with that. So far I have not found a way to do that in waf
> quickly. Any hint for that?

You could determine the command waf is using (make V=1 should do it),
and manually invoke gcc yourself.

It's not pretty, but it should allow you to handle this unusual case.

[Michael Adam]

On 2013-05-21 at 21:10 +1000, Andrew Bartlett wrote:
> On Tue, 2013-05-21 at 12:51 +0200, Volker Lendecke wrote:
> > On Tue, May 21, 2013 at 12:28:53PM +0200, Jelmer Vernooij wrote:
> > > IIRC
> > >
> > > ./buildtools/bin/waf build --targets=smbtorture
> > >
> > > should rebuild just smbtorture.
> >
> > Ah, another one: When coding larger changes, in autoconf I
> > like the ability to just compile one .o file. For example, I
> > frequently do a "make smbd/open.o" for example if I mess
> > with that. So far I have not found a way to do that in waf
> > quickly. Any hint for that?
>
> You could determine the command waf is using (make V=1 should do it),
> and manually invoke gcc yourself.
>
> It's not pretty, but it should allow you to handle this unusual case.

Well, it is not so very unusual for a developer working on a
certain set of code files to want to quickly check whether the
changes compile.

If I remember this correctly, the main difficulty in providing
something like this for the waf build lies in the fact that our
object files are generated as basename_<number>.o when the
source file is basename.c, and number is generated by waf.

So let me record a couple of wishes that target at reducing the
turnaround times for builds in a development process:

- Can we create targets for specific objects, to just
compile a single C file?

- When building a certain binary target, can we change
waf's mode to only compile the needed code files?
(linking is already reduced, as Volker confirmed)

- Is there a way to reduce the startup time waf needs
befor it (apparently) does anything?

It would be great if the poeple most knowledgeable about waf
could comment on the above. These are really things that make
development with autoconf faster. Let's not forget that on the
other side we have much advanced dependency tracking in waf that
greatly reduces the need to do make clean or git clean.

Cheers - Michael

[Andrew Bartlett]

On Tue, 2013-05-21 at 14:16 +0200, Michael Adam wrote:
> On 2013-05-21 at 21:10 +1000, Andrew Bartlett wrote:
> > On Tue, 2013-05-21 at 12:51 +0200, Volker Lendecke wrote:
> > > On Tue, May 21, 2013 at 12:28:53PM +0200, Jelmer Vernooij wrote:
> > > > IIRC
> > > >
> > > > ./buildtools/bin/waf build --targets=smbtorture
> > > >
> > > > should rebuild just smbtorture.
> > >
> > > Ah, another one: When coding larger changes, in autoconf I
> > > like the ability to just compile one .o file. For example, I
> > > frequently do a "make smbd/open.o" for example if I mess
> > > with that. So far I have not found a way to do that in waf
> > > quickly. Any hint for that?
> >
> > You could determine the command waf is using (make V=1 should do it),
> > and manually invoke gcc yourself.
> >
> > It's not pretty, but it should allow you to handle this unusual case.
>
> Well, it is not so very unusual for a developer working on a
> certain set of code files to want to quickly check whether the
> changes compile.

If you are only changing those files, and have done a full compile in
the recent past, then a simple 'make' will do just that. Yes, you may
not want to wait for it to link, but you can reduce that if you must by
specifying a target, but once it is at that point you know that it does

compile.

> If I remember this correctly, the main difficulty in providing
> something like this for the waf build lies in the fact that our
> object files are generated as basename_<number>.o when the
> source file is basename.c, and number is generated by waf.
>
> So let me record a couple of wishes that target at reducing the
> turnaround times for builds in a development process:
>
> - Can we create targets for specific objects, to just
> compile a single C file?
>
> - When building a certain binary target, can we change
> waf's mode to only compile the needed code files?
> (linking is already reduced, as Volker confirmed)

This is caused by the waf stages. The difficultly with removing these
is that we then must ensure our dependencies are absolutely perfect, as
otherwise the builds may not be as reliable.

Using ccache (with CC="ccache gcc" or the default installation on
Fedora, which puts it in the path) is what I do to keep the compile
delay under control.

> - Is there a way to reduce the startup time waf needs
> befor it (apparently) does anything?
>
> It would be great if the poeple most knowledgeable about waf
> could comment on the above. These are really things that make
> development with autoconf faster. Let's not forget that on the
> other side we have much advanced dependency tracking in waf that
> greatly reduces the need to do make clean or git clean.

This dependency calculation is essentially where the time goes. A great
deal of caching is done to keep this as short as possible, but the
calculations must be done to ensure a correct build, every time.

waf will never be as simple, nor as fast as a system without full
dependency tracking, but a lot of effort was put in to make it as fast
as it can be, within the constraint of always building correctly.

[Volker Lendecke]

On Tue, May 21, 2013 at 10:40:31PM +1000, Andrew Bartlett wrote:
> If you are only changing those files, and have done a full compile in
> the recent past, then a simple 'make' will do just that. Yes, you may
> not want to wait for it to link, but you can reduce that if you must by
> specifying a target, but once it is at that point you know that it does
> compile.

Well, there is no such thing in waf as a quick compile. The
startup time alone is much longer than in autoconf. Also,
figuring out the right target name is something I can never
really figure out instantaneously.

> This is caused by the waf stages. The difficultly with removing these
> is that we then must ensure our dependencies are absolutely perfect, as
> otherwise the builds may not be as reliable.
>
> Using ccache (with CC="ccache gcc" or the default installation on
> Fedora, which puts it in the path) is what I do to keep the compile
> delay under control.

Ah, another point: My default build is always with -O3 for
the warnings. When I have to use the debugger, I change the
Makefile to remove the -O3, recompile just the file in
question and I'm done. How do I do that with waf?

> This dependency calculation is essentially where the time goes. A great
> deal of caching is done to keep this as short as possible, but the
> calculations must be done to ensure a correct build, every time.
>
> waf will never be as simple, nor as fast as a system without full
> dependency tracking, but a lot of effort was put in to make it as fast
> as it can be, within the constraint of always building correctly.

Well, we have tons of CPUs. Can't we fork that into separate
processes?

[Matthieu Patou]

On 05/21/2013 03:49 AM, Volker Lendecke wrote:
> On Tue, May 21, 2013 at 12:28:53PM +0200, Jelmer Vernooij wrote:

Have you tried the gold linker ? Micheal suggested me at XP and I must
confess that it makes things way more faster when linking.
For the moment wafbuild is only able to use the system linker but it
should be very hard to teach him how to use another linker for the one
who don't want to use it by default (the gold linker)

Matthieu

--
Matthieu Patou
Samba Team
http://samba.org

[Volker Lendecke]

On Tue, May 21, 2013 at 06:59:04AM -0700, Matthieu Patou wrote:
> >That does save linking of all non-involved libs and
> >binaries, but it will still compile everything first. At
> >least it did, when I last tried.
> Have you tried the gold linker ? Micheal suggested me at XP and I
> must confess that it makes things way more faster when linking.
> For the moment wafbuild is only able to use the system linker but it
> should be very hard to teach him how to use another linker for the
> one who don't want to use it by default (the gold linker)

Yes, I did try. But it did not work for me, so I did not
pursue it further.

[Volker Lendecke]

On Tue, May 21, 2013 at 04:07:48PM +0200, Volker Lendecke wrote:
> On Tue, May 21, 2013 at 06:59:04AM -0700, Matthieu Patou wrote:
> > >That does save linking of all non-involved libs and
> > >binaries, but it will still compile everything first. At
> > >least it did, when I last tried.
> > Have you tried the gold linker ? Micheal suggested me at XP and I
> > must confess that it makes things way more faster when linking.
> > For the moment wafbuild is only able to use the system linker but it
> > should be very hard to teach him how to use another linker for the
> > one who don't want to use it by default (the gold linker)
>
> Yes, I did try. But it did not work for me, so I did not
> pursue it further.

Also, if I understand it correctly, the gold linker only
would speed up the linking phase. It would not help with the
problem that we have to compile everything before we even
start linking, right?

[Jelmer Vernooij]

On Tue, May 21, 2013 at 06:59:04AM -0700, Matthieu Patou wrote:

> >That does save linking of all non-involved libs and
> >binaries, but it will still compile everything first. At
> >least it did, when I last tried.
> Have you tried the gold linker ? Micheal suggested me at XP and I
> must confess that it makes things way more faster when linking.
> For the moment wafbuild is only able to use the system linker but it
> should be very hard to teach him how to use another linker for the
> one who don't want to use it by default (the gold linker)

You might also want to try --as-needed with the standard linker
(-Wl,--as-needed). The gold linker does that by default.

Cheers,

Jelmer

[Michael Adam]

On 2013-05-21 at 16:11 +0200, Volker Lendecke wrote:

> > Yes, I did try. But it did not work for me, so I did not
> > pursue it further.

There were some problems in the past with the oder of some
arguments as provided by our waf system, where the gold linker
was stricter than the standard ld and stumbled. I fixed that
(9ba44cc610426fb558b49aa9680b5bdf55c29082) and afterwards the
gold linker has worked like a charm for me.

> Also, if I understand it correctly, the gold linker only
> would speed up the linking phase. It would not help with the
> problem that we have to compile everything before we even
> start linking, right?

Correct.

Cheers - Michael

[Jeremy Allison]

On Tue, May 21, 2013 at 04:07:48PM +0200, Volker Lendecke wrote:
> On Tue, May 21, 2013 at 06:59:04AM -0700, Matthieu Patou wrote:
> > >That does save linking of all non-involved libs and
> > >binaries, but it will still compile everything first. At
> > >least it did, when I last tried.
> > Have you tried the gold linker ? Micheal suggested me at XP and I
> > must confess that it makes things way more faster when linking.
> > For the moment wafbuild is only able to use the system linker but it
> > should be very hard to teach him how to use another linker for the
> > one who don't want to use it by default (the gold linker)
>
> Yes, I did try. But it did not work for me, so I did not
> pursue it further.

Yeah, me too and it broke the build. I keep meaning to
log a bug with Ian the developer (he works here) but
haven't gotten to that yet.

I agree, being able to just compile one .o file corresponding
to a .c file I just changed is one of the things I wish I
could do easily with waf. Not a show-stopper for me though
as waf does have other benefits.

Jeremy.

[C.J. Adams-Collier KF7BMP]

Hey there Andrew, list,

Speaking as a build engineer responsible for compiling samba to run on
hundreds of systems, I would personally prefer not to phase autotools
out. Many distributions assume that there is an autoreconf
-i ; ./configure ; make ; make install option available in software they
build for their platforms, and build engineers know the system well
enough to produce binaries. Forcing build engineers to learn something
new will delay and in some cases remove support entirely for samba on
many platforms.

In the past, I've needed to provide alternate build options to various
communities and have contributed code to a project called 'Prebuild'
which takes build definitions from an XML file and generates various
other formats. Although this worked in most cases, it was cumbersome
and error prone.

I understand that many folks feel that m4, libtool, automake, autoconf,
etc. are also cumbersome and error prone, but they get the job done
using an interface that has become customary in the unix build world.
Build systems already have auto* deployed to them and do not require
testing and review to ensure that they do what they are expected to do.

So, my 2¢ is this:
1) autotools should remain the primary environment probing /
configuration system
2) failing that, it should remain an equal first class participant along
with the new alternative
3) failing that, build definitions should be stored in an easy-to-parse
file and used to generate an autotools as well as other build systems

Cheers,

C.J.
IMHO, IANAL, IDDQD

[Stefan (metze) Metzmacher]

Hi C.J.,

> Speaking as a build engineer responsible for compiling samba to run on
> hundreds of systems, I would personally prefer not to phase autotools
> out. Many distributions assume that there is an autoreconf
> -i ; ./configure ; make ; make install option available in software they
> build for their platforms, and build engineers know the system well
> enough to produce binaries. Forcing build engineers to learn something
> new will delay and in some cases remove support entirely for samba on
> many platforms.

What in './configure && make && make install' (using waf) is more
complicated
than './autogen.sh && ./configure && make && make install' (using autoconf)?

metze

[Andreas Schneider]

On Tuesday 21 May 2013 16:07:48 Volker Lendecke wrote:
> On Tue, May 21, 2013 at 06:59:04AM -0700, Matthieu Patou wrote:
> > >That does save linking of all non-involved libs and
> > >binaries, but it will still compile everything first. At
> > >least it did, when I last tried.
> >
> > Have you tried the gold linker ? Micheal suggested me at XP and I
> > must confess that it makes things way more faster when linking.
> > For the moment wafbuild is only able to use the system linker but it
> > should be very hard to teach him how to use another linker for the
> > one who don't want to use it by default (the gold linker)
>
> Yes, I did try. But it did not work for me, so I did not
> pursue it further.

Yes, there should be a way to disable the dependency checking.

make <target>/fast

It also shouldn't require to run the dependency checker every time if no
wscript file has changed at all.


-- andreas

--
Andreas Schneider GPG-ID: F33E3FC6
Samba Team a...@samba.org
www.samba.org

[Matthieu Patou]

On 05/22/2013 09:22 AM, Jeremy Allison wrote:

> On Tue, May 21, 2013 at 04:07:48PM +0200, Volker Lendecke wrote:
>> On Tue, May 21, 2013 at 06:59:04AM -0700, Matthieu Patou wrote:
>>>> That does save linking of all non-involved libs and
>>>> binaries, but it will still compile everything first. At
>>>> least it did, when I last tried.
>>> Have you tried the gold linker ? Micheal suggested me at XP and I
>>> must confess that it makes things way more faster when linking.
>>> For the moment wafbuild is only able to use the system linker but it
>>> should be very hard to teach him how to use another linker for the
>>> one who don't want to use it by default (the gold linker)
>> Yes, I did try. But it did not work for me, so I did not
>> pursue it further.

> Yeah, me too and it broke the build. I keep meaning to
> log a bug with Ian the developer (he works here) but
> haven't gotten to that yet.

How could it break the build for you and not for me ?

>
> I agree, being able to just compile one .o file corresponding
> to a .c file I just changed is one of the things I wish I
> could do easily with waf. Not a show-stopper for me though
> as waf does have other benefits.

The thing is that if you changed a .c then yes the associated .o needs
to be rebuild but also the .so or the binary and also all the libs that
link with it.
I understand that basically the autoconf build was not so great at
getting dependencies so you were getting fewer rebuilds because it
thoughts that the file didn't need to be rebuild (but it was wrong) so
we kind of trade speed for correctness.

As I pointed on IRC, we should reduce the number of groups so that
linking can kick up while some stuff are still compiling (if possible)
and potentially not do the symbol checks if we have specified --targets=
on the command line so that we don't force to rebuild every thing that
depends on this change.

This should make the speed of waf build --targets=smbtorture much faster
when you change something in talloc.c or whatsoever.

Matthieu.

[C.J. Adams-Collier KF7BMP]

Hello metze,

It's not a question of difficulty. It's a question of dependence on
external (or bundled) software which may or may not provide the expected
environmental details one comes to expect from auto*. Does waf process
a config.h.in file to produce a config.h which can be depended on to
provide detected environmental minutia? Are there multiple phases into
which the build systems can hook to assist in generating the correct
Makefile? The devil's in the details.

> metze

C.J.

[Matthieu Patou]

On 05/22/2013 09:58 AM, C.J. Adams-Collier KF7BMP wrote:
> Hey there Andrew, list,
<snip>

> So, my 2¢ is this:
> 1) autotools should remain the primary environment probing /
> configuration system

The thing is that it's already not anymore the case, you can't build the
AD part with autoconf you should follow discussion that occured ~2/3
years ago.

[C.J. Adams-Collier KF7BMP]

On Wed, 2013-05-22 at 10:25 -0700, Matthieu Patou wrote:
> On 05/22/2013 09:58 AM, C.J. Adams-Collier KF7BMP wrote:
> > Hey there Andrew, list,
> <snip>
> > So, my 2¢ is this:
> > 1) autotools should remain the primary environment probing /
> > configuration system
> The thing is that it's already not anymore the case, you can't build the
> AD part with autoconf you should follow discussion that occured ~2/3
> years ago.

Thank you. I had not read these.

Are these the threads you speak of?

https://lists.samba.org/archive/samba-technical/2012-May/084080.html
https://lists.samba.org/archive/samba-technical/2010-April/070867.html

Cheers,

C.J.

[Matthieu Patou]

On 05/21/2013 07:11 AM, Volker Lendecke wrote:
> On Tue, May 21, 2013 at 04:07:48PM +0200, Volker Lendecke wrote:
>> On Tue, May 21, 2013 at 06:59:04AM -0700, Matthieu Patou wrote:
>>>> That does save linking of all non-involved libs and
>>>> binaries, but it will still compile everything first. At
>>>> least it did, when I last tried.
>>> Have you tried the gold linker ? Micheal suggested me at XP and I
>>> must confess that it makes things way more faster when linking.
>>> For the moment wafbuild is only able to use the system linker but it
>>> should be very hard to teach him how to use another linker for the
>>> one who don't want to use it by default (the gold linker)
>> Yes, I did try. But it did not work for me, so I did not
>> pursue it further.

> Also, if I understand it correctly, the gold linker only
> would speed up the linking phase. It would not help with the
> problem that we have to compile everything before we even
> start linking, right?

Correct, part of a lot of recompilation might come from the check
symbols stuff that force every lib to be recompiled, I suspect that we
can add an option to not enable it (enabled by default in
./configure.developer) and with the use of --targets it should help.

As said on irc, if you are working on a lib like talloc or ldb, doing
waf --targets=<lib>.objlist will just compile the .o for this object
Matthieu.

[Herb Lewis]

On 05/22/2013 10:04 AM, Stefan (metze) Metzmacher wrote:
> Hi C.J.,
>
>> Speaking as a build engineer responsible for compiling samba to run on
>> hundreds of systems, I would personally prefer not to phase autotools
>> out. Many distributions assume that there is an autoreconf
>> -i ; ./configure ; make ; make install option available in software they
>> build for their platforms, and build engineers know the system well
>> enough to produce binaries. Forcing build engineers to learn something
>> new will delay and in some cases remove support entirely for samba on
>> many platforms.
> What in './configure && make && make install' (using waf) is more
> complicated
> than './autogen.sh && ./configure && make && make install' (using autoconf)?
>

> metze
>
what if your build system doesn't have waf and you have no control
on getting it installed?

[Michael Adam]

Hi Herb,

We ship our copy of waf. Only python is required.

Cheers - Michael

[Andrew Bartlett]

On Wed, 2013-05-22 at 10:21 -0700, Matthieu Patou wrote:

> As I pointed on IRC, we should reduce the number of groups so that
> linking can kick up while some stuff are still compiling (if possible)
> and potentially not do the symbol checks if we have specified --targets=
> on the command line so that we don't force to rebuild every thing that
> depends on this change.

The duplicate symbol checker is run on make test, not on make for this
reason.

The --symbol-check option is not enabled by default, nor is it turned on
by --enable-developer.

The ABI checker can be disabled by configuring with --abi-check-disable,
and as I understand the code only runs on libraries that have been
changed or re-linked. That said, I do agree that waf tends to prefer to
re-link over being sure that it doesn't need to.

> This should make the speed of waf build --targets=smbtorture much faster
> when you change something in talloc.c or whatsoever.

One of the key reasons to make this change is so that if we can find
ways to make waf both totally reliable and a bit faster for developers,
that we have those resources, rather than also spending those on
maintaining a parallel build system.

It is however really important that our build systems are totally
reliable. It is no benefit being a bit faster building if we loose that
time again debugging an 'impossible' Samba crash, that doesn't reproduce
anywhere but the customer's production environment, because they patched
Samba and didn't rebuild from 'git clean -x -f -d' or a fresh tarball
checkout.

The fact that waf always builds correctly, without special rules is a
key feature that *saves* developer time in the long run, by ensuring
that our users (who overall, build samba even more than we do) always
get a correct build.

[Andrew Bartlett]

On Tue, 2013-05-21 at 11:31 +0200, Michael Adam wrote:
> Hi Andrew,

> And maybe we can get some voices here of people using
> Samba file servers on various Unixes and of the distributors
> (of Linux, but also BSD flavours maybe).

I can speak to the process of packaging Samba 4.0, on Debian actually.

I've been involved in the effort to produce a Samba 4.0 package, and the
biggest challenge has not been the build system (because it is
still ./configure && make && DESTDIR=/tmp/installroot make install) but
in merging the two packing efforts because of the pre-existing samba4
packages.

There is more work in ensuring all the shared libraries are put into the
right packages (particularly because debian packages try to split things
up), but also great relief that Samba is finally using shared libs, and
is therefore smaller.

There is also work in adjusting to the slightly altered configure
syntax, but we did a lot of work to ensure options came across from
autoconf for 4.0, and this isn't difficult.

On the Red Hat side of the fence, I'll let the developers involved speak
for themselves, but we had a flurry of patches before 4.0 specifically
around tweaking the waf build to make it a little easier to package
correctly.

[C.J. Adams-Collier]

On May 21, 2013 4:11 AM, "Andrew Bartlett" <abar...@samba.org> wrote:

>
> On Tue, 2013-05-21 at 12:51 +0200, Volker Lendecke wrote:
> > On Tue, May 21, 2013 at 12:28:53PM +0200, Jelmer Vernooij wrote:

> > > IIRC
> > >
> > > ./buildtools/bin/waf build --targets=smbtorture
> > >
> > > should rebuild just smbtorture.
> >
> > Ah, another one: When coding larger changes, in autoconf I
> > like the ability to just compile one .o file. For example, I
> > frequently do a "make smbd/open.o" for example if I mess
> > with that. So far I have not found a way to do that in waf
> > quickly. Any hint for that?
>
> You could determine the command waf is using (make V=1 should do it),
> and manually invoke gcc yourself.

Yuck.

> It's not pretty, but it should allow you to handle this unusual case.

Citation needed. This is a common use case that should just work.

[Matthieu Patou]

On 05/22/2013 02:49 PM, Andrew Bartlett wrote:
> On Wed, 2013-05-22 at 10:21 -0700, Matthieu Patou wrote:
>
>> As I pointed on IRC, we should reduce the number of groups so that
>> linking can kick up while some stuff are still compiling (if possible)
>> and potentially not do the symbol checks if we have specified --targets=
>> on the command line so that we don't force to rebuild every thing that
>> depends on this change.
> The duplicate symbol checker is run on make test, not on make for this
> reason.
>
> The --symbol-check option is not enabled by default, nor is it turned on
> by --enable-developer.
>
> The ABI checker can be disabled by configuring with --abi-check-disable,
> and as I understand the code only runs on libraries that have been
> changed or re-linked.

True but let's say you modified talloc but want to see the changes only
smbtorture because you are iterating on this, as we have the abi-check
before the linking of the binaries (but after the linking of dynamic
libraries) waf will insists on relinking all the libraries even thoses
not needed by your target.

Also I suspect that we have too broad dependencies as sometime changing
stuff for DRS replication trigger relinking in the VFS modules.

> That said, I do agree that waf tends to prefer to
> re-link over being sure that it doesn't need to.
>
>> This should make the speed of waf build --targets=smbtorture much faster
>> when you change something in talloc.c or whatsoever.
> One of the key reasons to make this change is so that if we can find
> ways to make waf both totally reliable and a bit faster for developers,
> that we have those resources, rather than also spending those on
> maintaining a parallel build system.
>
> It is however really important that our build systems are totally
> reliable. It is no benefit being a bit faster building if we loose that
> time again debugging an 'impossible' Samba crash, that doesn't reproduce
> anywhere but the customer's production environment, because they patched
> Samba and didn't rebuild from 'git clean -x -f -d' or a fresh tarball
> checkout.
>
> The fact that waf always builds correctly, without special rules is a
> key feature that *saves* developer time in the long run, by ensuring
> that our users (who overall, build samba even more than we do) always
> get a correct build.

I do agree, I don't think that reducing groups or forcing the abi-check
to occur at the very last stage will actually change the reliability of
the build.
I think it will be harder to be as fast as the autoconf build (on
rebuilds) as it was skipping some stuff but we can most probably speed
up the --target= stuff without impacting reliability.

[Volker Lendecke]

On Thu, May 23, 2013 at 07:49:01AM +1000, Andrew Bartlett wrote:
> On Wed, 2013-05-22 at 10:21 -0700, Matthieu Patou wrote:
>
> > As I pointed on IRC, we should reduce the number of groups so that
> > linking can kick up while some stuff are still compiling (if possible)
> > and potentially not do the symbol checks if we have specified --targets=
> > on the command line so that we don't force to rebuild every thing that
> > depends on this change.
>
> The duplicate symbol checker is run on make test, not on make for this
> reason.
>
> The --symbol-check option is not enabled by default, nor is it turned on
> by --enable-developer.
>
> The ABI checker can be disabled by configuring with --abi-check-disable,
> and as I understand the code only runs on libraries that have been

> changed or re-linked. That said, I do agree that waf tends to prefer to

> re-link over being sure that it doesn't need to.
>
> > This should make the speed of waf build --targets=smbtorture much faster
> > when you change something in talloc.c or whatsoever.
>
> One of the key reasons to make this change is so that if we can find
> ways to make waf both totally reliable and a bit faster for developers,
> that we have those resources, rather than also spending those on
> maintaining a parallel build system.
>
> It is however really important that our build systems are totally
> reliable. It is no benefit being a bit faster building if we loose that
> time again debugging an 'impossible' Samba crash, that doesn't reproduce
> anywhere but the customer's production environment, because they patched
> Samba and didn't rebuild from 'git clean -x -f -d' or a fresh tarball
> checkout.

Our perception might just differ here. I do get those
crashes, yes, but for me they are very rare. Not more than
once or twice a year. And yes, if I get them, it takes me a
couple of minutes to realize I should retry with a clean
build. These few minutes for me are nothing compared to the
wait time for every single build.

> The fact that waf always builds correctly, without special rules is a
> key feature that *saves* developer time in the long run, by ensuring
> that our users (who overall, build samba even more than we do) always
> get a correct build.

I'm not sure about that. The people who track us with a git
pull should know that a git clean -dxf every now and then
helps. Has this imprecision really caused us so much trouble
before waf came along that we want to put delays on
everybody who writes code every day? The clean build that
our users build from (get the tar, unpack it and build) has
worked pretty well so far.

[Andrew Bartlett]

On Wed, 2013-05-22 at 21:41 -0700, Matthieu Patou wrote:

> I do agree, I don't think that reducing groups or forcing the abi-check
> to occur at the very last stage will actually change the reliability of
> the build.

You keep talking about the ABI checker. Please first compare the
results with and without this enabled, I think you will find that it
does not change what is linked or when.

Yes, waf is quite aggressive with re-linking, but it has always been
like that, including long before the ABI checker was added.

[Volker Lendecke]

On Wed, May 22, 2013 at 10:21:35AM -0700, Matthieu Patou wrote:

> On 05/22/2013 09:22 AM, Jeremy Allison wrote:
> >On Tue, May 21, 2013 at 04:07:48PM +0200, Volker Lendecke wrote:
> >>On Tue, May 21, 2013 at 06:59:04AM -0700, Matthieu Patou wrote:
> >>>>That does save linking of all non-involved libs and
> >>>>binaries, but it will still compile everything first. At
> >>>>least it did, when I last tried.
> >>>Have you tried the gold linker ? Micheal suggested me at XP and I
> >>>must confess that it makes things way more faster when linking.
> >>>For the moment wafbuild is only able to use the system linker but it
> >>>should be very hard to teach him how to use another linker for the
> >>>one who don't want to use it by default (the gold linker)
> >>Yes, I did try. But it did not work for me, so I did not
> >>pursue it further.

> >Yeah, me too and it broke the build. I keep meaning to
> >log a bug with Ian the developer (he works here) but
> >haven't gotten to that yet.
> How could it break the build for you and not for me ?
> >
> >I agree, being able to just compile one .o file corresponding
> >to a .c file I just changed is one of the things I wish I
> >could do easily with waf. Not a show-stopper for me though
> >as waf does have other benefits.
> The thing is that if you changed a .c then yes the associated .o
> needs to be rebuild but also the .so or the binary and also all the
> libs that link with it.
> I understand that basically the autoconf build was not so great at
> getting dependencies so you were getting fewer rebuilds because it
> thoughts that the file didn't need to be rebuild (but it was wrong)
> so we kind of trade speed for correctness.

Indeed. And my clear preference for me is to not have 100%
correctness if it will give me faster turnaround on
edit/build/run cycles. I always have the chance to rebuild
everything if things go wrong somehow. As I wrote in another
mail, I am willing to get confused once or twice a year
because the Makefile's missing dependencies broke my
binaries.

find .. -name \*.o | xargs rm

is typed quickly enough.

[David Disseldorp]

On Tue, 21 May 2013 19:08:06 +1000
Andrew Bartlett <abar...@samba.org> wrote:

> I'm not going to say that waf is perfect - it certainly isn't, but for
> Samba 4.1 and beyond, we should focus our energies on a single build
> system, and making it better, rather than continuing to build this large
> and complex code base with two independent build systems that we have to
> keep in sync.
>
> The autoconf build system has supported us for a long time, but just as
> it is time that we retire SWAT, Samba 4.1 is the appropriate time to
> retire autoconf, and move to a single way to build a unified Samba
> project.

+1 from me. Continuing to maintain both build systems is a heavy and
unnecessary burden.

Cheers, David

[Andrew Bartlett]

Thanks David,

I wonder if you would mind looking over the patch I've proposed (see my
other mail), to see if there is anything I've missed? It passes a full
autobuild for me.

It is at
https://git.samba.org/abartlet/samba.git/?p=abartlet/samba.git/.git;a=shortlog;h=refs/heads/remove-autoconf or I can mail you the patch (it's 800k, so I wasn't going to post it to the list).

Thanks,

[Volker Lendecke]

On Thu, May 23, 2013 at 12:00:14PM +0200, David Disseldorp wrote:
> On Tue, 21 May 2013 19:08:06 +1000
> Andrew Bartlett <abar...@samba.org> wrote:
>
> > I'm not going to say that waf is perfect - it certainly isn't, but for
> > Samba 4.1 and beyond, we should focus our energies on a single build
> > system, and making it better, rather than continuing to build this large
> > and complex code base with two independent build systems that we have to
> > keep in sync.
> >
> > The autoconf build system has supported us for a long time, but just as
> > it is time that we retire SWAT, Samba 4.1 is the appropriate time to
> > retire autoconf, and move to a single way to build a unified Samba
> > project.
>
> +1 from me. Continuing to maintain both build systems is a heavy and
> unnecessary burden.

Please hold back the patch until the issues I raised are
addressed. I am certainly willing to hack things up, but the
points are raised are really important to me.

Thanks,

Volker

--
SerNet GmbH, Bahnhofsallee 1b, 37081 G�ttingen
phone: +49-551-370000-0, fax: +49-551-370000-9
AG G�ttingen, HRB 2816, GF: Dr. Johannes Loxen
http://www.sernet.de, mailto:kon...@sernet.de

[Volker Lendecke]

On Thu, May 23, 2013 at 12:24:05PM +0200, Volker Lendecke wrote:
> On Thu, May 23, 2013 at 12:00:14PM +0200, David Disseldorp wrote:
> > On Tue, 21 May 2013 19:08:06 +1000
> > Andrew Bartlett <abar...@samba.org> wrote:
> >
> > > I'm not going to say that waf is perfect - it certainly isn't, but for
> > > Samba 4.1 and beyond, we should focus our energies on a single build
> > > system, and making it better, rather than continuing to build this large
> > > and complex code base with two independent build systems that we have to
> > > keep in sync.
> > >
> > > The autoconf build system has supported us for a long time, but just as
> > > it is time that we retire SWAT, Samba 4.1 is the appropriate time to
> > > retire autoconf, and move to a single way to build a unified Samba
> > > project.
> >
> > +1 from me. Continuing to maintain both build systems is a heavy and
> > unnecessary burden.
>
> Please hold back the patch until the issues I raised are
> addressed. I am certainly willing to hack things up, but the
> points are raised are really important to me.

s/are/I/

Sorry

Volker

--
SerNet GmbH, Bahnhofsallee 1b, 37081 Göttingen
phone: +49-551-370000-0, fax: +49-551-370000-9
AG Göttingen, HRB 2816, GF: Dr. Johannes Loxen
http://www.sernet.de, mailto:kon...@sernet.de

[David Disseldorp]

On Thu, 23 May 2013 20:18:57 +1000
Andrew Bartlett <abar...@samba.org> wrote:

> I wonder if you would mind looking over the patch I've proposed (see my
> other mail), to see if there is anything I've missed? It passes a full
> autobuild for me.

Sure, I'll take a look. Though I don't think I'll be able to get to it
until later this week given the size.

Cheers, David

[Andrew Bartlett]

Thanks. Most of it is straightforward, and it should be practical to do
in chunks once we decide because after the big 'rm', most of the long
tail is just removing stuff we no longer need.

[David Disseldorp]

On Thu, 23 May 2013 12:24:05 +0200
Volker Lendecke <Volker....@SerNet.DE> wrote:

> Please hold back the patch until the issues I raised are
> addressed. I am certainly willing to hack things up, but the
> points are raised are really important to me.

I won't push anything until those points are cleared up, but that
shouldn't keep me from reviewing the changes.

Cheers, David

[yabe...@ca.ibm.com]

Hi,

We are using Samba 3.6.x built by ourself on AIX with IBM XL C/C++
compiler, with the autoconf build system and it's working without issue.
We use Samba mainly for its file-server feature to share DFS, GPFS and
JFS2 filesystems to Windows systems and to Linux workstations.
We will be looking in the upcoming months/years (before you stop providing

security fixes for 3.6) to upgrade to Samba 4.x (file-server only) so we
hope to be able to build it in our environment.

We currently don't have Python on these systems as nothing requires it.
There is no IBM supported python package for AIX.
There is some sites providing Python packages (such as
http://www-frec.bull.com/ and http://www.perzl.org/aix/) but on a best
effort basis and not always at the latest release.
Being able to be on a supported version (or at least, an up-to-date
version for security fixes) is mandatory for us.
Therefore, the most logical solution remaining is to build Python ourself
and maintain it.

I believe waf requires Python 2.x so I've tried to compile Python 2.7.5 on

AIX with IBM XL C/C++.
"configure" and "make" worked well.
make test is throwing some error and wasn't able to complete
test_hotshot
make: 1254-059 The signal code from the last command is 11.
Stop.
I've did the make install anyway, tested it with a simple python script
and it worked.

I've downloaded Samba 4.0.6 and tried it with waf following this wiki:
https://wiki.samba.org/index.php/Waf#Using_waf_directly
cd lib/tdb
waf configure --enable-developer --prefix=$HOME/testprefix
waf
waf install

One of the "warning" I've got while running "waf" is that many options are

not supported by IBM XL C/C++ (but probably are by gcc)
/usr/vacpp/bin/xlc_r: 1501-289 (W) Option -Wall was incorrectly specified.
The option will be ignored.
/usr/vacpp/bin/xlc_r: 1501-289 (W) Option -Wshadow was incorrectly
specified. The option will be ignored.
/usr/vacpp/bin/xlc_r: 1501-289 (W) Option -Werror=strict-prototypes was
incorrectly specified. The option will be ignored.
/usr/vacpp/bin/xlc_r: 1501-289 (W) Option -Wstrict-prototypes was
incorrectly specified. The option will be ignored.
/usr/vacpp/bin/xlc_r: 1501-289 (W) Option -Werror=pointer-arith was
incorrectly specified. The option will be ignored.
/usr/vacpp/bin/xlc_r: 1501-289 (W) Option -Wpointer-arith was incorrectly
specified. The option will be ignored.
/usr/vacpp/bin/xlc_r: 1501-289 (W) Option -Wcast-align was incorrectly
specified. The option will be ignored.
/usr/vacpp/bin/xlc_r: 1501-289 (W) Option -Werror=write-strings was
incorrectly specified. The option will be ignored.
/usr/vacpp/bin/xlc_r: 1501-289 (W) Option -Wwrite-strings was incorrectly
specified. The option will be ignored.
/usr/vacpp/bin/xlc_r: 1501-289 (W) Option
-Werror-implicit-function-declaration was incorrectly specified. The
option will be ignored.
/usr/vacpp/bin/xlc_r: 1501-289 (W) Option -Wformat=2 was incorrectly
specified. The option will be ignored.
/usr/vacpp/bin/xlc_r: 1501-289 (W) Option -Wno-format-y2k was incorrectly
specified. The option will be ignored.
/usr/vacpp/bin/xlc_r: 1501-289 (W) Option -Wmissing-prototypes was
incorrectly specified. The option will be ignored.
/usr/vacpp/bin/xlc_r: 1501-289 (W) Option -Werror=address was incorrectly
specified. The option will be ignored.
/usr/vacpp/bin/xlc_r: 1501-289 (W) Option -Wcast-qual was incorrectly
specified. The option will be ignored.
/usr/vacpp/bin/xlc_r: 1501-289 (W) Option -Werror=format was incorrectly
specified. The option will be ignored.

I've did it again but from the samba-4.0.6 directory this time.
gzip -cd samba-4.0.6.tar.gz |tar xf -
cd samba-4.0.6
waf configure --enable-developer --prefix=/usr/local/samba-4.0.6
waf
waf install
I've had the following error while running "waf"
"../lib/socket/interfaces.c", line 150.42: 1506-045 (S) Undeclared
identifier IFF_UP.
"../lib/socket/interfaces.c", line 204.44: 1506-045 (S) Undeclared
identifier IFF_BROADCAST.
"../lib/socket/interfaces.c", line 204.58: 1506-045 (S) Undeclared
identifier IFF_LOOPBACK.
"../lib/socket/interfaces.c", line 208.51: 1506-045 (S) Undeclared
identifier IFF_POINTOPOINT.
Waf: Leaving directory `/usr/src/samba-4.0.6/bin'
Build failed: -> task failed (err #1):
{task: cc interfaces.c -> interfaces_1.o}
The problem is probably related to the code itself and not to the waf
build engine.
But as this haven't allowed me to complete the build, I don't know if I
would have any waf issue after that.



Conclusion
While I'm not 100% comfortable to see the autoconf build option removed, I

don't have any strong technical argument to be against.
I would however try to find time with one of the Samba team member to see
if it would be possible to fix the warnings/errors reported on AIX with
IBM XL C/C++.


Best regards,


Yannick Bergeron
450 534-7711
yabe...@ca.ibm.com
Advisory IT Specialist

Never say never, say "it depends" / Ne jamais dire jamais, dites "ca
dépend"

[Ricky Nance]

Give the build_with_python.sh script a try. I am at work right now and
can't confirm that is the correct name, but it should be similar, and it
will be in the base directory.

Ricky

[yabe...@ca.ibm.com]

Can't find it in the samba-4.0.6.tar.gz

It seems to be in master with the name "install_with_python.sh"
this script installs a private copy of python in the same prefix as Samba

This would definitely looks like an interesting alternative if the Samba
team keep this version up-to-date with security issues.
If I can find a few minutes, I might attempt to fetch the master tree and
see if I can build it with it

Best regards,

Yannick Bergeron
450 534-7711
yabe...@ca.ibm.com
Advisory IT Specialist

Never say never, say "it depends" / Ne jamais dire jamais, dites "ca
dépend"

From: Ricky Nance <ricky...@gmail.com>
To: yabe...@ca.ibm.com,
Cc: Samba Technical <samba-t...@lists.samba.org>, Andrew Bartlett

<abar...@samba.org>
Date: 05/23/2013 09:41 AM
Subject: Re: [PROPOSAL] To retire autoconf for 4.1

[Matthieu Patou]

it's in net/if.h in linux, is there this file in AIX? maybe the
detection test for this file is failing ?

>
>
> Conclusion
> While I'm not 100% comfortable to see the autoconf build option removed, I
>
> don't have any strong technical argument to be against.
> I would however try to find time with one of the Samba team member to see
> if it would be possible to fix the warnings/errors reported on AIX with
> IBM XL C/C++.
>

The AIX machine that we have in the build farm is a bit old, do you
think it would be possible to have one of your AIX server running the
buildfarm stuff so that we can have better reports ?
Having an up to date machine and with enough power should help us to get
accurate numbers and feedback.


Matthieu.

[Andrew Bartlett]

On Thu, 2013-05-23 at 12:24 +0200, Volker Lendecke wrote:
> On Thu, May 23, 2013 at 12:00:14PM +0200, David Disseldorp wrote:

> > On Tue, 21 May 2013 19:08:06 +1000
> > Andrew Bartlett <abar...@samba.org> wrote:
> >
> > > I'm not going to say that waf is perfect - it certainly isn't, but for
> > > Samba 4.1 and beyond, we should focus our energies on a single build
> > > system, and making it better, rather than continuing to build this large
> > > and complex code base with two independent build systems that we have to
> > > keep in sync.
> > >
> > > The autoconf build system has supported us for a long time, but just as
> > > it is time that we retire SWAT, Samba 4.1 is the appropriate time to
> > > retire autoconf, and move to a single way to build a unified Samba
> > > project.
> >
> > +1 from me. Continuing to maintain both build systems is a heavy and
> > unnecessary burden.
>

> Please hold back the patch until the issues I raised are
> addressed. I am certainly willing to hack things up, but the
> points are raised are really important to me.

Volker,

I appreciate your willingness to work on a solution for your concerns
here, as I think you understand your specific needs best. I'm always
happy to advise, and it will be really good to move this from the
general to the specific as we discuss what exact patches we could apply
to improve your situation.

Thanks,

[Volker Lendecke]

On Sun, May 26, 2013 at 08:14:52PM +1000, Andrew Bartlett wrote:
> I appreciate your willingness to work on a solution for your concerns
> here, as I think you understand your specific needs best. I'm always
> happy to advise, and it will be really good to move this from the
> general to the specific as we discuss what exact patches we could apply
> to improve your situation.

So to summarize, I have a few complaints:

Long startup time: No idea where the time is spent, but 6
seconds on a clean build seems really excessive.

Always compile everything before linking: I'm told that this
is due to broken dependencies, but I haven't followed this
exactly. I had thought that the main advantage of waf over
autoconf is that it is perfect with dependencies, so I
probably have just not listened right. If waf's way to
dependency perfection is to compile almost everything
always, I think that should be doable with autoconf as well.

Compile individual C files: It seems not possible without a
gross hack to compile just one file on demand without all
that long startup phase etc.

Change CFLAGS between runs: Nobody has commented on this one
so far.

The first two are certainly the most annoying ones, but
probably the two that are hardest to fix. If Tridge failed
to fix something (startup time), this is my very definition
of "unfixable". So I can only ask everybody to keep the old
build system around.

Thanks,

[C.J. Adams-Collier]

Is it possible to run AIX in a virtual environment? Perhaps IBM could
provide a build system to the team for development and testing purposes.

> Conclusion
> While I'm not 100% comfortable to see the autoconf build option removed, I
>
> don't have any strong technical argument to be against.
> I would however try to find time with one of the Samba team member to see
> if it would be possible to fix the warnings/errors reported on AIX with
> IBM XL C/C++.
>
>