Fw: [PROPOSAL] To retire autoconf for 4.1

[yabe...@ca.ibm.com]

nevermind, install_with_python.sh is part of samba-4.0.6.tar.gz

However, I'm not sure I like it.
It tries to use rsync (not available on our system but can be installed)
to download this file:
ftp://ftp.samba.org/pub/tridge/python/Python-2.6.5.tar
It doesn't look like something recent (2.6.5 was released in 2010, 2.6.8
and 2.7.5 are available)

I believe I would prefer to build and maintain it myself.


Best regards,

Yannick Bergeron
450 534-7711
yabe...@ca.ibm.com
Advisory IT Specialist

Never say never, say "it depends" / Ne jamais dire jamais, dites "ca
dépend"
----- Forwarded by Yannick Y Bergeron/Bromont/IBM on 05/23/2013 10:12 AM
-----

From: Yannick Y Bergeron/Bromont/IBM
To: Ricky Nance <ricky...@gmail.com>,
Cc: Andrew Bartlett <abar...@samba.org>, Samba Technical
<samba-t...@lists.samba.org>
Date: 05/23/2013 10:08 AM
Subject: Re: [PROPOSAL] To retire autoconf for 4.1


Can't find it in the samba-4.0.6.tar.gz

It seems to be in master with the name "install_with_python.sh"
this script installs a private copy of python in the same prefix as Samba

This would definitely looks like an interesting alternative if the Samba
team keep this version up-to-date with security issues.
If I can find a few minutes, I might attempt to fetch the master tree and
see if I can build it with it


Best regards,

Yannick Bergeron
450 534-7711
yabe...@ca.ibm.com
Advisory IT Specialist

Never say never, say "it depends" / Ne jamais dire jamais, dites "ca
dépend"



From: Ricky Nance <ricky...@gmail.com>
To: yabe...@ca.ibm.com,
Cc: Samba Technical <samba-t...@lists.samba.org>, Andrew Bartlett

<abar...@samba.org>
Date: 05/23/2013 09:41 AM
Subject: Re: [PROPOSAL] To retire autoconf for 4.1



Give the build_with_python.sh script a try. I am at work right now and

can't confirm that is the correct name, but it should be similar, and it
will be in the base directory.

Ricky


On Thu, May 23, 2013 at 8:27 AM, <yabe...@ca.ibm.com> wrote:
Hi,

We are using Samba 3.6.x built by ourself on AIX with IBM XL C/C++
compiler, with the autoconf build system and it's working without issue.
We use Samba mainly for its file-server feature to share DFS, GPFS and
JFS2 filesystems to Windows systems and to Linux workstations.
We will be looking in the upcoming months/years (before you stop providing

security fixes for 3.6) to upgrade to Samba 4.x (file-server only) so we
hope to be able to build it in our environment.

We currently don't have Python on these systems as nothing requires it.
There is no IBM supported python package for AIX.
There is some sites providing Python packages (such as
http://www-frec.bull.com/ and http://www.perzl.org/aix/) but on a best
effort basis and not always at the latest release.
Being able to be on a supported version (or at least, an up-to-date
version for security fixes) is mandatory for us.
Therefore, the most logical solution remaining is to build Python ourself
and maintain it.

I believe waf requires Python 2.x so I've tried to compile Python 2.7.5 on

AIX with IBM XL C/C++.
"configure" and "make" worked well.
make test is throwing some error and wasn't able to complete
test_hotshot
make: 1254-059 The signal code from the last command is 11.
Stop.
I've did the make install anyway, tested it with a simple python script
and it worked.

I've downloaded Samba 4.0.6 and tried it with waf following this wiki:
https://wiki.samba.org/index.php/Waf#Using_waf_directly
cd lib/tdb
waf configure --enable-developer --prefix=$HOME/testprefix
waf
waf install

One of the "warning" I've got while running "waf" is that many options are

not supported by IBM XL C/C++ (but probably are by gcc)
/usr/vacpp/bin/xlc_r: 1501-289 (W) Option -Wall was incorrectly specified.
The option will be ignored.
/usr/vacpp/bin/xlc_r: 1501-289 (W) Option -Wshadow was incorrectly
specified. The option will be ignored.
/usr/vacpp/bin/xlc_r: 1501-289 (W) Option -Werror=strict-prototypes was
incorrectly specified. The option will be ignored.
/usr/vacpp/bin/xlc_r: 1501-289 (W) Option -Wstrict-prototypes was
incorrectly specified. The option will be ignored.
/usr/vacpp/bin/xlc_r: 1501-289 (W) Option -Werror=pointer-arith was
incorrectly specified. The option will be ignored.
/usr/vacpp/bin/xlc_r: 1501-289 (W) Option -Wpointer-arith was incorrectly
specified. The option will be ignored.
/usr/vacpp/bin/xlc_r: 1501-289 (W) Option -Wcast-align was incorrectly
specified. The option will be ignored.
/usr/vacpp/bin/xlc_r: 1501-289 (W) Option -Werror=write-strings was
incorrectly specified. The option will be ignored.
/usr/vacpp/bin/xlc_r: 1501-289 (W) Option -Wwrite-strings was incorrectly
specified. The option will be ignored.
/usr/vacpp/bin/xlc_r: 1501-289 (W) Option
-Werror-implicit-function-declaration was incorrectly specified. The
option will be ignored.
/usr/vacpp/bin/xlc_r: 1501-289 (W) Option -Wformat=2 was incorrectly
specified. The option will be ignored.
/usr/vacpp/bin/xlc_r: 1501-289 (W) Option -Wno-format-y2k was incorrectly
specified. The option will be ignored.
/usr/vacpp/bin/xlc_r: 1501-289 (W) Option -Wmissing-prototypes was
incorrectly specified. The option will be ignored.
/usr/vacpp/bin/xlc_r: 1501-289 (W) Option -Werror=address was incorrectly
specified. The option will be ignored.
/usr/vacpp/bin/xlc_r: 1501-289 (W) Option -Wcast-qual was incorrectly
specified. The option will be ignored.
/usr/vacpp/bin/xlc_r: 1501-289 (W) Option -Werror=format was incorrectly
specified. The option will be ignored.

I've did it again but from the samba-4.0.6 directory this time.
gzip -cd samba-4.0.6.tar.gz |tar xf -
cd samba-4.0.6
waf configure --enable-developer --prefix=/usr/local/samba-4.0.6
waf
waf install
I've had the following error while running "waf"
"../lib/socket/interfaces.c", line 150.42: 1506-045 (S) Undeclared
identifier IFF_UP.
"../lib/socket/interfaces.c", line 204.44: 1506-045 (S) Undeclared
identifier IFF_BROADCAST.
"../lib/socket/interfaces.c", line 204.58: 1506-045 (S) Undeclared
identifier IFF_LOOPBACK.
"../lib/socket/interfaces.c", line 208.51: 1506-045 (S) Undeclared
identifier IFF_POINTOPOINT.
Waf: Leaving directory `/usr/src/samba-4.0.6/bin'
Build failed: -> task failed (err #1):
{task: cc interfaces.c -> interfaces_1.o}
The problem is probably related to the code itself and not to the waf
build engine.
But as this haven't allowed me to complete the build, I don't know if I
would have any waf issue after that.



Conclusion
While I'm not 100% comfortable to see the autoconf build option removed, I

don't have any strong technical argument to be against.
I would however try to find time with one of the Samba team member to see
if it would be possible to fix the warnings/errors reported on AIX with
IBM XL C/C++.


Best regards,


Yannick Bergeron
450 534-7711
yabe...@ca.ibm.com
Advisory IT Specialist

Never say never, say "it depends" / Ne jamais dire jamais, dites "ca
dépend"

[Ricky Nance]

If you get too new of a version of python (3.0+), it can break thinks, this
version is known to work with samba. I'd be willing to bet that 2.7.5 will
cause things to break, but I haven't tested it myself (I want to say 2.7.3
worked, so I very well could be wrong about this).

Good luck,
Ricky

On Thu, May 23, 2013 at 9:15 AM, <yabe...@ca.ibm.com> wrote:

> nevermind, install_with_python.sh is part of samba-4.0.6.tar.gz
>
> However, I'm not sure I like it.
> It tries to use rsync (not available on our system but can be installed)
> to download this file:
> ftp://ftp.samba.org/pub/tridge/python/Python-2.6.5.tar
> It doesn't look like something recent (2.6.5 was released in 2010, 2.6.8
> and 2.7.5 are available)
>
> I believe I would prefer to build and maintain it myself.
>
>
> Best regards,
>
> Yannick Bergeron
> 450 534-7711
> yabe...@ca.ibm.com
> Advisory IT Specialist
>
> Never say never, say "it depends" / Ne jamais dire jamais, dites "ca
> dépend"
> ----- Forwarded by Yannick Y Bergeron/Bromont/IBM on 05/23/2013 10:12 AM
> -----
>
> From: Yannick Y Bergeron/Bromont/IBM
> To: Ricky Nance <ricky...@gmail.com>,
> Cc: Andrew Bartlett <abar...@samba.org>, Samba Technical <
> samba-t...@lists.samba.org>
> Date: 05/23/2013 10:08 AM
> Subject: Re: [PROPOSAL] To retire autoconf for 4.1

> ------------------------------

>
>
> Can't find it in the samba-4.0.6.tar.gz
>
> It seems to be in master with the name "install_with_python.sh"

> *this script installs a private copy of python in the same prefix as Samba
> *

>
> This would definitely looks like an interesting alternative if the Samba
> team keep this version up-to-date with security issues.
> If I can find a few minutes, I might attempt to fetch the master tree and
> see if I can build it with it
>
>
> Best regards,
>
> Yannick Bergeron
> 450 534-7711
> yabe...@ca.ibm.com
> Advisory IT Specialist
>
> Never say never, say "it depends" / Ne jamais dire jamais, dites "ca
> dépend"
>
>
>
> From: Ricky Nance <ricky...@gmail.com>
> To: yabe...@ca.ibm.com,
> Cc: Samba Technical <samba-t...@lists.samba.org>, Andrew
> Bartlett <abar...@samba.org>
> Date: 05/23/2013 09:41 AM
> Subject: Re: [PROPOSAL] To retire autoconf for 4.1

> ------------------------------

>
>
>
> Give the build_with_python.sh script a try. I am at work right now and
> can't confirm that is the correct name, but it should be similar, and it
> will be in the base directory.
>
> Ricky
>
>

> On Thu, May 23, 2013 at 8:27 AM, <*yabe...@ca.ibm.com*<yabe...@ca.ibm.com>>

> wrote:
> Hi,
>
> We are using Samba 3.6.x built by ourself on AIX with IBM XL C/C++
> compiler, with the autoconf build system and it's working without issue.
> We use Samba mainly for its file-server feature to share DFS, GPFS and
> JFS2 filesystems to Windows systems and to Linux workstations.
> We will be looking in the upcoming months/years (before you stop providing
>
> security fixes for 3.6) to upgrade to Samba 4.x (file-server only) so we
> hope to be able to build it in our environment.
>
> We currently don't have Python on these systems as nothing requires it.
> There is no IBM supported python package for AIX.

> There is some sites providing Python packages (such as*
> **http://www-frec.bull.com/* <http://www-frec.bull.com/> and *
> http://www.perzl.org/aix/* <http://www.perzl.org/aix/>) but on a best

> effort basis and not always at the latest release.
> Being able to be on a supported version (or at least, an up-to-date
> version for security fixes) is mandatory for us.
> Therefore, the most logical solution remaining is to build Python ourself
> and maintain it.
>
> I believe waf requires Python 2.x so I've tried to compile Python 2.7.5 on
>
> AIX with IBM XL C/C++.
> "configure" and "make" worked well.
> make test is throwing some error and wasn't able to complete
> test_hotshot
> make: 1254-059 The signal code from the last command is 11.
> Stop.
> I've did the make install anyway, tested it with a simple python script
> and it worked.
>

> I've downloaded Samba 4.0.6 and tried it with waf following this wiki:*
> **https://wiki.samba.org/index.php/Waf#Using_waf_directly*<https://wiki.samba.org/index.php/Waf#Using_waf_directly>

> 450 534-7711*
> **yabe...@ca.ibm.com* <yabe...@ca.ibm.com>

[C.J. Adams-Collier KF7BMP]

Oh, god. Oh, god. This is *so* not a good idea people. REVERT REVERT.

[Ricky Nance]

What? Why is it a bad idea on a system without python?

Ricky

[Andrew Bartlett]

On Thu, 2013-05-23 at 09:33 -0500, Ricky Nance wrote:
> If you get too new of a version of python (3.0+), it can break thinks,
> this version is known to work with samba. I'd be willing to bet that
> 2.7.5 will cause things to break, but I haven't tested it myself (I
> want to say 2.7.3 worked, so I very well could be wrong about this).

Actually, 2.7.5 should be fine, I would be very supprised if it isn't.

We *do* actually need to update the version, as recent
Solaris/OpenIndiana versions break with the install_with_python.sh,
because they changed the way some network interface header worked, which
python compensated for in a more recent version.

What I've just not done is tested it, then uploaded the tarball to our
rsync server, then update the script. It just needs to be done, and on
the platforms where this is an issue, python and the development headers
are a native package. (Just make sure you install the math library
headers, which are missing from python as a package dep...).

Andrew Bartlett


--
Andrew Bartlett http://samba.org/~abartlet/
Authentication Developer, Samba Team http://samba.org

[Andrew Bartlett]

On Thu, 2013-05-23 at 10:15 -0400, yabe...@ca.ibm.com wrote:
> nevermind, install_with_python.sh is part of samba-4.0.6.tar.gz
>
> However, I'm not sure I like it.
> It tries to use rsync (not available on our system but can be
> installed) to download this file:
> ftp://ftp.samba.org/pub/tridge/python/Python-2.6.5.tar
> It doesn't look like something recent (2.6.5 was released in 2010,
> 2.6.8 and 2.7.5 are available)
>
> I believe I would prefer to build and maintain it myself.

Most administrators who see the script actually do that, using the
script as documentation of a known working process. Many of the sites
that need this (don't have python on their native unix) wisely do not
allow direct network access between their ancient Unix and the world.

However, these machines are most often tended by experienced Unix
administrators, for whom following the simple steps in this script, or
providing the unzipped tarball in the right location manually is
trivial.

Having an automated script allows us to test that this process works,
and that is incredibly valuable, as well as versatile to administrators
who are just stuck, and don't mind the incoming rsync.

[Jelmer Vernooij]

On Thu, May 23, 2013 at 09:33:56AM -0500, Ricky Nance wrote:
> If you get too new of a version of python (3.0+), it can break thinks, this
> version is known to work with samba. I'd be willing to bet that 2.7.5 will
> cause things to break, but I haven't tested it myself (I want to say 2.7.3
> worked, so I very well could be wrong about this).

On the contrary, I'd be very surprised if 2.7.5 broke things. Python is
conservative with bug fixes that go into minor releases. If you do hit
regressions like that please do tell us about them.

Python 3.0 makes a number of backward incompatible changes - intentionally,
and documented as such - and Samba 4 doesn't support it yet. Any versions
of Python 2 after 2.5 (which is ~5 years old at this point) should work with Samba 4.

Cheers,

Jelmer

> On Thu, May 23, 2013 at 9:15 AM, <yabe...@ca.ibm.com> wrote:
>
> > nevermind, install_with_python.sh is part of samba-4.0.6.tar.gz
> >
> > However, I'm not sure I like it.
> > It tries to use rsync (not available on our system but can be installed)
> > to download this file:
> > ftp://ftp.samba.org/pub/tridge/python/Python-2.6.5.tar
> > It doesn't look like something recent (2.6.5 was released in 2010, 2.6.8
> > and 2.7.5 are available)
> >
> > I believe I would prefer to build and maintain it myself.
> >
> >

> > Best regards,
> >
> > Yannick Bergeron
> > 450 534-7711
> > yabe...@ca.ibm.com
> > Advisory IT Specialist
> >
> > Never say never, say "it depends" / Ne jamais dire jamais, dites "ca

> > d�pend"

> > ----- Forwarded by Yannick Y Bergeron/Bromont/IBM on 05/23/2013 10:12 AM
> > -----
> >
> > From: Yannick Y Bergeron/Bromont/IBM
> > To: Ricky Nance <ricky...@gmail.com>,
> > Cc: Andrew Bartlett <abar...@samba.org>, Samba Technical <
> > samba-t...@lists.samba.org>
> > Date: 05/23/2013 10:08 AM
> > Subject: Re: [PROPOSAL] To retire autoconf for 4.1
> > ------------------------------
> >
> >
> > Can't find it in the samba-4.0.6.tar.gz
> >
> > It seems to be in master with the name "install_with_python.sh"
> > *this script installs a private copy of python in the same prefix as Samba
> > *
> >
> > This would definitely looks like an interesting alternative if the Samba
> > team keep this version up-to-date with security issues.
> > If I can find a few minutes, I might attempt to fetch the master tree and
> > see if I can build it with it
> >
> >
> > Best regards,
> >
> > Yannick Bergeron
> > 450 534-7711
> > yabe...@ca.ibm.com
> > Advisory IT Specialist
> >
> > Never say never, say "it depends" / Ne jamais dire jamais, dites "ca

> > d�pend"

> > d�pend"
> >
> >

--

[C.J. Adams-Collier KF7BMP]

you're throwing away a build system that has been dominant for longer
than you have been alive. You should re-consider the impact of your
decision. How well do you know autoconf? You should know what you're
getting rid of before you do so, so that you can replace all of the bits
and pieces that you did not realize you were getting rid of.

On Thu, 2013-05-23 at 16:18 -0500, Ricky Nance wrote:
> What? Why is it a bad idea on a system without python?
>
>

> Ricky
>
>

> On Thu, May 23, 2013 at 3:33 PM, C.J. Adams-Collier KF7BMP
> <cj...@colliertech.org> wrote:
> Oh, god. Oh, god. This is *so* not a good idea people.
> REVERT REVERT.
>

> On Thu, 2013-05-23 at 09:33 -0500, Ricky Nance wrote:
> > If you get too new of a version of python (3.0+), it can
> break thinks, this
> > version is known to work with samba. I'd be willing to bet
> that 2.7.5 will
> > cause things to break, but I haven't tested it myself (I
> want to say 2.7.3
> > worked, so I very well could be wrong about this).
> >

> > Good luck,
> > Ricky
> >
> >

> > On Thu, May 23, 2013 at 9:15 AM, <yabe...@ca.ibm.com>
> wrote:
> >
> > > nevermind, install_with_python.sh is part of
> samba-4.0.6.tar.gz
> > >
> > > However, I'm not sure I like it.
> > > It tries to use rsync (not available on our system but can
> be installed)
> > > to download this file:
> > > ftp://ftp.samba.org/pub/tridge/python/Python-2.6.5.tar
> > > It doesn't look like something recent (2.6.5 was released
> in 2010, 2.6.8
> > > and 2.7.5 are available)
> > >
> > > I believe I would prefer to build and maintain it myself.
> > >
> > >
> > > Best regards,
> > >
> > > Yannick Bergeron
> > > 450 534-7711
> > > yabe...@ca.ibm.com
> > > Advisory IT Specialist
> > >
> > > Never say never, say "it depends" / Ne jamais dire jamais,
> dites "ca

> > > dépend"

> > > dépend"

> > > dépend"
> > >
> > >
>
>
>
>

[Jelmer Vernooij]

On Thu, May 23, 2013 at 04:25:46PM -0700, C.J. Adams-Collier KF7BMP wrote:
> you're throwing away a build system that has been dominant for longer
> than you have been alive. You should re-consider the impact of your
> decision. How well do you know autoconf? You should know what you're
> getting rid of before you do so, so that you can replace all of the bits
> and pieces that you did not realize you were getting rid of.

It's not like we're just starting to consider this. waf has been used to
build Samba 4 with success for quite some time now. Even what we used before
was quite different from the traditional autoconf + automake + libtool + make
combo.

Either way, I don't see how the age of the technology matters other than
as a rough indicator of its maturity. The technologies the old and the new
build system were built on (Python and autoconf) both date from '91.

The waf-based build system has a number of advantages over the autoconf+make
based system. If there are regressions or concrete concerns, by all means,
please bring them up and we can discuss them.

Cheers,

Jelmer

> > > > d�pend"

> > > > d�pend"

> > > > d�pend"
> > > >
> > > >
> >
> >
> >
> >
>



--

[C.J. Adams-Collier KF7BMP]

On Fri, 2013-05-24 at 02:07 +0200, Jelmer Vernooij wrote:
> If there are regressions or concrete concerns, by all means,
> please bring them up and we can discuss them.
>
> Cheers,
>
> Jelmer

Okay. You've convinced me. But I reserve the right to say "I told you
so" ;-)

C.J.

[Michael Adam]

As Jelmer pointed out, the fundamental discussion you are
starting here has been (or should have been) taken place
several years ago.

We have been building the AD-Part (samba4) of Samba with waf for more
than three years now (iirc), the combined build followed soon.
And before introducing waf, the samba4 part was also not built by
a (pure) autoconf system for a considerable time.

What we are talking about now is the question whether to remove
the autoconf build system from the source3 subdirectory,
which is also built with the top level waf build, which is
declared default build in in Samba 4.0.

So maybe you should have told us several years ago. :-)

Cheers - Michael

[Andrew Bartlett]

Thanks Michael,

That describes where we are at very well.

[yabe...@ca.ibm.com]

A few points I would however consider:
When does Python 2.x will goes end of life? From there wiki: "Python 2.x
is the status quo, The final 2.x version 2.7 release is out, with a
statement of extended support for this end-of-life release. "
Which unix platform comes with Python 2.x instead of Python 3.x?
Is it possible to have a waf version working with both Python version?
If so, how much effort would it required?
If not, when do you plan to move waf from Python 2.x to 3.x?

Best regards,

Yannick Bergeron
450 534-7711
yabe...@ca.ibm.com
Advisory IT Specialist

Never say never, say "it depends" / Ne jamais dire jamais, dites "ca

dépend"

From: Jelmer Vernooij <jel...@samba.org>
To: Ricky Nance <ricky...@gmail.com>,
Cc: yabe...@ca.ibm.com, Samba Technical
<samba-t...@lists.samba.org>, Andrew Bartlett <abar...@samba.org>

Date: 05/23/2013 07:41 PM
Subject: Re: Fw: [PROPOSAL] To retire autoconf for 4.1

On Thu, May 23, 2013 at 09:33:56AM -0500, Ricky Nance wrote:
> If you get too new of a version of python (3.0+), it can break thinks,
this
> version is known to work with samba. I'd be willing to bet that 2.7.5
will
> cause things to break, but I haven't tested it myself (I want to say
2.7.3
> worked, so I very well could be wrong about this).

On the contrary, I'd be very surprised if 2.7.5 broke things. Python is
conservative with bug fixes that go into minor releases. If you do hit
regressions like that please do tell us about them.

Python 3.0 makes a number of backward incompatible changes -
intentionally,
and documented as such - and Samba 4 doesn't support it yet. Any versions
of Python 2 after 2.5 (which is ~5 years old at this point) should work
with Samba 4.

Cheers,

Jelmer

> On Thu, May 23, 2013 at 9:15 AM, <yabe...@ca.ibm.com> wrote:
>
> > nevermind, install_with_python.sh is part of samba-4.0.6.tar.gz
> >
> > However, I'm not sure I like it.
> > It tries to use rsync (not available on our system but can be
installed)
> > to download this file:
> > ftp://ftp.samba.org/pub/tridge/python/Python-2.6.5.tar
> > It doesn't look like something recent (2.6.5 was released in 2010,
2.6.8
> > and 2.7.5 are available)
> >
> > I believe I would prefer to build and maintain it myself.
> >
> >
> > Best regards,
> >
> > Yannick Bergeron
> > 450 534-7711
> > yabe...@ca.ibm.com
> > Advisory IT Specialist
> >
> > Never say never, say "it depends" / Ne jamais dire jamais, dites "ca

> > dépend"

> > dépend"

> > dépend"
> >
> >

--

[Jelmer Vernooij]

On Fri, May 24, 2013 at 07:34:22AM -0400, yabe...@ca.ibm.com wrote:
> A few points I would however consider:
> When does Python 2.x will goes end of life? From there wiki: "Python 2.x
> is the status quo, The final 2.x version 2.7 release is out, with a
> statement of extended support for this end-of-life release. "

Where on the wiki is that?

> Which unix platform comes with Python 2.x instead of Python 3.x?

Python2.x is still the default at this point. Those platforms that do ship
Python ship least python 2.x - although a few are looking now at what it would
take to migrate to 3.x.

> Is it possible to have a waf version working with both Python version?
> If so, how much effort would it required?

That would be possible - upstream waf supports this - but supporting two
major versions of Python is a lot of effort and a pain. We just
need to decide at some point - when Python3 is widely enough available, and
distributions start dropping support for 2.x - to migrate. I think
we're at least a year away from that point.

> If not, when do you plan to move waf from Python 2.x to 3.x?

We would move waf over at the same time we would move the rest of Samba from
Python 2.x to 3.x. There is no point in requiring users to install Python 2.x
for Samba itself *and* 3.x for waf or vice versa.

Cheers,

Jelmer

> > > d?pend"

> > > d?pend"

> > > d?pend"
> > >
> > >
>
> --
>
>

--

[yabe...@ca.ibm.com]

http://wiki.python.org/moin/Python2orPython3

I believe you're right, ie: when most major distributions will provide a
Python 3.x package in their repositories, Samba team should start working
on moving from Python 2.x to 3.x for Samba and Waf.
RHEL 6.4 is on 2.6.6
Debian 7 is on 2.7.3 but also have a package for Python 3.2.3

Best regards,

Yannick Bergeron
450 534-7711
yabe...@ca.ibm.com
Advisory IT Specialist

Never say never, say "it depends" / Ne jamais dire jamais, dites "ca

dépend"

[Simo]

On 05/24/2013 07:58 AM, yabe...@ca.ibm.com wrote:
> http://wiki.python.org/moin/Python2orPython3
>
> I believe you're right, ie: when most major distributions will provide a
> Python 3.x package in their repositories, Samba team should start working
> on moving from Python 2.x to 3.x for Samba and Waf.
> RHEL 6.4 is on 2.6.6
> Debian 7 is on 2.7.3 but also have a package for Python 3.2.3

At SambaXP I and Alexander started raising a concern about this.
Fedora is starting to plan to move to Python 3, so we need to start
thinking about moving samba as well.

Unfortunately we cannot just make a full switch. Because there are
distributions that will stay on Python 2.x for a long time, much longer
than Fedora's support for Python 2.x presumably.

So we should really look into what it will take to try to support both
2.x and 3.x especially for generated bindings as the binding interface,
I am told, changes quite some fundamental things.


A flag day where we switch fro 2 to 3 is highly unfeasible unless we
also decide to drop support for all Enterprise Linux distributions and
all other long term maintenance Unix flavors at the same time. I do not
think that would be a wise choice.


Simo.

[Jelmer Vernooij]

On Fri, May 24, 2013 at 08:30:10AM -0400, Simo wrote:
> On 05/24/2013 07:58 AM, yabe...@ca.ibm.com wrote:
> >http://wiki.python.org/moin/Python2orPython3
> >
> >I believe you're right, ie: when most major distributions will provide a
> >Python 3.x package in their repositories, Samba team should start working
> >on moving from Python 2.x to 3.x for Samba and Waf.
> >RHEL 6.4 is on 2.6.6
> >Debian 7 is on 2.7.3 but also have a package for Python 3.2.3
>
> At SambaXP I and Alexander started raising a concern about this.
> Fedora is starting to plan to move to Python 3, so we need to start
> thinking about moving samba as well.

When will Fedora drop support for Python 2.x? Just having the default changed
shouldn't be a problem, so long as Python2.x is still installable.

> Unfortunately we cannot just make a full switch. Because there are
> distributions that will stay on Python 2.x for a long time, much
> longer than Fedora's support for Python 2.x presumably.
>
> So we should really look into what it will take to try to support
> both 2.x and 3.x especially for generated bindings as the binding
> interface, I am told, changes quite some fundamental things.
>
> A flag day where we switch fro 2 to 3 is highly unfeasible unless we
> also decide to drop support for all Enterprise Linux distributions
> and all other long term maintenance Unix flavors at the same time. I
> do not think that would be a wise choice.

I've tried to do support for both python2 and python3 with a few projects. It
requires ugly hacks that make the code less readable, is a major pain to keep
up and prone to regressions even for smaller projects. It would be
a nightmare for a project the size of Samba.

What are the long term releases we should be worried about a year from now that
don't support Python3 yet? Debian oldstable and stable both have Python3, and
all LTS releases of Ubuntu that are still supported also have a version of
Python3.

Cheers,

Jelmer

[Volker Lendecke]

This means that Python 3 is just a different language. If
Fedora drops support for Python 2 it is almost the same as
if they drop support for C/C++ given the enormous popularity
of Python 2.

If Fedora drops support for Python 2, why should we bother?
If Fedora drops support for C, Samba would not switch to Ada
or Haskell, right? We would just say that Fedora should
really wake up and continue supporting us.

Yes, I know this is vastly exagerrated, but if Fedora wants
to drop supporting our infrastructure, it is up to them to
provide patches to upstream, right?

Volker

P.S: By the way, with configure/make we would not have this
discussion, right?

--
SerNet GmbH, Bahnhofsallee 1b, 37081 Göttingen
phone: +49-551-370000-0, fax: +49-551-370000-9
AG Göttingen, HRB 2816, GF: Dr. Johannes Loxen
http://www.sernet.de, mailto:kon...@sernet.de

[Jelmer Vernooij]

It is to some degree a different language. Python 3 is better in a lot of ways, but
I'm not a fan of the way the transition from 2.x to 3.0 was managed - and it shows
in the number of packages that have migrated to 3.0. Unfortunately there's not
much we can do about that now.

Python2.7 makes life slightly simpler if you want to support 2.x and 3.x in the
same application by supporting a number of features that were introduced in
Python3. However, 2.7 was released around the same time as 3.0. So if we can
afford to drop support for Python < 2.7 we might as well switch to 3.0.

> If Fedora drops support for Python 2, why should we bother?
> If Fedora drops support for C, Samba would not switch to Ada
> or Haskell, right? We would just say that Fedora should
> really wake up and continue supporting us.

> Yes, I know this is vastly exagerrated, but if Fedora wants
> to drop supporting our infrastructure, it is up to them to
> provide patches to upstream, right?

> P.S: By the way, with configure/make we would not have this
> discussion, right?

If Fedora were to consider dropping Python2.x support while there was still
a large number of packages relying on it, I don't see why they wouldn't
consider to do the same with m4 or autoconf. Neither seems realistic.

I'm assuming Simo is participating in this discussion because Fedora cares
about migrating upstreams to Python3 before dropping Python2.

Cheers,

Jelmer

[Simo]

On 05/24/2013 09:07 AM, Jelmer Vernooij wrote:
> On Fri, May 24, 2013 at 08:30:10AM -0400, Simo wrote:
>> On 05/24/2013 07:58 AM, yabe...@ca.ibm.com wrote:
>>> http://wiki.python.org/moin/Python2orPython3
>>>
>>> I believe you're right, ie: when most major distributions will provide a
>>> Python 3.x package in their repositories, Samba team should start working
>>> on moving from Python 2.x to 3.x for Samba and Waf.
>>> RHEL 6.4 is on 2.6.6
>>> Debian 7 is on 2.7.3 but also have a package for Python 3.2.3
>> At SambaXP I and Alexander started raising a concern about this.
>> Fedora is starting to plan to move to Python 3, so we need to start
>> thinking about moving samba as well.
> When will Fedora drop support for Python 2.x? Just having the default changed
> shouldn't be a problem, so long as Python2.x is still installable.

It is not that simple. Samba now is in the business of providing
bindings, that mean dependencies for other applications. It is very
likely we'll have to build bindings for both 2.x and 3.x at the same
time and make them parallel installable for a while. Otherwise it
becomes a multipackage flag day where all of samba and all dependent
packages need to switch at once.

>> Unfortunately we cannot just make a full switch. Because there are
>> distributions that will stay on Python 2.x for a long time, much
>> longer than Fedora's support for Python 2.x presumably.
>>
>> So we should really look into what it will take to try to support
>> both 2.x and 3.x especially for generated bindings as the binding
>> interface, I am told, changes quite some fundamental things.
>>
>> A flag day where we switch fro 2 to 3 is highly unfeasible unless we
>> also decide to drop support for all Enterprise Linux distributions
>> and all other long term maintenance Unix flavors at the same time. I
>> do not think that would be a wise choice.
> I've tried to do support for both python2 and python3 with a few projects. It
> requires ugly hacks that make the code less readable, is a major pain to keep
> up and prone to regressions even for smaller projects. It would be
> a nightmare for a project the size of Samba.

Blame python developers, 2 -> 3 *will* be nasty and hard for a lot of
projects, we are not alone.

> What are the long term releases we should be worried about a year from now that
> don't support Python3 yet? Debian oldstable and stable both have Python3, and
> all LTS releases of Ubuntu that are still supported also have a version of
> Python3.

No RHEL so far has Python 3, but see above, even if RHEL has python 3 it
wouldn't make a big difference we would still wan to build with python 2
due to the fact the samba bindings are now dependencies for other projects.

Simo.

[Ricky Nance]

Andrew, it might not hurt to have a check, look in /tmp for the tarball,
then use rsync (if available), if not, try wget, if not try XXX (not sure
what solaris, freebsd, or any others have by default), if those fail, then
error and tell the user to download the file manually to (say /tmp?) .

Just a thought,
Ricky


On Thu, May 23, 2013 at 5:08 PM, Andrew Bartlett <abar...@samba.org> wrote:

> On Thu, 2013-05-23 at 09:33 -0500, Ricky Nance wrote:
> > If you get too new of a version of python (3.0+), it can break thinks,
> > this version is known to work with samba. I'd be willing to bet that
> > 2.7.5 will cause things to break, but I haven't tested it myself (I
> > want to say 2.7.3 worked, so I very well could be wrong about this).
>

> Actually, 2.7.5 should be fine, I would be very supprised if it isn't.
>
> We *do* actually need to update the version, as recent
> Solaris/OpenIndiana versions break with the install_with_python.sh,
> because they changed the way some network interface header worked, which
> python compensated for in a more recent version.
>
> What I've just not done is tested it, then uploaded the tarball to our
> rsync server, then update the script. It just needs to be done, and on
> the platforms where this is an issue, python and the development headers
> are a native package. (Just make sure you install the math library
> headers, which are missing from python as a package dep...).
>

[Simo]

On 05/24/2013 09:29 AM, Volker Lendecke wrote:

> On Fri, May 24, 2013 at 03:07:33PM +0200, Jelmer Vernooij wrote:
>> On Fri, May 24, 2013 at 08:30:10AM -0400, Simo wrote:
>>> On 05/24/2013 07:58 AM, yabe...@ca.ibm.com wrote:
>>>> http://wiki.python.org/moin/Python2orPython3
>>>>
>>>> I believe you're right, ie: when most major distributions will provide a
>>>> Python 3.x package in their repositories, Samba team should start working
>>>> on moving from Python 2.x to 3.x for Samba and Waf.
>>>> RHEL 6.4 is on 2.6.6
>>>> Debian 7 is on 2.7.3 but also have a package for Python 3.2.3
>>> At SambaXP I and Alexander started raising a concern about this.
>>> Fedora is starting to plan to move to Python 3, so we need to start
>>> thinking about moving samba as well.
>> When will Fedora drop support for Python 2.x? Just having the default changed
>> shouldn't be a problem, so long as Python2.x is still installable.
>>

>>> Unfortunately we cannot just make a full switch. Because there are
>>> distributions that will stay on Python 2.x for a long time, much
>>> longer than Fedora's support for Python 2.x presumably.
>>>
>>> So we should really look into what it will take to try to support
>>> both 2.x and 3.x especially for generated bindings as the binding
>>> interface, I am told, changes quite some fundamental things.
>>>
>>> A flag day where we switch fro 2 to 3 is highly unfeasible unless we
>>> also decide to drop support for all Enterprise Linux distributions
>>> and all other long term maintenance Unix flavors at the same time. I
>>> do not think that would be a wise choice.
>> I've tried to do support for both python2 and python3 with a few projects. It
>> requires ugly hacks that make the code less readable, is a major pain to keep
>> up and prone to regressions even for smaller projects. It would be
>> a nightmare for a project the size of Samba.

> This means that Python 3 is just a different language. If
> Fedora drops support for Python 2 it is almost the same as
> if they drop support for C/C++ given the enormous popularity
> of Python 2.

Fedora is not going to drop Python 2 in the short term, but that is not
the issue.
See my other email.
The dependency chain is the issue.

> If Fedora drops support for Python 2, why should we bother?
> If Fedora drops support for C, Samba would not switch to Ada
> or Haskell, right? We would just say that Fedora should
> really wake up and continue supporting us.

I am warning you because I want to keep supporting samba and its python
bindings.

>
> Yes, I know this is vastly exagerrated, but if Fedora wants
> to drop supporting our infrastructure, it is up to them to
> provide patches to upstream, right?

Wrong, we are not going to drop Python 2, but all distributions will
have to go through a transition and we have dependencies now. If we
can't support building bindings for both 2.x and 3.x we will screw
things up.

> Volker

>
> P.S: By the way, with configure/make we would not have this
> discussion, right?
>

Wrong, we can have a flag day for WAF, it doesn't matter as it is
limited to building the code and has no ulterior dependency issues.
The problem is with the bindings we offer now.

Simo.

[Volker Lendecke]

On Fri, May 24, 2013 at 09:58:56AM -0400, Simo wrote:
> Blame python developers, 2 -> 3 *will* be nasty and hard for a lot
> of projects, we are not alone.

I don't think it is reasonable to ask an upstream project
for PHP bindings just because they support Python 2
bindings. Python 3 is a different language that only has a
superficial similarity with Python 2. We don't support PHP,
so why should we support Python 3?

Volker

[Volker Lendecke]

On Fri, May 24, 2013 at 10:08:56AM -0400, Simo wrote:
> >Yes, I know this is vastly exagerrated, but if Fedora wants
> >to drop supporting our infrastructure, it is up to them to
> >provide patches to upstream, right?
>
> Wrong, we are not going to drop Python 2, but all distributions will
> have to go through a transition and we have dependencies now. If we
> can't support building bindings for both 2.x and 3.x we will screw
> things up.

Wrong. The people who drop support for Python 2 screw things
up. Everything else is just an illusion. I don't know why
everybody wants to switch to a different language just
because it shares the same name.

[Jelmer Vernooij]

On Fri, May 24, 2013 at 09:58:56AM -0400, Simo wrote:

> On 05/24/2013 09:07 AM, Jelmer Vernooij wrote:

> >On Fri, May 24, 2013 at 08:30:10AM -0400, Simo wrote:
> >>On 05/24/2013 07:58 AM, yabe...@ca.ibm.com wrote:
> >>>http://wiki.python.org/moin/Python2orPython3
> >>>
> >>>I believe you're right, ie: when most major distributions will provide a
> >>>Python 3.x package in their repositories, Samba team should start working
> >>>on moving from Python 2.x to 3.x for Samba and Waf.
> >>>RHEL 6.4 is on 2.6.6
> >>>Debian 7 is on 2.7.3 but also have a package for Python 3.2.3
> >>At SambaXP I and Alexander started raising a concern about this.
> >>Fedora is starting to plan to move to Python 3, so we need to start
> >>thinking about moving samba as well.
> >When will Fedora drop support for Python 2.x? Just having the default changed
> >shouldn't be a problem, so long as Python2.x is still installable.

> It is not that simple. Samba now is in the business of providing
> bindings, that mean dependencies for other applications. It is very
> likely we'll have to build bindings for both 2.x and 3.x at the same
> time and make them parallel installable for a while. Otherwise it
> becomes a multipackage flag day where all of samba and all dependent
> packages need to switch at once.

Are there particular dependent packages you are referring to here?
What exactly do we promise external users can rely on? Just tdb and ldb?

We need to clarify what exactly external users can expect from our APIs
and changes to them. In 3.x the answer to this was just that everything in
"libsmbclient.h" was public, but with Python APIs in Samba 4 it's more
complicated.

> >>Unfortunately we cannot just make a full switch. Because there are
> >>distributions that will stay on Python 2.x for a long time, much
> >>longer than Fedora's support for Python 2.x presumably.
> >>
> >>So we should really look into what it will take to try to support
> >>both 2.x and 3.x especially for generated bindings as the binding
> >>interface, I am told, changes quite some fundamental things.
> >>
> >>A flag day where we switch fro 2 to 3 is highly unfeasible unless we
> >>also decide to drop support for all Enterprise Linux distributions
> >>and all other long term maintenance Unix flavors at the same time. I
> >>do not think that would be a wise choice.
> >I've tried to do support for both python2 and python3 with a few projects. It
> >requires ugly hacks that make the code less readable, is a major pain to keep
> >up and prone to regressions even for smaller projects. It would be
> >a nightmare for a project the size of Samba.

> Blame python developers, 2 -> 3 *will* be nasty and hard for a lot
> of projects, we are not alone.

I agree, see my other email. Python upstream has done a bad job
of managing this transition, and it shows in the number of packages
that have migrated so far. There is a chicken-egg problem; distributions won't
drop python2 until most packages have migrated to Python3, but packages won't
migrate while the rest of the ecosystem is still on Python 2.

There is no urgency for Samba to migrate so long as Python 2 is widely
available. It would still be nice to - a less fragmented ecosystem
is good for everybody, and there are some nice features in Python 3 -
but we need to consider the associated costs as well.

> >What are the long term releases we should be worried about a year from now that
> >don't support Python3 yet? Debian oldstable and stable both have Python3, and
> >all LTS releases of Ubuntu that are still supported also have a version of
> >Python3.
> No RHEL so far has Python 3, but see above, even if RHEL has python
> 3 it wouldn't make a big difference we would still wan to build with
> python 2 due to the fact the samba bindings are now dependencies for
> other projects.

Is that a burden that should be with the upstream Samba project?

Maintaining support for both versions takes (a significant chunk of) developer
resources that could be used for other things.

Cheers,

Jelmer

[yabe...@ca.ibm.com]

XXX could be a ftp client with a response file

Yannick Bergeron
450 534-7711
yabe...@ca.ibm.com
Advisory IT Specialist

Never say never, say "it depends" / Ne jamais dire jamais, dites "ca
dépend"

From: Ricky Nance <ricky...@gmail.com>
To: Andrew Bartlett <abar...@samba.org>,
Cc: yabe...@ca.ibm.com, Samba Technical
<samba-t...@lists.samba.org>

[Volker Lendecke]

On Fri, May 24, 2013 at 03:55:24PM +0200, Jelmer Vernooij wrote:
> It is to some degree a different language. Python 3 is better in a lot of ways, but
> I'm not a fan of the way the transition from 2.x to 3.0 was managed - and it shows
> in the number of packages that have migrated to 3.0. Unfortunately there's not
> much we can do about that now.

For that exact reason I don't believe that Python 2 will die
in any foreseeable time frame. Eventually someone might step
up and do a separate and parallel Python 3 implementation of
the bindings, but that should not affect the existing Samba
binding code. As I said -- it's the same as if someone
started to do bindings for Google go, Ruby or PHP.

Do you see the pattern here? This whole thread started
because the burden to support 2 build systems seems too high
for some developers. By choosing Python and not some fully
embeddable, shippable language like Lua or ECMAscript we
deliberately and knowingly chose to accept the very same
problem again.

> If Fedora were to consider dropping Python2.x support while there was still
> a large number of packages relying on it, I don't see why they wouldn't
> consider to do the same with m4 or autoconf. Neither seems realistic.

So -- there's no point in worrying about Python 3.

[Simo]

We use the python binding for dcerpc and related stuff in FreeIPA for
example.
I'll let Alexander give a list, as he was the one that did the research
and started using the bindings.

There is no urgency indeed, but we should start making plans on how to
address the migration.
It will happen at some point and because handling both Python 2 and
Python 3 at the same time is painful if we ever reach critical mass on
Python 3 then packages may rapidly start dropping v2 support and force
fast paced distributions hands into trying to move wholesale to Python 3.

We do not want to have to rush then.

>>> What are the long term releases we should be worried about a year from now that
>>> don't support Python3 yet? Debian oldstable and stable both have Python3, and
>>> all LTS releases of Ubuntu that are still supported also have a version of
>>> Python3.
>> No RHEL so far has Python 3, but see above, even if RHEL has python
>> 3 it wouldn't make a big difference we would still wan to build with
>> python 2 due to the fact the samba bindings are now dependencies for
>> other projects.
> Is that a burden that should be with the upstream Samba project?
>
> Maintaining support for both versions takes (a significant chunk of) developer
> resources that could be used for other things.
>

True, so my question is, given most of the bindings are partially
autogenerated, is there a way we can reduce the developer pain by using
some abstraction perhaps ?

Simo.

[David Collier-Brown]

>>> What are the long term releases we should be worried about a year from now that
>>> don't support Python3 yet? Debian oldstable and stable both have Python3, and
>>> all LTS releases of Ubuntu that are still supported also have a version of
>>> Python3.
>> No RHEL so far has Python 3, but see above, even if RHEL has python
>> 3 it wouldn't make a big difference we would still wan to build with
>> python 2 due to the fact the samba bindings are now dependencies for
>> other projects.
> Is that a burden that should be with the upstream Samba project?
>
> Maintaining support for both versions takes (a significant chunk of) developer
> resources that could be used for other things.
>

> Cheers,
>
> Jelmer
>
I can't help with maintaining two versions, but I have a toolset for
human-mediated automatic API changing, described briefly at
http://www.datacenterworks.com/stories/port.html

If the differences are well-known or perhaps even documented (:-)) we
can create a port database for them.

--dave

--
David Collier-Brown, | Always do right. This will gratify
System Programmer and Author | some people and astonish the rest
dav...@spamcop.net | -- Mark Twain
(416) 223-8968

[Jelmer Vernooij]

On Fri, May 24, 2013 at 05:40:14PM +0200, Volker Lendecke wrote:
> On Fri, May 24, 2013 at 03:55:24PM +0200, Jelmer Vernooij wrote:
> > It is to some degree a different language. Python 3 is better in a lot of ways, but
> > I'm not a fan of the way the transition from 2.x to 3.0 was managed - and it shows
> > in the number of packages that have migrated to 3.0. Unfortunately there's not
> > much we can do about that now.
>
> For that exact reason I don't believe that Python 2 will die
> in any foreseeable time frame. Eventually someone might step
> up and do a separate and parallel Python 3 implementation of
> the bindings, but that should not affect the existing Samba
> binding code. As I said -- it's the same as if someone
> started to do bindings for Google go, Ruby or PHP.

There are some big differences but they are still fundamentally the same
language. Migrating from Python 2 to Python 3 is nowhere as much work as
writing new bindings for another language.

> Do you see the pattern here? This whole thread started
> because the burden to support 2 build systems seems too high
> for some developers. By choosing Python and not some fully
> embeddable, shippable language like Lua or ECMAscript we
> deliberately and knowingly chose to accept the very same
> problem again.

I don't see how they are any different. Lua and ECMAscript are also evolving.
If we used Lua there would be people wanting to call Samba libraries from their
lua scripts, so we'd have to worry about supporting whatever version of lua
they are using. Nobody is forcing us to move away from Python 2.

> > If Fedora were to consider dropping Python2.x support while there was still
> > a large number of packages relying on it, I don't see why they wouldn't
> > consider to do the same with m4 or autoconf. Neither seems realistic.
> So -- there's no point in worrying about Python 3.

Unless we want to. It will probably make sense at some point.

Cheers,

Jelmer

[Jelmer Vernooij]

> We use the python binding for dcerpc and related stuff in FreeIPA
> for example.
> I'll let Alexander give a list, as he was the one that did the
> research and started using the bindings.

Assuming that's all you need, I suspect supporting those for both versions
is a lot more achievable than blanket support of all our Python code on
both 2.x and 3.x.

To manage expectations, I still think it would be good if we clarified what
we do and don't plan on doing with regard to the external use of various Python
APIs.

> >>>I've tried to do support for both python2 and python3 with a few projects. It
> >>>requires ugly hacks that make the code less readable, is a major pain to keep
> >>>up and prone to regressions even for smaller projects. It would be
> >>>a nightmare for a project the size of Samba.
> >>Blame python developers, 2 -> 3 *will* be nasty and hard for a lot
> >>of projects, we are not alone.
> >I agree, see my other email. Python upstream has done a bad job
> >of managing this transition, and it shows in the number of packages
> >that have migrated so far. There is a chicken-egg problem; distributions won't
> >drop python2 until most packages have migrated to Python3, but packages won't
> >migrate while the rest of the ecosystem is still on Python 2.
> >
> >There is no urgency for Samba to migrate so long as Python 2 is widely
> >available. It would still be nice to - a less fragmented ecosystem
> >is good for everybody, and there are some nice features in Python 3 -
> >but we need to consider the associated costs as well.
>

> There is no urgency indeed, but we should start making plans on how
> to address the migration.
> It will happen at some point and because handling both Python 2 and
> Python 3 at the same time is painful if we ever reach critical mass
> on Python 3 then packages may rapidly start dropping v2 support and
> force fast paced distributions hands into trying to move wholesale
> to Python 3.
>
> We do not want to have to rush then.

I agree. Being prepared is never a bad idea. :-)

> >>>What are the long term releases we should be worried about a year from now that
> >>>don't support Python3 yet? Debian oldstable and stable both have Python3, and
> >>>all LTS releases of Ubuntu that are still supported also have a version of
> >>>Python3.
> >>No RHEL so far has Python 3, but see above, even if RHEL has python
> >>3 it wouldn't make a big difference we would still wan to build with
> >>python 2 due to the fact the samba bindings are now dependencies for
> >>other projects.
> >Is that a burden that should be with the upstream Samba project?
> >
> >Maintaining support for both versions takes (a significant chunk of) developer
> >resources that could be used for other things.
> >
>

> True, so my question is, given most of the bindings are partially
> autogenerated, is there a way we can reduce the developer pain by
> using some abstraction perhaps ?

The DCE/RPC bindings are almost entirely generated. There are a lot of other
bindings written in C that are not, but it sounds like you guys don't actually
need those?

So in summary, if there is a small subset you need to be available for both
Python 2.x and 3.x then that is a lot more achievable than making everything
available for both - and considering the code you are using is probably more stable
and less prone to changes, there will probably also be less friction in keeping
it working with both.

Cheers,

Jelmer

[yabe...@ca.ibm.com]

If we go back to what triggered the discussion between Python 2 vs Python
3

http://wiki.python.org/moin/Python2orPython3

"Python 2.x is the status quo, The final 2.x version 2.7 release is out,
with a statement of extended support for this end-of-life release. "

Maybe you should ask the Python community when they expect to end the
extended support for Python 2.7.

As for the Linux distributions, I expect they might decide to provide
Python 3 instead Python 2 as their base Python package, but still provide
Python 2.7 as an optional package until the Python community stop
supporting it.
If they don't, they're not helping themself to keep their user base.


Best regards,

[Jelmer Vernooij]

On Sat, May 25, 2013 at 10:24:21AM -0400, yabe...@ca.ibm.com wrote:
> If we go back to what triggered the discussion between Python 2 vs Python
> 3
>
> http://wiki.python.org/moin/Python2orPython3
> "Python 2.x is the status quo, The final 2.x version 2.7 release is out,
> with a statement of extended support for this end-of-life release. "
>
> Maybe you should ask the Python community when they expect to end the
> extended support for Python 2.7.

See http://mail.python.org/pipermail/python-dev/2010-May/099971.html
It looks like there will be support for 2.7 at least until Mid 2016.

> As for the Linux distributions, I expect they might decide to provide
> Python 3 instead Python 2 as their base Python package, but still provide
> Python 2.7 as an optional package until the Python community stop
> supporting it.
> If they don't, they're not helping themself to keep their user base.

Yeah, I would expect the same - it's the default strategy for this kind of
transition. Ubuntu has plans to make Python3 the only Python on the desktop CD
for 14.04. https://wiki.ubuntu.com/Python/3

Cheers,

Jelmer

[Alexander Bokovoy]

On Fri, May 24, 2013 at 07:49:10PM +0200, Jelmer Vernooij wrote:
> > We use the python binding for dcerpc and related stuff in FreeIPA
> > for example.
> > I'll let Alexander give a list, as he was the one that did the
> > research and started using the bindings.
> Assuming that's all you need, I suspect supporting those for both versions
> is a lot more achievable than blanket support of all our Python code on
> both 2.x and 3.x.

Majority of what we use is indeed autogenerated. However, we rely on
security and param, and few others which are manually coded. So there is
still amount of work to port to any changed module API.

> To manage expectations, I still think it would be good if we clarified what
> we do and don't plan on doing with regard to the external use of various Python
> APIs.

I think actual issue is to find out what we use in our Python bindings
willingfully from Python module APIs and Python language features that
would change between 2 and 3. Strings are one part but most of other
APIs are hardly influencing complexity of maintaining the ports.
After all, we are not using CObject API so we don't need to migrate to
Capsule, and module initialization part is easy to rewrite.

Also, at Python level (as opposed to currently discussed Python C API) we
also don't need many of specific Python language features. We hardly are
beyond exceptions, iterators and dicts. Majority of new syntax is
backported as back as Python 2.6 which is what we rely on right now.

So I think the issue of maintaining support for both 2.6+ and 3.2+ is
rather question of having someone to come up and do actual work.

> > True, so my question is, given most of the bindings are partially
> > autogenerated, is there a way we can reduce the developer pain by
> > using some abstraction perhaps ?
> The DCE/RPC bindings are almost entirely generated. There are a lot of other
> bindings written in C that are not, but it sounds like you guys don't actually
> need those?

Some of those we need like smb.conf management and security.

> So in summary, if there is a small subset you need to be available for both
> Python 2.x and 3.x then that is a lot more achievable than making everything
> available for both - and considering the code you are using is probably more stable
> and less prone to changes, there will probably also be less friction in keeping
> it working with both.

I think what we need to proactively come up with is a research how to
perform bytes/strings management that is compliant to both Python 2.6+
and 3.2+. That should be a first step and it will show how large is the
overall task.

--
/ Alexander Bokovoy

[Michael Wood]

On 24 May 2013 16:04, Ricky Nance <ricky...@gmail.com> wrote:
>
> Andrew, it might not hurt to have a check, look in /tmp for the tarball,
> then use rsync (if available), if not, try wget, if not try XXX (not sure
> what solaris, freebsd, or any others have by default), if those fail, then
> error and tell the user to download the file manually to (say /tmp?) .

curl would be another one to try if you wanted to go in this
direction. On FreeBSD (and I think other BSDs too) there is "fetch".

> Just a thought,
> Ricky
>
>
> On Thu, May 23, 2013 at 5:08 PM, Andrew Bartlett <abar...@samba.org> wrote:
>
> > On Thu, 2013-05-23 at 09:33 -0500, Ricky Nance wrote:
> > > If you get too new of a version of python (3.0+), it can break thinks,
> > > this version is known to work with samba. I'd be willing to bet that
> > > 2.7.5 will cause things to break, but I haven't tested it myself (I
> > > want to say 2.7.3 worked, so I very well could be wrong about this).
> >
> > Actually, 2.7.5 should be fine, I would be very supprised if it isn't.
> >
> > We *do* actually need to update the version, as recent
> > Solaris/OpenIndiana versions break with the install_with_python.sh,
> > because they changed the way some network interface header worked, which
> > python compensated for in a more recent version.
> >
> > What I've just not done is tested it, then uploaded the tarball to our
> > rsync server, then update the script. It just needs to be done, and on
> > the platforms where this is an issue, python and the development headers
> > are a native package. (Just make sure you install the math library
> > headers, which are missing from python as a package dep...).
> >
> > Andrew Bartlett

--

Michael Wood <esio...@gmail.com>

[Andrew Bartlett]

On Sun, 2013-05-26 at 12:13 +0200, Michael Wood wrote:
> On 24 May 2013 16:04, Ricky Nance <ricky...@gmail.com> wrote:
> >
> > Andrew, it might not hurt to have a check, look in /tmp for the tarball,
> > then use rsync (if available), if not, try wget, if not try XXX (not sure
> > what solaris, freebsd, or any others have by default), if those fail, then
> > error and tell the user to download the file manually to (say /tmp?) .
>
> curl would be another one to try if you wanted to go in this
> direction. On FreeBSD (and I think other BSDs too) there is "fetch".

Honestly, so far there has been no demand for any of the above, but
tested patches are very welcome. To start with, we should at least just
check for a gzipped tarball, not just the uncompressed one, as folks on
these older systems seem quite likely to grab it with with web browser
on their PC and upload it to the restricted server, than have the server
even be able to talk to the 'net.

As I say, tested patches are very welcome.

[Simo]

On 05/26/2013 06:13 AM, Michael Wood wrote:
> On 24 May 2013 16:04, Ricky Nance <ricky...@gmail.com> wrote:
>> Andrew, it might not hurt to have a check, look in /tmp for the tarball,
>> then use rsync (if available), if not, try wget, if not try XXX (not sure
>> what solaris, freebsd, or any others have by default), if those fail, then
>> error and tell the user to download the file manually to (say /tmp?) .
> curl would be another one to try if you wanted to go in this
> direction. On FreeBSD (and I think other BSDs too) there is "fetch".

Well, whatever you do, make sure you generate and check a sha hash of
the tarball, to make sure the legitimate one is being downloaded.
Without a hash check this stuff should not be in samba sources.

Simo.

[C.J. Adams-Collier KF7BMP]

On Fri, 2013-05-24 at 10:25 +0200, Michael Adam wrote:
> On 2013-05-23 at 17:49 -0700, C.J. Adams-Collier KF7BMP wrote:
> > On Fri, 2013-05-24 at 02:07 +0200, Jelmer Vernooij wrote:
> > > If there are regressions or concrete concerns, by all means,
> > > please bring them up and we can discuss them.
> > >
> > > Cheers,
> > >
> > > Jelmer
> >
> > Okay. You've convinced me. But I reserve the right to say "I told you
> > so" ;-)
>
> As Jelmer pointed out, the fundamental discussion you are
> starting here has been (or should have been) taken place
> several years ago.
>
> We have been building the AD-Part (samba4) of Samba with waf for more
> than three years now (iirc), the combined build followed soon.
> And before introducing waf, the samba4 part was also not built by
> a (pure) autoconf system for a considerable time.
>
> What we are talking about now is the question whether to remove
> the autoconf build system from the source3 subdirectory,
> which is also built with the top level waf build, which is
> declared default build in in Samba 4.0.
>
> So maybe you should have told us several years ago. :-)
>
> Cheers - Michael

Yes, I agree. But I haven't been trolling this list for very long.

My concern is that there are probably many use cases which the autoconf
build environment is unwittingly making functional. Many folks tend to
complain about build problems in hallways at conferences and directly to
folks who can make changes to the environment directly. Some build
folks who might have had commit access to the source might have patched
things up without reporting bugs. Some issues might have been addresses
in other packages and fixed in autoconf before they reared their head in
samba specifically.

I'm just saying that there are likely a lot of edge cases which are
being made possible through the use of this time tested and hardened
system. Be aware that some folks may submit regression reports so that
they can be addressed, but in many cases, those reports won't get filed
and user adoption will instead diminish silently.

If I had more time to dedicate to the project, I would maintain an
alternate branch with an autoconf build system. But I don't, and I
trust that the Samba Team™ will continue to run a tight ship, so I'll
just be a curmudgeon and mumble about how back in my day we used
autoconf and it worked just fine for us.

Cheers,

C.J.

ps

> On Fri, 2013-05-24 at 02:07 +0200, Jelmer Vernooij wrote:
> Either way, I don't see how the age of the technology matters other than
> as a rough indicator of its maturity. The technologies the old and the new
> build system were built on (Python and autoconf) both date from '91.

you be frontin'. autoconf is built on m4 and the bourne shell (both
from 1977). waf may be built on Python, but the first commit to its
current source tree happened in 2011.

cjac@foxtrot:/usr/src/git/google/waf$ git log | tail
Author: Thomas Nagy <tnagy...@gmail.com>
Date: Sat Sep 10 12:20:18 2011 +0200

Unused variable

commit 44a967e326cc2e670a31b3712e4763b72d65e81b
Author: Thomas Nagy <tnagy...@gmail.com>
Date: Sat Sep 10 11:13:51 2011 +0200

Initial commit

Looks like the first commits to the present autoconf tree happened in
1992. On a Sunday. At 06:30 hrs. By a guy who still uses the same
email address twenty years later.

cjac@foxtrot:/usr/src/git/savannah/autoconf$ git log | tail
Author: Richard M. Stallman <r...@gnu.org>
Date: Fri Mar 13 19:54:03 1992 +0000

Initial revision

commit c05d1676aafc5afd689b7dc304e622450c33a450
Author: Richard M. Stallman <r...@gnu.org>
Date: Sun Mar 8 06:28:22 1992 +0000

Initial revision

[Jelmer Vernooij]

On Wed, May 29, 2013 at 12:56:54PM -0700, C.J. Adams-Collier KF7BMP wrote:
> On Fri, 2013-05-24 at 10:25 +0200, Michael Adam wrote:
> > On 2013-05-23 at 17:49 -0700, C.J. Adams-Collier KF7BMP wrote:
> > > On Fri, 2013-05-24 at 02:07 +0200, Jelmer Vernooij wrote:
> > > > If there are regressions or concrete concerns, by all means,
> > > > please bring them up and we can discuss them.
> > >

> > > Okay. You've convinced me. But I reserve the right to say "I told you
> > > so" ;-)
> >
> > As Jelmer pointed out, the fundamental discussion you are
> > starting here has been (or should have been) taken place
> > several years ago.
> >
> > We have been building the AD-Part (samba4) of Samba with waf for more
> > than three years now (iirc), the combined build followed soon.
> > And before introducing waf, the samba4 part was also not built by
> > a (pure) autoconf system for a considerable time.
> >
> > What we are talking about now is the question whether to remove
> > the autoconf build system from the source3 subdirectory,
> > which is also built with the top level waf build, which is
> > declared default build in in Samba 4.0.
> >
> > So maybe you should have told us several years ago. :-)

> Yes, I agree. But I haven't been trolling this list for very long.

> My concern is that there are probably many use cases which the autoconf
> build environment is unwittingly making functional. Many folks tend to
> complain about build problems in hallways at conferences and directly to
> folks who can make changes to the environment directly. Some build
> folks who might have had commit access to the source might have patched
> things up without reporting bugs. Some issues might have been addresses
> in other packages and fixed in autoconf before they reared their head in
> samba specifically.
>
> I'm just saying that there are likely a lot of edge cases which are
> being made possible through the use of this time tested and hardened
> system. Be aware that some folks may submit regression reports so that
> they can be addressed, but in many cases, those reports won't get filed
> and user adoption will instead diminish silently.

A thing to note is that the build system was not autoconf, nor is it waf now.
The old build system was built *using autoconf*, and the new build system is
built *using waf*. There is a fair amount of code on top of those systems required
still.

Sure, autoconf has been around longer and is more widely used. That doesn't
mean it's perfect. It just means that the chances of somebody running into
a particular design issue or bug *in autoconf* since it has been around is
greater. It also says nothing about whether those bugs or design issues have
been addressed, can be addressed or will be addressed.

Yes, waf is more of an unknown - but it's been around for more than a couple of
years, and we have used it in Samba for a while too. Early users of Samba 4
have run into issues with the build system (either in waf or unique to the
build system) that have been fixed. I'm sure people will continue to
occasionally hit minor bugs that we fix, just like we have always done
with the autoconf-based system. And waf has been a tremendous tool for
the Samba developers.

In the end, if users don't tell us about the problems they are experiencing
with Samba - at build time or at run time - and we're not experiencing
those problems, then ultimately there is nothing we can do.

> > On Fri, 2013-05-24 at 02:07 +0200, Jelmer Vernooij wrote:
> > Either way, I don't see how the age of the technology matters other than
> > as a rough indicator of its maturity. The technologies the old and the new
> > build system were built on (Python and autoconf) both date from '91.
>
> you be frontin'. autoconf is built on m4 and the bourne shell (both
> from 1977). waf may be built on Python, but the first commit to its
> current source tree happened in 2011.

The point of a build system is that it works, not that it is old. I'm sure
we can argue about which of the two systems has the oldest technologies behind
it, but that is ultimately not what matters.

> cjac@foxtrot:/usr/src/git/google/waf$ git log | tail
> Author: Thomas Nagy <tnagy...@gmail.com>
> Date: Sat Sep 10 12:20:18 2011 +0200
>
> Unused variable
>
> commit 44a967e326cc2e670a31b3712e4763b72d65e81b
> Author: Thomas Nagy <tnagy...@gmail.com>
> Date: Sat Sep 10 11:13:51 2011 +0200
>
> Initial commit

Waf has definitely been around longer than 2011. IIRC its origins are somewhere in 2005.
The system it is inspired by (SCons) and its predecessor (cons) date from '99.

Cheers,

Jelmer