scons build

[Greg Stein]

Hey guys,

The 1.2.x build continues to use the old, various build mechanisms. I
think if we end up releasing a 1.3.x (before a big 2.0), then we
should let the build mechanism switch over to scons.

Is there any particular reason to keep the same build system for the
1.x series? (it is unrelated to the published APIs, so it seems free
to change)

Thanks,
-g

[C. Michael Pilato]

Why did the project revert to the old serfmake stuff for the 1.2 release?
Are those reasons irrelevant when considering 1.3?

[Peter Samuelson]

[C. Michael Pilato]

> Why did the project revert to the old serfmake stuff for the 1.2 release?
> Are those reasons irrelevant when considering 1.3?

It wasn't a revert - it was the build system already in use in previous
releases. It looked like a revert because 1.2 was branched from trunk
rather than from 1.1. I think the reasons are the usual when you
contemplate changing something like this from one release to the next:

- Is it convenient for users to install scons?
- Does the scons-based build work in all scenarios?
- Does it support everyone's use cases - cross-compiling, custom or
unusual install paths, read-only source tree (out-of-tree build),
shared libraries on goofy platforms like Windows and OS X, etc.?

I don't have much of an opinion on these questions as my main interest
is Debian packaging, where these questions mostly don't apply - except
for custom install paths, and that's easy to work around if I have to.
And cross-compiling, but worst case, I don't _have_ to support that,
not all of our archive does.

Peter

[Greg Stein]

On Fri, Feb 22, 2013 at 3:11 PM, Peter Samuelson <pe...@p12n.org> wrote:
>
> [C. Michael Pilato]
>> Why did the project revert to the old serfmake stuff for the 1.2 release?
>> Are those reasons irrelevant when considering 1.3?
>
> It wasn't a revert - it was the build system already in use in previous
> releases. It looked like a revert because 1.2 was branched from trunk
> rather than from 1.1.

Right.

> I think the reasons are the usual when you
> contemplate changing something like this from one release to the next:
>
> - Is it convenient for users to install scons?

I believe this is fine. scons is a mature system, and I believe that
all packagers/developers have easy access to it.

> - Does the scons-based build work in all scenarios?

Nope. But we can't fix it unless we ship it and get feedback.

(believe me, this is the story of my life, w.r.t svn/serf)

I have tested the serf build on Mac OS, FreeBSD, Ubuntu, and Solaris.

The big missing piece is somebody getting the scons build to work on
Windows. I'm tempted to just rm serf.mak and force the Windows folks
to step up to the plate.

> - Does it support everyone's use cases - cross-compiling, custom or
> unusual install paths, read-only source tree (out-of-tree build),
> shared libraries on goofy platforms like Windows and OS X, etc.?

See above.

> I don't have much of an opinion on these questions as my main interest
> is Debian packaging, where these questions mostly don't apply - except
> for custom install paths, and that's easy to work around if I have to.
> And cross-compiling, but worst case, I don't _have_ to support that,
> not all of our archive does.

Theoretically, the new scons build supports a custom root for the
install. It *does* assume a "standard" layout underneath that. Are you
thinking about variant layouts? (doable, if you think that is needed)

For cross-compile... woof. No idea there. I would think that we just
wait for feedback/requests/patches when somebody tries that.

Cheers,
-g

[Justin Erenkrantz]

On Fri, Feb 22, 2013 at 3:43 PM, Greg Stein <gst...@gmail.com> wrote:

I believe this is fine. scons is a mature system, and I believe that

all packagers/developers have easy access to it.

As of right now, I can't seem to find a scons package for OmniOS...which I've been playing with for the last week or so.  I'll dig around to see if I'm just missing a package repository that has it somewhere, but this bit me on the cross-country flight as I couldn't test my changes with OmniOS.  Not a blocker...but, I do tend to find myself on weird platforms.  =)  (I may figure out how to package scons for OmniOS/Illumos...and solve this a better way.)

The one thing I really like to do is have VPATH-style builds with make (so I can just blow away the object files or test with different compiler version/settings) - I can't seem to figure out how to do that with scons - am I just missing something obvious?  -- justin

[Greg Stein]

On Mon, Feb 25, 2013 at 2:27 AM, Justin Erenkrantz
<jus...@erenkrantz.com> wrote:
> On Fri, Feb 22, 2013 at 3:43 PM, Greg Stein <gst...@gmail.com> wrote:
>>
>> I believe this is fine. scons is a mature system, and I believe that
>> all packagers/developers have easy access to it.
>
>
> As of right now, I can't seem to find a scons package for OmniOS...

Well, stop using OmniOS.

;-)

> which
> I've been playing with for the last week or so. I'll dig around to see if
> I'm just missing a package repository that has it somewhere, but this bit me
> on the cross-country flight as I couldn't test my changes with OmniOS. Not
> a blocker...but, I do tend to find myself on weird platforms. =) (I may
> figure out how to package scons for OmniOS/Illumos...and solve this a better
> way.)

SCons has a "local" version that can be shipped with your own package.
We could create a serf distro that includes scons for the fringe
platforms.

You could grab a copy of the local version right now, to get yourself
bootstrapped up. See the README for information.

> The one thing I really like to do is have VPATH-style builds with make (so I
> can just blow away the object files or test with different compiler
> version/settings) - I can't seem to figure out how to do that with scons -
> am I just missing something obvious? -- justin

No idea, to be honest. I'm sure it supports them, but I haven't
investigated. Let us know when you find it :-)

(or better yet, update README)

Cheers,
-g

[Justin Erenkrantz]

On Mon, Feb 25, 2013 at 3:44 PM, Greg Stein <gst...@gmail.com> wrote:

No idea, to be honest. I'm sure it supports them, but I haven't

investigated. Let us know when you find it :-)

(or better yet, update README)

After a few Google searches, "scons --Y ~/path/to/serf" would work...except for the fact that we're parsing serf.h to get the version information in SConstruct.  We can't call "open('serf.h')" and expect that to work.  If I hardcode the version number (a la below), it appears to build fine.

Perhaps a better way is to auto-gen the version info from SCons instead of parsing the .h file?  IOW, we add a builder that emits version information into a file?  But, I'm not quite clear how we can edit the Environment after it has been initialized...  -- justin

Index: SConstruct

===================================================================

--- SConstruct  (revision 1730)

+++ SConstruct  (working copy)

@@ -51,12 +51,7 @@

                None),

   )

 

-match = re.search('SERF_MAJOR_VERSION ([0-9]+).*'

-                  'SERF_MINOR_VERSION ([0-9]+).*'

-                  'SERF_PATCH_VERSION ([0-9]+)',

-                  open('serf.h').read(),

-                  re.DOTALL)

-MAJOR, MINOR, PATCH = [int(x) for x in match.groups()]

+MAJOR, MINOR, PATCH = [ 2, 0, 0 ]

 

 

 env = Environment(variables=opts,

[Greg Stein]

On Feb 25, 2013 4:42 PM, "Justin Erenkrantz" <jus...@erenkrantz.com> wrote:
>
> On Mon, Feb 25, 2013 at 3:44 PM, Greg Stein <gst...@gmail.com> wrote:
>>
>> No idea, to be honest. I'm sure it supports them, but I haven't
>>
>> investigated. Let us know when you find it :-)
>>
>> (or better yet, update README)
>
>
> After a few Google searches, "scons --Y ~/path/to/serf" would work

Cool.

> ...except for the fact that we're parsing serf.h to get the version information in SConstruct.  We can't call "open('serf.h')" and expect that to work.  If I hardcode the version number (a la below), it appears to build fine.

Oh, I'm sure we can get a proper path to serf.h. Should be a lot easier than generating or tweaking a file.

Cheers,
-g

[Justin Erenkrantz]

On Mon, Feb 25, 2013 at 4:49 PM, Greg Stein <gst...@gmail.com> wrote:

Oh, I'm sure we can get a proper path to serf.h. Should be a lot easier than generating or tweaking a file.

I spent 30 minutes hunting that down and I couldn't figure out a way.  Ideas?  It looks like the Repository object should have the right information - but I can't find any documentation about how to query it to find a path...  -- justin

[Greg Stein]

I'm away from my laptop, but it should be something like File('serf.h') to get an object representing the file. Then .path or something; that's the part I'd need to look up. There may even be an example in the SConstruct for fetching the underlying path.

Cheers,
-g

[Justin Erenkrantz]

On Mon, Feb 25, 2013 at 4:53 PM, Greg Stein <gst...@gmail.com> wrote:

I'm away from my laptop, but it should be something like File('serf.h') to get an object representing the file. Then .path or something; that's the part I'd need to look up. There may even be an example in the SConstruct for fetching the underlying path.

The other case in SConstruct uses env.File ... but, we need to have version resolved before we create the Environment.  Perhaps we should just delay assigning version info until after the env is created?  -- justin

[Justin Erenkrantz]

See r1731.  "scons -Y ~/work/serf" etc, etc, etc.  -- justin

[Greg Stein]

Nice!

[Justin Erenkrantz]

On Mon, Feb 25, 2013 at 7:23 PM, Greg Stein <gst...@gmail.com> wrote:

> See r1731.  "scons -Y ~/work/serf" etc, etc, etc.  -- justin

Nice!

Ok, I just spent some time with Theo on trying to get scons to build serf on OmniOS.  The problem we're running into is that scons doesn't understand the "-m64" flags (it doesn't give it to the linker - only to the compiler) and incorrectly sets the linker flags "-G" on gcc (where that specifies shared with Sun Forte, but not for GCC).  It's workable to hack around, but it's not likely to be very elegant and probably requires upstream fixes in scons to work out-of-the-box.

Ugh.  -- justin

[Greg Stein]

There are Solaris hacks in there already. This doesn't feel like a stopper, but more like Yet Another Patch. :-)

Real World Testing. gotta luv it!

Cheers,
-g

[Justin Erenkrantz]

On Tue, Feb 26, 2013 at 5:11 PM, Greg Stein <gst...@gmail.com> wrote:

There are Solaris hacks in there already. This doesn't feel like a stopper, but more like Yet Another Patch. :-)

I'll try to take a whack at coming up with a cleaner approach to the hackery later, but I need a long cold shower after diving into this pile of mud.  =)  -- justin 

[Greg Stein]

Heh. I was hoping that scons would save us from autotools hell. Let's discuss this weekend...

[Justin Erenkrantz]

Here in Portland, Brane was praising cmake, FWIW.  I know a few projects that are having good luck with that...a few others seem to use gyp.  -- justin

--
You received this message because you are subscribed to the Google Groups "Serf Development List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to serf-dev+u...@googlegroups.com.
To post to this group, send email to serf...@googlegroups.com.
Visit this group at http://groups.google.com/group/serf-dev?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Greg Stein]

Generally, the deal-breaker is Windows builds. Can they do that?

(I'm totally open; my goal is to kill our current tripartite build... serfmake, autotool, serf.mak)

[Justin Erenkrantz]

On Tue, Feb 26, 2013 at 6:03 PM, Greg Stein <gst...@gmail.com> wrote:

Generally, the deal-breaker is Windows builds. Can they do that?

(I'm totally open; my goal is to kill our current tripartite build... serfmake, autotool, serf.mak)

Yup - both product MS project files/solutions:

http://www.cmake.org/cmake/project/about.html
https://code.google.com/p/gyp/wiki/GypUserDocumentation
https://code.google.com/p/v8/wiki/BuildingWithGYP

GYP may be a little too barebone to support odd-ball platforms, but Chromium et al use it...  -- justin

[Ivan Zhakov]

On Wed, Feb 27, 2013 at 3:03 AM, Greg Stein <gst...@gmail.com> wrote:
> Generally, the deal-breaker is Windows builds. Can they do that?
>
> (I'm totally open; my goal is to kill our current tripartite build...
> serfmake, autotool, serf.mak)
>

Just for note: I'm happy with current NMake-based Windows build system
in serf. May be we can leave it for Windows and use other build system
for non-Windows platforms?


--
Ivan Zhakov

[Bert Huijben]

And all of them require one more dependency when building Subversion on Windows as none of the other projects required by Subversion use these tools.

 

So by removing a dependency for some users, you are adding another one for Windows users.

 

Nmake is currently required by both ZLib and OpenSSL, so we already have that as part of our build environments.

 

Once we can drop Visual C++ 6.0, Visual Studio 2002 and 2003 from the supported build environments for Subversion on Windows we could switch to a MSBuild file, which would require less maintenance from the project.

 

                Bert

 

--

[Justin Erenkrantz]

On Wed, Feb 27, 2013 at 7:30 AM, Bert Huijben <be...@qqmail.nl> wrote:

And all of them require one more dependency when building Subversion on Windows as none of the other projects required by Subversion use these tools.

 

So by removing a dependency for some users, you are adding another one for Windows users.

 

Nmake is currently required by both ZLib and OpenSSL, so we already have that as part of our build environments.

I'm fairly sure that cmake can produce nmake and MSBuild files; while I think that gyp only produces MSBuild.

 Once we can drop Visual C++ 6.0, Visual Studio 2002 and 2003 from the supported build environments for Subversion on Windows we could switch to a MSBuild file, which would require less maintenance from the project.

The thing I'd like to be able to do is be able to update one file and not worry about breaking the Windows build...which is where we are now.  Except we have four different build systems (autoconf, serfmake, scons, and Windows).  That's clearly unsustainable...  -- justin

[Bert Huijben]

Ok, so you can create and commit the NMake file? Problem solved.

 

I don’t have cmake, nor any of those other tools.

 

What I was trying to say by requiring those tools you make development on Windows harder; not easier.

 

Adding more external requirements is not going to reduce the complexity of building the total set.

 

Building Subversion on Windows is not hard; in fact it is the easiest part in building Subversion and all dependencies. All those requirements on third party projects is what makes it hard.

 

Adding one more dependency is not going to solve that we have so many dependencies.

 

Bert

 

Sent from Windows Mail

 

From: Justin Erenkrantz <jus...@erenkrantz.com>
Sent: February 27, 2013 6:28 PM
To: serf...@googlegroups.com
Subject: Re: [serf-dev] scons build

 

 

--

[Peter Samuelson]

[Bert Huijben]

> Ok, so you can create and commit the NMake file? Problem solved.

At least in Unix, cmake seems to work like scons and _unlike_ autoconf,
in that you need to install cmake itself in order to build something
that uses it. (Frankly that's one thing I like about autoconf and in
fact libtool and automake too: they go out of their way to not require
and end-user-with-a-release-tarball to install anything special.)

I don't know if cmake's generated Windows stuff can be shipped in
release tarballs, such that the end user in Windows doesn't need cmake
itself. As Bert says, that would be helpful.

[Bert Huijben]

> -----Original Message-----
> From: serf...@googlegroups.com [mailto:serf...@googlegroups.com]

> On Behalf Of Peter Samuelson
> Sent: woensdag 27 februari 2013 21:46
> To: serf...@googlegroups.com
> Subject: Re: [serf-dev] scons build
>
>

To me personally it doesn't really matter as long as it 'just works', (I
just script the next dependency like I did all the others) but I often hear
this argument of making things easier by adding another dependency.


Building Subversion on Windows is not easy and the only thing to make it
easier would be to remove dependencies, or to get our dependencies to work
together with similar systems. APR, Serf, Subversion and Httpd (and pocore?)
would be a good candidates to switch to a single system, which currently do
everything separately.

All these projects would be trivial to compile on Windows if they used
something like the Subversion project generator (or cmake, or whatever other
modern project generator)


There is not much else we can do as Subversion project. By making Subversion
itself easier to build, the situation doesn't really change. Solving this
inside serf is even harder.

Things that would really help is using Windows native libraries instead of
libraries like openssl that itself have multiple dependencies. The raw
support for deflate (which we use from zlib) and ssl layers is just
available in the OS, so those are the points where we can gain something.
But that would be by duplicating effort between different platforms.

Bert

[Justin Erenkrantz]

On Wed, Feb 27, 2013 at 3:45 PM, Peter Samuelson <pe...@p12n.org> wrote:

I don't know if cmake's generated Windows stuff can be shipped in
release tarballs, such that the end user in Windows doesn't need cmake
itself.  As Bert says, that would be helpful.

In talking with Bill Rowe (APR/httpd committer), he said that it's pretty straightforward to include the resulting project files in the release distributions so the Windows users don't need anything other than VS installed - and it permits easy building with nmake or the GUI.  Apparently, every version of cmake can produce every variant - though we did end up in a discussion about whether it makes sense to commit the cmake artifacts into version control.  Bill thought it was pretty easy to install cmake on Windows for developers checking out from trunk.

FWIW, Brane also recommended cmake as the way to go.

At the least, two more votes for cmake from the broader peanut gallery.  -- justin

[Bert Huijben]

Maybe we should look at this broader than serf.

 

Subversion uses X,

Apr uses Y,

Apache HTTPD uses Z,

Serf uses A.

 

And pocore (possible future dependency of serf) uses another thing.

 

So, great that serf moves to another thing… But can we please make slow progress towards a single system instead of side steps to more dependencies.

 

Reducing dependencies is what would helps. Adding more nonstandard dependencies, to manage more dependencies certainly won’t.

 

 

Would it help to provide a serf ssl bucket implementation that doesn’t need openssl on Windows, but uses the Windows secure channel system?

 

                Bert

 

 

From: serf...@googlegroups.com [mailto:serf...@googlegroups.com] On Behalf Of Justin Erenkrantz
Sent: donderdag 28 februari 2013 18:22
To: serf...@googlegroups.com
Subject: Re: [serf-dev] scons build

 

On Wed, Feb 27, 2013 at 3:45 PM, Peter Samuelson <pe...@p12n.org> wrote:

--

[Justin Erenkrantz]

On Thu, Feb 28, 2013 at 12:30 PM, Bert Huijben <be...@qqmail.nl> wrote:

Maybe we should look at this broader than serf.

 

Subversion uses X,

Apr uses Y,

Apache HTTPD uses Z,

Serf uses A.

 

And pocore (possible future dependency of serf) uses another thing.

At least in the informal conversation with Bill and Brane, they seemed very supportive of moving towards cmake.  Bill was already thinking of starting down this road for httpd (and APR by definition).  There is a lot of pain in Windows world...

 So, great that serf moves to another thing… But can we please make slow progress towards a single system instead of side steps to more dependencies.

 

Reducing dependencies is what would helps. Adding more nonstandard dependencies, to manage more dependencies certainly won’t.

 

 

Would it help to provide a serf ssl bucket implementation that doesn’t need openssl on Windows, but uses the Windows secure channel system?

Sure!  I know nothing about it though...but it can't be more complex than the OpenSSL implementation.  =)  -- justin

[Greg Stein]

On Feb 28, 2013 12:35 PM, "Justin Erenkrantz" <jus...@erenkrantz.com> wrote:
>
> On Thu, Feb 28, 2013 at 12:30 PM, Bert Huijben <be...@qqmail.nl> wrote:
>>
>> Maybe we should look at this broader than serf.
>>
>>  
>>
>> Subversion uses X,
>>
>> Apr uses Y,
>>
>> Apache HTTPD uses Z,
>>
>> Serf uses A.
>>
>>  
>>
>> And pocore (possible future dependency of serf) uses another thing.
>
>
> At least in the informal conversation with Bill and Brane, they seemed very supportive of moving towards cmake.  Bill was already thinking of starting down this road for httpd (and APR by definition).  There is a lot of pain in Windows world...

I looked briefly at cmake. Not entirely opposed to the concept, but I definitely like that scons allows for direct Python integrations.

>...

>> Would it help to provide a serf ssl bucket implementation that doesn’t need openssl on Windows, but uses the Windows secure channel system?
>
>
> Sure!  I know nothing about it though...but it can't be more complex than the OpenSSL implementation.  =)

I'd really like to see us move to connection types for this. We can have HTTP, HTTP/OpenSSL, and HTTP/Windows-security.

Cheers,
-g