at the last Franky meeting, we realized that our current attempt at creating a
combined Samba3/Samba4 build using git submodules was clumsy and fragile.
So we came up with a combined tree solution to address this issue.
Incidentally, this combined tree might be useful before we get the integrated
build sorted out already.
There is a small one-time effort involved migrating personal branches over,
more about that below. Let me first explain how the merged tree looks like.
As you all know, the Samba3 tree looks like this:
kai@blackjack:~/samba/samba3$ ls -1p
COPYING
docs-xml/
examples/
MAINTAINERS
Manifest
packaging/
pcp/
PFIF.txt
Read-Manifest-Now
README
README.Coding
release-scripts/
Roadmap
source/
swat/
testsuite/
WHATSNEW.txt
The Samba4 tree looks like this:
kai@blackjack:~/samba/samba4$ ls -1p
BUGS.txt
COPYING
howto.txt
NEWS
packaging/
PFIF.txt
prog_guide.txt
source/
swat/
swat2.txt
testdata/
testprogs/
TODO
WHATSNEW.txt
Now, it's obvious that a plain merge of these directories isn't possible as
there is a couple of clashes. Some of those are easy to fix (COPYING and
PFIF.txt), some are hard (source/).
As trying to combine the Samba3 and Samba4 source/ folder into a single folder
would be madness, we decided to merge the Samba3 source/ folder as source3/
and merge the Samba4 source/ folder as source4/. For my proof of concept
merge, I dropped the other conflicting files that weren't identical and as a
result, the combined tree looks like this:
kai@blackjack:~/samba/samba-devel$ ls -1p
BUGS.txt
COPYING
docs-xml/
examples/
howto.txt
MAINTAINERS
Manifest
merged_branches.txt
pcp/
PFIF.txt
prog_guide.txt
Read-Manifest-Now
README
README.Coding
release-scripts/
Roadmap
source3/
source4/
swat/
testdata/
testprogs/
testsuite/
TODO
The nice part about this is that nothing really changed for building Samba3 or
Samba4 standalone, just that to e.g. build Samba4, you now do "cd
source4; ./configure; make" instead of "cd source; ./configure; make".
So far so good, our first attempt looked pretty much like this. However, git's
tracking of file renames is very basic and breaks down if a lot of files are
renamed. For both Sambas, source is big enough to break. So while history
wasn't lost completely (e.g. git blame path/to/file still worked), git log
path/to/file didn't work anymore. The Git folks think this is a minor
annoyance, Volker hated it, and I tend to agree with Volker. There's a couple
of workarounds, but they were all pretty clumsy.
In the end, we found a solution that preserved the history across the merge
for a small one-time migration effort. Using some fancy git magic (git
filter-branch, in case you're interested), Volker created a Samba3 branch
that had all commits since the beginning use "source3/" instead of "source/",
and then repeated the same for a Samba4 branch. I then merged those two
branches using some more git magic Jelmer and I had discovered on Wednesday
(I'll describe the whole process in another email for people interested in
this) and uploaded the combined branches to git.samba.org [1].
Now, because we changed the commit objects doing this, the sha1-sums of the
commit objects don't work anymore, meaning git rebase or similar will blow up
attempting to rebase on the new, unknown objects. However, it's easy to work
around this with little effort, described on the Franky wiki page [2]
So while I don't propose to go live with this setup right now, I'm hoping for
some feedback on the general approach. The incredible advantage of this is
that (Franky considerations aside) code that is shared between Samba3 and
Samba4 can be pulled out of the source3/4 dirs and needs to be updated only
once instead of twice. So even though this looks like "yet one more branch"
right now, it will mean less branches to check things in in the long and even
in the medium term. Also, instead of having to get a separate Samba4 branch
to run smbtorture tests with Samba3, you just need to build source4 once, and
not change the repo.
We need to discuss how to handle the remaining duplicates from the old Samba3
and Samba4 trees. I'm looking forward to suggestions on how to improve the
migration guide as well.
Cheers,
Kai
[1] Check out the combined tree at git://git.samba.org/kai/samba/franky.git
[2] The migration guide is at
http://wiki.samba.org/index.php/Franky#Updating_your_branches_to_the_combined_build
--
Kai Blin
WorldForge developer http://www.worldforge.org/
Wine developer http://wiki.winehq.org/KaiBlin
Samba team member http://www.samba.org/samba/team/
--
Will code for cotton.
The only real clash I see is the swat/ directory. It should
be easy to rename swat/ to swat3/ for example.
For me running ./configure; make in source3 worked fine, so
in theory nothing technical would stop us from switching to
that combined tree, even if Jelmer's merged build patch is
not yet there. Or am I missing anything?
Volker
> Hi folks,
>
> at the last Franky meeting, we realized that our current attempt at
> creating a
> combined Samba3/Samba4 build using git submodules was clumsy and fragile.
>
> So we came up with a combined tree solution to address this issue.
> Incidentally, this combined tree might be useful before we get the
> integrated
> build sorted out already.
...
How do you handle the "install" target in the combined structure? Does a
"make install" in samba3 still, by default, overwrite a "make install" in
samba4 and vice versa, or have you come up (I hope!) with a nice solution so
that the two can both be installed and co-exist?
Derrell
Yes, as it stands now they overwrite each other. Future
discussion will for example have to solve which of the net
binaries will be used how and so on. There is a lot to be
solved/merged/migrated etc, this current effort is step zero
of a loong way.
Volker
I guess one of the 2 sources default prefix can be changed so that they
do not conflict ?
Simo.
--
Simo Sorce
Samba Team GPL Compliance Officer <si...@samba.org>
Senior Software Engineer at Red Hat Inc. <si...@redhat.com>
> I like the concept, but let us (me) redo the git-filter-branch thing so
> that each new commit has a reference to the old sha1 in it.
If you don't have time for this right now, I can give it a shot. It just takes
quite a while to rewrite the whole stuff, so I wanted to get more feedback
first, and then maybe do a final rewrite of the whole stuff.
Cheers,
Kai
No, I don't think you are missing anything.
Just the s3 packaging stuff will need some path changes, but
this should be very easy.
Let me stress here that the combined tree will be very useful
in its own right even without the merged build, since it will
allow us to reconcile common parts of both sambas more easily.
We will for instance be able to move for instance lib/tdb and
lib/talloc to the top level directory to eliminate the need to
sync these virtually identical folders between the two.
This way we would also have one only definitive source of these
subsystems.
Cheers - Michael
--
Michael Adam <m...@sernet.de>
SerNet GmbH, Bahnhofsallee 1b, 37081 Göttingen
phone: +49-551-370000-0, fax: +49-551-370000-9
AG Göttingen, HRB 2816, GF: Dr. Johannes Loxen
http://www.SerNet.DE, mailto: Info @ SerNet.DE
Or I could just finish wielding the axe on Samba4's SWAT :-)
I'm really intrigued by this work - in particular the hope that we could
start sharing code by default, rather than only as a result of
deliberate effort.
Andrew Bartlett
--
Andrew Bartlett
http://samba.org/~abartlet/
Authentication Developer, Samba Team http://samba.org
Samba Developer, Red Hat Inc.
Yeah, that was the gist of what I remembered of the last S4 SWAT discussion.
Which is why the proof of concept tree ships the s3 swat dir. :)
Cheers,
Kai
> So while I don't propose to go live with this setup right now, I'm hoping for
> some feedback on the general approach. The incredible advantage of this is
> that (Franky considerations aside) code that is shared between Samba3 and
> Samba4 can be pulled out of the source3/4 dirs and needs to be updated only
> once instead of twice. So even though this looks like "yet one more branch"
> right now, it will mean less branches to check things in in the long and even
> in the medium term. Also, instead of having to get a separate Samba4 branch
> to run smbtorture tests with Samba3, you just need to build source4 once, and
> not change the repo.
I really see a lot of value in this, if we can make this the new 'devel'
branch, and get Samba3 and Samba4 releases from the once place.
Having source code shared, rather than backported, should be much less
work in the long term (but will no doubt be a right pain in the short
term, we can't sustain what we have been doing).
I look forward to further progress on this.
Find the newly converted trees on
http://git.samba.org/?p=vl/samba.git/.git;a=summary
Volker
And the newly generated merge tree at:
http://gitweb.samba.org/?p=kai/samba/franky.git;a=summary
If people are fine with this tree, I'd propose we switch development to this
tree.
Cheers,
Kai
I'm downloading the tree now, but I think there are a few things we
should get straight, to ensure we gain a benefit from the upheaval.
Without changing our development practices, I don't see how this move
(which I fully support) will change anything.
Therefore, I suggest that this cannot be a success without the following
conditions being met. (and as we wish this project to be a success,
that we agree to abide by these).
That we have a defined goal release (in Samba3 release terms, as it has
a predictable timeline) that we expect to use the combined tree.
That a combined build system exist, such that in a reasonable manner, a
developer can build both Samba3 and Samba4, and run 'make test' that
uses binaries from both.
That both Samba3 and Samba4 be built as part of this build system by
default. If Samba4 critically depends on a system component not
available, that only then is Samba4 (or some Samba4 components) not
built. (Samba3 is expected to build at all times, simply because it has
historically required less from the system).
That there be an option such as --enable-developer which requires that
both trees be built. Developers will be expected to use this option at
all times.
That the goal is not to create a single release (at this stage), but to
integrate technology. As such, builds of Samba3 and Samba4, producing
similar user-visible binaries to current practice etc be available.
That 'make test' must pass against the combined work before commits, and
if (for whatever reason, developers being human) a change causes
breakage, even if the change 'fixes' a test, that 'make test' must be
made to pass again. (eg, a more strict torture test must come with a
fix for both file server implementations, or a very good reason
documented for a new test exemption).
That we work towards a common set of IDL again, and Samba3 use PIDL at
runtime for developers (at least) to ensure that IDL changes propagate
correctly.
That provided key functionality is not lost, changes are well tested,
and sensible user upgrade paths are available, that replacing code in
either branch with new or ported 'in common' code is strongly encouraged
(and the explicit purpose of the exercise).
That Python and Perl be an acceptable languages for implementation of
Samba infrastructure (build/test system etc) and installed scripts.
How does that sound?
(I don't want to bog us down with rules, but I'm also very wary of the
possible different expectations in this area).
Thanks,
There's more work to do before we can switch over...
1. we need to make v4-0-test and v3-devel readonly
update the sX-rename branches and to the merge again.
2. should be merged branch be named 'master'?
3. we need to update the build-farm scripts to
use source3 and source4
metze
> > If people are fine with this tree, I'd propose we switch development to
> > this tree.
[...]
> I suggest that this cannot be a success without the following
> conditions being met. (and as we wish this project to be a success,
> that we agree to abide by these).
>
> That we have a defined goal release (in Samba3 release terms, as it has
> a predictable timeline) that we expect to use the combined tree.
That's certainly doable. Karolin has the final word on this, of course, but I
guess that 3.4 would be branched off the combined tree. There's probably no
technical reason not to release 3.3.0 from the combined tree, apart from the
fact that the v3-3-test branch already was branched off.
> That a combined build system exist, such that in a reasonable manner, a
> developer can build both Samba3 and Samba4, and run 'make test' that
> uses binaries from both.
Jelmer was working on that one, but probably put it on hold until the merged
tree was finished. I agree this is the very next step.
> That both Samba3 and Samba4 be built as part of this build system by
> default. If Samba4 critically depends on a system component not
> available, that only then is Samba4 (or some Samba4 components) not
> built. (Samba3 is expected to build at all times, simply because it has
> historically required less from the system).
If I understood Jelmer correctly, his current patchset proposed in his "Allow
building Samba 4 inside the Samba 3 build process" already does this.
> That there be an option such as --enable-developer which requires that
> both trees be built. Developers will be expected to use this option at
> all times.
This might be problematic on build farm hosts that don't have python but run
with --enable-developer. Personally I don't see why I'd ever go back to only
building S3 or S4 if I can build both by a single "make"
> That the goal is not to create a single release (at this stage), but to
> integrate technology. As such, builds of Samba3 and Samba4, producing
> similar user-visible binaries to current practice etc be available.
I don't know what the required build system magic would be here, but I had the
impression the plan was to:
1) Get the trees merged, allowing to build an unchanged S3 from source3/ and
an unchanged S4 from source4/
2) Get a common build system set up, where "make samba3" builds S3, "make
samba4" builds S4 and use that as opposed to building from the respective
source directory.
3) Start consolidating the technology currently duplicated in S3 and S4
For the folks playing around with Franky, step 2) would likely include a "make
franky" that builds Franky. But I don't see any conflict of interest there.
After all integrating the technology is a core part of the Franky design as
well.
> That 'make test' must pass against the combined work before commits, and
> if (for whatever reason, developers being human) a change causes
> breakage, even if the change 'fixes' a test, that 'make test' must be
> made to pass again. (eg, a more strict torture test must come with a
> fix for both file server implementations, or a very good reason
> documented for a new test exemption).
I think as soon as "make" in the merged directory builds S3 and S4, "make
test" needs to run the S3 and S4 tests. Every other behavior sure would
confuse the heck out of people.
> That we work towards a common set of IDL again, and Samba3 use PIDL at
> runtime for developers (at least) to ensure that IDL changes propagate
> correctly.
Looking at the work gd put into this for netapi and others, I think this step
is a given, just to make his life easier. I'll gladly subscribe to this.
> That provided key functionality is not lost, changes are well tested,
> and sensible user upgrade paths are available, that replacing code in
> either branch with new or ported 'in common' code is strongly encouraged
> (and the explicit purpose of the exercise).
Full ack.
> That Python and Perl be an acceptable languages for implementation of
> Samba infrastructure (build/test system etc) and installed scripts.
I don't mind, but I figure this might be a problem for some of the hosts we
have in the build farm.
> How does that sound?
> (I don't want to bog us down with rules, but I'm also very wary of the
> possible different expectations in this area).
Most of this seems just explicitly stating sane development practices, so I
don't expect this will bog down people alot (apart maybe from the time a
combined "make && make test" run will take).
Personally, I'm just waiting for the combined build system to hit so I can
start converting over the utilities I'm working on.
Hm, good point.
> 2. should be merged branch be named 'master'?
This seems to be the convention for the main branch of git repositories, which
is why I decided to keep that.
> 3. we need to update the build-farm scripts to
> use source3 and source4
As an interrim solution, yes.
Looking at this, I figure that once we agree to switch to this tree, the
actual changeover work will be doable in a couple of hours, provided we don't
have to run the full filter-branch thing again, in which case it'll take a
day or so. But as we can't freeze the v3-devel and v4-0-test branches without
having agreed to switch to the combined tree, we need to decide about this
first.
I'm just saying that this is what the tree is going to look like and that I'm
looking forward to use the new setup.
I would like to leave v3-3-x and v3-2-x as they are and to branch 3.4 from
the new combined branch. Are there any arguments against this strategy?
Karolin
--
> > Personally, I'm just waiting for the combined build system to hit so I
> > can start converting over the utilities I'm working on.
>
> you can already start, you just need to rebase
> (or git format-patch && git am) when the final branch is ready...
I'm waiting for being able to "make" in the root dir, but you're right. It
should be possible to move the first utils now. I'm not short on todo list
items, though, so I don't lose much by waiting for Jelmer's updated build
system patch.
What utils do you want to move? Be careful as the same utils sometimes
don't do the same and operate on different databases.
metze
Well, we do have very different expectations on this one.
If merging trees makes Samba3 in the future require python
and perl to run, this will be problematic from my point of
view.
Volker
I was thinking about wbinfo, but I forgot that after creating libwbclient
wbinfo in S3 changed again. Last I looked at it, I had the differences
between S3 and S4 down to a couple of includes in the header file, but now
they're pretty different again.
Anyway, nsswitch/winbind_nss_* seem like good candidates. Looking at
winbind_nss_linux.c, they're pretty identical, not counting the thread-safety
changes that were not ported to the S4 copy.
The other utilities will need some extra steps, like getting a unified netapi
API for net. For a later milestone, I guess.
Given that both v4-0-test and v3-devel are used as they were 'master' I
think it would be advisable to sue this standard name for the master
branch.
> 3. we need to update the build-farm scripts to
> use source3 and source4
What about v3-0-stable/v3-2-stable ? Are they converted as well, or do
we keep them as is ?
I certainly think they need to be available to build and install, and I
think python at runtime is a reasonable requirement.
Otherwise, by having differing rules on what is acceptable between the
(former) branches, we start setting back up the barriers that this move
is designed to break down.
We'll see what the future brings. At least I would like to
keep up the option to build a fully compatible Samba 3 build
out of the combined tree for a while. This will not require
perl or python for the release tree.
I'd much rather go with "Hey, if you install python at build
time you get this super-cool feature", not force people into
it.
Volker
I think this is a very legitimate and doable approach.
We loose nothing and gain a lot by this.
It is important, to keep things working as they worked before
(from source3/ and source4) and add the merged build/test as
additional options.
Michael
--
Michael Adam <m...@sernet.de> <ob...@samba.org>
I'd say the same for the samba4 build.
The combined tree should also be the basis for future samba3 release
(after 3.3.x).
> I'd much rather go with "Hey, if you install python at build
> time you get this super-cool feature", not force people into
> it.
To depend on python at runtime, someone needs to fix the python build
for all machines in our build-farm.
I doesn't build on irix and some others the last time I tried to install
python there.
metze
> In the end, we found a solution that preserved the history across the merge
> for a small one-time migration effort. Using some fancy git magic (git
> filter-branch, in case you're interested), Volker created a Samba3 branch
> that had all commits since the beginning use "source3/" instead of "source/",
> and then repeated the same for a Samba4 branch. I then merged those two
> branches using some more git magic Jelmer and I had discovered on Wednesday
> (I'll describe the whole process in another email for people interested in
> this) and uploaded the combined branches to git.samba.org [1].
The rename seems to have had some casualties, as a number of the
blackbox test scripts in testprogs/blackbox presume that Samba4 can be
found under source, not source4.
This is hardly a blocker, but I wanted it noted, so we can get it fixed
once this becomes the real deal.
Regarding perl/python dependence, I think there are more levels to
this than just whether we depend on it for development or runtime use.
Right now Samba3 needs perl for compiling IDL for development work,
and Samba4 additionally needs python for installation and
administration tools.
I think it would be more of a problem if we needed an external
language like perl/python for the core functionality at runtime, as
that would potentially make it quite hard for some embedded uses of
Samba.
I originally chose the EJS scripting language to embed in Samba
largely because I wanted to avoid depending on external languages at
runtime, but too many other developers didn't like my choice
(particularly the lack of debug tools), so we ended up with
python. That seems to be working out fairly well, except that it does
tie us much more to having python installed on the system in order to
administer it. Given that python is very widely available now, I don't
see that as a showstopper.
Cheers, Tridge
As we now have the combined 'master' branch with source3/ and source4/
sub directories, we can think about where to move common code
and the build magic for a combined 'make'.
I'd propose to move them to source/, to that all 3 ways of building
code in the master branch have the same weight. I assume have the magic
of the combined build in the top level directory would give more weight
to the most experimental way of building samba.
Comments please!
metze
> I'd propose to move them to source/, to that all 3 ways of building
> code in the master branch have the same weight. I assume have the magic
> of the combined build in the top level directory would give more weight
> to the most experimental way of building samba.
Hm, I actually liked the idea of being able to build from the top dir, instead
of having to cd into the source/ dir. We could get started on moving the
tests from the various test dirs closer to the code they test and clean up
there as well if clutter in the root dir is an issue.
My proposal is to have a Makefile in the root dir and have that build s3
on "make samba3", s4 on "make samba4" and franky on "make franky". I'm open
for suggestions on what should fall out of the tree on a plain "make". :)
Having a simple Makefile in the root dir is easy,
but what about autogen.sh, configure.ac, VERSION?
metze
Keep them in each subsytsem?
I Frankly would like to see less magic with makefiles and just build
each separate piece separately.
Cheers,
Jelmer
This is my preference, too.
> Perhaps we can leave just the merged build configure + Makefile out of
> the top-level source directory for now?
Sounds like a good idea: this will make the still experimental
stuff not the most prominent choice. Subfolder "franky" for the
merged build?
Cheers - Michael
--
Michael Adam <m...@sernet.de>