Understanding Git workflow around DEP-14

[deb...@lewenberg.com]

I am packaging some upstream software for Debian and I am trying to
understand the workflow around "DEP-14: Recommended layout for Git
packaging repositories".

Looking at DEP-14 I might have these Git branches:

master
debian/master
debian/stretch
upstream/latest

I understand that the Debian packaging files in debian/ will appear in
the "debian/*" branch, but my general question is: what is the workflow
around all these branches? When and how do files get merged from one
branch to another?


More specifically:

1. Besides the debian/ directory, what is the difference between the
"debian/master" branch and the "upstream/latest" branch?

2. What should the "master" branch be used for?

3. When a new upstream tarball is released, where should it be imported?
In "upstream/latest"? In "debian/master"? Both?


Thanks, Adam Lewenberg

[Matt Zagrabelny]

I'm no expert. I've commented inline with things I've done and how I read and understand DEP-14.

On Mon, Mar 11, 2019 at 1:30 PM <deb...@lewenberg.com> wrote:

I am packaging some upstream software for Debian and I am trying to
understand the workflow around "DEP-14: Recommended layout for Git
packaging repositories".

Looking at DEP-14 I might have these Git branches:

   master
   debian/master
   debian/stretch
   upstream/latest

I understand that the Debian packaging files in debian/ will appear in
the "debian/*" branch, but my general question is: what is the workflow
around all these branches? When and how do files get merged from one
branch to another?

git checkout debian/master

git merge upstream/latest

You want to keep the upstream/latest branch free of Debian specific packaging bits.

You can use tags to mark any upstream releases. Here is an example:

$ git tag

debian/2.4-1

debian/2.4-2

upstream/2.4

More specifically:

1. Besides the debian/ directory, what is the difference between the
"debian/master" branch and the "upstream/latest" branch?

I don't believe anything.

 

2. What should the "master" branch be used for?

I don't use the master branch with DEP-14. I believe the DEP is stating that you'd use "master" for native packages - which from the sounds of it, yours is not. Therefore, I'd not use "master".

 

3. When a new upstream tarball is released, where should it be imported?

Assuming you have a remote named "github", I suppose you'd do something like:

git pull github upstream/latest

Then, you'd do:

git checkout debian/master

git merge upstream/latest

Cheers,

-m

[Geert Stappers]

On Mon, Mar 11, 2019 at 01:50:49PM -0500, Matt Zagrabelny wrote:
> On Mon, Mar 11, 2019 at 1:30 PM <deb...@lewenberg.com> wrote:
>
> > I am packaging some upstream software for Debian and I am trying to
> > understand the workflow around "DEP-14: Recommended layout for Git
> > packaging repositories".
> >
> > Looking at DEP-14 I might have these Git branches:

The DEP-14 we are talking about is https://dep-team.pages.debian.net/deps/dep14/

> > master
> > debian/master
> > debian/stretch
> > upstream/latest
> >
> > I understand that the Debian packaging files in debian/ will appear in
> > the "debian/*" branch, but my general question is: what is the workflow
> > around all these branches? When and how do files get merged from one
> > branch to another?
> >
>
> git checkout debian/master
> git merge upstream/latest
>
> You want to keep the upstream/latest branch free of Debian specific
> packaging bits.
>
> You can use tags to mark any upstream releases. Here is an example:
>
> $ git tag
> debian/2.4-1
> debian/2.4-2
> upstream/2.4
>
> More specifically:
> >
> > 1. Besides the debian/ directory, what is the difference between the
> > "debian/master" branch and the "upstream/latest" branch?
> >
>
> I don't believe anything.

True.
It is the idea about branches.

> > 2. What should the "master" branch be used for?

Consider the string "master" a label for _your leading branch_

> I don't use the master branch with DEP-14. I believe the DEP is stating
> that you'd use "master" for native packages - which from the sounds of it,
> yours is not. Therefore, I'd not use "master".

But you have your own leading branch

> > 3. When a new upstream tarball is released, where should it be imported?
> >
>
> Assuming you have a remote named "github", I suppose you'd do something
> like:
>
> git pull github upstream/latest

I think it should be (be warned _not tested_ ) avoid that your current branch gets pollueted.

} git checkout upstream/latest
} git pull github

> Then, you'd do:
>
> git checkout debian/master
> git merge upstream/latest

Groeten
Geert Stappers
--
DEP14 for the win

[Matt Zagrabelny]

Hi Geert!

On Mon, Mar 11, 2019 at 2:12 PM Geert Stappers <stap...@stappers.nl> wrote:

On Mon, Mar 11, 2019 at 01:50:49PM -0500, Matt Zagrabelny wrote:


> > 2. What should the "master" branch be used for?

Consider the string "master" a label for _your leading branch_


> I don't use the master branch with DEP-14. I believe the DEP is stating
> that you'd use "master" for native packages - which from the sounds of it,
> yours is not. Therefore, I'd not use "master".

But you have your own leading branch 

When I am packaging someone else's software (upstream/latest) for inclusion in Debian (debian/master), I don't feel like I have "my own" leading branch.

What am I missing for using (or not using) a "master" branch? 

 

> > 3. When a new upstream tarball is released, where should it be imported?
> >
>
> Assuming you have a remote named "github", I suppose you'd do something
> like:
>
> git pull github upstream/latest


I think it should be     (be warned  _not tested_ )   avoid that your current branch gets pollueted.

What makes you believe that the current branch would get polluted?

I believe a:

git pull repo refspec

is equivalent to:

git checkout refspec

git pull repo

Am I wrong?

Thanks for the dialog!

-m

[Andreas Ronnquist]

On Mon, 11 Mar 2019 10:42:35 -0700,
deb...@lewenberg.com<deb...@lewenberg.com> wrote:

>I am packaging some upstream software for Debian and I am trying to
>understand the workflow around "DEP-14: Recommended layout for Git
>packaging repositories".
>
>Looking at DEP-14 I might have these Git branches:
>
> master
> debian/master
> debian/stretch
> upstream/latest
>
>I understand that the Debian packaging files in debian/ will appear in
>the "debian/*" branch, but my general question is: what is the
>workflow around all these branches? When and how do files get merged
>from one branch to another?
>

In addition to what Matt and Geert has written:

>
>More specifically:
>
>1. Besides the debian/ directory, what is the difference between the
>"debian/master" branch and the "upstream/latest" branch?

debian/master is where your packaging work happens in the debian
folder, and also where the building of the package happens. (Here you
also should have the upstream source merged, to be able to build the
package with the standard debuild or dpkg-buildpackage).

upstream/latest (this branch is sometimes simply called upstream) is
most often used for unpacked upstream release tarballs. If you have a
pristine-tar branch, you can recreate the upstream tarball from these
two branches).

>2. What should the "master" branch be used for?

It can be used for upstream git branch from the upstream git repository
- this can be useful if you have imported an upstream version from a
tarball, and simply want to cherry-pick an upstream fix from a commit
that was just a tiny bit too late for the upstream release.

>
>3. When a new upstream tarball is released, where should it be
>imported? In "upstream/latest"? In "debian/master"? Both?
>

Simply use gbp import-orig - Its man-page mentions:

>The sources are placed on the upstream branch (default: upstream),
>tagged and merged onto the debian branch (default: master).

and I only have the addition that the defaults simply can be changed
using a debian/gbp.conf to match DEP-14.

-- Andreas Rönnquist
mailin...@gusnan.se
and...@ronnquist.net

[Ben Finney]

Andreas Ronnquist <mailin...@gusnan.se> writes:

> debian/master is where your packaging work happens in the debian
> folder, and also where the building of the package happens. (Here you
> also should have the upstream source merged, to be able to build the
> package with the standard debuild or dpkg-buildpackage).

So which branch contains *only* the Debian packaging files; that is, the
files that go into the “Debian package diff” (Debian Policy §3.3)?

Or does DEP-14 mandate mixing the Debian packaging with the upstream
files?

That would make DEP-14 incompatible with the “overlay” workflow, where
the Debian packaging is tracked in a separate VCS.

--
\ “If consumers even know there's a DRM, what it is, and how it |
`\ works, we've already failed.” —Peter Lee, Disney corporation, |
_o__) 2005 |
Ben Finney

[Andreas Ronnquist]

On Tue, 12 Mar 2019 12:49:35 +1100,
Ben Finney<big...@debian.org> wrote:

>Andreas Ronnquist <mailin...@gusnan.se> writes:
>
>> debian/master is where your packaging work happens in the debian
>> folder, and also where the building of the package happens. (Here you
>> also should have the upstream source merged, to be able to build the
>> package with the standard debuild or dpkg-buildpackage).
>
>So which branch contains *only* the Debian packaging files; that is,
>the files that go into the “Debian package diff” (Debian Policy §3.3)?
>
>Or does DEP-14 mandate mixing the Debian packaging with the upstream
>files?

Nah, it doesn't mandate it, but it does recommend it - see "What to
store in the packaging branches" a bit down the DEP-14 page,
https://dep-team.pages.debian.net/deps/dep14/

>That would make DEP-14 incompatible with the “overlay” workflow, where
>the Debian packaging is tracked in a separate VCS.
>

Sorry, I have only used gbp with the mixed branch workflow - does
anyone else have any hints for gbp with the overlay workflow?

[Geert Stappers]

On Mon, Mar 11, 2019 at 05:08:07PM -0500, Matt Zagrabelny wrote:

> On Mon, Mar 11, 2019 at 2:12 PM Geert Stappers wrote:
> > On Mon, Mar 11, 2019 at 01:50:49PM -0500, Matt Zagrabelny wrote:
> >
> >
> > > > 2. What should the "master" branch be used for?
> >
> > Consider the string "master" a label for _your leading branch_
> >
> >
> > > I don't use the master branch with DEP-14. I believe the DEP is stating
> > > that you'd use "master" for native packages - which from the sounds of it,
> > > yours is not. Therefore, I'd not use "master".
> >
> > But you have your own leading branch
> >
>
> When I am packaging someone else's software (upstream/latest) for inclusion
> in Debian (debian/master), I don't feel like I have "my own" leading branch.
>
> What am I missing for using (or not using) a "master" branch?

Nothing (and nothing)

The git term "clone" is a good term. Cloning is not just copying,
is creating a child with the exact DNA of its parent. The new repository
get its first branch named 'master'. Is it leading? It could.

Git being a distributed VCS makes it confuesing for newcomers.
We all start as child. There are parents. It takes a while before
we become parents. With `git` it goes much faster. Upon clone
there is a master.

> > > > 3. When a new upstream tarball is released, where should it be imported?
> > > >
> > >
> > > Assuming you have a remote named "github", I suppose you'd do something
> > > like:
> > >
> > > git pull github upstream/latest
> >
> >
> > I think it should be (be warned _not tested_ ) avoid that your
> > current branch gets pollueted.

> } git checkout upstream/latest
> } git pull github
> >
>

> What makes you believe that the current branch would get polluted?
>
> I believe a:
>
> git pull repo refspec
>
> is equivalent to:
>
> git checkout refspec
> git pull repo
>
> Am I wrong?

Don't know, _not tested_. But yes, It could work.
I try (way too much) to be on the safe side.


> Thanks for the dialog!

YW MP

Groeten
Geert Stappers
--
Leven en laten leven

[wf...@niif.hu]

deb...@lewenberg.com writes:

> Looking at DEP-14 I might have these Git branches:
>
> master
> debian/master
> debian/stretch
> upstream/latest
>
> I understand that the Debian packaging files in debian/ will appear in
> the "debian/*" branch, but my general question is: what is the
> workflow around all these branches? When and how do files get merged
> from one branch to another?

See below for my take on the subject. Don't fear all the complication
spelled out, I'm trying to explain the big picture in the most complex
setting, so things hopefully start to make sense. In practice it's
pretty intuitive once you get the guiding principles.

> 1. Besides the debian/ directory, what is the difference between the
> "debian/master" branch and the "upstream/latest" branch?

Nothing, if you use the "3.0 (quilt)" source format, which you should.
upstream/latest tracks the contents of the pristine upstream tarballs.
If you use pristine-tar, you can reproduce the exact upstream tarballs
from upstream/latest and the appropriate binary deltas (usually stored
in a disconnected/orphan branch called pristine-tar).

> 2. What should the "master" branch be used for?

Whatever your upstream uses it for. As a packager, you use it for
cherry-picking upstream commits into your quilt patch queue and for
creating upstream pull requests by cherry-picking from your patch queue.
In other words: for communicating commits with your upstream. The
"master" branch is not special in a packaging repository, it's just one
of the upstream branches. The default branch of a packaging repository
is usually debian/master. (Some use debian/sid; since uploads from here
can target experimental as well, I prefer debian/master.)

The point of DEP-14 is to work with a clone of the upstream repository,
adding extra branches under the debian/ namespace for packaging for
various Debian distributions, under the ubuntu/ namespace for Ubuntu
distributions and so on, keeping the package vendors separate while
storing upstream tarball contents under the shared upstream/ namespace.
The pristine-tar branch is basically an unordered container of binary
deltas, so it's unique.

> 3. When a new upstream tarball is released, where should it be
> imported? In "upstream/latest"? In "debian/master"? Both?

It's typically imported into upstream/latest, unless you package from
several version series concurrently and thus you have for example an
upstream/2.x branch as well: then you import 2.x tarballs into the
upstream/2.x branch and 3.x tarballs into upstream/latest. You can name
these branches however you please, the point is them being under the
upstream/ namespace for all package vendors' use.

For best effect these import commits should be formal merge commits:
their first parent is the previous import commit, and the second parent
is the upstream commit from which your upstream author created the
release tarball. This is all formal, just a "note of relations" in the
git history; the actual content of the commit is that of the imported
release tarball, not influenced by either parent in any way. However,
these relations are meaningful and git can use them answering queries
like git tag --contains 123456 and similar.

Returning to concrete branch names: in the simplest case your upstream
author tags commits on the master branch and generates tarballs from
them; you then import these tarballs into upstream/latest, formally
merging in the tags on the master branch. However, your upstream may as
well create their release tags on a release branch (say stable2), and
then you import those into upstream/latest all the same, or they can do
both and then you might package both release streams importing into
upstream/2.x and upstream/latest appropriately; DEP-14 is very flexible.

Once the import is done, you fuse it with your packaging work by merging
it into debian/master (assuming a single release stream again). This is
another formal merge: what actually happens is that you replace the
upstream part of the current contents of debian/master with the contents
of upstream/latest, keeping the debian directory (ever present in
debian/master only) unchanged[1]. Then you adjust your packaging files
by committing on debian/master; these commits mustn't touch anything
outside the debian/ directory. You can make upstream changes via files
under debian/patches.

Now, how do you do this all? Simple:

$ gbp import-orig --upstream-vcs-tag=v2.3 ../project-2.3.tgz

Issue the above command on your debian/master branch, where you
committed a debian/gbp.conf file with content like this[2]:

[DEFAULT]
debian-branch = debian/master
upstream-branch = upstream/latest
pristine-tar = True

[import-orig]
merge-mode = replace

Instead of specifying the upstream tarball, you can even use --uscan
most of the time, if you already have a working debian/watch file.
Besides all the git magic described above, this command also creates an
upstream/2.3 tag for future reference.

The usual packaging workflow is you hacking away on debian/master
producing new Debian revisions of the package (gbp buildpackage can
create debian/2.3-1 and similar tags for all your uploads), and
occasionally import new upstream releases by the above procedure. You
upload packages to unstable, they migrate to testing and eventually get
released in stable, for example into buster. You upload further
revisions in the new release cycle, targeting buster, then suddenly a
security issue is uncovered and you have to fix it in buster. This
can't happen through an unstable upload, you might already have a new
upstream release with lots of unrelated changes there. Supposing the
version in buster is 2.3-1, you create the debian/buster branch
starting at the debian/2.3-1 tag. You change the value of debian-branch
in your gbp.conf to debian/buster, and work as usual committing the
security fix and building your package. From this point, you use the
debian/buster branch for buster-security and buster uploads (if you
ever propose stable updates of this package). Similarly, debian/stretch
is used for stretch-security and stretch uploads (while there's still
security support, LTS or oldstable updates for stretch). These branches
stay unrelated, though, you don't usually merge or import new upstream
releases into them, even when you fix the same issue on both.

What if you want to upload backports of your package? Piece of cake:
you create a debian/buster-backports branch starting at the current
testing version of your package (this is typically debian/master).
Change debian-branch in gbp.conf, commit the backporting changes, build
the package, upload. You continue hacking away on debian/master. When
you want to refresh your backport, you merge the new version (typically
debian/master again) into debian/buster-backports, backport the new
changes if needed, build and upload. You can even add another layer for
debian/stretch-backports-sloppy on top of this by branching off from
debian/buster-backports. In principle you could branch off from
debian/master directly as well, but then you don't get the backporting
work already done on debian/buster-backports for free.

[1] With --merge-mode=replace, the new default in git-buildpackage.
[2] For completeness: I usually add these gbp dch and gbp pq settings as
well, look up these very useful tools with man!

[dch]
full = True
multimaint-merge = True
id-length = 7

[pq]
patch-numbers = False

Regards,
--
Feri