[perl #41386] MANIFEST must die.
Will Coleda
# Please include the string: [perl #41386]
# in the subject line of all future correspondence about this issue.
# <URL: http://rt.perl.org/rt3/Ticket/Display.html?id=41386 >
... at least in the repository. The primary purpose of the MANFEST is to
make sure that the generated release isn't missing any files, neh? So, when
you're working out of the repository, you already know the canonical list.
This can be handled when the release is generated: having this check run on
every configure by default complicates the process of patching; the tool
developed to simplify this doesn't work with non checked in files;
resulting, many times, in 2 commits, one to add the files to the repository,
then a cleanup of the MANIFEST.
Ditto MANIFEST.skip (use svn:ignore keyword).
There are many places that use MANIFEST that'd need to be updated to avoid
its use when working out of the repo. (Or those tasks can be migrated to a
release-only timeframe).
Allison Randal
>
> ... at least in the repository. The primary purpose of the MANFEST is to
> make sure that the generated release isn't missing any files, neh? So, when
> you're working out of the repository, you already know the canonical list.
The primary purpose of a MANIFEST in the repository is to tell you which
files out of the repository should be included in the distribution.
e.g., some languages, or the debian/ directory, or developer tools might
be excluded from the distribution, and so would be excluded from the
MANIFEST. Perhaps the usage needs to change, but I wouldn't remove it
from the repository.
> Ditto MANIFEST.skip (use svn:ignore keyword).
MANIFEST.skip could be simply generated from the svn:ignore keyword
before the distribution tarball is created. Do you have a list of places
MANIFEST.skip is used so we can review them?
Allison
Patrick R. Michaud via RT
manager this month :-)...
On Wed Jan 31 13:37:54 2007, all...@perl.org wrote:
> The primary purpose of a MANIFEST in the repository is to tell you
> which files out of the repository should be included in the
> distribution.
>
> MANIFEST.skip could be simply generated from the svn:ignore keyword
> before the distribution tarball is created.
Perhaps I'm just confused, but these two statements seem to be a
little at odds with each other, unless we say that MANIFEST.SKIP is
generated from "svn:ignore and other sources".
If I'm understanding correctly:
- MANIFEST indicates which files are to be in the distribution.
- svn:ignore indicates which files in a build tree are to be
ignored as part of "svn status" and other svn commands.
- MANIFEST.SKIP indicates the files in the repository (build tree?)
that are not part of the distribution -- i.e., not part of MANIFEST.
Since, as Allison says, there could be files in the repository
(i.e., svn:ignore *not* set) that are not part of the distribution,
MANIFEST.SKIP must contain more than just files with the svn:ignore
keyword.
Also, isn't it possible that MANIFEST could eventually include
(generated) files that belong in the distribution but aren't
in the repository? If yes, then the repository cannot be the
canonical list of files going into the distribution.
----
I would like to get rid of MANIFEST.SKIP, however, and I think
we may not be using our tool (svn) to its full advantage.
Brainstorming a bit ... isn't it possible for us to define
our own svn properties? (I'm not all that familiar with this
part of svn.) If yes, then can we just have a
"distribution:[yes|no]" svn property that indicates files to be
part of the distribution?
Then MANIFEST can be generated automatically when a release
is generated -- files with "distribution:yes" plus any
additional (generated) files that aren't part of the repository.
We can also have a test that checks that every file in the
repository has either "distribution:yes" or "distribution:no"
(and svn can probably do a lot of the work here).
Or, perhaps a little fancier, in that we have a property
"distribution:[parrot|main|doc|perl6|APL|TCL]", that can be
used to construct different distributions from a common
repository.
Or, if we don't want to use svn properties, perhaps simply
a master REPOSITORY.MANIFEST file that lists all of the files
found in the repository and the distribution disposition
of each (skip, release, parrot, whatever).
Ultimately I think we have to wrap our collective heads around
the idea that "repository" != "release", and that the MANIFEST
file is really tied to releases and not the repository. And
that MANIFEST.SKIP was intended to make it easier to manage
and test MANIFEST, but that we may now have better ways to
do this.
Either that, or let's leave things as they are and close this
ticket. (I did say I was doing this for bug day. :-)
Thoughts? I may have more to add to this discussion after
going through the release process this week. And my apologies
if I've muddied the waters by commenting on things from incomplete
knowledge.
Pm
Will Coleda
On Feb 17, 2007, at 10:14 AM, Patrick R. Michaud via RT wrote:
> Just working on bug-day tickets (especially since I'm the release-
> manager this month :-)...
>
> On Wed Jan 31 13:37:54 2007, all...@perl.org wrote:
>> The primary purpose of a MANIFEST in the repository is to tell you
>> which files out of the repository should be included in the
>> distribution.
Yes, but our MANIFEST is just a list of everything in the repository
(modulo a single directory that we started skipping in r12600).
>> MANIFEST.skip could be simply generated from the svn:ignore keyword
>> before the distribution tarball is created.
>
> Perhaps I'm just confused, but these two statements seem to be a
> little at odds with each other, unless we say that MANIFEST.SKIP is
> generated from "svn:ignore and other sources".
>
> If I'm understanding correctly:
>
> - MANIFEST indicates which files are to be in the distribution.
>
> - svn:ignore indicates which files in a build tree are to be
> ignored as part of "svn status" and other svn commands.
>
> - MANIFEST.SKIP indicates the files in the repository (build tree?)
> that are not part of the distribution -- i.e., not part of MANIFEST.
>
> Since, as Allison says, there could be files in the repository
> (i.e., svn:ignore *not* set) that are not part of the distribution,
> MANIFEST.SKIP must contain more than just files with the svn:ignore
> keyword.
>
> Also, isn't it possible that MANIFEST could eventually include
> (generated) files that belong in the distribution but aren't
> in the repository? If yes, then the repository cannot be the
> canonical list of files going into the distribution.
Yup. Makes sense. So "die" is too strong, but "fixed" isn't.
For the moment, disabling configure's manicheck by default would be a
good start. This isn't something we need every developer wasting
their time on given how we're using it right now; more of something
that should be handled by the release manager.
(How many times has the build been broken by a missing MANIFEST update?)
> ----
>
> I would like to get rid of MANIFEST.SKIP, however, and I think
> we may not be using our tool (svn) to its full advantage.
> Brainstorming a bit ... isn't it possible for us to define
> our own svn properties? (I'm not all that familiar with this
> part of svn.) If yes, then can we just have a
> "distribution:[yes|no]" svn property that indicates files to be
> part of the distribution?
"coming soon" to svn, not available yet.
--
Will "Coke" Coleda
wi...@coleda.com
Joshua Isom
On Feb 18, 2007, at 7:14 PM, Will Coleda wrote:
>
> On Feb 17, 2007, at 10:14 AM, Patrick R. Michaud via RT wrote:
>
> Yes, but our MANIFEST is just a list of everything in the repository
> (modulo a single directory that we started skipping in r12600).
It's best purpose is probably to make sure that your svn checkout was
complete, or that your tarball contained everything it was suppose to.
It's a nice neat way to figure out that the file you need to compile
everything isn't there because your checkout was bad or your tarball
was incomplete. It serves a good purpose for a fresh
checkout/download, even if after that it's annoying.
>> Also, isn't it possible that MANIFEST could eventually include
>> (generated) files that belong in the distribution but aren't
>> in the repository? If yes, then the repository cannot be the
>> canonical list of files going into the distribution.
>
> Yup. Makes sense. So "die" is too strong, but "fixed" isn't.
If you manually update MANIFEST, then yes, it's possible. If you use
one of the programs to generate MANIFEST which relies on svn doing the
dirty work, then no, since only those files in the repository would be
in the MANIFEST.
> For the moment, disabling configure's manicheck by default would be a
> good start. This isn't something we need every developer wasting their
> time on given how we're using it right now; more of something that
> should be handled by the release manager.
You mean you don't use a little script to automatically run
Configure.pl with your desired arguments, including --nomanicheck?
> (How many times has the build been broken by a missing MANIFEST
> update?)
If the MANIFEST is regenerated on the server whenever a file is added
or deleted, it should stay accurate. But I have no clue how badly that
would complicate things(add a file and instantly need to svn up).
Jerry Gay
>
> On Feb 18, 2007, at 7:14 PM, Will Coleda wrote:
>
> >
> > On Feb 17, 2007, at 10:14 AM, Patrick R. Michaud via RT wrote:
> >
> > Yes, but our MANIFEST is just a list of everything in the repository
> > (modulo a single directory that we started skipping in r12600).
>
> It's best purpose is probably to make sure that your svn checkout was
> complete, or that your tarball contained everything it was suppose to.
> It's a nice neat way to figure out that the file you need to compile
> everything isn't there because your checkout was bad or your tarball
> was incomplete. It serves a good purpose for a fresh
> checkout/download, even if after that it's annoying.
>
that means a release version. we should definitely include a manifest
with a parrot release. and it's straightforward to modify the release
instructions to make sure a manifest is generated, and that it's
correct and complete.
it's also possible using this strategy to have an automated daily
tarball of current trunk which includes a manifest as well, but i'd
like to leave the discussion of that extension for a future date.
> > For the moment, disabling configure's manicheck by default would be a
> > good start. This isn't something we need every developer wasting their
> > time on given how we're using it right now; more of something that
> > should be handled by the release manager.
>
> You mean you don't use a little script to automatically run
> Configure.pl with your desired arguments, including --nomanicheck?
>
> > (How many times has the build been broken by a missing MANIFEST
> > update?)
>
i don't think manifest checking should be disabled until we have a
replacement solution in place that makes sure the manifest is checked
before every release. fortunately, the solution is simple. updating
release instructions with a procedure to check manifest is sufficient,
and --nomanifest can be enabled on Configure.pl by default.
in the future, this option should be renamed to eliminate the negative.
> If the MANIFEST is regenerated on the server whenever a file is added
> or deleted, it should stay accurate. But I have no clue how badly that
> would complicate things(add a file and instantly need to svn up).
>
i don't recommend a repository hook for this--the overhead you mention
of required update after file addition/deletion isn't worth the
trouble. the main use of a manifest, as mentioned above, is for
verifying the contents of a release, not a repository. and that
requirement is for end users, not developers.
~jerry
Patrick R. Michaud
> >> For the moment, disabling configure's manicheck by default would be a
> >> good start.
>
> replacement solution in place that makes sure the manifest is checked
> before every release. fortunately, the solution is simple. updating
> release instructions with a procedure to check manifest is sufficient,
> and --nomanifest can be enabled on Configure.pl by default.
Except that for someone who downloads a tarball, the default
should probably be to check the manifest.
Or, perhaps we could use the shiny new Makefile.PL script (chromatic++)
to differentiate. Running Configure.pl defaults to --nomanifest,
running "perl Makefile.PL" checks the manifest.
Pm
Allison Randal via RT
> > If I'm understanding correctly:
> >
> > - MANIFEST indicates which files are to be in the distribution.
> >
> > - svn:ignore indicates which files in a build tree are to be
> > ignored as part of "svn status" and other svn commands.
> >
> > - MANIFEST.SKIP indicates the files in the repository (build tree?)
> > that are not part of the distribution -- i.e., not part of MANIFEST.
Aye.
MANIFEST.SKIP is just a safeguard to prevent extraneous files from
leaking into the distribution tarball. It may contain both files
committed to the repository but not included in the distribution and
files generated by the build process but not included in the distribution.
So, it doesn't entirely correspond to the svn:ignore property.
> For the moment, disabling configure's manicheck by default would be a
> good start. This isn't something we need every developer wasting
> their time on given how we're using it right now; more of something
> that should be handled by the release manager.
>
> (How many times has the build been broken by a missing MANIFEST update?)
I'd agree, but then we're likely to have developers bitten by files left
out of the distribution because they forgot to include them in the
MANIFEST. Requiring developers to explicitly note which checked in files
are excluded from the MANIFEST is good housekeeping. This falls in a
category of "developer tests" that don't need to be run by the users,
but do need to be run by the developers.
> > I would like to get rid of MANIFEST.SKIP, however,
It serves a useful purpose, as a "double accounting" check.
> > and I think
> > we may not be using our tool (svn) to its full advantage.
> > Brainstorming a bit ... isn't it possible for us to define
> > our own svn properties? (I'm not all that familiar with this
> > part of svn.) If yes, then can we just have a
> > "distribution:[yes|no]" svn property that indicates files to be
> > part of the distribution?
>
> "coming soon" to svn, not available yet.
Maybe someday. For now we're stuck with a manual MANIFEST.
Allison