info
Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss
[gentoo-dev] Using emerge-webrsync to simplify the handbook
286 views
Skip to first unread message
Maxim Kammerer
Nov 28, 2012, 9:20:02 AM11/28/12
to
On Wed, Nov 28, 2012 at 3:54 PM, Richard Yao <ry...@gentoo.org> wrote:
> We could slightly simplify the handbook installation procedure if we
> told people to use emerge-webrsync to fetch the initial snapshot.
Using emerge-webrsync also makes the installation process more robust,
> We could slightly simplify the handbook installation procedure if we
> told people to use emerge-webrsync to fetch the initial snapshot.
since it only requires HTTP access (whereas many firewalls restrict
RSYNC). Besides, emerge-webrsync can check PGP signatures, so I think
that it should be the primary recommended portage tree synchronization
method.
--
Maxim Kammerer
Liberté Linux: http://dee.su/liberte
Richard Yao
Nov 28, 2012, 10:20:02 AM11/28/12
to
On 11/28/2012 09:17 AM, Maxim Kammerer wrote:
> On Wed, Nov 28, 2012 at 3:54 PM, Richard Yao <ry...@gentoo.org> wrote:
>> We could slightly simplify the handbook installation procedure if we
>> told people to use emerge-webrsync to fetch the initial snapshot.
>
> Using emerge-webrsync also makes the installation process more robust,
> since it only requires HTTP access (whereas many firewalls restrict
> RSYNC). Besides, emerge-webrsync can check PGP signatures, so I think
> that it should be the primary recommended portage tree synchronization
> method.
>
The only downside of which I am aware is increased network traffic.
> On Wed, Nov 28, 2012 at 3:54 PM, Richard Yao <ry...@gentoo.org> wrote:
>> We could slightly simplify the handbook installation procedure if we
>> told people to use emerge-webrsync to fetch the initial snapshot.
>
> Using emerge-webrsync also makes the installation process more robust,
> since it only requires HTTP access (whereas many firewalls restrict
> RSYNC). Besides, emerge-webrsync can check PGP signatures, so I think
> that it should be the primary recommended portage tree synchronization
> method.
>
However, we could redesign emerge-webrsync to take advantage of GNU
Tar's incremental archive functionality.
That would permit us to mirror compressed diffs in addition to regular
portage snapshots. Doing that well could reduce bandwidth requirements.
Michał Górny
Nov 28, 2012, 1:00:02 PM11/28/12
to
the webrsync tarball. Therefore, it is very slow and not worth
the effort when syncing often.
However, I'm not aware of gnu tar's incremental archive. If it's much
faster than the above, then it should probably replace
emerge-delta-webrsync.
--
Best regards,
Michał Górny
Markos Chandras
Nov 29, 2012, 4:30:02 AM11/29/12
to
--
Regards,
Markos Chandras / Gentoo Linux Developer / Key ID: B4AFF2C2
Sven Vermeulen
Nov 30, 2012, 6:50:01 AM11/30/12
to
On Nov 29, 2012 10:24 AM, "Markos Chandras" <hwoa...@gentoo.org> wrote:
> > We could slightly simplify the handbook installation procedure if we
> > told people to use emerge-webrsync to fetch the initial snapshot. What
> > do people think?
> >
>
> Seems a good improvement to me.
I'm ok with it as well. I'll draft up something. I suspect emerge-webrsync is available on all platforms?
Richard Yao
Nov 30, 2012, 11:10:01 AM11/30/12
to
scheme does not seem like a bad choice to me. My main concern is that
maintaining weekly full snapshots would require too much space for the
mirrors. It might be better go monthly, with diffs on the following
intervals:
1 week
1 day
30 minutes
Doing that would eliminate the benefit of rsync entirely, with the
caveat that we now need to mirror a ton of diffs. This would make it
easy for us to provide the ability to obtain historical snapshots, which
would be nice.
Michael Mol
Nov 30, 2012, 11:20:04 AM11/30/12
to
allowing HTTP proxies to cache data, reducing load on mirrors. And
moves us in the direction of implementing mirrors themselves as a
network of caching proxies.
--
:wq
Ian Stakenvicius
Nov 30, 2012, 12:10:03 PM11/30/12
to
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Idea makes sense, I wonder if implementation would be better served by
leveraging the fact that we already produce daily full snapshots:
1 - continue to provide the daily snapshots we do now
2 - provide two weeks (more?) of daily diffs, such that a daily
snapshot from up to two weeks ago can be updated to present day
3 - provide hourly or 30-minute update diffs to get latest changes.
If the tree is more than two weeks old, emerge-webrsync would just
grab the latest daily plus the hourly diffs.
If the tree is less than two weeks old, grab the daily diffs and
hourly diffs. The local copy of the tree itself would need to be
rolled back to the best-available daily diff before these diff updates
could be applied; this may mean that a local cache of the latest
full-day snapshot needs to be kept and/or generated. Also if said
cache doesn't exist, then the whole full-day snapshot would be grabbed.
The advantage to this would be significantly fewer distfiles, although
the logic in emerge-webrsync would possibly be more complex.
Regarding rolling back the local tree to a known-good state, I think
that would be required regardless of the method as any local changes
made to the tree by users would need to be discarded, right?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)
iF4EAREIAAYFAlC45f0ACgkQ2ugaI38ACPChKgD9GOBptQ9jJ1/eYyq1NEl5Oq1E
dVy9UOab80bG5FZB9LwBAKwsifnT+iE3n/4d/ljnuT2qCnbtXNYr7yBjF/VcEpkq
=y9eB
-----END PGP SIGNATURE-----
Hash: SHA256
leveraging the fact that we already produce daily full snapshots:
1 - continue to provide the daily snapshots we do now
2 - provide two weeks (more?) of daily diffs, such that a daily
snapshot from up to two weeks ago can be updated to present day
3 - provide hourly or 30-minute update diffs to get latest changes.
If the tree is more than two weeks old, emerge-webrsync would just
grab the latest daily plus the hourly diffs.
If the tree is less than two weeks old, grab the daily diffs and
hourly diffs. The local copy of the tree itself would need to be
rolled back to the best-available daily diff before these diff updates
could be applied; this may mean that a local cache of the latest
full-day snapshot needs to be kept and/or generated. Also if said
cache doesn't exist, then the whole full-day snapshot would be grabbed.
The advantage to this would be significantly fewer distfiles, although
the logic in emerge-webrsync would possibly be more complex.
Regarding rolling back the local tree to a known-good state, I think
that would be required regardless of the method as any local changes
made to the tree by users would need to be discarded, right?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)
iF4EAREIAAYFAlC45f0ACgkQ2ugaI38ACPChKgD9GOBptQ9jJ1/eYyq1NEl5Oq1E
dVy9UOab80bG5FZB9LwBAKwsifnT+iE3n/4d/ljnuT2qCnbtXNYr7yBjF/VcEpkq
=y9eB
-----END PGP SIGNATURE-----
Maxim Kammerer
Nov 30, 2012, 12:20:01 PM11/30/12
to
On Fri, Nov 30, 2012 at 5:57 PM, Richard Yao <ry...@gentoo.org> wrote:
> My main concern is that
> maintaining weekly full snapshots would require too much space for the
> mirrors.
Mirrors already keep last 8 full daily snapshots, in 2 formats each, e.g.:
> My main concern is that
> maintaining weekly full snapshots would require too much space for the
> mirrors.
http://mirrors.kernel.org/gentoo/snapshots/
Alec Warner
Nov 30, 2012, 12:40:02 PM11/30/12
to
the stuff you just said?
Note that from an infra POV our existing system works fairly well and
requires no day-to-day tinkering.
I'm always happy to consider new options, but work needs to put in to
make it feasible. I'm sure if we switched to http with zsync or
something, we could make it feasible. I want to see a prototype, etc.
-A
Zac Medico
Nov 30, 2012, 12:40:02 PM11/30/12
to
On 11/28/2012 09:50 AM, Michał Górny wrote:
> On Wed, 28 Nov 2012 10:05:55 -0500
> Richard Yao <ry...@gentoo.org> wrote:
>
>> On 11/28/2012 09:17 AM, Maxim Kammerer wrote:
>>> On Wed, Nov 28, 2012 at 3:54 PM, Richard Yao <ry...@gentoo.org> wrote:
>>>> We could slightly simplify the handbook installation procedure if we
>>>> told people to use emerge-webrsync to fetch the initial snapshot.
>>>
>>> Using emerge-webrsync also makes the installation process more robust,
>>> since it only requires HTTP access (whereas many firewalls restrict
>>> RSYNC). Besides, emerge-webrsync can check PGP signatures, so I think
>>> that it should be the primary recommended portage tree synchronization
>>> method.
>>>
>>
>> The only downside of which I am aware is increased network traffic.
>> However, we could redesign emerge-webrsync to take advantage of GNU
>> Tar's incremental archive functionality.
>>
>> That would permit us to mirror compressed diffs in addition to regular
>> portage snapshots. Doing that well could reduce bandwidth requirements.
>
> There's emerge-delta-webrsync but it's mostly hand-work to reconstruct
> the webrsync tarball. Therefore, it is very slow and not worth
> the effort when syncing often.
At least because I maintain emerge-delta-webrsync, I use it regularly as
> On Wed, 28 Nov 2012 10:05:55 -0500
> Richard Yao <ry...@gentoo.org> wrote:
>
>> On 11/28/2012 09:17 AM, Maxim Kammerer wrote:
>>> On Wed, Nov 28, 2012 at 3:54 PM, Richard Yao <ry...@gentoo.org> wrote:
>>>> We could slightly simplify the handbook installation procedure if we
>>>> told people to use emerge-webrsync to fetch the initial snapshot.
>>>
>>> Using emerge-webrsync also makes the installation process more robust,
>>> since it only requires HTTP access (whereas many firewalls restrict
>>> RSYNC). Besides, emerge-webrsync can check PGP signatures, so I think
>>> that it should be the primary recommended portage tree synchronization
>>> method.
>>>
>>
>> The only downside of which I am aware is increased network traffic.
>> However, we could redesign emerge-webrsync to take advantage of GNU
>> Tar's incremental archive functionality.
>>
>> That would permit us to mirror compressed diffs in addition to regular
>> portage snapshots. Doing that well could reduce bandwidth requirements.
>
> There's emerge-delta-webrsync but it's mostly hand-work to reconstruct
> the webrsync tarball. Therefore, it is very slow and not worth
> the effort when syncing often.
my sync method. Latest versions of emerge-delta-webrsync use a temp
directory inside $PORTAGE_TMPDIR/portage, on which I have a tmpfs
filesystem mounted. With tmpfs, performance does not seem so bad (using
a sandy bridge core i5 here).
> However, I'm not aware of gnu tar's incremental archive. If it's much
> faster than the above, then it should probably replace
> emerge-delta-webrsync.
emerge-delta-webrsync, then it seems like a good idea. It would be nice
to integrate it directly into emerge-webrsync, and eventually deprecate
emerge-delta-webrsync.
--
Thanks,
Zac
Ian Stakenvicius
Nov 30, 2012, 1:10:02 PM11/30/12
to
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Hash: SHA256
On 30/11/12 12:30 PM, Alec Warner wrote:
>> How about we not change the docs until someone eagerly implements
>> all the stuff you just said?
>
Well, using emerge-webrsync for grabbing the initial snapshot during
>> How about we not change the docs until someone eagerly implements
>> all the stuff you just said?
>
an installation still makes sense regardless of whether or not we do
any of the above, iirc that was the original documentation request
change wasn't it?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)
QVpnQa7CRF33qVYvYkkBAKDUx6bB2LbPmGQgFm22su72i32tKfUqiKOgEs6tgboa
=CNSD
-----END PGP SIGNATURE-----
Alec Warner
Dec 1, 2012, 1:10:01 AM12/1/12
to
On Fri, Nov 30, 2012 at 10:06 AM, Ian Stakenvicius <a...@gentoo.org> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> On 30/11/12 12:30 PM, Alec Warner wrote:
>>> How about we not change the docs until someone eagerly implements
>>> all the stuff you just said?
>>
>
> Well, using emerge-webrsync for grabbing the initial snapshot during
> an installation still makes sense regardless of whether or not we do
> any of the above, iirc that was the original documentation request
> change wasn't it?
Zac can probably comment on where it fetches from. I can say with some
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> On 30/11/12 12:30 PM, Alec Warner wrote:
>>> How about we not change the docs until someone eagerly implements
>>> all the stuff you just said?
>>
>
> Well, using emerge-webrsync for grabbing the initial snapshot during
> an installation still makes sense regardless of whether or not we do
> any of the above, iirc that was the original documentation request
> change wasn't it?
certainty that we have enough rsync capacity, I can't say the same for
HTTP based services.
-A
Zac Medico
Dec 1, 2012, 3:10:01 AM12/1/12
to
On 11/30/2012 09:55 PM, Alec Warner wrote:
> On Fri, Nov 30, 2012 at 10:06 AM, Ian Stakenvicius <a...@gentoo.org> wrote:
> On 30/11/12 12:30 PM, Alec Warner wrote:
>>>>> How about we not change the docs until someone eagerly implements
>>>>> all the stuff you just said?
>>>>
>
> Well, using emerge-webrsync for grabbing the initial snapshot during
> an installation still makes sense regardless of whether or not we do
> any of the above, iirc that was the original documentation request
> change wasn't it?
>
>> Zac can probably comment on where it fetches from. I can say with some
>> certainty that we have enough rsync capacity, I can't say the same for
>> HTTP based services.
I just did a quick test on 107 http mirrors returned from mirrorselect
> On Fri, Nov 30, 2012 at 10:06 AM, Ian Stakenvicius <a...@gentoo.org> wrote:
> On 30/11/12 12:30 PM, Alec Warner wrote:
>>>>> How about we not change the docs until someone eagerly implements
>>>>> all the stuff you just said?
>>>>
>
> Well, using emerge-webrsync for grabbing the initial snapshot during
> an installation still makes sense regardless of whether or not we do
> any of the above, iirc that was the original documentation request
> change wasn't it?
>
>> Zac can probably comment on where it fetches from. I can say with some
>> certainty that we have enough rsync capacity, I can't say the same for
>> HTTP based services.
--list-only, and found that 102 returned a mirror://snapshots/index.html
containing the string "portage-2012".
--
Thanks,
Zac
Richard Yao
Dec 1, 2012, 3:40:01 AM12/1/12
to
On 11/30/2012 01:06 PM, Ian Stakenvicius wrote:
> On 30/11/12 12:30 PM, Alec Warner wrote:
>>> How about we not change the docs until someone eagerly implements
>>> all the stuff you just said?
>
>
> Well, using emerge-webrsync for grabbing the initial snapshot during
> an installation still makes sense regardless of whether or not we do
> any of the above, iirc that was the original documentation request
> change wasn't it?
>
>
>
That is correct.
> On 30/11/12 12:30 PM, Alec Warner wrote:
>>> How about we not change the docs until someone eagerly implements
>>> all the stuff you just said?
>
>
> Well, using emerge-webrsync for grabbing the initial snapshot during
> an installation still makes sense regardless of whether or not we do
> any of the above, iirc that was the original documentation request
> change wasn't it?
>
>
>
Richard Yao
Dec 1, 2012, 3:50:02 AM12/1/12
to
On 11/30/2012 12:30 PM, Alec Warner wrote:
> How about we not change the docs until someone eagerly implements all
> the stuff you just said?
>
> Note that from an infra POV our existing system works fairly well and
> requires no day-to-day tinkering.
> I'm always happy to consider new options, but work needs to put in to
> make it feasible. I'm sure if we switched to http with zsync or
> something, we could make it feasible. I want to see a prototype, etc.
>
> -A
The initial suggestion was to use emerge-webrsync to fetch the initial
> How about we not change the docs until someone eagerly implements all
> the stuff you just said?
>
> Note that from an infra POV our existing system works fairly well and
> requires no day-to-day tinkering.
> I'm always happy to consider new options, but work needs to put in to
> make it feasible. I'm sure if we switched to http with zsync or
> something, we could make it feasible. I want to see a prototype, etc.
>
> -A
portage snapshot, which requires no code changes.
A suggestion was made in response was that we should take this a step
further by using emerge-webrsync for everything. I feel that the code
needs improvements before it can formally replace `emerge --sync` as our
recommended way of updating the tree. Others seem to agree.
That does not impact our ability to use emerge-webrsync to fetch the
initial snapshot in the handbook, which is my original suggestion.
Robin H. Johnson
Dec 1, 2012, 5:50:02 PM12/1/12
to
On Fri, Nov 30, 2012 at 09:35:07AM -0800, Zac Medico wrote:
> > However, I'm not aware of gnu tar's incremental archive. If it's much
> > faster than the above, then it should probably replace
> > emerge-delta-webrsync.
> If it has benefits over the current diffball approach used by
> emerge-delta-webrsync, then it seems like a good idea. It would be nice
> to integrate it directly into emerge-webrsync, and eventually deprecate
> emerge-delta-webrsync.
I went and did a rough comparison of Tar incrementals vs the existing
> > However, I'm not aware of gnu tar's incremental archive. If it's much
> > faster than the above, then it should probably replace
> > emerge-delta-webrsync.
> If it has benefits over the current diffball approach used by
> emerge-delta-webrsync, then it seems like a good idea. It would be nice
> to integrate it directly into emerge-webrsync, and eventually deprecate
> emerge-delta-webrsync.
deltas.
TL;DR:
======
- Existing deltas are 8-9x better than other options.
- We should consider retaining monthly snapshots, plus all the deltas.
Results:
========
1.
Using bzip2 -9 compression:
- Existing deltas are 9x smaller than tar-incremental.
- Existing deltas are 8x smaller than rsync-batch.
2.
If you just want to save bandwidth, the average full snapshot,
compressed w/ BZIP2, is 55M. The average delta is 269k.
55M/269k = ~209.
Ergo it is LESS bandwidth to download ~180 deltas and apply those than
it is to download the full snapshot (assuming upstream side of the
transaction accounts for ~30 snapshots worth of overhead).
Notes:
======
1.
Extracting tar incrementals, you must be VERY careful to perform
operations in the correct order, otherwise removed files will not
actually be deleted.
2.
When the Git repo goes live, we should tag at the point we take the
daily snapshot, and use this to also consider git bundles.
Numbers:
========
Baseline tarball:
57919736 portage-20121123.0.tar.bz2
Tar incrementals, daily:
2554334 portage-20121123-20121124.1.tar.bz2
2045216 portage-20121124-20121125.1.tar.bz2
1936313 portage-20121125-20121126.1.tar.bz2
2355342 portage-20121126-20121127.1.tar.bz2
2063612 portage-20121127-20121128.1.tar.bz2
2582600 portage-20121128-20121129.1.tar.bz2
2720135 portage-20121129-20121130.1.tar.bz2
Rsync incrementals, daily:
2224311 portage-20121123-20121124.rsync-batch.bz2
1869241 portage-20121124-20121125.rsync-batch.bz2
1802648 portage-20121125-20121126.rsync-batch.bz2
1936937 portage-20121126-20121127.rsync-batch.bz2
1868771 portage-20121127-20121128.rsync-batch.bz2
2240386 portage-20121128-20121129.rsync-batch.bz2
2028207 portage-20121129-20121130.rsync-batch.bz2
Existing deltas, daily:
252400 snapshot-20121123-20121124.patch.bz2
267094 snapshot-20121124-20121125.patch.bz2
161136 snapshot-20121125-20121126.patch.bz2
225349 snapshot-20121126-20121127.patch.bz2
245804 snapshot-20121127-20121128.patch.bz2
232549 snapshot-20121128-20121129.patch.bz2
332835 snapshot-20121129-20121130.patch.bz2
Rsync incrementals, from baseline:
2224311 portage-20121123-20121124.rsync-batch.bz2
2536620 portage-20121123-20121125.rsync-batch.bz2
2700715 portage-20121123-20121126.rsync-batch.bz2
2986403 portage-20121123-20121127.rsync-batch.bz2
3258723 portage-20121123-20121128.rsync-batch.bz2
3824015 portage-20121123-20121129.rsync-batch.bz2
4232674 portage-20121123-20121130.rsync-batch.bz2
--
Robin Hugh Johnson
Gentoo Linux: Developer, Trustee & Infrastructure Lead
E-Mail : rob...@gentoo.org
GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
Sven Vermeulen
Dec 8, 2012, 3:50:02 PM12/8/12
to
First, I removed the references to the stage3 snapshots on the "Universal
CD" as I don't think we have a (recent enough) Universal CD, and we would
recommend the stage3 files from our mirrors anyway.
Second, the Portage tree snapshots are now installed through emerge-webrsync
(which means the entire section on downloading the tarballs, checking
integrity, extracting is now a single paragraph).
Finally, the section on updating the Portage tree (using emerge --sync) is
now marked as optional, with the remark that if you're behind a firewall you
can safely ignore this section as the user will already have a quite
up-to-date tree installed.
I don't know if we should remove the section altogether (about emerge
--sync) or not. It is a small step and users *will* create bug reports about
it if they don't notice it in the documentation anymore. Marking it optional
seems to be a good approach here.
Wkr,
Sven Vermeulen
PS Commit was made a few minutes ago, so please give it an hour before it
shows up on the site.
Duncan
Dec 8, 2012, 7:50:01 PM12/8/12
to
Sven Vermeulen posted on Sat, 08 Dec 2012 20:41:42 +0000 as excerpted:
> On Fri, Nov 30, 2012 at 10:44:31AM -0500, Richard Yao wrote:
>> On 11/30/2012 06:46 AM, Sven Vermeulen wrote:
>> > On Nov 29, 2012 10:24 AM, "Markos Chandras" <hwoa...@gentoo.org>
>> > wrote:
>> > > > We could slightly simplify the handbook installation procedure if
>> > > > we told people to use emerge-webrsync to fetch the initial
>> > > > snapshot. What do people think?
> Finally, the section on updating the Portage tree (using emerge --sync)
> is now marked as optional, with the remark that if you're behind a
> firewall you can safely ignore this section as the user will already
> have a quite up-to-date tree installed.
>
> I don't know if we should remove the section altogether (about emerge
> --sync) or not. It is a small step and users *will* create bug reports
> about it if they don't notice it in the documentation anymore. Marking
> it optional seems to be a good approach here.
++ on optional.
IMHO, one of the finer points of a good handbook-based install is that it
"sets the user up for success" by encouraging correct routine maintenance
practices in the future, stepping them thru the process once during the
initial install. Unfortunately, there's /way/ too many users that read
only the handbook's install section (and perhaps Working with Gentoo) and
never read the rest, so instilling at least some minimal basics of good
routine practices there is important.
Thus, even if the webrsync process grabbed them a reasonably fresh (24
hours) initial tree, keeping the now optional section on updating the
tree to the very latest is a good idea, since even if it's optional,
it'll step the new user thru the process at least once, establishing a
good idea of that routine basic as a good practice for future updates.
...
For the same reason (but likely more controversially), I'd actually argue
that the install section should (optionally?) step thru installing
gentoolkit and running revdep-rebuild and emerge --depclean (with an
appropriate --ask/--pretend first) as the (near) final steps of the
initial install, as well, even if gentoolkit is the only thing in @world
when they do so, and it's technically unnecessary as there should be
nothing to rebuild or depclean. For depclean especially, establishing
the practice right at the beginning will help to ensure that @world
always contains all desired "leaf" packages, and that as a result, users
will never accumulate a huge backlog of "uncovered" packages subject to
getting stale due to lack of @world updating, and that an initial depclean
run when they DO get around to it won't "accidentally" uninstall half
their system, because it never got in @world to begin with. (Of course,
a successful emerge run now includes a helpful depclean reminder anyway,
but stepping the user thru the process once during the initial guided
install certainly can't hurt.)
A quick review of the installation section in the manual suggests either
covering this as part of chapter 11, Finalizing your Gentoo Installation,
or instead, inserting it as chapter 11, bumping the current 11 and 12 to
12 and 13. I'd suggest the latter, with a title for the new chapter 11
of something like "Your first emerge --update @world". It could
encourage readers to look at the working with gentoo and working with
portage sections for more, but would be an initial step-thru of a basic
emerge --update @world and related maintenance (allowing a condensing of
the corresponding chapters in the other sections), for those who never
read past install in the handbook. Additionally, with such a title it'd
be easier for users to refer back to (and see the encouragement to read
the other sections again, when they maybe have more time to do so) for
their second update, etc, if they needed to.
One more suggestion while I'm at it. I remember all those many years ago
when I first installed gentoo, even after reading and absorbing the
handbook well enough to be pointing people to specific chapters and
subchapters in answer to specific questions on the user list, I was still
somewhat confused about the exact nature of the world list, and what
packages should be listed there vs. those that shouldn't. Unless I've
overlooked it, to this day there's still not a real good explanation of
the distinction between "leaf packages" and dependencies, and why "leaf
packages" belong in @world but dependencies don't. Something along the
lines of:
"""""
If you want to use and care about that specific package, put it in the
world list with emerge <package>, without --oneshot. If you don't really
care about that specific package and it's simply pulled in by something
else that you DO care about, but you happen to be specifically emerging
it anyway, use emerge --oneshot <package>, so it doesn't end up in the
world list and emerge --depclean can unmerge it in the future if the
packages pulling it in no longer need it.
"""""
(Actually, here, all my default emerge stub-scripts use --oneshot by
default. I use a special "2" (not oneshot) version when I WANT something
added to @world, which for me is a sort of package testing purgatory
between "I for sure want to keep it and it's in a set listed in
world_sets accordingly" and "I'm not sure I really want to keep it yet
and thus want to be reminded about it the next time I update and then run
depclean --ask to cleanup.")
As a result of my confusion, I ended up with more packages in the world
list than I really needed/wanted, and somewhere along the line, I had to
use some tool (I guess it'd be emaint --clean world, now) to weed out the
unnecessary cruft in my world list.
(Later, with kde4 and portage 2.2 with sets support, I actually went thru
and broke my world file into about a dozen individual sets by functional
category, starting with the kde sets which I sync to those in the kde
overlay with every 6-month kde4-minor bump, but also including sets such
as net-admin with traceroute, etc, and net-user with firefox, etc. This
was actually as a result of sorting out my existing world list to copy it
to my netbook, but I've kept everything in sets since then, and I've run
with an empty world list for about three years now, since everything is
in sets as enumerated in my world_sets file. That practice keeps the
sets that replace my world file REAL clean, since everything listed is in
its own functional category and each set is small enough it's immediately
clear upon cat-ing the set whether every package listed in the set is
something I actually care about or not. (For the kde sets I keep synced
with the ones in the overlay, I comment out the libs and apps I don't
need or care about, but keep them in the set, commented, for easier
syncing with the overlay versions at the next six-month bump.) I'm
really looking forward to the day when proper sets support appears in
stable portage, and sets can eliminate the unsorted world list mess for
others just as they have for me. Unfortunately, that looks to be
something of a "bluesky" enhancement bug for stable or even ~arch, but at
least 2.2's there to be unmasked for people who like me have real,
practical use for sets.)
--
Duncan - List replies preferred. No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master." Richard Stallman
> On Fri, Nov 30, 2012 at 10:44:31AM -0500, Richard Yao wrote:
>> On 11/30/2012 06:46 AM, Sven Vermeulen wrote:
>> > On Nov 29, 2012 10:24 AM, "Markos Chandras" <hwoa...@gentoo.org>
>> > wrote:
>> > > > We could slightly simplify the handbook installation procedure if
>> > > > we told people to use emerge-webrsync to fetch the initial
>> > > > snapshot. What do people think?
> Okay, I've updated the instructions.
> Second, the Portage tree snapshots are now installed through
> emerge-webrsync (which means the entire section on downloading the
> tarballs, checking integrity, extracting is now a single paragraph).
Thanks, swift. =:^)
> emerge-webrsync (which means the entire section on downloading the
> tarballs, checking integrity, extracting is now a single paragraph).
> Finally, the section on updating the Portage tree (using emerge --sync)
> is now marked as optional, with the remark that if you're behind a
> firewall you can safely ignore this section as the user will already
> have a quite up-to-date tree installed.
>
> I don't know if we should remove the section altogether (about emerge
> --sync) or not. It is a small step and users *will* create bug reports
> about it if they don't notice it in the documentation anymore. Marking
> it optional seems to be a good approach here.
IMHO, one of the finer points of a good handbook-based install is that it
"sets the user up for success" by encouraging correct routine maintenance
practices in the future, stepping them thru the process once during the
initial install. Unfortunately, there's /way/ too many users that read
only the handbook's install section (and perhaps Working with Gentoo) and
never read the rest, so instilling at least some minimal basics of good
routine practices there is important.
Thus, even if the webrsync process grabbed them a reasonably fresh (24
hours) initial tree, keeping the now optional section on updating the
tree to the very latest is a good idea, since even if it's optional,
it'll step the new user thru the process at least once, establishing a
good idea of that routine basic as a good practice for future updates.
...
For the same reason (but likely more controversially), I'd actually argue
that the install section should (optionally?) step thru installing
gentoolkit and running revdep-rebuild and emerge --depclean (with an
appropriate --ask/--pretend first) as the (near) final steps of the
initial install, as well, even if gentoolkit is the only thing in @world
when they do so, and it's technically unnecessary as there should be
nothing to rebuild or depclean. For depclean especially, establishing
the practice right at the beginning will help to ensure that @world
always contains all desired "leaf" packages, and that as a result, users
will never accumulate a huge backlog of "uncovered" packages subject to
getting stale due to lack of @world updating, and that an initial depclean
run when they DO get around to it won't "accidentally" uninstall half
their system, because it never got in @world to begin with. (Of course,
a successful emerge run now includes a helpful depclean reminder anyway,
but stepping the user thru the process once during the initial guided
install certainly can't hurt.)
A quick review of the installation section in the manual suggests either
covering this as part of chapter 11, Finalizing your Gentoo Installation,
or instead, inserting it as chapter 11, bumping the current 11 and 12 to
12 and 13. I'd suggest the latter, with a title for the new chapter 11
of something like "Your first emerge --update @world". It could
encourage readers to look at the working with gentoo and working with
portage sections for more, but would be an initial step-thru of a basic
emerge --update @world and related maintenance (allowing a condensing of
the corresponding chapters in the other sections), for those who never
read past install in the handbook. Additionally, with such a title it'd
be easier for users to refer back to (and see the encouragement to read
the other sections again, when they maybe have more time to do so) for
their second update, etc, if they needed to.
One more suggestion while I'm at it. I remember all those many years ago
when I first installed gentoo, even after reading and absorbing the
handbook well enough to be pointing people to specific chapters and
subchapters in answer to specific questions on the user list, I was still
somewhat confused about the exact nature of the world list, and what
packages should be listed there vs. those that shouldn't. Unless I've
overlooked it, to this day there's still not a real good explanation of
the distinction between "leaf packages" and dependencies, and why "leaf
packages" belong in @world but dependencies don't. Something along the
lines of:
"""""
If you want to use and care about that specific package, put it in the
world list with emerge <package>, without --oneshot. If you don't really
care about that specific package and it's simply pulled in by something
else that you DO care about, but you happen to be specifically emerging
it anyway, use emerge --oneshot <package>, so it doesn't end up in the
world list and emerge --depclean can unmerge it in the future if the
packages pulling it in no longer need it.
"""""
(Actually, here, all my default emerge stub-scripts use --oneshot by
default. I use a special "2" (not oneshot) version when I WANT something
added to @world, which for me is a sort of package testing purgatory
between "I for sure want to keep it and it's in a set listed in
world_sets accordingly" and "I'm not sure I really want to keep it yet
and thus want to be reminded about it the next time I update and then run
depclean --ask to cleanup.")
As a result of my confusion, I ended up with more packages in the world
list than I really needed/wanted, and somewhere along the line, I had to
use some tool (I guess it'd be emaint --clean world, now) to weed out the
unnecessary cruft in my world list.
(Later, with kde4 and portage 2.2 with sets support, I actually went thru
and broke my world file into about a dozen individual sets by functional
category, starting with the kde sets which I sync to those in the kde
overlay with every 6-month kde4-minor bump, but also including sets such
as net-admin with traceroute, etc, and net-user with firefox, etc. This
was actually as a result of sorting out my existing world list to copy it
to my netbook, but I've kept everything in sets since then, and I've run
with an empty world list for about three years now, since everything is
in sets as enumerated in my world_sets file. That practice keeps the
sets that replace my world file REAL clean, since everything listed is in
its own functional category and each set is small enough it's immediately
clear upon cat-ing the set whether every package listed in the set is
something I actually care about or not. (For the kde sets I keep synced
with the ones in the overlay, I comment out the libs and apps I don't
need or care about, but keep them in the set, commented, for easier
syncing with the overlay versions at the next six-month bump.) I'm
really looking forward to the day when proper sets support appears in
stable portage, and sets can eliminate the unsorted world list mess for
others just as they have for me. Unfortunately, that looks to be
something of a "bluesky" enhancement bug for stable or even ~arch, but at
least 2.2's there to be unmasked for people who like me have real,
practical use for sets.)
--
Duncan - List replies preferred. No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master." Richard Stallman
Rich Freeman
Dec 8, 2012, 8:20:01 PM12/8/12
to
On Sat, Dec 8, 2012 at 3:41 PM, Sven Vermeulen <sw...@gentoo.org> wrote:
> Second, the Portage tree snapshots are now installed through emerge-webrsync
> (which means the entire section on downloading the tarballs, checking
> integrity, extracting is now a single paragraph).
Uh, does emerge-webrsync have some kind of automagical detection of
> Second, the Portage tree snapshots are now installed through emerge-webrsync
> (which means the entire section on downloading the tarballs, checking
> integrity, extracting is now a single paragraph).
the fact that you're building a chroot? I would think that it would
sync /usr/portage, and not /mnt/gentoo/usr/portage. Perhaps you
should move that down into the chroot section, and mkdir /usr/portage
if that is needed?
Rich
Sven Vermeulen
Dec 9, 2012, 8:40:01 AM12/9/12
to
On Sat, Dec 08, 2012 at 08:10:32PM -0500, Rich Freeman wrote:
> Uh, does emerge-webrsync have some kind of automagical detection of
> the fact that you're building a chroot? I would think that it would
> sync /usr/portage, and not /mnt/gentoo/usr/portage. Perhaps you
> should move that down into the chroot section, and mkdir /usr/portage
> if that is needed?
Crap. Indeed, section moved towards the place where we optionally recommend
> Uh, does emerge-webrsync have some kind of automagical detection of
> the fact that you're building a chroot? I would think that it would
> sync /usr/portage, and not /mnt/gentoo/usr/portage. Perhaps you
> should move that down into the chroot section, and mkdir /usr/portage
> if that is needed?
"emerge --sync", and put in an mkdir /usr/portage.
Wkr,
Sven Vermeulen
0 new messages