Re: [gentoo-dev] Moving our/portage stuff to var

[Tomáš Chvátal]

2012/12/17 Diego Elio Pettenò <flam...@flameeyes.eu>:
> On 17/12/2012 11:19, Tomáš Chvátal wrote:
>>
>> I've always myself override these defaults in make.conf to point for
>> /var/portage/ (not /var/lib because I never bothered enough how to
>> make world and config files to be put elsewhere :P).
>
> I would say let's work on that so that portage can keep them there.
> Although I'm more for /var/cache/portage myself, as both distfiles and
> tree can be re-generated.
>
Thats right, it should be even better location. :-)

[Tomáš Chvátal]

Currently we put portage into /usr/portage and all related stuff is to
be in the subfolders there (distfiles, binpkg).

I've always myself override these defaults in make.conf to point for
/var/portage/ (not /var/lib because I never bothered enough how to
make world and config files to be put elsewhere :P).

The only reason why we have this currently in usr is that bsd ports
put their stuff in there and I suppose Daniel just did the same.

With respect to reality how stuff is done in the linux land all the
variable data should be in /var so we should adjust and move it in
there too.

What would you think?

Cheers

Tom

[Diego Elio Pettenò]

On 17/12/2012 11:19, Tomáš Chvátal wrote:
>

> I've always myself override these defaults in make.conf to point for
> /var/portage/ (not /var/lib because I never bothered enough how to
> make world and config files to be put elsewhere :P).

I would say let's work on that so that portage can keep them there.
Although I'm more for /var/cache/portage myself, as both distfiles and
tree can be re-generated.

--
Diego Elio Pettenò — Flameeyes
flam...@flameeyes.eu — http://blog.flameeyes.eu/

[Ben de Groot]

I prefer some location in /var/ as well.

--
Cheers,

Ben | yngwin
Gentoo developer
Gentoo Qt project lead, Gentoo Wiki admin

[Michał Górny]

On Mon, 17 Dec 2012 11:23:00 +0100
Diego Elio Pettenò <flam...@flameeyes.eu> wrote:

> On 17/12/2012 11:19, Tomáš Chvátal wrote:
> >
> > I've always myself override these defaults in make.conf to point for
> > /var/portage/ (not /var/lib because I never bothered enough how to
> > make world and config files to be put elsewhere :P).
>
> I would say let's work on that so that portage can keep them there.
> Although I'm more for /var/cache/portage myself, as both distfiles and
> tree can be re-generated.

+1 on /var/cache.

--
Best regards,
Michał Górny

[justin]

On 17/12/12 11:23, Diego Elio Pettenò wrote:
> On 17/12/2012 11:19, Tomáš Chvátal wrote:
>>
>> I've always myself override these defaults in make.conf to point for
>> /var/portage/ (not /var/lib because I never bothered enough how to
>> make world and config files to be put elsewhere :P).
>
> I would say let's work on that so that portage can keep them there.
> Although I'm more for /var/cache/portage myself, as both distfiles and
> tree can be re-generated.
>

fetch-restricted files are to be considered critical here. Do we want to
force the user to keep them twice? So an additional location which is
not a "cache"?
Of course PORTAGE_RO_DISTDIRS and friends are nice here, but they are
not part of a default setup.

justin

[Luca Barbato]

Agreed.

Bonus points if we consider suggesting to move it on a dedicated file
system ^^;

lu

[Diego Elio Pettenò]

On 17/12/2012 12:06, justin wrote:
> fetch-restricted files are to be considered critical here. Do we want to
> force the user to keep them twice? So an additional location which is
> not a "cache"?
> Of course PORTAGE_RO_DISTDIRS and friends are nice here, but they are
> not part of a default setup.

I would still think they are cache. I can re-download Oracle's JRE
binaries; Portage's copy is a cache because I don't need to back it up.

[Dirkjan Ochtman]

On Mon, Dec 17, 2012 at 11:23 AM, Diego Elio Pettenò
<flam...@flameeyes.eu> wrote:
> I would say let's work on that so that portage can keep them there.
> Although I'm more for /var/cache/portage myself, as both distfiles and
> tree can be re-generated.

+1.

Cheers,

Dirkjan

[Derek Dai]

+1 /var/cache

Derek Dai

[Markos Chandras]

+1 sounds good to me.

--
Regards,
Markos Chandras / Gentoo Linux Developer / Key ID: B4AFF2C2

[Philipp Riegger]

On 17.12.2012 11:23, Diego Elio Pettenò wrote:
> On 17/12/2012 11:19, Tomáš Chvátal wrote:
>>
>> I've always myself override these defaults in make.conf to point for
>> /var/portage/ (not /var/lib because I never bothered enough how to
>> make world and config files to be put elsewhere :P).
>
> I would say let's work on that so that portage can keep them there.
> Although I'm more for /var/cache/portage myself, as both distfiles and
> tree can be re-generated.
>

With this change distfiles and packages could be moved to a directory
which is not a subdir of portage. Something like

/var/cache/{portage,distfiles,packages}

or

/var/cache/portage/{tree,distfiles,packages}

since the file types and storage requirements are so different. At least
I prefer not to have too many filesystems mounted inside each other.

Philipp

[George Shapovalov]

On Monday 17 December 2012 11:19:20 Tomáš Chvátal wrote:
> I've always myself override these defaults in make.conf to point for
> /var/portage/ (not /var/lib because I never bothered enough how to
> make world and config files to be put elsewhere :P).

Finally!
And, while we are at it, lets more distfiles out of portage. That is the only
(prominent) dir inside that has drastically different storage requirements.
Having portage on a separate partition requires now changing defaults, bind
mounts or symlinks. It is better to have it done right from the onset then to
workaround every time.

Oh, and +1 on /var/cache/portage for the location.

George

[Diego Elio Pettenò]

On 17/12/2012 13:39, justin wrote:
> I am more thinking about packages which are not as easy accessible as
> JRE. There are a couple sci packages which are distributed on request by
> mail other inconvenient methods. Sometimes even not by your own, but by
> your PI or other seniors. And even sometimes upstreams doesn't
> distribute them anymore at all.

In which case you should keep them somewhere safer anyway, which is what
I do for that kind of files.

Remember that eclean acts on distfiles as well.

[justin]

On 17/12/12 12:17, Diego Elio Pettenò wrote:
> On 17/12/2012 12:06, justin wrote:
>> fetch-restricted files are to be considered critical here. Do we want to
>> force the user to keep them twice? So an additional location which is
>> not a "cache"?
>> Of course PORTAGE_RO_DISTDIRS and friends are nice here, but they are
>> not part of a default setup.
>
> I would still think they are cache. I can re-download Oracle's JRE
> binaries; Portage's copy is a cache because I don't need to back it up.
>

[Diego Elio Pettenò]

On 17/12/2012 13:42, Ulrich Mueller wrote:
> If we change the location, can we then move distfiles to some place
> outside of the tree? Something like:
>
> /var/cache/portage
> /var/cache/distfiles

What I do on my systems is

/var/cache/portage/tree
/var/cache/portage/distfiles

[Ulrich Mueller]

If we change the location, can we then move distfiles to some place
outside of the tree? Something like:

/var/cache/portage
/var/cache/distfiles

Rationale for this is that the tree with many small files and
distfiles (which tend to be large) have very different parameters for
filesystem optimisation, when one of them (or both) are kept on a
separate partition.

Ulrich

[Tiziano Müller]

Am Montag, den 17.12.2012, 11:19 +0100 schrieb Tomáš Chvátal:
> Currently we put portage into /usr/portage and all related stuff is to
> be in the subfolders there (distfiles, binpkg).
>
> I've always myself override these defaults in make.conf to point for
> /var/portage/ (not /var/lib because I never bothered enough how to
> make world and config files to be put elsewhere :P).

I always move the stuff as well:

* /var/cache/distfiles
* /var/cache/packages ... may not be the best choice since they can't
always be regenerated
* /var/db/repositories/portage
* /var/db/repositories/... (other portage repositories)
* /var/db/paludis/repositories/... (for paludis-specific repositories,
like layman)

Tiziano

[Diego Elio Pettenò]

On 17/12/2012 14:40, Kevin Chadwick wrote:
> /var sounds right but if /usr is still huge it may annoy some (like
> apt can) with smaller drives who now need lots of free space for new
> programs in both /usr and /var. Of course there is LVM.

Changing our defaults is unlikely to force users to change their settings.

> On OpenBSD the Auto partition map suggests /usr/ports /usr/src as
> seperate partitions as long as you have a fair amount of space. It
> possibly even suggests a seperate obj partition. The benefit being you
> can mkfs/newfs much quicker than deleting many many files. Security (DAC
> permission avoidance) and nuking more than what you wanted obviously
> needs consideration for that kind of function.

Honestly I would never take what OpenBSD does to face value.

But in general this is a call for users — myself I have been keeping
them split on the tinderbox host but merged into the rootfs for the laptops.

[Kevin Chadwick]

On Mon, 17 Dec 2012 11:19:20 +0100
Tomáš Chvátal <tomas....@gmail.com> wrote:

> The only reason why we have this currently in usr is that bsd ports
> put their stuff in there and I suppose Daniel just did the same.

>> +1 on /var/cache.
>>
>>> Agreed.

>>>Bonus points if we consider suggesting to move it on a dedicated
>>>file system ^^;

/var sounds right but if /usr is still huge it may annoy some (like
apt can) with smaller drives who now need lots of free space for new
programs in both /usr and /var. Of course there is LVM.

On OpenBSD the Auto partition map suggests /usr/ports /usr/src as
seperate partitions as long as you have a fair amount of space. It
possibly even suggests a seperate obj partition. The benefit being you
can mkfs/newfs much quicker than deleting many many files. Security (DAC
permission avoidance) and nuking more than what you wanted obviously
needs consideration for that kind of function.

So it's probably a user exercise?

[Rich Freeman]

On Mon, Dec 17, 2012 at 8:40 AM, Kevin Chadwick <ma1l...@yahoo.co.uk> wrote:
> So it's probably a user exercise?

It already is a user exercise. A stage3 doesn't even contain the
/usr/portage directory - you manually create it per the handbook (or
more likely let tar/etc do it for you.

I also would like to see distfiles moved. Ideally the package tree
should be a perfect copy of what is on the rsync mirrors. It seems a
bit odd to stick other stuff in there, which needs special treatment
as a result.

To the extent that this isn't already supported, portage should simply
let you set the location in make.conf.

I'd also suggest at least considering how paludis handles this. They
just have a directory containing config file per repository, with a
priority setting. The portage tree is just another overlay, which is
a good way to handle it. The sync mechanism handles the main tree
identically to overlays as a result, though you can specify what to
sync.

Rich

[Diego Elio Pettenò]

On 17/12/2012 14:49, Rich Freeman wrote:
> I'd also suggest at least considering how paludis handles this. They
> just have a directory containing config file per repository, with a
> priority setting. The portage tree is just another overlay, which is
> a good way to handle it. The sync mechanism handles the main tree
> identically to overlays as a result, though you can specify what to
> sync.

Let's not conflate two completely different changes with two completely
different work required to deal with them.

Changing our default locations, and the documentation is quick. Changing
the way the syncing/handling is done is a nightmare.

If we conflate the issue, we can stop discussing as we're never ever
going to go anywhere.

[Marc Schiffbauer]

Am Montag, 17. Dezember 2012, 11:23:00 schrieb Diego Elio Pettenò:
> On 17/12/2012 11:19, Tomáš Chvátal wrote:
> > I've always myself override these defaults in make.conf to point for
> > /var/portage/ (not /var/lib because I never bothered enough how to
> > make world and config files to be put elsewhere :P).
>
> I would say let's work on that so that portage can keep them there.
> Although I'm more for /var/cache/portage myself, as both distfiles and
> tree can be re-generated.

What about setups where portage tree is mounted via NFS to reduce traffic and
disk space?

FHS states[1] that /var/cache is *locally* generated. Which aould not be the
case for such setups...

I prefer /var as well, but I am not such if /var/cache would be the right
place...


[1]
From FHS 2.3:
"/var/cache is intended for cached data from applications. Such data is
locally generated as a result of time-consuming I/O or calculation."

-Marc
--
0x35A64134 - 8AAC 5F46 83B4 DB70 8317 3723 296C 6CCA 35A6 4134

[Diego Elio Pettenò]

On 17/12/2012 15:49, Marc Schiffbauer wrote:
> What about setups where portage tree is mounted via NFS to reduce traffic and
> disk space?

Since nothing in Gentoo and/or other distributions _enforces_ FHS,
you're allowed to do as you prefer....

> FHS states[1] that /var/cache is *locally* generated. Which aould not be the
> case for such setups...

Again, you can do as you prefer. Do you wish to violate FHS to just keep
in the usual place? You can. You want to move it somewhere else to
adhere to FHS? You can.

> I prefer /var as well, but I am not such if /var/cache would be the right
> place...

Any other suggestions on where to place it? And please don't say
/var/lib because that would usually be backed up.

[Marc Schiffbauer]

Am Montag, 17. Dezember 2012, 15:56:11 schrieb Diego Elio Pettenò:
> On 17/12/2012 15:49, Marc Schiffbauer wrote:
> > What about setups where portage tree is mounted via NFS to reduce traffic
> > and disk space?
>
> Since nothing in Gentoo and/or other distributions _enforces_ FHS,
> you're allowed to do as you prefer....
>
> > FHS states[1] that /var/cache is *locally* generated. Which aould not be
> > the case for such setups...
>
> Again, you can do as you prefer. Do you wish to violate FHS to just keep
> in the usual place? You can. You want to move it somewhere else to
> adhere to FHS? You can.
>
> > I prefer /var as well, but I am not such if /var/cache would be the right
> > place...
>
> Any other suggestions on where to place it? And please don't say
> /var/lib because that would usually be backed up.

FHS also states:

"[...] Other portions may be shared [between systems], notably /var/mail,
/var/cache/man, /var/cache/fonts, and /var/spool/news."

So I think this might indeed be interpreted like that /var/cache/portage would
be perfectly ok.

Another place I could imagine is /var/portage because of its fundamental
importance in gentoo.

FHS about that: "Applications must generally not add directories to the top
level of /var. Such directories should only be added if they have some system-
wide implication, and in consultation with the FHS mailing list."

I think this would also be ok because portage can be counted as "system-wide
implication" ...

[Anthony G. Basile]

+1 on the idea of moving portage and +1 on the idea of /var/cache

--
Anthony G. Basile, Ph.D.
Gentoo Linux Developer [Hardened]
E-Mail : blue...@gentoo.org
GnuPG FP : 8040 5A4D 8709 21B1 1A88 33CE 979C AF40 D045 5535
GnuPG ID : D0455535

[Brian Dolbec]

On Mon, 2012-12-17 at 11:23 +0100, Diego Elio Pettenò wrote:
> On 17/12/2012 11:19, Tomáš Chvátal wrote:
> >
> > I've always myself override these defaults in make.conf to point for
> > /var/portage/ (not /var/lib because I never bothered enough how to
> > make world and config files to be put elsewhere :P).
>
> I would say let's work on that so that portage can keep them there.
> Although I'm more for /var/cache/portage myself, as both distfiles and
> tree can be re-generated.
>

I've been pushing for the portage tree to move somewhere in /var for
awhile.

Also I think the tree and layman overlays should be under the same
master directory, then portage, pkgcore,... could just quickly scan the
master directory & sub directories to auto-add the valid overlays there.
No need to add them to PORTDIR_OVERLAY

something like

/var/cache/repositories/
/var/cache/repositories/gentoo
/var/cache/repositories/x11 <== overlay
/var/cache/repositories/local <== overlay
/var/cache/repositories/distfiles <== move it out of the tree dir.
/var/cache/repositories/packages <== move it out of the tree dir.

--
Brian Dolbec <dol...@gentoo.org>

[Brian Dolbec]

On Mon, 2012-12-17 at 13:47 +0100, Diego Elio Pettenò wrote:
> On 17/12/2012 13:39, justin wrote:
> > I am more thinking about packages which are not as easy accessible as
> > JRE. There are a couple sci packages which are distributed on request by
> > mail other inconvenient methods. Sometimes even not by your own, but by
> > your PI or other seniors. And even sometimes upstreams doesn't
> > distribute them anymore at all.
>
> In which case you should keep them somewhere safer anyway, which is what
> I do for that kind of files.
>
> Remember that eclean acts on distfiles as well.
>

which is why eclean has a config file where you can add files/pkgs,
patterns to exclude from being cleaned.
--
Brian Dolbec <dol...@gentoo.org>

[Brian Dolbec]

On Mon, 2012-12-17 at 15:56 +0100, Diego Elio Pettenò wrote:
> On 17/12/2012 15:49, Marc Schiffbauer wrote:
> > What about setups where portage tree is mounted via NFS to reduce traffic and
> > disk space?
>
> Since nothing in Gentoo and/or other distributions _enforces_ FHS,
> you're allowed to do as you prefer....
>
> > FHS states[1] that /var/cache is *locally* generated. Which aould not be the
> > case for such setups...
>
> Again, you can do as you prefer. Do you wish to violate FHS to just keep
> in the usual place? You can. You want to move it somewhere else to
> adhere to FHS? You can.
>
> > I prefer /var as well, but I am not such if /var/cache would be the right
> > place...
>
> Any other suggestions on where to place it? And please don't say
> /var/lib because that would usually be backed up.
>

then /var/repositories/ similar to my previous reply. It is very clear
by the name what it's purpose is. Also name the portage tree dir gentoo
like it's repo_name and all but one of the layman overlays available to
install.
--
Brian Dolbec <dol...@gentoo.org>

[Diego Elio Pettenò]

On 17/12/2012 16:51, Brian Dolbec wrote:
>> >
> then /var/repositories/ similar to my previous reply. It is very clear
> by the name what it's purpose is. Also name the portage tree dir gentoo
> like it's repo_name and all but one of the layman overlays available to
> install.

Erm, why should we invent something new on top of var which is something
that about everywhere it's suggested NOT to do?

Using /var/lib clearly spells "back this up yourself".
Using /var/cache clearly spells "can be lost without consequences".
Using /var/tmp clearly spells "things will be deleted".

/var/repositories sounds just like "NIH", seriously.

Somebody already proposed "/var/db" — that probably makes more sense if
you want to go that route, although I wouldn't put distfiles or packages
there.

[Brian Dolbec]

On Mon, 2012-12-17 at 15:02 +0100, Diego Elio Pettenò wrote:
> On 17/12/2012 14:49, Rich Freeman wrote:
> > I'd also suggest at least considering how paludis handles this. They
> > just have a directory containing config file per repository, with a
> > priority setting. The portage tree is just another overlay, which is
> > a good way to handle it. The sync mechanism handles the main tree
> > identically to overlays as a result, though you can specify what to
> > sync.
>
> Let's not conflate two completely different changes with two completely
> different work required to deal with them.
>
> Changing our default locations, and the documentation is quick. Changing
> the way the syncing/handling is done is a nightmare.
>
> If we conflate the issue, we can stop discussing as we're never ever
> going to go anywhere.
>

I agree.

But for the purpose of answering Rich, layman is also capable of syncing
the portage tree, although, not using the round robin and other user
sync variations portage is capable of. Once I finish the python 3
compatibility code changes. I will re-visit getting layman's api use
into portage, pkgcore for them to use to sync the overlays from within
the package manager. I have already demonstrated how easy it is for
portage to run layman's api. An in tree example is app-portage/esearch
which adding the -l parameter will use the layman api if available or
fall back to running layman in a subprocess to sync the overlays as well
as the portage tree.
--
Brian Dolbec <dol...@gentoo.org>

[Ciaran McCreesh]

On Mon, 17 Dec 2012 15:56:11 +0100

Diego Elio Pettenò <flam...@flameeyes.eu> wrote:

> Any other suggestions on where to place it? And please don't say
> /var/lib because that would usually be backed up.

/var/db

--
Ciaran McCreesh

["Paweł Hajdan, Jr."]

On 12/17/12 2:19 AM, Tomáš Chvátal wrote:
> With respect to reality how stuff is done in the linux land all the
> variable data should be in /var so we should adjust and move it in
> there too.
>
> What would you think?

Fully seconded.

+1 to /var/cache/portage and having distfiles outside of the portage
tree directory.

Paweł

[Ultrabug]

+1

Ultra

[Zac Medico]

On 12/17/2012 02:19 AM, Tomáš Chvátal wrote:
> Currently we put portage into /usr/portage and all related stuff is to
> be in the subfolders there (distfiles, binpkg).
>

> I've always myself override these defaults in make.conf to point for
> /var/portage/ (not /var/lib because I never bothered enough how to
> make world and config files to be put elsewhere :P).
>

> The only reason why we have this currently in usr is that bsd ports
> put their stuff in there and I suppose Daniel just did the same.
>

> With respect to reality how stuff is done in the linux land all the
> variable data should be in /var so we should adjust and move it in
> there too.
>
> What would you think?

I like the idea. As noted in bug #378603 [1], I'd like the portage
ebuild to ensure that the locations don't unexpectedly change for
existing installs.

Will it break catalyst? If so, we might begin the migration by fixing
catalyst and having to set the new default locations in the make.conf
that it generates.

[1] https://bugs.gentoo.org/show_bug.cgi?id=378603#c1
--
Thanks,
Zac

[Zac Medico]

[Rick "Zero_Chaos" Farina]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Yes, it will break catalyst. However, if the folder that the portage
tree goes in is still something/something/portage then it shouldn't be
hard to fix.

- -ZC
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)
Comment: Using GnuPG with undefined - http://www.enigmail.net/

iQIcBAEBAgAGBQJQ0MjSAAoJEKXdFCfdEflKTZ4P/2h5iKIGbNWhxAGkWEDRez31
YiCVUf5lJeDGAYpJ0hM20aWebzrR+HITqcKENKfGwerr+2/VeKVKsyYR8M1tNbQs
9jCNTMBfoYOEm8ElOu0Vzny5uWV3LflP0BMsnOjBDUpxw9ESXuz+54tkknsyEaLD
yBoVoe4zj4GfY/B3aXO08gqig/w7sGR3iLPyhy3iJimznK/qMszyMJ9LzCFVSmwz
zXVMoaM1VKTwVI7E/CZmxBWIF4KVoQ1fl4xJoqt4l82HK/5IDoRLu+s5c6rJLDhp
SPDsFQa+XtdHOZvaXhn+15zAcWFJm8UBdXtnfi8tJmpoukEJr3BscC9DRUCdRzbR
mzE0ptqDayoJ8tKuxUR+ppU987PPIAJ03h/X3yTWsIc7QPB1TtFI5W6mk4/iVPSy
FF9LrcxYvAM/+Cqu+LlhKIapqK+XAA7nw6txpEjiw3HcTZ+0O/GGHTlKM0gxbx0i
kMGe00zwodoN7DwGYenmulDk3eX71moZ/ioXMMNisDAazmcbOKOEXC/pmuUk342H
9oUtgYCH/PJGjdZHAlQnBwyfIP2YCNRZEnL6lN5t6V+p0XPzMB97tJCYsB2qEifx
m1qOPZhQX5dXovKJZyrs93SbpkUhMF8TXi6IsosVtAQFldRd2LJOawRBEkZzF2Yn
mn0gQl0ORyorOMEYsxbL
=lZrh
-----END PGP SIGNATURE-----

[Rick "Zero_Chaos" Farina]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 12/18/2012 02:49 PM, Rick "Zero_Chaos" Farina wrote:
> On 12/18/2012 01:38 PM, Zac Medico wrote:
>> On 12/17/2012 02:19 AM, Tomáš Chvátal wrote:
>>> Currently we put portage into /usr/portage and all related stuff is to
>>> be in the subfolders there (distfiles, binpkg).
>>>
>>> I've always myself override these defaults in make.conf to point for
>>> /var/portage/ (not /var/lib because I never bothered enough how to
>>> make world and config files to be put elsewhere :P).
>>>
>>> The only reason why we have this currently in usr is that bsd ports
>>> put their stuff in there and I suppose Daniel just did the same.
>>>
>>> With respect to reality how stuff is done in the linux land all the
>>> variable data should be in /var so we should adjust and move it in
>>> there too.
>>>
>>> What would you think?
>
>> I like the idea. As noted in bug #378603 [1], I'd like the portage
>> ebuild to ensure that the locations don't unexpectedly change for
>> existing installs.
>
>> Will it break catalyst? If so, we might begin the migration by fixing
>> catalyst and having to set the new default locations in the make.conf
>> that it generates.
>
>> [1] https://bugs.gentoo.org/show_bug.cgi?id=378603#c1
>
> Yes, it will break catalyst. However, if the folder that the portage
> tree goes in is still something/something/portage then it shouldn't be
> hard to fix.
>

It should probably be mentioned (since most of us don't use the
snapshots every day) that the snapshots actually contain a folder called
"portage" in the tarball. Not that it would be impossible to change,
but if we migrate from /usr/portage to /var/whatever/portage then the
changes are trivial, if we migrate to /var/repositories/gentoo that
makes things like unpacking the snapshots significantly non-trivial.

I really don't care what everyone wants to do here (although I'm
generally for sticking closer to FHS), but I warn that if the path
doesn't end in "portage" the changes are going to be significantly
non-trivial.

Thanks,
Zero

> -ZC
>

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)
Comment: Using GnuPG with undefined - http://www.enigmail.net/

iQIcBAEBAgAGBQJQ0MwqAAoJEKXdFCfdEflKF0YQAJ1mBkAPgDcIdXL+2eIdc5GO
yk7t2qSOtSaU159C8d2xI2AdB3b5jm5AXWrrvpAovNIAMTN/fFa781oKQVkFprsS
AZsT3UFluLep3ngLVTvpv8nry2t0AhT+/k1u1GAeurnTT5b4GDGXu7EF+QHzeaLC
+8VF8/N0TvRp+PwZ9lEJGXd+RVXxQ6AJXo1XsIbgKnA/L5mCKCfYn3o+9oQCqHbW
fPVeGyqEAKVYFG8rNUHwxPHYUvxhIC7xxuBl81ErXv5VqhcgNNjX4SMnKXeT3Z/V
FRxtnLJMz69mlX9C+CMf9boFVfveMjGj7pZOzopJGGhJM3d8Q/rAbgHejEmtrlAd
n9FBLc7ihGGBGGcTHrmhG+1DalmY/c2WaR63NfR+Le5mIpPpC0mibhK5TYeWRAcF
zFJhjmLwBfyAlBLnQk6gd4x/c8jMH68wgvRqPNMJbWETkT3tntgl7cUcoesv6TdT
fGNuISGCqMSEHHTKxX9XvqBl5EDSNEIS4I7rbFwO3qtuHuR2+88H506E9RQU4Xdd
+/bV6EW3Z7s/9hp7LhvK5CqEXAfJwN278aJOKAuyNX/YmwKa1VXy/ZDMXEUrTa6x
r+LcgyNHUkY3Sr4OFR7yoIY/oH4FrP1Y5H45GQUNje/XeL7bnLoN9h4QfeIwwt/L
6KZZtj3hmJa7I/m5hBwv
=VeR7
-----END PGP SIGNATURE-----

[Florian D.]

Am Mon, 17 Dec 2012 13:45:35 +0100
schrieb Diego Elio Pettenò <flam...@flameeyes.eu>:

> On 17/12/2012 13:42, Ulrich Mueller wrote:
> > If we change the location, can we then move distfiles to some place
> > outside of the tree? Something like:
> >
> > /var/cache/portage
> > /var/cache/distfiles
>
> What I do on my systems is
>
> /var/cache/portage/tree
> /var/cache/portage/distfiles
>

+1 for moving the location

my suggestion:

/var/gentoo/distfiles
/var/gentoo/packages
/var/gentoo/repositories

/var/gentoo/repositories/layman
/var/gentoo/repositories/portage
.
.
this could be extended, like:
/var/gentoo/repositories/sth_paludis_related

It would be nice, if you'd adopted my scheme, then I don't need to change anything.. ;)

[Diego Elio Pettenò]

No /var/gentoo. No /var/repositories.

/var/db/gentoo, /var/db/repositories, /var/cache/portage ... as long as Zac is fine with one whatever, but let's not invent any new top-level.

Diego Elio Pettenò — Flameeyes
flam...@flameeyes.eu — http://blog.flameeyes.eu/

[Zac Medico]

On 12/18/2012 01:33 PM, Diego Elio Pettenò wrote:
> No /var/gentoo. No /var/repositories.
>
> /var/db/gentoo, /var/db/repositories, /var/cache/portage ... as long as
> Zac is fine with one whatever, but let's not invent any new top-level.

Yeah, /var/db or /var/cache sounds good to me.

I would encourage all those interested to coordinate with the catalyst
developers to have the new defaults written in the make.conf that it
generates. Having the new defaults explicitly recorded there does not
require any changes to portage (though I will bring portage's internal
defaults into sync as soon as possible). Also, I think it might help
avoid confusion for users if they are able to see the new defaults
explicitly recorded in make.conf (as opposed to
/usr/share/portage/config/make.globals which is somewhat obscure).
--
Thanks,
Zac

[Joshua Saddler]

On Mon, 17 Dec 2012 11:19:20 +0100

Tomáš Chvátal <tomas....@gmail.com> wrote:

> Currently we put portage into /usr/portage and all related stuff is to
> be in the subfolders there (distfiles, binpkg).
>
> I've always myself override these defaults in make.conf to point for
> /var/portage/ (not /var/lib because I never bothered enough how to
> make world and config files to be put elsewhere :P).
>
> The only reason why we have this currently in usr is that bsd ports
> put their stuff in there and I suppose Daniel just did the same.
>
> With respect to reality how stuff is done in the linux land all the
> variable data should be in /var so we should adjust and move it in
> there too.
>
> What would you think?

do it. stick it somewhere in /var. i have a small SLC SSD just for /var and /usr/portage partitions, since those consistently incur high writes. dropping to just one partition for all that i/o would be real nice.

if this proposed change is made, please make sure to contact the GDP. while we don't update things like manpages or elog announcements, we would have a ton of stuff to fix in gentoo.org/doc/en/ . also, make sure stuff is sorted out on the catalyst/releng end well in advance, so users aren't stuck with bad stages.

[Marc Schiffbauer]

Am Dienstag, 18. Dezember 2012, 22:33:06 schrieb Diego Elio Pettenò:
> No /var/gentoo. No /var/repositories.
>
> /var/db/gentoo, /var/db/repositories, /var/cache/portage ... as long as Zac
> is fine with one whatever, but let's not invent any new top-level.

Why not? From a FHS pov it seems ok...

I would suggest /var/portage ...

[Diego Elio Pettenò]

On 19/12/2012 13:44, Marc Schiffbauer wrote:
> Why not? From a FHS pov it seems ok...
>
> I would suggest /var/portage ...

Seriously, mine is going to be a huge veto here with as much power I can
put.

There is a _reason_ why stuff is added to /var/lib instead of having

/var/postgres
/var/mysql
/var/foobar
/var/wtf
/var/wth
/var/imtired

...

As I said on other messages before (which you probably missed since you
ask "Why not?"), putting it in /var/lib or /var/db or /var/cache makes
it explicit how you should handle its backup.

/var/portage ? I have to look it up manually.

Let's to things properly instead of how it looks cooler.

--

[Ulrich Mueller]

>>>>> On Wed, 19 Dec 2012, Diego Elio Petten� wrote:

> On 19/12/2012 13:44, Marc Schiffbauer wrote:
>> I would suggest /var/portage ...

> Seriously, mine is going to be a huge veto here with as much power I
> can put.

Why? The portage tree is of central importance for Gentoo, so IMHO a
second-level directory would be acceptable for it. Besides, it
currently is in /usr/portage, so it wouldn't be new but would only
move from /usr to /var.

> There is a _reason_ why stuff is added to /var/lib instead of having

> /var/postgres
> /var/mysql
> /var/foobar
> /var/wtf
> /var/wth
> /var/imtired

> ...

I don't understand how this is related to the discussion. None of the
above have any relevance for Gentoo that would be comparable to
Portage.

This doesn't mean that /var/portage is the only possible choice. But
IMHO it's better than some of the other suggestions that I've seen
here, like /var/cache/portage/repositories/gentoo/tree and so on.

> As I said on other messages before (which you probably missed since
> you ask "Why not?"), putting it in /var/lib or /var/db or /var/cache
> makes it explicit how you should handle its backup.

Yes, these are certainly fine, as long as we don't add additional
useless subdirectory levels.

> /var/portage ? I have to look it up manually.

Please, stay serious. ;-)

Ulrich

[Diego Elio Pettenò]

On 19/12/2012 14:43, Ulrich Mueller wrote:
> Why? The portage tree is of central importance for Gentoo, so IMHO a
> second-level directory would be acceptable for it. Besides, it
> currently is in /usr/portage, so it wouldn't be new but would only
> move from /usr to /var.

I'm irked enough by /usr/portage that using that as a reason is just
going to make me feel even more strongly that it should not be /var/portage.

> I don't understand how this is related to the discussion. None of the
> above have any relevance for Gentoo that would be comparable to
> Portage.

See above.

> This doesn't mean that /var/portage is the only possible choice. But
> IMHO it's better than some of the other suggestions that I've seen
> here, like /var/cache/portage/repositories/gentoo/tree and so on.

I'm not arguing that it should go 5 levels deep. But two or three deep
is fine for me. Is it going to be /var/db/portage/master ? Fine. Is it
going to be /var/cache/portage/tree? Fine. /var/cache/portage +
/var/cache/distfiles ? Fine.

Just mv /usr/portage /var/portage ? FFS no. Among other things, as many
said before, we should really take distfiles out of the tree itself, and
packages the same. And I don't want /var/packages or /var/distfiles at all.

>> /var/portage ? I have to look it up manually.
>
> Please, stay serious. ;-)

I am serious. If it's my first time backing up a system, and I encounter
a directory "/var/portage", it doesn't make it clear what it contains.
Is it re-generable? Should it be backed up entirely?

That's why my suggestion is to use /var/cache: it makes it clear that
there is no definitive reason to back it up (as Justin said there is an
issue with distfiles you can't re-download but that's a different story
I'd say — maybe setting a default read-only distdir for said packages
might make sense, but I don't want to get there at all).

Also, I usually keep /var/cache in a more "unsafe" disk — I don't care
if I lose cache because the drive dies, while /var/lib is fully backed
up. I don't usually split /var/db but I can see what people were saying
about having different allocation requirements for the tree compared to
distfiles, and I guess that if we put the tree there we could gain
something even for /var/db/pkg by splitting it.

Tree hierarchies are there to make things more easily organized, not
just to look nice.

[Ian Stakenvicius]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 19/12/12 08:56 AM, Diego Elio Pettenò wrote:
>
> That's why my suggestion is to use /var/cache: it makes it clear
> that there is no definitive reason to back it up (as Justin said
> there is an issue with distfiles you can't re-download but that's a
> different story I'd say — maybe setting a default read-only distdir
> for said packages might make sense, but I don't want to get there
> at all).

In terms of the fact that a current copy of the portage tree is always
available, no it isn't necessary to back it up. However, if one isn't
constantly maintaining their system via -uDN and doing say, updates on
a monthly cycle (ie, production systems), then it is very useful to
maintain the same portage tree snapshot as the system's last -uDN ...
As such I would argue that it is worthwhile to back it up.
Similarly, 'packages' should probably stay synchronized with the tree.

So in terms of the above, would that mean /var/lib is a better fit?
or would that mean /var/cache and it is up to the user to add their
own backup of /var/cache/portage ?


Distfiles, imo, are definitely just cache and can be discarded at any
time. There are issues if one has a very old tree that some distfiles
disappear from the mirrors (especially gentoo patchset tarballs) but
such is life -- personally I'd like to see all such files stored on a
dev's webspace in perpetuity so that SRC_URI could grab it from there
after it's dropped from the mirrors. As for "special" distfiles
(fetch-restricted etc), these would need to be downloaded manually
anyways and if they are of value they should be backed up elsewhere
(ie, not rely on the distfiles dir to keep them).


Either of i.e. /var/cache/{distfiles,packages,portage} or
/var/cache/portage/{distfiles,packages,tree} works for me; i can see
the extra directory level keeping all portage bits together as looking
nicer for the end user but meh.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)

iF4EAREIAAYFAlDRy8kACgkQ2ugaI38ACPAiLQD9HeENg+cPrkQcHhPF54h1AaPG
hvTvaq4GaghMNXCKV7sBAKz8cKR6LD8grvuTnftWVJiRYYbhYM+HANTaE5xWs6f+
=WPmW
-----END PGP SIGNATURE-----

[Diego Elio Pettenò]

On 19/12/2012 15:14, Ian Stakenvicius wrote:
> So in terms of the above, would that mean /var/lib is a better fit?
> or would that mean /var/cache and it is up to the user to add their
> own backup of /var/cache/portage ?

I would say it's up to the user. When I do that kind of setup I actually
use a single rsync source and tar it up from there.

[Brian Dolbec]

Yes, Catalyst and the portage defaults will be changed in a co-ordinated
manner. I've started poking around catalyst, but it has paths
hard-coded nearly everywhere, despite it having passed config variables
around most places. So, It is going to need code cleanup first. It
will take me some time to get familiar enough with the code before I
make many changes to clean it up.

For the documentation, primarily the install handbook, perhaps it would
be better to mention both locations, at least mention the old location
so existing and old users won't be thrown for a loop. It will also need
some attention in the forums where tons of threads will be
referencing /usr/portage. Perhaps the docs team could start
preparing the docs changes in a manner that will be easy to
search/replace with the final correct locations when that decision is
made.

And YES Diego, it won't be /var/portage or /var/repositories, we heard
you.

From my rough tracking, I believe somewhere under /var/cache was
majority vote. Anyway, once catalyst is ready it will be easy to set it
to whatever is finally decided.
--
Brian Dolbec <dol...@gentoo.org>

[Duncan]

Ulrich Mueller posted on Wed, 19 Dec 2012 14:43:56 +0100 as excerpted:

>>>>>> On Wed, 19 Dec 2012, Diego Elio Pettenò wrote:
>
>> On 19/12/2012 13:44, Marc Schiffbauer wrote:
>>> I would suggest /var/portage ...
>
>> Seriously, mine is going to be a huge veto here with as much power I
>> can put.
>
> Why? The portage tree is of central importance for Gentoo, so IMHO a
> second-level directory would be acceptable for it. Besides, it currently
> is in /usr/portage, so it wouldn't be new but would only move from /usr
> to /var.
>
>> There is a _reason_ why stuff is added to /var/lib instead of having
>
>> /var/postgres /var/mysql /var/foobar /var/wtf /var/wth /var/imtired
>
>> ...
>
> I don't understand how this is related to the discussion. None of the
> above have any relevance for Gentoo that would be comparable to Portage.

Seriously, people, we're talking about a user-level default, not
something there's no user-level way to change. A ...

>> huge veto here with as much power I can put

... seems rather strong for a simple default, for something already
DESIGNED for the user to put wherever they want!


FWIW, here, my tree's fully resolved to /usr/src/portage, but:

PORTDIR=/p
DISTDIR=${PORTDIR}/src
PKGDIR=/m/pw

/p -> /usr/src/portage

/p because I wanted as short a path as possible. DISTDIR is set inside
it, but to a customized srcdir name as makes more sense to me as well.
My layman toplevel is inside as well, as $PORTDIR/layman. (Yes, rsync-
exclude is set appropriately too.)

/usr/src is a dedicated partition containing PORTDIR, the kernel git tree
and a local automated kernel-patches dir (similar to /etc/portage/patches
but for the kernel), and the system ccache. Actually, it contains a
second system ccache as well, for the 32-bit chroot that's the build-
image for my netbook.

PKGDIR, /m/pw, is its own dedicated partition, with /mnt being a symlink
to /m (short paths), and pw abreviating pkg-workstation. There's another
dedicated partition, /m/nr/pn, that's the PKGDIR for my netbook's build-
image (nr indicating netbook root, pn indicating pkg-netbook).

The gentoo tree and overlays, along with DISTDIR, are common to both the
workstation and the netbook, and directly net-downloadable, so keeping src
aka DISTDIR and the layman tree inside PORTDIR, on the /usr/src
partition, makes sense to me. But I keep separate PKGDIRs in dedicated
partitions so I can have backup partitions for them, thus keeping
installable binpkgs for both the workstation and the netbook, even if the
working partitions get fscked.


The point of all this being, these settings are DESIGNED to be set by the
user, who can have an as crazy-for-others-but-makes-sense-to-the-user
configuration as they want!

Based on that, yes, the defaults have some level of symbolic importance,
but even if they're something like
/some/crazy/nth/level/weirdness/subdir/portage by default, it's *JUST*
the defaults, and if users feel strongly about it at all, they can change
them. IMO there's thus no reason to feel so strongly that one issues a...

>>huge veto here with as much power I can put

In fact, it could be argued that...

/some/crazy/nth/level/weirdness/subdir/portage

... *should* be the default, precisely to MAKE that point, that it's just
a default for something intended to be set by the user.

Not that /I'd/ argue for it, especially since the gentoo way is
customizable configuration but sane "just works" defaults where possible
and I'd hardly call that sane, but it /would/ make the point.


Anyway, yes, getting the default out of /usr/portage into /var somewhere
seems reasonable, and if I were to choose, I'd choose
/var/cache/portage as that seems to me to be the clearest labeled intent,
but it's a default INTENDED to be set to something else, should a user
feel strongly about it, and as such, the default really isn't that big of
a deal. Yes let's move it into /var, but /var/db/portage vs.
/var/cache/portage vs. /var/portage ... if people have strong feelings
about it, they'll move it anyway, so /whatever/ the default, even
something as insane as the path suggested to make the point above, it's
simply not worth having a coronary (or incinerating opposition with a
flaming stare) over.

--
Duncan - List replies preferred. No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master." Richard Stallman

[Diego Elio Pettenò]

On 19/12/2012 16:54, Brian Dolbec wrote:
>
> And YES Diego, it won't be /var/portage or /var/repositories, we heard
> you.

Thanks, it's appreciated :)

[Marc Schiffbauer]

Am Mittwoch, 19. Dezember 2012, 14:43:56 schrieb Ulrich Mueller:

> >>>>> On Wed, 19 Dec 2012, Diego Elio Pettenò wrote:
> > On 19/12/2012 13:44, Marc Schiffbauer wrote:
> >> I would suggest /var/portage ...
> >
> > Seriously, mine is going to be a huge veto here with as much power I
> > can put.
>
> Why? The portage tree is of central importance for Gentoo, so IMHO a
> second-level directory would be acceptable for it. Besides, it
> currently is in /usr/portage, so it wouldn't be new but would only
> move from /usr to /var.

+1

FHS 2.3: "Applications must generally not add directories to the top level of
/var. Such directories should only be added __if they have some system-wide
implication__, and in consultation with the FHS mailing list."

[Diego Elio Pettenò]

On 19/12/2012 17:19, Marc Schiffbauer wrote:
> FHS 2.3: "Applications must generally not add directories to the top level of
> /var. Such directories should only be added __if they have some system-wide
> implication__, and in consultation with the FHS mailing list."

Since you like adding emphasis:

FHS 2.3: "Applications must generally not add directories to the top

level of /var. Such directories should only be added if they have some
system-wide implication, *and in consultation with the FHS mailing list.*"

It's always nice to cherry-pick half a phrase from a standard, right?

[George Shapovalov]

On Wednesday 19 December 2012 15:52:28 Duncan wrote:
> >> There is a _reason_ why stuff is added to /var/lib instead of having
> >>
> >> /var/postgres /var/mysql /var/foobar /var/wtf /var/wth /var/imtired
> >>
> >> ...
> >
> > I don't understand how this is related to the discussion. None of the
> > above have any relevance for Gentoo that would be comparable to Portage.
>
> Seriously, people, we're talking about a user-level default, not
> something there's no user-level way to change. A ...

Um, no. Dont forget the "educate by example" aspect.
Seriously, how many people are going to read the fine print rather than just
skimming over and going with the defaults? Oh, sure, for those who know all
the fine details of FS layout and who manage to notice this is off and care to
change it they totally are able to do so. But most of the users (who even
notice)? It is going to be - "devs set it so, it must be right" and then we
have to combat to set things right again?

Oh, and clearly I am here with Diego: /var/portage or /var/gentoo is plain
wrong.

George

[Alan McKinnon]

On Wed, 19 Dec 2012 14:56:44 +0100

Diego Elio Pettenò <flam...@flameeyes.eu> wrote:

> Just mv /usr/portage /var/portage ? FFS no. Among other things, as
> many said before, we should really take distfiles out of the tree
> itself, and packages the same. And I don't want /var/packages
> or /var/distfiles at all.

If we are going to move distfiles out of the tree into, what are the
odds of getting /some/path/portage/local to move somewhere else too?

That one has irked me for ages, its the one thing left on my systems
that stops the local tree dir being an exact replica of the upstream
master.

--
Alan McKinnon
alan.m...@gmail.com

[Zac Medico]

On 12/19/2012 02:01 PM, Alan McKinnon wrote:
> On Wed, 19 Dec 2012 14:56:44 +0100
> Diego Elio Pettenņ <flam...@flameeyes.eu> wrote:
>
>> Just mv /usr/portage /var/portage ? FFS no. Among other things, as
>> many said before, we should really take distfiles out of the tree
>> itself, and packages the same. And I don't want /var/packages
>> or /var/distfiles at all.
>
> If we are going to move distfiles out of the tree into, what are the
> odds of getting /some/path/portage/local to move somewhere else too?

What program uses this "local" directory? It's not used directly by
portage itself, though portage has an exclude for it in the default
PORTAGE_RSYNC_OPTS setting (in /usr/share/portage/config/make.globals).

> That one has irked me for ages, its the one thing left on my systems
> that stops the local tree dir being an exact replica of the upstream
> master.

For portage's defaults, I won't settle for anything less than having
them all refer to separate directories which are *not* nested within one
other. These are the current default settings which violate my requirements:

PORTDIR=/usr/portage
DISTDIR=${PORTDIR}/distfiles
PKGDIR=${PORTDIR}/packages
RPMDIR=${PORTDIR}/rpm
--
Thanks,
Zac

[Mike Gilbert]

On Wed, Dec 19, 2012 at 5:19 PM, Zac Medico <zme...@gentoo.org> wrote:
> On 12/19/2012 02:01 PM, Alan McKinnon wrote:
>> On Wed, 19 Dec 2012 14:56:44 +0100
>> Diego Elio Pettenņ <flam...@flameeyes.eu> wrote:
>>
>>> Just mv /usr/portage /var/portage ? FFS no. Among other things, as
>>> many said before, we should really take distfiles out of the tree
>>> itself, and packages the same. And I don't want /var/packages
>>> or /var/distfiles at all.
>>
>> If we are going to move distfiles out of the tree into, what are the
>> odds of getting /some/path/portage/local to move somewhere else too?
>
> What program uses this "local" directory? It's not used directly by
> portage itself, though portage has an exclude for it in the default
> PORTAGE_RSYNC_OPTS setting (in /usr/share/portage/config/make.globals).
>

layman used to use /usr/portage/local for storing overlays. See code
listing 2.2.

http://www.gentoo.org/proj/en/overlays/userguide.xml

[Marc Schiffbauer]

Am 19.12.2012 17:25, schrieb Diego Elio Pettenò:
> On 19/12/2012 17:19, Marc Schiffbauer wrote:
>> FHS 2.3: "Applications must generally not add directories to the top
>> level of
>> /var. Such directories should only be added __if they have some
>> system-wide
>> implication__, and in consultation with the FHS mailing list."
>
> Since you like adding emphasis:
>
> FHS 2.3: "Applications must generally not add directories to the top
> level of /var. Such directories should only be added if they have some
> system-wide implication, *and in consultation with the FHS mailing
> list.*"
>
> It's always nice to cherry-pick half a phrase from a standard, right?

No. That would be the next step. But I do not see any problem here. Do
you?

--

[Brian Dolbec]

On Wed, 2012-12-19 at 17:33 -0500, Mike Gilbert wrote:
> On Wed, Dec 19, 2012 at 5:19 PM, Zac Medico <zme...@gentoo.org> wrote:
> > On 12/19/2012 02:01 PM, Alan McKinnon wrote:
> >> On Wed, 19 Dec 2012 14:56:44 +0100
> >> Diego Elio Pettenņ <flam...@flameeyes.eu> wrote:
> >>
> >>> Just mv /usr/portage /var/portage ? FFS no. Among other things, as
> >>> many said before, we should really take distfiles out of the tree
> >>> itself, and packages the same. And I don't want /var/packages
> >>> or /var/distfiles at all.
> >>
> >> If we are going to move distfiles out of the tree into, what are the
> >> odds of getting /some/path/portage/local to move somewhere else too?

I am already making this configurable in catalyst which builds the
stages and install media. It is my intention to move it to the new
repositories location alongside the gentoo tree directory and any layman
installed overlays.

> >
> > What program uses this "local" directory? It's not used directly by
> > portage itself, though portage has an exclude for it in the default
> > PORTAGE_RSYNC_OPTS setting (in /usr/share/portage/config/make.globals).
> >
>
> layman used to use /usr/portage/local for storing overlays. See code
> listing 2.2.
>
> http://www.gentoo.org/proj/en/overlays/userguide.xml
>

I plan to migrate layman to the new portage/catalyst defaults when they
are done. The current default is /var/lib/layman


As I will not be the sole dictator as to what these new defaults will
be. I am working on the code to make it easy to set to whatever the
final decision is made to be.

just to clarify, I'm voting for...

/var/cache/distfiles
/var/cache/packages
/var/cache/repositories/
/var/cache/repositories/gentoo <== the main portage tree
/var/cache/repositories/local <== the new location for a local overlay
/var/cache/repositories/some-overlay <== layman installed overlay


--
Brian Dolbec <dol...@gentoo.org>

[Ulrich Mueller]

>>>>> On Wed, 19 Dec 2012, Brian Dolbec wrote:

> just to clarify, I'm voting for...

> /var/cache/distfiles
> /var/cache/packages

Fine.

> /var/cache/repositories/
> /var/cache/repositories/gentoo <== the main portage tree
> /var/cache/repositories/local <== the new location for a local overlay
> /var/cache/repositories/some-overlay <== layman installed overlay

So the Portage tree would move from the second level (/usr/portage) to
the fourth level? IMHO this is unacceptable. Make it /var/cache/gentoo
or /var/cache/portage; the /var/cache directory really isn't so
overpopulated that there's a need for hiding things in subdirs.

Also I wonder if local overlays should be in /var/cache? It might not
always be possible to restore them.

Ulrich

[Duncan]

Ulrich Mueller posted on Thu, 20 Dec 2012 09:11:39 +0100 as excerpted:

> Also I wonder if local overlays should be in /var/cache? It might not
> always be possible to restore them.

Good point. My local overlay's in /usr/local/ (which is a dedicated
partition I actually keep several backups of, including my netbook's
copy), for exactly that reason.

(Actually, /usr/local is a symlink, to a shorter path that's the actual
partition, /l, and both PORTDIR and my local overlay are simply "p" (/p
being a symlink to the gentoo tree in /usr/src), so I configure and
access the overlay as /l/p, and simply /p for the gentoo tree, but the /l
partition is also reachable via /usr/local symlink. But however it's
identified, the partition reachable as either /l or /usr/local contains
the local overlay, and is backed up multiple ways including on an
entirely separate machine, which its own copy of the same /l aka /usr/
local.)

[George Shapovalov]

On Thursday 20 December 2012 09:11:39 Ulrich Mueller wrote:
> > /var/cache/repositories/local <== the new location for a local overlay
>

> Also I wonder if local overlays should be in /var/cache? It might not
> always be possible to restore them.

So, this is the present /usr/local/portage? What's wrong with it?
According to FHS, /usr/local/ is reserved for local admin meddling, which fits
the bill rather nicely fo local overlay. This is the one location that was
actually making sense from the beginning.

Now, trying to recall its history, I seem to remember that it was not there
from the onset, but rather added when we first had this idea of overlays, and,
faintly (remember), even having some discussion with some technical arguments.
I guess this is how it ended up in the place that makes sense :).

George

[Alan McKinnon]

On Wed, 19 Dec 2012 14:19:52 -0800
Zac Medico <zme...@gentoo.org> wrote:

> On 12/19/2012 02:01 PM, Alan McKinnon wrote:
> > On Wed, 19 Dec 2012 14:56:44 +0100
> > Diego Elio Pettenò <flam...@flameeyes.eu> wrote:
> >
> >> Just mv /usr/portage /var/portage ? FFS no. Among other things, as
> >> many said before, we should really take distfiles out of the tree
> >> itself, and packages the same. And I don't want /var/packages
> >> or /var/distfiles at all.
> >
> > If we are going to move distfiles out of the tree into, what are the
> > odds of getting /some/path/portage/local to move somewhere else too?
>
> What program uses this "local" directory? It's not used directly by
> portage itself, though portage has an exclude for it in the default
> PORTAGE_RSYNC_OPTS setting
> (in /usr/share/portage/config/make.globals).

It goes back a long time, and is basically a poor man's local overlay
without having to use layman. As I understand it, portage will treat
the directory like any other when looking for ebuilds and resolving
deps, but exclude it from a sync.

>
> > That one has irked me for ages, its the one thing left on my systems
> > that stops the local tree dir being an exact replica of the upstream
> > master.
>
> For portage's defaults, I won't settle for anything less than having
> them all refer to separate directories which are *not* nested within
> one other. These are the current default settings which violate my
> requirements:
>
> PORTDIR=/usr/portage
> DISTDIR=${PORTDIR}/distfiles
> PKGDIR=${PORTDIR}/packages
> RPMDIR=${PORTDIR}/rpm

/usr/portage/local has the taste feel and smell of a hacky workaround:
shove a directory in the tree and exclude it from sync.

I suspect the best solution all round is to move all support for local
overlays into layman. I'd be happy with that. Probably make the portage
code cleaner too.



--
Alan McKinnon
alan.m...@gmail.com

[Rich Freeman]

On Thu, Dec 20, 2012 at 4:25 AM, George Shapovalov <geo...@gentoo.org> wrote:
> On Thursday 20 December 2012 09:11:39 Ulrich Mueller wrote:
>> > /var/cache/repositories/local <== the new location for a local overlay
>>
>> Also I wonder if local overlays should be in /var/cache? It might not
>> always be possible to restore them.
> So, this is the present /usr/local/portage? What's wrong with it?
> According to FHS, /usr/local/ is reserved for local admin meddling, which fits
> the bill rather nicely fo local overlay. This is the one location that was
> actually making sense from the beginning.

I actually like the /var/cache/repositories approach. You can always
add a symlink to it if you want to for convenience (as I already do
for /var/lib/portage/world). There is really nothing special about
/usr/portage, other than it being the lowest-priority overlay.

As far as the local one goes - again this is configurable in
/etc/make.conf, and we don't really need to pre-create that directory
either.

Rich

[Ulrich Mueller]

>>>>> On Thu, 20 Dec 2012, Rich Freeman wrote:

> I actually like the /var/cache/repositories approach. You can always
> add a symlink to it if you want to for convenience (as I already do
> for /var/lib/portage/world). There is really nothing special about
> /usr/portage, other than it being the lowest-priority overlay.

There's also nothing special about Earth amongst other planets,
so please add "Earth / Solar System" when addressing your letters.

SCNR,
Ulrich

[Graham Murray]

Zac Medico <zme...@gentoo.org> writes:

> On 12/19/2012 02:01 PM, Alan McKinnon wrote:
>> If we are going to move distfiles out of the tree into, what are the
>> odds of getting /some/path/portage/local to move somewhere else too?
>
> What program uses this "local" directory? It's not used directly by
> portage itself, though portage has an exclude for it in the default
> PORTAGE_RSYNC_OPTS setting (in /usr/share/portage/config/make.globals).

It is useful for 'site' local ebuilds. It allows a 'master' repository
to sync with the main Gentoo one without disturbing local changes but
allow other systems on-site to fetch the modified tree with a simple
'emerge --sync'.

[Brian Dolbec]

My idea for having all repos under one directory is to make it easier
for a pkg manager to simply scan the directory to know all installed
overlays. Currently each one has to be listed in a configured variable
in make.conf. So if you wanted your local overlay somewhere else, then
a symlink would work (provided the PM can/will autoscan repos), or add
it to the PORTDIR_OVERLAY variable (current behavior). I don't
otherwise have a strong desire for it to be there.

If and only if the tree and all overlays (not other directories) are not
under one directory, then an autoscan cannot easily happen.

For an example of what I am referring to. In layman-2.0.0 I have added
a /etc/layman/overlays directory. On start it will scan it for any
*.xml files and add those specifications to the overlays variable.
Layman-1.4 and previous required you add each entry to that overlays
variable in layman.cfg. Essentially this makes the overlays directory a
plug-in directory, so if you want to install an overlay not listed in
the main repositories list, just download the xml spec to

/etc/layman/overlays/${meaningful-name}.xml
then layman -f (adds it to it's cache)

no editing required :)

If repositories is too long a name, repos is shorter and still
meaningful, although still puts them at a 4th level.

--
Brian Dolbec <dol...@gentoo.org>

[Ian Stakenvicius]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 20/12/12 10:37 AM, Brian Dolbec wrote:
>>
>>> /var/cache/repositories/ /var/cache/repositories/gentoo <==
>>> the main portage tree /var/cache/repositories/local <== the
>>> new location for a local overlay
>>> /var/cache/repositories/some-overlay <== layman installed
>>> overlay
>

> My idea for having all repos under one directory is to make it
> easier for a pkg manager to simply scan the directory to know all
> installed overlays. Currently each one has to be listed in a
> configured variable in make.conf. So if you wanted your local
> overlay somewhere else, then a symlink would work (provided the PM
> can/will autoscan repos), or add it to the PORTDIR_OVERLAY variable
> (current behavior). I don't otherwise have a strong desire for it
> to be there.
>
> If and only if the tree and all overlays (not other directories)
> are not under one directory, then an autoscan cannot easily
> happen.
>

You could do this while not having the portage tree be in that
directory. IE, portage goes in /var/cache/portage , and all the
overlays go into /var/cache/repositories.

The tree is separate enough IMO that autoscan can still happen easily,
and also I believe that it can be assumed that the tree is in place.
For instance, if the tree's location is defined to be elsewhere, it
isn't done so via PORTDIR_OVERLAYS but rather PORTDIR.


On an unrelated note, I would never treat my "local" overlays as
cache. Ebuilds that (as a user) I wrote and installed by hand are not
likely to be kept in a repository someplace, but rather the overlay
dir would most likely be it's only location. IIRC the reason for
/usr/portage/local/ was to have a path within the portage tree that
rsync wouldn't kill; given that what you're suggesting is already not
under the proposed portage tree location, emerge --sync couldn't touch
it, and so I don't see a need at all to provide a 'local' repository
destination by default.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)

iF4EAREIAAYFAlDTNlQACgkQ2ugaI38ACPCxQAEApT/7CaIbuVTwnDQk93hhDjGu
mXKPdCJg4h1iMECtdoABAJj2601LuRPUKFJ+BJa/FqrdRTsjSpBRiEd8pvO2042P
=W3T9
-----END PGP SIGNATURE-----

[Michael Mol]

It's sounding like the nearly the optimal solution would be:

/var/cache/portage/distfiles
/var/cache/portage/repositories/gentoo
/var/cache/portage/repositories/{sunrise,kde,gnome,whatever,layman,grabs}
/var/db/portage/repositories/{non-cache,repo,names,go,here}

Clearly, some data in question needs to be treated as persistent, and
others can be treated as cache. So it should probably be divided up
that way. The placement of tree and overlays as subfolders of the same
folder strikes me as appropriate, too.

The only thing I can't see an elegant workaround for are how to avoid
or handle repo name collisions between
/var/cache/portage/repositories/* and /var/db/portage/repositories/*

--
:wq

[Diego Elio Pettenò]

On 20/12/2012 17:16, Michael Mol wrote:
>
> /var/cache/portage/distfiles
> /var/cache/portage/repositories/gentoo
> /var/cache/portage/repositories/{sunrise,kde,gnome,whatever,layman,grabs}
> /var/db/portage/repositories/{non-cache,repo,names,go,here}

+1

[Rich Freeman]

On Thu, Dec 20, 2012 at 11:19 AM, Diego Elio Pettenò
<flam...@flameeyes.eu> wrote:
> On 20/12/2012 17:16, Michael Mol wrote:
>>
>> /var/cache/portage/distfiles
>> /var/cache/portage/repositories/gentoo
>> /var/cache/portage/repositories/{sunrise,kde,gnome,whatever,layman,grabs}
>> /var/db/portage/repositories/{non-cache,repo,names,go,here}
>
> +1

Agreed.

Look, we're not going to find any place that makes everybody happy.
This one seems to be logical from a design standpoint. As long as we
make sure that everything is set in configuration then individuals can
move it wherever they want to. Symlinks also work.

Rich

[Rick "Zero_Chaos" Farina]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

> It's sounding like the nearly the optimal solution would be:
>
> /var/cache/portage/distfiles
> /var/cache/portage/repositories/gentoo
> /var/cache/portage/repositories/{sunrise,kde,gnome,whatever,layman,grabs}
> /var/db/portage/repositories/{non-cache,repo,names,go,here}

Not to oversimplify but why exactly can't we leave /usr/local/portage
where it is? I'm not going to want to cd
/var/db/portage/repositories/local every time I want to edit a local
ebuild...

- -ZC

>
> Clearly, some data in question needs to be treated as persistent, and
> others can be treated as cache. So it should probably be divided up
> that way. The placement of tree and overlays as subfolders of the same
> folder strikes me as appropriate, too.
>
> The only thing I can't see an elegant workaround for are how to avoid
> or handle repo name collisions between
> /var/cache/portage/repositories/* and /var/db/portage/repositories/*
>
> --
> :wq
>
>

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)

Comment: Using GnuPG with undefined - http://www.enigmail.net/

iQIcBAEBAgAGBQJQ0zvlAAoJEKXdFCfdEflKfiEP/0uLujqLlv1DVydqj3xXZUVq
t/c5mDsg3iJgt5T7Hm+ER949r2GUqju4veed4JQWFlVSaOoLEViL1Me/jPco5fC8
v064ktt2hOLPb+tR2IWaK3tR8i+LhcFEcIyANhl62ENPWgvOAR6V0KNFuudQLicS
QUFaJYKZkkYuPSTTqPld3QXzFwH1X6RCQaOtjCOqZKAZr9iW8HRNTTLpoa4bSMgr
VBswHyH+q0C9TzIVv5u8G8s8cYNdqHf1wrSTeMjq961tVzF3Tno5s1zk1MOyQ7cQ
MkQCxiMAum0d9PX87UkPuvHKgLdZ7e+tW26B9bS3M9yGu66lsHB7+sOTxFAJ9kqp
YKLuO2XPmpIMyDNc/5rQtTl5ygA9CmqSpUZEjMgwvCmemOHO3CsPXXQxfq6Ze7kK
/aNfCHJVEP/x8bY7PdWoexaScW/Qnqrqm6R+GCd6B3LGmTinGaDWYzJj+pAkpGUn
OAHcxATC9gX3AZr9atTFHRaPkD3L3FdYothVDZq5DDkW2qAmuhbqaEhzytDI3GLl
R+MEWYcMqvNLV5eYlNPe4OOaYfFTr/1mP0k/3ixjxJFwMDXxmIaJGTKoMOyoPu3O
diZlo0m2EPCH7Ggl9Fh0xf4P/wDSQB0AkfyldQhnVNbR5DRcTrkU6IfJBVLdcJ40
XljVqWuG27XmMXwjLGgV
=NLJg
-----END PGP SIGNATURE-----

[Ian Stakenvicius]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

> -ZC
>

IMO local user overlays would always end up being defined and placed
wherever the user decides them to be -- ie, /usr/local/portage as Rick
so nicely pointed out.

It may be a nice idea to try and enforce a structure to it/them, but
since PORTDIR_OVERLAY can be defined to include any path at all I
don't really see a point to it.


..I'm still with ulm about the tree not being in the repositories
subdir though. If I wanted to nuke all the overlays installed, "rm
- -Rf /var/cache/portage/repos/*" is very easy. If the tree is also in
that dir then it becomes less easy: "find /var/cache/portage/repos
- -maxdepth 1 -mindepth 1 -not -name gentoo -exec rm -Rf {} \+"

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)

iF4EAREIAAYFAlDTPlUACgkQ2ugaI38ACPBtPAD8DJ6BPjL6xY8alB5pbo7vQ5kb
dzRO9Z32F3r84RyVccABAIEu+k+ztw0ipoCwhmLlBHyiU6aEOsExixNvnMMLLu9X
=2gWT
-----END PGP SIGNATURE-----

[Ulrich Mueller]

>>>>> On Thu, 20 Dec 2012, Diego Elio Pettenň wrote:

>> /var/cache/portage/distfiles
>> /var/cache/portage/repositories/gentoo
>> /var/cache/portage/repositories/{sunrise,kde,gnome,whatever,layman,grabs}
>> /var/db/portage/repositories/{non-cache,repo,names,go,here}

> +1

-1

The subdirs are too deeply nested. (Ebuilds would be at the eighth
level then...)

Let's put the tree in /var/cache/portage please, and distfiles in
/var/cache/distfiles. Layman overlays can stay where they are, or move
to /var/cache/layman.

Ulrich

[Michael Mol]

On Thu, Dec 20, 2012 at 12:05 PM, Ulrich Mueller <u...@gentoo.org> wrote:
>>>>>> On Thu, 20 Dec 2012, Diego Elio Pettenň wrote:
>
>>> /var/cache/portage/distfiles
>>> /var/cache/portage/repositories/gentoo
>>> /var/cache/portage/repositories/{sunrise,kde,gnome,whatever,layman,grabs}
>>> /var/db/portage/repositories/{non-cache,repo,names,go,here}
>
>> +1
>
> -1
>
> The subdirs are too deeply nested. (Ebuilds would be at the eighth
> level then...)

Maybe I missed something...but what's wrong with that?

>
> Let's put the tree in /var/cache/portage please, and distfiles in
> /var/cache/distfiles. Layman overlays can stay where they are, or move
> to /var/cache/layman.

--
:wq

[William Hubbs]

On Thu, Dec 20, 2012 at 06:58:11AM -0500, Rich Freeman wrote:
> On Thu, Dec 20, 2012 at 4:25 AM, George Shapovalov <geo...@gentoo.org> wrote:
> > On Thursday 20 December 2012 09:11:39 Ulrich Mueller wrote:
> >> > /var/cache/repositories/local <== the new location for a local overlay
> >>
> >> Also I wonder if local overlays should be in /var/cache? It might not
> >> always be possible to restore them.
> > So, this is the present /usr/local/portage? What's wrong with it?
> > According to FHS, /usr/local/ is reserved for local admin meddling, which fits
> > the bill rather nicely fo local overlay. This is the one location that was
> > actually making sense from the beginning.
>
> I actually like the /var/cache/repositories approach. You can always
> add a symlink to it if you want to for convenience (as I already do
> for /var/lib/portage/world). There is really nothing special about
> /usr/portage, other than it being the lowest-priority overlay.

If we do this, I don't like the name repositories -- what kind of
repositories? should I put git repositories in there?

> As far as the local one goes - again this is configurable in
> /etc/make.conf, and we don't really need to pre-create that directory
> either.

Right, there is nothing to pre-create for this; it should be left to the
user.

William

[Ulrich Mueller]

>>>>> On Thu, 20 Dec 2012, Michael Mol wrote:

>>>> /var/cache/portage/distfiles
>>>> /var/cache/portage/repositories/gentoo
>>>> /var/cache/portage/repositories/{sunrise,kde,gnome,whatever,layman,grabs}
>>>> /var/db/portage/repositories/{non-cache,repo,names,go,here}

>> -1
>>
>> The subdirs are too deeply nested. (Ebuilds would be at the eighth
>> level then...)

> Maybe I missed something...but what's wrong with that?

There's no good reason for nesting it so deeply. As it is proposed
above, /var/cache/portage would contain only two subdirectories, and
/var/cache/portage/repositories only a single "gentoo" on a stable
system. Also /var/cache itself isn't overpopulated; I count about ten
entries on my systems.

We should go with a shorter (easier to remember, easier to type) path
and move things at least one level up. Two would be even better.

>> Let's put the tree in /var/cache/portage please, and distfiles in
>> /var/cache/distfiles. Layman overlays can stay where they are, or
>> move to /var/cache/layman.

Ulrich

[Vaeth]

>>> /var/cache/portage/distfiles
>>> /var/cache/portage/repositories/gentoo
>>> /var/cache/portage/repositories/{sunrise,kde,gnome,whatever,layman,grabs}
>>> /var/db/portage/repositories/{non-cache,repo,names,go,here}
>

>> +1

>
> -1
>
> The subdirs are too deeply nested.

There is a practical advantage of having the main gentoo tree
and all overlays in the same directories which was not mentioned yet:

In this way it is easy to put them in a common filesystem appropriate
for this type of data (e.g. squashfs+aufs or whatever).
The same argument holds if you want to export the tree to several
systems: Probably all overlays should then be exported as well.

However, I have strong objections against using /var/cache at all:
FHS explicitly states:

"Such data is locally generated ... The application must be able to
regenerate or restore the data."

In my opinion it is rather clear that "regenerate" does not mean
"download". Not to speak about download-restricted data which
"the application" is certainly not able to regenerate.
(That locally maintained overlays do not fall into this category
is clear anyway; but again, it would be nice to have these possibly
on the same filesystem as all other ebuilds: Especially compressing
or other deduplicating filesystem would treat this better).

Therefore I would prefer another location.

Of course, on many systems, something under /srv might be the appropriate
place (especially if the tree should be exported); this should perhaps be
suggested to the user, although it is probably not a good idea to make
this the default, because any default can just be plain wrong for the
user's system.

What is really so bad about the current location under /usr as default?
Or maybe only split somewhat differently than currently like e.g.
/usr/portage/{gentoo,sunrise,...,local}
/usr/distiles

This would avoid the deep nesting, although it would allow a
common filesystem for the data of the same type.

As far as I can see, the argument against /usr/... is only that the
data is not "constant". But actually, it is as constant as all
other data in /usr: Usually emerge --sync is only called to do
an emerge afterwards which means that the data only needs to be
modified if you intend to modify /usr anyway...

Martin

[Ciaran McCreesh]

On Thu, 20 Dec 2012 07:37:52 -0800
Brian Dolbec <dol...@gentoo.org> wrote:
> My idea for having all repos under one directory is to make it easier
> for a pkg manager to simply scan the directory to know all installed
> overlays.

That's going to cause trouble, unless we start forcing overlays to
contain enough information for a PM to configure them properly...

--
Ciaran McCreesh

[Ian Stakenvicius]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 20/12/12 12:52 PM, Vaeth wrote:
>
> However, I have strong objections against using /var/cache at all:
> FHS explicitly states:
>
> "Such data is locally generated ... The application must be able
> to regenerate or restore the data."
>

emerge --sync <-- regenerates/restores the portage tree
layman -s <-- regenerates/restores overlays
emerge -f @world <-- regenerates/restores any distfiles of consequence

download-restricted data whose sole location is in a cache dir is a
decision made by the user. Note that current portage has a
PORTAGE_RO_DISTDIR var which could be used to specify the path to
these fetch-restricted downloads rather than the user putting them
directly into /var/cache/distfiles/. Maybe it would be pertinent to
suggest and/or include a default path for this?

> What is really so bad about the current location under /usr as
> default?

There were some suggestions about this in the initial thread posts.
But /usr/portage is certainly OK to remain as-is, we do not *need* to
make any changes.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)

iF4EAREIAAYFAlDTU/AACgkQ2ugaI38ACPDmmwD/XIrHlMiDBU7WPUjp9DILJwuq
fM/y5zUVZcoD2W7kcuEA/jPV9l1BH8qsw3UQxJKi6zbjBweD0uJU28jxiX2ybdjo
=lBlR
-----END PGP SIGNATURE-----

[Ian Stakenvicius]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 20/12/12 01:18 PM, George Shapovalov wrote:

> On Thursday 20 December 2012 13:36:27 Alan McKinnon wrote:
>>> What program uses this "local" directory? It's not used
>>> directly by portage itself, though portage has an exclude for
>>> it in the default PORTAGE_RSYNC_OPTS setting (in
>>> /usr/share/portage/config/make.globals).
>>
>> It goes back a long time, and is basically a poor man's local
>> overlay without having to use layman. As I understand it, portage
>> will treat the directory like any other when looking for ebuilds
>> and resolving deps, but exclude it from a sync.

> Nope, he means /usr/portage/local, not /usr/local/portage.

Alan's description *was* for /usr/portage/local

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)

iF4EAREIAAYFAlDTVxYACgkQ2ugaI38ACPAEnQD/fo3/VbYD32yZMUuaEB0zZHES
71qGChzegSgxLqV01FoA/i583ha2AX+xLdw9/tyC7HUkzQ+9jXbqWKoT4Jay6bHc
=dt7h
-----END PGP SIGNATURE-----

[George Shapovalov]

On Thursday 20 December 2012 13:36:27 Alan McKinnon wrote:

> > What program uses this "local" directory? It's not used directly by
> > portage itself, though portage has an exclude for it in the default
> > PORTAGE_RSYNC_OPTS setting
> > (in /usr/share/portage/config/make.globals).
>
> It goes back a long time, and is basically a poor man's local overlay
> without having to use layman. As I understand it, portage will treat
> the directory like any other when looking for ebuilds and resolving
> deps, but exclude it from a sync.

Nope, he means /usr/portage/local, not /usr/local/portage. It is rarely (if
ever, nowadays) present, as I understand, but you can find it mentioned in
that config file. It may be just a legacy definition by now, looking at how
only layman was mentioned with relation to it and even that one appears not to
use it any more.

BTW, /usr/local/portage is hardly ""poor man's". Where are you going to store
your local changes that are of interest only to you and not present in any
other overlays? (like, you want to keep some old version of some package after
it has been cleaned, or your personal mods). The location even accords to FHS,
which is, apparently, a rarity :).

George

[Markos Chandras]

Yeah +1 to that. Makes more sense to me

--
Regards,
Markos Chandras / Gentoo Linux Developer / Key ID: B4AFF2C2

[Ciaran McCreesh]

On Thu, 20 Dec 2012 18:44:06 +0100
Ulrich Mueller <u...@gentoo.org> wrote:
> There's no good reason for nesting it so deeply. As it is proposed
> above, /var/cache/portage would contain only two subdirectories, and
> /var/cache/portage/repositories only a single "gentoo" on a stable
> system. Also /var/cache itself isn't overpopulated; I count about ten
> entries on my systems.

You really think most people don't have to use overlays?

> We should go with a shorter (easier to remember, easier to type) path
> and move things at least one level up. Two would be even better.

You shouldn't ever be typing that path in...

--
Ciaran McCreesh

[Michael Mol]

On Thu, Dec 20, 2012 at 12:31 PM, William Hubbs <will...@gentoo.org> wrote:
> On Thu, Dec 20, 2012 at 06:58:11AM -0500, Rich Freeman wrote:
>> On Thu, Dec 20, 2012 at 4:25 AM, George Shapovalov <geo...@gentoo.org> wrote:
>> > On Thursday 20 December 2012 09:11:39 Ulrich Mueller wrote:
>> >> > /var/cache/repositories/local <== the new location for a local overlay
>> >>
>> >> Also I wonder if local overlays should be in /var/cache? It might not
>> >> always be possible to restore them.
>> > So, this is the present /usr/local/portage? What's wrong with it?
>> > According to FHS, /usr/local/ is reserved for local admin meddling, which fits
>> > the bill rather nicely fo local overlay. This is the one location that was
>> > actually making sense from the beginning.
>>
>> I actually like the /var/cache/repositories approach. You can always
>> add a symlink to it if you want to for convenience (as I already do
>> for /var/lib/portage/world). There is really nothing special about
>> /usr/portage, other than it being the lowest-priority overlay.
>
> If we do this, I don't like the name repositories -- what kind of
> repositories? should I put git repositories in there?

I was pondering this, too. How about 'pms', for trees and overlays?

--
:wq

[Dale]

I was thinking tab completion myself. Just hit a couple letters and hit
tab. That tab key types much faster and more accurately than I can. ;-)

Dale

:-) :-)

--
I am only responsible for what I said ... Not for what you understood or how you interpreted my words!

[Ulrich Mueller]

>>>>> On Thu, 20 Dec 2012, Ciaran McCreesh wrote:

>> We should go with a shorter (easier to remember, easier to type) path
>> and move things at least one level up. Two would be even better.

> You shouldn't ever be typing that path in...

Ebuilds tell users to do so:

pkg_nofetch() {
einfo "Please download ${foo} and place it in ${DISTDIR}"
}

[George Shapovalov]

On Thursday 20 December 2012 13:21:11 Ian Stakenvicius wrote:
> > Nope, he means /usr/portage/local, not /usr/local/portage.
>
> Alan's description *was* for /usr/portage/local

Really? It matches /usr/local/portage pretty well. How did it come around
then? We had /usr/local/portage for ages for storing local changes..

[Zac Medico]

This usage is slightly annoying, because it tends to give people the
impression that it's safe to store random things inside $PORTDIR, while
it's somewhat fragile given that it relies on special rsync options.
Occasionally, we get bug reports from people who have lost files because
of this sort of confusion:

https://bugs.gentoo.org/show_bug.cgi?id=131030
https://bugs.gentoo.org/show_bug.cgi?id=392565
--
Thanks,
Zac

[Ian Stakenvicius]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

/usr/local/portage has always been a convention or recommendation;
it's not a directory that portage (package or tree) ever created,
enforced, or did anything in particular to support.

/usr/portage/local/ came around (i think -- i was around at this time
but was not a dev and was not privy to decision making) so that
locally modified ebuilds could be stored and distributed (ie via
netmount or manual rsync) along with the rest of the portage tree
without worries of the changes being wiped out on the next --sync.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)

iF4EAREIAAYFAlDTZhEACgkQ2ugaI38ACPBEXwD+IuFOgsHcQDNaqUCUfSZW53ca
7gsST6Prls/7rPmpGqcBAKnnUIH48UPcDYrwexlNbmPzRN9CjYaeR36/2qo/hC47
=r5C9
-----END PGP SIGNATURE-----

[Vaeth]

Ian Stakenvicius wrote:

>> "Such data is locally generated ... The application must be able
>> to regenerate or restore the data."
>
> emerge --sync <-- regenerates/restores the portage tree

Perhaps my English is too poor, but IMHO
download != regenerate/restore.

Indeed, the FHS also emphasizes only the *time* which it costs
to regenerate the data in /var/cache and that it can be deleted
without data loss (e.g. if your filesystem runs full).

However, if you would do this e.g. on a laptop because you
blindly trusted FHS you may have a lot of problems: Not only
can it be quite expensive to "regenerate" the data
(e.g. if you have to pay a lot for the bandwidth where you
currently are), it might even be impossible if you have no
internet connection at all or e.g. if you realize only after
the deletion that you would have to (re)emerge some package to
make the internet connection work (e.g. because you did not
revdep-rebuild before).

(And, yes, if you deal with distdir carefully, there are several
occassions when you might want to (re)emerge a package even if
you have no or only very limited internet connection; not only for
revdep-rebuild but also e.g. if you need to change certain USE-Flags.)

Of course, an experienced user knows and can care about this all,
but an experienced user will choose the location anyway as he needs:
We are speaking about the default which should mainly help also the
less experienced user.

> download-restricted data whose sole location is in a cache dir is a
> decision made by the user.

If you make /var/cache/... the default, this suggests the wrong
decision to the user. At least, it should be documented then in
a prominent place that the default path where the data is
required is probably not the appropriate path to keep it
(which IMHO is a bit strange).

> [...] PORTAGE_RO_DISTDIR [...] Maybe it would be pertinent to

> suggest and/or include a default path for this?

This would be a good idea anyway, also because of theoretically
possible name collisions in distdir.

Martin

[Zac Medico]

On 12/20/2012 03:36 AM, Alan McKinnon wrote:

> On Wed, 19 Dec 2012 14:19:52 -0800
> Zac Medico <zme...@gentoo.org> wrote:
>
>> On 12/19/2012 02:01 PM, Alan McKinnon wrote:

>>> On Wed, 19 Dec 2012 14:56:44 +0100
>>> Diego Elio Pettenņ <flam...@flameeyes.eu> wrote:
>>>
>>>> Just mv /usr/portage /var/portage ? FFS no. Among other things, as
>>>> many said before, we should really take distfiles out of the tree
>>>> itself, and packages the same. And I don't want /var/packages
>>>> or /var/distfiles at all.

>>>
>>> If we are going to move distfiles out of the tree into, what are the
>>> odds of getting /some/path/portage/local to move somewhere else too?
>>
>> What program uses this "local" directory? It's not used directly by
>> portage itself, though portage has an exclude for it in the default
>> PORTAGE_RSYNC_OPTS setting
>> (in /usr/share/portage/config/make.globals).
>

> It goes back a long time, and is basically a poor man's local overlay
> without having to use layman. As I understand it, portage will treat
> the directory like any other when looking for ebuilds and resolving
> deps, but exclude it from a sync.

Portage doesn't have any special handling for this directory, aside from
the exclude in the default PORTAGE_RSYNC_OPTS setting. I would not
encourage people to use this directory for anything, because it tends to

give people the impression that it's safe to store random things inside
$PORTDIR, while it's somewhat fragile given that it relies on special
rsync options. Occasionally, we get bug reports from people who have
lost files because of this sort of confusion:

https://bugs.gentoo.org/show_bug.cgi?id=131030
https://bugs.gentoo.org/show_bug.cgi?id=392565

>>

>>> That one has irked me for ages, its the one thing left on my systems
>>> that stops the local tree dir being an exact replica of the upstream
>>> master.
>>
>> For portage's defaults, I won't settle for anything less than having
>> them all refer to separate directories which are *not* nested within
>> one other. These are the current default settings which violate my
>> requirements:
>>
>> PORTDIR=/usr/portage
>> DISTDIR=${PORTDIR}/distfiles
>> PKGDIR=${PORTDIR}/packages
>> RPMDIR=${PORTDIR}/rpm
>
> /usr/portage/local has the taste feel and smell of a hacky workaround:
> shove a directory in the tree and exclude it from sync.

Right.

> I suspect the best solution all round is to move all support for local
> overlays into layman. I'd be happy with that. Probably make the portage
> code cleaner too.

As mentioned, portage doesn't have any special handling for this
directory (aside from the rsync exclude).
--
Thanks,
Zac

[Alan McKinnon]

On Thu, 20 Dec 2012 19:18:56 +0100
George Shapovalov <geo...@gentoo.org> wrote:

> > It goes back a long time, and is basically a poor man's local
> > overlay without having to use layman. As I understand it, portage
> > will treat the directory like any other when looking for ebuilds
> > and resolving deps, but exclude it from a sync.
> Nope, he means /usr/portage/local, not /usr/local/portage. It is
> rarely (if ever, nowadays) present, as I understand, but you can find
> it mentioned in that config file. It may be just a legacy definition
> by now, looking at how only layman was mentioned with relation to it
> and even that one appears not to use it any more.

I thought about this some more, and now realise I've been using
essentially the same set of config files since about 2004. I've never
lost them and always had a current copy so each time I build a new host
I just copy, tweak CFLAGS, maybe MAKEOPTS, and let 'er rip.

My "local" is probably years out of date. Serves me right for not
reading 50 screens of man page with every new host :-)

This sub-thread is probably just noise, sorry for that.

--
Alan McKinnon
alan.m...@gmail.com

[Ciaran McCreesh]

I believe Unix has a facility for taking bits of text that are on the
screen and copying them without the need to type the whole thing in.
But in any case, DISTDIR shouldn't be under the repository dir at all.

--
Ciaran McCreesh