Say "hi" to mini-manifests [BETA]
Daniel Robbins
Mini-manifests and a minified Portage tree are here here and ready for you to try out.
Here's how you do it:
# emerge --sync (to get latest portage required for the mini-tree)
# emerge portage (should be 2.2_rc67-r1)
# git clone git://github.com/funtoo/portage-mini-2010.git /usr/portage-mini-2010
or
# git clone https://fun...@github.com/funtoo/portage-mini-2010.git /usr/portage-mini-2010
# cd /usr/portage-mini-2010
# git checkout funtoo.org
add PORTDIR=/usr/portage-mini-2010 to /etc/make.conf
add FEATURES="mini-manifest" to /etc/make.conf
# emerge --sync (to populate the metadata cache)
And there you go... updates will be pushed to this tree along with the offiiclal "big" tree.
If you ever have lots of conflicts when doing an emerge --sync, it's likely that I've rebuilt the tree and you will need to re-clone it. Check the ML for an announcement when I do this.
The portage-mini-2010 tree has no ChangeLogs and mini-manifests and can be fully cloned using ~55MB of network traffic. This is a lot smaller than the current tree.
Please report bugs and issues here.
The latest portage should also have a slightly faster "emerge --sync" for all users do to the disabling of the diffstat after sync.
-Daniel
Oleg
mini-portage works without issues
oleg@oleg ~ $ sudo emerge --sync
>>> Starting git pull in /usr/portage-mini-2010...
Already up-to-date.
>>> Git pull in /usr/portage-mini-2010 successful
>>> Synchronizing timestamps...
but there is a eix problems (many users like it). I do have few
overlays, so an error looks like:
oleg@oleg ~ $ sudo eix-update
Reading Portage settings ..
Building database (/var/cache/eix) ..
[0] "funtoo" /usr/portage-mini-2010/ (cache: metadata-flat)
Reading category 154|154 (100%) Finished
[1] "oss-overlay" /var/lib/layman/oss-overlay (cache: parse|
ebuild*#metadata-flat#assign)
Reading category 35|154 ( 22%): dev-libs .. * ERROR: dev-libs/
libmowgli-9999 failed:
* flag-o-matic.eclass could not be found by inherit()
*
* Call stack:
* ebuild.sh, line 1877: Called source '/var/lib/layman/
oss-overlay/dev-libs/libmowgli/libmowgli-9999.ebuild'
* libmowgli-9999.ebuild, line 5: Called inherit 'flag-o-matic'
'mercurial'
* ebuild.sh, line 1290: Called die
* The specific snippet of code:
* [ ! -e "$location" ] && die "${1}.eclass could not be found by
inherit()"
*
* If you need support, post the output of 'emerge --info =dev-libs/
libmowgli-9999',
* the complete build log and the output of 'emerge -pqv =dev-libs/
libmowgli-9999'.
* This ebuild is from an overlay: '/var/lib/layman/oss-overlay/'
* S: '/libmowgli-9999'
and so on for every single ebuild within all overlays. Again eclasses
hell, I think :)
shayes
Portage full manifest -> ~1go with mini portage ~500mb.
Shayes.
David Brown
> Mini-manifests and a minified Portage tree are here here and ready
> for you to try out.
Portage seems to have a problem when the manifest ends up with nothing
in it. For example, trying to merge some kde stuff gives me. This
seems to stop the emerge.
!!! Manifest is empty: '/usr/portage-mini-2010/kde-base/kde-env/Manifest'
!!! Manifest is empty: '/usr/portage-mini-2010/dev-db/mysql-init-scripts/Manifest'
!!! Manifest is empty: '/usr/portage-mini-2010/virtual/perl-Sys-Syslog/Manifest'
!!! Manifest is empty: '/usr/portage-mini-2010/virtual/perl-Storable/Manifest'
!!! Manifest is empty: '/usr/portage-mini-2010/virtual/perl-File-Spec/Manifest'
!!! Manifest is empty: '/usr/portage-mini-2010/virtual/glu/Manifest'
!!! Manifest is empty: '/usr/portage-mini-2010/virtual/opengl/Manifest'
!!! Manifest is empty: '/usr/portage-mini-2010/kde-base/kdebase-pam/Manifest'
!!! Manifest is empty: '/usr/portage-mini-2010/virtual/mysql/Manifest'
!!! Manifest is empty: '/usr/portage-mini-2010/kde-base/kdebase-runtime-meta/Manifest'
Guy Fontaine
Works fine here as well as eix-sync. No issues for the moment.
Guy F, (aramisqc)
--
You received this message because you are subscribed to the Google Groups "Funtoo" group.
To post to this group, send email to funto...@googlegroups.com.
To unsubscribe from this group, send email to funtoo-dev+...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/funtoo-dev?hl=en.
Martin 'golodhrim' Scholz
>> funtoo-dev+...@googlegroups.com<funtoo-
dev%2Bunsu...@googlegroups.com>
>> .
>> For more options, visit this group at
>> http://groups.google.com/group/funtoo-dev?hl=en.
>>
>>
Works fine for me too, nice that it is so small now...
---
Martin 'golodhrim' Scholz
Auf dem Sattler 4
34516 Ederbringhausen
Phone: +49 6454 799623
mobile: +49 176 63301749
Fax: +49 6454
e-Mail: schol...@googlemail.com
All great discoveries are made by mistake.
-- Young
Giuseppe `ferdy` Miceli
just fast as hell. attached the session log.
cheers,
--
p.s. what /usr/portage is good for now?
>----- Original Message -----
>From: drob...@funtoo.org
>To: funto...@googlegroups.com
>Data: Sat, 26 Jun 2010 19:38:22 -0600
>Subject: [funtoo] Say "hi" to mini-manifests [BETA]
>Hi All,
>
>Mini-manifests and a minified Portage tree are here here and ready for you to try out.
>
>--
>You received this message because you are subscribed to the Google Groups "Funtoo" group.
>To post to this group, send email to funto...@googlegroups.com.
>To unsubscribe from this group, send email to funtoo-dev+...@googlegroups.com.
David Brown
> Works fine here as well as eix-sync. No issues for the moment.
The problem with blank manifests only occurs when you actually try to
emerge a package with a blank manifest. I've not had problems with
other packages.
Try one from this list:
find /usr/portage-mini-2010 -name Manifest -size 0
It's mostly virtual packages and packages that exist to hold
dependencies, so most people probably have all of the ones that they
need to install.
David
Daniel Robbins
> Everything works perfectly for me :)
>
> Portage full manifest -> ~1go with mini portage ~500mb.
Next step in this effort I will be working on is to get it even 10x smaller than it is now. ~50MB. And about 20 inodes.
-Daniel
Daniel Robbins
Thanks for this bug report - I will disable the "empty Manifest" message when mini-manifest is enabled.
-Daniel
Daniel Robbins
Quick fix to this is to edit /usr/lib*/portage/pym/portage/package/ebuild/digestcheck.py and change line 37 to read:
if manifest_empty and "mini-manifest" not in mysettings.features:
Committing this to git now for the next release.
-Daniel
Tonko Mulder
masked portage installed:
---------------------------------------------------------
tonko-desktop ~ # eix-sync
* Running emerge --sync
>>> Starting git pull in /usr/portage-mini-2010...
Already up-to-date.
>>> Git pull in /usr/portage-mini-2010 successful
* An update to portage is available. It is _highly_ recommended
* that you update portage now, before any other packages are updated.
* To update portage, run 'emerge portage' now.
* Copying old /var/cache/eix cache to /var/cache/eix.previous
* Running eix-update
Reading Portage settings ..
Building database (/var/cache/eix) ..
[0] "" /usr/portage-mini-2010/ (cache: metadata-flat)
Reading category 0|0 (100%) EMPTY!
Applying masks ..
Calculating hash tables ..
Writing database file /var/cache/eix ..
Database contains 0 packages in 0 categories.
* Calling eix-diff
Diffing databases (0 -> 0 packages)
* Time statistics:
5 seconds for syncing
6 seconds total
--------------------------------------------
tonko-desktop ~ # emerge -s portage
* sys-apps/portage
Latest version installed: 2.2_rc67-r1
Homepage: http://www.gentoo.org/proj/en/portage/index.xml
Description: Portage is the package management and
distribution system for Gentoo
License: GPL-2
-----------------------------------------------
But when I try to update portage:
-------------------------------------------------
tonko-desktop ~ # emerge portage -pv
These are the packages that would be merged, in order:
Calculating dependencies... done!
emerge: there are no ebuilds to satisfy "sys-apps/portage".
---------------------------------------------------
So I'm not sure what the issue is here
> --
> You received this message because you are subscribed to the Google Groups "Funtoo" group.
> To post to this group, send email to funto...@googlegroups.com.
> To unsubscribe from this group, send email to funtoo-dev+...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/funtoo-dev?hl=en.
>
>
--
Met vriendelijke groet,
Tonko Mulder
Ik ben ook op LinkedIn http://nl.linkedin.com/in/tonkomulder
robbat2
> # emerge --sync (to get latest portage required for the mini-tree)
> # emerge portage (should be 2.2_rc67-r1)
On the gentoo-scm list, "thin Manifests" were proposed for the Git
migration, but they got somewhat stalled,
A quick glance at your modified Portage source looks like you've got
most of it covered, but I'd love a clear description of how it works.
What wasn't solved yet on the scm list, was how to denote that the
tree was using thin Manifests.
Daniel Robbins
> Have you got a summary of what mini-manifest does?
> On the gentoo-scm list, "thin Manifests" were proposed for the Git
> migration, but they got somewhat stalled,
>
> A quick glance at your modified Portage source looks like you've got
> most of it covered, but I'd love a clear description of how it works.
>
> What wasn't solved yet on the scm list, was how to denote that the
> tree was using thin Manifests.
In a nutshell, here's how to migrate full Manifests to mini-manifests:
find -iname Manifest -exec sed -n -i -e "/DIST/p" {} \;
Basically they are just regular manifests with only the DIST lines retained.
-Daniel
d2_racing
funtootux usr # pwd
/usr
funtootux usr # du -sh portage
2,6G portage
funtootux usr # du -sh portage-mini-2010/
580M portage-mini-2010/
That's a lot smaller :P
David Brown
> funtootux usr # pwd
> /usr
> funtootux usr # du -sh portage
> 2,6G portage
> funtootux usr # du -sh portage-mini-2010/
> 580M portage-mini-2010/
>
> That's a lot smaller :P
Keep in mind that the distfiles are still being stored under
/usr/portage/distfiles, and you need to subtract out the size of this
directory in order to make any kind of meaningful comparison.
In my particular case, I get 624M for portage-distfiles, and 157M for
the new mini tree, which is still a good savings. The discrepancy
between different results will mostly be due to filesystem choice and
how well it deals with lots of small files.
David
d2_racing
And Daniel, if something goes wrong, how can we return to the actual
Funtoo tree, I would like to know which command do I need to run to
return to the Funtoo tree without your new tweak ?
Thanks :P
We will post a news at Funtoo-Quebec.org about that tomorrow.
Daniel Robbins
> Nice job, it's working pretty good on my box :P
>
> And Daniel, if something goes wrong, how can we return to the actual
> Funtoo tree, I would like to know which command do I need to run to
> return to the Funtoo tree without your new tweak ?
>
> Thanks :P
>
> We will post a news at Funtoo-Quebec.org about that tomorrow.
To change back to the normal Portage tree, just move it in place of the mini-portage tree, or update PORTDIR in /etc/make.conf.
To turn full Manifest creation/checking back on, remove "mini-manifest" from FEATURES.
-Daniel
Daniel Robbins
> Have you got a summary of what mini-manifest does?
> On the gentoo-scm list, "thin Manifests" were proposed for the Git
> migration, but they got somewhat stalled,
>
> A quick glance at your modified Portage source looks like you've got
> most of it covered, but I'd love a clear description of how it works.
>
> What wasn't solved yet on the scm list, was how to denote that the
> tree was using thin Manifests.
Also note that more is coming in regards to minimizing the portage tree. New extensions will be coming to reduce the on-disk footprint from 495MB to ~55MB (mini-portage tree) and from ~124,000 inodes to ~50 inodes. Zac and I will be working on this and it will be integrated into Gentoo's portage 2.2_rc series.
-Daniel
Tonko Mulder
need do a git checkout ;)
After that everything works just fine :)
David Brown
> Mini-manifests and a minified Portage tree are here here and ready
> for you to try out.
Just a brief status report. I did a fresh install from stage 3 and
checking out the mini-manifests tree as /usr/portage. So far
everything seems to be working fine. So, it seems to work fine to
install with.
David
robbat2
The question STILL remains about how to denote that a given tree
contains
only thin Manifests, and should not be checked for full manifests
(Basically
avoiding the need for your FEATURES addition).
For other size reductions in the tree, look at the MetaManifest GLEPs
that
were accepted. They are the offshoots of my years of treesigning
GLEP work.
Thin-manifest+MetaManifest together can give you a SINGLE Manifest
file,
at the top level of the tree. Saves ~14k inodes for all the other
Manifest files.
robbat2
compressed tree. We have more than 50 categories...
Daniel Robbins
devbox-ve portage-mini-2010 # du -m --max-depth=1
55 ./.git
55 .
devbox-ve portage-mini-2010 # find | wc -l
44
-Daniel
Daniel Robbins
> The question STILL remains about how to denote that a given tree
> contains
> only thin Manifests, and should not be checked for full manifests
> (Basically
> avoiding the need for your FEATURES addition).
zmedico and ferringb have suggested putting a setting in layout.conf
since it is a per-repo setting.
-Daniel
David Brown
In other words, portage will be able to use an unchecked-out, or bare
git repo. That's quite cool.
Of course, I'll probably end up checking it out anyway, since I end up
looking at ebuilds frequently.
David
chickenkiller
> Hi All,
> Mini-manifests and a minified Portage tree are here here and ready for you to try out.
> -Daniel
Working perfectly here, on an ~amd64 arch.
Thanks and keep up the great innovations,
Lionel
aramis_qc
Now installed and running on both PC's (laptop ~amd64 and desktop x86)
without any problems.
Regards,
Guy F.