[gentoo-user] Hoping someone can help explain distcc to me
Paul Hartman
I have a fast desktop computer and a slow laptop. Both are ~amd64
Gentoo. After some of the recent discussions about Gentoo on slow
devices, I thought I'd dust off the laptop and try to bring it up to
date.
I'd like to use distcc to make the desktop do all the compiling during
emerges. I've never been able to get distcc working properly, or, at
least, I've never been able to get it working to the point where using
distcc is any faster than not using it at all.
Desktop: 192.168.0.100
Laptop: 192.168.0.107
My goal is to just don't use the laptop at all. Do all compiling on
the desktop. So far, while it appears to be working (some things are
being compiled on the desktop, a very insignificant amount -- CPU
usage on the desktop is basically 0% during emerges on the laptop),
compile times are no better than just compiling everything on the
laptop. The laptop still has 100% CPU activity during emerges. I
realize the laptop is still going to do work during emerges, but
thought at least distcc might be able to help offload most of the
heavy part (the compiling).
Both machines contain "distcc" in FEATURES. It's not using
-march=native. I've tried various -jN values with no real difference
in performance.
On the desktop, /etc/conf.d/distcc contains (among other things):
DISTCCD_OPTS="${DISTCCD_OPTS} --allow 192.168.0.0/24"
DISTCCD_OPTS="${DISTCCD_OPTS} --listen 192.168.0.100"
And /etc/distccd/hosts contains:
192.168.0.0/24
On the laptop, /etc/conf.d/distcc contains (among other things):
DISTCCD_OPTS="${DISTCCD_OPTS} --allow 192.168.0.0/24"
DISTCCD_OPTS="${DISTCCD_OPTS} --listen 192.168.0.107"
And /etc/distccd/hosts contains:
192.168.0.100
Does anyone know any tricks or can tell me if I'm doing something
wrong? How much of a speed-up should I realistically expect? Should
distcc be able to exploit the fast machine to the fullest of its
abilities?
Thanks
Peter Humphrey
> I have a fast desktop computer and a slow laptop. Both are ~amd64
> Gentoo. After some of the recent discussions about Gentoo on slow
> devices, I thought I'd dust off the laptop and try to bring it up to
> date.
>
> I'd like to use distcc to make the desktop do all the compiling during
> emerges. I've never been able to get distcc working properly, or, at
> least, I've never been able to get it working to the point where using
> distcc is any faster than not using it at all.
Your laptop isn't sending its compilation jobs to your desktop. Have you
used gkrellm or similar to check for network traffic?
I think I'd take a different approach, one that I dare say you can
anticipate. By all means take the advice you're about to be offered to get
distcc working, but why not create a chroot on your desktop, NFS export the
packages directory from the laptop to it and do the whole job there? Then
return to the laptop and emerge -k. It does need more keystrokes but it uses
far less CPU. You just need to copy /var/lib/portage/world, make.conf and
the /etc/portage tree to the chroot before you start.
I did once get distcc working, but as Neil has observed re Atoms, a lot of
work is still done before compilation is farmed out to the distcc server,
rather diluting the benefit of distributed compilation.
--
Rgds
Peter Linux Counter 5290, 1994-04-23
Paul Hartman
<pe...@humphrey.ukfsn.org> wrote:
> On Saturday 20 August 2011 23:56:08 Paul Hartman wrote:
>
>> I have a fast desktop computer and a slow laptop. Both are ~amd64
>> Gentoo. After some of the recent discussions about Gentoo on slow
>> devices, I thought I'd dust off the laptop and try to bring it up to
>> date.
>>
>> I'd like to use distcc to make the desktop do all the compiling during
>> emerges. I've never been able to get distcc working properly, or, at
>> least, I've never been able to get it working to the point where using
>> distcc is any faster than not using it at all.
>
> Your laptop isn't sending its compilation jobs to your desktop. Have you
> used gkrellm or similar to check for network traffic?
Well, there are definitely some distcc jobs going to the desktop, I
see them, it's just that they finish so fast and there's a "long" time
between them. The logs indicate it is successfully compiling things.
But, it is in no way saturating the resources of the desktop, which is
what I was (perhaps naively) hoping to see.
> I think I'd take a different approach, one that I dare say you can
> anticipate. By all means take the advice you're about to be offered to get
> distcc working, but why not create a chroot on your desktop, NFS export the
> packages directory from the laptop to it and do the whole job there? Then
> return to the laptop and emerge -k. It does need more keystrokes but it uses
> far less CPU. You just need to copy /var/lib/portage/world, make.conf and
> the /etc/portage tree to the chroot before you start.
>
> I did once get distcc working, but as Neil has observed re Atoms, a lot of
> work is still done before compilation is farmed out to the distcc server,
> rather diluting the benefit of distributed compilation.
That was going to be my next approach if distcc just doesn't do enough. :)
Could I just export the entire laptop - everything from the root
directory and below - and chroot into that over the network? Then I
wouldn't even need to emerge -k...
victor romanchuk
> Both machines contain "distcc" in FEATURES. It's not using
> -march=native. I've tried various -jN values with no real difference
> in performance.
only client (your laptop) machine should be distcc featured. for server
(desktop) that feature is useless
> On the desktop, /etc/conf.d/distcc contains (among other things):
> DISTCCD_OPTS="${DISTCCD_OPTS} --allow 192.168.0.0/24"
> DISTCCD_OPTS="${DISTCCD_OPTS} --listen 192.168.0.100"
this is server distcc daemon configuration, one just instructs daemon on what
network interface to listen for incoming connections (interface with ip
192.168.0.100 in your case) and filter incoming distcc connections by source
address: allow only those coming from local network machines with ip addresses
192.168.0.1 to 192.168.0.254
then distccd have to be started: /etc/init.d/distccd start
> And /etc/distccd/hosts contains:
> 192.168.0.0/24
this file configures distcc client behavior (actually the configuration can be
complex, see distcc man page for details), but in trivial case (for home
computing) it might look like:
e.g the client is able to send up to 6 distcc jobs to 192.168.0.100 and limit to
one job at local machine. client's /etc/make.conf has to have distcc feature
enabled (FEATURES="distcc"). surely you can play with job distribution rules
around the network. `distcc --show-hosts` command displays what you configured.
the number of cuncurrently running jobs (-j flag) has to be not less than sum of
local and remote jobs
i had noticed that distcc is peevish about CFLAGS: these should be compatible on
both client and server. in my case i made these similar on both machines (laptop
is core2duo and desktop is core2quad; both are running amd64 arch)
yet another way to install packages on weak notebook running it on the same arch
as desktop runs, - is to create binaries at powerful machine (while emerging or
with quickpkg utility) and share $PKGDIR with laptop
hth
Peter Humphrey
> Could I just export the entire laptop - everything from the root
> directory and below - and chroot into that over the network? Then I
> wouldn't even need to emerge -k...
No, I tried that and got myself tied in knots - well, actually it was the
whole portage tree that I exported, not the entire system. I forget what
went wrong now, but it's definitely cleaner to tell the server to build the
packages and the client to install from them. The emerge -k step is quick
too, and you have the advantage that you can see whether the packages are
actually there, unless you've switched colours off or not specified -v. (I
once found that they weren't there, which prompted me to go looking for the
config problem. Like Dale, I'm quite a good tester!)
You just have to make sure that the chroot is identical to the client.
Dale
Since you mentioned me. I wish I could set up a quicky from my 4 core
64 bit machine to compile 32 bit packages for a older 2GHz machine that
belongs to a friend. I was going to put Mandriva on it but the CD won;t
boot up properly. It stops at starting udev. Grrrrr.
How hard is it to set up a 64 bit machine to compile programs for a 32
bit system?
Dale
:-) :-)
Matthew Finkel
Peter Humphrey
Yes, just follow this guide:
http://www.gentoo.org/proj/en/base/amd64/howtos/chroot.xml
I did that and it was straightforward as far as I remember. I did spend some
time thinking at a few stages, to get an understanding of what I was doing
rather than just blindly following somebody else's prescription.
Then it's a matter of writing some simple scripts to mount the packages
directory on the big host. Here's mine, most of which I scrounged from
somewhere:
$ cat /etc/init.d/atom
#!/sbin/runscript
depend() {
need localmount
need bootmisc
}
start() {
ebegin "Mounting 32-bit chroot dirs"
mount -o bind /dev /mnt/atom/dev >/dev/null
mount -o bind /dev/pts /mnt/atom/dev/pts >/dev/null
mount -o bind /dev/shm /mnt/atom/dev/shm >/dev/null
mount -t proc /proc /mnt/atom/proc >/dev/null
mount -o bind /sys /mnt/atom/sys >/dev/null
mount -o bind /tmp /mnt/atom/tmp >/dev/null
mount -t nfs -o vers=3 192.168.2.2:/usr/portage/packages
/mnt/atom/usr/portage/packages
eend $? "An error occurred while attempting to mount 32-bit chroot
directories"
ebegin "Copying 32-bit chroot files"
cp -pf /etc/resolv.conf /mnt/atom/etc/ >/dev/null
cp -pf /etc/passwd /mnt/atom/etc/ >/dev/null
cp -pf /etc/shadow /mnt/atom/etc/ >/dev/null
cp -pf /etc/group /mnt/atom/etc/ >/dev/null
cp -pf /etc/hosts /mnt/atom/etc/ > /dev/null
cp -Ppf /etc/localtime /mnt/atom/etc/ >/dev/null
eend $? "An error occurred while attempting to copy 32-bit chroot files."
}
stop() {
ebegin "Unmounting 32-bit chroot dirs"
umount -f /mnt/atom/dev/pts >/dev/null
umount -f /mnt/atom/dev/shm >/dev/null
umount -f /mnt/atom/dev >/dev/null
umount -f /mnt/atom/proc >/dev/null
umount -f /mnt/atom/sys >/dev/null
umount -f /mnt/atom/tmp >/dev/null
umount -f /mnt/atom/usr/portage/packages >/dev/null
eend $? "An error occurred while attempting to unmount 32-bit chroot
directories"
}
I could list the steps of my daily routine to upgrade both the client and
the chroot if that would help.
Alex Schuster
> > Both machines contain "distcc" in FEATURES. It's not using
> > -march=native. I've tried various -jN values with no real difference
> > in performance.
-jN in make.conf's MAPEOPTS variable I assume, not as argument to emerge,
which does something different. It also hides the normal compile messages,
in case of problems you should see something like 'failed to distribute to
<host>'.
And syslog on the desktop should list every distributed job.
Distcc helps a lot here when I use Gentoo on slow systems. Although they are
still slow, because of time spent for configuring, or linking.
And maybe you are also using ccache, and your tests were recompiles of stuff
already cached, so the slow machine is fast at this?
> i had noticed that distcc is peevish about CFLAGS: these should be
> compatible on both client and server. in my case i made these similar on
> both machines (laptop is core2duo and desktop is core2quad; both are
> running amd64 arch)
I don't think this is true - as long as the CHOST is identical, there should
be no problem.
> yet another way to install packages on weak notebook running it on the
> same arch as desktop runs, - is to create binaries at powerful machine
> (while emerging or with quickpkg utility) and share $PKGDIR with laptop
This means some extra work, and also use flags need to be compatible, but
the speedup would be much bigger than with just distcc.
What about exporting the whole root file system and mounting it on the fast
desktop, chrooting and emerging?
And then there's Sabayon, based on Gentoo, but with binary packages. And
when you want to customize things (use different use flags), you can still
compile stuff manually. I will try this for a slow desktop PC now, all this
compiling is so annoying.
Wonko
Joost Roeleveld
That would help as I'm planning on setting this up myself as well for my
netbook.
Is there a way to automate the steps inside the chroot without having to have
a script inside the chroot?
--
Joost
victor romanchuk
>
>> i had noticed that distcc is peevish about CFLAGS: these should be
>> compatible on both client and server. in my case i made these similar on
>> both machines (laptop is core2duo and desktop is core2quad; both are
>> running amd64 arch)
> I don't think this is true - as long as the CHOST is identical, there should
> be no problem.
CHOST defines the arch (i686, amd64, arm ..) whilst CFLAGS control gcc behavior
and the binary code generation produced by compiler. in my case core2quad
(q8300) i'm using in the desktop supports sse4.1 instruction set and notebook
powered with core2duo (t7600) does not have that cpu feature. having option
`-msse4.1' set in CFLAGS at desktop side will causes frequent compilation
failures (initiated by distcc) or, in worst case - arbitrary crashes at notebook
when running binaries compiled in distributed distcc environment
>> yet another way to install packages on weak notebook running it on the
>> same arch as desktop runs, - is to create binaries at powerful machine
>> (while emerging or with quickpkg utility) and share $PKGDIR with laptop
> This means some extra work, and also use flags need to be compatible, but
> the speedup would be much bigger than with just distcc.
>
> What about exporting the whole root file system and mounting it on the fast
> desktop, chrooting and emerging?
>
i do not insist the distcc is the only or most efficient way to maintain gentoo
installation on a slow machine (having something more powerful nearby). just
tried to explain how does distcc work
victor
Peter Humphrey
> That would help as I'm planning on setting this up myself as well for my
> netbook.
Right. I have two Konsoles open on my workstation, which is the compilation host. In one I "su -" and in the
other I "ssh serv" (this is the client Atom box, which among other things runs http-replicator to serve the
portage tree to the LAN). Naming is going to get confusing if I'm not careful, so I'll refer to the ssh session as
"Atom" and the compilation host as "Host". Then my steps are:
Host: # /etc/init.d/atom start (this is the script I showed yesterday)
# linux32 chroot /mnt/atom /bin/bash
# env-update && . /etc/profile
Atom: $ sudo emerge --sync && sudo eix-update
Host: # emerge --sync && emerge -auvD -j 5 --changed-use --keep-going world
Atom: $ sudo emerge -auDkv --jobs=3 --changed-use --with-bdeps y --keep-going world
Host: (various clean-up operations such as depclean, eclean and localepurge)
# exit
# /etc/init.d/atom stop
Atom: (similar cleaning up)
$ exit
That's it as far as I remember.
> Is there a way to automate the steps inside the chroot without having to
> have a script inside the chroot?
I'd be reluctant to try to automate it any more than this. It's about as simple to use as can be and as I want it.
I've set up aliases for most of those long commands to save my wrists, and of course command-line recall is
wonderful. The task that takes longest is portage on the Atom calculating what packages it needs to emerge
from.
I try not to forget to copy any USE-flag changes etc between the Atom and the chroot, but of course I'm no
more than human.
Neil Bothwick
> Is there a way to automate the steps inside the chroot without having
> to have a script inside the chroot?
This is the script I use. You can call it with "ch hostname" or symlink
it to chhostname. That way I can use the same script for multiple
computers. The -w option causes it to enter the chroot, emerge world and
exit.
--
Neil Bothwick
Printer: (n.) a small box attached to a computer and used to start
fires in cold weather.
Paul Hartman
<paul.hart...@gmail.com> wrote:
>
> I'd like to use distcc to make the desktop do all the compiling during
> emerges. I've never been able to get distcc working properly, or, at
> least, I've never been able to get it working to the point where using
> distcc is any faster than not using it at all.
Thanks to all who replied for your suggestions and info! After
experimenting with various combinations of settings it seems to be
working now. I think I was too aggressive with the number of parallel
make jobs, the laptop just couldn't coordinate that many. I also
stopped distccd on the laptop, in case it was picking up jobs
unintentionally.
Here's the results, a real-life test with ImageMagick 6.7.1.0. The
machines were idle, files were already downloaded, and I don't use
ccache, so it should be actual compile time comparison only:
Normal emerge on laptop, no parallel jobs in MAKEOPTS, no distcc:
merge time: 8 minutes and 27 seconds.
With -j3 in MAKEOPTS, using distcc:
merge time: 4 minutes and 40 seconds.
Great!
Of course, on the desktop it only takes 1 minutes 17 seconds, but an
almost 50% reduction in emerge time on the laptop is still great to
see.
Alex Schuster
> >> i had noticed that distcc is peevish about CFLAGS: these should be
> >> compatible on both client and server. in my case i made these
> >> similar on both machines (laptop is core2duo and desktop is
> >> core2quad; both are running amd64 arch)
> >
> > I don't think this is true - as long as the CHOST is identical, there
> > should be no problem.
>
> CHOST defines the arch (i686, amd64, arm ..) whilst CFLAGS control gcc
> behavior and the binary code generation produced by compiler. in my case
> core2quad (q8300) i'm using in the desktop supports sse4.1 instruction
> set and notebook powered with core2duo (t7600) does not have that cpu
> feature. having option `-msse4.1' set in CFLAGS at desktop side will
> causes frequent compilation failures (initiated by distcc) or, in worst
> case - arbitrary crashes at notebook when running binaries compiled in
> distributed distcc environment
But how could the gcc on the desktop, called via distcc, even see the CFLAGS
in the desktops make.conf? This just cannot be.
I have been using march=athlon-xp on my desktop, and yet used it as distcc
server for my Pentium-III notebook for years. Worked fine.
Wonko
Bill Longman
Bill Longman
>>> > > should be no problem.
>> >
>> > CHOST defines the arch (i686, amd64, arm ..) whilst CFLAGS control gcc
>> > behavior and the binary code generation produced by compiler. in my case
>> > core2quad (q8300) i'm using in the desktop supports sse4.1 instruction
>> > set and notebook powered with core2duo (t7600) does not have that cpu
>> > feature. having option `-msse4.1' set in CFLAGS at desktop side will
>> > causes frequent compilation failures (initiated by distcc) or, in worst
>> > case - arbitrary crashes at notebook when running binaries compiled in
>> > distributed distcc environment
> But how could the gcc on the desktop, called via distcc, even see the CFLAGS
> in the desktops make.conf? This just cannot be.
> I have been using march=athlon-xp on my desktop, and yet used it as distcc
> server for my Pentium-III notebook for years. Worked fine.
The arguments sent to gcc tell it what kind of object gcc is supposed to
create. If gcc cannot create the object you request because it's not the
right "kind" of gcc, implies that, within the same gcc version, it is
unable to reproduce object files.
Dale
I followed the guide and some info here and it seems to have worked
great. I'm glad. The cooler on the heatsink is VERY whimpy. It's
aluminum and has a little 60mm fan on it. I didn't measure it but its
certainly not as big as my 80mm. I can barely feel the air it moves.
Compiling on htat would likely be a problem pretty quick.
Now to get the install moved over to the other rig. DVD and a tarball
maybe? Hmmmm, I got to think on this one. It's almost 5Gbs worth.
Maybe man scp will help. ;-)
Thanks for the link. This was awesome!!
Dale
:-) :-)
P. S. Now I can build binaries for my old rig too. :-D