General question: icecc actively supported ?

[Haio Maio]

Is icecc still actively supported?

Ive been observing some documentation statements / commands of how to use icecc and it often hasnt been the case in reality...

How reliable is the documentation?

Unfortunately, Ive lost too much time trying to achieve basics...

Regards,

[Henry Miller]

I try to support it, but there is a lot that I don't know much about. 

The documentation is okay, I've updated it where I had problems but I'm sure there are gaps. If you could be more specific about what isn't right I'll fix it. Or better yet open a pull request. 

I'm not sure how long it will be supported though. 32 core machines are becoming affordable. In my testing icecc has enough overhead that no amount of networked machines can be faster than a stand alone 32 core machine with enough ram. Ymmv, but it is worth asking if you can afford a really fast machine. 

-- 

  Henry Miller

  ha...@millerfarm.com

--

You received this message because you are subscribed to the Google Groups "icecream-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to icecream-user...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Haio Maio]

Are you saying the current version supports only up to 32 cores? (or why did you mention this number?) Hopefully not, that 32 will be pretty low as a limit (32000 ok, but 32 only...)

Otherwise, if I know something for sure, I will make a pull request.

[Henry Miller]

I have a icecc network with 60 8 core machines. A single stand alone 32 core machine always builds our system faster by a significant margin. That is what I mean.

Our networked machines all have users doing something unknown, are connected by a network with other traffic, have different disks... There is a lot of variation in the ice-cream builds that are hard to isolate. Still it is worth asking if icecc is worth it for you. 

-- 

  Henry Miller

  ha...@millerfarm.com

[Haio Maio]

This is really strange ... What are you saying is that icecc with 608 cores do worse than 32 cores without icecc ... This only means, that icecc is worth nothing in general or at least has severe flows. Is this what you are saying?

[Henry Miller]

Not exactly. First, I have 60 machines, each with 8 cores. I'm saying that none of the above machines can build as fast with icecc as a 32 core AWS machine that our CI system uses. This is not an equal comparison . Not only are the cups different, these are all user's machines that someone is doing work on - some of the 60 machines are doing builds of thir own in all tests. Our network also has other traffic, and not all machines are on the same subnet. 

Icecc does have overhead, this is documented. If you have no other machines in the network, icecc is always slower because it breaks apart build steps the compiler is more efficient doing as one. When you only have a few cores adding icecc in helps. When you already have a lot of cores locally the overhead of icecc is not worth it. You need to figure out how this applies to you.

-- 

  Henry Miller

  ha...@millerfarm.com

[Haio Maio]

Ok, 60x8=480 cores still have to bring significant difference to only one 32 AWS machine... Is it an icecc drawback only ? (Im wondering how distcc would do).

It is clear, that with only one machine, slow network etc. it has disadvantages, but still 480 is much bigger than 32 ... something is rotten in iceccMark....

[Oswald Buddenhagen]

On Tue, Mar 05, 2019 at 07:08:59AM -0500, Henry Miller wrote:
> Not exactly. First, I have 60 machines, each with 8 cores. I'm saying
> that none of the above machines can build as fast with icecc as a 32
> core AWS machine that our CI system uses. This is not an equal
> comparison . Not only are the cups different, these are all user's
> machines that someone is doing work on - some of the 60 machines are
> doing builds of thir own in all tests. Our network also has other
> traffic, and not all machines are on the same subnet.
>

the upshot is that single machines scale better than clusters, esp. when
the nodes are not allocated exclusively.

> If you have no other machines in the network, icecc is always slower
> because it breaks apart build steps the compiler is more efficient
> doing as one.
>

that can't be it, because that's a linear factor, so (say) 2*x icecc
cores would still beat x local ones.

> Icecc does have overhead, this is documented.

> [...]

> When you only have a few cores adding icecc in helps. When you already
> have a lot of cores locally the overhead of icecc is not worth it.
>

non-linear overhead is created only when disproportionately much work is
done locally before distributing, as is the case with the preprocessing.
that's an architectural limitation of icecc's current implementation.

this can be addressed to a high degree by full distribution (see
distcc's pump mode) built on top of efficient distributed caching.
lubos did an experiment in that direction, but i'm convinced he didn't
go far enough to make it pay off; see
https://github.com/icecc/icecream/issues/138 for the discussion (i
would actually have more to add there, but it's kinda pointless when
nobody is going to implement it anyway).

[Henry Miller]

On Tue, Mar 5, 2019, at 8:04 AM, Haio Maio wrote:

Ok, 60x8=480 cores still have to bring significant difference to only one 32 AWS machine... Is it an icecc drawback only ? (Im wondering how distcc would do).

It is clear, that with only one machine, slow network etc. it has disadvantages, but still 480 is much bigger than 32 ... something is rotten in iceccMark....

I don't know how to benchmark this, but my gut feeling is disk IO is the bottleneck. No amount of additional CPUs can speed up our builds if CPU is not the limit.  The 32 core machines have a different disk setup that probably is faster.

The 8 core machines continue to use icecc because builds are significantly faster that way. However they also see the downsides of icecc. Sometimes someone shuts off their computer while building, and the compile needs to be rerun elsewhere.  Sometimes the network is slow (IT puts some rules in place that may be a factor). Sometimes there are more people building at once than there are CPUs to go around. Sometimes we see icecc errors that make no sense to us - icecc retrys and the build finishes but it is slower.

[Diogo Guerra]

Im interested to know how is the icecream arquitectured. together with the distcc.
Do you have any knowledge to explain this? A diagram would be nice

[Lubos Lunak]

I don't think there's any diagram. File services/comm.h near the top has a
summary of messages that's the best documentation there is besides reading
the source. In short, daemons connect to the scheduler, when a client wants
to do a compile, it asks its local daemon for information about the build
(including on which node to build, which the daemon asks the scheduler), and
then it tells the selected remote daemon to do the compile.

--
Lubos Lunak