icecc suddenly did not distribute jobs anymore - only local build.

[Abhijit Sovakar]

Hi,

I have the following strange behavior of the icecream.

From one day to another the distribution of jobs did not work anymore.
Instead every job is build locally only.
And my machine is the only one with that problem.

We are using openSuSE 13.1 with icecream-1.0.1-2.1.2.x86_64
We are using ICECC_VERSION with a custom configured gcc 4.8.3

My iceccd connects correctly to the scheduler.
The icemon shows my machine.
The scheduler also get's the information that my machine is building (but locally).

Spend already 1 day to figure out why. But at the moment I am clueless.

Thankful for any hints.

Regards,

Abi

[Abhijit Sovakar]

Additional remarks,

my machine is used by others in the same icecream network. So the scheduler gives jobs to my machine.
So I assume it's something local to my machine. For some reason my iceccd thinks the job has to be done locally only:

But so far I couldn't find anything specific.

part of the iceccd log with option -vvv:
---- snip
[18729] 16:28:49: Connected to scheduler (I am known as .... )
[18729] 16:32:03: accepted 10 108.127.0.0
[18729] 16:32:03: send JobLocalBeginMsg to client
[18729] 16:32:03: pushed local job 1
[18729] 16:32:03: eof && !HAS_MSG
[18729] 16:32:03: scheduler->send_msg( JobLocalDoneMsg( 1) );
[18729] 16:32:03: accepted 10 0.0.0.0
[18729] 16:32:03: send JobLocalBeginMsg to client
[18729] 16:32:03: pushed local job 2
[18729] 16:32:03: eof && !HAS_MSG
[18729] 16:32:03: scheduler->send_msg( JobLocalDoneMsg( 2) );
[18729] 16:32:04: accepted 10 0.0.0.0
[18729] 16:32:04: accepted 11 0.0.0.0
[18729] 16:32:04: accepted 12 0.0.0.0
[18729] 16:32:04: send JobLocalBeginMsg to client
[18729] 16:32:04: pushed local job 3
[18729] 16:32:04: send JobLocalBeginMsg to client
[18729] 16:32:04: pushed local job 4
[18729] 16:32:04: send JobLocalBeginMsg to client
[18729] 16:32:04: pushed local job 5
[18729] 16:32:04: accepted 13 108.127.0.0
[18729] 16:32:04: send JobLocalBeginMsg to client
[18729] 16:32:04: pushed local job 6
[18729] 16:32:04: accepted 14 108.127.0.0
[18729] 16:32:04: send JobLocalBeginMsg to client
[18729] 16:32:04: pushed local job 7
[18729] 16:32:04: accepted 15 108.127.0.0
[18729] 16:32:04: accepted 16 0.0.0.0
[18729] 16:32:04: accepted 17 0.0.0.0
[18729] 16:32:04: accepted 18 0.0.0.0
[18729] 16:32:04: accepted 19 108.127.0.0
[18729] 16:32:04: accepted 20 108.127.0.0
[18729] 16:32:04: accepted 21 108.127.0.0
[18729] 16:32:04: accepted 22 108.127.0.0
---- snap

[David Vest]

On Thu, Apr 9, 2015 at 4:55 PM, Abhijit Sovakar
<a.so...@googlemail.com> wrote:
> Additional remarks,
>
> my machine is used by others in the same icecream network. So the scheduler
> gives jobs to my machine.
> So I assume it's something local to my machine. For some reason my iceccd
> thinks the job has to be done locally only:

I'm far from knowledgeable about icecc in general but it would be
interesting to know what the client (icecc) reports with
ICECC_DEBUG=debug set.

David

[Abhijit Sovakar]

I tried that, too. Here is the output of the icc with "ICECC_DEBUG=debug" set. Does not say to much to me.
---- snip
ICECC[5304] 19:43:43: connected to /var/run/icecc/iceccd.socket
ICECC[5304] 19:43:43: <building_local>
ICECC[5304] 19:43:43: invoking: /usr/cae/gcc48/bin/g++
ICECC[5304] 19:43:44: </building_local: 889ms>
ICECC[5292] 19:43:43: connected to /var/run/icecc/iceccd.socket
ICECC[5292] 19:43:43: <building_local>
ICECC[5292] 19:43:43: invoking: /usr/cae/gcc48/bin/g++
ICECC[5292] 19:43:44: </building_local: 1563ms>
ICECC[5311] 19:43:43: connected to /var/run/icecc/iceccd.socket
ICECC[5311] 19:43:43: <building_local>
ICECC[5311] 19:43:43: invoking: /usr/cae/gcc48/bin/g++
ICECC[5311] 19:43:45: </building_local: 1747ms>
ICECC[5309] 19:43:43: connected to /var/run/icecc/iceccd.socket
ICECC[5309] 19:43:43: <building_local>
ICECC[5309] 19:43:43: invoking: /usr/cae/gcc48/bin/g++
ICECC[5309] 19:43:45: </building_local: 1748ms>
ICECC[5295] 19:43:43: connected to /var/run/icecc/iceccd.socket
ICECC[5295] 19:43:43: <building_local>
ICECC[5295] 19:43:44: invoking: /usr/cae/gcc48/bin/g++
ICECC[5295] 19:43:45: </building_local: 2138ms>
ICECC[5307] 19:43:43: connected to /var/run/icecc/iceccd.socket
ICECC[5307] 19:43:43: <building_local>
ICECC[5307] 19:43:43: invoking: /usr/cae/gcc48/bin/g++
ICECC[5307] 19:43:46: </building_local: 2963ms>
---- snap

[Abhijit Sovakar]

And that is what the scheduler says with option -vvv:

----snip
[22802] 19:53:05: login asovakar protocol version: 29 [runtime48(x86_64), ]
[22802] 19:53:05: handle_local_job <file...> 36
[22802] 19:53:05: handle_local_job <file...> 40
[22802] 19:53:05: handle_local_job <file...> 29
----snap

[David Vest]

On Thu, Apr 9, 2015 at 7:45 PM, Abhijit Sovakar
<a.so...@googlemail.com> wrote:
> I tried that, too. Here is the output of the icc with "ICECC_DEBUG=debug"
> set. Does not say to much to me.
> ---- snip

Right, that didn't say much. icecc usually moans when I've messed up
ICECC_VERSION or passed a compile flag it isn't compatible with
distributed compilation forcing it to compile locally.

Anything special in your environment that may cause this (e.g. env |
grep ICECC)? Otherwise I'm out of ideas... Possibly recompile the
client with CLIENT_DEBUG=1 set in arg.cpp, if you suspect a compiler
flag. But since it works on other computers, I'd double check the
environment first.

David

[Abhijit Sovakar]

Got it it was the environment. Compared my one with those of others. For some reason -march=native was set.

[Stefan Reinhold]

I have exactly the same problem but without -march=native being set.

icecc worked for me a long time with clang-3.5 then suddenly it stopped distributing my jobs to the other machines.

Same debug messages etc. as above.