Working ccache configuration for buildworld on amd64?

[Maxim Khitrov]

This is an old problem, but so far I haven't been able to find a
solution. When ccache is used to build world on amd64, the process
fails when /usr/src/lib/csu/i386-elf/crt1.c is compiled. If
WITHOUT_LIB32 is added to src.conf, this problem does not happen.
Likewise, building without ccache works fine.

Has anyone out there been able to find a ccache configuration that
would work in this situation? I tried disabling ccache for some of the
directories under /usr/src and /usr/obj, but it only caused problems
in other stages of the build process.

Here are the default ccache settings from make.conf:

.if exists(/usr/local/libexec/ccache) && !defined(NOCCACHE) && \
(!empty(.CURDIR:M/usr/src*) || !empty(.CURDIR:M/usr/obj*))
CC= /usr/local/libexec/ccache/world-cc
CXX= /usr/local/libexec/ccache/world-c++
.endif

And here is what I tried using to isolate the problem, albeit with no success:

.if exists(/usr/local/libexec/ccache) && !defined(NOCCACHE) && \
(!empty(.CURDIR:M/usr/src*) || !empty(.CURDIR:M/usr/obj*)) && \
empty(.CURDIR:M/usr/src/lib/csu*) && \
empty(.CURDIR:M/usr/obj/usr/src/lib/csu*) && \
empty(.CURDIR:M/usr/obj/lib32/usr/src/lib/csu*)
CC= /usr/local/libexec/ccache/world-cc
CXX= /usr/local/libexec/ccache/world-c++
.endif

Please let me know if you have a working solution.

- Max

[RW]

On Thu, 14 Aug 2008 16:04:52 -0400
"Maxim Khitrov" <mkhi...@gmail.com> wrote:

> This is an old problem, but so far I haven't been able to find a
> solution. When ccache is used to build world on amd64, the process
> fails when /usr/src/lib/csu/i386-elf/crt1.c is compiled. If
> WITHOUT_LIB32 is added to src.conf, this problem does not happen.
> Likewise, building without ccache works fine.

I take it that you've already tried removing any unnecessary settings
such as CFLAGS.

What interesting about this is that it's failing on a compile; i.e. on a
cache miss, when ccache is doing next to nothing. That suggests that
there's either a problem in the way that the real compiler is invoked
by ccache, or that the real failure occurred during the building
of the toolchain and it's dependencies.

I'd try setting CCACHE_RECACHE temporarily in the environment, to flush
out the old cached files, and see if it makes a difference.

[Maxim Khitrov]

I only set the CPUTYPE in make.conf, CFLAGS are untouched. Clearing
out the repository (ccache -C) doesn't help.

I think what's happening is that there is a collision in hash values
generated by ccache. That's the only thing I can think of, because
crt1.c is compiled twice; once from /usr/src/lib/csu/amd64/crt1.c, and
a second time from /usr/src/lib/csu/i386-elf/crt1.c. If LIB32 is
disabled in src.conf, only the first compilation takes place. If the
generated hash values are the same, by some chance, then the actual
problem is that the file is not compiled a second time when, in fact,
it should be. This is only a guess, however.

- Max

[RW]

That collision isn't going to happen for several reasons, but it's
missing the point I made earlier, that the build is failing on a cache
miss. If the kind of situation you're describing is happening, then it's
happening earlier, and the observed error is just a side-effect.

[John Silva]

>
> On Fri, 15 Aug 2008 11:22:01 -0400

> "Maxim Khitrov" <mkhitrov at gmail.com <http://lists.freebsd.org/mailman/listinfo/freebsd-questions>> wrote:
>
>
> >/ I think what's happening is that there is a collision in hash values
> />/ generated by ccache. That's the only thing I can think of, because
> />/ crt1.c is compiled twice; once from /usr/src/lib/csu/amd64/crt1.c, and
> />/ a second time from /usr/src/lib/csu/i386-elf/crt1.c. If LIB32 is
> />/ disabled in src.conf, only the first compilation takes place. If the
> />/ generated hash values are the same, by some chance, then the actual
> />/ problem is that the file is not compiled a second time when, in fact,
> />/ it should be. This is only a guess, however.
> /

> That collision isn't going to happen for several reasons, but it's
> missing the point I made earlier, that the build is failing on a cache
> miss. If the kind of situation you're describing is happening, then it's
> happening earlier, and the observed error is just a side-effect.
>

Nothing is wrong with ccache; it works just fine.

What's happening is that the build system is passing compiler options in
the CC variable when invoking make for lib32, and the recommended
make.conf config for ccache is blowing those away.

Here's how the lib32 build is invoked with ccache enabled:

cd /usr/src; MAKEOBJDIRPREFIX=/usr/obj/lib32
_SHLIBDIRPREFIX=/usr/obj/usr/src/lib32 MACHINE=i386 MACHINE_ARCH=i386
INSTALL="sh /usr/src/tools/install.sh"
PATH=/usr/obj/usr/src/tmp/legacy/usr/sbin:/usr/obj/usr/src/tmp/legacy/usr/bin:/usr/obj/usr/src/tmp/legacy/usr/games:/usr/obj/usr/src/tmp/usr/sbin:/usr/obj/usr/src/tmp/usr/bin:/usr/obj/usr/src/tmp/usr/games:/sbin:/bin:/usr/sbin:/usr/bin
CC="/usr/local/libexec/ccache/world-cc -m32 -march=k8 -mfancy-math-387
-DCOMPAT_32BIT -iprefix /usr/obj/usr/src/lib32/usr/
-L/usr/obj/usr/src/lib32/usr/lib32 -B/usr/obj/usr/src/lib32/usr/lib32"
CXX="/usr/local/libexec/ccache/world-c++ -m32 -march=k8 -mfancy-math-387
-DCOMPAT_32BIT -iprefix /usr/obj/usr/src/lib32/usr/
-L/usr/obj/usr/src/lib32/usr/lib32 -B/usr/obj/usr/src/lib32/usr/lib32"
OBJC="cc -m32 -march=k8 -mfancy-math-387 -DCOMPAT_32BIT -iprefix
/usr/obj/usr/src/lib32/usr/ -L/usr/obj/usr/src/lib32/usr/lib32
-B/usr/obj/usr/src/lib32/usr/lib32" LD="ld -m elf_i386_fbsd -Y
P,/usr/obj/usr/src/lib32/usr/lib32" AS="as --32" LIBDIR=/usr/lib32
SHLIBDIR=/usr/lib32 make -DNO_CPU_CFLAGS -DCOMPAT_32BIT -DWITHOUT_BIND
-DWITHOUT_MAN -DWITHOUT_NLS -DWITHOUT_INFO -DWITHOUT_HTML
DESTDIR=/usr/obj/usr/src/lib32 -f Makefile.inc1 libraries
cd /usr/src; make -f Makefile.inc1 _prereq_libs; make -f Makefile.inc1
_startup_libs; make -f Makefile.inc1 _prebuild_libs; make -f
Makefile.inc1 _generic_libs;

Here's how the compiler gets called from the 2nd level make with the
recommended settings:

/usr/local/libexec/ccache/world-cc -O2 -fno-strict-aliasing -pipe
-I/usr/src/lib/libc/include -I/usr/src/lib/libc/../../include
-I/usr/src/lib/libc/i386 -D__DBINTERFACE_PRIVATE
-I/usr/src/lib/libc/../../contrib/gdtoa -DINET6
-I/usr/obj/lib32/usr/src/lib/libc -I/usr/src/lib/libc/resolv
-DPOSIX_MISTAKE -I/usr/src/lib/libc/locale -DBROKEN_DES -DPORTMAP
-DDES_BUILTIN -I/usr/src/lib/libc/rpc -DYP -DNS_CACHING
-DSYMBOL_VERSIONING -Wsystem-headers -Wall -Wno-format-y2k
-Wno-uninitialized -Wno-pointer-sign -c

Here's what's supposed to be happening (without ccache):

cc -m32 -march=k8 -mfancy-math-387 -DCOMPAT_32BIT -iprefix
/usr/obj/usr/src/lib32/usr/ -L/usr/obj/usr/src/lib32/usr/lib32
-B/usr/obj/usr/src/lib32/usr/lib32 -O2 -fno-strict-aliasing -pipe
-I/usr/src/lib/libc/include -I/usr/src/lib/libc/../../include
-I/usr/src/lib/libc/i386 -D__DBINTERFACE_PRIVATE
-I/usr/src/lib/libc/../../contrib/gdtoa -DINET6
-I/usr/obj/lib32/usr/src/lib/libc -I/usr/src/lib/libc/resolv
-DPOSIX_MISTAKE -I/usr/src/lib/libc/locale -DBROKEN_DES -DPORTMAP
-DDES_BUILTIN -I/usr/src/lib/libc/rpc -DYP -DNS_CACHING
-DSYMBOL_VERSIONING -Wsystem-headers -Wall -Wno-format-y2k
-Wno-uninitialized -Wno-pointer-sign -c
/usr/src/lib/libc/i386/gen/_ctx_start.S

Notice the difference?

Try this instead:

# Special ccache for buildworld
.if exists(/usr/local/libexec/ccache/world-cc) && !defined(NOCCACHE) && \
(!empty(.CURDIR:M/usr/src*) || !empty(.CURDIR:M/usr/obj*))
CC := ${CC:C,^cc,/usr/local/libexec/ccache/world-cc,1}
CXX := ${CXX:C,^c\+\+,/usr/local/libexec/ccache/world-c++,1}
.endif

Instead of hard-configuring CC/CXX, it selectively modifies it only on
the first invocation of make.

Subsequent invocations are allowed to modify CC as needed using the
original (modified) values as a seed.

-j