-c opt and -fPIC

[glin...@gmail.com]

We are building ffmpeg inside of bazel, and it needs to be compiled with -fPIC. We've discovered two potential bugs:

1. Even when specifying '-fPIC' in the copts, bazel tries to do a non PIC build. So we end up building each source file twice, but they are in fact the same. Once bazel includes it's own '-fPIC', and once it doesn't, but since it's included as part of the copts, it is always passed to the compiler.

2. There is some sort of fun dependency issue, where including '-fPIC' causes the rules to become non-deterministic somehow. Haven't narrowed it down to a small repro yet, but the general steps are adding a comment like '# testing' to the BUILD file for ffmpeg, and then it rebuilds the entire thing. When analyzing the dependencies, we see something very strange, the include files are ordered differently. eg.

before adding comment:
863 Declared include source: third_party/ffmpeg/libavformat/yuv4mpeg.h
864 Declared include source: third_party/ffmpeg/libavutil/aarch64/bswap.h
865 Declared include source: third_party/ffmpeg/libavutil/aarch64/cpu.h
866 Declared include source: third_party/ffmpeg/libavutil/aarch64/neontest.h
867 Declared include source: third_party/ffmpeg/libavutil/adler32.h

after adding comment:
862 Declared include source: third_party/ffmpeg/libavformat/yuv4mpeg.h
863 Declared include source: third_party/ffmpeg/libavutil/adler32.h
864 Declared include source: third_party/ffmpeg/libavutil/aes.h
865 Declared include source: third_party/ffmpeg/libavutil/atomic.h
866 Declared include source: third_party/ffmpeg/libavutil/atomic_gcc.h

[James Philbin]

I believe I have self-contained reproduction of this issue and have filed a bug here:

https://github.com/bazelbuild/bazel/issues/2304

fPIC is a red-herring. The real issue seems to be the use of globs within a target.

Thanks,

James

[Damien Martin-guillerez]

Gary, can you confirm it is really issue #2304? It seems to be related to an older version of Bazel.

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/e7be46e4-1887-4952-9191-b8671de5623d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[glin...@gmail.com]

The second issue does appear to be #2304. However, the first issue (double building of the source files when -c opt specified) is still there.

Is this behavior intentional? It looks like it might be: https://github.com/bazelbuild/bazel/blob/master/src/main/java/com/google/devtools/build/lib/rules/cpp/CppConfiguration.java#L418-L419

[Damien Martin-guillerez]

I guess it is but I cannot say for sure. +Lukács T. Berki might know and answer when he will be back from holidays.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/0db0cf07-d556-480b-ac13-5d435560053b%40googlegroups.com.

[Lukács T. Berki]

Building non-PIC code for binaries is intentional, although I'm somewhat surprised that binaries are built *twice*. Are you by any chance building dynamic libraries and binaries on the same command line?

This behavior is indeed surprising. Marcel opined that it was trading startup time for faster code, which makes sense, at least on the surface.

You can use the --force_pic command line option to build binaries in PIC mode, too. We should give you more control over picness, but in the meantime, I hope this'll do.

To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discuss+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/0db0cf07-d556-480b-ac13-5d435560053b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Lukács T. Berki | Software Engineer | lbe...@google.com | 

Google Germany GmbH | Erika-Mann-Str. 33  | 80636 München | Germany | Geschäftsführer: Matthew Scott Sucherman, Paul Terence Manicle | Registergericht und -nummer: Hamburg, HRB 86891

[glin...@gmail.com]

On Monday, January 2, 2017 at 3:39:21 AM UTC-10, Lukács T. Berki wrote:
> Building non-PIC code for binaries is intentional, although I'm somewhat surprised that binaries are built *twice*. Are you by any chance building dynamic libraries and binaries on the same command line?

Yes, we are building dynamic libraries and binaries in the same command line. We usually build a "meta" target, which has all the binaries/libraries needed for our system.

> This behavior is indeed surprising. Marcel opined that it was trading startup time for faster code, which makes sense, at least on the surface.

Yes, it's an expensive trade-off, but it would be worth it at Google scale probably :).

> You can use the --force_pic command line option to build binaries in PIC mode, too. We should give you more control over picness, but in the meantime, I hope this'll do.

Cool, I'll look into that. Thanks!

[yunjia...@gmail.com]

On Monday, January 2, 2017 at 5:39:21 AM UTC-8, Lukács T. Berki wrote:
> Building non-PIC code for binaries is intentional, although I'm somewhat surprised that binaries are built *twice*. Are you by any chance building dynamic libraries and binaries on the same command line?
>
>
> This behavior is indeed surprising. Marcel opined that it was trading startup time for faster code, which makes sense, at least on the surface.
>
>
> You can use the --force_pic command line option to build binaries in PIC mode, too. We should give you more control over picness, but in the meantime, I hope this'll do.
>

As of bazel 0.16.1, even with --force_pic I still get this error:

/usr/bin/ld: bazel-out/k8-opt/bin/tensorflow_serving/model_servers/_objs/tensorflow_model_server/tensorflow_serving/model_servers/version.o: relocation R_X86_64_32 against `.rodata' can not be used when making a shared object; recompile with -fPIC