Successfully Cross-Compiling GRPC for Linux/ARM - Some Open Questions

[abhinava.s...@gmail.com]

Hello Folks!

Across this forum and the GRPC GitHub issue archives, I don't think I see any guide from the maintainers on how to cross-compile GRPC and the dependencies. 

Background: For an embedded project, I was trying to get GRPC to run on Linux/ARM. I am able to successfully cross-compile GRPC, Protobuf and build some applications to run on the target. However, I am still not sure of the exact order of steps and it seems to me like black magic! The Makefile is at a whopping 25k LoC! The section that deals with cross-compiling is very small. Thankfully, I found [1], [2] and [3] very helpful but I still have a bunch of things which I don't quite understand and how the dependencies are resolved.

Here are some of the questions I have and would be eternally grateful if somebody can help me better understand :)

NOTE: 

• In this case, --host=arm-linux-gnueabihf --build=x86_64-pc-linux-gnu
  
• I have ProtoBuf and the protoc compiler (with all the GRPC plugins installed on my build machine)

1. I don't want to use BoringSSL and would prefer using OpenSSL. The reason being that for purposes of "security conformance" (certifications such as NIST's FIPS or Common Criteria), it's better (we've already invested efforts to "sanitize" OpenSSL)
1. I got OpenSSL from GitHub (https://github.com/openssl/openssl) and cross-compiled them for the Linux/ARM
2. Produced all the libssl and libcrypto .so and .a libraries
2. I am able to cross-compile ProtoBuf for Linux/ARM and get all the .so and .a libraries
3. It also seems that for the "make run_dep_checks" to work, I had to separately individually cross-compile (for Linux/ARM) the following projects:
1. gperftools (https://github.com/gperftools/gperftools) - For the "-l profiler"
2. zlib (grpc/third_party/zlib already as part of the grpc 'sub module')
3. c-ares (grpc/third_party/cares/cares) - For the "-l cares"
   
4. protobuf (grpc/third_party/protobuf)
5. OpenSSL - Like I mentioned in (1).(1) above
4. So with all these "dependencies" already compiled for my Linux/ARM target, when I try compiling the GRPC source, I am unable to figure out a few things
1. With the right GRPC_CROSS_LDOPTS pointing to the paths of the compiled dependencies above, the Makefile figures out that it doesn't need to build BoringSSL --> That's great!
2. I also use the EMBED_OPENSSL=false flag --> Is this correct?
3. As per [4], should I also be using HAS_SYSTEM_OPENSSL_ALPN=false HAS_SYSTEM_OPENSSL_NPN=true?
   
4. I'm also using the following flags: GRPC_CROSS_COMPILE=true HAS_PKG_CONFIG=false GRPC_CROSS_AROPTS="cr --target=elf32-little" 
5. One of the problems is that the compilation skips compiling Protobuf (as needed - since it's already compiled), BUT, it tries to build all the GRPC language plugins using the arm-linux-gcc toolchain and this invariably fails on the build machine host since the CPU architectures don't match --> How can I skip the building of the GRPC language plugins? Using HAS_SYSTEM_PROTOBUF=false and this skips the language plugins, BUT it re-builds the entire protobuf source again :|
1. How can I avoid re-building protobuf from source again during GRPC building, avoid building protoc and the GRPC plugins (since it's already available on the host)? What make flags should I use?

This is what I'm using to cross-compile and works - barring that the protobuf source is again re-compiled (due to the HAS_SYSTEM_PROTOBUF=false flag)

GRPC_CROSS_COMPILE=true HAS_PKG_CONFIG=false EMBED_OPENSSL=false GRPC_CROSS_AROPTS="cr --target=elf32-little" HAS_SYSTEM_PROTOBUF=false PROTOBUF_CONFIG_OPTS="--host=arm-linux-gnueabihf --build=x86_64-pc-linux-gnu --with-protoc=/usr/local/bin/protoc"  HOST_CC=gcc HOST_CXX=g++ HOST_LD=gcc HOST_LDXX=g++ HOST_LDLIBSXX="-L<path-to-the-build-machine's-protobuf-libs>" HOST_LDLIBS_PROTOC="-L/usr/local/lib -lprotoc -lprotobuf" CC=arm-linux-gnueabihf-gcc CFLAGS="-I<path-to-openssl-header-files> -I<path-to-cares-header-files> -I<path-to-gperftools-header-files> -I<path-to-zlib-header-files>" CXX=arm-linux-gnueabihf-g++ CXXFLAGS="-I<path-to-openssl-header-files> -I<path-to-cares-header-files> -I<path-to-gperftools-header-files> -I<path-to-zlib-header-files>"  RANLIB=arm-linux-gnueabihf-ranlib LD=arm-linux-gnueabihf-gcc LDXX=arm-linux-gnueabihf-g++ AR=arm-linux-gnueabihf-ar GRPC_CROSS_LDOPTS="-L<path-to-cares-compiled-Linux/ARM-libs> -L<path-to-gperftools-compiled-Linux/ARM-libs> -L<path-to-zlib-compiled-Linux/ARM-libs>  -L<path-to-openssl-compiled-Linux/ARM-libs> -L<path-to-protobuf-compiled-Linux/ARM-libs>" make

(Apologies for the long post!)

Oh, another thing I figured out was that it's easier (in fact, the Makefile assumes this) to use gcc/arm-linux-gnueabihf-gcc and g++/arm-linux-gnueabihf-g++ for the linker (instead of invoking ld/arm-linux-gnueabihf-ld).

Reference:

[1] https://github.com/grpc/grpc/issues/9719

[2] https://groups.google.com/forum/#!topic/grpc-io/848KwP13fPc

[3] https://github.com/grpc/grpc/issues/13436

[4] https://github.com/grpc/grpc/issues/11655

[abhinava.s...@gmail.com]

Also, HAS_SYSTEM_PROTOBUF=false not only results in re-building protobuf, but generates static libprotobuf.a and libprotoc.a archives. Anyway to use shared objects?