Cross Compiling gRPC on ARM with buildroot

Mark Fine

unread,

Aug 30, 2018, 12:05:57 PM8/30/18

to grpc.io

Hi!

We're having issues cross compiling gRPC in buildroot. There's been some prior work on getting gRPC into buildroot:

https://gfiber.googlesource.com/buildroot/+/refs/heads/master/package/grpc

http://lists.busybox.net/pipermail/buildroot/2017-April/189714.html

https://patchwork.ozlabs.org/patch/917779/

but nothing seems to have made it to upstream. In particular, we're running into issues around host compiling having target libraries included and linked against. Here's an example of host linking against target libraries:

c++ -L/piksi_buildroot/buildroot/output/host/bin/../arm-buildroot-linux-gnueabihf/sysroot/usr//lib -Wl,-rpath,/piksi_buildroot/buildroot/output/host/bin/../arm-buildroot-linux-gnueabihf/sysroot/usr//lib -flto -flto -g -fPIC -Llibs/opt -pthread -L/piksi_buildroot/buildroot/output/host/bin/../arm-buildroot-linux-gnueabihf/sysroot/usr/lib -L/piksi_buildroot/buildroot/output/host/bin/../arm-buildroot-linux-gnueabihf/sysroot/usr/lib /piksi_buildroot/buildroot/output/build/grpc_custom-v1.12.0/objs/opt/src/compiler/cpp_plugin.o /piksi_buildroot/buildroot/output/build/grpc_custom-v1.12.0/libs/opt/libgrpc_plugin_support.a -lprotoc -lprotobuf -ldl -lrt -lm -lpthread -lz -lcares -lprotoc -lprotobuf -o /piksi_buildroot/buildroot/output/build/grpc_custom-v1.12.0/bins/opt/grpc_cpp_plugin

Here's an example of host including target libraries:

/usr/bin/g++ -std=c++11 -I/piksi_buildroot/buildroot/output/host/bin/../arm-buildroot-linux-gnueabihf/sysroot/usr/include -pthread -Ithird_party/googletest/googletest/include -Ithird_party/googletest/googlemock/include -I/piksi_buildroot/buildroot/output/host/bin/../arm-buildroot-linux-gnueabihf/sysroot/usr/include -Ithird_party/address_sorting/include -g -Wall -Wextra -Werror -Wno-long-long -Wno-unused-parameter -DOSATOMIC_USE_INLINED=1 -Wno-deprecated-declarations -O2 -Wframe-larger-than=16384 -fPIC -I. -Iinclude -I/piksi_buildroot/buildroot/output/build/grpc_custom-v1.12.0/gens -DPB_FIELD_16BIT -DNDEBUG -DINSTALL_PREFIX=\"/usr/local\" -I/piksi_buildroot/buildroot/output/host/bin/../arm-buildroot-linux-gnueabihf/sysroot/usr/include -I/piksi_buildroot/buildroot/output/host/bin/../arm-buildroot-linux-gnueabihf/sysroot/usr/include -MMD -MF /piksi_buildroot/buildroot/output/build/grpc_custom-v1.12.0/objs/opt/src/compiler/cpp_generator.dep -c -o /piksi_buildroot/buildroot/output/build/grpc_custom-v1.12.0/objs/opt/src/compiler/cpp_generator.o src/compiler/cpp_generator.cc

In file included from /piksi_buildroot/buildroot/output/host/bin/../arm-buildroot-linux-gnueabihf/sysroot/usr/include/features.h:447:0,

from /usr/include/x86_64-linux-gnu/c++/6/bits/os_defines.h:39,

from /usr/include/x86_64-linux-gnu/c++/6/bits/c++config.h:507,

from /usr/include/c++/6/bits/stl_algobase.h:59,

from /usr/include/c++/6/bits/stl_tree.h:63,

from /usr/include/c++/6/map:60,

from src/compiler/cpp_generator.cc:19:

/piksi_buildroot/buildroot/output/host/bin/../arm-buildroot-linux-gnueabihf/sysroot/usr/include/gnu/stubs.h:7:29: fatal error: gnu/stubs-soft.h: No such file or directory

# include <gnu/stubs-soft.h>

^

Has anyone had success cross compiling gRPC on buildroot? Or just cross compiling gRPC on ARM? The interactions between the host and target compiling with and without the GRPC_CROSS_COMPILE flag are causing us issues, especially around the host plugins. Thanks for any help!

Mark

Nicolas Noble

unread,

Aug 30, 2018, 12:34:38 PM8/30/18

to mark...@gmail.com, grpc.io

Yes, it's definitely doable to cross compile for ARM - in fact we do it regularly for some of our prebuilt binaries.

Now each environment and case is different. What steps are you following exactly?

--
You received this message because you are subscribed to the Google Groups "grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+u...@googlegroups.com.
To post to this group, send email to grp...@googlegroups.com.
Visit this group at https://groups.google.com/group/grpc-io.
To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/6ef7f44c-60f9-4afe-92ff-2ba139febf12%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Mark Fine

unread,

Aug 30, 2018, 1:00:08 PM8/30/18

to Nicolas Noble, grpc.io

Hi, thanks for the reply - we've got a simple buildroot mk file down to:

################################################################################

#

# grpc

#

################################################################################

GRPC_CUSTOM_VERSION = v1.12.0

GRPC_CUSTOM_SITE = https://github.com/grpc/grpc.git

GRPC_CUSTOM_SITE_METHOD = git

GRPC_CUSTOM_LICENSE = BSD-3-Clause

GRPC_CUSTOM_LICENSE_FILES = LICENSE

GRPC_CUSTOM_DEPENDENCIES = gflags gtest c-ares openssl protobuf zlib

GRPC_CUSTOM_INSTALL_STAGING = YES

GRPC_CUSTOM_MAKE_ENV = \

$(TARGET_MAKE_ENV) \

CC="$(TARGET_CC)" \

CXX="$(TARGET_CXX)" \

LD="$(TARGET_CC)" \

CFLAGS="$(TARGET_CFLAGS)" \

LDFLAGS="$(TARGET_LDFLAGS)" \

STRIP="$(TARGET_STRIP)"

GRPC_CUSTOM_MAKE_OPTS = \

PROTOC="$(HOST_DIR)/usr/bin/protoc"

GRPC_CUSTOM_INSTALL_TARGET_OPTS = \

prefix="$(TARGET_DIR)/usr"

GRPC_CUSTOM_INSTALL_STAGING_OPTS = \

prefix="$(STAGING_DIR)/usr"

define GRPC_CUSTOM_BUILD_CMDS

$(GRPC_CUSTOM_MAKE_ENV) $(MAKE) $(GRPC_CUSTOM_MAKE_OPTS) -C $(@D) \

static

endef

define GRPC_CUSTOM_INSTALL_STAGING_CMDS

$(GRPC_CUSTOM_MAKE_ENV) $(MAKE) $(GRPC_CUSTOM_INSTALL_STAGING_OPTS) -C $(@D) \

install-headers install-static_c install-static_cxx

endef

define GRPC_CUSTOM_INSTALL_TARGET_CMDS

$(GRPC_CUSTOM_MAKE_ENV) $(MAKE) $(GRPC_CUSTOM_INSTALL_TARGET_OPTS) -C $(@D) \

install-static_c install-static_cxx

endef

$(eval $(generic-package))

My understanding is that this should be calling make static against grpc's root Makefile with the env setup. The target compiling seems to go fine:

/piksi_buildroot/buildroot/output/host/bin/arm-linux-g++ -I/piksi_buildroot/buildroot/output/host/bin/../arm-buildroot-linux-gnueabihf/sysroot/usr/include -pthread -Ithird_party/googletest/googletest/include -Ithird_party/googletest/googlemock/include -I/piksi_buildroot/buildroot/output/host/bin/../arm-buildroot-linux-gnueabihf/sysroot/usr/include -Ithird_party/address_sorting/include -g -Wall -Wextra -Werror -Wno-long-long -Wno-unused-parameter -DOSATOMIC_USE_INLINED=1 -Wno-deprecated-declarations -O2 -Wframe-larger-than=16384 -fPIC -I. -Iinclude -I/piksi_buildroot/buildroot/output/build/grpc_custom-v1.12.0/gens -DPB_FIELD_16BIT -DNDEBUG -DINSTALL_PREFIX=\"/usr/local\" -I/piksi_buildroot/buildroot/output/host/bin/../arm-buildroot-linux-gnueabihf/sysroot/usr/include -I/piksi_buildroot/buildroot/output/host/bin/../arm-buildroot-linux-gnueabihf/sysroot/usr/include -std=c++11 -MMD -MF /piksi_buildroot/buildroot/output/build/grpc_custom-v1.12.0/objs/opt/src/cpp/common/auth_property_iterator.dep -c -o /piksi_buildroot/buildroot/output/build/grpc_custom-v1.12.0/objs/opt/src/cpp/common/auth_property_iterator.o src/cpp/common/auth_property_iterator.cc

But then we end up running into issues once we move to host compiling around the plugins

[HOSTCXX] Compiling src/compiler/cpp_plugin.cc

[HOSTCXX] Compiling src/compiler/node_plugin.cc

[HOSTCXX] Compiling src/compiler/csharp_plugin.cc

[HOSTCXX] Compiling src/compiler/objective_c_plugin.cc

[HOSTCXX] Compiling src/compiler/php_plugin.cc

[HOSTCXX] Compiling src/compiler/python_plugin.cc

[HOSTCXX] Compiling src/compiler/ruby_plugin.cc

[HOSTLD] Linking /piksi_buildroot/buildroot/output/build/grpc_custom-v1.12.0/bins/opt/grpc_cpp_plugin

And have tried any number of settings get them to work. For instance, with the above configuration we error on host linking:

c++ -L/piksi_buildroot/buildroot/output/host/bin/../arm-buildroot-linux-gnueabihf/sysroot/usr//lib -Wl,-rpath,/piksi_buildroot/buildroot/output/host/bin/../arm-buildroot-linux-gnueabihf/sysroot/usr//lib -flto -flto -g -fPIC -Llibs/opt -pthread -L/piksi_buildroot/buildroot/output/host/bin/../arm-buildroot-linux-gnueabihf/sysroot/usr/lib -L/piksi_buildroot/buildroot/output/host/bin/../arm-buildroot-linux-gnueabihf/sysroot/usr/lib /piksi_buildroot/buildroot/output/build/grpc_custom-v1.12.0/objs/opt/src/compiler/cpp_plugin.o /piksi_buildroot/buildroot/output/build/grpc_custom-v1.12.0/libs/opt/libgrpc_plugin_support.a -lprotoc -lprotobuf -ldl -lrt -lm -lpthread -lz -lcares -lprotoc -lprotobuf -o /piksi_buildroot/buildroot/output/build/grpc_custom-v1.12.0/bins/opt/grpc_cpp_plugin

is there some ENV we're missing here or should add to help things out? Thanks for any help or advice!

Mark

To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+unsubscribe@googlegroups.com.

Mark Fine

unread,

Aug 30, 2018, 6:54:23 PM8/30/18

to Nicolas Noble, grpc.io

We've had some luck with patching grpc's Makefile to update host flags:

diff -ur grpc_custom-v1.12.0/Makefile grpc_custom-v1.12.0.new/Makefile
--- grpc_custom-v1.12.0/Makefile        2018-05-15 17:10:37.000000000 +0000
+++ grpc_custom-v1.12.0.new/Makefile    2018-08-30 20:28:31.357410038 +0000
@@ -439,11 +439,11 @@
 DEFINES += $(EXTRA_DEFINES)
 LDLIBS += $(EXTRA_LDLIBS)

-HOST_CPPFLAGS = $(CPPFLAGS)
-HOST_CFLAGS = $(CFLAGS)
-HOST_CXXFLAGS = $(CXXFLAGS)
-HOST_LDFLAGS = $(LDFLAGS)
-HOST_LDLIBS = $(LDLIBS)
+HOST_CPPFLAGS += $(CPPFLAGS)
+HOST_CFLAGS += $(CFLAGS)
+HOST_CXXFLAGS += $(CXXFLAGS)
+HOST_LDFLAGS += $(LDFLAGS)
+HOST_LDLIBS += $(LDLIBS)

does that sound reasonable? Thanks!

Mark

Nicolas Noble

unread,

Aug 30, 2018, 7:35:54 PM8/30/18

to mark...@gmail.com, grpc.io

That's... more than reasonable actually, yes.

To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+u...@googlegroups.com.

To post to this group, send email to grp...@googlegroups.com.
Visit this group at https://groups.google.com/group/grpc-io.
To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/6ef7f44c-60f9-4afe-92ff-2ba139febf12%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "grpc.io" group.

To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+u...@googlegroups.com.

To post to this group, send email to grp...@googlegroups.com.
Visit this group at https://groups.google.com/group/grpc-io.

To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/CANRZ_fmNKqZXZE-8Yx%2Bgu%3DWKpouDw6c32o_nrSUm_WH4pOHjcw%40mail.gmail.com.

ja...@swift-nav.com

unread,

Aug 31, 2018, 12:15:34 PM8/31/18

to grpc.io

We put up a pull request for the changes here:

https://github.com/grpc/grpc/pull/16527

Reply all

Reply to author

Forward