Cross compiling to ARM on X64 fails, uses ARM js_embed

[Maxx Becker]

I'm a newbie, but I have successfully built the protocol buffers sources (version 3.1.0) on Ubuntu 14.04 x86_64 and installed, as per https://github.com/google/protobuf/blob/master/src/README.md

I am now attempting to cross-compile to an ARM target, using the cross-compiling info from the README above and the following as references: https://groups.google.com/forum/#!topic/protobuf/ZWuTzg-N0E0 

After running 'make clean' and re-configuring to use the arm-linux-gnueabihf compilers and the previously installed x64 protoc, I encounter a error that indicates it is attempting to use the js_embed binary built for ARM in the process.

Relevant info as follows:

$ which protoc js_embed

/usr/bin/protoc

/usr/bin/js_embed

$ file /usr/bin/protoc /usr/bin/js_embed

/usr/bin/protoc:   ELF 64-bit LSB  executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.24, BuildID[sha1]=637edeade5ad13f462551a45ae1285e55a2a1f3c, not stripped

/usr/bin/js_embed: ELF 64-bit LSB  executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.24, BuildID[sha1]=e49255de4711facf660f838ad7bad9b01525cf12, not stripped

$ arm-linux-gnueabihf-gcc --version

arm-linux-gnueabihf-gcc (Ubuntu/Linaro 4.8.4-2ubuntu1~14.04.1) 4.8.4

$ arm-linux-gnueabihf-g++ --version

arm-linux-gnueabihf-g++ (Ubuntu/Linaro 4.8.4-2ubuntu1~14.04.1) 4.8.4

$ g++ --version

g++ (Ubuntu 4.8.4-2ubuntu1~14.04.3) 4.8.4

$ gcc --version

gcc (Ubuntu 4.8.4-2ubuntu1~14.04.3) 4.8.4

$ make clean

$ ./configure --host=arm-linux-gnueabihf CC=arm-linux-gnueabihf-gcc CXX=arm-linux-gnueabihf-g++ --with-protoc=/usr/bin/protoc --prefix=/programs/protobuf

(Can post output if desired)

$ make

make  all-recursive

make[1]: Entering directory `/storage/tools/protobuf'

Making all in .

make[2]: Entering directory `/storage/tools/protobuf'

make[2]: Leaving directory `/storage/tools/protobuf'

Making all in src

make[2]: Entering directory `/storage/tools/protobuf/src'

/usr/bin/protoc -I. --cpp_out=. google/protobuf/any_test.proto google/protobuf/compiler/cpp/cpp_test_bad_identifiers.proto google/protobuf/map_lite_unittest.proto google/protobuf/map_proto2_unittest.proto google/protobuf/map_unittest.proto google/protobuf/unittest_arena.proto google/protobuf/unittest_custom_options.proto google/protobuf/unittest_drop_unknown_fields.proto google/protobuf/unittest_embed_optimize_for.proto google/protobuf/unittest_empty.proto google/protobuf/unittest_enormous_descriptor.proto google/protobuf/unittest_import_lite.proto google/protobuf/unittest_import.proto google/protobuf/unittest_import_public_lite.proto google/protobuf/unittest_import_public.proto google/protobuf/unittest_lite_imports_nonlite.proto google/protobuf/unittest_lite.proto google/protobuf/unittest_mset.proto google/protobuf/unittest_mset_wire_format.proto google/protobuf/unittest_no_arena_lite.proto google/protobuf/unittest_no_arena_import.proto google/protobuf/unittest_no_arena.proto google/protobuf/unittest_no_field_presence.proto google/protobuf/unittest_no_generic_services.proto google/protobuf/unittest_optimize_for.proto google/protobuf/unittest_preserve_unknown_enum2.proto google/protobuf/unittest_preserve_unknown_enum.proto google/protobuf/unittest.proto google/protobuf/unittest_proto3_arena.proto google/protobuf/unittest_proto3_arena_lite.proto google/protobuf/unittest_proto3_lite.proto google/protobuf/unittest_well_known_types.proto google/protobuf/util/internal/testdata/anys.proto google/protobuf/util/internal/testdata/books.proto google/protobuf/util/internal/testdata/default_value.proto google/protobuf/util/internal/testdata/default_value_test.proto google/protobuf/util/internal/testdata/field_mask.proto google/protobuf/util/internal/testdata/maps.proto google/protobuf/util/internal/testdata/oneofs.proto google/protobuf/util/internal/testdata/proto3.proto google/protobuf/util/internal/testdata/struct.proto google/protobuf/util/internal/testdata/timestamp_duration.proto google/protobuf/util/internal/testdata/wrappers.proto google/protobuf/util/json_format_proto3.proto google/protobuf/util/message_differencer_unittest.proto google/protobuf/compiler/cpp/cpp_test_large_enum_value.proto

touch unittest_proto_middleman

depbase=`echo google/protobuf/compiler/js/embed.o | sed 's|[^/]*$|.deps/&|;s|\.o$||'`;\

arm-linux-gnueabihf-g++ -std=c++11 -DHAVE_CONFIG_H -I. -I..    -pthread -DHAVE_PTHREAD=1  -Wall -Wno-sign-compare -O2 -g -DNDEBUG -MT google/protobuf/compiler/js/embed.o -MD -MP -MF $depbase.Tpo -c -o google/protobuf/compiler/js/embed.o google/protobuf/compiler/js/embed.cc &&\

mv -f $depbase.Tpo $depbase.Po

/bin/bash ../libtool  --tag=CXX   --mode=link arm-linux-gnueabihf-g++ -std=c++11 -pthread -DHAVE_PTHREAD=1  -Wall -Wno-sign-compare -O2 -g -DNDEBUG -pthread  -o js_embed google/protobuf/compiler/js/embed.o  

libtool: link: arm-linux-gnueabihf-g++ -std=c++11 -pthread -DHAVE_PTHREAD=1 -Wall -Wno-sign-compare -O2 -g -DNDEBUG -pthread -o js_embed google/protobuf/compiler/js/embed.o  -pthread

oldpwd=`pwd` && cd . && \

$oldpwd/js_embed google/protobuf/compiler/js/well_known_types/any.js google/protobuf/compiler/js/well_known_types/struct.js google/protobuf/compiler/js/well_known_types/timestamp.js > $oldpwd/google/protobuf/compiler/js/well_known_types_embed.cc

/bin/bash: line 1: /tools/protobuf/src/js_embed: cannot execute binary file: Exec format error

make[2]: *** [google/protobuf/compiler/js/well_known_types_embed.cc] Error 126

make[2]: Leaving directory `/storage/tools/protobuf/src'

make[1]: *** [all-recursive] Error 1

make[1]: Leaving directory `/storage/tools/protobuf'

make: *** [all] Error 2

$ file src/js_embed 

src/js_embed: ELF 32-bit LSB  executable, ARM, EABI5 version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.32, BuildID[sha1]=74fb68528ae9db46637e1570eeda640ae9f318f1, not stripped

It's unclear how I can direct the compilation process to use the js_embed binary in /usr/bin, instead of the js_embed binary it builds for ARM. Any help or direction is appreciated. Thanks! 

[Feng Xiao]

+acozzette

The js_embed binary was added very recently. The cross-build support for that binary is missing at the moment. As a temporary solution, you can use this version of protobuf (which doesn't have the js_embed binary):

https://github.com/google/protobuf/releases/tag/v3.1.0

--
You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscribe@googlegroups.com.
To post to this group, send email to prot...@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

[Adam Cozzette]

I hadn't thought about the cross-compilation use case for this. It seems to me that there are a few different ways to solve this:

1. Just check well_known_types_embed.cc into Git and we can update it manually when necessary. This is probably the easiest solution but is a little bit messy since it requires us to check in another build artifact.

2. Fix the Autotools build to do the cross-compilation properly on its own. For this we would probably want to use the AX_PROG_CXX_FOR_BUILD macro from here as a way of finding the right C++ compiler to use for building binaries that need to run as part of the build.

3. Rely on Bazel for cross-compilation since it is pretty good at handling this automatically: https://bazel.build/versions/master/docs/bazel-user-manual.html#configurations

4. Rewrite js/embed.cc as a Python or Perl or shell script so that it doesn't require compilation. The problem with this is that it would add another dependency to the build and might make it less portable (e.g. for Windows).

Let me try using the AX_PROG_CXX_FOR_BUILD macro and see if that will work.

[Adam Cozzette]

I updated the Autotools build so that it should now correctly build js_embed in a form that's executable on the build machine. It is not in master yet but it is on my cross-compilation branch here: https://github.com/acozzette/protobuf/tree/cross-compilation Maxx, when you have a chance you could you try building on that branch and let me know if it works?

[Adam Cozzette]

I went ahead and merged this to master with pull request 2565 and so cross compilation should now work again.

[Maxx Becker]

Thanks so much for your replies! I was able to limp along initially by copying the x64 js_embed binary from /usr/bin to src/ before cross-compiling. 

make check failed all tests in that case but the tools appear to work!

I tried both of your suggestions, with success, as described below. Before doing so, I ran "make uninstall" and "make clean" for both cross-compile and host build configurations so everything was wiped out, then I re-ran the configure and build steps for the host arch before attempting to cross-compile using the same configuration commands noted previously.

@Feng Xiao, I obtained the protobuf-cpp-3.1.0 files from https://github.com/google/protobuf/releases and had success.

@Adam Cozzette I checked out your branch, verified the macro change, built and had success.

In both cases, "make check" failed all tests because the host machine was unable to execute the binaries in .lib, as they were built in this case for the ARM arch.

-Maxx

https://github.com/google/protobuf/releases/tag/v3.1.0

To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+u...@googlegroups.com.

To post to this group, send email to prot...@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+u...@googlegroups.com.

[Adam Cozzette]

Great, I'm glad it's working now. Now that you mention it I think there was another bug that js_embed was getting installed to /usr/bin, which doesn't really make sense since that program is just used as part of the build and isn't useful for anything else. I believe that is now fixed, too.

To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscribe@googlegroups.com.

[Alex Biddulph]

@Adam Cozzette I have a further need to also control which libraries js_embed uses at run time as the necessary run time libraries are not in the standard system search paths. Is there a way to add an LD_LIBRARY_PATH_FOR_BUILD type environment variable, or some other way of controlling this?

[Adam Cozzette]

The js_embed binary shouldn't need anything but the C++ standard library. What kind of error are you getting?

On Tue, Dec 12, 2017 at 4:55 PM, Alex Biddulph <bid...@gmail.com> wrote:

@Adam Cozzette I have a further need to also control which libraries js_embed uses at run time as the necessary run time libraries are not in the standard system search paths. Is there a way to add an LD_LIBRARY_PATH_FOR_BUILD type environment variable, or some other way of controlling this?

--

[Alex Biddulph]

We ultimately found an alternative method of handling this issue.

The problem we were getting was that the binary (more accurately the dynamic loader) was finding the wrong version of C++ standard library, basically it was picking up our ARM libstdc++ because that was in the LD_LIBRARY_PATH.