generate big endian protobuf code on little endian machine

[kerry crouse]

I have been through the online documentation, readmes, and protoc help and I'm missing how to do this: generate big endian protobuf code on little endian machine. Can someone point me to a howto? My build machine is a Linux Debian x86 machine. My target machine is a 32 bit PowerPC device. Thanks!

[David L. Jones]

(I'm assuming this is for C++...)

The protobuf libraries (including the code generated by the protobuf compiler) should internally contain the necessary #ifdef/etc. guards to handle endian issues. For example:

https://github.com/protocolbuffers/protobuf/blob/master/src/google/protobuf/wire_format_lite.cc#L319-L369

In other words, it should Just Work (TM) with your cross-build toolchain.

If you use the Debian packages, make sure you grab the backports (otherwise you'll likely get a very old version).

If you want to build completely from source, both the autoconf and CMake builds should work as usual for cross-buliding. (If you also build and install natively, you can pass `--with-protoc` to configure.)

If you use a different build system, the changes are a tiny bit out of scope for our instructions. Again, the major host tool consideration is the `protoc` binary, which generates .pb.cc, etc. files.

[David L. Jones]

(I think Kerry's reply went to a different thread...)

On Thursday, April 23, 2020 at 5:11:05 PM UTC-7, Crouse, Kerry wrote:

David,

    Thank you for your very prompt reply! I am required (for compatibility) to use version 2.6.0. I'm not sure this makes much of a difference. What I am attempting to ultimately accomplish is turn the .proto files into code I can feed into the PowerPC cross compiler to generate code as part of the static image to load onto the PowerPC.

Yes, this is something that (conceptually) should work. (But there are more details...)

 

Perhaps I'm not thinking about this process in the correct manner. If I generate Protoc using that cross compiler, Protoc will have to be run on the PowerPC? If so, my goal is defeated. I don't think, however, that running the x86 Protoc will give me the correct data structures and code. Perhaps I'm completely backwards. 

This is notionally correct. However, the compatibility and ABI surfaces within the protobuf suite are ... let's say, "complex." :-)

If the PPC32 target uses version 2.6.0, then you'll need to use the protoc from that version. The generated protobuf code is version-locked, in the sense that the generated code is effectively API- and ABI-locked to internal APIs in the (non-generated) protobuf runtime libraries. (This is somewhat similar to Qt and moc, if you are familiar with cross-building Qt-based projects... protobuf has a few more notional "API surfaces," but the idea is similar.)

If you have the sources that were used to build the ppc32 runtimes, you should be able to build for your host using those. Alternatively, you can grab a tarball from that release here:

https://github.com/protocolbuffers/protobuf/releases/tag/v2.6.0

I would also suggest upgrading from 2.6.0 if you can (but it sounds like this may be a hard constraint for you... in which case, you have my sympathies).

--dlj

[kerry crouse]

If I run the PowerPC cross compiler and create a Protoc with it, that Protoc will only run on a PowerPC. I am trying to set up the PowerPC Protobuf files, the .pb.cc files, on the x86 build machine to cross compile the image for the PowerPC machine. Is not the cross compiled Protoc meant to run directly on the PowerPC machine?

[David L. Jones]

The sources generated by protoc do not vary by platform... protoc isn't a cross-compiler itself, it's just a source code generator.

The goal would be to get the protoc binary for your build environment, and use it to generate the sources to feed into your host/target platform.

In other words, you'll need to build the protobuf suite once to get `protoc` for the builder. Then, you'll use that `protoc` to build a second time for the target.

If you are using the autoconf-based build, you can pass `--with-protoc` to the configure script to point to pick the correct protoc binary to use.

I was able to do this using two build trees. The "-build" tree is x86-64, and targets the builder:

~/protobuf-2.6.0-build$ (./autogen.sh && ./configure && make -j32) >& make.log

Then, use that to build for ppc:

~/protobuf-2.6.0-build$ PROTOC=$PWD/src/protoc

~/protobuf-2.6.0-build$ cd ../protobuf-2.6.0

~/protobuf-2.6.0$ (./autogen.sh && ./configure --with-protoc=$PROTOC --host=powerpc-linux-gnu && make -j32) >& make.log

~/protobuf-2.6.0$ readelf -h src/.libs/libprotobuf.so.9.0.0

ELF Header:

  Magic:   7f 45 4c 46 01 02 01 00 00 00 00 00 00 00 00 00

  Class:                             ELF32

  Data:                              2's complement, big endian

  Version:                           1 (current)

  OS/ABI:                            UNIX - System V

  ABI Version:                       0

  Type:                              DYN (Shared object file)

  Machine:                           PowerPC

(etc.)

(Of course, you can install the x86-64 build if you want, but I would recommend keeping it separate. Version 2.6.0 is really old, so you wouldn't want to use it for new projects.)