grpc-node AMRv6 cross-compilation with Yocto

[Maciej Pijanowski]

Hello,

I'm using [Yocto 2.6.1](https://www.yoctoproject.org/)
with
[meta-raspberrypi](http://git.yoctoproject.org/cgit/cgit.cgi/meta-raspberrypi)
layer to build the image for RaspberryPi Zero.

The gcc is `8.2.0` and the `node` is `8.12.0`.

The `grpc-node` is a dependency for a Node package I'm building. However, I'm now focused
on getting the `grpc-node` to work. It is the only package which contains native
code to be compiled. The precompiled version is for `armv7` as explained here:
https://github.com/googleapis/nodejs-speech/issues/12

I'm well aware of the issue with `node-pre-gyp` mentioned there:
https://github.com/mapbox/node-pre-gyp/issues/348
However, I think that it might not be the problem in my case since I'm building
my own toolchain, specifically tweaked for the target (RPI 0) architecture.

In fact, I'm already getting the `armv6` binary crosscompiled:

```
Attribute Section: aeabi
File Attributes
  Tag_CPU_name: "6"
  Tag_CPU_arch: v6
  Tag_ARM_ISA_use: Yes
  Tag_THUMB_ISA_use: Thumb-1
  Tag_FP_arch: VFPv2
  Tag_ABI_PCS_wchar_t: 4
  Tag_ABI_FP_denormal: Needed
  Tag_ABI_FP_exceptions: Needed
  Tag_ABI_FP_number_model: IEEE 754
  Tag_ABI_align_needed: 8-byte
  Tag_ABI_align_preserved: 8-byte, except leaf SP
  Tag_ABI_enum_size: int
  Tag_ABI_VFP_args: VFP registers
  Tag_CPU_unaligned_access: v6
```

While the `arm` precompiled binary from the npm registry looks like:

```
Attribute Section: aeabi
File Attributes
  Tag_CPU_name: "7-A"
  Tag_CPU_arch: v7
  Tag_CPU_arch_profile: Application
  Tag_ARM_ISA_use: Yes
  Tag_THUMB_ISA_use: Thumb-2
  Tag_FP_arch: VFPv3-D16
  Tag_ABI_PCS_wchar_t: 4
  Tag_ABI_FP_denormal: Needed
  Tag_ABI_FP_exceptions: Needed
  Tag_ABI_FP_number_model: IEEE 754
  Tag_ABI_align_needed: 8-byte
  Tag_ABI_align_preserved: 8-byte, except leaf SP
  Tag_ABI_enum_size: int
  Tag_ABI_VFP_args: VFP registers
  Tag_CPU_unaligned_access: v6
```

The runtime error I'm getting right now (`Illegall instruction` error is gone)
is:

```
root@raspberrypi0-wifi:~# node
> require('/usr/lib/node/firebase/node_modules/grpc')
Error: /usr/lib/node/firebase/node_modules/grpc/src/node/extension_binary/node-v57-linux-arm-glibc/grpc_node.node: undefined symbol: CRYPTO_set_locking_callback
    at Object.Module._extensions..node (module.js:682:18)
    at Module.load (module.js:566:32)
    at tryModuleLoad (module.js:506:12)
    at Function.Module._load (module.js:498:3)
    at Module.require (module.js:597:17)
    at require (internal/module.js:11:18)
```

However, both pre-compiled (for `armv7`) and the cross-compiled (for `armv6`)
binaries seem to have this symbol undefined:

precompiled:

```
    25: 00000000     0 NOTYPE  GLOBAL DEFAULT  UND CRYPTO_set_locking_callba
```

cross-compiled:

```
    27: 00000000     0 NOTYPE  GLOBAL DEFAULT  UND CRYPTO_set_locking_callba
```

Do I need anything more than the `grpc_node.node` binary correctly compiled for
the target architecture?

Now I'm wondering what's the OpenSSL requirement of the `grpc`. Does it require
OpenSSL at compile time / at runtime?

I can see that there is an option for `BoringSSL` (I assume linked statically?)
or to use the `OpenSSL` dynamic library from the system - is that correct?

How one can choose which one to use? What would be the "default" setting for
the Node package on ARM Linux system, which depends on the `grpc-node` package?

As a side note, I faced some issues as descriebd in various GCC 8 issues like this one:
https://github.com/grpc/grpc/issues/18689 and silenced them down with gcc
flags: `"-Wno-class-memaccess -Wno-ignored-qualifiers -Wno-cast-function-type"`
to keep going.

Please note that I'm far from Node expert / dev. I work on various build
systems, mostly in embedded area. I'm willing to help in resolving the
cross-compilation issues. However, I would need to get some better understanding
of how the `grpc` is being built - especially it's requirements on the OpenSSL.

Thanks for any feedback

[Nicolas Noble]

OpenSSL is supposed to be provided by your nodejs runtime. If the file you're building can't find this symbol at runtime, it means your nodejs runtime isn't providing its necessary dependencies properly.

Where does this runtime come from?

--
You received this message because you are subscribed to the Google Groups "grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+u...@googlegroups.com.
To post to this group, send email to grp...@googlegroups.com.
Visit this group at https://groups.google.com/group/grpc-io.
To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/26285314-420b-41cf-905d-1dd9eac8632a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Nicolas Noble]

Well, yes, the problem is with the compilation process, somehow :-)

I mean, you probably want to have a look at https://github.com/nodejs/node-gyp/wiki/Linking-to-OpenSSL and https://github.com/nicolasnoble/openssl-nodejs-ubuntu-demo for a baseline openssl-in-nodejs-native-module test. Unfortunately, it's not really easy to diagnose what's going on remotely like this, so you're going to have to poke at it yourselves. But the gist is that these should work.

My repository can be built and ran the following way:

$ npm install -g node-gyp

$ npm install nan

$ node-gyp configure

$ node-gyp build

$ node ./openssl-example.js

This may narrow your spelunking down a bit.

On Wed, Apr 17, 2019 at 3:23 PM Maciej Pijanowski <maciej.p...@3mdeb.com> wrote:

On 17.04.2019 16:56, Nicolas Noble wrote:

OpenSSL is supposed to be provided by your nodejs runtime. If the file you're building can't find this symbol at runtime, it means your nodejs runtime isn't providing its necessary dependencies properly.

Where does this runtime come from?

Thanks a lot for interest in this subject.

A `nodejs` itself is also cross-compiled as part of the build. In fact, each system
component is cross-compiled as part of the Yocto build.

Here we can see how the nodejs is configured:
https://github.com/openembedded/meta-openembedded/blob/thud/meta-oe/recipes-devtools/nodejs/nodejs_8.12.0.bb#L49

The `shared-openssl` configure switch is passed, so IIUC, the node and it's
modules should refer to the shared OpenSSL libraries in the system?:

```
    --shared-openssl    link to a shared OpenSSl DLL instead of static linking
```

Nevertheless, neither the runtime, nor the system itself does not seem to be
the problem.

I've compiled the grpc_node.node binary natively on the RPI0 (Raspbian) using
the `npm rebuild --build-from-source grpc`. After copying the natively compiled binary
to the system built with Yocto, I can correctly load the grpc modules (or the
modules which depends on it) in the node. I believe that suggests that the
issue is within the compilation process rather than the system components.

I think one more test point would be to compile natively on the RPI0, but using the
system built with Yocto, rather than the Raspbian. I would have the same
version of toolchain as in case of the cross-compilation, but it would be the
native toolchain on the target, not the cross-toolchain on the host. To do that
I would need to build the native toolchain and deploy it to the RPI0 image as
well.

Binaries attached below, if someone would be interested:

* binary compiled natively on RPI0 (Raspbian):
https://cloud.3mdeb.com/index.php/s/bXnSwARKbReS9kZ

* binary cross-compiled in Yocto:
https://cloud.3mdeb.com/index.php/s/GpxYcLxcWH8LWEo

* readelf dump (natively compiled binary):
https://cloud.3mdeb.com/index.php/s/FkBpQm4nEqkBRfi

* readelf dump (cross-compiled binary):
https://cloud.3mdeb.com/index.php/s/SocJX2wn3iTwroP

-- 
Maciej Pijanowski
Embedded Systems Engineer
https://3mdeb.com | @3mdeb_com