NodeJS and Protobuf 3

[Nick Pavlica]

All,

   I'm trying to use Nats(0.9.6) with NodeJS(6.9.2) and Google Protocol Buffers v3.  When I try sending the encoded buffer I get the following error:

Error [nats://localhost:4222]: 'Unknown Protocol Operation' 

The encoded message is a Uint 8 Array, for example:

Uint8Array [ 10, 4, 78, 105, 99, 107 ] 

I'm connecting with:

var nats = require ('nats').connect({'encoding': 'binary'});

The publish function is:

nats.publish(subject, bytes, function() {

   process.exit();

});

I've found a Golang example here(https://jacobmartins.com/2016/06/06/practical-golang-getting-started-with-nats-and-related-patterns/), but haven't been able to find any with NodeJS. If there are any examples that demonstrate the use of Nats, Node, and Protobufs it would be greatly appreciated.  

Thanks!

--Nick

[derek]

Could you provide more of the source code?

Also, you can run the server with -DV flag and send the results of what the server logs when you run the app.

[Nick Pavlica]

Hi Derek,

  Thanks for taking some time to help me figure this out!  I'm new to this combination, so I'm sure I've made many simple errors.  

On Tuesday, December 20, 2016 at 6:17:20 PM UTC-7, derek wrote:

Could you provide more of the source code?

Here is the code that I have so far:

<publish.js> ----------------------------------------------------------------------------------- Begin ------------------------------------------------------------------------

'use strict';

// Connect to Nats and set encoding to binary

var nats = require('nats').connect({

  'encoding': 'binary'

});

nats.on('error', function (e) {

  console.log('Error [' + nats.options.url + ']: ' + e);

  process.exit();

});

// Import Protobuf

var pbmessages = require('./helloworld_pb');

// Create a new Protobuf messages

var pbmessage = new pbmessages.HelloRequest();

// Set pbmessage name attribute

pbmessage.setName("Protobufs with Nats.io Server");

console.log("Origional-Message: " + pbmessage);

// Serializes pbmessage to a UInt8Array

var bytes = pbmessage.serializeBinary();

console.log('Serialized-Bytes : ' + bytes);

// Test pbmessge decode

var decodedmsg = proto.helloworld.HelloRequest.deserializeBinary(bytes);

console.log("Decoded Message Test: " + decodedmsg.getName());

// Get the channel subject fromt the command line

var subject = process.argv[2];

if (subject) {

  console.log("The Channel Subject : " + subject)

}

if (!subject) {

  console.log('Usage: node-pub <subject> ');

  process.exit();

}

// Publish the binary pbmessage to Nats

nats.publish(subject, bytes, function() {

  console.log("sent" + bytes);

  process.exit();

});

<publish.js> ----------------------------------------------------------------------------------- END ------------------------------------------------------------------------

<subscribe.js> ----------------------------------------------------------------------------------- Begin ------------------------------------------------------------------------

'use strict';

// Connect to Nats and set encoding to binary

var nats = require ('nats').connect({'encoding': 'binary'});

nats.on('error', function(e) {

  console.log('Error [' + nats.options.url + ']: ' + e);

  process.exit();

});

nats.on('close', function() {

  console.log('CLOSED');

  process.exit();

});

// Set subscription topic

var subject = process.argv[2];

if (!subject) {

  console.log('Usage: node-sub <subject>');

  process.exit();

}

console.log('Listening on [' + subject + ']');

// Import Protobuf

var messages = require('./helloworld_pb');

// Receive encoded pbmessage, and print to console

nats.subscribe(subject, function(bytes) {

  var pbmsg = proto.helloworld.HelloRequest.deserializeBinary(bytes);

  console.log('Received "' + pbmsg.getName() + '"');

});

<subscribe.js> ----------------------------------------------------------------------------------- END ------------------------------------------------------------------------

<helloworld.proto>------------------------------------------------- Begin ---------------------------------------------------------------------------------------

syntax = "proto3";

package helloworld;

// The request message containing the user's name.

message HelloRequest {

  string name = 1;

}

// The response message containing the greetings

message HelloReply {

  string message = 1;

}

<helloworld.proto>------------------------------------------------- END ---------------------------------------------------------------------------------------

** I'm using the Google protobuf lib here: https://github.com/google/protobuf/tree/master/js

** This is my protoc command: protoc --js_out=import_style=commonjs,binary:. helloworld.proto

** $ protoc --version is libprotoc 3.1.0

** Here is the example console output when running the message publisher:

$node publish.js pbmsg

Origional-Message: Protobufs with Nats.io Server

Serialized-Bytes : 10,29,80,114,111,116,111,98,117,102,115,32,119,105,116,104,32,78,97,116,115,46,105,111,32,83,101,114,118,101,114

Decoded Message Test: Protobufs with Nats.io Server

The Channel Subject : pbmsg

Error [nats://localhost:4222]: 'Unknown Protocol Operation'

** Here is the example console ouput when running message subscriber:

$ node subscribe.js pbmsg

Listening on [pbmsg]

 

Also, you can run the server with -DV flag and send the results of what the server logs when you run the app.

** Here is a sample of the Nats server log to standard out:

[10107] 2016/12/21 14:10:42.053124 [TRC] 127.0.0.1:60204 - cid:34 - ->> [PING]

[10107] 2016/12/21 14:10:42.053131 [TRC] 127.0.0.1:60204 - cid:34 - <<- [PONG]

[10107] 2016/12/21 14:10:42.053554 [TRC] 127.0.0.1:60204 - cid:34 - ->> [PUB pbmsg 31]

[10107] 2016/12/21 14:10:42.053563 [TRC] 127.0.0.1:60204 - cid:34 - <<- [-ERR Unknown Protocol Operation]

[10107] 2016/12/21 14:10:42.053604 [ERR] 127.0.0.1:60204 - cid:34 - Error reading from client: Client Parser ERROR, state=30, i=127: proto='""...'

[10107] 2016/12/21 14:10:42.053612 [TRC] 127.0.0.1:60204 - cid:34 - <<- [-ERR Parser Error]

[10107] 2016/12/21 14:10:42.053626 [DBG] 127.0.0.1:60204 - cid:34 - Client connection closed

[10107] 2016/12/21 14:12:03.067197 [DBG] 127.0.0.1:60190 - cid:32 - Client Ping Timer

[10107] 2016/12/21 14:12:03.067224 [TRC] 127.0.0.1:60190 - cid:32 - <<- [PING]

[10107] 2016/12/21 14:12:03.068162 [TRC] 127.0.0.1:60190 - cid:32 - ->> [PONG]

[10107] 2016/12/21 14:13:06.458202 [DBG] 127.0.0.1:60234 - cid:35 - Client connection created

[10107] 2016/12/21 14:13:06.460588 [TRC] 127.0.0.1:60234 - cid:35 - ->> [CONNECT {"lang":"node","version":"0.6.8","verbose":false,"pedantic":false}]

[10107] 2016/12/21 14:13:06.460687 [TRC] 127.0.0.1:60234 - cid:35 - ->> [PING]

[10107] 2016/12/21 14:13:06.460695 [TRC] 127.0.0.1:60234 - cid:35 - <<- [PONG]

[10107] 2016/12/21 14:13:06.461253 [TRC] 127.0.0.1:60234 - cid:35 - ->> [PUB pbmsg 31]

[10107] 2016/12/21 14:13:06.461263 [TRC] 127.0.0.1:60234 - cid:35 - <<- [-ERR Unknown Protocol Operation]

[10107] 2016/12/21 14:13:06.461289 [ERR] 127.0.0.1:60234 - cid:35 - Error reading from client: Client Parser ERROR, state=30, i=127: proto='""...'

[10107] 2016/12/21 14:13:06.461298 [TRC] 127.0.0.1:60234 - cid:35 - <<- [-ERR Parser Error]

[10107] 2016/12/21 14:13:06.461312 [DBG] 127.0.0.1:60234 - cid:35 - Client connection closed

[10107] 2016/12/21 14:13:08.594766 [DBG] 127.0.0.1:60236 - cid:36 - Client connection created

[10107] 2016/12/21 14:13:08.597238 [TRC] 127.0.0.1:60236 - cid:36 - ->> [CONNECT {"lang":"node","version":"0.6.8","verbose":false,"pedantic":false}]

[10107] 2016/12/21 14:13:08.597307 [TRC] 127.0.0.1:60236 - cid:36 - ->> [PING]

[10107] 2016/12/21 14:13:08.597316 [TRC] 127.0.0.1:60236 - cid:36 - <<- [PONG]

[10107] 2016/12/21 14:13:08.597748 [TRC] 127.0.0.1:60236 - cid:36 - ->> [PUB pbmsg 31]

[10107] 2016/12/21 14:13:08.597761 [TRC] 127.0.0.1:60236 - cid:36 - <<- [-ERR Unknown Protocol Operation]

[10107] 2016/12/21 14:13:08.597784 [ERR] 127.0.0.1:60236 - cid:36 - Error reading from client: Client Parser ERROR, state=30, i=127: proto='""...'

[10107] 2016/12/21 14:13:08.597791 [TRC] 127.0.0.1:60236 - cid:36 - <<- [-ERR Parser Error]

[10107] 2016/12/21 14:13:08.597804 [DBG] 127.0.0.1:60236 - cid:36 - Client connection closed

[10107] 2016/12/21 14:14:03.067437 [DBG] 127.0.0.1:60190 - cid:32 - Client Ping Timer

[10107] 2016/12/21 14:14:03.067459 [TRC] 127.0.0.1:60190 - cid:32 - <<- [PING]

[10107] 2016/12/21 14:14:03.067925 [TRC] 127.0.0.1:60190 - cid:32 - ->> [PONG]

[10107] 2016/12/21 14:16:03.067676 [DBG] 127.0.0.1:60190 - cid:32 - Client Ping Timer

[10107] 2016/12/21 14:16:03.067697 [TRC] 127.0.0.1:60190 - cid:32 - <<- [PING]

[10107] 2016/12/21 14:16:03.067959 [TRC] 127.0.0.1:60190 - cid:32 - ->> [PONG]

[10107] 2016/12/21 14:18:03.067901 [DBG] 127.0.0.1:60190 - cid:32 - Client Ping Timer

[10107] 2016/12/21 14:18:03.067931 [TRC] 127.0.0.1:60190 - cid:32 - <<- [PING]

[10107] 2016/12/21 14:18:03.068216 [TRC] 127.0.0.1:60190 - cid:32 - ->> [PONG]

[10107] 2016/12/21 14:20:03.068118 [DBG] 127.0.0.1:60190 - cid:32 - Client Ping Timer

[10107] 2016/12/21 14:20:03.068138 [TRC] 127.0.0.1:60190 - cid:32 - <<- [PING]

[10107] 2016/12/21 14:20:03.068421 [TRC] 127.0.0.1:60190 - cid:32 - ->> [PONG]

[10107] 2016/12/21 14:22:03.068356 [DBG] 127.0.0.1:60190 - cid:32 - Client Ping Timer

[10107] 2016/12/21 14:22:03.068381 [TRC] 127.0.0.1:60190 - cid:32 - <<- [PING]

[10107] 2016/12/21 14:22:03.068660 [TRC] 127.0.0.1:60190 - cid:32 - ->> [PONG]

[10107] 2016/12/21 14:24:03.068600 [DBG] 127.0.0.1:60190 - cid:32 - Client Ping Timer

[10107] 2016/12/21 14:24:03.068625 [TRC] 127.0.0.1:60190 - cid:32 - <<- [PING]

[10107] 2016/12/21 14:24:03.068911 [TRC] 127.0.0.1:60190 - cid:32 - ->> [PONG]

[10107] 2016/12/21 14:24:49.023743 [DBG] 127.0.0.1:60350 - cid:37 - Client connection created

[10107] 2016/12/21 14:24:49.026231 [TRC] 127.0.0.1:60350 - cid:37 - ->> [CONNECT {"lang":"node","version":"0.6.8","verbose":false,"pedantic":false}]

[10107] 2016/12/21 14:24:49.026316 [TRC] 127.0.0.1:60350 - cid:37 - ->> [PING]

[10107] 2016/12/21 14:24:49.026323 [TRC] 127.0.0.1:60350 - cid:37 - <<- [PONG]

[10107] 2016/12/21 14:24:49.026743 [TRC] 127.0.0.1:60350 - cid:37 - ->> [PUB pbmsg 31]

[10107] 2016/12/21 14:24:49.026752 [TRC] 127.0.0.1:60350 - cid:37 - <<- [-ERR Unknown Protocol Operation]

[10107] 2016/12/21 14:24:49.026778 [ERR] 127.0.0.1:60350 - cid:37 - Error reading from client: Client Parser ERROR, state=30, i=127: proto='""...'

[10107] 2016/12/21 14:24:49.026786 [TRC] 127.0.0.1:60350 - cid:37 - <<- [-ERR Parser Error]

[10107] 2016/12/21 14:24:49.026802 [DBG] 127.0.0.1:60350 - cid:37 - Client connection closed

[10107] 2016/12/21 14:26:03.068820 [DBG] 127.0.0.1:60190 - cid:32 - Client Ping Timer

[10107] 2016/12/21 14:26:03.068844 [TRC] 127.0.0.1:60190 - cid:32 - <<- [PING]

[10107] 2016/12/21 14:26:03.069099 [TRC] 127.0.0.1:60190 - cid:32 - ->> [PONG]

[10107] 2016/12/21 14:26:25.960588 [DBG] 127.0.0.1:60358 - cid:38 - Client connection created

[10107] 2016/12/21 14:26:25.962914 [TRC] 127.0.0.1:60358 - cid:38 - ->> [CONNECT {"lang":"node","version":"0.6.8","verbose":false,"pedantic":false}]

[10107] 2016/12/21 14:26:25.962992 [TRC] 127.0.0.1:60358 - cid:38 - ->> [PING]

[10107] 2016/12/21 14:26:25.962999 [TRC] 127.0.0.1:60358 - cid:38 - <<- [PONG]

[10107] 2016/12/21 14:26:25.963410 [TRC] 127.0.0.1:60358 - cid:38 - ->> [PUB pbmsg 31]

[10107] 2016/12/21 14:26:25.963422 [TRC] 127.0.0.1:60358 - cid:38 - <<- [-ERR Unknown Protocol Operation]

[10107] 2016/12/21 14:26:25.963444 [ERR] 127.0.0.1:60358 - cid:38 - Error reading from client: Client Parser ERROR, state=30, i=127: proto='""...'

[10107] 2016/12/21 14:26:25.963451 [TRC] 127.0.0.1:60358 - cid:38 - <<- [-ERR Parser Error]

[10107] 2016/12/21 14:26:25.963464 [DBG] 127.0.0.1:60358 - cid:38 - Client connection closed

Thanks Again!

--Nick

[Ivan Kozlovic]

Nick,

I don't know much about Node.js, but looking at our node-nats-streaming repo that does use protobuf, I wonder if you should replace:

var bytes = pbmessage.serializeBinary();

with

var bytes = new Buffer(pbmessage.serializeBinary());

That being said, it looks like node-nats-streaming is using a different way to generate the protobuf:

** I'm using the Google protobuf lib here: https://github.com/google/protobuf/tree/master/js

** This is my protoc command: protoc --js_out=import_style=commonjs,binary:. helloworld.proto

protoc -I=. -I=$GOPATH/src --gogofaster_out=. protocol.proto

Again, I don't know much about Node so I hope I am not just adding noise here ;-)

Ivan.

[Nick Pavlica]

Hi Ivan,

   Thanks for taking a look at this as well!

On Wednesday, December 21, 2016 at 3:05:11 PM UTC-7, Ivan Kozlovic wrote:

Nick,

I don't know much about Node.js, but looking at our node-nats-streaming repo that does use protobuf, I wonder if you should replace:

var bytes = pbmessage.serializeBinary();

with

var bytes = new Buffer(pbmessage.serializeBinary());

  This caused the Google Protobuf lib(google-protobuf.js) to throw this error: ....  AssertionError: Failure: Type not convertible to Uint8Array. ......

 

That being said, it looks like node-nats-streaming is using a different way to generate the protobuf:

** I'm using the Google protobuf lib here: https://github.com/google/protobuf/tree/master/js

** This is my protoc command: protoc --js_out=import_style=commonjs,binary:. helloworld.proto

protoc -I=. -I=$GOPATH/src --gogofaster_out=. protocol.proto

  With protocol buffers you generate the "protobuf" based on an idl(your_idl.proto)/message definition file for language that you are using.  In your example, you are telling the protoc idl compiler to emit go code rather than javascript or xyz-lang... 

 

Again, I don't know much about Node so I hope I am not just adding noise here ;-)

Ivan.

Thanks for taking a shot at this!!

-- Nick 

[Ivan Kozlovic]

 Nick,

  With protocol buffers you generate the "protobuf" based on an idl(your_idl.proto)/message definition file for language that you are using.  In your example, you are telling the protoc idl compiler to emit go code rather than javascript or xyz-lang... 

My mistake indeed. I guess Alberto did not update the proto file with the proper `protoc` command he actually used.

Sorry for the noise, I will now let people with actual Node.js knowledge chime in ;-)

Ivan.