Handling buffers, file and function nodes

[Josh Conway]

Greetings,

I've been racking my brain trying to figure out how to handle a buffer that's being passed by the File node to a function node, so the data can be manipulated.

The data in the file is an HTML format, whereas normally javascript throws errors on things like " and \ . I do not want to have to modify any of my stored files, so the buffer seems to be the best way forward. To that, I've tried:

manipulating msg.payload directly (type errors, or no method indexOf found)

var titlePos = msg.payload.indexOf("title:")+6; // and have the position in that variable. I get 'no method indexOf' doing this.

copying the buffer into a created buffer and working on that (random jibberish)


I've been following the documentation here: https://nodejs.org/api/buffer.html
And everything seems to work if I don't touch msg.payload . Once I do that, all I get are endless amounts of failure modes.

All I want to do is search the buffer in msg.payload for text and save it as individual buffers passed out of msg.payload

Can someone help me what I'm doing wrong?

[Mark Setrem]

Have you tried

 msg.payload = msg.payload.toString();

[Josh Conway]

That's what I had the file node doing initially, with a string output. It unfortunately cannot handle HTML encoding, as the variable gets clobbered by " and \ . That's why I had to switch to handling a buffer. Ideally, I'd much rather deal with string data, but it would require non-compliant HTML using extra escape codes.

[Nicholas O'Leary]

Josh, can you give an example of what you mean by 'clobbered by " and \ ' ? I assume this is a file encoding thing (not using UTF-8 or some such).

--
http://nodered.org
---
You received this message because you are subscribed to the Google Groups "Node-RED" group.
To unsubscribe from this group and stop receiving emails from it, send an email to node-red+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Josh Conway]

Sure. Here's an example if I

const buf1 = new Buffer(msg.payload);
buf1[0] = 255;
msg.payload = buf1 ;

resultant:

31 Jan 15:05:46 - [error] [function:70fe7b4a.640084] TypeError: Object P�� p�� Y { �ߜl� 8 � ��� ,� �� � h2} �1 ��� ��} � h p�� c �/80k p�� -�� � h ��� �� ��� ��� p�� ��� 0�� p�� �v� X� 8 � � � � � ( � � � 0 � � �� �
� H
   � (
      � �
� 8� � �� � � 0 � x � ��� � h ��������� v �g�v� ��� � h �������� �g�v� �� ��� ���� �g�v� ��� H0k P�� p�� ��� �g�v� p �v� � h p�� ������g�v� �v� �0k � �v�   �0k �m�v� @�� � �v� ] ��� � �v� �0k ��� ��� H h �0k ����`p�� � ��� �� a�g�v� 0
                  �v� X� � ��� ��� X0k ���p�� �g�v� �� 8 � � � � � ( � p �v�       h0k ��� p � has no method 'indexOf'
31 Jan 15:05:46 - [info] [file:2f6200a9.621f7] appended to file: /home/nodeuser/bloglist.txt
31 Jan 15:05:46 - [error] [function:70fe7b4a.640084] TypeError: Object Z�"I�H2k ARH�E�H�� �� �� H�X��{
                      ��� �{ �� H�x H���    I�EP�� H�� �<$H�@3�3�H�t�@�� �O H�^��[
      �㇀� �; N
                �2 p h �~� �|� �0 h 0s�v� ���v� }� ���l� �}� �}� ~� �~� � �������� �/k ���l� @y� ��� L� ��H��H�����;�������> �0k � � ���+�N
                                                                      H�v �� � �A��J�L �J�L �I�� L� ��H�0k � � ��L $H�D�H��H�t$ �^$ �FH�t$ �˃� � L���� �H�A��J�L �J�L �I �� L� ��H��H��I;������H�4ЋN
                                     H�v �� � L���� �H�A��J�L �J�L �I �� ��� L� � H�p h Hv� � � A0 h 0s�v� ���v� � ����l� ��� ��� Ȁ� H�� ��������� �/k �L)�L���E�H�p4k �CH�E�YZ�&H�EȨL �p � r
                 � H�p4k I��u�H�E ZH��H
                                       ���
����!�*�������    H����3�H�E�� �u(H�H�E�H�E��I;E�� has no method 'indexOf'

[Nicholas O'Leary]

And what is the format of the file you're loading? Is it text? Do you know what encoding has been used?

Can you send me one (directly, off-list) to see what you've got?

Nick

[Josh Conway]

I have it currently on http://crankylinuxuser.net/test.file

[Josh Conway]

I am unexpectedly mobile right now.

The gist is that I waned to parse a JSON file for 4 fields listed. JSON wouldn't work because I'd have to use non compliant HTML to pass the data.

The idea turned into getting the file contents as a bigger and then inserting said buffer in a moustache file at the end.

Its not possible for me to get the flow at the moment, because my NR instance is only accessible via tor.

[Nicholas O'Leary]

For me, a file-in node set to output UTF8 string is able to load that file fine. If I send it straight to debug, it all looks fine to me.

In the example code you provided:

  const buf1 = new Buffer(msg.payload);
  buf1[0] = 255;
  msg.payload = buf1 ;

I assume this Function is immediately following your File In node? What format is the File In node set to output? Why are you setting buf1[0] to 255?

Buffer.indexOf was only added in a recent version of node.js - don't immediately recall if thats node 4.x or 5.x - so I guess that will be why you're hitting that error.

I don't understand your comment about non compliant HTML... you can store anything you want in JSON if you encode it properly. As it is, you've now got a file format you're going to have to manually parse to break out the fields, rather than let JSON do all the heavy lifting.

Nick

On 31 January 2016 at 22:25, Josh Conway <jwcr...@gmail.com> wrote:

I have it currently on http://crankylinuxuser.net/test.file

[Josh Conway]

Egads :/ I feel like a heel.

The problem was that the Node.js that was installed was the ancient .10 version installed by default from ubuntu 14.04 server. For some reason, I thought I had installed the 4.x or 5.x branch, but alas, I did not.

It was your hint Nick, that I checked the version. Now, after updating to a 5.x stack (and verifying to make sure), everything is indeed working as I thought it should :)

Thank you, everyone.

Sincerely,
Josh Conway

[Dave C-J]

Josh,

glad you got it sorted... but please note we are currently only recommending / testing against node v4.2.x - the LTS version.  (just in case you are now too far ahead of the curve ;-)

[Josh Conway]

I think I spoke too soon. And I think I've found a bug with the Template Node. If it's a bug (and not me misusing it), I'll file a ticked if you'd like.

First, sample code:

[{"id":"ec77da4d.e464b8","type":"function","z":"a7cf70f3.0cff98","name":"","func":"msg.payload = \"./img/someimage.jpg\";\nreturn msg;","outputs":1,"noerr":0,"x":266,"y":203,"wires":[["ab50041c.186db","e471922.7f2ff7"]]},{"id":"d3bebcbc.ba23a","type":"inject","z":"a7cf70f3.0cff98","name":"","topic":"","payload":"","payloadType":"str","repeat":"","crontab":"","once":false,"x":110,"y":202.5,"wires":[["ec77da4d.e464b8"]]},{"id":"ab50041c.186db","type":"template","z":"a7cf70f3.0cff98","name":"","field":"payload","fieldType":"msg","format":"text","template":"<img src=\"{{payload}}\">","x":414,"y":202.5,"wires":[["78af7852.0e7138"]]},{"id":"78af7852.0e7138","type":"debug","z":"a7cf70f3.0cff98","name":"","active":true,"console":"false","complete":"false","x":577,"y":203,"wires":[]},{"id":"e471922.7f2ff7","type":"debug","z":"a7cf70f3.0cff98","name":"","active":true,"console":"false","complete":"false","x":367,"y":118,"wires":[]}]


The gist: the text "./img/someimage.jpg" goes in, and ".&#x2F;img&#x2F;someimage.jpg" comes out. It appears to be the Template node mangling data.

Tested on (NR 0.12.5 on Node 0.10.29), (NR 0.13.1 on Node 5.5.0) to the same effect. I can test on more systems if you'd like for me to do so, but I fear the same results.

Sincerely,
Josh Conway

[Josh Conway]

Never mind... Turns out to keep the Template Node from escaping out HTML and similar syntax, you need to use {{{data}}} (triple curly brackets) instead of the usual {{data}} .

And this is a standard and planned feature. I missed it until I searched here and found it : https://groups.google.com/forum/#!searchin/node-red/template$20node/node-red/xR4ojiJbJ6I/mrmCz_-s6kcJ

Sincerely,
Josh Conway