Node red diagnostic and backup / good housekeeping practices

[Csongor Varga]

Hi All,

I am doing something on my Node red on the daily basis. Today I wanted to check a flow and the editor was not loading. Nor the UI. Could not get in touch with the my over SSH.

I had no idea what I can do. I unplugged it and plugged it back in. SSH was still not working, I could not see it on the network.

Unplugged again, moved the setup to the TV to hook it up to the HDMI.

This time it was booting. I stopped OpenHAB service and the Apache service (later is only used for the phpliteadmin). And everything seems to be back to normal.

I had no idea what has happened. I even thought my pi was hacked. I am using self signed certificates, user/pass for editor and admin as well (but not from the start). But I have no idea how to check if anything else is running on the PI which should not.

Do you guys do any particular monitoring of your PI? What about backup of flows, databases?

Regards,

Csongor

[Colin Law]

For backup I make an occasional full image of the card, then use git
to keep track of the flow files themselves with the master repository
on my PC. So if I mess up my flow I can restore to the previous
version with git, and if the card dies then I can restore from the
latest image and update to the latest flow in git.

To make it so that diffs of the flow file work well I have
flowFilePretty: true in settings.js and in addition, in order to make
it easier to see what changes have been made inside function nodes, I
run the command
sed -r 's/\\n/\n/g' flowfile.json > flowfile.json.formatted
which formats the functions into multiple lines, and with that file
also committed to git one can run diffs on it to see what has been
changed before committing.

Colin

> --
> http://nodered.org
>
> Join us on Slack to continue the conversation: http://nodered.org/slack
> ---
> You received this message because you are subscribed to the Google Groups
> "Node-RED" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to node-red+u...@googlegroups.com.
> To post to this group, send email to node...@googlegroups.com.
> Visit this group at https://groups.google.com/group/node-red.
> To view this discussion on the web, visit
> https://groups.google.com/d/msgid/node-red/475741c5-61c0-4d56-9206-450423312207%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[steve rickus]

This is a great tip -- I am constantly using diff tools to keep track of changes in the code for my regular work projects. Now I can do the same for my N-R flows.

But I am wondering why the pretty printing switch does not apply to Function node text (and presumably any other template text)? Perhaps we can request a new feature that supports pretty printing of blocks of text within the flow as well the flow JSON itself. Ideally, it would support more than just the body of a Function node, like JSON text in the Inject nodes as well, and probably others I haven't used.

I realize there could be lots of edge cases (e.g. JSON blocks of text in an Inject node, Template nodes which can show multiple language syntax highlighting, UI Templates for chunks of HTML/RSS/SVG, etc, etc). But if the ACE editor can correctly display all of these "strings" of text in the editor window, perhaps one of those methods can be called on each chunk of text BEFORE the flow pretty print method is invoked? I'm just thinking out loud here -- as always, i'm interested in whether this community thinks this would be possible.

--

Steve

[Nicholas O'Leary]

Steve,

JSON does not support newlines within a string - they must be encoded as '\n'. 

If we 'pretty printed' those blocks as described, the content would no longer be valid JSON. You wouldn't be able to use standard JSON tools for validation, you wouldn't be able to import the flow via copy/paste etc etc.

Nick

--
http://nodered.org
 
Join us on Slack to continue the conversation: http://nodered.org/slack
---
You received this message because you are subscribed to the Google Groups "Node-RED" group.

To unsubscribe from this group and stop receiving emails from it, send an email to node-red+unsubscribe@googlegroups.com.

To post to this group, send email to node...@googlegroups.com.
Visit this group at https://groups.google.com/group/node-red.

To view this discussion on the web, visit https://groups.google.com/d/msgid/node-red/0a2c8950-255b-439d-a3d0-a44edb4eb4a2%40googlegroups.com.

[steve rickus]

Ah, of course -- so the conversion of \n to new lines has to take place during the comparison/display of the JSON flow. Makes sense, but I don't think I have any tools that support that by default. I guess that's why Colin chose to keep a formatted version of the file next to the original...

[steve rickus]

... then again, what about supporting valid JSON blobs, by outputting them as embedded JSON objects? I agree that body of the Function node needs to remain in string format, but if (for instance) my Inject node payloadType = JSON, that wouldn't have to be stringified, right?

If this is true, then in the "embedded object" version of my data-generator node, the template field would look like this (instead of a string):

How many other nodes (core or otherwise) hold text that represents a JSON object? Is there a generic way to identify those, so they can be handled differently during export and import? Inquiring minds want to know...

[Nicholas O'Leary]

There is no standard meta data that would help the runtime determine what can and cannot be encoded like this.

Nick

--
http://nodered.org
 
Join us on Slack to continue the conversation: http://nodered.org/slack
---
You received this message because you are subscribed to the Google Groups "Node-RED" group.
To unsubscribe from this group and stop receiving emails from it, send an email to node-red+unsubscribe@googlegroups.com.
To post to this group, send email to node...@googlegroups.com.
Visit this group at https://groups.google.com/group/node-red.

To view this discussion on the web, visit https://groups.google.com/d/msgid/node-red/dbe6ca83-4be6-4f04-815e-ccc85908a157%40googlegroups.com.

[Csongor Varga]

Hi Colin,

Thanks for the response. Just to make sure I am getting this right. We are talking about the flows_xxxx.json file in .node-red folder, right? And I am guessing settings.js is also quite important.

I am not too familiar with git other than how to download packages. You can commit to git from command line? And you have a cron job to do all this?

Regards,

Csongor

[Colin Law]

On 5 December 2016 at 20:17, Csongor Varga <csongo...@gmail.com> wrote:
> Hi Colin,
>
> Thanks for the response. Just to make sure I am getting this right. We are
> talking about the flows_xxxx.json file in .node-red folder, right? And I am
> guessing settings.js is also quite important.

I commit all files in the node-red folder except *.backup

>
> I am not too familiar with git other than how to download packages. You can
> commit to git from command line? And you have a cron job to do all this?

Git is a command line utility, though there are some excellent gui
addons, in particular git-gui and gitk. I don't commit automatically,
just when I feel it appropriate. So I run the sed script, check the
diffs (using git diff or git gui if in gui environment) to make sure
that what has changed looks right and then commit it. There are some
great advantages when using git. For example you have been doing some
restructuring of the flow but have not quite got it fully working, and
time has run out. Simple run
git stash
and git will save your changes and restore the flows to the previous
committed state (which presumably was a working system). Restart node
red to get it to pick up the changes and you are back with a working
system. Then when you are able to continue simply
git stash apply
and git will restore your work at the point you stashed it.

Once you get the hang of how to use git you will wonder how you
managed without it.

Cheers.

Colin

[steve rickus]

For the record, it appears that JSON does support line and paragraph separators, specifically for passing multiline/multipara fields through JSPONP, while remaining a valid JS string that can be parsed by the browser:

http://timelessrepo.com/json-isnt-a-javascript-subset

Just an idea -- maybe all JSON fields in "pretty formatted" flow files can be serialized with the "\u2028" escape sequence instead of "\n"

I'll experiment with some of my flows and see what else is affected. Even if it's not practical (say maybe the ACE editor cannot handle it), I just found this to be an interesting trip to a little cul-de-sac off of the internet highway...

On Monday, December 5, 2016 at 9:22:10 AM UTC-5, Nick O'Leary wrote:

[steve rickus]

Oh, and here is the link where I found this information:

https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/JSON/stringify#Issue_with_plain_JSON.stringify_for_use_as_JavaScript

The page also gives a code example for extending JSON.stringify() to handle these special characters.

[Csongor Varga]

After this long discussion I decided to just back up my files to an FTP server. I am using this shell command and an Exec node to trigger from node red:

find /home/pi/ -type f -exec curl -u <user>:<password> --ftp-create-dirs -T {} ftp://192.168.1.90/backup/{} \;

This is an obviously slow process, causes the PI to open a connection, log in and upload every single file in the home/pi/ directory which has loads of stuff.

Can you recommend a better way of doing this? I would still like to trigger this from node red, so any python, shell script would be ideal. I tried googeling it but could not find a better way that I could actually make work. I wanted to use FTP in shell script, but could not figure out what I need to install to make it work. Sorry, I am still quite noob to linux.

Regards,

Csongor

[Sebastian Barwe]

you have so many options for your backup target:

1. ftp as you mentioned, or you can use rsync which has some advanced options (filecopy over ssh with differentials etc.)

2. windows smb (network folders)

3. just an usb mass storage attached to the Pi (you can use rsync as well to speed up the process, as it will just sync changed files)

https://wiki.ubuntuusers.de/rsync/

[Colin Law]

Am I reading it correctly in that if you accidentally corrupt a file
(perhaps copy one file over another for example) and then run the
backup, that you will overwrite your backup with the corrupted
version? If so then that seems not to be a good idea to me.

I suggest using a source control system for the flow files (git for
example) and then a proper backup method for off-site (or at least
off-machine) routine backups. I find backintime excellent for this on
my PC. On my Pis I make an image of the SD card occasionally, and
since the git repository is on my PC not the card then I can restore
the latest card image and then update the flow with git to get it back
to within at most an hour or two of work from where the problem arose.

Git may seem a bit daunting at first, but once you get the hang of the
three or four commands used regularly you will wonder how you got on
without it.

Remember a disc, SD card or machine can go up in smoke (literally or
metaphorically) at any time.

Colin

> --
> http://nodered.org
>
> Join us on Slack to continue the conversation: http://nodered.org/slack
> ---
> You received this message because you are subscribed to the Google Groups
> "Node-RED" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to node-red+u...@googlegroups.com.

> To post to this group, send email to node...@googlegroups.com.
> Visit this group at https://groups.google.com/group/node-red.
> To view this discussion on the web, visit

> https://groups.google.com/d/msgid/node-red/561377de-af51-4ea3-84bd-71ee2a2b52ba%40googlegroups.com.

[Csongor Varga]

Hi Colin,

Yes, if I corrupt some files, I overwrite the backup with the corrupted version. But I did not have more time to implement any other method for the time being. 

So far I just want to fix the fact that the current method runs for ours with all the small files. So I try to implement differential backup first and probably go over to git next.

Btw, the FTP is on a NAS with Raid 1 driver, so that offers more protection than the SD card.

Sebastian: thanks for the tip, I will check out rsync.

Csongor

[Colin Law]

Just to clarify, when I said make an image backup of the card, I did
not mean to another card, I meant to the PC, which is then itself
backed up to an off-site server, using backintime.

Colin

> https://groups.google.com/d/msgid/node-red/e9817a49-0bb7-491d-a0c4-d41c61d13c88%40googlegroups.com.

[Julian Knight]

If you hunt around, you will find a number of Linux backup tools/scripts that use rsync. Some of them also do multi-version backups.