Simper

unread,

Feb 26, 2016, 4:16:35 AM2/26/16

to Node-RED

Hi,

I have a problem with a shell I created and that I'm calling from an exec node.
The script (forgive me for the shell but it is the first time I try to create one) tries to download a list of gas stations and their prices and create a list of those most economical in towns around me.
It works correctly if run it from linux prompt but from the exec node I'm getting this error:
/home/lele/prezzi_carb/Prezzi_carb.sh: Syntax error: "(" unexpected

Does someone have an idea about the reason?

Thanks for your help.

Prezzi_carb.sh

Zenofmud

unread,

Feb 26, 2016, 4:51:18 AM2/26/16

to node...@googlegroups.com

Can you also provide the code from the function that calls this.

> On Feb 26, 2016, at 4:16 AM, Simper <er_s...@yahoo.it> wrote:
>
> <Prezzi_carb.sh>

Simper

unread,

Feb 26, 2016, 6:37:39 AM2/26/16

to Node-RED

This is the node:

Auto Generated Inline Image 1

Giovanni Angoli

unread,

Feb 26, 2016, 6:49:49 AM2/26/16

to node...@googlegroups.com

I've downloaded and tried the script both in the command line (on a raspberry pi) and via Node-Red.

on the command line on the Pi it works fine only if i run it like this bash /path/to/script.sh

it doesn't work if called directly (even after changing the shell used inside the script)

I didn't investigate further (it has something to do with the declaration of the array but that's the whole point)

From a NodeRed point, it works fine for me if I tick "use spawn() instead of exec()" ...

I'm puzzled too, but it works.

On Fri, Feb 26, 2016 at 12:37 PM, Simper <er_s...@yahoo.it> wrote:

This is the node:

--
http://nodered.org
---
You received this message because you are subscribed to the Google Groups "Node-RED" group.
To unsubscribe from this group and stop receiving emails from it, send an email to node-red+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Giovanni Angoli

Simper

unread,

Feb 26, 2016, 10:56:32 AM2/26/16

to Node-RED

I tried to simplify and it seems related to array assignment. I created a new shell script with:

#! /bin/bash
declare -a Unix=('Debian' 'Red hat' 'Red hat' 'Suse' 'Fedora');

and I have same error from Node red.

Mark Setrem

unread,

Feb 26, 2016, 11:01:33 AM2/26/16

to Node-RED

What version of nodered are you running?
With what version of nodejs?
What is your entire flow?

Mark Setrem

unread,

Feb 26, 2016, 11:30:45 AM2/26/16

to Node-RED

If I run the following script

[{"id":"66ced73a.993128","type":"inject","z":"ec4d1ad9.13b2e8","name":"","topic":"","payload":"","payloadType":"date","repeat":"","crontab":"","once":false,"x":103.5,"y":533,"wires":[["7c57572c.83a8a8"]]},{"id":"7c57572c.83a8a8","type":"exec","z":"ec4d1ad9.13b2e8","command":"/home/pi/testshellscript.sh","addpay":false,"append":"","useSpawn":"","name":"","x":285.5,"y":577.5,"wires":[["78a0c3e1.875f3c"],["78a0c3e1.875f3c"],["78a0c3e1.875f3c"]]},{"id":"78a0c3e1.875f3c","type":"debug","z":"ec4d1ad9.13b2e8","name":"","active":true,"console":"false","complete":"false","x":381.5,"y":702,"wires":[]}]

where testshellscript.sh is

#! /bin/bash
declare -a Unix=('Debian' 'Red hat' 'Red hat' 'Suse' 'Fedora');

echo Unix

and the permissions are set to make it an executable

I get the result I would expect I'm running
Node-RED version: v0.13.2
Node.js version: v4.3.1

Simper

unread,

Jun 9, 2016, 1:16:02 PM6/9/16

to Node-RED

Hi,

It tooks some time but I did a test :)

I'm running:

Node-RED version: v0.13.4

Node.js version: v0.10.29

Linux 4.1.17+ arm LE

Using your flow and your shell script I get always the same error: /home/pi/testshellscript.sh: 3: /home/pi/testshellscript.sh: Syntax error: "(" unexpected

I tried to upgrade node.js to latest stable and now I have:

Node-RED version: v0.13.4

Node.js version: v6.2.1

Linux 4.1.17+ arm LE

But unfortunately same result :(

Mark Setrem

unread,

Jun 9, 2016, 1:48:44 PM6/9/16

to Node-RED

Running Node-red on nodejs v6 isn't recommended yet.

The recommended version of nodejs is the LTS version of node v4, see http://nodered.org/docs/getting-started/installation

But if we forget about node-red. Can you successfully run the script from the command line?

Simper

unread,

Jun 9, 2016, 2:06:07 PM6/9/16

to Node-RED

Hi,

thanks for your replay. Yes from command line I run it without problem.

I can try again with v4.3.1 as you specified some messages before, and see.

Simper

unread,

Jun 9, 2016, 2:30:37 PM6/9/16

to Node-RED

With node.js 4.4.5 same output.

I removed declare -a Unix=('Debian' 'Red hat' 'Red hat' 'Suse' 'Fedora'); from script and I see Unix on NodeRed debug...

Julian Knight

unread,

Jun 10, 2016, 3:51:08 AM6/10/16

to Node-RED

I didn't even know that BASH understood arrays!!

However, a quick read seems to indicate that you can't just do an "echo Unix" but rather have to do:

echo ${Unix[@]}

Simper

unread,

Jun 10, 2016, 4:17:45 AM6/10/16

to Node-RED

Hi,

Thanks for replying.

In this example:

#! /bin/bash

declare -a Unix=('Debian' 'Red hat' 'Red hat' 'Suse' 'Fedora');

echo Unix

it is declaring an array but then is just an echo of a simple string. This was just for testing the array presence in a shell called by a function node in node-red.
The shell scipt itself works correctly if executed by command prompt.

Dave C-J

unread,

Jun 10, 2016, 5:12:00 AM6/10/16

to node...@googlegroups.com

this is odd - using that simple script seems to work fine for me... (on Mac)

Giovanni Angoli

unread,

Jun 10, 2016, 5:30:46 AM6/10/16

to node...@googlegroups.com

@simper

maybe my original reply was lost, i did some test when you first reported this issue and it works for me but I had to tick "use spawn() instead of exec()"

I did the tests with a wheezy install and a relatively old node.js but an up to date node-red installa at the time.

cheers.

On Fri, Jun 10, 2016 at 11:11 AM, Dave C-J <dce...@gmail.com> wrote:

this is odd - using that simple script seems to work fine for me... (on Mac)

--
http://nodered.org

Join us on Slack to continue the conversation: http://nodered.org/slack

---
You received this message because you are subscribed to the Google Groups "Node-RED" group.
To unsubscribe from this group and stop receiving emails from it, send an email to node-red+u...@googlegroups.com.

To post to this group, send email to node...@googlegroups.com.
Visit this group at https://groups.google.com/group/node-red.

For more options, visit https://groups.google.com/d/optout.

--

Giovanni Angoli

Simper

unread,

Jun 10, 2016, 6:19:33 AM6/10/16

to Node-RED

Ciao Giovanni,

I saw your comment and I tried but I got always same problem. Since it is seems something replicable on my environment only I'll try to find a workaround.

Dave C-J

unread,

Jun 10, 2016, 6:25:59 AM6/10/16

to node...@googlegroups.com

I guess there are "other" shells available also... sh, ksh, dash etc ...

Giovanni Angoli

unread,

Jun 10, 2016, 6:34:45 AM6/10/16

to node...@googlegroups.com

@Dave, Simper

I dit try the script at least with two different versions of bash, and my default shell (zsh). since some months passed i can only say what I've written in my last mail:

on the command line on the Pi it works fine only if i run it like this bash /path/to/script.sh

it doesn't work if called directly (even after changing the shell used inside the script)

So I'm quite lost at the moment.. :(

On Fri, Jun 10, 2016 at 12:25 PM, Dave C-J <dce...@gmail.com> wrote:

I guess there are "other" shells available also... sh, ksh, dash etc ...

--

http://nodered.org

Join us on Slack to continue the conversation: http://nodered.org/slack
---
You received this message because you are subscribed to the Google Groups "Node-RED" group.
To unsubscribe from this group and stop receiving emails from it, send an email to node-red+u...@googlegroups.com.
To post to this group, send email to node...@googlegroups.com.
Visit this group at https://groups.google.com/group/node-red.
For more options, visit https://groups.google.com/d/optout.

--

Giovanni Angoli

Julian Knight

unread,

Jun 10, 2016, 9:13:55 AM6/10/16

to Node-RED

OK, how about going back to basics.

You've used a BASH script to do something that could be done purely with NR I think. If I've understood it right:

Create a manual list of stations
Grab a couple of CSV files from a website
Extract the data for the stations in 1
... some other stuff that my Italian is letting me down on ;) ...

Is that correct?

If so, this is much better done in NR.

Trigger node to kick things off
http request node to get the file
csv node to convert to a JavaScript object
...

The trick is combining the two datasets. That is actually easiest done in a SQL database so if you already have a db server running, dig out and install the right node for it. Otherwise, use npm install node-red-node-sqlite

The installation is slow because it has to compile the libraries for sqlite3.

You can, of course, do the data merge purely in JS but it is a pain to work out and SQL is designed for that kind of thing. If you REALLY insist on BASH, you can use sqlite from it directly.

Simper

unread,

Jun 10, 2016, 9:36:01 AM6/10/16

to Node-RED

Yes, you understood right what I'm trying to achieve and proposed me a challenging suggestion, thanks for that ;)
I'll try to do everything from NR and SQL.

Thanks!

Dave C-J

unread,

Jun 10, 2016, 11:17:34 AM6/10/16

to node...@googlegroups.com

Actually looking at those files... they aren't sooo big that you couldn't just hold in memory objects.... :-)

Dave C-J

unread,

Jun 10, 2016, 11:18:33 AM6/10/16

to node...@googlegroups.com

Couldn't resist...

[{"id":"a815c116.0fdb5","type":"inject","z":"ff08c5aa.63eb38","name":"","topic":"","payload":"","payloadType":"date","repeat":"","crontab":"","once":false,"x":107.5,"y":60,"wires":[["dc6bb467.d0a578"]]},{"id":"dc6bb467.d0a578","type":"http request","z":"ff08c5aa.63eb38","name":"","method":"GET","ret":"txt","url":"http://www.sviluppoeconomico.gov.it/images/exportCSV/anagrafica_impianti_attivi.csv","tls":"","x":169.5,"y":120,"wires":[["a3f180bc.0e2bb"]]},{"id":"b33442ff.ba0ac","type":"csv","z":"ff08c5aa.63eb38","name":"","sep":";","hdrin":true,"hdrout":"","multi":"one","ret":"\\n","temp":"","x":362.5,"y":246,"wires":[["333e5cca.860044"]]},{"id":"a3f180bc.0e2bb","type":"function","z":"ff08c5aa.63eb38","name":"remove first line","func":"msg.payload = msg.payload.substring(msg.payload.indexOf(\"\\n\") + 1);\nreturn msg;","outputs":1,"noerr":0,"x":313.5,"y":181,"wires":[["b33442ff.ba0ac"]]}]

Dave C-J

unread,

Jun 10, 2016, 11:21:33 AM6/10/16

to node...@googlegroups.com

Dave C-J

unread,

Jun 10, 2016, 12:17:35 PM6/10/16

to node...@googlegroups.com

Dave C-J

unread,

Jun 10, 2016, 5:07:03 PM6/10/16

to node...@googlegroups.com

complete flow from above - needs node-red-contrib-web-worldmap

Julian Knight

unread,

Jun 10, 2016, 5:52:14 PM6/10/16

to Node-RED

Urrgggghhhh! Well, that's a way to do it.

You need a new hobby Dave!

Sadly though, it doesn't seem to work for me and it all but kills the Pi! It also seems to all but kill my i7 Surface Pro 4 as well! At least Chrome goes into serious overdrive - actually, I think it is the NR admin interface that is causing that.

I also get loads of buffer exceeded messages but no output on the map.

Anyway, I am going to sign-off for tonight because I just know I'm about to try and pick all that apart and I really need some sleep!

Dave C-J

unread,

Jun 10, 2016, 6:26:41 PM6/10/16

to node...@googlegroups.com

yeah its pushing 13800 or so messages to the map that is killing it... the actual fetch and join isn't to bad.. you can turn down the rate in the delay node and it should then work. The >1000 messages are just warnings... (and I've just pushed a fix to calm that down to 0.14 ;-)

Simper

unread,

Jun 10, 2016, 6:49:03 PM6/10/16

to Node-RED

Hi guys,

Wow I truly appreciate your help...

Dave your flow is a lesson that I have to study but unfortunately, as Julian, I tried your flow on a Pi and after 12' of elaboration I get same result.

Since I would like to get the 10 most convenient stations in an area ordered by date of upload (They should do it on a daily base but it is not always true) I think an sql could be more easy for me to implement but I'm open to suggestion.

Dave C-J

unread,

Jun 10, 2016, 7:03:46 PM6/10/16

to node...@googlegroups.com

Or add in node-red-node-geofence to the flow before sending to map 😀

Simper

unread,

Jun 10, 2016, 7:05:53 PM6/10/16

to Node-RED

you are the man!

Dave C-J

unread,

Jun 11, 2016, 3:39:39 AM6/11/16

to node...@googlegroups.com

yeah the geofence works rather well doesn't it :-) no need for the rate limit any more.

Of course if this is the case then there are other optimisations that could be done. Like geofencing just after the fetch of the list of stations and only creating an object with the interesting ones in... then getting the price and only updating the few stations needed.

And of course you should only need to get the stations once as they won't change too much etc...

And yes of course you can use a database and sql and display in some other format and ... will leave all that to you :-) love to see what you come up with.

Walter Kraembring

unread,

Jun 11, 2016, 2:14:38 PM6/11/16

to Node-RED

1) How is the flow above supposed to work? Should I click on both the trigger-node buttons?

2) Where in the flow is the geofence best positioned?

BR Walter

Dave C-J

unread,

Jun 11, 2016, 2:53:40 PM6/11/16

to node...@googlegroups.com

load and deploy the flow, Load the map in another tab/window - zoom out and pan to Italy.

In flow window - click the top inject... wait while it fetches data etc... should then appear on map automatically.

If you reload map and need to refresh you can then hit second inject just to display already fetched data.

A version with the geofence set around Toscana

Walter Kraembring

unread,

Jun 13, 2016, 12:05:41 PM6/13/16

to Node-RED

Dear Dave,

This was fun! I am modifying the flow a bit to split & save the results for easier re-use and to make it somehow working on a RPi. So far the map is updated correctly when running it chunk-by-chunk but of course it takes some time to run. What I have done so far:

- separately creating the station locations object already with geofence applied (to reduce size of object)

- separately creating the gas price array

- saving results to local files on the SD card

In my opinion, what actually takes most time is inserting prices into stations:

var p = context.global.stations;

if (p.hasOwnProperty(msg.payload.idImpianto)) {

    
    var s = p[msg.payload.idImpianto];

var t = msg.payload.descCarburante;

    s[t] = {prezzo:msg.payload.prezzo, isSelf:msg.payload.isSelf, dtComu:msg.payload.dtComu};

    context.global.stations[msg.payload.idImpianto] = s;
}
else {
    //console.log(msg.payload.idImpianto);
}
return {payload:1};

So just for the intellectual challenge, I am trying to figure out how this eventually could be made more efficient...

BR Walter

Julian Knight

unread,

Jun 13, 2016, 12:14:45 PM6/13/16

to Node-RED

Without wishing to sound like a broken record, that is what SQL is for. You probably want to find a way to "diff" the prices file so that you only end up with changed records and then update your main table with those. If course, if your tables and indexes end up larger than the available memory, you will always have a slower response. The RPi only has 1GB RAM (a limitation of the ARM chipsets) so memory is always going to be tight.

Another thing you could do to reduce the data sizes to manageable chunks would be to split the data into regions.

Dave C-J

unread,

Jun 13, 2016, 5:25:21 PM6/13/16

to Node-RED

Yes SQL is great - but 18000 records is not exactly massive (even at say 1k per record ..) - Doing the diff to find the ones to insert should be no worse than doing the same to update the in memory object... and ought to be faster. But yes that code was in no way optimised. All good fun.

Julian Knight

unread,

Jun 14, 2016, 6:10:58 AM6/14/16

to Node-RED

I think that it is more about knowledge and ease of reasoning than raw performance.

Though the performance may be different due to the way the tools are coded - this used to be much bigger of course but the excellent work done on optimising the V8 engine has certainly closed or maybe even eliminated such gaps?

Reply all

Reply to author

Forward

Syntax error with exec node but

Simper

Zenofmud

Simper

Giovanni Angoli

Simper

Mark Setrem

Mark Setrem

Simper

Mark Setrem

Simper

Simper

Julian Knight

Simper

Thanks for replying.

Dave C-J

Giovanni Angoli

Simper

Dave C-J

Giovanni Angoli

Julian Knight

Simper

Dave C-J

Dave C-J

Dave C-J

Dave C-J

Dave C-J

Julian Knight

Dave C-J

Simper

Dave C-J

Simper

Dave C-J

Walter Kraembring

Dave C-J

Walter Kraembring

Julian Knight

Dave C-J

Julian Knight