Help Needed with Node Red HTML Node

[tcb...@gmail.com]

OK I have spent nearly four days (in between watching rugby to maintain my sanity)  trying to figure this out.

I am trying to get a Fire Danger Rating for a given location from this site. https://www.emergency.wa.gov.au/index.html#firedangerratings using node-red

I should explain the address is that shown AFTER selecting the appropriate button in the MORE drop down menu ( in this case the relevant button is Fire Danger ratings for the current date).

Anyway I used Google Chrome F12  to see the HTML which looks like this (see the link for the actual and the context) 

<firedangerrating-list>

  div< class="list-title">Fire Danger Ratings Today(37)</div>

    <ul class="item-list firedangerratings-list">

       <li class= item-content-wrapper:...</li>

etc with a  bunch more list items for each location

I am using a node-red http request node followed by an html node and am using the CSS select within that.  I also used F12 and Ctrl F to search for and select "item-list" but when I typed that into the CSS Selector using $(".item-list") it just returns unmatched selector in the debug node.  The only thing that does seem to work is <item-list> in the CSS box which does return an object but not the underlying array.  I do not understand why this works.

I have had a look at the cheerio documentation and being a beginner that is a bit sparse for me.  I have checked out how to write Jquery etc and have tried every combination I could think of and nothing has worked like they say it should. I should also mention that I have managed to get this to work for similar information on a site without drop downs.  

I am thinking this might have something to do with the drop down feature. If so how do I manage that and if not can anybody explain how to get the fire danger rating for a location out of this page using Node-red

Thanks in advance

Tom Barker

[steve rickus]

Hi Tom,

Seems to be a bit of confusion caused by the fusion of CSS and node-red. The selector string is just the portion ".item-list" inside of the Jquery $(...) function. However, since there IS only one element with this class (the <ul> group, not each <li> within that group), you will still not get all 37 locations, if that is what you are trying to extract. Instead, you can try stringing the selectors together, separated by spaces, something like this:

ul.item-list li div.item-content div.item-summary

However, when I try to get the simplest html element from that page, there is NO output from the html node ?!? very strange, and probably a bug.

But I think the underlying problem is that the html page is being generated on the fly by some riot front-end libs and back-end ajax calls. You may be able to find out what the underlying XHR calls are, and reverse engineer the page, but the html parsing approach does not appear that it will work for you -- sorry for the lost 4 days...

--

Steve

[Dave C-J]

I think Steve is right in that the neat RIOT framework they are using is stopping us seeing the underlying data ... 

but as I'm feeling sorry for you guys down under after the match yesterday... :-)

If you inspect the source you can see it mentions ... https://www.emergency.wa.gov.au/app/incident-store.js

which if you then look a the embedded http request in there it points to 'var source = ConstantsService.PATH.dataPath + ConstantsService.DATA.Folder + ConstantsService.DATA.FCADIncidents.fileName;'

which in turn points to a file - https://www.emergency.wa.gov.au/app/constants-service.html which has all the pieces necessary to locate the following url

https://www.emergency.wa.gov.au/data/incident_FCAD.json

which actually then returns nicely formatted json in the first place .... (in fact it looks like GeoJSON)

of course this is fragile in that they could easily move / change their architecture at any time...  a bit like the England Rugby team... but hey while it's working it's great... ;-)

[Dave C-J]

Here is a flow (using the node-red-contrib-web-worldmap )

[{"id":"a1f4f0aa.a68a1","type":"inject","z":"f03b57d5.e525f8","name":"","topic":"","payload":"","payloadType":"date","repeat":"","crontab":"","once":false,"x":120,"y":800,"wires":[["ca79b166.111ca"]]},{"id":"ca79b166.111ca","type":"http request","z":"f03b57d5.e525f8","name":"","method":"GET","ret":"obj","url":"https://www.emergency.wa.gov.au/data/incident_FCAD.json","tls":"","x":290,"y":800,"wires":[["7e1b7314.21c88c"]]},{"id":"7e1b7314.21c88c","type":"function","z":"f03b57d5.e525f8","name":"","func":"msg.payload.features.forEach(function(f) {\n    p = f.properties;\n    p.name = f.properties.DFESUniqueNumber;\n    p.lat = f.geometry.coordinates[1];\n    p.lon = f.geometry.coordinates[0];\n    p.icon = \"fire\";\n    node.send({payload:p});\n});\nreturn null;","outputs":1,"noerr":0,"x":450,"y":800,"wires":[["8091e862.ba1a48"]]},{"id":"8091e862.ba1a48","type":"worldmap","z":"f03b57d5.e525f8","name":"","lat":"-22","lon":"116","zoom":"5","layer":"","cluster":"1","maxage":"","usermenu":"show","layers":"show","panit":"false","x":610,"y":800,"wires":[]}]

[steve rickus]

Dave, so what should the HTML node do is the situation where the page is not parseable as an html document?

You will notice that on this particular site, if you select the "return json parsed data" from the Http Request node, it throws an error saying the results are not parseable. Ok, fine -- not every site is strict in its html compliance... so in this case we get the page source as a string sent into the HTML node. But, there is NO output msg from the HTML node. I didn't try using a catch node, but shouldn't a core node that expects html page source detect when it can't parse the incoming string? Does this look like a bug to you?

--

Steve

[Nick O'Leary]

Steve, 

> You will notice that on this particular site, if you select the "return json parsed data" from the Http Request node, it throws an error saying the results are not parseable. Ok, fine -- not every site is strict in its html compliance

If you select 'return json parsed data' then the node expects the url to respond with parsable JSON. That has nothing to do with 'html compliance'.

>  But, there is NO output msg from the HTML node

How exactly have you configured the HTML node? If it doesn't find the content you've asked for, it doesn't send anything on. That isn't a bug - that's just what the node does (or doesn't do in this case).

It's a reasonable question to ask whether there ought to be an option to pass on an empty payload if nothing matches so that a flow can handle the 'no match' scenario - but that would have to be an option on the node as there may well be flows out there that rely on this behaviour. At the very least we should update the help text to make it clear that it doesn't pass anything on if nothing matches.

Nick

--
http://nodered.org
 
Join us on Slack to continue the conversation: http://nodered.org/slack
---
You received this message because you are subscribed to the Google Groups "Node-RED" group.
To unsubscribe from this group and stop receiving emails from it, send an email to node-red+unsubscribe@googlegroups.com.
To post to this group, send email to node...@googlegroups.com.
Visit this group at https://groups.google.com/group/node-red.
To view this discussion on the web, visit https://groups.google.com/d/msgid/node-red/e38f0f75-05f0-45fe-97cd-b51043beb586%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Dave C-J]

True - in this particular case (https://www.emergency.wa.gov.au/#firedangerratings) - actually easier if we use https://www.emergency.wa.gov.au/textonly.html as an example...

If you look at that page using inspect you can see a div container - full of loads of good info..

But what is fetched by the http request node is the template page prior to it's population by the RIOT server side

<div id='container'>

</div>

is an empty container waiting to be populated... so when you follow it with an html node looking for anything useful you get nothing. Nothing the html node can do about that when there is nothing to parse.

Not sure what the http-request could do to help get round this

[Tom Barker]

Gentlemen

 

Thanks for the comprehensive help.  Way above what I expected.  I was able to use the method suggested by Dave to get the Fire Incident Data at https://www.emergency.wa.gov.au/data/incident_FDR_20112017.json.

 

Steve

 

I did actually try using all the known ways of iterating down the list including that which you suggested.  Being a newbie when they didn’t work I thought it was me! And don’t worry about the four days being lost because they weren’t.  Look at how much I learned!  Also made more progress on the project of desensitizing the wife to my cussing.

 

Cheers and thanks again

--

http://nodered.org
 
Join us on Slack to continue the conversation: http://nodered.org/slack
---

You received this message because you are subscribed to a topic in the Google Groups "Node-RED" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/node-red/Ads5GBVOyu8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to node-red+u...@googlegroups.com.

To post to this group, send email to node...@googlegroups.com.
Visit this group at https://groups.google.com/group/node-red.

To view this discussion on the web, visit https://groups.google.com/d/msgid/node-red/CACXWFw%2BZ3hhYcbQyjdFdmvg1FMkrKdk6j4px3iNeqHbhiD3n3w%40mail.gmail.com.

[Tom Barker]

Dave

 

I had to modify this by adding a layer to get it to work.  As it was I could not see the fire icons.  Am I missing something perhaps?

 

From: node...@googlegroups.com [mailto:node...@googlegroups.com] On Behalf Of Dave C-J
Sent: Monday, 20 November 2017 1:06 AM
To: node...@googlegroups.com
Subject: Re: [node-red] Re: Help Needed with Node Red HTML Node

 

Here is a flow (using the node-red-contrib-web-worldmap )

 

[{"id":"a1f4f0aa.a68a1","type":"inject","z":"f03b57d5.e525f8","name":"","topic":"","payload":"","payloadType":"date","repeat":"","crontab":"","once":false,"x":120,"y":800,"wires":[["ca79b166.111ca"]]},{"id":"ca79b166.111ca","type":"http request","z":"f03b57d5.e525f8","name":"","method":"GET","ret":"obj","url":"https://www.emergency.wa.gov.au/data/incident_FCAD.json","tls":"","x":290,"y":800,"wires":[["7e1b7314.21c88c"]]},{"id":"7e1b7314.21c88c","type":"function","z":"f03b57d5.e525f8","name":"","func":"msg.payload.features.forEach(function(f) {\n    p = f.properties;\n    p.name = f.properties.DFESUniqueNumber;\n    p.lat = f.geometry.coordinates[1];\n    p.lon = f.geometry.coordinates[0];\n    p.icon = \"fire\";\n    node.send({payload:p});\n});\nreturn null;","outputs":1,"noerr":0,"x":450,"y":800,"wires":[["8091e862.ba1a48"]]},{"id":"8091e862.ba1a48","type":"worldmap","z":"f03b57d5.e525f8","name":"","lat":"-22","lon":"116","zoom":"5","layer":"","cluster":"1","maxage":"","usermenu":"show","layers":"show","panit":"false","x":610,"y":800,"wires":[]}]

--

http://nodered.org
 
Join us on Slack to continue the conversation: http://nodered.org/slack
---
You received this message because you are subscribed to a topic in the Google Groups "Node-RED" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/node-red/Ads5GBVOyu8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to node-red+u...@googlegroups.com.
To post to this group, send email to node...@googlegroups.com.
Visit this group at https://groups.google.com/group/node-red.

To view this discussion on the web, visit https://groups.google.com/d/msgid/node-red/CACXWFw%2B_rMBKRQqajzAMS0nQzuoYmn8aas9h1e-tii7roqhOOA%40mail.gmail.com.