Getting content from password protected webpage

[Jokkemokke]

Hi.

I want to use Node-Red to get the content from a webpage and then send the content to a MQTT node. However the webpage is a https page were the content is protected by a login.

The login form is presented as a html form in the webpage, where I have to enter username and password. The html form uses the post method:

<form action="checklogin.asp" method="post" name="login">
Username:
<input type="text" name="username">
Password:
<input type="password" name="password">
<input type="submit" value="Login">
</form> 

Is there somehow I can make Node-Red submit my username and password in this form, so that it can get access to the protected content on this webpage?

Thanks

[Julian Knight]

You probably don't need to try and handle the form as such but rather send a POST direct from NR with the appropriate body content. It is possible however that the form might do some back-end checks to see if the data actually came from the form.

I'm afraid my brain has gone blank regarding how to format the POST. Hopefully some googling will turn up a generic answer which you can adapt.

[Greg EVA]

I would tend to agree with Julian.  You should just format the post body appropriately, and this is part of the HTTP protocol, not Node-RED, so you should be able to find examples of people doing this with other languages by Googling a bit.

[Nicholas O'Leary]

I suspect the hard part will be what to do after submitting the post.

The normal way this sort of thing works is that the POST request, if successful, sets some session cookies so that subsequent requests are recognised as coming from an authorised user. So even if you get a flow that successfully POSTs the user/password, it would then need to capture the cookie responses and then ensure they are included in any subsequent request to the page. It is doable, just quite fiddly to get right.

Nick

On 28 October 2015 at 09:44, Greg EVA <ge...@ge-volution.eu> wrote:

I would tend to agree with Julian.  You should just format the post body appropriately, and this is part of the HTTP protocol, not Node-RED, so you should be able to find examples of people doing this with other languages by Googling a bit.

--
http://nodered.org
---
You received this message because you are subscribed to the Google Groups "Node-RED" group.
To unsubscribe from this group and stop receiving emails from it, send an email to node-red+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Julian Knight]

Agreed. I expect there is a NodeJS library somewhere that would pick up the heavy stuff though. If so, you could add that to the NR settings so it is available to all script nodes.

However, I actually think this would be a good case for creating a separate NodeJS micro-service that just polls that web site as needed and then passes the output direct to MQTT. Sometimes keeping things separate is easier. This would be easier than using NR I think as long as you are only providing one set of credentials to the page. If you need different users to pass credentials then it might be different.

[Edward Vielmetti]

The alternative approach is to script this with an "exec" command, and then to use something like "curl" to do the tricky work of handlng forms submission and cookie management.

This tutorial is pretty reasonable

http://curl.haxx.se/docs/httpscripting.html

for approaching the problem from the command line. Once you have it all wrapped up you can stick it into a shell script and use exec to execute it.

thanks

Ed

--

http://nodered.org
---
You received this message because you are subscribed to the Google Groups "Node-RED" group.
To unsubscribe from this group and stop receiving emails from it, send an email to node-red+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Edward Vielmetti +1 734 330 2465

edward.v...@gmail.com