Downloading DOM from password protected website

[Julie Zhang]

Hi all,

I was wondering on how you can use the DOM parser from pattern for websites that you need to login before gaining access. I do have a username and password for the site, but I don't know how to pass that information through the URL or DOM call (url.username and url.password didn't work).

Any tips?

Thanks,

J

[Tom De Smedt]

Hi Julie,

If it is basic HTTP authentication, you can pass the arguments to URL.download():

from pattern.web import URL

print URL("http://...").download(authentication=(username, password))

If you need to fill in an actual login form it's more complex, since it may or may not involve cookies etc. You can try to locate the login request in the source HTML and send it by post-method:

from pattern.web import URL, POST

URL("http://domain.com?login", query={"username_fieldname": username, "password_fieldname": password}, method=POST).open()

But the chance that this will actually work is quite small I think..

I haven't looked at this kind of situation yet. Let me know how it goes so we can learn from it.

Best,

Tom

--
 
---
You received this message because you are subscribed to the Google Groups "Pattern" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pattern-for-pyt...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.