HTTP, client cookie and client form handling

[Yannick Turgeon]

Hello,

I've posted this message on february 24th. No reply. I try another time...

-----------------
Hello all,

I've got a script written in Perl that I have to modify. I'm planning to
completely rewrite it using Ruby. The three main reasons to do this are:
1- My script has to be modified.
2- Perl is too "unclear" for me to find it pleasant to work with.
3- I want to give a try programming with Ruby. I've been charmed recently
with Python and now looking at Ruby doc it seems to be a pretty interesting
language.

What my script is doing:
1- It logs in Yahoo website using my user and password
2- It tooks some webpages there and do some process to extract information.
3- It updates a database with this information.

While looking at Ruby doc, I feel at ease to jump in with almost all the
work to be done except the Yahoo login and session management. Perl has a
great module to do this kind of work (LWP::UserAgent) but I suppose Ruby too
can acheive this. How? Is there any module more specific than Net::HTTP or
Net::HTTPS? If not, how am I suppose to use it? I mean, I get Yahoo login
page, extract all hidden parameter and "post" it with my username and
password, then... I don't understand the big picture to how should I manage
sessions in this case. Are cookies used?

How Yahoo knows if I'm logged or not and what do I have to do to manage
this? Any help or suggestion is welcome.

Thanks

Yannick
----------------------

I recently found that someone has developped two Python modules:
ClientCookie (now included in the standard library) and ClientForm. Maybe
you know, this is exacly what I need. But I would prefer to use Ruby.

Thanks again.

Yannick

[Aleksi]

Yannick Turgeon wrote:
> Hello,
>
> I've posted this message on february 24th. No reply. I try another time...

I don't know why it wasn't answered, as it's all polite and so...

> What my script is doing:
> 1- It logs in Yahoo website using my user and password
> 2- It tooks some webpages there and do some process to extract information.
> 3- It updates a database with this information.
>
> While looking at Ruby doc, I feel at ease to jump in with almost all the
> work to be done except the Yahoo login and session management. Perl has a
> great module to do this kind of work (LWP::UserAgent) but I suppose Ruby too
> can acheive this. How? Is there any module more specific than Net::HTTP or
> Net::HTTPS? If not, how am I suppose to use it? I mean, I get Yahoo login
> page, extract all hidden parameter and "post" it with my username and
> password, then... I don't understand the big picture to how should I manage
> sessions in this case. Are cookies used?

Session handling is probably no different from your Perl way of doing
it. If that used cookies, by all means use those with Ruby as well.

Net::HTTP is quite versatile, handles proxies for you and so on.

Of these libraries, http://ruby.yi.org/raa/en/library.html#WWW, you
might probably want to check

http://www.acc.umu.se/~r2d2/programming/ruby/webfetcher/

for it's very nice API, although it can't do everything.

http://sourceforge.net/project/showfiles.php?group_id=65749

as it works with Yahoo to some extent.

Another one which might be of interest is Michael's Mechanize with
another nice API.

http://www.ntecs.de/blog/Blog/WWW-Mechanize.rdoc

- Aleksi

[Yannick Turgeon]

Hello Aleksi,

Thank you very much for your answer. Mechanize did the trick.

Yannick