Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss

How do I debug WWW::Mechanize problems?

2 views
Skip to first unread message

Dan Kohn

unread,
Dec 2, 2005, 1:50:05 PM12/2/05
to
Hi, I'm trying to scrape my mileage summary from United. Mechanize
starts out correctly, but then gets redirected back to the United
homepage instead of the Mileage Summary page.

Is there an rdoc for mechanize, as gems doesn't install one? The only
examples I could find (besides the one it ships with) were
<http://www.ntecs.de/blog/Blog/WWW-Mechanize.rdoc> and
<http://www.zenspider.com/pipermail/ruby/2005-July/002068.html>. So,
I'm having trouble just figuring out the process to debug these errors.
Is there anything equivalent to the mech-dump that Perl's Mechanize
ships with
<http://search.cpan.org/dist/WWW-Mechanize/lib/WWW/Mechanize/FAQ.pod>?

Anyway, here's the code and the debugging output (run with my username
and password). (If you have a United frequent flyer account, you can
replace username and password with your own to recreate.)

require 'mechanize'
agent = WWW::Mechanize.new{|a| a.log = Logger.new(STDERR) }

page = agent.get('http://www.united.com')
agent.user_agent_alias = 'Windows IE 6'

form = page.forms.find {|f| f.name == 'MileagePlus'}
form.fields.find {|f| f.name == 'userId'}.value = username
form.fields.find {|f| f.name == 'password'}.value = password
puts page.forms
page = agent.submit(form, form.buttons.first)

output = File.open("output.html", "w")
output.puts page.body
puts page.body

>ruby united.rb
I, [2005-12-02T10:02:10.791000 #4816] INFO -- : GET:
http://www.united.com
D, [2005-12-02T10:02:10.961000 #4816] DEBUG -- : request-header: accept
=> */*
D, [2005-12-02T10:02:11.021000 #4816] DEBUG -- : header: connection :
close
D, [2005-12-02T10:02:11.021000 #4816] DEBUG -- : header: content-type :
text/html
D, [2005-12-02T10:02:11.021000 #4816] DEBUG -- : header: date : Fri, 02
Dec 2005 18:02:12 GMT
D, [2005-12-02T10:02:11.021000 #4816] DEBUG -- : header: server :
Netscape-Enterprise/6.0
I, [2005-12-02T10:02:11.512000 #4816] INFO -- : status: 200
I, [2005-12-02T10:02:18.091000 #4816] INFO -- : POST:
https://www.ua2go.com/ci/DoLogin.jsp?stamp=NEWCOOKY*itn/ord=NEWREC,itn/air/united
warning: peer certificate won't be verified in this SSL session
D, [2005-12-02T10:02:19.313000 #4816] DEBUG -- : query:
{"submit2"=>"Login", "sel_return_to"=>"&return_to=ff_acct_hist",
"userId"=>"username", "password"=>"password"}
D, [2005-12-02T10:02:19.313000 #4816] DEBUG -- : request-header: accept
=> */*
D, [2005-12-02T10:02:19.313000 #4816] DEBUG -- : request-header:
content-type => application/x-www-form-urlencoded
D, [2005-12-02T10:02:19.313000 #4816] DEBUG -- : request-header:
user-agent => Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)
D, [2005-12-02T10:02:19.313000 #4816] DEBUG -- : request-header:
referer => http://www.united.com
D, [2005-12-02T10:02:19.313000 #4816] DEBUG -- : request-header:
content-length => 92
D, [2005-12-02T10:02:19.553000 #4816] DEBUG -- : cookie received:
JSESSIONID=DQMtsIaIkMsQbuTwzMK9Nciy064tQh1KBNY1EB1Q3M3ibLa5g7J9!151650267;
path=/
D, [2005-12-02T10:02:19.553000 #4816] DEBUG -- : cookie received:
testcookie=1
D, [2005-12-02T10:02:19.553000 #4816] DEBUG -- : header: connection :
close
D, [2005-12-02T10:02:19.553000 #4816] DEBUG -- : header: content-type :
text/html
D, [2005-12-02T10:02:19.553000 #4816] DEBUG -- : header: date : Fri, 02
Dec 2005 18:02:21 GMT
D, [2005-12-02T10:02:19.553000 #4816] DEBUG -- : header: server :
Netscape-Enterprise/6.0
D, [2005-12-02T10:02:19.553000 #4816] DEBUG -- : header: set-cookie :
JSESSIONID=DQMtsIaIkMsQbuTwzMK9Nciy064tQh1KBNY1EB1Q3M3ibLa5g7J9!151650267;
path=/, testcookie=1
D, [2005-12-02T10:02:19.553000 #4816] DEBUG -- : header: location :
https://www.ua2go.com/ci/DoLogin.jsp?setCookie=1&stamp=NEWCOOKY*itn/ord=NEWREC,itn/air/united
I, [2005-12-02T10:02:19.553000 #4816] INFO -- : status: 302
I, [2005-12-02T10:02:19.553000 #4816] INFO -- : follow redirect to:
https://www.ua2go.com/ci/DoLogin.jsp?setCookie=1&stamp=NEWCOOKY*itn/ord=NEWREC,itn/air/united
I, [2005-12-02T10:02:19.553000 #4816] INFO -- : GET:
https://www.ua2go.com/ci/DoLogin.jsp?setCookie=1&stamp=NEWCOOKY*itn/ord=NEWREC,itn/air/united
warning: peer certificate won't be verified in this SSL session
D, [2005-12-02T10:02:19.824000 #4816] DEBUG -- : use cookie:
JSESSIONID=DQMtsIaIkMsQbuTwzMK9Nciy064tQh1KBNY1EB1Q3M3ibLa5g7J9!151650267;
testcookie=1
D, [2005-12-02T10:02:19.824000 #4816] DEBUG -- : request-header: accept
=> */*
D, [2005-12-02T10:02:19.824000 #4816] DEBUG -- : request-header:
user-agent => Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)
D, [2005-12-02T10:02:19.824000 #4816] DEBUG -- : request-header: cookie
=>
JSESSIONID=DQMtsIaIkMsQbuTwzMK9Nciy064tQh1KBNY1EB1Q3M3ibLa5g7J9!151650267;
testcookie=1
D, [2005-12-02T10:02:19.824000 #4816] DEBUG -- : request-header:
referer =>
https://www.ua2go.com/ci/DoLogin.jsp?stamp=NEWCOOKY*itn/ord=NEWREC,itn/air/united
D, [2005-12-02T10:02:20.174000 #4816] DEBUG -- : cookie received:
rememberMP=-1; domain=.ua2go.com; expires=Friday, 02-Dec-2005 18:02:23
GMT; path=/
D, [2005-12-02T10:02:20.174000 #4816] DEBUG -- : cookie received:
ciToken=792ad6bc2fb42b7d178c34e2f98fab5c6931842a1f415c7d123f896c82db46a0;
domain=.ua2go.com; path=/
D, [2005-12-02T10:02:20.174000 #4816] DEBUG -- : cookie received:
ciToken=792ad6bc2fb42b7d178c34e2f98fab5cf8cf3f5e1ce627d58e2ed9576b828ef1;
domain=.ua2go.com; path=/
D, [2005-12-02T10:02:20.174000 #4816] DEBUG -- : header: connection :
close
D, [2005-12-02T10:02:20.174000 #4816] DEBUG -- : header: content-type :
text/html
D, [2005-12-02T10:02:20.174000 #4816] DEBUG -- : header: date : Fri, 02
Dec 2005 18:02:21 GMT
D, [2005-12-02T10:02:20.224000 #4816] DEBUG -- : header: server :
Netscape-Enterprise/6.0
D, [2005-12-02T10:02:20.224000 #4816] DEBUG -- : header: set-cookie :
rememberMP=-1; domain=.ua2go.com; expires=Friday, 02-Dec-2005 18:02:23
GMT; path=/,
ciToken=792ad6bc2fb42b7d178c34e2f98fab5c6931842a1f415c7d123f896c82db46a0;
domain=.ua2go.com; path=/,
ciToken=792ad6bc2fb42b7d178c34e2f98fab5cf8cf3f5e1ce627d58e2ed9576b828ef1;
domain=.ua2go.com; path=/
D, [2005-12-02T10:02:20.224000 #4816] DEBUG -- : header: location :
http://www.united.com/index/Home?stamp=NEWCOOKY*itn/ord=NEWREC,itn/air/united
I, [2005-12-02T10:02:20.234000 #4816] INFO -- : status: 302
I, [2005-12-02T10:02:20.234000 #4816] INFO -- : follow redirect to:
http://www.united.com/index/Home?stamp=NEWCOOKY*itn/ord=NEWREC,itn/air/united
I, [2005-12-02T10:02:20.234000 #4816] INFO -- : GET:
http://www.united.com/index/Home?stamp=NEWCOOKY*itn/ord=NEWREC,itn/air/united
D, [2005-12-02T10:02:20.325000 #4816] DEBUG -- : use cookie:
JSESSIONID=DQMtsIaIkMsQbuTwzMK9Nciy064tQh1KBNY1EB1Q3M3ibLa5g7J9!151650267;
testcookie=1; rememberMP=-1;
ciToken=792ad6bc2fb42b7d178c34e2f98fab5c6931842a1f415c7d123f896c82db46a0;
ciToken=792ad6bc2fb42b7d178c34e2f98fab5cf8cf3f5e1ce627d58e2ed9576b828ef1
D, [2005-12-02T10:02:20.325000 #4816] DEBUG -- : request-header: accept
=> */*
D, [2005-12-02T10:02:20.325000 #4816] DEBUG -- : request-header:
user-agent => Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)
D, [2005-12-02T10:02:20.325000 #4816] DEBUG -- : request-header: cookie
=>
JSESSIONID=DQMtsIaIkMsQbuTwzMK9Nciy064tQh1KBNY1EB1Q3M3ibLa5g7J9!151650267;
testcookie=1; rememberMP=-1;
ciToken=792ad6bc2fb42b7d178c34e2f98fab5c6931842a1f415c7d123f896c82db46a0;
ciToken=792ad6bc2fb42b7d178c34e2f98fab5cf8cf3f5e1ce627d58e2ed9576b828ef1
D, [2005-12-02T10:02:20.325000 #4816] DEBUG -- : request-header:
referer =>
https://www.ua2go.com/ci/DoLogin.jsp?setCookie=1&stamp=NEWCOOKY*itn/ord=NEWREC,itn/air/united
D, [2005-12-02T10:02:20.435000 #4816] DEBUG -- : cookie received:
united=DQMujVlTtYtjKEjdBoIwk2nLn1vdm82kMn1pSndGtFNa7nIBOEFy!-232476364;
path=/
D, [2005-12-02T10:02:20.435000 #4816] DEBUG -- : cookie received:
sessionid=NEWCOOKY*itn/ord=NEWREC; domain=.united.com; path=/
D, [2005-12-02T10:02:20.435000 #4816] DEBUG -- : header: connection :
close
D, [2005-12-02T10:02:20.435000 #4816] DEBUG -- : header: content-type :
text/html
D, [2005-12-02T10:02:20.435000 #4816] DEBUG -- : header: date : Fri, 02
Dec 2005 18:02:21 GMT
D, [2005-12-02T10:02:20.435000 #4816] DEBUG -- : header: server :
Netscape-Enterprise/6.0
D, [2005-12-02T10:02:20.435000 #4816] DEBUG -- : header: set-cookie :
united=DQMujVlTtYtjKEjdBoIwk2nLn1vdm82kMn1pSndGtFNa7nIBOEFy!-232476364;
path=/, sessionid=NEWCOOKY*itn/ord=NEWREC; domain=.united.com; path=/
D, [2005-12-02T10:02:20.435000 #4816] DEBUG -- : header: content-length
: 0
D, [2005-12-02T10:02:20.435000 #4816] DEBUG -- : header: location :
http://www.united.com/
I, [2005-12-02T10:02:20.435000 #4816] INFO -- : status: 302
I, [2005-12-02T10:02:20.435000 #4816] INFO -- : follow redirect to:
http://www.united.com/
I, [2005-12-02T10:02:20.435000 #4816] INFO -- : GET:
http://www.united.com/
D, [2005-12-02T10:02:20.665000 #4816] DEBUG -- : use cookie:
JSESSIONID=DQMtsIaIkMsQbuTwzMK9Nciy064tQh1KBNY1EB1Q3M3ibLa5g7J9!151650267;
testcookie=1; rememberMP=-1;
ciToken=792ad6bc2fb42b7d178c34e2f98fab5c6931842a1f415c7d123f896c82db46a0;
ciToken=792ad6bc2fb42b7d178c34e2f98fab5cf8cf3f5e1ce627d58e2ed9576b828ef1;
united=DQMujVlTtYtjKEjdBoIwk2nLn1vdm82kMn1pSndGtFNa7nIBOEFy!-232476364;
sessionid=NEWCOOKY*itn/ord=NEWREC
D, [2005-12-02T10:02:20.665000 #4816] DEBUG -- : request-header: accept
=> */*
D, [2005-12-02T10:02:20.665000 #4816] DEBUG -- : request-header:
user-agent => Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)
D, [2005-12-02T10:02:20.665000 #4816] DEBUG -- : request-header: cookie
=>
JSESSIONID=DQMtsIaIkMsQbuTwzMK9Nciy064tQh1KBNY1EB1Q3M3ibLa5g7J9!151650267;
testcookie=1; rememberMP=-1;
ciToken=792ad6bc2fb42b7d178c34e#<WWW::Form:0x34644c0>
#<WWW::Form:0x35fcd78>
#<WWW::Form:0x35d1530>
#<WWW::Form:0x35a71e8>
#<WWW::Form:0x35a3c78>
#<WWW::Form:0x35e8300>

- dan
--
Dan Kohn <mailto:d...@dankohn.com>
<http://www.dankohn.com/> <tel:+1-415-233-1000>

0 new messages