Browser session dump

[Theo Markettos]

I want to record all my interactions with a website - in other words what I
send it, what is given back to me, rinse and repeat. The difficulty is the
website uses SSL, Javascript and Flash to do the interaction. So basic
tcpdump won't work due to the SSL. If I spoof the SSL certificate, the
Flash will probably fall over. I can't use a perl script because of the
javascript and Flash that need to run. It's extremely fragile and has a
tendency to fail at a whim, which means that any measuring thing may well
make it keel over.

So is there any man-in-the-browser thing I can use to record a session
without interfering with what the browser does? Or is there some way I can
make a browser divulge the SSL keys so that I can decrypt tcpdump traces?
Don't really mind what browser it is (though the pile of JS/Flash mess is
probably also fussy about browser too).

Any ideas?

Thanks
Theo

[Jim Price]

It's a lot like hard work, but once you have the session cookie, you
should be able to download each page with wget. What you won't get with
that approach is any object subrequests etc., which are dynamically
loaded - you'll only get stuff which loads as a new page. It might be
possible to see the results of any on-page javascript if you open the
file with a browser, but it's quite possible anything clever won't work
properly. It really depends on how the site works as to what you would
be able to do with it. Adblock Plus is useful for digging into the site
a bit to see if there are any subrequests doing the heavy lifting. I'd
speed the thing up a bit by installing the Launcher 1.1 Firefox
extension, so I could go round the loop of wget-ing a link and opening
it in a new browser tab from the right click menu. You'll need to set
all that up though, and if the links you click on won't open in new
tabs, forget that whole approach. Before abandoning it completely
though, do check the links shown via Adblock Plus, as they can expose
things which work inside Flash containers. Set it up right and you
should end up with a series of files which are a record of what you did
on that web site, and which should open in a browser, but maybe not work
entirely depending on the content. Well, you asked for ideas...

--
╔═╦═╦═════╦═══╗
║ ║ ║ ║ ║
╔═╝ ║ ║ ║ ║ ║ ╔═╝
╚═══╩═╩═╩═╩═╩═╝ -- JimP.

[Chris Davies]

Theo Markettos <theom...@chiark.greenend.org.uk> wrote:
> I want to record all my interactions with a website [...]

> basic tcpdump won't work due to the SSL. If I spoof the SSL certificate,
> the Flash will probably fall over.

> So is there any man-in-the-browser thing I can use to record a session
> without interfering with what the browser does?

I can outline the principle of a MitM type approach. But I haven't tried
it myself. (Yet.)

1. Create a certificate authority. Install CA in your webbrowser in as
many places as necessary so that Flash can find it.

2. Create web site locally with same name as target website. Use homegrown
CA to create and sign an SSL certificate. Check that sample pages on
said website are delivered to browser

3. Create a MitM web proxy that listens on SSL/443 but simply forwards
the requests over a new https connection to the true target website. Oh,
and while it's forwarding you can have it dump the traffic to a series
of timestamped files, or a log, or whatever.

Unfortunately these things exist in the corporate world as web
proxies - and as corporate (Windows) machines tend to accept the
certificate authority implemented by Active Directory it becomes pretty
straightforward to have the web browsers trust your proxy.

Chris

[Justin C]

On 2012-10-26, Theo Markettos <theom...@chiark.greenend.org.uk> wrote:
>
> I want to record all my interactions with a website - in other words what I
> send it, what is given back to me, rinse and repeat. The difficulty is the
> website uses SSL, Javascript and Flash to do the interaction. So basic
> tcpdump won't work due to the SSL. If I spoof the SSL certificate, the
> Flash will probably fall over. I can't use a perl script because of the
> javascript and Flash that need to run.

Don't write perl off just yet. There is a javascript plugin for
the WWW::Mechanize module, and there are ways of interacting with
flash in the Mech module too:

<URL:http://search.cpan.org/~sprout/WWW-Mechanize-Plugin-JavaScript-0.010/lib/WWW/Mechanize/Plugin/JavaScript.pm>

<URL:http://search.cpan.org/~corion/WWW-Mechanize-Firefox-0.31/lib/WWW/Mechanize/Firefox/Cookbook.pod>


Justin.