pcap2har converter and HAR Visualizer for SPDY packets

[Ashiwan]

Hi,

I am a researcher and im trying to use HAR visualizer to visualize web requests. I understand that both the pcap2har converter and HAR visualizer support only HTTP. Since I am recently working on SPDY packets, I was wondering if it's easy to add support for SPDY in pcap2har converter (which is in python). Please let me know if you foresee any issues in adding support for SPDY.

Thanks,

Ashiwan

[Libo Song]

SPDY is working over SSL, so all the requests and responses are
encrypted. There is no easy way to capture and decrypt the packets.

If you want to see the visualized web requests, their are many
other tools: firebug, chrome devtools, and webpagetest.org. Those
tools works with the browser, so they have more information
than what pcap has.

Best,

[Ashiwan.S.]

Thanks for the immediate reply. I understand that as of now SPDY works
over SSL. But I have an application with a SPDY server and a client.
If both the server and the client are implemented by me, I believe I
can run SPDY by default without any encryption. Wireshark doesn't
parse the un encrypted SPDY packets correct. My requirement is to
analyze the packets between this SPDY server and client. So I can't
really use the browser tools. I am thinking of working on the pcap2har
converter to add the functionality of parsing SPDY packets. Please let
me know if I am missing something. Any suggestions or help will be
highly appreciated.

Thanks,
Ashiwan

--
-Ashiwan

[Ashiwan.S.]

Do someone have any feedback or suggestions on this requirement?
Highly appreciate any help in this regard,

Thanks,
Ashiwan

--
-Ashiwan

[Jake Holland]

I think this would be a valuable addition.

I'm not going to be able to do this right now. If I were, I expect it
would take me something like 1-3 days, depending on how easy it is to
turn my vague understanding from skimming the spdy spec once into a
function that starts with one data stream and make several data streams.

That said, I'm glad to see somebody using this, and even more glad to
see somebody looking to contribute. So if the hint below is enough of a
hint to get you started, I'll take a good look at what you've got and
try to help, if you end up stuck after spending some reasonable effort.

It's remotely possible I will run into this need and maybe even
eventually figure out how to upload my solution if I write it, now that
I'm aware of the need. But if you want it within the next year, you're
probably better off trying to write it.

------

As a hint on getting started, you'll want to look at the way
http/message.py uses tcpdir.data. (That's more or less the relevant
bottom of the stack from where you call http.Flow(f) from main.py.)

That tcpdir object has type tcp.Direction from tcp/direction.py, and its
"data" member holds the in-order TCP-transmitted data that got extracted
from the pcap.

Those tcpdir objects are accessible from main.py as f.rev and f.fwd,
where currently there's a http.Flow(f) call (at which point f is a
tcp.Flow).

Probably what you want to do is, before that loop, figure out whether
any of the tcp flows in flows.flowdict are transporting SPDY instead of
HTTP, and if they are, replace those tcp flows with something like a
spdy.Flow object, with a .rev and a .fwd which are spdy.Direction
objects, which contains an API similar to tcp.Direction.

Then when you pass flows to http.Flow(), your spdy flows would just have
to look sufficiently similar, and the spdy.Flow/spdy.Direction objects
would result in an effectively identical set of outputs.

As a first guess, (with no guarantee I'm right) I think that all you
would need is to implement byte_to_seq, seq_arrival, and
seq_final_arrival (plus the .data member) in the Direction object. In
the Flow object, you'd need fwd and rev, and you might be OK with a
pass-through on the other data members (I doubt you need any of the
functions). I think the intent on those functions in the Flow is:
Byte_to_seq: the first data byte of the flow is byte index 0. This takes
a byte index and returns its tcp sequence number.
Seq_arrival: takes a tcp sequence number and returns the time it first
went to client.
Seq_final_arrival: takes a tcp sequence number and returns the last time
it went to the client before client acked it.

I think you can implement these functions by maintaining an offset
mapping from the data in a spdy flow to the data in the tcp flow, and
forwarding these functions (with your offset) to the tcp.Direction
object that the spdy.Direction object came from.

-----

If that level of explanation is useful to you, and you're willing to do
the actual digging into details and debugging (at least mostly), I'm
very happy to write something similarly helpful a few more times, if it
turns out you need hints on what to try next. But if that looked like
crazy talk that made no sense, then I probably can't help you at
present.

Hope that helps.
Jake -- jholland at fastsoft dot com

[Ashiwan.S.]

Jake,

Thanks a lot for your reply and interest in helping me. Highly
appreciate it.. I am currently reviewing the spdy spec in detail. The
hint which you gave definitely helps me to get started. I will work on
it as soon as I get a good understanding of the spdy spec. If I need
any help during the effort, I shall check with you and the forum.

Thanks,
Ashiwan

--
-Ashiwan