Is there any HTTP layer manipulation example available?

[kroval...@gmail.com]

PCapPlusPlus is a very useful lib,it make packet sniff easier than before, and I want ask

1  Is there any HTTP layer manipulation example available?

2  Can PcapPlusPlus manipulate HTTP body but not just HTTP request&response header?

[PcapPlusPlus Support]

Unfortunately there's isn't an HTTP manipulation example yet. I'm planning to create one but it'll take some time.
In the meantime you can look at the unit-tests to see some code:
1. You can take a look at Packet++Test/main.cpp at the following tests:
    - HttpRequestLayerParsingTest
    - HttpRequestLayerCreationTest
    - HttpRequestLayerEditTest
    - HttpResponseLayerParsingTest
    - HttpResponseLayerCreationTest
    - HttpResponseLayerEditTest

2. You can read the HttpLayer documentation: http://seladb.github.io/PcapPlusPlus-Doc/Documentation/a00095.html


Currently PcapPlusPlus can only parse and manipulate HTTP headers. The main reason is that HTTP body usually consists of more than 1 packet and PcapPlusPlus doesn't yet have the infrastructure to support parsing messages spreading over more than 1 packet.
I hope I'll be able to add this in the future

[kroval...@gmail.com]

Thanks again,Packet reconstruction of HTTP layer is very difficult since HTTP packet have no serial No.. Parsing message over more than 1 packet maybe solved by some algorithm depends on ethernet packet reconstruction.

[PcapPlusPlus Support]

For parsing HTTP messages spreading over more than 1 packet one must first reconstruct the TCP stream. TCP stream reconstruction isn't a trivial task but it's a feasible one
But unfortunately PcapPlusPlus doesn't have this infra yet

[kroval...@gmail.com]

Now I have an idea which we can just identify HTTP stream by IP + Port infomation,then we forward this stream to a http proxy to process it.

[PcapPlusPlus Support]

Yes, this is possible, depends on what you want to achieve

[Richard Howell-Peak]

So just to clarify then, I'm assuming you're talking about the fact that a TCP packet has a max size, and therefore if an HTTP message has a size larger than that then the HTTP message will be split into two or more TCP packets.

And what you're saying is the PcapPlusPlus library does not currently manage to reassemble the multiple TCP packets into one HTTP message again?

If that's the case, what does it do?  As in, if the callback function reports a packet incoming, and it is the first of two packets for an HTTP message, and I go up through the layers to find the HTTP layer, what will I find?

Would it just be the first half of the HTTP message, and the other half is just floating around somewhere else and will be in-coming later in the day?  Or does the information get lost completely?

[PcapPlusPlus Support]

You're statement is accurate.
PcapPlusPlus parses each packet on its own, regardless of other packets that may be related to it. In the case of HTTP the discovery of HTTP is done by TCP ports (80 or 8080) and discovering of an HTTP method (GET, POST, etc.) at the beginning of the HTTP layer. So in the case of HTTP message that is split between several packets the following will happen:
- When the first packet arrives, PcapPlusPlus will recognize it as HTTP because the TCP port is 80 or 8080 and because the HTTP layer begins with an HTTP method (GET, POST, etc.)
- When the other packets arrive, PcapPlusPlus won't recognize them as HTTP because although the TCP port is 80 or 8080 the HTTP layer doesn't begin with an HTTP method. So it will only recognize the Ethernet, IP and TCP layers, and won't recognize the HTTP, so you won't be able to find the HttpLayer for these packets. If you want to get the HTTP raw data for these packets you can look at the payload of the TcpLayer by using the TcpLayer::getLayerPayload method. But it's obviously the raw data (a byte array), not a parsed layer.

The way to solve it is to write a TCP re-assembly module that will gather all packets of the HTTP message into 1 big packet and let PcapPlusPlus parse it. If this feature is demanded I can write it, or someone else can write it and contribute it to PcapPlusPlus code-base

Thanks.

[Richard Howell-Peak]

Hi,

To be honest that would be an extremely useful feature for me, as at the moment it is dropping the later part of any image files, which means they get all corrupted which is a major problem for the project I'm working on.

Richard.

[PcapPlusPlus Support]

I can work on such feature but it would take time. How urgent is it for you? If it's urgent maybe you can write it yourself and merge it to PcapPlusPlus code base

[Richard Howell-Peak]

Hi.  I'm currently looking at implementing this feature.  Just as a note on how to do it, will I be doing something like this:

1) Each packet that comes through, check for the existence of an HTTP layer.

2) Somehow determine if this packet contains all the data or not (by looking at the payload size and seeing if it's less than max packet length?)

3) If I get one, start looking for TCP packets with port 80 or 8080, but don't have an HTTP layer.  keep these in a list

4) Somehow know when to stop collecting TCP packets (is there some indication in the TCP header to say when the whole chain is complete?  I hope so!)

5) Now, put the payload of the whole list of packets into one place - that should be the entire payload of the original HTTP packet.

Is that right?  Not sure how I would then parse it into PcapPlusPlus, or why I would even need to do that.  Your thought would be much appreciated. :)

Richard.

[Richard Howell-Peak]

Ok guys, so I've made some progress on this.

Essentially what I've done is get the content-length field out of the response header, and compared it to the payload size.  That's how to tell if there is more data to come, also check for Transfer-Encoding:chunked, that also means there's more to come.

I'm working on the rest of it now

[PcapPlusPlus Support]

The basic concept you propose is ok, here are my comments:

• You need to sort packets per connection first, and do the work you mention per connection. You can use the hash5Tuple() method in Packet++ for that
  
• Please keep in mind that an HTTP connection contains both HTTP requests and responses and they may come simultaneously. So if you want to parse only responses you need to filter only packets with source port 80 or 8080
• In some cases TCP packets get mixed on the way from side to side, meaning that if an HTTP response is assembled from TCP packet 1,2 and 3 you may see 2 before 1 or even 3 first, then 1 and then 2. In order to solve that issue you need first to do TCP re-assembly, meaning verify TCP packets order and re-order them if needed. Only then you can get to HTTP parsing
• TCP re-assembly is also good for removing packets with duplicate data, TCP re-transmission packet and packet with overlap data
  
• Please keep in mind that in most cases HTTP responses are compressed with gzip so in order to get the HTML/JS/etc data you need to decompress the gzip data
• You're absolutely right regarding content-length and chunked. These are the 2 ways I know to identify the end of an HTTP message
• So my proposal for the general architecture is:
• Have a component that does TCP re-assembly. This component should receive TCP packets, re-order them if needed, remove duplicates, re-transmissions and overlaps. It should output the TCP payload only in an orderly fashion (much like sockets do)
  
• Have another component that does HTTP parsing. It should receive the TCP payload from the TCP re-assembly component and do the HTTP parsing. You can use the parsing currently done in HttpLayer but you need this payload to look like a packet (still need to think what is the best way to do this). You should extend the HttpLayer capabilities to do not only header parsing but also body parsing including looking at content-length, chunked, use gzip to decompress data atc. 

What do you think?

[Richard Howell-Peak]

Hmm, I hadn't considered the problem of TCP reassembly, I thought TCP guaranteed the order of packets.  I'll have to rethink how I'm doing this!

I do have one question though, one issue I've got is getting data out of the TCP Layer.  I'm really not sure which methods to call, like getNextLayer, parseNextLayer, getData etc.  For the HTTP Message layer it worked fine to do this:

 p = (pcpp::PayloadLayer*) httpResponse->getNextLayer();

Then p->getPayload() 

and this got me a pointer to the data.  When the images are sufficiently small, this does indeed return the full image.

Of course the next packets coming through are TCP packets and have no HTTP header.  So I'm currently doing this:

tcpLay->parseNextLayer();

p =  (pcpp::PayloadLayer*)  tcpLay->getNextLayer();

p->getPayload();

And it doesn't seem to be working really.  Am I using the correct combination of functions or have I misunderstood the documentation here?  If this is the correct way to do this then that means the problem is elsewhere (i.e. packet reassembly or some such problem), just wanted to make sure I was doing this correctly.

Thanks.

[PcapPlusPlus Support]

Regarding TCP reassembly - TCP does guarantee the order of packets. That's why each packet contains sequence number and ack number. But from all sorts of reasons, packets may arrive to a machine at an incorrect order. When working with the OS (for example using sockets) the OS is taking care of packet reordering, removing duplicates, etc. But when getting packets directly from the NIC you need to take care of that yourself.

Regarding your question about the payload - it seems you're doing the right thing. I can't understand why it doesn't work. If you want you can send me the code and a pcap example and I can try to help you find the problem

[667...@gmail.com]

Hi,  

Has the problem been solved?

在 2016年5月6日星期五 UTC+8下午7:26:25，PcapPlusPlus Support写道：

[PcapPlusPlus Support]

Which problem are you referring to?

Thanks,

PcapPlusPlus maintainer

--
You received this message because you are subscribed to the Google Groups "PcapPlusPlus support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pcapplusplus-sup...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pcapplusplus-support/a234afaa-53b9-4eea-a23f-12c352c731db%40googlegroups.com.