Divide tcpflow output files to HTTP request-response flows

[tcpflow user]

Hi,

Sorry if that is not the correct forum...

I am wondering if it possible to order the output files of tcpflow so i could easily see a stream of http request-response files.

Is there is a way to determine which request the server response is belongs to?

Thank you very much

[Simson Garfinkel]

Not at the moment. 

--
You received this message because you are subscribed to the Google Groups "tcpflow-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tcpflow-user...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[tcpflow user]

And if read the files content by script is it possible to determine which request the server response is belongs to?

Thanks a lot.

[Simson Garfinkel]

Sure. The ports are flipped. 

[tcpflow user]

Thank you very much.

I just didn't understand how could I determine which response is belongs to which request as it's store in seperated files, usually with some requests store in the same file.

Thanks again.

[Simson Garfinkel]

If you have one request per file, it's not an issue. If you have multiple requests, then  you can use the lengths to decode it. If you want to deal with a single connection, you can use the console mode output.

[tcpflow user]

Just want to be sure:

I have, for example, file with 30 requests from my ip (seperated by empty lines) and 30 response files of "HTTPBODY" (with connection number at the end of filename).

How can I map the requests to the HTTPBODY files? Does the connection numbers match the requests order?

Thank you very much

[Simson Garfinkel]

I suggest that you look carefully in the files. You should be able to figure it out. All of the information that you need is present.

[tcpflow user]

Thanks.

I made a mistake in the previous quastion and i write it here for other useres. the numbers at the end of "HTTPBODY" file are not a connection number but sequaence numbers.

a cursory glance at the files shows that the sequence is match the requests sequence (hopefully. that is what I want to know, and therfore I opened this post)

[Simson Garfinkel]

Great. Glad you got it.

[tcpflow user]

Thanks.

Some notes and another question:
Using tcpflow for windows (downloaded from http://www.digitalcorpora.org/downloads/tcpflow/)  I see significant differents between the output of the 64bit version and the 32bit version, on the same pcap file with the same command.

The command:

 tcpflow32.exe -a -r test.pcap -o .\tc32

and 

 tcpflow64.exe -a -r test.pcap -o .\tc64

yields completely different output:

1. There is only 1 HTTPBODY file for each response flow in the 32bit version (in the 64bit ver there are many file, depends on the responses number). that could be explained by the different version (1.3.0 for 32bit and 1.4.0_a2 for 64bit)
   
2. When I compare the same flow files between the two version i see great differents. for example- in the 32bit version there are 31 requests in one file while in the 64bit version there are only 2 requests (which included also in the 32bit file) on the same file.

Take the above into account i want to write a python script that builds request-response files from the 32bit version output.

My question is:

If there is no response for some requests (due to, for example, network error)- could I see it in the flow file which contains the server responses?

Tahnk you very much

[Simson Garfinkel]

Yes, this is explained by the version number. You should not be using version 1.3.0.

Why do you want to use the 32-bit version?

On Oct 9, 2013, at 6:03 AM, tcpflow user <tcpflo...@gmail.com> wrote:

Thanks.

Some notes and another question:
Using tcpflow for windows (downloaded from http://www.digitalcorpora.org/downloads/tcpflow/)  I see significant differents between the output of the 64bit version and the 32bit version, on the same pcap file with the same command.

The command:

 tcpflow32.exe -a -r test.pcap -o .\tc32

and 

 tcpflow64.exe -a -r test.pcap -o .\tc64

yields completely different output:

1. There is only 1 HTTPBODY file for each response flow in the 32bit version (in the 64bit ver there are many file, depends on the responses number). that could be explained by the different version (1.3.0 for 32bit and 1.4.0_a2 for 64bit)
   
2. When I compare the same flow files between the two version i see great differents. for example- in the 32bit version there are 31 requests in one file while in the 64bit version there are only 2 requests (which included also in the 32bit file) on the same file.

Take the above into account i want to write a python script that builds request-response files from the 32bit version output.

My question is:

If there is no response for some requests (due to, for example, network error)- could I see it in the flow file which contains the server responses?

Tahnk you very much

On Saturday, October 5, 2013 11:23:51 PM UTC+3, SLG wrote:

Great. Glad you got it.

On Oct 5, 2013, at 3:00 PM, tcpflow user <tcpflo...@gmail.com> wrote:

> Thanks.
> I made a mistake in the previous quastion and i write it here for other useres. the numbers at the end of "HTTPBODY" file are not a connection number but sequaence numbers.
> a cursory glance at the files shows that the sequence is match the requests sequence (hopefully. that is what I want to know, and therfore I opened this post)

[tcpflow user]

because the 64bit version gives me wrong results.

[Simson Garfinkel]

Can you provide me a copy of the packet flow?

What happens when you run on Linux?

[tcpflow user]

Unfortunately i can't provide you a copy because it's including some private data.

However I just did the same test with verstion 1.4.0b1 (both 32bit and 64bit versions) and it seems to be OK (I made the previous test with version 1.3.0 because it is the only file that including "win" in its name at http://digitalcorpora.org/downloads/tcpflow/ and now I have noticed that there are win versions for all of the releases. Sorry).

[Simson Garfinkel]

Sorry about the confusion. I cleaned up the downloads directory.

[tcpflow user]

Great.Thank you. 

Be aware that the the link to beta vaersion in 

https://groups.google.com/forum/#!topic/tcpflow-users/xXI_C3azjTs

is now broken.

[tcpflow user]

Ok. Now things seems to be ok but I encountered another problem:

When I run tcpflow on windows with the following arguments-

tcpflow64.exe -a -r .\test.pcap -o .\tc64
the tcpflow throw exception when it try to save files of ipv6 address:

"Cannot open:  .\tc64/****:0000:0000:0000:*****.*****-******:0000:0000:0000..."

My guess is that such filenames are forbidden in windows.

Is there a workaround for this problem?

[Simson Garfinkel]

Filenames with stars are forbidden under windows, as are filenames with colons. How would you like to address this? Nobody has reported this problem before, You're the first!

[tcpflow user]

The stars is my editing (private ip). the colons are part of ipv6 address.

You can replace the colon with dots,undescores or something else...

BTW (for other users): My workaround is to filter the capture program for ipv4 packets only (as I run tcpflow on windows and therfore only on pcap files). Existing pcap files could be filtered by wireshark (filter the displayed packet and then export). 

[Simson Garfinkel]

We have code in bulk_extractor that turns colons and such to underscores. I'll migrate that to tcpflow and get out a new release within a day or so. Thanks for testing the beta. I hope that the 1.4.0 release will ship shortly.

[tcpflow user]

Hi,
Sorry I reopen this thread again...

I wrote a python script that make a pairs of specific http requests and responses, based on the output of the last tcpflow beta version for windows (I'm using the requests flow files and the response HTTPBODY files).
This works fine, but in some rare cases there are more requests than responses files. This problem could happens due to network, capture software or the winpcap driver errors.

Anyway, when I make the pairs I can't figure out which of the responses are missing, and therfore there are mismatches with the built pairs.

My question is:
Using tcpflow, is there a way to figure out which of the responses are missing?
If not, could I ask to handles such situations in the next versions of tcpflow? The most intuitive way is to add the information about the request address of HTTPBODY file to the XML report file.
I think that this information would be welcomed by other users too.

Thank you very much.