REST contents missing

78 views
Skip to first unread message

pdebo...@gmail.com

unread,
Apr 29, 2019, 4:57:27 PM4/29/19
to Pcap.Net Q&A
I'm by no means TCP/IP fluent. So, don't assume I know the basics.

I'm trying to capture Some REST command traffic and have succeeded in capturing the traffic I send in Postman.

However, when I'm send using HttpClient in C# the traffic snooper finds http packets with no JSON content.

Here is the snooper thread, cobbled together from the other posts in this group.

private void MonitorTraffic()
{
// Retrieve the device list from the local machine
IList<LivePacketDevice> allDevices = LivePacketDevice.AllLocalMachine;

if (allDevices.Count == 0)
{
WriteLog("No interfaces found! Make sure WinPcap is installed.");
return;
}

// Print the list
for (int i = 0; i != allDevices.Count; ++i)
{
LivePacketDevice device = allDevices[i];
WriteLog((i + 1) + ". " + device.Name);
if (device.Description != null)
WriteLog("(" + device.Description + ")");
else
WriteLog("(No description available)");
}
int deviceIndex = 2;

// Take the selected adapter
PacketDevice selectedDevice = allDevices[deviceIndex - 1];

// Open the device
using (PacketCommunicator communicator =
selectedDevice.Open(65536, // portion of the packet to capture
// 65536 guarantees that the whole packet will be captured on all the link layers
PacketDeviceOpenAttributes.Promiscuous, // promiscuous mode
1000)) // read timeout
{
communicator.SetFilter($"tcp && ( ( ip dst {ATS_IP} && ip src {AIO_IP} ) || ( ip dst {AIO_IP} && ip src {ATS_IP} ) )");

WriteLog("Listening on " + selectedDevice.Description + "...");

// Retrieve the packets
Packet packet;
do
{
PacketCommunicatorReceiveResult result = communicator.ReceivePacket(out packet);
//WriteLog(packet.ToString() + "\r\n");
if (result == PacketCommunicatorReceiveResult.Ok)
{
if (packet.DataLink.Kind == DataLinkKind.Ethernet)
{
if (packet.Ethernet.EtherType == PcapDotNet.Packets.Ethernet.EthernetType.IpV4)
{
if (packet.Ethernet.IpV4.Protocol == PcapDotNet.Packets.IpV4.IpV4Protocol.Tcp)
{
if (packet.Ethernet.IpV4.Tcp.Http != null)
{
TcpDatagram tcp = packet.Ethernet.IpV4.Tcp;
HttpDatagram http = packet.Ethernet.IpV4.Tcp.Http;
if (http.IsRequest && http.IsValid)
{
String msg = http.Decode(Encoding.UTF8).Split('\n')[0];
if (msg.StartsWith("GET ") || msg.StartsWith("POST "))
{
string payload0 = packet.Ethernet.Decode(Encoding.ASCII).ToString();
string payload1 = packet.Ethernet.Payload.Decode(Encoding.UTF8).ToString();
string payload = tcp.Payload.Decode(Encoding.UTF8).ToString();
//WriteLog(payload.ToString() + "\r\n");
//if (payload.Contains("{"))
{
reqJSON = payload.Substring(payload.IndexOf("{") + 1, payload.LastIndexOf("}") - payload.IndexOf("{") - 1);
reqJSON = reqJSON.Trim();
WriteLog(reqJSON);
}
}
}
if (http.IsResponse && http.IsValid)
{
String msg = http.Decode(Encoding.UTF8).Split('\n')[0];
if (msg.StartsWith("HTTP/1.1 200 OK"))
{
string payload = tcp.Payload.Decode(Encoding.UTF8).ToString();
//WriteLog(payload.ToString() + "\r\n");
rspJSON = payload.Substring(payload.IndexOf("{") + 1, payload.LastIndexOf("}") - payload.IndexOf("{")-1);
rspJSON = rspJSON.Trim();
WriteLog(rspJSON);
if (reqJSON != null)
ParseJSON(reqJSON, rspJSON);
else
WriteLog("reqJSON was NULL!");
reqJSON = null;
rspJSON = null;
}
}
}
}
}
}
}
} while (!this.IsDisposed);
}
}

pdebo...@gmail.com

unread,
Apr 29, 2019, 5:05:37 PM4/29/19
to Pcap.Net Q&A
The pcappng file here shows the two packets from 10.180.8.160

first being the bad one sent by HttpClient

the second being the good one from Postman

https://drive.google.com/open?id=1uiO1jhtONudOJi4jfQxJdT59DfT55-kJ

In the Snooper, packet.Ethernet.IpV4.Tcp.Payload decodes to

"POST /remote/GetSignal HTTP/1.1\r\nAccept: application/json\r\nContent-Type: application/json; charset=utf-8\r\nHost: 10.180.8.160:5050\r\nContent-Length: 42\r\nExpect: 100-continue\r\nConnection: Keep-Alive\r\n\r\n"

Ending before I get any JSON data.

Any ideas?

pdebo...@gmail.com

unread,
May 16, 2019, 10:35:36 AM5/16/19
to Pcap.Net Q&A
Seems the missing content is in a second segment. How do I reassemble a segmented packet?

Pablo Andréi

unread,
May 25, 2019, 4:58:11 PM5/25/19
to Pcap.Net Q&A
I have the same problem.
Were you able to solve your problem?

pdebo...@gmail.com

unread,
May 26, 2019, 10:23:39 PM5/26/19
to Pcap.Net Q&A
No, I did not. I moved on to use purely HTTP monitors like titanium proxy. Still has problems but doesn't require pcap and you don't need to muck around in TCP layer. I would still love to have a solution here as its simpler to use.

Pablo Andréi

unread,
May 27, 2019, 8:10:33 AM5/27/19
to Pcap.Net Q&A
I did an algorithm yesterday to capture, and it worked!

Basically, comparing frames, storing frames containing the JSON or HTPP a list, and for last, I waited for the frame containing PUSH + ACK to concatenate the list and have the result.

Example:

             IpV4Datagram ip = packet.Ethernet.IpV4;
            TcpDatagram tcp = ip.Tcp;

            Datagram datagram = tcp.Payload;

            HttpDatagram httpDatagram = packet.Ethernet.IpV4.Tcp.Http;

            if (null != datagram)
            {
                string payloadHTTP = Common.RxToPayloadHTTP(datagram);

                if (tcp.PayloadLength != 0 && tcp.IsValid)
                {
                    listReassembledPacket.Add(
                        new ReassembledPacket()
                        {
                            Frame = Monitor.frame,
                            Destination = ip.Destination.ToString(),
                            DestinationPort = tcp.DestinationPort,
                            Source = ip.Source.ToString(),
                            SourcePort = tcp.SourcePort,
                            NextSequenceNumber = tcp.NextSequenceNumber,
                            SequenceNumber = tcp.SequenceNumber,
                            PayloadLenght = tcp.PayloadLength,
                            PayLoadHTTP = payloadHTTP,
                            Datagram = tcp.Payload,
                            Protocol = ip.Protocol.ToString(),
                            IsValid = tcp.IsValid,
                            IsPush = tcp.IsPush
                        }
                        );

                    if (listReassembledPacket.Count > 1)
                    {
                        found = Common.CheckReassembledPacket();
                    }
                }

                if (listReassembledPacket.Count > 0)
                {
                    if (listReassembledPacket[listReassembledPacket.Count - 1].IsValid &&
                        listReassembledPacket[listReassembledPacket.Count - 1].IsPush)
                    {
                        if (payloadHTTP.Length > 0) //Return ACK is Lenght ==0
                        {
                            string @message = ParseHTTP.ParseTransaction(Common.ConcatPayload());

The @message is listReassembledPacket concated.

Sincerely,
Pablo

pdebo...@gmail.com

unread,
May 28, 2019, 8:43:01 AM5/28/19
to Pcap.Net Q&A
You are correct Pablo. I managed to find the rest of the segments. Somehow I did not see that they too were HTTP packets, just they don't start with "POST". Thanks for responding. I'm actually returning to use this library now.
Reply all
Reply to author
Forward
0 new messages