Network capture of a Quarkus application

[Georgios Andrianakis]

Hi folks,

I did a super quick and dirty experiment to see whether or not we could generate a .pcap file for a running Quarkus application.

It turns out that this is possible, but it will require a patch in Vert.x (and maybe Netty), so before moving forward with it, I am wondering if you think this is something worthwhile and also if we can leverage this file in any meaningful way in Quarkus.

WDYT?

[George Gastaldi]

I think that would be useful, assuming you could open it with WireShark or similar tools

--
You received this message because you are subscribed to the Google Groups "Quarkus Development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CALeTM-kAx04AWYcpidRyZ2siS8YwFB3oD8s0BkgZPmQXHshbMQ%40mail.gmail.com.

[Georgios Andrianakis]

Yes, I verified that I can open the captured file in Wireshark.

My main concern is whether we can come up with an interesting way to use this, otherwise I am not sure if the patches are justified.

[Max Rydahl Andersen]

what is actually in that .pcap file - what does it enable ?

Asking naively here as I always end up manually configuring wireshark :)

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CALeTM-mzryb5cLMjH1g0dkgubTRymqjY_X%2BjoX9a2Ar45JXSdQ%40mail.gmail.com.

[George Gastaldi]

Would it do something different compared to setting up Wireshark manually to an existing Quarkus app? If all that takes is enabling it through a configuration property, I'm sold :)

[Georgios Andrianakis]

On Wed, Dec 22, 2021 at 4:18 PM Max Rydahl Andersen <mand...@redhat.com> wrote:

what is actually in that .pcap file - what does it enable ?

Asking naively here as I always end up manually configuring wireshark :)

It's the file format used by tcpdump and Wireshark. So if we produce it, Wireshark can open it

[Georgios Andrianakis]

On Wed, Dec 22, 2021 at 4:19 PM George Gastaldi <ggas...@redhat.com> wrote:

Would it do something different compared to setting up Wireshark manually to an existing Quarkus app? If all that takes is enabling it through a configuration property, I'm sold :)

The way I see it is that if we have the file, we can potentially use tools to parse it and hopefully come up with some interesting use case for using the captured data

[Loïc MATHIEU]

This is interesting, this will be even more interesting if we could open it inside the dev ui and filter on source/target IP/port ... So if we have wireshark inside the dev UI :)

But anyway, it's interesting if you can trigger the creation of the file filtered on packets received/sent via the application, as any time I use tcpdump or whireshark there are dozens (or hundreds) of frames coming from my tens of opened browser tabs ...

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CALeTM-mXMrHTHjC_AEyhSGq3qh43GbworHHv7iOLfOizKVuESA%40mail.gmail.com.

[Max Rydahl Andersen]

On 22 Dec 2021, at 15:26, Georgios Andrianakis wrote:

On Wed, Dec 22, 2021 at 4:18 PM Max Rydahl Andersen <mand...@redhat.com>
wrote:

what is actually in that .pcap file - what does it enable ?

Asking naively here as I always end up manually configuring wireshark :)

It's the file format used by tcpdump and Wireshark. So if we produce it,
Wireshark can open it

oooooh - it's about capturing the network traffic we generate into a buffered file
which then can be introspected. I for some reason thought it was some custom config
to make wireshark good to capture - my bad.

So I think its a nice idea and has some technical debug value, but if that pcap does
not contain all network traffic from the javavm then I think its more misleading than good to be honest?
Also, does this .pcap approach provide something that HTTP Toolkit or even just straight
up Wireshark snooping can't bring ?

i.e. will .pcap show decoded https traffic ?

/max

Yes, I verified that I can open the captured file in Wireshark.

My main concern is whether we can come up with an interesting way to use
this, otherwise I am not sure if the patches are justified.

On Wed, Dec 22, 2021 at 4:10 PM George Gastaldi <ggas...@redhat.com>
wrote:

I think that would be useful, assuming you could open it with WireShark
or similar tools

On Wed, Dec 22, 2021 at 11:00 AM Georgios Andrianakis <
gand...@redhat.com> wrote:

Hi folks,

I did a super quick and dirty experiment to see whether or not we could
generate a .pcap file for a running Quarkus application.
It turns out that this is possible, but it will require a patch in
Vert.x (and maybe Netty), so before moving forward with it, I am wondering
if you think this is something worthwhile and also if we can leverage this
file in any meaningful way in Quarkus.

WDYT?

--
You received this message because you are subscribed to the Google
Groups "Quarkus Development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to quarkus-dev...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/quarkus-dev/CALeTM-kAx04AWYcpidRyZ2siS8YwFB3oD8s0BkgZPmQXHshbMQ%40mail.gmail.com

<https://groups.google.com/d/msgid/quarkus-dev/CALeTM-kAx04AWYcpidRyZ2siS8YwFB3oD8s0BkgZPmQXHshbMQ%40mail.gmail.com?utm_medium=email&utm_source=footer>
.

--

You received this message because you are subscribed to the Google Groups
"Quarkus Development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to quarkus-dev...@googlegroups.com.
To view this discussion on the web visit

https://groups.google.com/d/msgid/quarkus-dev/CALeTM-mzryb5cLMjH1g0dkgubTRymqjY_X%2BjoX9a2Ar45JXSdQ%40mail.gmail.com
<https://groups.google.com/d/msgid/quarkus-dev/CALeTM-mzryb5cLMjH1g0dkgubTRymqjY_X%2BjoX9a2Ar45JXSdQ%40mail.gmail.com?utm_medium=email&utm_source=footer>
.

[Georgios Andrianakis]

On Wed, Dec 22, 2021 at 4:58 PM Max Rydahl Andersen <mand...@redhat.com> wrote:

On 22 Dec 2021, at 15:26, Georgios Andrianakis wrote:

On Wed, Dec 22, 2021 at 4:18 PM Max Rydahl Andersen <mand...@redhat.com>
wrote:

what is actually in that .pcap file - what does it enable ?

Asking naively here as I always end up manually configuring wireshark :)

It's the file format used by tcpdump and Wireshark. So if we produce it,
Wireshark can open it

oooooh - it's about capturing the network traffic we generate into a buffered file
which then can be introspected. I for some reason thought it was some custom config
to make wireshark good to capture - my bad.

So I think its a nice idea and has some technical debug value, but if that pcap does
not contain all network traffic from the javavm then I think its more misleading than good to be honest?
Also, does this .pcap approach provide something that HTTP Toolkit or even just straight
up Wireshark snooping can't bring ?

It will capture all traffic that goes through Vertx - so basically the entirety of our blessed IO stack.

 

i.e. will .pcap show decoded https traffic ?

I haven't tried, I'll need to check it out.

[Georgios Andrianakis]

On Wed, Dec 22, 2021 at 4:49 PM Loïc MATHIEU <loik...@gmail.com> wrote:

This is interesting, this will be even more interesting if we could open it inside the dev ui and filter on source/target IP/port ... So if we have wireshark inside the dev UI :)

I would love to have something like that, but I wasn't able to find anything ready made.

But anyway, it's interesting if you can trigger the creation of the file filtered on packets received/sent via the application, as any time I use tcpdump or whireshark there are dozens (or hundreds) of frames coming from my tens of opened browser tabs ...

Yeah, providing out of the box capturing is useful in itself for that exact reason.

[Max Rydahl Andersen]

On 22 Dec 2021, at 16:02, Georgios Andrianakis wrote:

> On Wed, Dec 22, 2021 at 4:49 PM Loïc MATHIEU <loik...@gmail.com> wrote:
>
>> This is interesting, this will be even more interesting if we could open
>> it inside the dev ui and filter on source/target IP/port ... So if we have
>> wireshark inside the dev UI :)
>>
>
> I would love to have something like that, but I wasn't able to find
> anything ready made.
>
>>
>> But anyway, it's interesting if you can trigger the creation of the file
>> filtered on packets received/sent via the application, as any time I use
>> tcpdump or whireshark there are dozens (or hundreds) of frames coming from
>> my tens of opened browser tabs ...
>>
>
> Yeah, providing out of the box capturing is useful in itself for that exact
> reason.

Yeah - so thats a good point - focused capturing.
That is though what tools like HTTP Toolkit gives you but for everything in the jvm;
no matter if vert.x or not.

I would say its worth it if the https traffic is visible - if not ...then not
that useful ?

/max

>>>>>>> <https://groups.google.com/d/msgid/quarkus-dev/CALeTM-kAx04AWYcpidRyZ2siS8YwFB3oD8s0BkgZPmQXHshbMQ%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>>>>>> .

>>>>>>>
>>>>>> --
>>> You received this message because you are subscribed to the Google Groups
>>> "Quarkus Development mailing list" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an
>>> email to quarkus-dev...@googlegroups.com.
>>> To view this discussion on the web visit

>>> https://groups.google.com/d/msgid/quarkus-dev/CALeTM-mXMrHTHjC_AEyhSGq3qh43GbworHHv7iOLfOizKVuESA%40mail.gmail.com
>>> <https://groups.google.com/d/msgid/quarkus-dev/CALeTM-mXMrHTHjC_AEyhSGq3qh43GbworHHv7iOLfOizKVuESA%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>> .

>>>
>>
>
> --
> You received this message because you are subscribed to the Google Groups "Quarkus Development mailing list" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CALeTM-nBO5CjB_rndyT6WdNmKV%3DZ20t3h2H7WeJWYHN0u8kuHA%40mail.gmail.com.

[Den]

Hi,

recently I had to capture the GELF messages sent from quarkus app via UDP, so it probably would have been helpful, provided it can handle UDP and gzipped traffic

den

[Georgios Andrianakis]

On Wed, Dec 22, 2021 at 5:05 PM Max Rydahl Andersen <mand...@redhat.com> wrote:

On 22 Dec 2021, at 16:02, Georgios Andrianakis wrote:

> On Wed, Dec 22, 2021 at 4:49 PM Loïc MATHIEU <loik...@gmail.com> wrote:
>
>> This is interesting, this will be even more interesting if we could open
>> it inside the dev ui and filter on source/target IP/port ... So if we have
>> wireshark inside the dev UI :)
>>
>
> I would love to have something like that, but I wasn't able to find
> anything ready made.
>
>>
>> But anyway, it's interesting if you can trigger the creation of the file
>> filtered on packets received/sent via the application, as any time I use
>> tcpdump or whireshark there are dozens (or hundreds) of frames coming from
>> my tens of opened browser tabs ...
>>
>
> Yeah, providing out of the box capturing is useful in itself for that exact
> reason.

Yeah - so thats a good point - focused capturing.
That is though what tools like HTTP Toolkit gives you but for everything in the jvm;
no matter if vert.x or not.

I would say its worth it if the https traffic is visible - if not ...then not
that useful ?

I just did another totally hacky patch and yes, when the Reactive Rest Client sends a request to an HTTPS service (in my example to https://stage.code.quarkus.io/api), the capture shows all the information

[Sanne Grinovero]

This is pretty cool, but is it worth maintaining? Honestly I suspect that if I were to need a network capture, I might as well just fire up Wireshark - I think I'd prefer that as it would give me higher confidence about seeing the real picture.

One aspect that might be interesting of this approach is to be able to run integration tests on expectations of network traffic. But I've never had such a need in practice.

Thanks,

Sanne

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CALeTM-mCOzVYbP2GRjwsYjMm8AW7doDsB%3DQndV7oh1pyRoa-Fw%40mail.gmail.com.

[Georgios Andrianakis]

On Wed, Dec 22, 2021 at 6:06 PM Sanne Grinovero <sa...@hibernate.org> wrote:

This is pretty cool, but is it worth maintaining? Honestly I suspect that if I were to need a network capture, I might as well just fire up Wireshark - I think I'd prefer that as it would give me higher confidence about seeing the real picture.

One aspect that might be interesting of this approach is to be able to run integration tests on expectations of network traffic. But I've never had such a need in practice.

Exactly, this is the kind of thing I would like us to explore. I agree that just being able to capture the packets is neat, but probably does not warrant the cost. If however the capture unlocked new potential, then I think it would be totally worth it

[romain...@gmail.com]

Hi,

This could be useful, as we needed to find someone with right permissions to run tcpdump in an OpenShift pod.

However in our case the issue was with the Quarkus REST client based on Apache Http client (TCP FIN packet from server blocked by network firewall), thus it wasn't going through the Vert.x/Netty stack.

Cheers,

Romain

[Georgios Andrianakis]

Interesting use case. 

But as you already mentioned, this would only work for Netty / Vert.x IO

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/93daf382-bb76-4bb5-a132-d822b067229bn%40googlegroups.com.

[Julien Viet]

Hi Georgios,

this is a feature that we wanted to add in Vert.x when we saw the
existing Netty PCAP handler
(https://github.com/netty/netty/issues/10385)

It is listed here https://github.com/vert-x3/issues/issues/566 as "pcap logging"

I think it is quite easy to implement with a few modifications to
vertx options and the building of the Netty pipeline.

we would be happy to have such contribution :-)

Julien

> --
> You received this message because you are subscribed to the Google Groups "Quarkus Development mailing list" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CALeTM-kAx04AWYcpidRyZ2siS8YwFB3oD8s0BkgZPmQXHshbMQ%40mail.gmail.com.

[Georgios Andrianakis]

Hey Julien,

On Thu, Dec 23, 2021 at 11:44 PM Julien Viet <jul...@julienviet.com> wrote:

Hi Georgios,

this is a feature that we wanted to add in Vert.x when we saw the
existing Netty PCAP handler
(https://github.com/netty/netty/issues/10385)

This is exactly what I am using in my hacks :)

It is listed here https://github.com/vert-x3/issues/issues/566 as "pcap logging"

I think it is quite easy to implement with a few modifications to
vertx options and the building of the Netty pipeline.

we would be happy to have such contribution :-)

I didn't know about the Vert.x issue, thanks for pointing it out.

I will prepare a draft PR when we get back from the holidays, as I see this is useful for Vert.x regardless of Quarkus

[Georgios Andrianakis]

On a slightly related note, since Max has been playing with Http Toolkit and wanted an easy way to configure our Reactive REST Client to use it, I opened https://github.com/httptoolkit/jvm-http-proxy-agent/pull/9 which will enable the use of their agent with our stack