REST Client Proxy demo

[Georgios Andrianakis]

Hi folks,

I wanted to share a demo of a little something I did.

With a few changes to the Quarkus REST Client, my change allows Quarkus to automatically start an HTTP proxy which is used for all the calls for that REST Client.

The reason I did this was that I wanted to capture network traffic between a Quarkus application and a downstream service that was using HTTPS - doing this under normal circumstances requires some heroics, however if we insert a proxy in the middle, the capture is trivial, as shown in the video.

Take a look if you are interested and let me know if you think this is something that makes sense to have or if you think of other use cases where this approach could be beneficial.

Thanks

--

Georgios Andrianakis

Independent Contractor

[Holly Cummins]

This is very cool! Lots of potential use cases for automated testing. (Well, with the caveat that for some test applications, the test classloading needs to be sorted out first.)

We can call it WireQuark. :) 

--

Holly Cummins

Senior Principal Quarkus Software Engineer, Java Champion

Red Hat 

--
You received this message because you are subscribed to the Google Groups "Quarkus Development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CALeTM-%3Dc4iVw_izTGYjV3TOPsjefAzQ%2BKr06TVO3b6EO-7kzZQ%40mail.gmail.com.

[Alex Soto Bueno]

I love the idea, but I am thinking in terms of maintenance (and my idea might not work), I see that now is just capturing traffic, then there will be a request about injecting errors to test, then there will be a request about a mock. So, my fear is that what was started like a simple logging system, ends up with a rewrite of wiremock.

So I don't know that, but does wiremock support an embedded mode? The idea would be that when the property is set to true instead of starting our own HTTP proxy, it starts the wiremock in embedded mode.

Maybe it is not that easy, maybe my idea does not make much sense.

Said that, I love the idea of course.

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/9B7BD5C5-887A-41B5-A14D-BF9EB5CF0FFB%40redhat.com.

[Georgios Andrianakis]

That's part of what I want to figure out - what should be the scope of this?

Based on a similar model of this, you could easily use Wiremock instead of the proxy (which at this point does absolutely nothing but shuttle the requests)

[Max Rydahl Andersen]

cool stuff - reminds me of https://httptoolkit.com/ ; but just only for resteasy client.

My main questions are security concerned - as this is effectively disabling or rather circumventing https.

this is devtime only, right? so no way by accidentally expose it in prod?

is it intended to be something you enable per restclient and not just globally?

also...would it make more sense to have the proxy devservice something that is part of vertx extension and then
any http based client (even raw vert.x) can reroute? i.e. why need for special restclient reactive setup?

/max

[Georgios Andrianakis]

On Wed, Jul 10, 2024 at 1:38 PM Max Rydahl Andersen <mand...@redhat.com> wrote:

cool stuff - reminds me of https://httptoolkit.com/ ; but just only for resteasy client.

My main questions are security concerned - as this is effectively disabling or rather circumventing https.

For what I had in mind, that's the whole point

 

this is devtime only, right? so no way by accidentally expose it in prod?

No way to enable it in prod

is it intended to be something you enable per restclient and not just globally?

Per client

 

also...would it make more sense to have the proxy devservice something that is part of vertx extension and then
any http based client (even raw vert.x) can reroute? i.e. why need for special restclient reactive setup?

The problem is that with anything not managed (Vertx HTTP Client, JDK HTTP Client, Jakarta REST Client) you don't know what the URL is.

With the REST Client you already have all the information available to make this a one flag opt-in process

/max

On 10 Jul 2024, at 9:33, Georgios Andrianakis wrote:

Hi folks,

I wanted to share a demo of a little something I did.

With a few changes to the Quarkus REST Client, my change allows Quarkus to automatically start an HTTP proxy which is used for all the calls for that REST Client.

The reason I did this was that I wanted to capture network traffic between a Quarkus application and a downstream service that was using HTTPS - doing this under normal circumstances requires some heroics, however if we insert a proxy in the middle, the capture is trivial, as shown in the video.

Take a look if you are interested and let me know if you think this is something that makes sense to have or if you think of other use cases where this approach could be beneficial.

Thanks

--

Georgios Andrianakis

Independent Contractor

--
You received this message because you are subscribed to the Google Groups "Quarkus Development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CALeTM-%3Dc4iVw_izTGYjV3TOPsjefAzQ%2BKr06TVO3b6EO-7kzZQ%40mail.gmail.com.

[William Burke]

If Georgios's proxy adds value and can only be done with Quarkus, why not grow it?

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CAOaD6kL_O82ZQB3JCks%3DDooEXLCH17Rj0_inv7%3DXcS6SoR62Xg%40mail.gmail.com.

--

Bill Burke

Red Hat

[Georgios Andrianakis]

I completely agree with this: If there are good use cases for this kind of thing, it should be pursued. Furthermore it builds on the Vert.x HTTP Proxy which we can bend to our own will should the need arise

[Thomas SEGISMONT]

Cool stuff Georgios!

I assume this can't be a Quarkiverse extension because it requires access to the REST Client setup code. Correct?

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CALeTM-k3Dvg6GX5n9a70aNuS9HMxk%3DNN87vr%3DyUyaVoUwiXgwA%40mail.gmail.com.

[Georgios Andrianakis]

On Thu, Jul 11, 2024 at 11:37 AM Thomas SEGISMONT <tsegi...@gmail.com> wrote:

Cool stuff Georgios!

Thanks!

I assume this can't be a Quarkiverse extension because it requires access to the REST Client setup code. Correct?

That's an interesting question. I can probably split this up some more so that the foundational parts are in the quarkus-rest-client extension but allowing for Quarkiverse extensions to hook into this feature and implement their own proxy

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CACiEr_Qq8MLgWwMnkm_6qF3x_aYGTmVcZTkjYZRWSwvuWBBkdA%40mail.gmail.com.

[Thomas Darimont]

Cool stuff indeed!

Another use case for this could be fault injection for chaos engineering - you could use this as a wrapper for tools like https://chaostoolkit.org/drivers/toxiproxy/ to inject faults like additional latency, invalid headers etc. into the request processing.

[Georgios Andrianakis]

Oh, that's a very intriguing idea, thanks!

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/182bbad6-273f-44da-9364-60895cff67b1n%40googlegroups.com.

[Holly Cummins]

That's an interesting question. I can probably split this up some more so that the foundational parts are in the quarkus-rest-client extension but allowing for Quarkiverse extensions to hook into this feature and implement their own proxy 

+1 for this, it would be a sweet little capability to offer to extensions. 

[Georgios Andrianakis]

On Thu, Jul 11, 2024 at 1:16 PM Holly Cummins <hcum...@redhat.com> wrote:

That's an interesting question. I can probably split this up some more so that the foundational parts are in the quarkus-rest-client extension but allowing for Quarkiverse extensions to hook into this feature and implement their own proxy 

+1 for this, it would be a sweet little capability to offer to extensions. 

This will soon be ready :)

[Max Rydahl Andersen]

toxiproxy usecase could be done at vert.x http level could it not?

I'm still blind to why this has to be defined at restclient level ;)

/max

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CALeTM-nCyYfuJVCeb0tFHVoLYXN0faopgGUgVUrp_BjnQfVHdA%40mail.gmail.com.

[Sergey Beryozkin]

Hey Georgios

On Wed, Jul 10, 2024 at 11:43 AM Georgios Andrianakis <gand...@redhat.com> wrote:

On Wed, Jul 10, 2024 at 1:38 PM Max Rydahl Andersen <mand...@redhat.com> wrote:

cool stuff - reminds me of https://httptoolkit.com/ ; but just only for resteasy client.

My main questions are security concerned - as this is effectively disabling or rather circumventing https.

For what I had in mind, that's the whole point

 

This is a localhost server which acts as a proxy, right ? But it may make sense to harden it a little bit and enforce the same CORS policy that the DevUI server does, that 3rd party origin requests to it get blocked, only requests made from local host to it are allowed. It can mitigate some concerns, as otherwise, even if it is an opt-in, once it is enabled, there is a window on the local host into what is meant to be a secure HTTPS target, and then it is not too hard to imagine situations, even if somewhat theoretical, that one downloads some advanced Wireshark from not the main Wireshark site, which will do something bad when tracing etc. A copy and paste of https://github.com/quarkusio/quarkus/blob/main/extensions/vertx-http/runtime/src/main/java/io/quarkus/devui/runtime/LocalHostOnlyFilter.java at some point can put a lot of concerns to bed.

 

this is devtime only, right? so no way by accidentally expose it in prod?

No way to enable it in prod

is it intended to be something you enable per restclient and not just globally?

Per client

 

also...would it make more sense to have the proxy devservice something that is part of vertx extension and then
any http based client (even raw vert.x) can reroute? i.e. why need for special restclient reactive setup?

The problem is that with anything not managed (Vertx HTTP Client, JDK HTTP Client, Jakarta REST Client) you don't know what the URL is.

With the REST Client you already have all the information available to make this a one flag opt-in process

Hopefully, once you enable an SPI into it, extensions which use Vert.x WebClient can tap into it, like others said, it would be great, I can think of a few extensions which may benefit :-)

Thanks, Sergey

 

/max

On 10 Jul 2024, at 9:33, Georgios Andrianakis wrote:

Hi folks,

I wanted to share a demo of a little something I did.

With a few changes to the Quarkus REST Client, my change allows Quarkus to automatically start an HTTP proxy which is used for all the calls for that REST Client.

The reason I did this was that I wanted to capture network traffic between a Quarkus application and a downstream service that was using HTTPS - doing this under normal circumstances requires some heroics, however if we insert a proxy in the middle, the capture is trivial, as shown in the video.

Take a look if you are interested and let me know if you think this is something that makes sense to have or if you think of other use cases where this approach could be beneficial.

Thanks

--

Georgios Andrianakis

Independent Contractor

--
You received this message because you are subscribed to the Google Groups "Quarkus Development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CALeTM-%3Dc4iVw_izTGYjV3TOPsjefAzQ%2BKr06TVO3b6EO-7kzZQ%40mail.gmail.com.

--

Georgios Andrianakis

Independent Contractor

--
You received this message because you are subscribed to the Google Groups "Quarkus Development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CALeTM-m00%2BsHvPK4oBUEiJid7PZ4XhQ5ydvORtPEDZS00WNZCA%40mail.gmail.com.

[Georgios Andrianakis]

The PR has been overhauled and taken out of draft. 

The description on it hopefully provides a good idea of how advanced integrations can use it

[Georgios Andrianakis]

On Thu, Jul 11, 2024 at 3:28 PM Sergey Beryozkin <sbia...@redhat.com> wrote:

Hey Georgios

On Wed, Jul 10, 2024 at 11:43 AM Georgios Andrianakis <gand...@redhat.com> wrote:

On Wed, Jul 10, 2024 at 1:38 PM Max Rydahl Andersen <mand...@redhat.com> wrote:

cool stuff - reminds me of https://httptoolkit.com/ ; but just only for resteasy client.

My main questions are security concerned - as this is effectively disabling or rather circumventing https.

For what I had in mind, that's the whole point

 

This is a localhost server which acts as a proxy, right ? But it may make sense to harden it a little bit and enforce the same CORS policy that the DevUI server does, that 3rd party origin requests to it get blocked, only requests made from local host to it are allowed. It can mitigate some concerns, as otherwise, even if it is an opt-in, once it is enabled, there is a window on the local host into what is meant to be a secure HTTPS target, and then it is not too hard to imagine situations, even if somewhat theoretical, that one downloads some advanced Wireshark from not the main Wireshark site, which will do something bad when tracing etc. A copy and paste of https://github.com/quarkusio/quarkus/blob/main/extensions/vertx-http/runtime/src/main/java/io/quarkus/devui/runtime/LocalHostOnlyFilter.java at some point can put a lot of concerns to bed.

 

this is devtime only, right? so no way by accidentally expose it in prod?

No way to enable it in prod

is it intended to be something you enable per restclient and not just globally?

Per client

 

also...would it make more sense to have the proxy devservice something that is part of vertx extension and then
any http based client (even raw vert.x) can reroute? i.e. why need for special restclient reactive setup?

The problem is that with anything not managed (Vertx HTTP Client, JDK HTTP Client, Jakarta REST Client) you don't know what the URL is.

With the REST Client you already have all the information available to make this a one flag opt-in process

Hopefully, once you enable an SPI into it, extensions which use Vert.x WebClient can tap into it, like others said, it would be great, I can think of a few extensions which may benefit :-)

This is very unlikely to happen as I mentioned to Max

[Sergey Beryozkin]

On Thu, Jul 11, 2024 at 1:29 PM Georgios Andrianakis <gand...@redhat.com> wrote:

On Thu, Jul 11, 2024 at 3:28 PM Sergey Beryozkin <sbia...@redhat.com> wrote:

Hey Georgios

On Wed, Jul 10, 2024 at 11:43 AM Georgios Andrianakis <gand...@redhat.com> wrote:

On Wed, Jul 10, 2024 at 1:38 PM Max Rydahl Andersen <mand...@redhat.com> wrote:

cool stuff - reminds me of https://httptoolkit.com/ ; but just only for resteasy client.

My main questions are security concerned - as this is effectively disabling or rather circumventing https.

For what I had in mind, that's the whole point

 

This is a localhost server which acts as a proxy, right ? But it may make sense to harden it a little bit and enforce the same CORS policy that the DevUI server does, that 3rd party origin requests to it get blocked, only requests made from local host to it are allowed. It can mitigate some concerns, as otherwise, even if it is an opt-in, once it is enabled, there is a window on the local host into what is meant to be a secure HTTPS target, and then it is not too hard to imagine situations, even if somewhat theoretical, that one downloads some advanced Wireshark from not the main Wireshark site, which will do something bad when tracing etc. A copy and paste of https://github.com/quarkusio/quarkus/blob/main/extensions/vertx-http/runtime/src/main/java/io/quarkus/devui/runtime/LocalHostOnlyFilter.java at some point can put a lot of concerns to bed.

 

this is devtime only, right? so no way by accidentally expose it in prod?

No way to enable it in prod

is it intended to be something you enable per restclient and not just globally?

Per client

 

also...would it make more sense to have the proxy devservice something that is part of vertx extension and then
any http based client (even raw vert.x) can reroute? i.e. why need for special restclient reactive setup?

The problem is that with anything not managed (Vertx HTTP Client, JDK HTTP Client, Jakarta REST Client) you don't know what the URL is.

With the REST Client you already have all the information available to make this a one flag opt-in process

Hopefully, once you enable an SPI into it, extensions which use Vert.x WebClient can tap into it, like others said, it would be great, I can think of a few extensions which may benefit :-)

This is very unlikely to happen as I mentioned to Max

I was thinking about SPI providing a full target URL, etc. But I guess I don't appreciate the whole complexity involved into it 🙂

[Georgios Andrianakis]

On Thu, Jul 11, 2024 at 3:36 PM Sergey Beryozkin <sbia...@redhat.com> wrote:

On Thu, Jul 11, 2024 at 1:29 PM Georgios Andrianakis <gand...@redhat.com> wrote:

On Thu, Jul 11, 2024 at 3:28 PM Sergey Beryozkin <sbia...@redhat.com> wrote:

Hey Georgios

On Wed, Jul 10, 2024 at 11:43 AM Georgios Andrianakis <gand...@redhat.com> wrote:

On Wed, Jul 10, 2024 at 1:38 PM Max Rydahl Andersen <mand...@redhat.com> wrote:

cool stuff - reminds me of https://httptoolkit.com/ ; but just only for resteasy client.

My main questions are security concerned - as this is effectively disabling or rather circumventing https.

For what I had in mind, that's the whole point

 

This is a localhost server which acts as a proxy, right ? But it may make sense to harden it a little bit and enforce the same CORS policy that the DevUI server does, that 3rd party origin requests to it get blocked, only requests made from local host to it are allowed. It can mitigate some concerns, as otherwise, even if it is an opt-in, once it is enabled, there is a window on the local host into what is meant to be a secure HTTPS target, and then it is not too hard to imagine situations, even if somewhat theoretical, that one downloads some advanced Wireshark from not the main Wireshark site, which will do something bad when tracing etc. A copy and paste of https://github.com/quarkusio/quarkus/blob/main/extensions/vertx-http/runtime/src/main/java/io/quarkus/devui/runtime/LocalHostOnlyFilter.java at some point can put a lot of concerns to bed.

 

this is devtime only, right? so no way by accidentally expose it in prod?

No way to enable it in prod

is it intended to be something you enable per restclient and not just globally?

Per client

 

also...would it make more sense to have the proxy devservice something that is part of vertx extension and then
any http based client (even raw vert.x) can reroute? i.e. why need for special restclient reactive setup?

The problem is that with anything not managed (Vertx HTTP Client, JDK HTTP Client, Jakarta REST Client) you don't know what the URL is.

With the REST Client you already have all the information available to make this a one flag opt-in process

Hopefully, once you enable an SPI into it, extensions which use Vert.x WebClient can tap into it, like others said, it would be great, I can think of a few extensions which may benefit :-)

This is very unlikely to happen as I mentioned to Max

I was thinking about SPI providing a full target URL, etc. But I guess I don't appreciate the whole complexity involved into it 🙂

If we have a way of knowing the URL (which I can't think of), it might be possible

[Stephane Epardaud]

This looks great, but if the use-case is really to be able to see traffic, and given that I find wireshark confusing and complex, I am not sure this is the best way to do it.

A while ago I filed https://github.com/quarkusio/quarkus/issues/22213 and I still think that I'd prefer if a single config property would allow me to dump every http request (client and server) to a text file. Or, frankly, better yet, to some place in the DEV UI that looked like the browser's dev tools' Network panel.

It could be done via your proxy, or via hooking up to Vert.x to dump traffic. I suppose it doesn't really matter how we obtain the traffic, but IMO what's important is how we present it to users. And I get confused by wireshark every single time. Mostly because the request and response are in different places, and there's always protocols I do not care about.

Not to mention that it'd be even more amazing if we could log other kinds of traffic, such as SQL requests/responses, or graphql, or protobuf, in the same place using the same UI. Now, _that_ is something that would be jaw-dropping for demos.

--
You received this message because you are subscribed to the Google Groups "Quarkus Development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CALeTM-%3Dc4iVw_izTGYjV3TOPsjefAzQ%2BKr06TVO3b6EO-7kzZQ%40mail.gmail.com.

--

Stéphane Épardaud

[Paul Robinson]

Is this something that's already captured by traces or could be if they were enabled? If so, could we build on that to give users the nice view you propose?

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CAKU9E9udEy6M%3DyXq7_z4BbuAdZJ_JiJj7W8Yxq-2ZE-q0mO%2BQg%40mail.gmail.com.

--

Paul Robinson

Senior Engineering Manager

MicroProfile & Quarkus

He/Him/His

Red Hat

Red Hat respects your work life balance. Therefore there is no need to answer this email out of your office hours.

[Georgios Andrianakis]

On Fri, Jul 12, 2024 at 11:33 AM Stephane Epardaud <stephane...@gmail.com> wrote:

This looks great, but if the use-case is really to be able to see traffic, and given that I find wireshark confusing and complex, I am not sure this is the best way to do it.

A while ago I filed https://github.com/quarkusio/quarkus/issues/22213 and I still think that I'd prefer if a single config property would allow me to dump every http request (client and server) to a text file. Or, frankly, better yet, to some place in the DEV UI that looked like the browser's dev tools' Network panel.

This is not the same thing - this is for the client which is talking to other services

It could be done via your proxy, or via hooking up to Vert.x to dump traffic. I suppose it doesn't really matter how we obtain the traffic, but IMO what's important is how we present it to users. And I get confused by wireshark every single time. Mostly because the request and response are in different places, and there's always protocols I do not care about.

Not to mention that it'd be even more amazing if we could log other kinds of traffic, such as SQL requests/responses, or graphql, or protobuf, in the same place using the same UI. Now, _that_ is something that would be jaw-dropping for demos.

If Vert.x provides TCP proxy, we could stand it up and hook it into other managed clients

On Wed, 10 Jul 2024 at 09:33, Georgios Andrianakis <gand...@redhat.com> wrote:

Hi folks,

I wanted to share a demo of a little something I did.

With a few changes to the Quarkus REST Client, my change allows Quarkus to automatically start an HTTP proxy which is used for all the calls for that REST Client.

The reason I did this was that I wanted to capture network traffic between a Quarkus application and a downstream service that was using HTTPS - doing this under normal circumstances requires some heroics, however if we insert a proxy in the middle, the capture is trivial, as shown in the video.

Take a look if you are interested and let me know if you think this is something that makes sense to have or if you think of other use cases where this approach could be beneficial.

Thanks

--

Georgios Andrianakis

Independent Contractor

--
You received this message because you are subscribed to the Google Groups "Quarkus Development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CALeTM-%3Dc4iVw_izTGYjV3TOPsjefAzQ%2BKr06TVO3b6EO-7kzZQ%40mail.gmail.com.

--

Stéphane Épardaud

[Georgios Andrianakis]

On Fri, Jul 12, 2024 at 11:37 AM Paul Robinson <prob...@redhat.com> wrote:

Is this something that's already captured by traces or could be if they were enabled? If so, could we build on that to give users the nice view you propose?

Wed don't have anything that captures traffic currently

[Stephane Epardaud]

On Fri, 12 Jul 2024 at 11:38, Georgios Andrianakis <gand...@redhat.com> wrote:

This is not the same thing - this is for the client which is talking to other services

My issue was about the client and the server both.

 

If Vert.x provides TCP proxy, we could stand it up and hook it into other managed clients

Well, yes… but I don't want TCP-level logs, I want HTTP-equivalent layer logs.