Simplifying streaming of reactive messaging channel

[Georgios Andrianakis]

Hi folks,

A few of us have been thinking about how to improve the integration of RESTEasy Reactive with Reactive Messaging.

Currently when we a Channel needs to be streamed, our documentation proposes this:

@Path("/quotes")
public class QuotesResource {

    @Channel("quotes")
    Multi<Quote> quotes;

    @GET
    @Produces(MediaType.SERVER_SENT_EVENTS)
    public Multi<Quote> stream() {
        return quotes;
    }
}

Although this is simple, it feels a little too ceremonious and can likely be improved.

One thought is allowing something like this:

@Path("/quotes")
public class QuotesResource {

    @GET
    @Produces(MediaType.SERVER_SENT_EVENTS)
    public native Multi<Quote> stream();
}

Note the use of native here, which is the only modifier that allows us to not have to supply an implementation of the method without requiring the class to be abstract.

Also note that the following solution would be nice but does not work:

@Path("/quotes")
public class QuotesResource {
    @Produces(MediaType.SERVER_SENT_EVENT)
    @Channel("quotes")
    Multi<Quote> quotes
}

The reason is that JAX-RS 's @Produces cannot be used on a field.

What do you think of the proposed solution? Any other options that come to mind?

[Loïc MATHIEU]

I would not use native as it have a strong meaning of platform dependent code (and usually C/C++), see its definition https://docs.oracle.com/javase/specs/jls/se8/html/jls-8.html#jls-8.4.3.4

On your example on native, you miss the @Channel annotation right ?

We can use an abstract method instead, but this would need the classes will be an abstract one, not sure it's a better design as develop would be a little surprised by it

@Path("/quotes")
public abstract class QuotesResource {

    @GET
    @Produces(MediaType.SERVER_SENT_EVENTS)
    @Channel("quotes")
    public abstract Multi<Quote> stream();
}

Maybe we can have an API for this purpose

@Path("/quotes")
public class QuotesResource {

    @GET
    @Produces(MediaType.SERVER_SENT_EVENTS)
    public Multi<Quote> stream() {
        return Channel.of("quotes", Quote.class);
  }
}
Note that the opposite can also be improved, it's usually written as

@Channel("quotes") Emitter<Quote> emitter;

@POST
@Consumes(MediaType.APPLICATION_JSON)
public void receive(Quote quote){

    emitter.send(quote);
}

--
You received this message because you are subscribed to the Google Groups "Quarkus Development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CALeTM-mQ7-e-d-vRXU%2BhmMOQWcLiZoD_%3DZJrfBdvi4Q2AjRBbg%40mail.gmail.com.

[Georgios Andrianakis]

On Tue, Jul 13, 2021, 18:43 Loïc MATHIEU <loik...@gmail.com> wrote:

I would not use native as it have a strong meaning of platform dependent code (and usually C/C++), see its definition https://docs.oracle.com/javase/specs/jls/se8/html/jls-8.html#jls-8.4.3.4

Yeah, we know it's a contentious point 😉. But we use it in Panache as well, don't we?

On your example on native, you miss the @Channel annotation right ?

Correct, good catch.

We can use an abstract method instead, but this would need the classes will be an abstract one, not sure it's a better design as develop would be a little surprised by it

Yeah I thinn we don't want to go down that route, it's to much for a user to do.

@Path("/quotes")
public abstract class QuotesResource {

    @GET
    @Produces(MediaType.SERVER_SENT_EVENTS)
    @Channel("quotes")
    public abstract Multi<Quote> stream();
}

Maybe we can have an API for this purpose

@Path("/quotes")
public class QuotesResource {

    @GET
    @Produces(MediaType.SERVER_SENT_EVENTS)
    public Multi<Quote> stream() {
        return Channel.of("quotes", Quote.class);
  }
}

Is this really an improvement over what we already have?

[George Gastaldi]

Can the QuotesResource class be an interface (similar to the CRUD endpoints) and then we auto-generate the boring boiling code?

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CALeTM-mdppQmjiY1qPsEuLs%2BYWkG5j%2BAZhZ6%2Bcb1HTcTaos3hw%40mail.gmail.com.

[Georgios Andrianakis]

On Tue, Jul 13, 2021, 18:50 George Gastaldi <ggas...@redhat.com> wrote:

Can the QuotesResource class be an interface (similar to the CRUD endpoints) and then we auto-generate the boring boiling code?

Not if we want users to be able to continue using JAX-RS as they know it - I mean they should be able to add other JAX-RS methods to these classes.

[Loïc MATHIEU]

Le mar. 13 juil. 2021 à 17:50, George Gastaldi <ggas...@redhat.com> a écrit :

Can the QuotesResource class be an interface (similar to the CRUD endpoints) and then we auto-generate the boring boiling code?

This is a different story because usually the example from Georgio and my addition with the Emitter are used to bridge the gap between messaging and REST endpoint, so you'll have more operations in the same entry point that deals with the Quote object but not the stream of messages.

But the idea to transform a stream of messages to at least GET/POST automatically as we did for CRUD JPA is interesting also, we just need a way to express that object send to the POST method will go to this channel and when calling the GET method you will read from the stream from this topic.

[William Burke]

IMO, bad idea on using native.  You shouldn't be overriding a core JDK construct for a tiny bit more productivity, especially for an edge case like this.

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CALeTM-n5JsS%3DQdyOoPcd1gv-TY7M4CYtVTQHDyUvsPFwZFPJyg%40mail.gmail.com.

--

Bill Burke

Red Hat

[Georgios Andrianakis]

Yeah, that's a fair point, one we did indeed anticipate 

[Stuart Douglas]

Is this really worth the effort? How common is it to directly stream messages as SSE events?

Stuart

[Georgios Andrianakis]

It's certainly debatable

[Ladislav Thon]

Dne út 13. 7. 2021 17:21 uživatel Georgios Andrianakis <gand...@redhat.com> napsal:

Hi folks,

A few of us have been thinking about how to improve the integration of RESTEasy Reactive with Reactive Messaging.

Currently when we a Channel needs to be streamed, our documentation proposes this:

@Path("/quotes")
public class QuotesResource {

    @Channel("quotes")
    Multi<Quote> quotes;

    @GET
    @Produces(MediaType.SERVER_SENT_EVENTS)
    public Multi<Quote> stream() {
        return quotes;
    }
}

Although this is simple, it feels a little too ceremonious

Seriously?

I mean, this is simplicity distilled. In my eyes, removing from this would be actively adding complexity.

Also the practical benefit is virtually zero, I'd say. The moment you need to do anything more, such as transform the data in some way, or send the client some initial data at the time of subscription, you need to go back to this.

Frankly, I'm not a fan of shortcuts that stop working when you need anything custom, and you have to go back to the non-shortcut way, and you have to _know_ there's a non-shortcut way.  And here, it isn't even a shortcut, it's almost as long as the original.

LT

and can likely be improved.

One thought is allowing something like this:

@Path("/quotes")
public class QuotesResource {

    @GET
    @Produces(MediaType.SERVER_SENT_EVENTS)
    public native Multi<Quote> stream();
}

Note the use of native here, which is the only modifier that allows us to not have to supply an implementation of the method without requiring the class to be abstract.

Also note that the following solution would be nice but does not work:

@Path("/quotes")
public class QuotesResource {
    @Produces(MediaType.SERVER_SENT_EVENT)
    @Channel("quotes")
    Multi<Quote> quotes
}

The reason is that JAX-RS 's @Produces cannot be used on a field.

What do you think of the proposed solution? Any other options that come to mind?

[Georgios Andrianakis]

On Wed, Jul 14, 2021 at 9:59 AM Ladislav Thon <lad...@gmail.com> wrote:

Dne út 13. 7. 2021 17:21 uživatel Georgios Andrianakis <gand...@redhat.com> napsal:

Hi folks,

A few of us have been thinking about how to improve the integration of RESTEasy Reactive with Reactive Messaging.

Currently when we a Channel needs to be streamed, our documentation proposes this:

@Path("/quotes")
public class QuotesResource {

    @Channel("quotes")
    Multi<Quote> quotes;

    @GET
    @Produces(MediaType.SERVER_SENT_EVENTS)
    public Multi<Quote> stream() {
        return quotes;
    }
}

Although this is simple, it feels a little too ceremonious

Seriously?

Seriously.

It's not like I enjoy writing emails. If I thought what I saw made perfect sense, I'd leave it at that

I mean, this is simplicity distilled. In my eyes, removing from this would be actively adding complexity.

Also the practical benefit is virtually zero, I'd say. The moment you need to do anything more, such as transform the data in some way, or send the client some initial data at the time of subscription, you need to go back to this.

Indeed, but Clement says that most processing will happen at the processor level anyway.

Frankly, I'm not a fan of shortcuts that stop working when you need anything custom, and you have to go back to the non-shortcut way, and you have to _know_ there's a non-shortcut way.  And here, it isn't even a shortcut, it's almost as long as the original.

Yeah, that's understandable. 

Given that most people seem to be happy with the state of things, it probably doesn't make sense to pursue this any further

[Ladislav Thon]

Dne st 14. 7. 2021 9:02 uživatel Georgios Andrianakis <gand...@redhat.com> napsal:

On Wed, Jul 14, 2021 at 9:59 AM Ladislav Thon <lad...@gmail.com> wrote:

Dne út 13. 7. 2021 17:21 uživatel Georgios Andrianakis <gand...@redhat.com> napsal:

Hi folks,

A few of us have been thinking about how to improve the integration of RESTEasy Reactive with Reactive Messaging.

Currently when we a Channel needs to be streamed, our documentation proposes this:

@Path("/quotes")
public class QuotesResource {

    @Channel("quotes")
    Multi<Quote> quotes;

    @GET
    @Produces(MediaType.SERVER_SENT_EVENTS)
    public Multi<Quote> stream() {
        return quotes;
    }
}

Although this is simple, it feels a little too ceremonious

Seriously?

Seriously.

It's not like I enjoy writing emails. If I thought what I saw made perfect sense, I'd leave it at that

Fair enough, though not everything we write has to be serious :-) The line between useless ceremony and expressing the essentials may be thin, but it's there.

I mean, this is simplicity distilled. In my eyes, removing from this would be actively adding complexity.

Also the practical benefit is virtually zero, I'd say. The moment you need to do anything more, such as transform the data in some way, or send the client some initial data at the time of subscription, you need to go back to this.

Indeed, but Clement says that most processing will happen at the processor level anyway.

I'm thinking presentation-level transformations. It's been a while since I worked on something actually end-user-facing, but I still remember that system of record data and data presented to users or other systems are often almost-the-same-but-not-quite. It would surely be possible to do such transformations on an in-memory channel, but I'd personally do them in the JAX-RS resource method, it just makes more sense to me.

Ah, one more thing. Don't we have an HTTP connector for Reactive Messaging in Quarkus? I think it allows both producing and consuming messages, though not sure how the consumption stuff works. Maybe for the super-simple cases, that would be easier?

LT

[Georgios Andrianakis]

On Wed, Jul 14, 2021 at 11:55 AM Ladislav Thon <lad...@gmail.com> wrote:

Dne st 14. 7. 2021 9:02 uživatel Georgios Andrianakis <gand...@redhat.com> napsal:

On Wed, Jul 14, 2021 at 9:59 AM Ladislav Thon <lad...@gmail.com> wrote:

Dne út 13. 7. 2021 17:21 uživatel Georgios Andrianakis <gand...@redhat.com> napsal:

Hi folks,

A few of us have been thinking about how to improve the integration of RESTEasy Reactive with Reactive Messaging.

Currently when we a Channel needs to be streamed, our documentation proposes this:

@Path("/quotes")
public class QuotesResource {

    @Channel("quotes")
    Multi<Quote> quotes;

    @GET
    @Produces(MediaType.SERVER_SENT_EVENTS)
    public Multi<Quote> stream() {
        return quotes;
    }
}

Although this is simple, it feels a little too ceremonious

Seriously?

Seriously.

It's not like I enjoy writing emails. If I thought what I saw made perfect sense, I'd leave it at that

Fair enough, though not everything we write has to be serious :-) The line between useless ceremony and expressing the essentials may be thin, but it's there.

Yeah, hence the conversation as opposed to me just going out and implementing it :) 

I mean, this is simplicity distilled. In my eyes, removing from this would be actively adding complexity.

Also the practical benefit is virtually zero, I'd say. The moment you need to do anything more, such as transform the data in some way, or send the client some initial data at the time of subscription, you need to go back to this.

Indeed, but Clement says that most processing will happen at the processor level anyway.

I'm thinking presentation-level transformations. It's been a while since I worked on something actually end-user-facing, but I still remember that system of record data and data presented to users or other systems are often almost-the-same-but-not-quite. It would surely be possible to do such transformations on an in-memory channel, but I'd personally do them in the JAX-RS resource method, it just makes more sense to me.

Ah, one more thing. Don't we have an HTTP connector for Reactive Messaging in Quarkus? I think it allows both producing and consuming messages, though not sure how the consumption stuff works. Maybe for the super-simple cases, that would be easier?

I had this question myself, but as far as I could gather from the Reactive Messaging docs, the http connector is simply a sink for sending messages, it's not for exposing the contents of a channel. If we do have what you say, then it would be nice to document it.

[Ladislav Thon]

Dne st 14. 7. 2021 11:01 uživatel Georgios Andrianakis <gand...@redhat.com> napsal:

Ah I misremembered it again! Yeah, it can receive messages on an HTTP endpoint (when acting as an HTTP server), and it can push messages to another HTTP endpoint (when acting as an HTTP client), but it can't do the opposite (in both directions). Sorry for the noise.

(Well maybe we could add that? Not sure if that's necessary though.)

LT

[clement escoffier]

Hello,

To simplify the streaming, we can add support for something like this:

@Channel("...")

@SSEStream (pingValue="{}", pingPeriod=Duration.ofSeconds(10), path="/stream")

Multi<Quote> quotes;

This would be a field in a REST endpoint. 

- Path should be optional (inheriting the path from the resource, or appending if set)

- the pingValue and pingPeriod is where things get fun. Most proxies and routers cut SSE connection after a few seconds of inactivity.  That’s why when using this feature you often need to send periodic “ping” values to keep the connection opened. However, it requires the client side to be aware of the value and handle these ping events (typically, it should detect and drop them).

The previous code could generate the following method:

    @GET

    @Produces(MediaType.SERVER_SENT_EVENTS)

    @Path(“/stream”)

    public Multi<String> quotes() {

        Multi<String> pings = Multi.createFrom().ticks().every(Duration.ofSeconds(10));

        return Multi.createBy().merging()

                .streams(

                        quotes.map(q -> toJson(q)),

                        pings

                );

    }

Clement

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CALbocOmJZgg5OdPBcoJLHw4PmMU3UtHO_MkTonOxSyUf_gevNg%40mail.gmail.com.

[Georgios Andrianakis]

I personally like this suggestion.

My only concern is what happens if users try to use the field, but it should be fine

[Stuart Douglas]

I guess my main question is, how common is this pattern going to be in an application? How many SSE streams backed by reactive messaging are we expecting (and also is SSE even that popular these days compared to websockets)?

If there are only going to be a couple of these per app at most I really don't think special syntax is warranted. It's one more thing for users to learn, and makes it harder for people to read the code (as if you are not familiar with the special syntax you need to go look it up).

I am not really convinced that this is worth it.

Stuart

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CAKW6fidjkQK1izDyLVC2c464WixSNMvvYX-MrU1vhmDCMcWqyA%40mail.gmail.com.

[William Burke]

+1000 to a WebSocket and SSE connector for Reactive Messaging over any other suggestion posted here.

 

[Michał Szynkiewicz]

The quarkus-http connector (for reactive messaging) that was already mentioned can consume WebSockets and HTTP, and can feed a WebSocket or HTTP endpoint.

It's rather basic but it can be extended to also expose SSE and WebSocket endpoints.

That was the original plan but I never had time to do this. 

If we think it's worth it, I can get back to this.

--

You received this message because you are subscribed to the Google Groups "Quarkus Development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CAL%3DE%3DjTK_JY4vZo1WijkBW_CBsows-RaJN%3D%3DTQXafApi9qOZqA%40mail.gmail.com.

[Georgios Andrianakis]

On Mon, Jul 19, 2021 at 11:55 AM Michał Szynkiewicz <michal.l.s...@gmail.com> wrote:

The quarkus-http connector (for reactive messaging) that was already mentioned can consume WebSockets and HTTP, and can feed a WebSocket or HTTP endpoint.

It's rather basic but it can be extended to also expose SSE and WebSocket endpoints.

That was the original plan but I never had time to do this. 

If we think it's worth it, I can get back to this.

I personally think that a no-hassle way of exposing it makes sense

[Stephane Epardaud]

Yeah, any example of what it would look like?

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CALeTM-nfB%2BJx7fh94eFvH%2BaF9dhsQX8VUaoKGLBBLiEvE3O4Ug%40mail.gmail.com.

--

Stéphane Épardaud

[Michał Szynkiewicz]

I think it would have to be something along the lines of:

@Incoming("some-source-channel")
@Outgoing("my-http-sink")
SomeType passThroughWithCustomSerializer(SomeMaybeDifferentType foo) {
 ...
}

mp.messaging.outgoing.my-http-sink.connector=quarkus-http
mp.messaging.outgoing.my-http-sink.path=/my-http-sink
mp.messaging.outgoing.my-http-sink.method=POST

Two downsides of this vs exposing it via RR are

• the method and path are in the configuration and not in a java file
• converters. Quarkus-http connector can currently serialize JsonObject, JsonArray, Number, String, Buffer. Anything else it will try to convert to a JSON String. User can create a custom serializer but it's quakrus-http specific. RestEasy (Reactive) is more powerful here.

NOTE: the proposed configuration would probably mean that we need to change what we currently have a bit, right now configuration for feeding a remote http endpoint is really similar to that.

[Guillaume Smet]

I'm -1 on using native too. When I use it in Eclipse, I have all sorts of errors when refactoring things.

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CAL%3DE%3DjSrMv9TQLhFpJTgV6NDCNvqqeh8VB9aExm0iVdgT%2B1Ofg%40mail.gmail.com.

[Stephane Epardaud]

On Tue, 20 Jul 2021 at 11:50, Michał Szynkiewicz <michal.l.s...@gmail.com> wrote:

I think it would have to be something along the lines of:

@Incoming("some-source-channel")
@Outgoing("my-http-sink")
SomeType passThroughWithCustomSerializer(SomeMaybeDifferentType foo) {
 ...
}

mp.messaging.outgoing.my-http-sink.connector=quarkus-http
mp.messaging.outgoing.my-http-sink.path=/my-http-sink
mp.messaging.outgoing.my-http-sink.method=POST

This is really suboptimal.

So, in light of the discussion here, I think that the original example is actually alright:

@Path("/quotes")
public class QuotesResource {

    @Channel("quotes")
    Multi<Quote> quotes;

    @GET
    @Produces(MediaType.SERVER_SENT_EVENTS)
    public Multi<Quote> stream() {
        return quotes;
    }
}

Even though I'd simplify it to:

@Path("/quotes")
public class QuotesResource {

    @GET
    @Produces(MediaType.SERVER_SENT_EVENTS)
    public Multi<Quote> stream(@Channel("quotes") Multi<Quote> quotes) {

        return quotes;
    }
}

 If we can. Perhaps this needs https://github.com/quarkusio/quarkus/issues/13593 and we should Just Do It™.

This sort of method is decently minimal, because even though it's tempting to put the annotations on the field, since we're not actually doing anything with it:

@Path("/quotes")
public class QuotesResource {

    @GET
    @Produces(MediaType.SERVER_SENT_EVENTS)
    @Channel("quotes")

    Multi<Quote> quotes;

}

We have to acknowledge that this is probably an edge-case, and in most other cases we will want to do something to the channel data as we send it through, so perhaps this is a demo-feature that may not be actually useful, so methods will be required in most cases?

[Stuart Douglas]

I think that most of the time you want a method, and the extra complexity of supporting fields is not worth it (e.g. how do you do interceptors, security support etc).

Stuart

 

--
You received this message because you are subscribed to the Google Groups "Quarkus Development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CAKU9E9u1FBucdxotNh%3DpytgjNaBcQqz8ou-12NZW-ciWwHaXRQ%40mail.gmail.com.

[Stephane Epardaud]

Well, I agree most of the time we want a method, but as for the field support, we could just turn them into methods and call it a day, so all interceptors and such keep on working on the generated method. But I think it's probably not worth it at this point.

--

Stéphane Épardaud