Distributed Logs with Quarkus

[david evano]

Hello,

I'm using Quarkus in production.

There is 2 ways to communicate: API REST (between Front/Back) and Kafka (between Infra Microservices).

For API REST (mutiny) and Kafka (smallrye-reactive-messaging-kafka), we use Reactive model.

So I need to have a correlator Id (Jaeger) from a HTTP request to a Kafka record to be able to follow a request.

Problem: Both OpenTracing and OpenTelemetry extensions provided by Quarkus seems not able to manage Reactive model.

How can we do to have a correlator id (jaeger) with Quarlkus in reactive mode?

OpenTelemetry seems to be the future of distributed logs, so when is it planned to have 

distributed logs with Quarkus (Reactive mode)?

Which of OpenTracing and OpenTelemetry should be considered to have distributed logs with Quarkus (Reactive mode) quickly?

Thanks,

David

[Ken Finnigan]

David,

What do you mean by "manage Reactive model"?

For instance, with OpenTelemetry spans are created for @Incoming and @Outgoing processing with reactive messaging, but bridging between imperative and reactive, or vice versa, currently requires manual propagation of the span details. In the future, we intend to look at ways to improve it. This is what it would look like to go from a REST endpoint to reactive messaging with an Emitter:

@Inject
@Channel("names")
Emitter<String> nameEmitter;

@GET
@Path("greeting")
public Response greeting() {
    String name = nameService.name();

    nameEmitter.send(Message.of(name, getKafkaMetadata(Context.current())));

    return Response.ok()
            .entity(new Greeting(String.format("Hello %s", name)))
            .build();
}

private Metadata getKafkaMetadata(Context current) {
    return Metadata.of(TracingMetadata.withCurrent(current));
}

Notice we need to create TracingMetadata with the current OpenTelemetry context to propagate the trace on the Message.

Can you clarify what you mean by "distributed logs"? Is it the correlation id being present in logs or something else?

Either way, as OpenTracing is not under active development, we would likely focus any improvements towards OpenTelemetry.

Ken

--
You received this message because you are subscribed to the Google Groups "Quarkus Development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/dbaaa72e-8962-44a1-9f77-831a4414576cn%40googlegroups.com.

[david evano]

Hi Ken,

Thanks for your feedback.

So I have to focus to OpenTelemetry.

I would like be sure that OpenTelemetry extension is able to manage correlator id (spanId)

In the Quarkus documentation, it seems working fine with RestEasy and Kafka Client.

The minimal requirement to adopt OpenTelemetry is the following:

A microservice exposes an API REST endpoint /sayhello.

On request:

   -  process the request With ReastEasy Mutiny, get the Jaeger traceID

   - do some staff (it's seems that crete span is not yet available (https://github.com/quarkusio/quarkus/issues/18750)

   - produce to a Kafka topic with the current traceId (should be possible manually with getKafkaMetadata(Context.current())) 

   - calling REST Client Reactive with current traceId propagation (is it working?) 

So I would like to be sure that OpenTelemetry is a mature extension with Reactive (RestEasy+SmallRye Kafka).

Your feedback about adding manually  getKafkaMetadata(Context.current()) is interesting, 

What about ReastEasy Mutiny?

See below an example of code with endpoitn (RestEasy Mutiny and SmallRye Kafka)

Can you confirm that the traceId coming from the request /sayhello will be propagated to the kafka topic?

And when another microservice will consume this topic (with SmallRye Kafka), the traceId will be propagated?

Thanks,

David

[Ken Finnigan]

On Tue, Aug 24, 2021 at 1:59 PM david evano <david...@gmail.com> wrote:

Hi Ken,

Thanks for your feedback.

So I have to focus to OpenTelemetry.

I would like be sure that OpenTelemetry extension is able to manage correlator id (spanId)

In the Quarkus documentation, it seems working fine with RestEasy and Kafka Client.

The minimal requirement to adopt OpenTelemetry is the following:

A microservice exposes an API REST endpoint /sayhello.

On request:

   -  process the request With ReastEasy Mutiny, get the Jaeger traceID

I think this is there today. We have tests for RESTEasy Reactive and OpenTelemetry

   - do some staff (it's seems that crete span is not yet available (https://github.com/quarkusio/quarkus/issues/18750)

It's available, but a possible issue has been raised. I haven't had an opportunity to investigate it fully yet

 

   - produce to a Kafka topic with the current traceId (should be possible manually with getKafkaMetadata(Context.current())) 

Manual propagation is only needed when going from REST to Kafka, or vice versa. Kafka to Kafka propagates the span correctly

 

   - calling REST Client Reactive with current traceId propagation (is it working?) 

OpenTelemetry is integrated with the REST Client

 

So I would like to be sure that OpenTelemetry is a mature extension with Reactive (RestEasy+SmallRye Kafka).

Your feedback about adding manually  getKafkaMetadata(Context.current()) is interesting, 

What about ReastEasy Mutiny?

It should work as Mutiny just provides additional types, the main work is done with RESTEasy

 

See below an example of code with endpoitn (RestEasy Mutiny and SmallRye Kafka)

Can you confirm that the traceId coming from the request /sayhello will be propagated to the kafka topic?

I don't know what kafkaService is so it's hard to say. If it's a REST Client interface, then yes the trace is automatically propagated.

If it's sending a message to Kafka, then it's likely manual propagation is needed

 

And when another microservice will consume this topic (with SmallRye Kafka), the traceId will be propagated?

If you have a consumer with @Incoming, then the trace will be on the incoming RecordHeaders from Kafka

 

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/cd3ec5de-bdbd-4cb3-974e-7dfa1cfa9007n%40googlegroups.com.

[david evano]

So I will check with Resteasy Mutiny to confirm.

Thanks, 

David