Jaeger Performance Tests

[vteja...@gmail.com]

Hello,

We are trying to use jaeger in our application, but we want to know about jaeger performance & behavior with different sampling rates, different number of spans. 

• How would jaeger measure it? It would be good if you could share us any release note with these stuff or some tool or test tools used in git hub

            Intention is to understand and make sure it is good enough and doesn't consume too much compute resources

• Also is there is any formula or trend observation with number of spans, sampling rate w.r.t cpu & memory for agent and collector?
  
  
• How much memory one trace point/span consume?

I also believe these values are independent of instrumented language.

BR,

Teja

[Juraci Paixão Kröhling]

Hello Teja,

Kevin is working on some performance tests and storing the code here:
https://github.com/jaegertracing/jaeger-performance

The idea is to get "some" numbers and, more importantly, make sure we
don't have regressions when we release a new version. If you want to
collaborate, do get in touch with him.

That said, "performance" is something really broad and quite dependent
on specific deployments. It's unlikely that the scenarios from the code
above would cover your use case.

Here are some leading questions, just to give an idea how your usage
might influence the performance:

* Are you adding tags/baggage items/logs to your spans?
* Are you making your client connect directly to the collector behind
an auth proxy, or is it sending spans to the Agent via UDP?
* Is everything on bare metal, or are you using tools like Kubernetes?
* Do you intend to use Cassandra or Elasticsearch?

In the end, I'd say that you should really be measuring your own
environment and make adjustments as you go. If you have a high traffic
scenario, start sampling a small percentage of requests and increase it
bit by bit. You can do that by adjusting env vars if your target
application is using the Go or Java clients.

- Juca.

> --
> You received this message because you are subscribed to the Google
> Groups "Jaeger Tracing" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to jaeger-tracin...@googlegroups.com.
> To post to this group, send email to jaeger-...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/jaeger-tracing/a2c9d001-e6a5-45af-a6e4-
> 9b0190e96ebf%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[vteja...@gmail.com]

Hi Juca,

Thanks for the answer.

Have to think about contributing back to jaeger :)

I am looking for a trend and approximate value for a particular client language in K8s. But its okay, I completely agree to your statements stated down!

But a quick question, will jaeger publish these test results somewhere? It would be good if it can publish its cpu, memory, storage, query latency's for every major release atleast.

BR,

Teja

[Juraci Paixão Kröhling]

On Wed, 2018-04-11 at 23:01 -0700, vteja...@gmail.com wrote:
>
> But a quick question, will jaeger publish these test results
> somewhere? It would be good if it can publish its cpu, memory,
> storage, query latency's for every major release atleast.
>

I don't think we planned on how to make it public yet, but we have no
intention in keeping it private. Perhaps we could try to convince Kevin
to write a blog on this topic once it's done ;)

- Juca.

[Pandiaraj A]

Hi juca, 

I m using the performance testing tool with insecure endpoint.

Is there any capability to test with secure endpoint?

I had tested with my secure collector endpoint in the testing tool by adding self signed certificate, but trace/spans are not flowing through the endpoint.

what I'm missing in the tool, please provide the steps to test the secure endpoint in the performance tool.

Thanks,

Pandya.

[Jeeva Kandasamy]

Hi Pandya,

Can you give the URL of the performance tool repository?

If you are talking about jaegertracing/jaeger-performance [1]

* We have not tested yet with https collector port.

* Feel free to submit a PR if changes are required.

[1] - https://github.com/jaegertracing/jaeger-performance

Thanks,

- Jeeva

[Pandiaraj A]

Hi Jeeva,

Thanks for the response.

Using the jaeger performance tool https://github.com/jaegertracing/jaeger-performance

Tested my own application with secured collector endpoint, it working , can able to send traces. But I can't able to test it with bulk load. Attached the screenshot.

Same way, I'm trying in jaeger performance tool using secured endpoint, Collector endpoint  accepted my .pem / .crt file. But span/trace is not flowing through it.

 private JaegerTracer createJaegerTracer(String serviceName) {

        Sender sender;

        if (config.getSender().equalsIgnoreCase("udp")) {

            sender = new UdpSender(

                    config.getJaegerAgentHost(),

                    config.getJaegerAgentPort(),

                    config.getJaegerMaxPocketSize());

            logger.info("Using UDP sender, sending to: {}:{}",

                    config.getJaegerAgentHost(), config.getJaegerAgentPort());

        } else {

            // use the collector

        

        String certificatesTrustStorePath = "/Library/Java/JavaVirtualMachines/************/Contents/lib/security/cacerts";

        System.setProperty("javax.net.ssl.trustStore", certificatesTrustStorePath);

        

            String httpEndpoint = "http://" + config.getJaegerCollectorHost() + ":" + config.getJaegerCollectorPort()

                    + "/api/traces";

            logger.info("Using HTTP sender, sending to endpoint: {}", httpEndpoint);

            sender = new HttpSender.Builder(httpEndpoint).build();

        }

      logger.info("Flush interval {}, queue size {}", config.getJaegerFlushInterval(),

                config.getJaegerMaxQueueSize());

        RemoteReporter reporter = new RemoteReporter.Builder()

                .withSender(sender)

                .withMaxQueueSize(config.getJaegerMaxQueueSize())

                .withFlushInterval(config.getJaegerFlushInterval())

                

                .build();

        return new JaegerTracer.Builder(serviceName)

                .withReporter(reporter)

                .withSampler(new ConstSampler(true))

                .build();

    }

As mentioned above code,  had configured my certs.

Guide me Where/What I am missing.

Thanks & Regards,

Pandya.

[Pandiaraj A]

Hi team,

Remainder for the below query.

can I get a help?

Thanks,

Pandya.

[Pandiaraj A]

[Jeeva Kandasamy]

Hi Pandya,

Sorry for the delayed response.

We never did a test with certificates on this tool.

I try to replicate your setup by this week.

Thanks,

-Jeeva

[Pandiaraj A]

Hi Jeeva,

Any update on my request?

Thanks

Pandya.