X-trace-id
Kristin Klodzinski
When executed as an inbound policy, this action will remove headers on an incoming http request before reaching the upstream server withthe configured headers. When executed as an outbound policy, the configured headers are removed to the response from the upstream server.
We have been getting a lot of requests with x-client-version and x-trace-idand it is starting to overwhelm our logging system. We want to remove these headersfrom all requests. The following example demonstrates how to remove these headersusing the remove-headers action.
The following variables are made available for use in subsequent expressions andCEL interpolations after the action has run. Variable values will only applyto the last action execution, results are not concatenated.
NetScaler Observability Exporter supports OpenTracing (OpenTracing is a part of OpenTelemetry now) using Zipkin as the endpoint. NetScaler Observability Exporter transforms the tracing data collected from NetScalers into supported formats suitable for OpenTracing and exports them to Zipkin. Zipkin is a distributed tracing system that helps to gather the timing data required to troubleshoot latency problems in microservice architectures. Elasticsearch is used for long-term retention of trace data and the traces can be visualized using the Zipkin UI or Kibana.
When the tracing is enabled, initially, it adds additional open-tracing headers: x-trace-id, x-span-id, and x-parent-span-id to HTTP packet, before it forwards the packet to the next microservice pod.
The information about this communication or transaction is pushed to NetScaler Observability Exporter. The information includes the details about the headers, the timestamp (time when this request is initiated and the entire duration of the process), and annotations (annotations include HTTP, SSL, and TCP associated with that request).
Zipkin API stores the trace data in the Elasticsearch database, and finally stitch the complete trace to the given HTTP request and visualize it in the visualization tool such as Kibana. You can view the time that the request spent on each microservices.
Based on your NetScaler deployment, you can deploy NetScaler Observability Exporter either outside or inside Kubernetes clusters. You can deploy NetScaler Observability Exporter as a pod inside the Kubernetes cluster or enable the configuration on NetScaler MPX or VPX form factor outside the cluster. You can deploy NetScaler Observability Exporter using the Kubernetes YAML file provided by NetScaler.
The following diagram illustrates NetScaler as an ingress gateway with the NetScaler Ingress Controller as a sidecar. NetScaler Observability Exporter sends the tracing data collected from NetScalers to Zipkin API. The tracing data is, then, uploaded to the Elasticsearch server. From Elasticsearch, the data is sent to Zipkin UI or Kibana UI for visualization.
While deploying NetScaler CPX, you can modify the deployment YAML file cpx-ingress-tracing.yaml to include the configuration information that is required for the NetScaler Observability Exporter support.
If you have used a namespace other than default, change coe-zipkin.default.svc.cluster.local to coe-zipkin..svc.cluster.local. If ADC is outside the Kubernetes cluster, then you must specify IP address and Nodport address of NetScaler Observability Exporter.
In the following image, you can view the traces of the Watches application. The Watches application has multiple microservices for each watches type, communicating with each other to serve the application data. The trace data shows application FASTTRACK took more time to serve when compare to other micro services. In this way, you can identify the slow performing workloads and troubleshoot it.
NodeJS is a single threaded runtime for JavaScript. This means that NodeJS cannot take advantage of threads to determine which context belongs to a certain request. Request specific context can be very useful for logging because it allows us to assign a unique id to a request. With that it is possible to easily view all logging messages related to a specific request in a synchonous way thereby making debugging significantly easier. So how to we solve this in NodeJS? Enter Async Hooks!
NodeJS is a runtime that heavily relies on asynchronous calls. Almost everything inside NodeJS is asynchronous. The use of promises and timeouts for example. The eventloop is in charge of handling these asynchronous resources. the async hooks module provides an API to track these asynchronous resources. This API allows us to keep track of which asynchronous resource is is associated with a request.
In this example we will be using AsyncLocalStorage (a little abstraction on top of async_hooks). AsyncLocalStorage allows us to create a storage for each request with little to no effort. The example used in the NodeJS documentation explains this quite well.
We create a method called LoggingContext. This is essentially a wrapper around the creation of our AsyncLocalStorage and returns an object with methods we want to expose to the client code. The init method initializes the context for each request and calls a callback function when the initialization has completed. The context is initialized with an object that conforms to IContext and contains a requestId, traceId and the ipAddress from where the request was send. We can now create a simple express application that uses this LoggingContext
Three changes were made to the LoggingContext. The first change is the addition of ILogObj. This interfaces describes the layout of our log message. I choose to add operation as a way to describe what action is being performed and data to add a Record of useful properties that should be in the log message. The second change is the addition of transport to the LoggingContext method. This is essentially the transport to be used for the log messages. In this case I use console to transport the messages to console. The third and last change is the addition of the log method which allows us to log ILogObj to console with the request context and timestamp.
Note how each request got its own requestId and traceId assigned even though the requests were handled at the same time. We now created a great way to keep track of the log messages scoped to a request.
In the example above we added a axios interceptor, this interceptor uses the getTraceId method on loggingContext to get the traceId the context was initialized with. Whenever a request is send with Axios the interceptor is executed and x-trace-id is added to our request. The middleware we already had then ensures the x-trace-id is used in the loggingContext. The above example uses axios to send a request to /hello and you can see in the result that for each request the requestId changes but the different clients can still be traced through traceId.
93ddb68554