We have 40,000 servers, hundreds of metrics, and hundreds of analytics on each metric.
If I up the UID bytes for metrics to 4 bytes, I can push the server name into the metric (metric = server.metric). Right now I have to use the server-id as the metric and the metric as a tag to make things performant which makes things like Grafana frustrating to use.
The UID creation is incredibly slow though. Right now, 150-million metric IDs are taking like 36 hours to make (with a multi-threaded app making rest calls). Is there any way to create metrics in bulk that is vastly faster than the REST API? Why is it so slow?