relabeling in remote read

[Johny]

I've a peculiar use case where there are large number of time series published with a label set, that is causing overhead at query time when the filter includes shared label value.  e.g.

my_metric1{l1="fixed",l2="..",l3=".."}

my_metric2{l1="fixed",l2="..",l3=".."}

my_metric3{l1="fixed",l2="..",l3=".."}

...

query: my_metric1{l1="fixed",...}

The overhead is not dependent on number of samples query, but due to the way prometheus loads metric metadata, specifically label postings and finds an intersection of these lists to find the time series to load from DB. If I remove the l1 filter above, query runs much faster and overhead is minimal.

Due to number of queries hitting the system and high friction of changes in user applications, I am looking for ways to solve this problem in query processing. One option I was considering was filtering this label out before sending query to remote read destination but the remote read configuration does not support metric relabeling. Can you suggest alternatives? Modifying the ingestion path or metric modeling is not feasible at this point and will take a long time to implement.