Data not showing up intermittently on the UI

[Hrishikesh Barua]

We are running some high volume tests on OpenTSDB (2.3.0) with BigTable, and a curious problem surfaces from time to time. For some metrics, an hour of data stops showing up on the web UI when we run a query. The span of "missing" data is very clearcut and borders on the hour (UTC). After a while, while rerunning the same query, the data shows up. There does not seem to be any pattern that we can deduce here, other than the hour span. Any pointers on what to look for and debug this?

[ManOLamancha]

On Thursday, July 6, 2017 at 10:44:29 AM UTC-7, Hrishikesh Barua wrote:

We are running some high volume tests on OpenTSDB (2.3.0) with BigTable, and a curious problem surfaces from time to time. For some metrics, an hour of data stops showing up on the web UI when we run a query. The span of "missing" data is very clearcut and borders on the hour (UTC). After a while, while rerunning the same query, the data shows up. There does not seem to be any pattern that we can deduce here, other than the hour span. Any pointers on what to look for and debug this?

That sounds odd. One thing to do is to disable the TSD compactions and see if that resolves the issue. set this to false: tsd.storage.enable_compaction