Forgive me if this is not the right forum for thanos questions.
I recently set up a test instance of thanos, using the S3 interface to ceph for the long term storage. In my ceph rgw logs, I can see the occasional PUT request to put data into the thanos bucket. But I am having trouble using s3cmd and the rbd cli to look at what has gone into it.
Attempts to list the contents of the thanos bucket with s3cmd returns no results. Attempts to use the ceph rbc cli to list the contents of the storage pools show no results.
If I point my grafana dashboard directly at the prometheus server, it shows the expected lookback (6h). If I point the dashboard at the thanos querier, I see the full lookback. This tells me that thanos is indeed storing the long term data.
How can I poke at the storage system directly to view what has gone into it?
--Mike