thanos sidecar cannot find prometheus

[mel]

Hello I am trying to run thanos sidecar on a on-prem prometheus instance to ship logs to AWS S3, and I will run thanos in AWS. My understanding is that this should work

# thanos sidecar --tsdb.path /data/prometheus/ --prometheus.url http://localhost:9090 --objstore.config-file /etc/prometheus/bucket_config.yaml

But I get this error

ts=2024-09-04T20:49:45.622990448Z caller=options.go:26 level=info protocol=gRPC msg="disabled TLS, key and cert must be set to enable"
ts=2024-09-04T20:49:45.623495042Z caller=factory.go:53 level=info msg="loading bucket configuration"
ts=2024-09-04T20:49:45.623870275Z caller=sidecar.go:426 level=info msg="starting sidecar"
ts=2024-09-04T20:49:45.624053759Z caller=reloader.go:262 level=info component=reloader msg="nothing to be watched"
ts=2024-09-04T20:49:45.624159982Z caller=intrumentation.go:75 level=info msg="changing probe status" status=healthy
ts=2024-09-04T20:49:45.624263372Z caller=http.go:73 level=info service=http/server component=sidecar msg="listening for requests and metrics" address=0.0.0.0:10902
ts=2024-09-04T20:49:45.624543225Z caller=tls_config.go:313 level=info service=http/server component=sidecar msg="Listening on" address=[::]:10902
ts=2024-09-04T20:49:45.625111798Z caller=tls_config.go:316 level=info service=http/server component=sidecar msg="TLS is disabled." http2=false address=[::]:10902
ts=2024-09-04T20:49:45.626098488Z caller=sidecar.go:193 level=warn msg="failed to validate prometheus flags. Is Prometheus running? Retrying" err="found that TSDB Max time is 1d12h and Min time is 2h. Compaction needs to be disabled (storage.tsdb.min-block-duration = storage.tsdb.max-block-duration)"
^Cts=2024-09-04T20:49:47.877678349Z caller=main.go:182 level=info msg="caught signal. Exiting." signal=interrupt
ts=2024-09-04T20:49:47.877756234Z caller=intrumentation.go:67 level=warn msg="changing probe status" status=not-ready reason=null
ts=2024-09-04T20:49:47.877774686Z caller=http.go:91 level=info service=http/server component=sidecar msg="internal server is shutting down" err=null
ts=2024-09-04T20:49:47.877876118Z caller=http.go:110 level=info service=http/server component=sidecar msg="internal server is shutdown gracefully" err=null
ts=2024-09-04T20:49:47.877982816Z caller=intrumentation.go:81 level=info msg="changing probe status" status=not-healthy reason=null
ts=2024-09-04T20:49:47.87814256Z caller=grpc.go:174 level=info service=gRPC/server component=sidecar msg="internal server is shutting down" err=null
ts=2024-09-04T20:49:47.878219725Z caller=grpc.go:187 level=info service=gRPC/server component=sidecar msg="gracefully stopping internal server"
ts=2024-09-04T20:49:47.878326514Z caller=grpc.go:200 level=info service=gRPC/server component=sidecar msg="internal server is shutdown gracefully" err=null
ts=2024-09-04T20:49:47.878405123Z caller=main.go:174 level=info msg=exiting

Prometheus is definitely running

# prometheus --version
prometheus, version 2.47.2 (branch: HEAD, revision: 3f3172cde1ee37f1c7b3a5f3d9b031190509b3ad)
build user: root@79f2ad339b75
build date: 20231012-16:07:10
go version: go1.21.3
platform: linux/amd64
tags: netgo,builtinassets,stringlabels



# curl http://localhost:9090
<a href="/graph">Found</a>.



# ls /data/prometheus
01J5SMXQ9NH8YFQHBGQ3MNS6J3 01J5ZEAABQQ5FKN10M497V9PY9 01J657Q89PQ2VG1Q4ECAT5G5YZ 01J6B13VKC5WZ0YZ30VH6Z8ES9 01J6GTG80PTKBQ1A4H5T5VY9HC 01J6PKWV8RZDDHTNFZQBYG3RZS 01J6WD9S20Z5Q6HS9N5BS40DMJ 01J6YZN4MMM7ZM5W9NXFEMZ9MT chunks_head wal
01J5VJQ8662PZNG6HBTTP1MC8B 01J61C451ERBCVBA53BAJRPQJM 01J675GSSK1ZVQ8NXWT8DKZQ4S 01J6CYXCVCYY9XVE9TZAXTKX5X 01J6JRA0XYMJE7CE9P5ZKF6TTV 01J6RHPNFX8XKMM29M2TN3FN1M 01J6YB31NZTV4J4N9C3PVG8487 01J6YZNQN83DAM0H1BVYC5A2KC lock
01J5XGGSA7A4WXT252HSRQJ3PC 01J639XPHKXAEGWGYS1CSS5NF0 01J693AB1A89Y7G5MRBZ6SBZT5 01J6EWPYS460286NA24MN5NC2T 01J6MP3FETWR1GJV3N6M0FHK01 01J6TFG4RD0KQKEK2RM220N1XZ 01J6YRSDCMDJDM2XN91MAN58QY 01J6Z6GVWNNHPNX3C1DKRN5GWR queries.active

[mel]

The "is prometheus running" msg was a red herring. The real issue was that err shown next. After setting storage.tsdb.max-block-duration and storage.tsdb.max-block-duration to 2h, I was able to run sidecar.