WAL reply duration increase

[dc3o]

Trying to understand why WAL reply increased after Prometheus was upgraded  to  2.33.5. Any advice where to start troubleshooting? 

Mar 10 03:22:05 prometheus prometheus[1038]: level=info ts=2022-03-10T10:22:05.756Z caller=head.go:583 component=tsdb msg="WAL replay completed" checkpoint_replay_duration=6.713852792s wal_replay_duration=32.700594357s total_replay_duration=41.759587541s
Mar 14 06:57:45 prometheus prometheus[3552]: ts=2022-03-14T13:57:45.386Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=5m10.693729136s wal_replay_duration=26.884695487s total_replay_duration=5m37.925437434s
Mar 28 08:50:56 prometheus prometheus[14944]: ts=2022-03-28T15:50:56.080Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=5m48.120257674s wal_replay_duration=27.001835591s total_replay_duration=6m15.504431742s
Mar 28 08:57:11 prometheus prometheus[15305]: ts=2022-03-28T15:57:11.374Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=5m46.213594484s wal_replay_duration=27.471572247s total_replay_duration=6m14.053022786s

[Brian Candler]

The WAL replay duration depends on the amount of WAL you have, so unless you can show the WAL was the same size at the time you restarted prometheus on Mar 14 and Mar 28, I don't think you can conclude very much from this.