Promethues v2.x and Prometheus v3.x Tracks

[Harpreet Singh]

Hello Community

Since Promethues v3 has been released and currently I am using v2.53. And to upgrade to v3, I first have to update to v2.55 to make the TSDB ready for v3.

• Will Promethues be maintaining 2 seperate tracks on v2 and v3 ?
• How long (duration) will these seperate tracks be maintained ?
• And what will be maintained in these 2 seperate tracks like Fault correction, CVE etc.?

[Ben Kochie]

On Mon, Jan 20, 2025 at 11:01 AM Harpreet Singh <harpre...@gmail.com> wrote:

Hello Community

Since Promethues v3 has been released and currently I am using v2.53. And to upgrade to v3, I first have to update to v2.55 to make the TSDB ready for v3.

It is not strictly required to upgrade to v2.55.x. It is only necessary to run v2.55.x if you need to downgrade from v3.x. Only v2.55.x supports some small changes to the TSDB format. 

• Will Promethues be maintaining 2 seperate tracks on v2 and v3 ?

No 

• How long (duration) will these seperate tracks be maintained ?

Only the 2.53.x LTS track will be supported.

https://prometheus.io/docs/introduction/release-cycle/ 

 

• And what will be maintained in these 2 seperate tracks like Fault correction, CVE etc.?

No, only for the LTS release.

 

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/prometheus-users/fef76c24-9707-4199-affb-b9052a214570n%40googlegroups.com.

[Harpreet Singh]

Yes, we do have a requirement to support rollback from v3.x to a version below v2.55 , for example v2.53.

Does it cause any downtime or imcompatibility between releases?

And till when is the support for v2.55 ?

[Ben Kochie]

On Mon, Jan 20, 2025 at 2:06 PM Harpreet Singh <harpre...@gmail.com> wrote:

Yes, we do have a requirement to support rollback from v3.x to a version below v2.55 , for example v2.53.

Does it cause any downtime or imcompatibility between releases?

This is not supported. The storage format is incompatible.

 

And till when is the support for v2.55 ?

There is none, it is now an old release.

 

On Monday, 20 January 2025 at 15:36:34 UTC+5:30 Ben Kochie wrote:

On Mon, Jan 20, 2025 at 11:01 AM Harpreet Singh <harpre...@gmail.com> wrote:

Hello Community

Since Promethues v3 has been released and currently I am using v2.53. And to upgrade to v3, I first have to update to v2.55 to make the TSDB ready for v3.

It is not strictly required to upgrade to v2.55.x. It is only necessary to run v2.55.x if you need to downgrade from v3.x. Only v2.55.x supports some small changes to the TSDB format. 

• Will Promethues be maintaining 2 seperate tracks on v2 and v3 ?

No 

• How long (duration) will these seperate tracks be maintained ?

Only the 2.53.x LTS track will be supported.

https://prometheus.io/docs/introduction/release-cycle/ 

 

• And what will be maintained in these 2 seperate tracks like Fault correction, CVE etc.?

No, only for the LTS release.

 

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/prometheus-users/fef76c24-9707-4199-affb-b9052a214570n%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.

To view this discussion visit https://groups.google.com/d/msgid/prometheus-users/71a94fdf-e6de-400e-beb1-78408e65dfbbn%40googlegroups.com.

[Harpreet Rekhi]

Hello Ben

Two steps rollback is also not supported ? Like rollback from v3.x to v2.55 and then rollback from v2.55 to v2.53 without breaking the storage format or storage compatibility?

Regards

HarpreetR

[Gavine Xue]

Sorry to weigh in here, but do you foresee any issues if we develop a conversion tool to change the version number in index files back to 2 before a rollback? Prometheus V2 (starting from some older versions) has already stopped reading/parsing index label index 1...N and the label index table, correct? Is there anything else we should consider? Thanks!

Best regards

Gavine  

[Brian Candler]

On Monday, 20 January 2025 at 13:32:11 UTC Harpreet Rekhi wrote:

Two steps rollback is also not supported ? Like rollback from v3.x to v2.55 and then rollback from v2.55 to v2.53 without breaking the storage format or storage compatibility?

That is exactly what Ben said. v2.55 knows how to work with the data format from v3, but will not convert it back to the older format.

It sounds to me like you would be best served by staying on the LTS version v2.53, which still has bug and security fix support.  Why do you want to run anything later?

If you *absolutely* need new features introduced in later versions, then you will have to remain between versions v2.55 and v3.x - as there is no rollback to anything earlier than v2.55 without losing data. But that's fine, because if you *need* those new features then you wouldn't be able to rollback anyway.  Bear in mind that you will also have to keep upgrading to all v3.x versions to get bugfixes and security fixes, until the next LTS release comes along.

If you just want to test the newer versions then do that on a non-production system where you know you'll never need to roll back prior to v2.55

Gavine Xue wrote:

> do you foresee any issues if we develop a conversion tool to change the version number in index files back to 2 before a rollback?

I think it's fairly obvious that won't work. If the *only* difference between the data formats was the version number, then there would have been no need for a change in the version number.  The change in the version number implies that the way that the data is stored has been changed in an incompatible way.

[Gavine Xue]

Hi Brian,

Thanks for your prompt reply.

We were thinking the same — 'it's fairly obvious that won't work.' However, when we quickly reviewed the changes in v2.55.0, we noticed that it primarily makes FormatV3 recognizable while applying the same logic as FormatV2 (Backward compatibility with upcoming index v3 · prometheus/prometheus@5ccb069 · GitHub ). Other changes under the tsdb folder seem irrelevant (though we could be mistaken, as we didn’t dive into the details).

Additionally, Prometheus V2, starting from some versions, has already stopped reading/parsing Label Index 1...N and the Label Index Table — both of which are removed in Format V3. All this makes us wonder if simply changing the version number would suffice.

Are there any other changes in v2.55.0 that we should inspect further?

Another question: Have we decided which v3 version will make this change effective? The current v3 is still generating FormatV2 TSDB.

Thanks!

Best regards

Gavine

[Brian Candler]

The release notes for v2.55 describe this as an "upcoming index v3", and links to

https://github.com/prometheus/prometheus/pull/14934

which in turn links to

https://github.com/prometheus/prometheus/issues/14749

That has the full details of what's compatible. If I read this correctly, the proposal is to remove some obsolete fields. Once the format change has been made (and it hasn't yet), in principle it could still be compatible as far back as prometheus 2.15.

However, not only prometheus reads these chunk files (there's also thanos, mimir, ...) so good engineering practice says that along with changing the format, the version number will be bumped.

[Jian Xue]

Hey Brian,

Thanks for the links — they are exactly what we’ve been referring to.

It’s great to hear that 'in principle, it could still be compatible as far back as Prometheus 2.15.' If I’m interpreting this correctly, our initial thought should work, right? Specifically, simply changing the version from 3 to 2 in the index file for a V3-formatted TSDB block should make it recognizable and properly handled by Prometheus versions prior to 2.55, going back to 2.15.  Of course, this is theoretical, and proper testing will be necessary once Prometheus releases a version that generates V3-formatted TSDB blocks.

Bumping the version is absolutely justified given the breaking changes in the format. We are, however, seeking a feasible solution to meet our requirements — enabling rollback from Prometheus v3 to a version prior to v2.55 (though not as old as 2.15).

Thanks!

Best regards

Gavine

--

You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.

To view this discussion visit https://groups.google.com/d/msgid/prometheus-users/1fa127fb-9c9f-4018-9859-18106cb8354bn%40googlegroups.com.

[Gavine Xue]

Hi Brian,

Hope you had a good weekend!

I’d like to confirm the exact behavior when rolling back from Prometheus v3.x (where FormatV3 TSDB is generated) to a pre-v2.55 version, such as v2.54.1.

The v3 migration guide states:

"... you will only be able to downgrade to v2.55, not lower, without losing your TSDB persistent data."

This gives me the impression that downgrading to v2.54.1 is still possible, but I’d appreciate clarification on the expected behavior:

1. Would Prometheus v2.54.1 start after rollback, ignoring / losing FormatV3 TSDB data, while FormatV2 blocks (if present) remain intact?
2. Or is downgrading to v2.54.1 entirely unsupported, e.g. Prometheus v2.54.1 won't start at all?

From reviewing the source code, this block:
🔗 db.go#L1600-L1615
suggests that Prometheus may fail even though openBlocks() returns OK with FormatV3 blocks put into corrupted. Could you confirm if this is the intended behavior?

Thanks!

Best regards

Gavine

[Brian Candler]

I am not a developer of prometheus, just a satisfied user who sometimes shares some PromQL tips.

I do know that Ben Kochie is a major figure in the project, therefore I suggest you follow his advice, and the documented advice, which I would summarise as: "Don't do that".

It is of course open source, and you are free to do what you like with it, but if you don't use it in the recommended way then you support yourself. I think that's fair.

[Gavine Xue]

Thanks, Brian, for the heads-up.

Hi Ben,

Could you please help clarify the following questions?

1. Regarding the statement in the V3 migration guide: "... you will only be able to downgrade to v2.55, not lower, without losing your TSDB persistent data." — does this mean that downgrading from v3.x to a pre-v2.55.0 version is completely unsupported, or is it technically possible but would result in the loss of FormatV3 data after rollback? This distinction is crucial for our customers.

2. Is there a version — e.g., in a dev branch — where this change is already effective (i.e., generating FormatV3 TSDB data) so that we can test it?

Thanks!

Best regards

Gavine

[Brian Candler]

I think your problem has been solved in Prometheus v2.53.5:

[ENHANCEMENT] TSDB: Add backward compatibility with the upcoming TSDB block index v3 #16762