Is tiered storage feature fully completed and ready for production use?

[Goku]

Hi WiredTiger Community,

I have been exploring WiredTiger tiered storage feature and recently read the arch-tiered-storage.dox documentation (from the develop branch). The document provides a great overview of the architecture for tiered storage, which has been really helpful for my work.

However, I noticed a key detail in the documentation: it mentions that "While there is a mechanism to create new objects, there is no removal,

or <em>garbage collection</em> of objects that become redundant.", Garbage Collection (GC) functionality for tiered storage had not yet been integrated at the time the document was written. This made me curious about the current development status of two areas, and I'd appreciate any updates the team can share:

1. First, is the core tiered storage feature set now fully completed and ready for production use?
2. Second, regarding Garbage Collection for tiered storage specifically: Has this functionality been integrated since the document was published? If not, is there a planned timeline or roadmap for its development and release?

I understand documentation might not always reflect the latest code changes, so any clarity on these points would help me better plan how to leverage tiered storage in my project. If additional context (e.g., my use case, specific WiredTiger version I am testing) would be useful, please let me know and I would be happy to share more.

Thank you in advance for your time and insights!

[Will Korteland]

Hi Goku,

Thanks for your interest in WiredTiger! I'm glad you found the documentation useful.

We still don't have that garbage collection functionality, sorry. Tiered storage development is on pause for now -- while it solves a problem, we've decided to solve that problem another way. As such, tiered storage as a whole may be removed in the medium to long term.

I appreciate that's probably not what you were hoping to hear, so if you want to share your use case and get some advice then please feel free.

Regards,

 - Will

--
You received this message because you are subscribed to the Google Groups "wiredtiger-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wiredtiger-use...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/wiredtiger-users/bc51c4e7-09ae-46f1-8c46-268674005f21n%40googlegroups.com.

[Goku]

Hi Korteland,

Thank you so much for your prompt and transparent response—it’s really helpful to understand the current status of tiered storage.

I wanted to follow up on two points, if you don’t mind. First, you mentioned that the team has decided to “solve that problem another way” instead of continuing with tiered storage. Could you share a bit more about what this alternative approach might entail? Even a high-level direction (e.g., new APIs, integration with existing features, or a different architectural design) would help us align our plans.

Second, to provide context on our use case: we were looking to leverage tiered storage specifically for hot-cold data tiering. Our goal is to move less frequently accessed (cold) data to lower-cost storage media, while keeping frequently used (hot) data on high-performance storage—all to optimize overall storage costs without sacrificing access speed for critical data.

We’d greatly appreciate any insights or advice you might have on how to address this use case given the pause on tiered storage.

Thanks again for your time and support!

[Will Korteland]

Hi Goku,

All I'm really able to say right now about our future solution to this problem is that it's an architectural change, sorry.

As for the hot/cold data tiering problem -- it depends which product you're using. If you're using MongoDB, then online archive is probably the way to go here. I can't really help with Mongo-specific questions on this mailing list, but I can point you at the docs: https://www.mongodb.com/docs/atlas/online-archive/overview/

If you want to use WiredTiger itself for this (without MongoDB), I'd probably start with the custom filesystem extension API (https://source.wiredtiger.com/develop/custom_file_systems.html and https://source.wiredtiger.com/develop/struct_w_t___f_i_l_e___s_y_s_t_e_m.html). You could use this to (say) try to access local content first, and then reach out to your colder tier if that fails.

 - Will

To view this discussion visit https://groups.google.com/d/msgid/wiredtiger-users/94098a99-a353-4d98-b449-f1376ba2b500n%40googlegroups.com.