|
This is definitely a scalability concern and we are working on a plan for addressing it. However, we don't yet have a target date.
There are two approaches we can take to mitigate this. The first is an operational approach, where Tephra would provide the ability for an administrator to truncate the invalid list up to a given point. The idea is that, as part of a normal operational policy handling major compactions, an admin would know up to what time all tables in the cluster have been major compacted. Since Tephra transaction IDs are time based, you could then manually issue a command to truncate the invalid list up to this time, since you know, by virtue of the major compactions completing, that any data from invalid transactions prior to that point have been purged. This isn't ideal, as it requires some operational coordination, but it is doable.
The second approach would build this processing and tracking into Tephra itself, so that it could make the determination to automatically truncate the invalid list. This will require quite a bit more complexity to do the tracking, and needs a detailed design around it.
|
