RabbitMQ 4.2.2: Quorum queue Raft logs grew until disk exhaustion; Khepri metadata unavailable

[Swathi Mocharla]

The same quorum queue replica on each of the three nodes eventually filled its PVC completely, resulting in all RabbitMQ PVCs reaching 100% utilization. Once the disks were full, the cluster became unusable:

• list_users returned no users
• list_vhosts returned no vhosts
• rabbit_db_user:get_all(). returned []
• cluster_status showed no running/disk nodes
• rabbit_khepri:status(). returned []
• khepri:get([]). returned {error,noproc}

The quorum queue data was still present on disk.

1. Is it expected for a quorum queue replica to retain thousands of Raft segment files and dozens of checkpoints, or should compaction eventually reclaim them?
2. Under what conditions can Raft log compaction or checkpoint cleanup stop progressing?
   • Lagging follower?
   • Snapshot installation failure?
   • Frequent membership changes (grow/shrink)?
   • Heavy sustained workload?
3. Can a disk-full condition cause Khepri metadata to become unavailable while the broker is still running? Is the observed behavior (empty users/vhosts and rabbit_khepri:status() -> []) expected in this scenario?
4. Are there any known issues in RabbitMQ 4.2.2 (Ra 2.17.1) related to Raft log compaction, checkpoint cleanup, or Khepri behavior under disk-full conditions?
5. Are there recommended metrics or diagnostics to detect stalled Raft compaction before PVC exhaustion?

[Lucas Weis Polesello]

Hi Swathi,

One thing you didn't mention was. Did the queue was ever completely empty? Did you had any unacked messages? One of the issues before 4.3x is that Quorum Queues won't reclaim disk space (since the checkpoint/snapshot will never update) if you have any OLD unacked messages. Usually, a good measurement is finding the segment file which has the lowest raft index. That should point you why RMQ doesn't reclaim disk space.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/rabbitmq-users/b00fd5ab-2e94-482e-a280-3ad029467a47n%40googlegroups.com.

[Swathi Mocharla]

hi Lucas,

Thanks for your response and inputs.

I inspected the on-disk data for the offending quorum queue (EVENT_QUEUE).

Findings:

• max_checkpoints => 3
• about 6134 .segment files (~20 GB)
• about 30 checkpoint directories

• Active snapshot:

  snapshots/0000000000000001_00000000002F54BD

• Latest checkpoint:

  checkpoints/0000000000000001_0000000001A78D56

The checkpoint index advanced from approximately 3.1M to 27.8M, so snapshots appear to have continued progressing.

Given that snapshots were advancing, why would old checkpoints and Raft segments not be reclaimed? Are there conditions other than old unacknowledged messages that can prevent checkpoint/segment cleanup?

[Lucas Weis Polesello]

Hi Swathi,

Yeah. I believe this confirms the unacked messages may not be the issue itself. I don't recall any other scenario that would prevent compaction.

I believe we've seen a similar case—also on the same minor version (4.2.x) - which was triggered after an upgrade and Khepri was also enabled during that same upgrade.

On that occasion, the resolution was deleting a member and re-adding them to the Quorum. Using `rabbitmq-queues delete_member|add_member` helped. Hope it helps on your case!

Regards,

To view this discussion visit https://groups.google.com/d/msgid/rabbitmq-users/6517ad4e-4490-4e33-93cf-f9e051531ee4n%40googlegroups.com.