Seqno bug in IngestExternalFiles

[Fred Zhang]

Hello,

We are trying to use `IngestExternalFiles` for bulk workloads, but run into some funky assertion errors in tests.

Glancing over the source, there is an obvious bug in `consumed_seqno_count` because the debug build alters its behavior. Debug build include `ConsumedSequenceNumbersCount` from all IngestionJob, but the prod build only include the first one. [1]

It is also questionable whether this assertion make sense any more. Skimping through the source code (AssignLevelAndSeqnoForIngestedFile in particular), a seqno can be consumed for various reasons:

- overlapping range.

- snapshot_consistent && a snapshot created.

There is hardly any reason for the invariant of either all or none ingestion jobs consume seqno. [2]

We ran some experiments and rocksdb crash afterwards due to failed `force_consistency_checks` [3]

Can someone familiar with the `IngestExternalFiles` API comment on whether the bug/assertion can be fixed and if the potentially impact of potentially wrong(duplicated?) `seqno`?

[1] https://github.com/facebook/rocksdb/pull/5539/files#r600840656

[2] https://github.com/facebook/rocksdb/blob/master/db/db_impl/db_impl.cc#L4573-L4588

[3] https://github.com/facebook/rocksdb/blob/7457c7cd00f936ee4e3d37a48811a73851bee182/HISTORY.md#public-api-change-5

Thanks in advance!

-

Fred Zhang

[Yanqin Jin]

[1] Yes, this is a bug.

[2] Also seems a bug to me. Due to the fact that assigned_seq can be the target level's largest seq, the assertion here https://github.com/facebook/rocksdb/blob/f06b761185ca4ed67883670c68726f4ce2d16540/db/external_sst_file_ingestion_job.cc#L378 may also be problematic.

Could you contribute a fix?

facebook/rocksdb A library that provides an embeddable, persistent key-value store for fast storage. - facebook/rocksdb github.com

---

From: 'Fred Zhang' via rocksdb <roc...@googlegroups.com>
Sent: Wednesday, March 24, 2021 1:56 PM
To: rocksdb <roc...@googlegroups.com>
Subject: Seqno bug in IngestExternalFiles

 

--
You received this message because you are subscribed to the Google Groups "rocksdb" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rocksdb+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/rocksdb/f7d5da99-8100-4eac-8b83-a14353112ad3n%40googlegroups.com.

[Yanqin Jin]

Actually, that assertion should be fine because if (assigned_seq > last_seq)​.

---

From: 'Yanqin Jin' via rocksdb <roc...@googlegroups.com>
Sent: Wednesday, March 24, 2021 10:06 PM
To: rocksdb <roc...@googlegroups.com>; Fred Zhang <fre...@google.com>
Subject: Re: Seqno bug in IngestExternalFiles

 

To view this discussion on the web visit https://groups.google.com/d/msgid/rocksdb/BYAPR15MB23416C666BE036E4013CE4D0B1639%40BYAPR15MB2341.namprd15.prod.outlook.com.