Is there a way to fetch only the deleted documents?

[Brendan Duddridge]

I need to be able to fetch the deleted documents. I've setup an allDocumentsQuery with kCBLIncludeDeleted as the allDocsMode

But I don't want the non-deleted documents to be fetched. Because if I have 10,000 documents and 1 has been deleted somewhere, I don't want to have to loop through all 10,000 to find the one deleted one.

Also, if I call existingLocalDocumentWithID, will it return a document even if it's deleted? Hopefully so. I need this because I'm hoping to get an array of dictionaries that represent the deleted documents. Then let the user pick which ones they want to restore. I'll fetch the document given the ID and then get its leaf revisions, then restore it.

I'm using CBL 1.4 on macOS and iOS.

Thanks,

Brendan

[Jens Alfke]

On Oct 22, 2017, at 11:37 PM, Brendan Duddridge <bren...@gmail.com> wrote:

But I don't want the non-deleted documents to be fetched. Because if I have 10,000 documents and 1 has been deleted somewhere, I don't want to have to loop through all 10,000 to find the one deleted one.

Good point. There isn’t really a solution for this. But I think the overhead shouldn’t be too high because the query isn’t fetching the body of each document, just the metadata. Have you experienced this being slow or are you just worried?

Also, if I call existingLocalDocumentWithID, will it return a document even if it's deleted? Hopefully so. I need this because I'm hoping to get an array of dictionaries that represent the deleted documents. Then let the user pick which ones they want to restore. I'll fetch the document given the ID and then get its leaf revisions, then restore it.

You mean existingDocumentWithID, right? (Local docs are different.) Yes, I’m pretty sure it returns the document (though I’m getting rusty on 1.x so don’t take that as gospel.)

—Jens

[Brendan Duddridge]

You mean existingDocumentWithID, right? (Local docs are different.) Yes, I’m pretty sure it returns the document (though I’m getting rusty on 1.x so don’t take that as gospel.)

Actually yes, I did mean existingDocumentWithID. I mistyped. Sorry about that. What I found out that existingDocumentWithID returns nil if the document is deleted. So it doesn't find it. But I found that documentWithID does load the deleted document so I can create a new non-deleted revision from it.

Good point. There isn’t really a solution for this. But I think the overhead shouldn’t be too high because the query isn’t fetching the body of each document, just the metadata. Have you experienced this being slow or are you just worried?

The specific database I'm working on took between 7 and 10 seconds to fetch and filter them to get only the deleted documents. I can live with that. But it will vary depending on the database. 

What took MUCH longer is to go through all of those deleted documents getting their leaf revisions and then getting the revision properties. I'm talking 9 minutes or so on my MacBook Pro to process 88,380 deleted documents. 

This is my code that processes the deleted documents:

for (CBLQueryRow *row in deletedRows) {

 

 if (progress.isCancelled) {

TFFLog(@"Fetch deleted documents cancelled...");

break;

 }

 

 NSArray *leafRevisions = [row.document getLeafRevisions:&error];

 

 CBLSavedRevision *revision = [leafRevisions firstObject];

 

 

if (revision && !error && revision.propertiesAvailable) {
   CBLJSONDict *properties = revision.properties;
   if ([(NSNumber *)properties[@"_deleted"] isEqual:@1]) {
       revision = revision.parentRevision;

       if (revision.propertiesAvailable) {

           properties = revision.properties;

       } else {

         properties = nil;

       }

    }

    NSString *objectType = properties[@"type"];

    if (objectType.length > 0 && properties.count > 0) {

        NSMutableArray *revisionsForType = deletedDocs[objectType];

        if (!revisionsForType) {

          revisionsForType = [NSMutableArray array];

          deletedDocs[objectType] = revisionsForType;

        }

        [revisionsForType addObject:properties];

    }                    

 }

Does that look like the right way to get the properties for the deleted documents? I need to display to the user a list of objects that they can restore, so I do need to dig into the revision properties to get that data.

Thanks!

Brendan

[Jens Alfke]

On Oct 24, 2017, at 8:12 PM, Brendan Duddridge <bren...@gmail.com> wrote:

if (revision && !error && revision.propertiesAvailable) {
   CBLJSONDict *properties = revision.properties; 
   if ([(NSNumber *)properties[@"_deleted"] isEqual:@1]) { 

You can replace this with just

if (revision.isDeletion) {

—Jens

[Brendan Duddridge]

You can replace this with just

if (revision.isDeletion) {

—Jens

Thanks for the tip Jens. I've just done that.

But I still get the WARNING: Couldn't load body/sequence of CBLSavedRevision error.

I've modified my code as follows:

 NSArray *leafRevisions = [row.document getLeafRevisions:&error];

 

 CBLSavedRevision *revision = [leafRevisions firstObject];

 

 if (revision && !error) {

CBLJSONDict *properties = nil;

if (revision.isDeletion) {

revision = revision.parentRevision;

if (revision.propertiesAvailable) {

properties = revision.properties;

}

} else if (revision.propertiesAvailable) {

properties = revision.properties;

}

.... and so on

The problem is that "propertiesAvailable" comes back with YES and then calling revision.properties results in a NULL value and I get the WARNING error message too. I'd like to avoid trying to get the properties and generating that warning if there are no properties available to get. It might speed things up.

The documentation in the header for propertiesAvalable states:

Are this revision's properties available? They may not be if the revision is an ancestor and

either the database has been compacted, or the revision was replicated from another db.

Is it possible it's returning false positives even if there's no properties to fetch?

I'd like to speed this process up as much as possible. Right now it's taking around 10 minutes to fully determine all the data that's available to recover. And that's on a fast MacBook Pro. I haven't yet tried it on an iOS device.

Thanks,

Brendan

[Jens Alfke]

On Oct 25, 2017, at 10:42 AM, Brendan Duddridge <bren...@gmail.com> wrote:

The problem is that "propertiesAvailable" comes back with YES and then calling revision.properties results in a NULL value and I get the WARNING error message too.

Honestly that sounds like a bug. Could you file an issue?

—Jens

[Brendan Duddridge]

Sure, I can do that. But I know you're going to want a sample application that demonstrates the problem with a sample database. It'll take me some time to put that together.