PouchDB and deleted true

[John Mandia]

Hi,

I really like pouchdb but when I try to delete docs I am still seeing them appear?

Scenario:

I create 5 docs and save them into pouchdb (websql).

I see entries in the document-store and in the by-sequence table.

I trigger an action on the UI that says delete a doc (passing an id). 

I get the document and all it's children (not the full doc just the id, revision cut down versions) and add the key _deleted with the value true and do a bulk update.

In the by-sequence table:

I see my entries but the doc ids are now in json and the doc_id column has a GUID instead of my original document id.

The json has _deleted:true and the deleted column has a 1.

Questions:

1) Is this the correct behaviour.

2) Will the sequences entries get deleted (as they will take up space)?

3) Is there a way of clearing away the sequence entries?

I am passing {adapter: ‘websql’, auto_compaction:true } as the options when creating the database.

In the document-store table:

1) I still see my docs and they point to the old seq id entries.

2) I have a new set of entries that have a guid and deleted:true in the json. The number reflects the number of documents I deleted.

Question:

When I try to get the documents by calling allDocs and passing in the id from the startKey I still get the parent doc and all it's children. 

I also have a query that returns docs in a certain state and that still shows the doc that I requested for deletion.

Am I deleting things the wrong way?

Any help/advise would be appreciated.

Cheers,

John

[Calvin Metcalf]

when pouchdb deletes a document it doesn't really delete it it just gives it a deleted tag, this is in order to correctly sync with another database, if we didn't leave it in then if you deleted it, but you sync from somebody who didn't the db would think it was a new document and recreate it over here, instead if you delete it and the other db doesn't touch it then then deletion is replicated over.

--
You received this message because you are subscribed to the Google Groups "PouchDB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pouchdb+u...@googlegroups.com.
To post to this group, send email to pou...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pouchdb/62edb88c-7598-4d7a-a93e-29e191b0f058%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

-Calvin W. Metcalf

[John Mandia]

I understand that the meta data saying the record should be deleted should remain but all the data that was within the json should be removed to free up space?

Would auto compaction/compaction get rid of the sequence entries (or at least the data within them) as it doesn't seem to be happening...should I call compact directly?

Any ideas why I would be getting the documents and their data back even when I've called delete?

Thanks,

John

[no...@nolanlawson.com]

Deleted documents are replicated, but they shouldn't show up when you do get() or allDocs().

If that's happening, then something is very wrong. Please open a bug with a test case; that would be super. :)

IIRC there may be issues with deleted docs in 3.0.6. You may want to try the master branch: http://pouchtest.com/nightly.

all the data that was within the json should be removed to free up space?

PouchDB does this automatically under the hood when you delete. No sense in keeping the old data around.

should I call compact

Compaction most def has bugs in 3.0.6. Use master for the time being (we're releasing 3.1.0 either today or tomorrow).

Cheers,
Nolan

[John Mandia]

Thanks for the responses.

Can I just confirm something (sorry for the basic questions):

1) Compaction should delete the data within the sequence entries for old sequences? If not is there a way of deleting the data in the sequence table as it builds up over time?

2) If all things go well when you save a number of documents with the deleted flag a record will remain with deleted = true (for sync purposes) but all other data should go?

3) With persistent queries I believe there was a bug raised on github that would return a document from the index even if that document had been deleted. I am thinking of switching to the old query approach until that is resolved (is that fixed in the nightly)?

I really like the project and appreciate the effort you guys put into it. Hopefully the first two points are correct because I really don't want old data hanging around as some of it can be quite large.

Thanks,

John

[John Mandia]

Sorry one last thing.

With the documents showing up in the document store with doc ids of guids........is that expected behaviour for a deletion or an indication that there is a problem?

Thanks,

John

[no...@nolanlawson.com]

On Thursday, November 6, 2014 3:23:35 PM UTC-5, John Mandia wrote:

Thanks for the responses.

Can I just confirm something (sorry for the basic questions):

1) Compaction should delete the data within the sequence entries for old sequences? If not is there a way of deleting the data in the sequence table as it builds up over time?

Compaction removes all the "leaf" revisions of a document: picture a branching tree, where the "leafs" are possibly conflicting latest revisions to a document. In the simple case this is just a single lineage.

So with deleted stubs, after compaction only a stub containing the document ID and deleted: true is kept around. It's very efficient.
 
All of this applies to PouchDB master, not 3.0.6; there's some stuff that was fixed recently. >_<

2) If all things go well when you save a number of documents with the deleted flag a record will remain with deleted = true (for sync purposes) but all other data should go?

Yup, basically.
 

3) With persistent queries I believe there was a bug raised on github that would return a document from the index even if that document had been deleted. I am thinking of switching to the old query approach until that is resolved (is that fixed in the nightly)?

I saw that issue, but wasn't able to reproduce. If you can reproduce, that would be swell, but our tests deal pretty heavily with deleted documents, and we never saw it.
 

With the documents showing up in the document store with doc ids of guids........is that expected behaviour for a deletion or an indication that there is a problem?

Not sure what you mean. Documents have random guids if you post() rather than put(); that's totally normal. Also the database has to remember WHICH document IDs were deleted; that's point of keeping them around for forever. :)

I really like the project and appreciate the effort you guys put into it. Hopefully the first two points are correct because I really don't want old data hanging around as some of it can be quite large.

Cheers, glad ya like it. :)

[John Mandia]

Hi,

Thanks again for the responses.

I tried with the nightly and still seeing the documents exist:

So here is a better example:

_pouch_mydb

    by-sequence

       seq 1 & 2 are entries for my views.

       seq 3 is doc1 json e.g. { "mydata":"x"} deleted = 0 doc_id = MY-CUSTOM-ID

       seq 4 is doc 2 json e.g. { "mydata":"y"} deleted = 0 doc_id = MY-CUSTOM-ID-2

      seq 5 is doc 1 updated json e.g. {"mydata":"xy"} deleted = 0 doc_id = MY-CUSTOM-ID

I then update doc 1 and 2 and set the _deleted flag to true and put them back in with bulkDocs.

I look at the database in chrome and see: 2 new entries in the sequence table.

     seq 6 json contains {"id":"MY-CUSTOM-ID", "key":"MY-CUSTOM-ID", "value":{"rev":"2-5c1f006c5ed375ab55066f8fa0118bac"},"_deleted":true} deleted = 1, doc_id = "DE9C4D42-096F-F22F-9C7C-01F3A9ECB26E"

similar thing in seq 7 where doc 2 was deleted.

I go to the document-store table and I see 2 entries for the views I created, 2 entries for the docs and 2 entries for the deletion whose keys are guids e.g:

{"deleted":true,"id":"0AC51156-16D6-A457-BFC1-DB19F539C904","rev_tree":[{"pos":1,"ids":["fbfb0ee325ad558317d5eec21391291d",{"status":"available","deleted":true},[]]}],"seq":6}

 I always use put to have ids I can search by :)

Thanks,

John

[John Mandia]

Hi all,

Just an update. Nothing wrong with pouchdb, more a human error on my part.

I was deleting in bulk by first using allDocs to get a set of docs (passing an array of ids) and then looping through that and adding the _deleted = true property. 

The reason I was seeing new entries was because allDocs returns the docs I am interested (when I don't specify return the full document) as:

 {"id":docid, "key":dockey, value: { "rev":revisionid }} 

so simply adding _deleted property to the object means I end up with an array of new documents to add to pouchDB when I call bulkDocs.

The docs get added (without an _id so pouchDb generates a guid for them, and because they have the _deleted property they get marked as deleted) and a success is returned.

What I needed to do was build up an array of new objects that the _deleted = true property and have _id and _rev properties using the id and value.rev properties from the items in the allDocs response (as I loop through I also make sure that error and deleted are undefined as I just want to request current records are deleted).

Thanks for the answers and sorry for wasting your time. Still learning all the ins and outs but as I mentioned before I am really liking what you have done with pouchdb.

John

[Nick Colley]

Hey John,

Glad you sorted your issue, thanks for posting in detail what went wrong hopefully it helps someone else too!

Take it easy,
Nick