Purged documents replicated back in

[wbi...@attic.nl]

I have an iOS app running on devices A and B, both configured with a continuous push replication to the other. The databases on the devices are in sync with 300 documents that won't be modified. When 100 docs are purged from A (in a 'backgroundTellDatabaseNamed' block) they are replicated back in from B. 

Is that to be expected ? 

I thought that replication was done by considering changes after a previous replication - but a purge is not a change (unlike a delete), or is it ?

[Jens Alfke]

On Aug 23, 2014, at 10:06 AM, wbi...@attic.nl wrote:

I have an iOS app running on devices A and B, both configured with a continuous push replication to the other. The databases on the devices are in sync with 300 documents that won't be modified. When 100 docs are purged from A (in a 'backgroundTellDatabaseNamed' block) they are replicated back in from B. 

Is that to be expected ? 

Yes, this is actually correct behavior. Purge is sort of weird — it completely forgets about the document and its history, so the next time the replicator hears about the document it will think it's new and will download it. This has a couple of unexpected side effects:

• If A has never pulled from B before, then during the initial scan of B's changes feed it will find the purged docs and since it doesn't have them yet it'll download them.
• If A has pulled from B before, nothing will happen until the next time one of those docs is modified on B; then it'll show up in B's changes feed and A will pull it again because it doesn't have it.

We inherited the purge feature from CouchDB, but it's becoming apparent that we need something smarter, like something in between a purge and a delete. I think what would work is a purge that leaves behind a marker saying "don't re-download this".

—Jens

[wbi...@attic.nl]

Op zaterdag 23 augustus 2014 20:08:48 UTC+2 schreef Jens Alfke:

On Aug 23, 2014, at 10:06 AM, wbi...@attic.nl wrote:

I have an iOS app running on devices A and B, both configured with a continuous push replication to the other. The databases on the devices are in sync with 300 documents that won't be modified. When 100 docs are purged from A (in a 'backgroundTellDatabaseNamed' block) they are replicated back in from B. 

Is that to be expected ? 

Yes, this is actually correct behavior. Purge is sort of weird — it completely forgets about the document and its history, so the next time the replicator hears about the document it will think it's new and will download it. This has a couple of unexpected side effects:

But the replicator won't hear about the document because it never changes. And its not the first replication: the 300 documents were created on A and replicated to B. 

This is the background: the documents are restaurant orders. Once they are paid they become irrelevant for our application and to keep the db clean we periodically purge old paid orders - but they returned back from peers.

I've solved the problem by adding a replication filter that skips old/paid orders but I don't understand why that should be necessary.

[Jens Alfke]

On Aug 24, 2014, at 1:13 AM, wbi...@attic.nl wrote:

But the replicator won't hear about the document because it never changes. And its not the first replication: the 300 documents were created on A and replicated to B. 

How soon after pushing to B do you purge the documents? There's an 'echo' where the corresponding puller from B will see the documents that were just pushed to B. But that wouldn't cause a problem unless you purged the docs the instant you pushed them.

Since you mentioned 'peers' — if you've got a mesh of P2P replications, another scenario is that A pushes to B, then A purges the docs, but then meanwhile B pushes to C. If A later pulls from C it'll see and download the docs that it previously purged.

Otherwise, the only thing that would cause this would be if the replication checkpoint got lost somehow and the puller had to start from the beginning of the changes feed.

I've solved the problem by adding a replication filter that skips old/paid orders but I don't understand why that should be necessary.

That's the safest solution, I think. In general there's no guarantee that a purged doc will stay purged, due to the effects described above.

—Jens