Backup & Restore from within iOS App

[goo...@marco.betschart.name]

I'm not quite sure where to post this issue; I've tried it on GitHub - which seems to be the wrong place.

Afterwards I've posted this issue in the Couchbase Forums, but there seems to be no activity in this topic.

As I need to resolve this, I'm now trying my luck in this Google Group - and would be open for suggestions where to post which issues best ;)

I'm trying to implement a user driven backup & restore functionality within my iOS app.

The basic steps to get to a solution are:

For backing up the database locally:

1. Stop the sync process
2. Close the connection to the currently used database
3. Copy the *.cblite and the corresponding * attachments folder
4. Open again the connection to the currently used database
5. Kick off the sync process

To restore the database:

1. Close the connection to the currently used database
2. Replace the currently used database with a backup done before
3. Open the connection to the just restored database
4. Create new revisions of every document in the restored database to force the client's data be synced to the server - how? (see below)
5. Kick of the sync process
6. Doe some sort of cleanup after the initial sync - how? (see below)

Unfortunately I have no clue, how to create new revisions of each document without modifying it's data. How could this be done?

Also I think I have to delete all documents which are newer than the backup after the sync process has finished the first time, to avoid cluttering the local database with newer remote documents? How could I be sure the initial sync has been completed after the restore has taken place?

[Jens Alfke]

On Nov 14, 2015, at 1:55 AM, goo...@marco.betschart.name wrote:

Unfortunately I have no clue, how to create new revisions of each document without modifying it's data. How could this be done?

Well, you are modifying its data, by restoring it to an earlier state. (You wouldn’t need to do this to documents that haven’t changed since the backup.)

To restore the earlier state:

1. Read the old copy of the document from the backup
2. Check its revID against the revID of the current document; if they’re the same, skip this document.
3. Update the current document with the properties of the old document, except for the “_rev” property.

Afterwards I've posted this issue in the Couchbase Forums, but there seems to be no activity in this topic.

That’s odd; I watch the mobile sub-forums there but I don’t think I saw your post.

—Jens

[Brendan Duddridge]

Hi Jens,

Are you implying in your response to this question that you can't just replace the cblite2 database package and that you must read through the backup file, restoring to the current database with the contents of the backup file? Does that mean that you would then also have to delete all the documents created since the most recent document in the backup database?

Thanks,

Brendan

On Saturday, November 14, 2015 at 12:35:23 PM UTC-7, Jens Alfke wrote:

[Jens Alfke]

On Nov 21, 2015, at 7:19 PM, Brendan Duddridge <bren...@gmail.com> wrote:

Are you implying in your response to this question that you can't just replace the cblite2 database package and that you must read through the backup file, restoring to the current database with the contents of the backup file? Does that mean that you would then also have to delete all the documents created since the most recent document in the backup database?

Yes, if your purpose in backing up is to reset the database back to the state it was in at some past time (as opposed to just being able to restore the database later on in case of data loss.) In resetting back to a past state, you are fighting against the replicator, whose job it is to update to the latest state available on the server. That’s why you can’t just restore the database file — it contains old revisions that will be replaced by newer ones from the server.

—Jens

[Brendan Duddridge]

Well that definitely complicates things. Perhaps another way to reset back to a previous state would be to purge all the documents on the server for a specific channel? Given that in my case a user can have multiple cblite2 database files, each belonging to their own channel. And then restore the cblite2 database file to a previous version?

Thanks,

Brendan

[Brendan Duddridge]

I forgot to add that I would think then the sync process could start from scratch, uploading the data from the client to the server. Basically the client gets to decide who the "source of truth" is in this circumstance.

[goo...@marco.betschart.name]

I've now tried the following approach:

1. Close any database connection

2. Delete the local database

3. Move an older backup to it's place

4. Open the connection to the restored, local database

5. Kick off the sync process

6. Wait till the initial replication has completed

7. Purge all documents provided by the pull replicator

Unfortunately I'm stuck in the steps 6 + 7. It seems, the replicator does not execute a full replication after restoring the database. Although the kCBLReplicationChangeNotification gets called multiple times, it's always printing that there are 0 pendingDocumentIDs that need a sync (see code below).

But after killing the app and restart it from scratch, all documents get synced from the server.

Any idea what's wrong?

The following method is waiting till the initial replication has completed (registered to the kCBLReplicationChangeNotification):

func databaseReplicationChanged(notification: NSNotification){

guard let replication = notification.object as? CBLReplication else {

return

}

if let error = replication.lastError{

print(error)

NSLog(error.localizedDescription)

}

let replications = replication.localDatabase.allReplications()

var pendingDocumentIDCount = 0

var idleReplicationCount = 0

for repl in replications{

if let pendingIDs = repl.pendingDocumentIDs{

pendingDocumentIDCount += pendingIDs.count

}

if repl.status == .Idle{

idleReplicationCount++

}

}

print("pendingDocumentIDCount: \(pendingDocumentIDCount) - idleReplicationCount: \(idleReplicationCount)")

if replications.count == idleReplicationCount && pendingDocumentIDCount == 0 {

NSNotificationCenter.defaultCenter().postNotificationName(PersistenceManagerNotification.DatabaseCompletedReplication.rawValue, object: replication.localDatabase)

}

}

This additional purge method, gets called from the PersistenceManagerNotification.DatabaseCompletedReplication handler:

func databasePurgePulledDocuments(database: CBLDatabase) throws{

for repl in database.allReplications(){

if repl.pull {

if let documentIDs = repl.documentIDs{

print("documentIDs to delete: \(documentIDs)")

for documentID in documentIDs{

try database.deleteLocalDocumentWithID(documentID)

}

}

break

}

}

}

[goo...@marco.betschart.name]

@Brendand

Maybe deleting all documents from a given channel on the server is the easier approach. Any idea how to accomplish this?

[goo...@marco.betschart.name]

*shit* can't edit my previous post :(

Forgot to add the console output when restoring the database (there are a few extra lines from other methods present in there).

Maybe these warnings have something in common with the broken initial sync after restoring?

hasPullReplication: false

hasPushReplication: false

pendingDocumentIDCount: 0 - idleReplicationCount: 0

pendingDocumentIDCount: 0 - idleReplicationCount: 0

2015-11-22 11:39:53.468 biz[5379:11135826] _BSMachError: (os/kern) invalid capability (20)

2015-11-22 11:39:53.468 biz[5379:11135826] _BSMachError: (os/kern) invalid name (15)

pendingDocumentIDCount: 0 - idleReplicationCount: 1

pendingDocumentIDCount: 0 - idleReplicationCount: 2

initialReplicationCompleted: Optional((Function))

2015-11-22 11:39:53.710 biz[5379:11135826] Running database update...

11:39:53.903‖ WARNING: CBL_Pusher[https://sync.mandelkind.biz:4984/biz]: Couldn't get local contents of {-hZTLhYojFIQjEkOxMFgRJB #1-cefce94614ae803e05594b95a7f60f47}

2015-11-22 11:39:53.922 biz[5379:11135826] Database finished.

pendingDocumentIDCount: 0 - idleReplicationCount: 1

11:39:54.396‖ WARNING: CBL_Pusher[https://sync.mandelkind.biz:4984/biz]: Couldn't get local contents of {-_8gOVW9U7S-LxzfIBgOkbz #1-f30d143725f00a5cca94b61ed5902576}

pendingDocumentIDCount: 0 - idleReplicationCount: 2

[Mark]

I spent some time thinking about this type of solution on another project I was working on so I think I understand what you are trying to do.

There cannot be a true "source of truth” database in a master-master cluster, which is the idea on which CouchDB’s protocols are based. The source of truth is within the individual documents. So, I think your options are:

1. Clear/purge the database from all other nodes in the syncing system (esp. the “master server”/Couchbase/CouchDB) and resync from your local backup. I don’t know how you accomplish this in your environment.

2. If you know the individual bad doc: 1) make a copy of the correct version when restoring from backup 2) sync from master 3) push an edit of the ‘bad doc’ from the ‘good copy’ made from your backup.

Neither are ideal, but it’s the way the technology is structured, not a failing.

Cheers.

> --
> You received this message because you are subscribed to the Google Groups "Couchbase Mobile" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to mobile-couchba...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/mobile-couchbase/1bb08566-99cc-490c-a4ef-1187daef9b93%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Jens Alfke]

On Nov 22, 2015, at 2:57 AM, goo...@marco.betschart.name wrote:

Although the kCBLReplicationChangeNotification gets called multiple times, it's always printing that there are 0 pendingDocumentIDs that need a sync (see code below).

pendingDocumentIDs is only available in push replications — it returns the local docs that have revisions that have not yet been pushed to the server.

If you want to find out when a replication completes, wait for its status to change to idle (if it’s continuous) or stopped, with no error.

—Jens

[Jens Alfke]

On Nov 22, 2015, at 2:31 AM, Brendan Duddridge <bren...@gmail.com> wrote:

Well that definitely complicates things. Perhaps another way to reset back to a previous state would be to purge all the documents on the server for a specific channel?

Yeah, I’d recommend that too, although it does cause everything to get uploaded back to the server, which can take a while.

(But there’s the problem that for some reason SG hasn’t implemented the _purge call. It’s pretty easy to do a purge using CB Server APIs though: just delete the docs.)

—Jens

[Jens Alfke]

On Nov 22, 2015, at 3:05 AM, goo...@marco.betschart.name wrote:

11:39:53.903‖ WARNING: CBL_Pusher[https://sync.mandelkind.biz:4984/biz]: Couldn't get local contents of {-hZTLhYojFIQjEkOxMFgRJB #1-cefce94614ae803e05594b95a7f60f47}

Hm. This warning indicates that the replicator has identified this revision as one that should be pushed, (it exists locally but not on the server), but then failed to read the contents of the revision to send it.

Sounds like a race condition, where you purged the doc while a push replication was running? This is sort of inevitable; purge is a blunt instrument, kind of a “memory eraser” as in Eternal Sunshine Of The Spotless Mind, and it can have odd (but non-fatal) side effects like this.

—Jens

[goo...@marco.betschart.name]

Hi Jens

Thank you very much for pointing me into the right direction. I'd like the "delete-all-documents-of-given-channel-from-server" approach.

Unfortunately I can't figure out how I'd exactly do this.

I've found the Delete design documents API description, but can't see how I'd pass the channel id as argument?

Hoping you can give me a brief hint.

Thank you very much!!

[goo...@marco.betschart.name]

Ok; I think I got it.

It seems I'm able to read all documents using this:

http://developer.couchbase.com/documentation/mobile/1.1.0/develop/references/sync-gateway/rest-api/database/get-all-docs/index.html

And afterwards deleting every single document using the following request for each one:

http://developer.couchbase.com/documentation/mobile/1.1.0/develop/references/sync-gateway/rest-api/document/delete---db---doc-/index.html

[goo...@marco.betschart.name]

*grml* - still no luck using those API's. And I truly running out of ideas :(

Do you have any idea what's going wrong here?

The sad news are, the deletion of the remote documents doesn't seem to have any effect at all :(

I've got the following databaseRestoreBackup func:

func databaseRestoreBackup(database: CBLDatabase, from date: NSDate, withManager databaseManager: CBLManager) throws -> CBLDatabase?{

...

//delete the old database

try databaseDelete(database, withManager: databaseManager)

//Unzip backup to the manager directory

SSZipArchive.unzipFileAtPath(sourcePath, toDestination: databaseManager.directory)

//create new revisions of the restored documents

let database = try databaseManager.databaseNamed(database.name)

let enumerator = try database.createAllDocumentsQuery().run()

for i in 0..<enumerator.count{

if let doc = enumerator.rowAtIndex(i).document {

try doc.newRevision().save()

}

}

//purge all remote documents

try PersistenceManager.sharedInstance().purgeRemoteDocuments(channel: Identity.sharedInstance().tenant, completion: { (result: AnyObject?) -> Void in

do{

try self.databaseStartReplication(database, completion: { (notification) -> Void in

PersistenceManager.sharedInstance().databaseSetup(database, withManager: databaseManager)

})

} catch let error as NSError{

NSLog(error.localizedDescription)

}

})

...

}

Which calls the following purgeRemoteDocuments func (Is there a better way reading only the docs of the given channel?):

func purgeRemoteDocuments(channel channel: String, completion: CompletionHandler? = nil) throws{

guard let serverURL = self.replicationConfig["url"] as? NSURL,

let serverDatabase = self.replicationConfig["database"] as? String,

let serverUsername = self.replicationConfig["username"] as? String,

let serverPassword = self.replicationConfig["password"] as? String

else { throw PersistenceManagerError.ConfigInvalid }

Alamofire.request(.GET,

serverURL

.URLByAppendingPathComponent(serverDatabase)

.URLByAppendingPathComponent("_all_docs"),

parameters:[ "channels":"true" ])

.authenticate(user: serverUsername, password: serverPassword)

.responseJSON { response in switch response.result {

case .Success(let data):

let json = JSON(data)

if let documents = json["rows"].array{

for document in documents{

if let channels = document["value"]["channels"].arrayObject as? [String]{

if channels.indexOf(channel) != nil{

if let docID = document["id"].string{

Alamofire.request(.DELETE,

serverURL

.URLByAppendingPathComponent(serverDatabase)

.URLByAppendingPathComponent(docID))

.authenticate(user: serverUsername, password: serverPassword)

.responseJSON { response in switch response.result {

case .Success(_):

NSLog("Deleted remote document with id '\(docID)'")

case .Failure(let error):

NSLog(error.localizedDescription)

}

}

}

}

}

}

}

if let handler = completion{

handler(result: data)

}

case .Failure(let error):

NSLog(error.localizedDescription)

if let handler = completion{

handler(result: error)

}

}

}

}

[Jens Alfke]

On Nov 22, 2015, at 11:41 AM, goo...@marco.betschart.name wrote:

And afterwards deleting every single document using the following request for each one:

http://developer.couchbase.com/documentation/mobile/1.1.0/develop/references/sync-gateway/rest-api/document/delete---db---doc-/index.html

Deleting a replicated doc adds a new ‘tombstone’ revision that marks the deletion. You don’t want that. (In a nutshell, this tombstone will be newer than your old revision you’re restoring, so it takes precedence over it.)

What you want to do is purge the server-side doc; this literally deletes it and all its revisions from storage, causing the server to forget about it completely. The problem is that Sync Gateway currently doesn’t implement the _purge call, and even if it did, it would be a privileged call (part of the admin-only API) not something a client could use.

What you’ll need to do is implement a bit of app-server code that can respond to an (authorized) request  to purge a user’s docs, and will use a Couchbase Server ‘Delete' call to erase the document from the bucket, which is pretty much equivalent to a purge.

—Jens

[goo...@marco.betschart.name]

Ok I see why deleting using the sync gateway does not work. I've now tried to delete it directly using the Couchbase Java SDK (wrapped into the ColdFusion cfcouchbase library).

But for some reason I still seem to doing something wrong - as I get back documents from the query but as soon as I try to delete them, Couchbase returns the getStatus().getMessage() as 'Not Found'.

Can you please tell me, where I've done a locigal mistake?

FYI: the couchbaseClient variable is a ColdFusion wrapper for the Java Class com.couchbase.client.CouchbaseClient.

This code is in my added view:

function(doc,meta){

if( !doc._deleted && doc.channels && doc.channels.length && doc.type != 'Setting' ){

for( var i = 0; i < doc.channels.length; i++ ){

emit(doc.channels[i],doc);

}

}

}

This is saved with the following name (designDocumentName, viewName, mapFunction):

couchbaseClient.asyncSaveView('channels','channel',... map function code from above ...)

Then I'm querying this view using the following query (designDocumentName, viewName, options):

couchbaseClient.query('channels','channel',{ key=arguments.channelId })

Last but not least: I'm looping over the result set and call the delete function for the document.id:

<cfloop array="#local.docs#" item="local.doc" index="local.index">

    <cfset future = couchbaseClient.delete(local.doc.id) />
    <cfoutput>#future.getStatus().getMessage()#</cfoutput><cfabort> <!--- this outputs 'Not Found' ---->

</cfloop>

Am Montag, 23. November 2015 00:23:25 UTC+1 schrieb Jens Alfke:

[goo...@marco.betschart.name]

Anybody any idea whats wrong? I'm totally lost and should complete the new App Version by today :(

Thanks in advance

[goo...@marco.betschart.name]

FWIW: The documents I'm trying to delete look like the the following. Wondering why the 'DOCUMENT' key is empty...?

{
DOCUMENT = '',
ID = '--2szFNTpPIAxX1X68LZo3u',

KEY = '9c0bb124365c49d48e5e0c5577f0000d',

VALUE = '{"_sync":{"rev":"2-be773aa928134657f796ce90a02caaf1","sequence":34678,"recent_sequences":[34611,34678],"history":{"revs":["1-575fde964b77a7e916f7b3f395acb4e6","2-be773aa928134657f796ce90a02caaf1"],"parents":[-1,0],"channels":[["9c0bb124365c49d48e5e0c5577f0000d"],["9c0bb124365c49d48e5e0c5577f0000d"]]},"channels":{"9c0bb124365c49d48e5e0c5577f0000d":null},"time_saved":"2015-11-24T21:38:46.558978545+01:00"},"channels":["9c0bb124365c49d48e5e0c5577f0000d"],"type":"CostCenter","value":"Marketing"}'

}

[Jens Alfke]

> On Nov 24, 2015, at 8:22 AM, goo...@marco.betschart.name wrote:
>
> Anybody any idea whats wrong? I'm totally lost and should complete the new App Version by today :(

Nope, sorry. I’m not familiar with either ColdFusion or the Couchbase SDK for it. Deleting the doc by its ID field should work. (The ‘DOCUMENT’ key isn’t part of the Couchbase document so it must be something added by the SDK?)

You might try asking on the Couchbase web forum — not in the mobile section but in a section for server stuff.

—Jens