Pull Replication Stuck on TouchDB

[Sameer Segal]

Hi, I am trying to do filtered pull replication from CouchDB (on AWS) to TouchDB-Android. When I install the app it works perfectly then I exit the activity/service and from then on when I try to re-initiate the pull replication it just looks stuck to me:

V/TDDatabase(20946): TDPusher [http://couch.domain.com/dbname] STARTING ...

V/TDDatabase(20946): TDPusher [http://couch.domain.com/dbname]: GET ./_local/1aa6cee354449b761ccd8acc8a26878349ed03d2

V/TDDatabase(20946): com.couchbase.touchdb.support.TDRemoteRequest@406377b8: GET .http://couch.domain.com/dbname/_local/1aa6cee354449b761ccd8acc8a26878349ed03d2

V/TDDatabase(20946): TDPuller [http://couch.domain.com/dbname] STARTING ...

V/TDDatabase(20946): TDPuller [http://couch.domain.com/dbname]: GET ./_local/960868f6495d3584f9f42fd560ee63f6850bfe85

V/TDDatabase(20946): com.couchbase.touchdb.support.TDRemoteRequest@40574748: GET .http://couch.domain.com/dbname/_local/960868f6495d3584f9f42fd560ee63f6850bfe85

V/TDDatabase(20946): com.couchbase.touchdb.replicator.TDReplicator$4@4056e930: Replicating from lastSequence=177

V/TDDatabase(20946): com.couchbase.touchdb.replicator.TDReplicator$4@405e4520: lastSequence mismatch: I had 101214, remote had 101195

W/TDDatabase(20946): TDPuller [http://couch.domain.com/dbname] starting ChangeTracker with since=null

V/TDDatabase(20946): Making request to http://couch.domain.com/dbname/_changes?feed=longpoll&heartbeat=300000&filter=common%2FbyContext&context=XYZ

V/TDDatabase(20946): Re-indexing view common/byName ...

V/TDDatabase(20946): Begin transaction (level 1)...

V/TDDatabase(20946): Committing transaction (level 1)...

V/TDDatabase(20946): Query common/byName: SELECT key, value, docid FROM maps, revs, docs WHERE maps.view_id=? AND revs.sequence = maps.sequence AND docs.doc_id = revs.doc_id ORDER BY key LIMIT ? OFFSET ?

V/TDDatabase(20946): TDPusher [http://couch.domain.com/dbname] STARTING ...

V/TDDatabase(20946): TDPusher [http://couch.domain.com/dbname]: GET ./_local/1aa6cee354449b761ccd8acc8a26878349ed03d2

V/TDDatabase(20946): com.couchbase.touchdb.support.TDRemoteRequest@4066d660: GET .http://couch.domain.com/dbname/_local/1aa6cee354449b761ccd8acc8a26878349ed03d2

V/TDDatabase(20946): TDPuller [http://couch.domain.com/dbname] STARTING ...

V/TDDatabase(20946): TDPuller [http://couch.domain.com/dbname]: GET ./_local/960868f6495d3584f9f42fd560ee63f6850bfe85

V/TDDatabase(20946): com.couchbase.touchdb.support.TDRemoteRequest@407646e0: GET .http://couch.domain.com/dbname/_local/960868f6495d3584f9f42fd560ee63f6850bfe85

V/TDDatabase(20946): com.couchbase.touchdb.replicator.TDReplicator$4@405772a8: Replicating from lastSequence=177

V/TDDatabase(20946): com.couchbase.touchdb.replicator.TDReplicator$4@4070be78: lastSequence mismatch: I had 101214, remote had 101195

W/TDDatabase(20946): TDPuller [http://couch.domain.com/dbname] starting ChangeTracker with since=null

V/TDDatabase(20946): Making request to http://couch.domain.com/dbname/_changes?feed=longpoll&heartbeat=300000&filter=common%2FbyContext&context=XYZ

Additionally, is there a way to progress of replication (2000 out of 6000) docs/changes done? Is there a way to listen to changes of the Replication Status?

Best, Sameer

[Marty Schoch]

On Fri, Jul 6, 2012 at 1:50 PM, Sameer Segal <sam...@artoo.in> wrote:

> V/TDDatabase(20946):
> com.couchbase.touchdb.replicator.TDReplicator$4@4070be78: lastSequence
> mismatch: I had 101214, remote had 101195
>
> W/TDDatabase(20946): TDPuller [http://couch.domain.com/dbname] starting
> ChangeTracker with since=null

It's possible there is a bug in the way we handle the sequence number
mismatches here. It looks like it started over again from the
beginning, and that might make it look like it was stuck for a long
period of time, until it starting comparing revisions newer than
101195. Can you put this info in an issue on github?

>
> V/TDDatabase(20946): Making request to
> http://couch.domain.com/dbname/_changes?feed=longpoll&heartbeat=300000&filter=common%2FbyContext&context=XYZ
>
> Additionally, is there a way to progress of replication (2000 out of 6000)
> docs/changes done? Is there a way to listen to changes of the Replication
> Status?

Our TDRouter implementation implements the _active_tasks API which has
some information about the progress, but Ektorp doesn't have an API to
query it. So for right now, the only way is to query the TDReplicator
objects directly which isn't something I really want to support.

I think when we add support for the _replicator db, you'll be able to
query the progress there as well. This is will probably where this
feature shows up first.

marty

[Sameer Segal]

On Friday, July 6, 2012 11:37:40 PM UTC+5:30, Marty Schoch wrote:

On Fri, Jul 6, 2012 at 1:50 PM, Sameer Segal <sam...@artoo.in> wrote:

> V/TDDatabase(20946):
> com.couchbase.touchdb.replicator.TDReplicator$4@4070be78: lastSequence
> mismatch: I had 101214, remote had 101195
>
> W/TDDatabase(20946): TDPuller [http://couch.domain.com/dbname] starting
> ChangeTracker with since=null

It's possible there is a bug in the way we handle the sequence number
mismatches here.  It looks like it started over again from the
beginning, and that might make it look like it was stuck for a long
period of time, until it starting comparing revisions newer than
101195.  Can you put this info in an issue on github?

Reported this issues as https://github.com/couchbaselabs/TouchDB-Android/issues/47 

>
> V/TDDatabase(20946): Making request to
> http://couch.domain.com/dbname/_changes?feed=longpoll&heartbeat=300000&filter=common%2FbyContext&context=XYZ
>
> Additionally, is there a way to progress of replication (2000 out of 6000)
> docs/changes done? Is there a way to listen to changes of the Replication
> Status?

Our TDRouter implementation implements the _active_tasks API which has
some information about the progress, but Ektorp doesn't have an API to
query it.  So for right now, the only way is to query the TDReplicator
objects directly which isn't something I really want to support.

I think when we add support for the _replicator db, you'll be able to
query the progress there as well.  This is will probably where this
feature shows up first.

How do I query TDReplicator? I am using ReplicationCommand to trigger push/pull replication.

Additionally, it had been some time since I had taken TouchDB code. I update my local code and ever since I get the following error android.database.sqlite.SQLiteException due to:

E/Database(22856): Failure 1 (table docs already exists) on 0x250528 when executing 'CREATE TABLE docs (         doc_id INTEGER PRIMARY KEY,         docid TEXT UNIQUE NOT NULL)'

marty

[Marty Schoch]

On Fri, Jul 6, 2012 at 2:43 PM, Sameer Segal <sam...@artoo.in> wrote:

> How do I query TDReplicator? I am using ReplicationCommand to trigger
> push/pull replication.

After starting the replication with ReplicationCommand, you can
directly get the TDDatabse from the TDServer, then get the TDRepliator
by calling tdDatabase.getReplicator(). If you pass in the same
parameters as you used to create the ReplicationCommand, you'll get
the existing replicator, and not a new one. You can then look at
changesTotal and changesProcessed to estimate progress.

I don't recommend you rely on this, the TouchDB-Android package is not
intended to be used directly by applications.

> Additionally, it had been some time since I had taken TouchDB code. I update
> my local code and ever since I get the following error
> android.database.sqlite.SQLiteException due to:
>
> E/Database(22856): Failure 1 (table docs already exists) on 0x250528 when
> executing 'CREATE TABLE docs ( doc_id INTEGER PRIMARY KEY,
> docid TEXT UNIQUE NOT NULL)'

Looks like it attempted a schema upgrade and failed. CREATE TABLE
docs should only ever run on initial schema creation though. Can you
run "pragma user_version;" within sqlite3 and see what it reports?

marty

[Sameer Segal]

On Saturday, July 7, 2012 12:30:33 AM UTC+5:30, Marty Schoch wrote:

On Fri, Jul 6, 2012 at 2:43 PM, Sameer Segal <sam...@artoo.in> wrote:

> How do I query TDReplicator? I am using ReplicationCommand to trigger
> push/pull replication.

After starting the replication with ReplicationCommand, you can
directly get the TDDatabse from the TDServer, then get the TDRepliator
by calling tdDatabase.getReplicator().  If you pass in the same
parameters as you used to create the ReplicationCommand, you'll get
the existing replicator, and not a new one.  You can then look at
changesTotal and changesProcessed to estimate progress.

I don't recommend you rely on this, the TouchDB-Android package is not
intended to be used directly by applications.

>> I understand that this will change as the APIs mature and may even break all of a sudden. I understand the risk but require this for the short-term 

> Additionally, it had been some time since I had taken TouchDB code. I update
> my local code and ever since I get the following error
> android.database.sqlite.SQLiteException due to:
>
> E/Database(22856): Failure 1 (table docs already exists) on 0x250528 when
> executing 'CREATE TABLE docs (         doc_id INTEGER PRIMARY KEY,
> docid TEXT UNIQUE NOT NULL)'

Looks like it attempted a schema upgrade and failed.  CREATE TABLE
docs should only ever run on initial schema creation though.  Can you
run "pragma user_version;" within sqlite3 and see what it reports?

>> Interesting it started working all of a sudden 

marty