Two objects from _active_tasks with replication?

[Ryan Meyer]

The problem I'm having is that I can't monitor total/completed documents when replicating.

I'm using CouchCocoa, BigCouch, Couchbase Mobile (maybe soon to switch to TouchDB).  

When I do a replication the documents come down fine, but I can't observe the status of the replication.  When I look at the HTTP traffic I see calls to /_active_tasks and I see two objects in a JSON array.  Both are JSON objects and both have type "Replication".  When I look at the CouchCocoa code I see it loops through this array and then checks the status field.

The first element in the array has a very bizarre status value of something like this "Processed [113,      <<"g1AAAAAA...".  The second element in the JSON array is what I want.  The status value there is "Processed 27 / 1 changes".  I can tell CouchCocoa can parse that just fine.

Just curious if this is an issue with BigCouch or ... something else?  I'd prefer not to have to try and fork CouchCocoa for my own use to parse the correct element.

[Jens Alfke]

On Jan 24, 2013, at 5:01 PM, Ryan Meyer <ryan...@gmail.com> wrote:

The problem I'm having is that I can't monitor total/completed documents when replicating.

By “I can’t monitor” do you mean that the progress API in CouchReplication doesn’t do what you want, or isn’t working, or…?

Just curious if this is an issue with BigCouch or ... something else?

The _active_tasks URL that CouchCocoa is checking is on your local embedded CouchDB, not BigCouch.

The first element in the array has a very bizarre status value of something like this "Processed [113,      <<"g1AAAAAA...".  The second element in the JSON array is what I want.

Here are the docs on what the output of _active_tasks looks like:

http://wiki.apache.org/couchdb/HttpGetActiveTasks

But what you’re describing doesn’t sound anything like that.

—Jens

[Ryan Meyer]

Man you reply fast!  =)

Sorry my original post was light on details.  Here is the JSON being returned:

[
    {
        "type": "Replication",
        "task": "`514f47c512e356180861e7d38034a005+continuous`: `http://<some ip>:5984/feeds/` -> `feeds`",
        "started_on": 1359077238,
        "updated_on": 1359077241,
        "status": "Processed [113,\n           <<\"g1AAAAGTeJzLYWBg4MhgTmGQTMpMT84vTc4wdMgs0DU00DU10DW0NNe1MMoBKmFKZEiS____f1YiFxGKkxSAZJI9WD0bMeodQOrjweqZiVGfAFJfD1bPSoT6PBYgydAApIBa5mclihKtZwFEz_6sRB6i9RyA6LlPnF8geh5A9AD9Y5oFAOzYcW8\">>] / [113,\n                                                                                                                                                                                                             <<\"g1AAAAGZeJzLYWBg4MhgTmGQSspMT84vTc4wcsgs0DU00DW0NNU1MjLTNTTOAaphSmRIkv___39WIhcxqpMUgGSSPVgDG1EaHEAa4sEamInSkADSUA_WwEqMhjwWIMnQAKSAeuZnJYoSr2kBRNP-rEQekCZJmCZDqCZTkD5zXQsjJD0HIHruQ71DlJ4HED1AH5lmAQAB1nKJ\">>] changes",
        "pid": "<0.189.0>"
    },
    {
        "type": "Replication",
        "task": "`48d6e75cb8fbf72d8d9f5af5523e51ea+continuous`: `feeds` -> `http://<some ip>:5984/feeds/`",
        "started_on": 1359077239,
        "updated_on": 1359077240,
        "status": "Processed 27 / 1 changes",
        "pid": "<0.214.0>"
    }
]

As best I can tell, CouchCocoa loops through this array and checks for type = Replication.  We have two of those and when it goes through the first element in the array, it can't parse the "status" field.  I found a different post from a few years ago saying this is a known issue with Cloudant.

But if it could focus on the second element in the array ... that status field seems to be parseable from CouchCocoa.  Is that what I would need?

[Jens Alfke]

On Jan 24, 2013, at 5:33 PM, Ryan Meyer <ryan...@gmail.com> wrote:

        "status": "Processed [113,\n           <<\"g1AAAAGTeJzLYWBg4MhgTmGQTMpMT84vTc4wdMgs0DU00DU10DW0NNe1MMoBKmFKZEiS____f1YiFxGKkxSAZJI9WD0bMeodQOrjweqZiVGfAFJfD1bPSoT6PBYgydAApIBa5mclihKtZwFEz_6sRB6i9RyA6LlPnF8geh5A9AD9Y5oFAOzYcW8\">>] / [113,\n                                                                                                                                                                                                             <<\"g1AAAAGZeJzLYWBg4MhgTmGQSspMT84vTc4wcsgs0DU00DW0NNU1MjLTNTTOAaphSmRIkv___39WIhcxqpMUgGSSPVgDG1EaHEAa4sEamInSkADSUA_WwEqMhjwWIMnQAKSAeuZnJYoSr2kBRNP-rEQekCZJmCZDqCZTkD5zXQsjJD0HIHruQ71DlJ4HED1AH5lmAQAB1nKJ\">>] changes",

Oh, I see. Everything in the “[…]” is a sequence ID from Cloudant that marks the current checkpoint of the replication. Regular CouchDB uses integers as sequence IDs, but Cloudant being clustered has a more complex format that’s actually a JSON array. It looks like whatever code in CouchDB that formats this isn’t doing a good job — the “<<…>>” syntax is actually from Erlang.

The code in CouchReplication that picks apart that “Processed…” string would have to be tweaked to handle this. Or you could just upgrade to TouchDB, which doesn’t have this problem.

But if it could focus on the second element in the array ... that status field seems to be parseable from CouchCocoa.  Is that what I would need?

Well, there are two replications (a push and a pull). Each item in the array describes one of them. The one you’re having trouble with is the pull. CouchCocoa is probably parsing the other one just fine already.

—Jens

[Ryan Meyer]

Makes perfect sense.  One object for pull and one for push.  I wasn't sure ... I thought maybe BigCouch introduced one for managing documents across the cluster.

Looks like it's time to switch over to TouchDB.  

Thanks for the info!

[Jens Alfke]

On Jan 25, 2013, at 9:40 AM, Ryan Meyer <ryan...@gmail.com> wrote:

Makes perfect sense.  One object for pull and one for push.  I wasn't sure ... I thought maybe BigCouch introduced one for managing documents across the cluster.

No; you’re seeing the activities list for the local TouchDB server, not for BigCouch.

—Jens

[Ami Kapadia]

Hi! I am using CouchBaseLite database. When I request with "_activity_tasks", I get this array :

[

{

continuous: true

source: "test1"

status: "Idle"

target: "xxx"

task: "repl003"

type: "Replication"

}, 

{

continuous: true

progress: 21

source: "xxx"

status: "Processed 10 / 48 changes"

target: "test1"

task: "repl004"

type: "Replication"

}

]

I don't get data like "started_on" or "updated_on". I want to display data like "Last sync time" and "All data are synced or not". So, how will I get them?

Thanks!

Ami

[Jens Alfke]

On Dec 17, 2014, at 9:46 AM, Ami Kapadia <ami....@gmail.com> wrote:

I don't get data like "started_on" or "updated_on". Any idea on this? I want to display data like "Last sync time" and "All data are synced or not". So,  those data will be helpful.

Are those properties that are in CouchDB's _active_tasks? I don't think we've implemented them.

We may have ignored "started_on" because your app starts the replication itself, and if capturing the time is important, it can do that itself.

You can tell if all data is synced: the "status" property will change to "Idle". (Or in the case of a non-continuous replication, the task will stop.)

—Jens

[Ami Kapadia]

On Thursday, December 18, 2014 12:00:22 AM UTC+5:30, Jens Alfke wrote:

On Dec 17, 2014, at 9:46 AM, Ami Kapadia <ami....@gmail.com> wrote:

I don't get data like "started_on" or "updated_on". Any idea on this? I want to display data like "Last sync time" and "All data are synced or not". So,  those data will be helpful.

Are those properties that are in CouchDB's _active_tasks? I don't think we've implemented them.

We may have ignored "started_on" because your app starts the replication itself, and if capturing the time is important, it can do that itself.

What if I want last sync time? Is there any property or direct way to get it? 

You can tell if all data is synced: the "status" property will change to "Idle". (Or in the case of a non-continuous replication, the task will stop.)

If we get offline, and I am adding some data, do we get status like "these many records are remaining to sync"? 

—Jens

[Jens Alfke]

On Dec 18, 2014, at 5:31 AM, Ami Kapadia <ami....@gmail.com> wrote:

What if I want last sync time? Is there any property or direct way to get it? 

No, the replicator doesn't track that. You can store it yourself.

If we get offline, and I am adding some data, do we get status like "these many records are remaining to sync"? 

No. There's an unmerged branch named "feature/unpushed" that adds a [native] API to get that status, though, but it doesn't expose that via the REST API. (I should merge that branch for 1.1…)

—Jens

[Traun Leyden]

No. There's an unmerged branch named "feature/unpushed" that adds a [native] API to get that status, though, but it doesn't expose that via the REST API. (I should merge that branch for 1.1…)

That is a really useful feature.  What is blocking getting this merged into master?