FYI: Some nice sync-status icons

[Jens Alfke]

Google just released a set of 750 black-and-white icons for mobile and web apps. They're pretty nice, and I noticed a group of 'cloud' icons that were just what I'd been looking for as toolbar indicators of sync status:

These would be great for indicating sync in progress, sync complete, unable to sync, etc.

—Jens

[Seung Chan Lim]

What's your algorithm for setting these?

If you query the _active_tasks and check the stats could you get all of these figured out?

Are there any exceptions?

slim

[CouchbaseLover]

I agree that this would be very nice but how do you accomplish reading the sync?

[Jens Alfke]

On Oct 24, 2014, at 3:29 AM, CouchbaseLover <sharess...@gmail.com> wrote:

I agree that this would be very nice but how do you accomplish reading the sync?

CBLReplication.status

[Dominique Legault]

How could this be achieved using the phonegap app ?

I have not been able to find a way to hook into the notifyChangeListenersStateTransition callback does anyone know how ?

[Jens Alfke]

On Nov 5, 2014, at 3:19 PM, Dominique Legault <deefac...@gmail.com> wrote:

How could this be achieved using the phonegap app ?

GET /_active_tasks to find the status of the replication.

—Jens

[Juan Hernandez]

Hi Jens,

do you know if is possible to get more information about the status of the replication using the REST API?

Currently retrieving the active tasks (GET /_active_tasks) only gives something like:

{
 continuous: true
 source: "mydb"
 status: "Idle"
 target: "http://server.local/mydb"
 task: "repl002"
 type: "Replication"
}

It'd be very helpful to get the timestamp of the last status change.

It's impossible to know if my local changes have been replicated already to the server as I don't know if the status "Idle" happened before or after I made those changes.

Maybe we could have the last Source sequence ID (Checkpoint) from the replication in the active tasks object.

What do you think?

Juan

[Jens Alfke]

On Nov 10, 2014, at 10:12 AM, Juan Hernandez <juanherna...@gmail.com> wrote:

It's impossible to know if my local changes have been replicated already to the server as I don't know if the status "Idle" happened before or after I made those changes.

Yes, this same issue has come up in the native API; I added some methods (still on a branch) to ask a replication whether a specific document has been pushed yet.

Maybe we could have the last Source sequence ID (Checkpoint) from the replication in the active tasks object.

That sounds reasonable. (Do you know whether CouchDB's _active_tasks already includes this?) Please file an issue to request it.

—Jens

[Dominique]

What I would like to see is access to the Replication Change Listener through the REST API, similar to the Document Change Listener.

--
You received this message because you are subscribed to a topic in the Google Groups "Couchbase Mobile" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/mobile-couchbase/JRZbjSSDd8s/unsubscribe.
To unsubscribe from this group and all its topics, send an email to mobile-couchba...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mobile-couchbase/7EF26060-0180-474F-A157-564CB34221A6%40couchbase.com.
For more options, visit https://groups.google.com/d/optout.

[Jens Alfke]

> On Nov 10, 2014, at 10:56 AM, Dominique <deefac...@gmail.com> wrote:
>
> What I would like to see is access to the Replication Change Listener through the REST API, similar to the Document Change Listener.

If you add "?feed=continuous" to _active_tasks it'll send push updates just like the _changes feed.

—Jens

[Juan Hernandez]

Maybe we could have the last Source sequence ID (Checkpoint) from the replication in the active tasks object.

That sounds reasonable. (Do you know whether CouchDB's _active_tasks already includes this?) Please file an issue to request it.

—Jens

Was going to file the issue but it was already there:

https://github.com/couchbase/couchbase-lite-ios/issues/290

[Dominique Legault]

Hi Jens,

Thanks for hinting towards that, it brings me a little closer to what I need but, I'm still having issues with it.
In android the response I get is

{"progress":0,
 "target":"openmoney",
 "source":"https://sync.url",
 "type":"Replication",
 "status":"Processed 0 / 0 changes",
 "task":"repl002"}

this same response is sent continuously until a change happens,
then status gets updated with :
Processed 0 / 1 changes
then
Processed 1 / 1 changes
then
Processed 1 / 2 changes
then
Processed 2 / 2 changes
then
Processed 2 / 3 changes
then
Processed 2 / 2 changes
...
How can it go back to 2 / 2 changes once it has hit 2 / 3 changes ?

The changes feed for documents only sends data when it changes, the active_tasks is different in that it sends data continuously.
When I turn off access to the internet the stream of continuous responses stops.

How do I detect the various different states that the replication is in with the active_tasks API ?
Idle when there isn't a change ?
Active when changes are not equal ?
and Stopped when the stream stops ?

How do I know if a change has completed ?
do I track the number of changes I have made and compare that to the number of changes completed ?

[Jens Alfke]

On Nov 11, 2014, at 1:04 PM, Dominique Legault <deefac...@gmail.com> wrote:

How can it go back to 2 / 2 changes once it has hit 2 / 3 changes ? 
The changes feed for documents only sends data when it changes, the active_tasks is different in that it sends data continuously.

These sound like Android specific issues; hopefully Traun can respond to them. It definitely shouldn't be sending changes constantly.

How do I detect the various different states that the replication is in with the active_tasks API ?

By looking at the "status" string — it'll be either "Stopped", "Offline", "Idle", or "Processed %u / %u changes".

Not the cleanest API … it's based on CouchDB's REST API. It may be that CouchDB has added some more properties since I last looked, though.

How do I know if a change has completed ?

Not sure what you mean. How to tell when a local change has been pushed to the server? That would require the "update_seq" property which we don't provide yet, as discussed here yesterday (was it a different thread?)

—Jens

[Traun Leyden]

See inline responses below:

How can it go back to 2 / 2 changes once it has hit 2 / 3 changes ?

That's definitely a bug.  Can you file a github issue here and mention:

* Is it a pull or push replication?

* Which version of Couchbase Lite are you using?  (or a link to where you downloaded it and I can probably figure out from there)

* Is this sync'ing with a CouchDB database or a Sync Gateway?

* Is it possible to put the database or an equivalent database on a public URL so that we can test against it and try to reproduce the issue?

 

The changes feed for documents only sends data when it changes, the active_tasks is different in that it sends data continuously.
When I turn off access to the internet the stream of continuous responses stops.

I'm not sure I understand this, can you describe it in more details?  Are you saying that the HTTP response never finishes?

 

How do I detect the various different states that the replication is in with the active_tasks API ?
Idle when there isn't a change ?
Active when changes are not equal ?
and Stopped when the stream stops ?

How do I know if a change has completed ?
do I track the number of changes I have made and compare that to the number of changes completed ?

On Monday, 10 November 2014 11:12:03 UTC-8, Jens Alfke wrote:

> On Nov 10, 2014, at 10:56 AM, Dominique <deefac...@gmail.com> wrote:
>
> What I would like to see is access to the Replication Change Listener through the REST API, similar to the Document Change Listener.

If you add "?feed=continuous" to _active_tasks it'll send push updates just like the _changes feed.

—Jens

--
You received this message because you are subscribed to the Google Groups "Couchbase Mobile" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mobile-couchba...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mobile-couchbase/24bf3b32-bb3a-4df4-ac5e-78481382345d%40googlegroups.com.

[Jens Alfke]

On Nov 11, 2014, at 2:08 PM, Traun Leyden <traun....@gmail.com> wrote:

I'm not sure I understand this, can you describe it in more details?  Are you saying that the HTTP response never finishes?

He's using GET /_active_tasks?feed=continuous — this is an enhancement Chris and I came up with that provides a "push" style REST API to replication progress. Just like the _changes feed, it keeps the response open and sends a JSON object followed by a newline every time the status changes. (See -[CBLRouter replicationChanged:] in the iOS code.) It sounds like the Android equivalent code may just be using a timer to send periodic updates.

—Jens

[Dominique Legault]

See my responses inline:

On Tuesday, 11 November 2014 14:08:25 UTC-8, Traun Leyden wrote:

See inline responses below:

How can it go back to 2 / 2 changes once it has hit 2 / 3 changes ?

That's definitely a bug.  Can you file a github issue here and mention:

* Is it a pull or push replication?

* Which version of Couchbase Lite are you using?  (or a link to where you downloaded it and I can probably figure out from there)

* Is this sync'ing with a CouchDB database or a Sync Gateway?

* Is it possible to put the database or an equivalent database on a public URL so that we can test against it and try to reproduce the issue?

I'll try and reproduce the issue to see if it comes up. It may have been that push replication surpassed the pull replication at that time, I'll get the logs this time.
Most recent version by doing this four days ago.

phonegap plugin remove com.couchbase.lite.phonegap
phonegap local plugin add https://github.com/couchbaselabs/Couchbase-Lite-PhoneGap-Plugin.git

I am doing a sync to a sync_gateway
The DB is on a public url, here is the source code:
https://github.com/deefactorial/openmoney-mobile/

Let me do some preliminary testing to see if I can reproduce the error and I'll create a bug report with the logcat.

 

The changes feed for documents only sends data when it changes, the active_tasks is different in that it sends data continuously.
When I turn off access to the internet the stream of continuous responses stops.

I'm not sure I understand this, can you describe it in more details?  Are you saying that the HTTP response never finishes?

 

When I get the active_tasks with feed=continuous I get a stream of responses, often the data hasn't changed. It will be the same response every time, until a document is changed in my local db, Why does it send continuous responses when the response data has not changed.
Here is an example log:

11-12 11:58:44.569: I/Web Console(3378): task{"progress":100,"target":"https://deefactorial%2B8%40gmail.com:123456@cloud.openmoney.cc:4984/openmoney_shadow/","source":"openmoney","type":"Replication","status":"Processed 168 / 168 changes","task":"repl001"} at file:///android_asset/www/js/index.js:5861
11-12 11:58:44.569: I/Web Console(3378): push sync connected handler called at file:///android_asset/www/js/index.js:5861
11-12 11:58:44.620: I/Web Console(3378): task{"progress":100,"target":"openmoney","source":"https://deefactorial%2B8%40gmail.com:123456@cloud.openmoney.cc:4984/openmoney_shadow/","type":"Replication","status":"Processed 6 / 6 changes","task":"repl002"} at file:///android_asset/www/js/index.js:5861
11-12 11:58:44.620: I/Web Console(3378): pull sync connected handler called at file:///android_asset/www/js/index.js:5861
11-12 11:58:44.846: I/Web Console(3378): task{"progress":100,"target":"https://deefactorial%2B8%40gmail.com:123456@cloud.openmoney.cc:4984/openmoney_shadow/","source":"openmoney","type":"Replication","status":"Processed 168 / 168 changes","task":"repl001"} at file:///android_asset/www/js/index.js:5861
11-12 11:58:44.846: I/Web Console(3378): push sync connected handler called at file:///android_asset/www/js/index.js:5861
11-12 11:58:44.858: I/Web Console(3378): task{"progress":100,"target":"openmoney","source":"https://deefactorial%2B8%40gmail.com:123456@cloud.openmoney.cc:4984/openmoney_shadow/","type":"Replication","status":"Processed 6 / 6 changes","task":"repl002"} at file:///android_asset/www/js/index.js:5861
11-12 11:58:44.858: I/Web Console(3378): pull sync connected handler called at file:///android_asset/www/js/index.js:5861
11-12 11:58:45.124: I/Web Console(3378): task{"progress":100,"target":"https://deefactorial%2B8%40gmail.com:123456@cloud.openmoney.cc:4984/openmoney_shadow/","source":"openmoney","type":"Replication","status":"Processed 168 / 168 changes","task":"repl001"} at file:///android_asset/www/js/index.js:5861
11-12 11:58:45.128: I/Web Console(3378): push sync connected handler called at file:///android_asset/www/js/index.js:5861

Notice how all those responses happened within 1 second, and there isn't anything new in each of the responses.

I have not seen any responses where the status has been anything but "Processed [number] / [number] changes" in the Android version.
When the replication stops (wifi connection disabled) the stream of responses stops, there is no response that indicates it has stopped other than the stream of responses has stopped.

[Dominique Legault]

What about adding:

String status = String.format("Processed %d / %d changes", processed, total);
if (replicator.getStatus().name() != ReplicationStatus.REPLICATION_ACTIVE) {
    status = replicator.getStatus().name();
}

to this line
https://github.com/couchbase/couchbase-lite-java-core/blob/master/src/main/java/com/couchbase/lite/router/Router.java#L698

At least I would be able to get the current status of the replication, and I could assume if it was Processed... that it was Active.

[Dominique Legault]

sorry string compare in java is different.

String status = String.format("Processed %d / %d changes", processed, total);

if (! replicator.getStatus().name().equals( ReplicationStatus.REPLICATION_ACTIVE ) ) {
   status = replicator.getStatus().name();
}

[Dominique Legault]

I have finally been dynamically update my UI using phonegap. I had to update couchbase-lite-java library REST API to implement the feed=continous and feed=longpoll parameters:

Here are the changes I made to the Router.java file to make it work:

    public Status do_GET_Document_active_tasks(Database _db, String _docID, String _attachmentName) {
        return do_GET_active_tasks(_db, _docID, _attachmentName);
    }
   
    private boolean longpollReplication = false;
   
    public Status do_GET_active_tasks(Database _db, String _docID, String _attachmentName) {
        // http://wiki.apache.org/couchdb/HttpGetActiveTasks
        String feed = getQuery("feed");
         
        longpollReplication = "longpoll".equals(feed);
        boolean continuous = !longpollReplication && "continuous".equals(feed);
       
        String session_id = getQuery("session_id");
       
        ChangeListener listener = new ChangeListener() {
           
            ChangeListener self = this;
               
            @Override
            public void changed(ChangeEvent event) {
               
                Map<String,Object> activity = getActivity( event.getSource() );
               
                if (event.getTransition() != null ) {
                       
                    activity.put("transition_source", event.getTransition().getSource());
                    activity.put("transition_destination", event.getTransition().getDestination());
                    activity.put("trigger", event.getTransition().getTrigger() );
                    Log.d(Log.TAG_ROUTER, "do_GET_active_tasks Transition [" + event.getTransition().getTrigger() + "] Source:" + event.getTransition().getSource() + ", Destination:" + event.getTransition().getDestination() );
                   
                }
               
                if(longpollReplication) {
                    Log.w(Log.TAG_ROUTER, "Router: Sending longpoll replication response");
                    sendResponse();
                   
                    if(callbackBlock != null) {
                        byte[] data = null;
                        try {
                            data = Manager.getObjectMapper().writeValueAsBytes(activity);
                        } catch (Exception e) {
                            Log.w(Log.TAG_ROUTER, "Error serializing JSON", e);
                        }
                        OutputStream os = connection.getResponseOutputStream();
                        try {
                            os.write(data);
                            os.close();
                        } catch (IOException e) {
                            Log.e(Log.TAG_ROUTER, "IOException writing to internal streams", e);
                        }
                    }
                    //remove this change listener because a new one will be added when this responds
                    event.getSource().removeChangeListener(self);
                } else {
                    Log.w(Log.TAG_ROUTER, "Router: Sending continous replication change chunk");
                    sendContinuousReplicationChanges(activity);
                }
               
               
               
            }
        };
       
       
        List<Map<String,Object>> activities = new ArrayList<Map<String,Object>>();
        for (Database db : manager.allOpenDatabases()) {
            List<Replication> allReplicators = db.getAllReplications();
            if(allReplicators != null) {
                for (Replication replicator : allReplicators) {
                   
                    Map<String,Object> activity = getActivity( replicator );
                   
                    activities.add(activity);
                   
                    if (continuous || longpollReplication) {
                       
                        if (session_id != null) {
                            if (replicator.getSessionID().equals(session_id)){
                                replicator.addChangeListener(listener);
                            }
                        } else {
                            replicator.addChangeListener(listener);
                        }
                    }
                }
            }
        }
       

        if (continuous || longpollReplication) {
           
            connection.setChunked(true);
            connection.setResponseCode(Status.OK);
            sendResponse();
           
            if (continuous && !activities.isEmpty()) {
                for (Map<String,Object> activity : activities) {
                    sendContinuousReplicationChanges(activity);
                }
            }
           
            // Don't close connection; more data to come
            return new Status(0);
        } else {
            connection.setResponseBody(new Body(activities));
            return new Status(Status.OK);
        }
    }
   
    private Map<String,Object> getActivity(Replication replicator) {
       
        Map<String,Object> activity = new HashMap<String,Object>();
       
        String source = replicator.getRemoteUrl().toExternalForm();
        String target = replicator.getLocalDatabase().getName();        
       
        if(!replicator.isPull()) {
            String tmp = source;
            source = target;
            target = tmp;
        }
        int processed = replicator.getCompletedChangesCount();
        int total = replicator.getChangesCount();

        String status = String.format("Processed %d / %d changes", processed, total);

        if (! replicator.getStatus().name().equals( ReplicationStatus.REPLICATION_ACTIVE.name() ) ) {
           status = replicator.getStatus().name();
        }
        int progress = (total > 0) ? Math.round(100 * processed / (float)total) : 0;
       
        activity.put("type", "Replication");
        activity.put("task", replicator.getSessionID());
        activity.put("source", source);
        activity.put("target", target);
       
        if (replicator.getLastError() != null) {
            String msg = String.format("Replicator error: %s.  Repl: %s.  Source: %s, Target: %s",
                    replicator.getLastError(), replicator, source, target);
            Log.e(Log.TAG_ROUTER, msg);
            Throwable error = replicator.getLastError();
            int statusCode = 400;
            if (error instanceof HttpResponseException) {
                statusCode = ((HttpResponseException)error).getStatusCode();
            }
            Object[] errorObjects = new Object[]{ statusCode, replicator.getLastError().toString() };
            activity.put("error", errorObjects);
            activity.put("status", Replication.ReplicationStatus.REPLICATION_STOPPED);
            activity.put("progress", null);
            activity.put("change_count", null);
            activity.put("completed_change_count", null);
        } else {
            activity.put("status", status);
            activity.put("progress", progress);
            activity.put("change_count", total);
            activity.put("completed_change_count", processed);
        }

        return activity;
    }
   
    public void sendContinuousReplicationChanges(Map<String,Object> activity) {
        try {
            String jsonString = Manager.getObjectMapper().writeValueAsString(activity);
            if(callbackBlock != null) {
                byte[] json = (jsonString + "\n").getBytes();
                OutputStream os = connection.getResponseOutputStream();
                try {
                    os.write(json);
                    os.flush();
                } catch (Exception e) {
                    Log.e(Log.TAG_ROUTER, "IOException writing to internal streams", e);
                }
            }
        } catch (Exception e) {
            Log.w("Unable to serialize change to JSON", e);
        }
    }

If you would like to see how I implemented it in my phonegap app. you can check out my source code here.
https://github.com/deefactorial/openmoney-mobile/

I was not able to get feed=continuous to work in my phonegap app because it does not work for the changes API, But longpoll works quite nicely.

[Jens Alfke]

On Nov 16, 2014, at 1:33 PM, Dominique Legault <deefac...@gmail.com> wrote:

I have finally been dynamically update my UI using phonegap. I had to update couchbase-lite-java library REST API to implement the feed=continous and feed=longpoll parameters:

If you want to propose changes to our code, you should submit a pull request. Thanks!

—Jens

[Dominique Legault]

Here is the link to the pull request.

https://github.com/couchbase/couchbase-lite-java-core/pull/333

[Traun Leyden]

Thanks, I'll take a look.

--

You received this message because you are subscribed to the Google Groups "Couchbase Mobile" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mobile-couchba...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/mobile-couchbase/d05c7ef7-9861-464a-b9d0-2e187fdef3c1%40googlegroups.com.