Sync Failing - JSON error parsing _changes feed

[Steve]

Couchbase Lite is failing to sync with the following error:

2014-11-28 14:06:50.537 GoGas[26614:1110004] self.push is NOT nil
2014-11-28 14:06:50.537 GoGas[26614:1110004] Push sync progress: 0 / 0
2014-11-28 14:06:50.538 GoGas[26614:1110004] Push sync is idle.
14:06:50.982‖ WARNING*** : JSON error parsing _changes feed: parse error: client cancelled parse via callback return value

14:06:50.985‖ WARNING*** : CBLSocketChangeTracker[0x7f8c1265b790 gogas]: Can't connect, giving up: Error Domain=CBLChangeTracker Code=587 "The operation couldn’t be completed. (CBLChangeTracker error 587.)"

I have tried restarted the Sync Gateway and Couchbase Server (running on AWS), this did not help. Any ideas what might cause this, what I should be checking in my app?

Version of Sync Gateway:

01:06:18.554680 ==== Couchbase Sync Gateway/1.0.3(81;fa9a6e7) ====

[Adam Fraser]

Hi Steve,

I don't see an obvious cause based on the information you've provided.  Is there anything in the SG log when the error occurs?  Also - what client and build are you using when you see this?  

Beyond that - getting a look at the HTTP traffic would probably be the next step, to understand whether this is a case of the SG sending unexpected data in the _changes feed, or a client issue processing valid data.

Thanks,

Adam

[Jens Alfke]

I wonder if this comes from the _changes feed entries for user-access changes? Those entries don't have an _id property, and it looks as though the high-level parser (CBLChangeMatcher) will reject an entry without an _id, which aborts the parse.

—Jens

[Traun Leyden]

I checked couchbase-lite-android, and there is code to ignore _change entries without "id" fields.

Should it be doing anything with these changes, or is ignoring them the correct behavior?

On Mon, Dec 1, 2014 at 8:34 AM, Jens Alfke <je...@couchbase.com> wrote:

I wonder if this comes from the _changes feed entries for user-access changes? Those entries don't have an _id property, and it looks as though the high-level parser (CBLChangeMatcher) will reject an entry without an _id, which aborts the parse.

—Jens

--
You received this message because you are subscribed to the Google Groups "Couchbase Mobile" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mobile-couchba...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mobile-couchbase/4CAF7C00-7567-4241-86D5-A18C102DBE23%40couchbase.com.

For more options, visit https://groups.google.com/d/optout.

[Jens Alfke]

> On Dec 1, 2014, at 1:32 PM, Traun Leyden <traun....@gmail.com> wrote:
>
> Should it be doing anything with these changes, or is ignoring them the correct behavior?

Ignoring is correct.

—Jens

[Steve]

Running the app now, I get the same error(1):

2014-12-03 00:27:40.163 GoGas[29228:1565883] self.push is NOT nil
2014-12-03 00:27:40.163 GoGas[29228:1565883] Push sync progress: 0 / 0
2014-12-03 00:27:40.163 GoGas[29228:1565883] Push sync is idle.
00:27:40.534‖ WARNING*** : JSON error parsing _changes feed: parse error: client cancelled parse via callback return value

00:27:40.536‖ WARNING*** : CBLSocketChangeTracker[0x7fcc28d20660 gogas]: Can't connect, giving up: Error Domain=CBLChangeTracker Code=587 "The operation couldn’t be completed. (CBLChangeTracker error 587.)"

From the Sync Gateway log:

11:27:27.336022 ==== Couchbase Sync Gateway/1.0.3(81;fa9a6e7) ====
11:27:27.336092 Opening db /gogas as bucket "gogas", pool "default", server <http://localhost:8091>
11:27:27.336127 Opening Couchbase database gogas on <http://localhost:8091>
11:27:27.421076 Using default sync function 'channel(doc.channels)' for database "gogas"
11:27:27.421339     Reset guest user to config
11:27:27.421349 Starting admin server on 127.0.0.1:4985
11:27:27.423772 Starting server on :4984 ...
11:27:38.363695 HTTP:  #001: GET /gogas/_local/1aa559184c1fcb046e5c985c2750b8e974446e5b
11:27:38.364023 HTTP: #001:     --> 404 missing  (1.9 ms)
11:27:38.608456 HTTP:  #002: GET /gogas/_local/9f3c15f82f509e2420bd1bfbb27ddae5babc8a57
11:27:38.608638 HTTP: #002:     --> 404 missing  (0.3 ms)
11:27:38.782813 HTTP:  #003: POST /gogas/_changes
2014/12/02 11:27:38 go-couchbase: call to ViewCustom("sync_gateway", "channels") in github.com/couchbaselabs/sync_gateway/base.couchbaseBucket.ViewCustom took 207.483582ms
11:27:38.992145 changes_view: Query took 208.144095ms to return 1420 rows, options = db.Body{"stale":false, "startkey":[]interface {}{"*", 0x1}, "endkey":[]interface {}{"*", 0x59a}}

(1) = I just made some code changes to my iOS app to tidy up the code/remove duplicated code. Previously to the change I was getting a "network connection lost" error.

[Steve]

It maybe just an issue with Amazon Web Services as syncing is working if I change the Couchbase URL in my app to point to Cloudant or the local database on my laptop.

From the Sync Gateway log when syncing with the local database on my laptop:

00:51:35.059387 ==== Couchbase Sync Gateway/1.00 (1.0.1-18; commit 81a69a16) ====
00:51:35.059464 Configured Go to use all 4 CPUs; setenv GOMAXPROCS to override this
00:51:35.059485 Opening db /gogas as bucket "gogas", pool "default", server <http://blahblah.local:8091>
00:51:35.059526 Opening Couchbase database gogas on <http://blahblah.local:8091>
00:51:35.116853 Using default sync function 'channel(doc.channels)' for database "gogas"
00:51:35.119595     Reset guest user to config
00:51:35.119607 Starting admin server on 127.0.0.1:4985
00:51:35.121077 Starting server on :4984 ...
00:51:43.494157 HTTP:  #001: GET /gogas/_local/ec68e9f927a86a472d235152dde3322978611468
00:51:43.494450 HTTP: #001:     --> 404 missing  (0.0 ms)
00:51:43.495794 HTTP:  #002: GET /gogas/_local/88e749e379a9a8be0a700d89e2d1f5218c1eb2e5
00:51:43.496158 HTTP: #002:     --> 404 missing  (0.0 ms)
00:51:43.522040 HTTP:  #003: POST /gogas/_changes
00:51:43.611762 HTTP:  #004: POST /gogas/_bulk_get?revs=true&attachments=true
00:51:43.611861 HTTP:  #005: GET /gogas/_local/c71ab5c7aeaae3bc8e48576ff0171373182ee6d8
00:51:43.612696 HTTP: #005:     --> 404 missing  (0.0 ms)
00:51:43.661130 HTTP:  #007: POST /gogas/_bulk_get?revs=true&attachments=true
00:51:43.662123 HTTP:  #008: POST /gogas/_bulk_get?revs=true&attachments=true
00:51:43.667257 HTTP:  #006: POST /gogas/_bulk_get?revs=true&attachments=true
...

Why the "404 missing" entries?

What I don't understand is that previously the app was able to successfully sync with AWS; not sure what has changed.

On Friday, 28 November 2014 14:15:38 UTC+13, Steve wrote:

[Steve]

I have another app which I created for development purposes to source the data for my database and populate my remote Couchbase Server. This app now fails to populate the remote database on AWS:

2014-12-03 01:04:20.466 GoGasSync[31141:1581097] Error syncing: Error Domain=CBLHTTP Code=401 "401 unauthorized" UserInfo=0x7fdcb1516400 {NSURL=http://ec2-54-148-103-232.us-west-2.compute.amazonaws.com:4984/gogas/_local/94408eca1b65df2d376060eba75abb555b765c62, NSLocalizedFailureReason=unauthorized, NSLocalizedDescription=401 unauthorized}

This seems to me to be a AWS problem rather than a Couchbase/Sync Gateway problem.

On Friday, 28 November 2014 14:15:38 UTC+13, Steve wrote:

[Steve]

As mentioned previously I have another app which I created for development purposes to source the data for my database and populate my remote Couchbase Server. Experiencing with this app has uncovered a couple of things.

This app happily performs a push sync with the remote Couchbase Server on AWS. However, if I perform a flush of the relevant bucket and then run the app, the push sync fails with the following:

2014-12-06 01:15:28.606 GoGasSync[33802:1682508] Sync is idle.
2014-12-06 01:15:28.606 GoGasSync[33802:1682508] Error syncing: Error Domain=CBLHTTP Code=401 "401 unauthorized" UserInfo=0x7fc9ea48c770 {NSURL=http://ec2-blahblah.us-west-2.compute.amazonaws.com:4984/gogas/_local/1ab6816c4ecbf74479205114371b77f5da567bfa, NSLocalizedFailureReason=unauthorized, NSLocalizedDescription=401 unauthorized}

The reason it fails is because the flush deletes ALL documents from the bucket, including the three Sync Gateway-related documents which are added to the bucket when Sync Gateway is started (please see screenshot attached). This is also why the app runs fine upon stopping and restarting the Sync Gateway.

However, my original app is still unable to sync with the remote Couchbase Server on AWS:

2014-12-06 01:29:04.650 GoGas[33874:1688042] Inside pushReplicationProgress:]
2014-12-06 01:29:04.650 GoGas[33874:1688042] self.push is NOT nil
2014-12-06 01:29:04.651 GoGas[33874:1688042] Push sync progress: 0 / 0
2014-12-06 01:29:04.651 GoGas[33874:1688042] Push sync is idle.
01:29:05.999‖ WARNING*** : JSON error parsing _changes feed: parse error: client cancelled parse via callback return value

01:29:06.001‖ WARNING*** : CBLSocketChangeTracker[0x7fea7c8c0000 gogas]: Can't connect, giving up: Error Domain=CBLChangeTracker Code=587 "The operation couldn’t be completed. (CBLChangeTracker error 587.)"
2014-12-06 01:29:06.115 GoGas[33874:1688042] Inside replicationProgress:]
2014-12-06 01:29:06.115 GoGas[33874:1688042] Inside pullReplicationProgress:]
2014-12-06 01:29:06.116 GoGas[33874:1688042] self.pull is NOT nil
2014-12-06 01:29:06.116 GoGas[33874:1688042] Pull sync progress: 0 / 0
2014-12-06 01:29:06.116 GoGas[33874:1688042] Pull sync is idle.
2014-12-06 01:29:06.116 GoGas[33874:1688042] Pull sync error: Error Domain=CBLChangeTracker Code=587 "The operation couldn’t be completed. (CBLChangeTracker error 587.)"
2014-12-06 01:29:06.116 GoGas[33874:1688042] Inside pushReplicationProgress:]
2014-12-06 01:29:06.116 GoGas[33874:1688042] self.push is NOT nil
2014-12-06 01:29:06.122 GoGas[33874:1688042] Push sync progress: 0 / 0
2014-12-06 01:29:06.122 GoGas[33874:1688042] Push sync is idle.

[Jens Alfke]

On Dec 5, 2014, at 4:35 AM, Steve <s.ande...@gmail.com> wrote:

This app happily performs a push sync with the remote Couchbase Server on AWS. However, if I perform a flush of the relevant bucket and then run the app, the push sync fails with the following:

Um, it sounds like you flushed the bucket while Sync Gateway was running. Don't do that! It will totally mess up the Gateway, since it uses the bucket for all its storage. Always stop the gateway first.

Also, yes, flushing the bucket will completely reset the Gateway's state, including user accounts.

However, my original app is still unable to sync with the remote Couchbase Server on AWS:

#541.

—Jens

[Steve]

Thanks Jens, makes sense - you articulated it well.

I woke up today thinking if one app is syncing and the other app is not then there must be an issue with my app rather than Couchbase. So I removed the Couchbase Framework and other dependencies from the project and added them again. The app is now running without any syncing errors. I think were was some sort of version mismatch with the Couchbase Frame.

However, even though the syncing is no longer failing the app is not actually pulling any the changes from the remote Couchbase Server on AWS. I need to go over my code with a fine tooth comb, I suspect the CBLReplication (pull) object is inadvertently being set to nil.

One thing I am puzzled by, when I do a build I get 135 warnings - "unable to open object file." The path in question is /User/jenkins/jenkins/... which will definitely not exist on my MacBook. I assume jenkins is the continuous integration tool.

[Jens Alfke]

On Dec 6, 2014, at 5:49 PM, Steve <s.ande...@gmail.com> wrote:

I woke up today thinking if one app is syncing and the other app is not then there must be an issue with my app rather than Couchbase. So I removed the Couchbase Framework and other dependencies from the project and added them again. The app is now running without any syncing errors. I think were was some sort of version mismatch with the Couchbase Frame. 

Weird! Is the version you added the same as the version you removed?

However, even though the syncing is no longer failing the app is not actually pulling any the changes from the remote Couchbase Server on AWS. I need to go over my code with a fine tooth comb, I suspect the CBLReplication (pull) object is inadvertently being set to nil.

Turning on logging (the "Sync" keyword) may help; see the wiki for details.

One thing I am puzzled by, when I do a build I get 135 warnings - "unable to open object file."

This is a CBL issue … I changed the way the library is built, and it has the side effect of generating a bunch of (harmless) warnings when an app is linked.

—Jens

[Steve]

On Monday, 8 December 2014 12:45:52 UTC+13, Jens Alfke wrote:

Weird! Is the version you added the same as the version you removed?

Nope - I was using 1.0-38, but later switched to 1.0.3.1. (I moved the 1.0-38 folder too, though I would not have thought that would make a difference).

I notice from time to time Xcode complains about a path which references the 1.0-38 folder. I guess it's complaining because I moved the folder, that said I am puzzled why it is still referencing 1.0.38 and not 1.0.3.1.
  

Turning on logging (the "Sync" keyword) may help; see the wiki for details.

The wiki?

It turned out to be a bug in my code which was inadvertently setting the pull object to nil - a bit of a programming 101 mistake. To fix it I could either pass the pull replication object by reference to the class method in question or simply change the method to return the new pull replication object (I chose the latter).

So in the end it was neither an AWS issue or a Couchbase issue.

This is a CBL issue … I changed the way the library is built, and it has the side effect of generating a bunch of (harmless) warnings when an app is linked.

Cool.

[Jens Alfke]

> On Dec 7, 2014, at 6:43 PM, Steve <s.ande...@gmail.com> wrote:
>
> Nope - I was using 1.0-38, but later switched to 1.0.3.1. (I moved the 1.0-38 folder too, though I would not have thought that would make a difference).

"1.0-38"? Not sure what that is. You mean just 1.0?

> I notice from time to time Xcode complains about a path which references the 1.0-38 folder. I guess it's complaining because I moved the folder, that said I am puzzled why it is still referencing 1.0.38 and not 1.0.3.1.

Take a look at the search paths specified in the target's build settings.

> The wiki?

https://github.com/couchbase/couchbase-lite-ios/wiki/Logging

—Jens

[Steve]

On Monday, 8 December 2014 19:25:37 UTC+13, Jens Alfke wrote:

Take a look at the search paths specified in the target's build settings.

The fix was to change the search paths in the target's build settings to:

$(inherited) $(PROJECT_DIR)

Previously the search paths included the absolute path to my Couchbase Lite folder.

[Jens Alfke]

On Dec 16, 2014, at 2:46 AM, Steve <s.ande...@gmail.com> wrote:

Previously the search paths included the absolute path to my Couchbase Lite folder.

Yeah, Xcode has this annoying feature/bug that, when you drag a framework or library into the project, it adds the absolute path of its parent directory to the target library paths. It's good to keep this in mind, because there are several ways this can bite you later on. You ran into one of them, and another is that if you check your project into source control you'll find that no one else can build it because of that search path that only makes sense on your computer.)

—Jens