Replication timeout and internal server errors

[wbi...@attic.nl]

We have a setup where 3-6 iPads in a LAN need to be aware of changes in a set of documents. When our app is started, the following steps are performed: 

• start a CBLListener 
  
• start a NSNetServiceBrowser
  
• on receipt of a netServiceBrowser:didFindService a 'peer' object is created and resolved with resolveWithTimeout
  
• on netServiceDidResolveAddress a continuous push replication is started with the newly found peer.
  
• on receipt of netServiceBrowser:didRemoveService: replication is stopped with corresponding peer
  
• observe kCBLReplicationChangeNotification
  
• start a continuous push/pull replication with a remote CouchDB server (at Cloudant) (allows us to see what's happening. We do not want to depend on working internet so this replication is redundant)

This works fine but every few days the replication stops and we see replication errors "Request time out" and "500 Internal server error" in our logs. There is no relation with the workload or the number of ipads. 

Is something wrong with the procedure described above ?

Are we maybe overloading the internal webserver and/or the replication task with up to 7 replications ?

How can we investigate this issue ?

[Jens Alfke]

On Jul 22, 2014, at 12:09 PM, wbi...@attic.nl wrote:

This works fine but every few days the replication stops and we see replication errors "Request time out" and "500 Internal server error" in our logs. There is no relation with the workload or the number of ipads. 

Which replication stops? A peer one, or the one with Cloudant?

If it’s a continuous replication, it shouldn’t stop even if there are intermittent errors like that; it should back off and retry. You should file an issue on Github, with a link to logs (probably separately uploaded to Gist or something, because they’re long.) Turn on ‘Sync’ and ‘RemoteRequest’ logging. Thanks!

—Jens

[wbi...@attic.nl]

Which replication stops? A peer one, or the one with Cloudant? 

If it’s a continuous replication, it shouldn’t stop even if there are intermittent errors like that; it should back off and retry.

The replication does continue but fails with the errors mentioned.

 

You should file an issue on Github, with a link to logs (probably separately uploaded to Gist or something, because they’re long.) Turn on ‘Sync’ and ‘RemoteRequest’ logging.

Is there a better way to get the console log other than hooking a Mac to an iPad ? Ideally, I would like to be able to add a listener to the couch logging system.

Thanks !

[Jens Alfke]

On Jul 23, 2014, at 3:22 AM, wbi...@attic.nl wrote:

The replication does continue but fails with the errors mentioned.

Intermittent failures are a fact of life with networking. As long as the replication continues, and is able to pick up where it left off the next time it’s able to connect, it’s working correctly. 

Is there a better way to get the console log other than hooking a Mac to an iPad ? Ideally, I would like to be able to add a listener to the couch logging system.

I believe you can look in Xcode’s Devices window while the devices is tethered and browse recent logs.

There isn’t a facility to redirect log messages to your own handlers, although that would be a good feature to add (you’re welcome to file an issue to request it.)

—Jens

[wbi...@attic.nl]

I believe you can look in Xcode’s Devices window while the devices is tethered and browse recent logs.

There isn’t a facility to redirect log messages to your own handlers, although that would be a good feature to add (you’re welcome to file an issue to request it.)

The Couch logging is only shown in the XCode IDE but not in XCode Organizer or the iPhone Config Utility. When I try to capture the log by redirecting stderr to file (so I can later mail the log) the resulting log does not contain the Couch logging either. Looking at the MYUtil source it looks like the msg is sent to stderr (sLoggingTo = 2). 

The only way to get the log seems to run the app in XCode ?

Three logdumps of the same session; only the first shows the Couch logging (running app under debugger on device with usb cable):

XCode IDE console

2014-07-24 16:45:54.015 Nappkin[460:60b] XXX v2.1 started with server http://devcellarapp.apphb.com 

2014-07-24 16:45:54.292 Nappkin[460:60b] GET: /api/v1/menucard/-1   

16:45:54.316| Logging mode 2 enabled in domains: {Sync, SyncVerbose, View} 

2014-07-24 16:45:54.348 Nappkin[460:60b] Database 'nappkin_193' initialized with 9 documents 

XCode Organizer - Devices

Jul 24 16:45:54 MiniPad Nappkin[460] <Warning>: XXX v2.1 started with server http://devcellarapp.apphb.com

Jul 24 16:45:54 MiniPad Nappkin[460] <Warning>: GET: /api/v1/menucard/-1  

Jul 24 16:45:54 MiniPad Nappkin[460] <Warning>: Database 'nappkin_193' initialized with 9 documents

iPhone Configuration Utility

Jul 24 16:45:54 MiniPad Nappkin[460] <Warning>: XXX v2.1 started with server http://devcellarapp.apphb.com

Jul 24 16:45:54 MiniPad Nappkin[460] <Warning>: GET: /api/v1/menucard/-1  

Jul 24 16:45:54 MiniPad Nappkin[460] <Warning>: Database 'nappkin_193' initialized with 9 documents

[Len Kawell]

You're right that Couchbase Lite logging is currently only going to the Xcode debugger and not the Xcode Organizer or iPhone Config Utility. That's because the logging is being output to sderr instead of the Apple System Logging (ASL). I recently filed an issue (#395) on that and Jens said he'd look into changing it. In the mean time, if you do your own build of CBL, you can change line #186 in Logging.m from fputs([finalMsg UTF8String], stderr); to NSLog(@"%@", finalMsg);. 

If you'd like to include the console log with user feedback, you can use something like we use in our app: https://gist.github.com/lenkawell/f930a62ea0b21368d163

[Jens Alfke]

On Jul 24, 2014, at 8:30 AM, wbi...@attic.nl wrote:

Looking at the MYUtil source it looks like the msg is sent to stderr (sLoggingTo = 2). 

The only way to get the log seems to run the app in XCode ?

Yes, that was filed a few days ago as an issue (by you?) I’ll need to switch to calling ASL or NSLog, I think.

There’s also been a request to allow redirecting log messages to an app callback, which is also a good idea.

—Jens

[wbi...@attic.nl]

Op woensdag 23 juli 2014 17:34:49 UTC+2 schreef Jens Alfke:

On Jul 23, 2014, at 3:22 AM, wbi...@attic.nl wrote:

The replication does continue but fails with the errors mentioned.

Intermittent failures are a fact of life with networking. As long as the replication continues, and is able to pick up where it left off the next time it’s able to connect, it’s working correctly. 

I incorrectly interpreted (and logged) each notification with a non-nil .lastError as a new error.

Sorry for the confusion !