Invalid JSON during replication?

[dyowee]

Good day,

Could an invalid json cause this error? I am a little bit lost trying to figure out the cause of the crash.

Thread 6 name:  CouchbaseLite

Thread 6 Crashed:

0   libsystem_kernel.dylib         0x1c86dacc __pthread_kill + 8

1   libsystem_c.dylib             0x1c80295b abort + 109

2   myApp                  0x00ed159f mono_handle_native_crash (mini-exceptions.c:2548)

3   myApp                  0x00eda60f mono_sigsegv_signal_handler (mini-runtime.c:2838)

4   libsystem_platform.dylib       0x1c920259 _sigtramp + 25

5   myApp                  0x0106171b yajl_buf_append (yajl_buf.c:77)

6   myApp                  0x0106171b yajl_buf_append (yajl_buf.c:77)

7   myApp                  0x01061279 yajl_string_encode (yajl_encode.c:69)

8   myApp                  0x01062c95 yajl_gen_string (yajl_gen.c:260)

9   myApp                  0x010482d9 __53-[NSString(CBJSONEncoder) cbjson_encodeTo:canonical:]_block_invoke (CBJSONEncoder.m:167)

10  myApp                  0x0100c36f CBLWithStringBytes (CBLMisc.m:85)

11  myApp                  0x01048277 -[NSString(CBJSONEncoder) cbjson_encodeTo:canonical:] (CBJSONEncoder.m:169)

12  myApp                  0x010486c3 -[NSDictionary(CBJSONEncoder) cbjson_encodeTo:canonical:] (CBJSONEncoder.m:249)

13  myApp                  0x01047f6d -[CBJSONEncoder encode:] (CBJSONEncoder.m:102)

14  myApp                  0x01047d47 +[CBJSONEncoder canonicalEncoding:error:] (CBJSONEncoder.m:67)

15  myApp                  0x0106d7c7 -[CBL_SQLiteStorage encodeDocumentJSON:] (CBL_SQLiteStorage.m:1842)

16  myApp                  0x0106cfed __78-[CBL_SQLiteStorage forceInsert:revisionHistory:validationBlock:source:error:]_block_invoke (CBL_SQLiteStorage.m:1726)

17  myApp                  0x010678f1 -[CBL_SQLiteStorage inTransaction:] (CBL_SQLiteStorage.m:489)

18  myApp                  0x0106ca2d -[CBL_SQLiteStorage forceInsert:revisionHistory:validationBlock:source:error:] (CBL_SQLiteStorage.m:1660)

19  myApp                  0x01043da7 -[CBLDatabase(Insertion) forceInsert:revisionHistory:source:error:] (CBLDatabase+Insertion.m:264)

20  myApp                  0x01000eef __30-[CBL_Puller insertDownloads:]_block_invoke (CBL_Puller.m:677)

21  myApp                  0x010678f1 -[CBL_SQLiteStorage inTransaction:] (CBL_SQLiteStorage.m:489)

22  myApp                  0x01000a5d -[CBL_Puller insertDownloads:] (CBL_Puller.m:660)

23  myApp                  0x0100bf35 -[CBLBatcher processNow] (CBLBatcher.m:83)

24  Foundation                     0x1d9d202f __NSFireDelayedPerform + 459

25  CoreFoundation                 0x1d05c637 __CFRUNLOOP_IS_CALLING_OUT_TO_A_TIMER_CALLBACK_FUNCTION__ + 15

26  CoreFoundation                 0x1d05c339 __CFRunLoopDoTimer + 833

27  CoreFoundation                 0x1d05bd37 __CFRunLoopDoTimers + 189

28  CoreFoundation                 0x1d059dd5 __CFRunLoopRun + 781

29  CoreFoundation                 0x1cfad1af CFRunLoopRunSpecific + 471

30  CoreFoundation                 0x1cfacfd1 CFRunLoopRunInMode + 105

31  Foundation                     0x1d901ab5 -[NSRunLoop(NSRunLoop) runMode:beforeDate:] + 259

32  myApp                  0x00ff6b2d -[CBL_Server runServerThread] (CBL_Server.m:113)

33  Foundation                     0x1d9e58ab __NSThread__start__ + 1123

34  libsystem_pthread.dylib       0x1c92493b _pthread_body + 217

35  libsystem_pthread.dylib       0x1c92485d _pthread_start + 235

36  libsystem_pthread.dylib       0x1c922468 thread_start + 8

Thanks.

[dyowee]

Or this error:

Thread 6 name:  CouchbaseLite

Thread 6 Crashed:

0   libsystem_kernel.dylib         0x1c86dacc __pthread_kill + 8

1   libsystem_c.dylib             0x1c80295b abort + 109

2   myApp                  0x00e9d59f mono_handle_native_crash (mini-exceptions.c:2548)

3   myApp                  0x00ea660f mono_sigsegv_signal_handler (mini-runtime.c:2838)

4   libsystem_platform.dylib       0x1c920259 _sigtramp + 25

5   ???                           0000000000 0 + 0

6   Foundation                     0x1da6279f _writeJSONString + 93

7   Foundation                     0x1da615f3 _writeJSONValue + 93

8   Foundation                     0x1da636a9 ___writeJSONObject_block_invoke + 197

9   CoreFoundation                 0x1d0072fd __65-[__NSDictionaryI enumerateKeysAndObjectsWithOptions:usingBlock:]_block_invoke + 63

10  CoreFoundation                 0x1cff877f -[__NSDictionaryI enumerateKeysAndObjectsWithOptions:usingBlock:] + 185

11  Foundation                     0x1da62f1f _writeJSONObject + 451

12  Foundation                     0x1da6175b _writeJSONValue + 453

13  Foundation                     0x1da6155f -[_NSJSONWriter dataWithRootObject:options:error:] + 127

14  Foundation                     0x1da623db +[NSJSONSerialization dataWithJSONObject:options:error:] + 333

15  myApp                  0x01030029 +[CBLJSON dataWithJSONObject:options:error:] (CBLJSON.m:50)

16  myApp                  0x00fc017b -[CBL_Body asJSON] (CBL_Body.m:120)

17  myApp                  0x00fc04b5 -[CBL_Body compact] (CBL_Body.m:173)

18  myApp                  0x00fcc7d5 -[CBL_Puller queueDownloadedRevision:] (CBL_Puller.m:649)

19  myApp                  0x00fcb493 __33-[CBL_Puller pullRemoteRevision:]_block_invoke (CBL_Puller.m:440)

20  myApp                  0x00fd6a73 -[CBLRemoteRequest respondWithResult:error:] (CBLRemoteRequest.m:183)

21  myApp                  0x00fdac91 -[CBLMultipartDownloader connectionDidFinishLoading:] (CBLMultipartDownloader.m:107)

22  CFNetwork                     0x1d7ba9bd __65-[NSURLConnectionInternal _withConnectionAndDelegate:onlyActive:]_block_invoke + 43

23  CFNetwork                     0x1d7ba985 -[NSURLConnectionInternal _withConnectionAndDelegate:onlyActive:] + 179

24  CFNetwork                     0x1d7baa77 -[NSURLConnectionInternal _withActiveConnectionAndDelegate:] + 37

25  CFNetwork                     0x1d70510b ___ZN27URLConnectionClient_Classic26_delegate_didFinishLoadingEU13block_pointerFvvE_block_invoke + 71

26  CFNetwork                     0x1d7038e3 ___ZN27URLConnectionClient_Classic18_withDelegateAsyncEPKcU13block_pointerFvP16_CFURLConnectionPK33CFURLConnectionClientCurrent_VMaxE_block_invoke_2 + 71

27  libdispatch.dylib             0x1c76d783 _dispatch_client_callout + 23

28  libdispatch.dylib             0x1c777e21 _dispatch_block_invoke_direct + 295

29  CFNetwork                     0x1d79b9b7 RunloopBlockContext::_invoke_block(void const*, void*) + 19

30  CoreFoundation                 0x1cfacbd5 CFArrayApplyFunction + 37

31  CFNetwork                     0x1d79b889 RunloopBlockContext::perform() + 173

32  CFNetwork                     0x1d79c865 MultiplexerSource::perform() + 209

33  CFNetwork                     0x1d79c677 MultiplexerSource::_perform(void*) + 47

34  CoreFoundation                 0x1d05bfdd __CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE0_PERFORM_FUNCTION__ + 13

35  CoreFoundation                 0x1d05bb05 __CFRunLoopDoSources0 + 425

36  CoreFoundation                 0x1d059f51 __CFRunLoopRun + 1161

37  CoreFoundation                 0x1cfad1af CFRunLoopRunSpecific + 471

38  CoreFoundation                 0x1cfacfd1 CFRunLoopRunInMode + 105

39  Foundation                     0x1d901ab5 -[NSRunLoop(NSRunLoop) runMode:beforeDate:] + 259

40  myApp                  0x00fc2b2d -[CBL_Server runServerThread] (CBL_Server.m:113)

41  Foundation                     0x1d9e58ab __NSThread__start__ + 1123

42  libsystem_pthread.dylib       0x1c92493b _pthread_body + 217

43  libsystem_pthread.dylib       0x1c92485d _pthread_start + 235

44  libsystem_pthread.dylib       0x1c922468 thread_start + 8

We are on CBL 1.1.1

[dyowee]

Or is this a memory issue?

[Jens Alfke]

No, invalid JSON is handled safely and won't cause a crash.
I've never seen a crash like this before; I'm not even sure what a SIGTRAMP signal is.

—Jens

[dyowee]

Ok, just weird that the crash occurs during _writeJSONString

[dyowee]

This seems to be a memory issue in 1.1.1. I am getting a malloc: *** mach_vm_map(size=4505600) failed (error code=3)

On Monday, September 4, 2017 at 1:36:11 PM UTC-7, Jens Alfke wrote:

[Jens Alfke]

On Sep 4, 2017, at 5:41 PM, dyowee <csharpen...@gmail.com> wrote:

This seems to be a memory issue in 1.1.1. I am getting a malloc: *** mach_vm_map(size=4505600) failed (error code=3)

Sounds like you've run out of memory.

FYI, we don't support 1.1.1 anymore. (The support period for 1.2 just ended.)

—Jens

[dyowee]

We are unable to update beyond 1.1.1, due to lesser support for CouchDB, such as the filter functions.

[Jens Alfke]

On Sep 10, 2017, at 9:13 PM, dyowee <csharpen...@gmail.com> wrote:

We are unable to update beyond 1.1.1, due to lesser support for CouchDB, such as the filter functions.

You mean the inconsistency in how CouchDB accepts _changes parameters in a POST vs a GET? If you're willing to make a few changes in CBL to be compatible with whatever CouchDB is expecting, it would probably save you trouble in the long run; there have been so many fixes and improvements in CBL since 1.1.1.

When we added support for sending _changes as a POST (in CBL and Sync Gateway), we simply moved all the URL parameters to equivalent POST body parameters. But for some reason when CouchDB made the same change, they only enabled a small subset of the parameters to go in the POST; the rest still have to go in the URL. I have no idea why.

In the ChangeTracker class in CBL you can disable the use of POST if the server isn't known to be SG; that would be the easiest and safest approach. Or if you need to send POST, you can figure out which parameters CouchDB wants, and then modify the code to put only those parameters in the body.

—Jens

[dyowee]

Are you referring sir to the filter function?

[dyowee]

We really don't want to move away from Couchbase lite, we have been using it since TouchDB. 

Also, do you have ideas why NSJSONSerialization dataWithJSONObject can cause a crash in Couchbase lite during pull replication? It is really critical since it causes our app to crash during replication.

[Jens Alfke]

On Sep 10, 2017, at 10:54 PM, dyowee <csharpen...@gmail.com> wrote:

I was actually referring to the filter function sir, which I believe is not yet supported anymore since Couchbase lite 2.0

Couchbase Lite 2.0 hasn’t been released yet (and will not be compatible with CouchDB at all; it doesn’t use the same replication protocol.)

Is there a reason you can’t upgrade to 1.4?

—Jens

[Jens Alfke]

On Sep 10, 2017, at 11:04 PM, dyowee <csharpen...@gmail.com> wrote:

Also, do you have ideas why NSJSONSerialization dataWithJSONObject can cause a crash in Couchbase lite during pull replication? It is really critical since it causes our app to crash during replication.

No; have you filed a bug report with Apple? That API should not crash regardless of the input.

—Jens

[dyowee]

Sorry sir, I mean Couchbase lite 1.2. It says in the Couchbase website that:

"Since Couchbase Lite 1.2, filter functions in pull replications with non-Couchbase databases are no longer available."

[Jens Alfke]

> On Sep 10, 2017, at 11:32 PM, dyowee <csharpen...@gmail.com> wrote:
>
> Sorry sir, I mean Couchbase lite 1.2. It says in the Couchbase website that:
>
> "Since Couchbase Lite 1.2, filter functions in pull replications with non-Couchbase databases are no longer available."

That's news to me! I don't think it's true. Where does it say that?

—Jens

[Jens Alfke]

On Sep 11, 2017, at 10:06 AM, Jens Alfke <je...@couchbase.com> wrote:

That's news to me! I don't think it's true. Where does it say that?

Actually it is true. James Nocentini pointed me at the relevant Github issue, #1139. That's the one I talked about earlier, but I didn't remember it going back as far as 1.2.

It should be relatively easy to fix, by someone who is willing to do some experimenting to figure out exactly how CouchDB wants _changes request parameters sent to it. I've given up, and Couchbase doesn't have a business reason to keep supporting CouchDB, but we're happy to take a pull request that resolves the issue.

—Jens

[joey.s...@gmail.com]

Thanks sir. Will probably do that, but I need to figure out first the JSON crashing issue

Noong Martes, Setyembre 12, 2017 ng 1:25:53 AM UTC+8, si Jens Alfke ay sumulat:

[dyowee]

When does CBL check if the json is valid or not during pull replication?

[Jens Alfke]

On Sep 11, 2017, at 9:52 PM, dyowee <csharpen...@gmail.com> wrote:

When does CBL check if the json is valid or not during pull replication?

When it parses it. If the JSON is invalid, the parser will return an error/exception and we’ll handle it.

—Jens

[dyowee]

So isValidJSONObject called as indicated here?

https://developer.apple.com/documentation/foundation/nsjsonserialization/1418461-isvalidjsonobject?language=objc

[Jens Alfke]

On Sep 12, 2017, at 2:43 AM, dyowee <csharpen...@gmail.com> wrote:

So isValidJSONObject called as indicated here?

https://developer.apple.com/documentation/foundation/nsjsonserialization/1418461-isvalidjsonobject?language=objc

No, that's the other direction — it's for preflighting generating JSON from objects. Not parsing.

As I said, we don't preflight the JSON. We just call +[NSJSONSerialization JSONObjectWithData:error:] to parse it, and rely on it to return an error if the JSON is invalid.

—Jens

[dyowee]

I understand sir. What about the size/length of the json string? Would that cause an issue/crash?

[Jens Alfke]

On Sep 12, 2017, at 2:03 PM, dyowee <csharpen...@gmail.com> wrote:

I understand sir. What about the size/length of the json string? Would that cause an issue/crash?

No. The method is not supposed to crash, period. (Unless it's given a bad pointer, or the heap has been corrupted or something, of course.)

There are three possibilities:

1. CBL is passing a bad pointer or a dealloced object or something like that.
2. There's a bug in Apple's NSJSONSerialization implementation.
3. Some code in your app has corrupted the heap.

1 seems unlikely because you're the only one to have reported this bug, so apparently it doesn't happen in other people's apps. Also, a bug like this tends to crash immediately in the Obj-C runtime the first time the bad object is called.

2 also seems unlikely given how many developers Apple has, and that the NSJSONSerialization code has been around for about six years.

As for 3, have you used the Xcode Address Sanitizer and Static Analyzer to look for potential memory bugs?

—Jens

[dyowee]

I understand sir. Thank you for your input. Will try your suggestions.

[dyowee]

We are actually using Xamarin with CBL iOS, since we have started with TouchDB even before CBL.NET was around.