Debugging binder trouble

[satur9nine]

At the company I work we have shipped several devices running various versions of Android. One device runs 4.2.2 (with kernel 3.1.10), another runs 4.4.2 (with kernel 3.4.66). We ship the devices with a number of APKs that use interprocess communication extensively: bound services, broadcast intents, etc. On the device running 4.2.2 I have seen a much higher number of TransactionTooLargeExceptions which rarely happen on the device running 4.4.2. I am wondering if there are some android or kernel patches I might need to merge in to fix the issue, maybe there is kernel configuration problem, or if it could simply be performance related. The device running 4.2.2 has a much older application processor that is not nearly as fast, as a result I expect that when things get busy multiple threads may running binder transactions simultaneously and end up hitting the overall per-process limit. Perhaps there is some kind leak going on? Maybe our device has insufficient memory or memory configuration is incorrect? Any tips on how to figure this out?

Some bad things I see in dmesg from the binder driver look like:

[15405.056815] binder: 4776: binder_alloc_buf failed to map pages in userspace, no vma

[15483.637481] binder: 454:471 transaction failed 29201, size 352-4

[15485.754927] binder: 5560:5584 refcount change on invalid ref 48

[15485.767215] binder: transaction release 363289 bad handle 48

Some bad things I see in kernel debug logs look like:

adb shell cat /sys/kernel/debug/binder/failed_transaction_log
0: call  from 721:777 to 0:0 node 2253 handle 13 size 76:0

0: call  from 721:776 to 0:0 node 2219 handle 15 size 76:0

0: call  from 721:782 to 0:0 node 2253 handle 13 size 76:0

0: call  from 721:721 to 0:0 node 39622 handle 57 size 64:0

0: call  from 721:721 to 0:0 node 39730 handle 60 size 64:0

0: call  from 721:721 to 0:0 node 39327 handle 44 size 64:0

0: call  from 721:721 to 0:0 node 39391 handle 48 size 64:0

0: call  from 721:721 to 0:0 node 39627 handle 58 size 64:0

0: call  from 721:721 to 0:0 node 39525 handle 54 size 64:0

See more complete dumps at http://pastebin.com/aGCi3b9r