Stress Test with CouchDB backend result in EADDRINUSE error

[Markus Essl]

Hello!

I have my PouchDB in nodejs configured to either use LevelDB or CouchDB (depending on deployment scenarios).

When I stress test my app with LevelDB, everything works fine. If I use CouchDB as backend, I get EADDRINUSE errors.

Apparently, this is because the client TCP connection requires a unique port (epheremal) and this port does not get released until some time out (see also http://blog.gluwer.com/2014/03/story-of-eaddrinuse-and-econnreset-errors...)

As a solution, I read that I need to apply a pool and set the maxConnections for the require node module. I did this (like) this:


var db=pouchdb({name:"http://127.0.0.1:5984/db", ajax: { pool: { maxSockets: 500 } });

But this did not help. I still get errors when I put a lot of documents in it (simulating client behaviour - bulkDocs is not an option).

Could someone provide an example how this is correctly set up? I did look at the common errors, and the various API calls, but the description was always pretty generic - the ajax parameters are passed to require npm module (except cache)

Markus

[Calvin Metcalf]

can you post the stack trace (error.stack) you are getting? this might be a couchdb error or could be a node http error

--
You received this message because you are subscribed to the Google Groups "PouchDB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pouchdb+u...@googlegroups.com.
To post to this group, send email to pou...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pouchdb/1f5c2db9-11e8-4b75-b535-dc11e15f8a01%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

-Calvin W. Metcalf

[Markus Essl]

Hello,

I made a small program that simulates the behaviour. It updates 1000 documents in total. 100 documents are processed in parallel. Each document gets updated 100 times (so there are 100.000 put requests). If connection pooling is working, I would expect pouchdb to have 100 TCP/IP connections open (since the TCP/IP connections get reused).

"C:\dev\svnwork\ItemBank\experiments\stress_couchdb\stress_couchdb.zip"

BEGIN Test 0
BEGIN Test 1
BEGIN Test 2
BEGIN Test 3
BEGIN Test 4
BEGIN Test 5
BEGIN Test 6
BEGIN Test 7
BEGIN Test 8
BEGIN Test 9
BEGIN Test 10
BEGIN Test 11
BEGIN Test 12
BEGIN Test 13
BEGIN Test 14
BEGIN Test 15
BEGIN Test 16
BEGIN Test 17
BEGIN Test 18
BEGIN Test 19
BEGIN Test 20
BEGIN Test 21
BEGIN Test 22
BEGIN Test 23
BEGIN Test 24
BEGIN Test 25
BEGIN Test 26
BEGIN Test 27
BEGIN Test 28
BEGIN Test 29
BEGIN Test 30
BEGIN Test 31
BEGIN Test 32
BEGIN Test 33
BEGIN Test 34
BEGIN Test 35
BEGIN Test 36
BEGIN Test 37
BEGIN Test 38
BEGIN Test 39
BEGIN Test 40
BEGIN Test 41
BEGIN Test 42
BEGIN Test 43
BEGIN Test 44
BEGIN Test 45
BEGIN Test 46
BEGIN Test 47
BEGIN Test 48
BEGIN Test 49
BEGIN Test 50
BEGIN Test 51
BEGIN Test 52
BEGIN Test 53
BEGIN Test 54
BEGIN Test 55
BEGIN Test 56
BEGIN Test 57
BEGIN Test 58
BEGIN Test 59
BEGIN Test 60
BEGIN Test 61
BEGIN Test 62
BEGIN Test 63
BEGIN Test 64
BEGIN Test 65
BEGIN Test 66
BEGIN Test 67
BEGIN Test 68
BEGIN Test 69
BEGIN Test 70
BEGIN Test 71
BEGIN Test 72
BEGIN Test 73
BEGIN Test 74
BEGIN Test 75
BEGIN Test 76
BEGIN Test 77
BEGIN Test 78
BEGIN Test 79
BEGIN Test 80
BEGIN Test 81
BEGIN Test 82
BEGIN Test 83
BEGIN Test 84
BEGIN Test 85
BEGIN Test 86
BEGIN Test 87
BEGIN Test 88
BEGIN Test 89
BEGIN Test 90
BEGIN Test 91
BEGIN Test 92
BEGIN Test 93
BEGIN Test 94
BEGIN Test 95
BEGIN Test 96
BEGIN Test 97
BEGIN Test 98
BEGIN Test 99
DONE Test 9
BEGIN Test 999
DONE Test 92
BEGIN Test 998
DONE Test 30
BEGIN Test 997
DONE Test 52
BEGIN Test 996
DONE Test 39
BEGIN Test 995
DONE Test 0
BEGIN Test 994
DONE Test 32
BEGIN Test 993
DONE Test 56
BEGIN Test 992
DONE Test 35
BEGIN Test 991
DONE Test 16
BEGIN Test 990
DONE Test 7
BEGIN Test 989
DONE Test 65
BEGIN Test 988
DONE Test 17
BEGIN Test 987
DONE Test 89
BEGIN Test 986
DONE Test 48
BEGIN Test 985
DONE Test 64
BEGIN Test 984
DONE Test 57
BEGIN Test 983
DONE Test 87
BEGIN Test 982
DONE Test 2
BEGIN Test 981
DONE Test 14
BEGIN Test 980
DONE Test 94
BEGIN Test 979
DONE Test 99
BEGIN Test 978
DONE Test 34
BEGIN Test 977
DONE Test 10
BEGIN Test 976
DONE Test 21
BEGIN Test 975
DONE Test 19
BEGIN Test 974
DONE Test 44
BEGIN Test 973
DONE Test 77
BEGIN Test 972
DONE Test 4
BEGIN Test 971
DONE Test 12
BEGIN Test 970
DONE Test 43
BEGIN Test 969
DONE Test 58
BEGIN Test 968
DONE Test 37
BEGIN Test 967
DONE Test 33
BEGIN Test 966
DONE Test 95
BEGIN Test 965
DONE Test 69
BEGIN Test 964
DONE Test 67
BEGIN Test 963
DONE Test 13
BEGIN Test 962
DONE Test 93
BEGIN Test 961
DONE Test 36
BEGIN Test 960
DONE Test 27
BEGIN Test 959
DONE Test 55
BEGIN Test 958
DONE Test 81
BEGIN Test 957
DONE Test 29
BEGIN Test 956
DONE Test 60
BEGIN Test 955
DONE Test 82
BEGIN Test 954
DONE Test 85
BEGIN Test 953
DONE Test 23
BEGIN Test 952
DONE Test 79
BEGIN Test 951
DONE Test 40
BEGIN Test 950
DONE Test 54
BEGIN Test 949
DONE Test 5
BEGIN Test 948
DONE Test 96
BEGIN Test 947
DONE Test 28
BEGIN Test 946
DONE Test 11
BEGIN Test 945
DONE Test 6
BEGIN Test 944
DONE Test 26
BEGIN Test 943
DONE Test 31
BEGIN Test 942
DONE Test 74
BEGIN Test 941
DONE Test 41
BEGIN Test 940
DONE Test 15
BEGIN Test 939
DONE Test 18
BEGIN Test 938
DONE Test 25
BEGIN Test 937
DONE Test 84
BEGIN Test 936
DONE Test 3
BEGIN Test 935
DONE Test 53
BEGIN Test 934
DONE Test 59
BEGIN Test 933
DONE Test 46
BEGIN Test 932
DONE Test 45
BEGIN Test 931
DONE Test 49
BEGIN Test 930
DONE Test 62
BEGIN Test 929
DONE Test 24
BEGIN Test 928
DONE Test 76
BEGIN Test 927
DONE Test 42
BEGIN Test 926
DONE Test 22
BEGIN Test 925
DONE Test 51
BEGIN Test 924
DONE Test 8
BEGIN Test 923
DONE Test 47
BEGIN Test 922
DONE Test 71
BEGIN Test 921
DONE Test 88
BEGIN Test 920
DONE Test 91
BEGIN Test 919
DONE Test 97
BEGIN Test 918
DONE Test 72
BEGIN Test 917
DONE Test 90
BEGIN Test 916
DONE Test 63
BEGIN Test 915
DONE Test 1
BEGIN Test 914
DONE Test 50
BEGIN Test 913
DONE Test 66
BEGIN Test 912
DONE Test 80
BEGIN Test 911
DONE Test 73
BEGIN Test 910
DONE Test 61
BEGIN Test 909
DONE Test 70
BEGIN Test 908
DONE Test 98
BEGIN Test 907
DONE Test 38
BEGIN Test 906
DONE Test 68
BEGIN Test 905
DONE Test 86
BEGIN Test 904
DONE Test 75
BEGIN Test 903
DONE Test 83
BEGIN Test 902
DONE Test 20
BEGIN Test 901
DONE Test 78
BEGIN Test 900
node : Possibly unhandled Error: connect EADDRINUSE
In Zeile:1 Zeichen:1
+ node .\app.js 2>&1|clip
+ ~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (Possibly unhand...nect EADDRINUSE:String) [], RemoteException
    + FullyQualifiedErrorId : NativeCommandError
 
    at onError (C:\dev\svnwork\ItemBank\experiments\stress_couchdb\node_modules\pouchdb\lib\deps\ajax.js:78:18)
    at Request._callback
(C:\dev\svnwork\ItemBank\experiments\stress_couchdb\node_modules\pouchdb\lib\deps\ajax.js:144:14)
    at self.callback
(C:\dev\svnwork\ItemBank\experiments\stress_couchdb\node_modules\pouchdb\node_modules\request\request.js:123:22)
    at Request.emit (events.js:95:17)
    at ClientRequest.self.clientErrorHandler
(C:\dev\svnwork\ItemBank\experiments\stress_couchdb\node_modules\pouchdb\node_modules\request\request.js:232:10)
    at ClientRequest.emit (events.js:95:17)
    at Socket.socketErrorListener (http.js:1551:9)
    at Socket.emit (events.js:95:17)
    at net.js:440:14
    at process._tickCallback (node.js:419:13)

Thanks,
Markus

[Calvin Metcalf]

doesn't couchdb have a max listener limit or something similar? looks like you have like 800 concurrent tests going at once looks like 

To view this discussion on the web visit https://groups.google.com/d/msgid/pouchdb/6b43f5a0-8671-441c-82b3-09dbadba89a1%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

-Calvin W. Metcalf

[Markus Essl]

Hi!

Yes, exactly. That's the problem. There should be only 100 outgoing connections. But currently it goes into the thousands (until the resources are exhausted).

Using HTTP Pipelining, the TCP/IP connection should get reused - which is called pooling. When enabled - and everything works fine - only 100 connections should be there (under windows, using netstat you see that it uses all local ports until the port numbers get exhausted).

How do you configure PouchDB in a way that the http calls that the outgoing calls are pipelined? I did not see any samples of that. I see that in 'request' npm module - which is used by pouchDB, there is this "pool" config option. And the pouchDB config says, that the 'ajax' config option is passed to the request module. So I think I would need to call "var db=pouchdb({name:..,ajax:{pool:{maxConnections:300}}}) to definitly enable pooling (I would have guessed that it actually reuses TCP/IP connections on default). The request module documentation says that configuring the pool parameter must be done outside the loop ==> therefore I specified it in AJAX constructor parameter.

As of yet, I did not look into the pouchdb or request source code, I just looked at the documentation. So do you have an actual example how to configure it with couchdb in real life (in nodejs)?

I think I will try to make a test using plain request module to see if this is a pouchDB problem or request problem.

[Calvin Metcalf]

I believe pooling is done by default in node, request is just a wrapper around the http module, though looking at what you are doing, ... why are you using requrejs instead of the node module loader?

--

You received this message because you are subscribed to the Google Groups "PouchDB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pouchdb+u...@googlegroups.com.
To post to this group, send email to pou...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/pouchdb/3c581480-3a8c-4963-b763-b209120ed882%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

-Calvin W. Metcalf

[Markus Essl]

Because the code block I write run in browser as well - I don't think that this has anything to do with that, though (requireJS will use the node module loader). But I'll give it a try.

[Calvin Metcalf]

on a side note, you should look into browserify for running code that works both places, it's what we use for pouchdb 

but I believe the error has to do with pouchdb making a copy of the options object, which is usually the right thing to do except for the pool option for request that explicitly needs copies.  I believe this error has been fixed in the newer version of node (Which should be coming out really soon) but in the mean time you can try using the dev version (0.11) via something like http://npmjs.org/n and see if the problem goes away.

On Thu, Nov 13, 2014 at 10:47 AM, Markus Essl <thunder...@gmail.com> wrote:

Because the code block I write run in browser as well - I don't think that this has anything to do with that, though (requireJS will use the node module loader). But I'll give it a try.

--
You received this message because you are subscribed to the Google Groups "PouchDB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pouchdb+u...@googlegroups.com.
To post to this group, send email to pou...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/pouchdb/fd45021d-150d-4142-a29f-95457cf3a0d9%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

-Calvin W. Metcalf