views failing due to fabric_worker_timeout and OS process timed out

[Gustavo Delfino]

Hi, I am evaluating using CouchDB and all worked well with a small test database. Now I am trying to use it with a much larger database and I am having an issue creating views. My view map function is very simple:

function (doc) {
var trw_id;
if(doc.customer_id){
emit(doc.customer_id, doc._id);
}
}

With a few hundred documents it works well but not as the size of the db grows (or maybe I have an issue with the function above).

I can see in the log how the shards start working:

[info] 2017-02-21T22:38:58.786000Z couchdb@localhost <0.29209.6> -------- Starting index update for db: shards/20000000-3fffffff/vw.1487715840 idx: _design/appname
[info] 2017-02-21T22:38:58.786000Z couchdb@localhost <0.29194.6> -------- Starting index update for db: shards/00000000-1fffffff/vw.1487715840 idx: _design/appname
[info] 2017-02-21T22:38:58.786000Z couchdb@localhost <0.29191.6> -------- Starting index update for db: shards/60000000-7fffffff/vw.1487715840 idx: _design/appname
[info] 2017-02-21T22:38:58.786000Z couchdb@localhost <0.29205.6> -------- Starting index update for db: shards/80000000-9fffffff/vw.1487715840 idx: _design/appname
[info] 2017-02-21T22:38:58.786000Z couchdb@localhost <0.29218.6> -------- Starting index update for db: shards/40000000-5fffffff/vw.1487715840 idx: _design/appname
[info] 2017-02-21T22:38:58.786000Z couchdb@localhost <0.29228.6> -------- Starting index update for db: shards/a0000000-bfffffff/vw.1487715840 idx: _design/appname
[info] 2017-02-21T22:38:58.786000Z couchdb@localhost <0.29225.6> -------- Starting index update for db: shards/c0000000-dfffffff/vw.1487715840 idx: _design/appname
[info] 2017-02-21T22:38:58.788000Z couchdb@localhost <0.29208.6> -------- Starting index update for db: shards/e0000000-ffffffff/vw.1487715840 idx: _design/appname

I see high CPU activity signaling that the index is being created and suddenly it stops:

[error] 2017-02-21T22:39:58.931000Z couchdb@localhost <0.19734.6> d4985a33d1 fabric_worker_timeout map_view,couchdb@localhost,<<"shards/00000000-1fffffff/vw.1487715840">>
[error] 2017-02-21T22:39:58.931000Z couchdb@localhost <0.19734.6> d4985a33d1 fabric_worker_timeout map_view,couchdb@localhost,<<"shards/20000000-3fffffff/vw.1487715840">>
[error] 2017-02-21T22:39:58.931000Z couchdb@localhost <0.19734.6> d4985a33d1 fabric_worker_timeout map_view,couchdb@localhost,<<"shards/40000000-5fffffff/vw.1487715840">>
[error] 2017-02-21T22:39:58.931000Z couchdb@localhost <0.19734.6> d4985a33d1 fabric_worker_timeout map_view,couchdb@localhost,<<"shards/60000000-7fffffff/vw.1487715840">>
[error] 2017-02-21T22:39:58.931000Z couchdb@localhost <0.19734.6> d4985a33d1 fabric_worker_timeout map_view,couchdb@localhost,<<"shards/80000000-9fffffff/vw.1487715840">>
[error] 2017-02-21T22:39:58.931000Z couchdb@localhost <0.19734.6> d4985a33d1 fabric_worker_timeout map_view,couchdb@localhost,<<"shards/a0000000-bfffffff/vw.1487715840">>
[error] 2017-02-21T22:39:58.931000Z couchdb@localhost <0.19734.6> d4985a33d1 fabric_worker_timeout map_view,couchdb@localhost,<<"shards/c0000000-dfffffff/vw.1487715840">>
[error] 2017-02-21T22:39:58.931000Z couchdb@localhost <0.19734.6> d4985a33d1 fabric_worker_timeout map_view,couchdb@localhost,<<"shards/e0000000-ffffffff/vw.1487715840">>
[error] 2017-02-21T22:39:59.000000Z couchdb@localhost <0.19734.6> d4985a33d1 req_err(1329706011) unknown_error : function_clause
[<<"couch_mrview_show:list_cb/2 L212">>,<<"fabric_view_map:go/7 L52">>,<<"couch_query_servers:with_ddoc_proc/2 L421">>,<<"chttpd:process_request/1 L293">>,<<"chttpd:handle_request_int/1 L229">>,<<"mochiweb_http:headers/6 L122">>,<<"proc_lib:init_p_do_apply/3 L237">>]
[notice] 2017-02-21T22:39:59.002000Z couchdb@localhost <0.19734.6> d4985a33d1 127.0.0.1:5984 127.0.0.1 undefined GET /dbname/_design/appname/_list/data/customer_id?key=%22PRIV-SE270_FC_AZT10L16_016%22 500 ok 60218

In the web browser, I get this:

{"error":"unknown_error","reason":"function_clause","ref":1329706011}

When the error happened, also I was replicating from another CouchDB server that has a large number of documents. I was running a test requesting the views as the db was getting filled in to see at what point I started to get the issue. So I started seeing the issue with about 17k documents (0.7GB).

I have just reloaded the page and it now works, but I have not been able to make the view work on another machine with my complete DB which is much bigger (1/2 million docs, 22GB)

This is what I see in the log in the machine with the large database:

[info] 2017-02-21T23:08:26.127000Z couchdb@localhost <0.4854.995> -------- Starting index update for db: shards/00000000-1fffffff/vw.1481754819 idx: _design/adag
[info] 2017-02-21T23:08:26.127000Z couchdb@localhost <0.4141.984> -------- Starting index update for db: shards/20000000-3fffffff/vw.1481754819 idx: _design/adag
[info] 2017-02-21T23:08:26.127000Z couchdb@localhost <0.5037.995> -------- Starting index update for db: shards/a0000000-bfffffff/vw.1481754819 idx: _design/adag
[info] 2017-02-21T23:08:26.127000Z couchdb@localhost <0.3463.995> -------- Starting index update for db: shards/80000000-9fffffff/vw.1481754819 idx: _design/adag
[info] 2017-02-21T23:08:26.127000Z couchdb@localhost <0.4588.995> -------- Starting index update for db: shards/60000000-7fffffff/vw.1481754819 idx: _design/adag
[info] 2017-02-21T23:08:26.127000Z couchdb@localhost <0.1391.995> -------- Starting index update for db: shards/40000000-5fffffff/vw.1481754819 idx: _design/adag
[info] 2017-02-21T23:08:26.127000Z couchdb@localhost <0.5295.995> -------- Starting index update for db: shards/c0000000-dfffffff/vw.1481754819 idx: _design/adag
[info] 2017-02-21T23:08:26.127000Z couchdb@localhost <0.31235.994> -------- Starting index update for db: shards/e0000000-ffffffff/vw.1481754819 idx: _design/adag
[notice] 2017-02-21T23:08:29.332000Z couchdb@localhost <0.18522.1766> 24e91bc164 servername:5984 149.223.224.36 undefined GET /dbname/_changes?feed=continuous&style=all_docs&since=%22482896-g1AAAAJ7eJzLYWBg4MhgTmEQTM4vTc5ISXLIyU9OzMnILy7JAUoxJTIkyf___z8rgzmJgeF1by5QjD3NwDjFPNkEmx48JiUpAMkke4Rha8CGJVmkmSWam5NqmAPIsHiEYS0QlxlbpiUakmxYAsiweoRh-8CGWSSaWliaGpBoWB4LkGRoAFJA8-ZDDTwHNtDYzCI1McWCLAMXQAzcDzHwTRAk7IzNU5LSUsky8ADEwPtQF94FG5honmRsaUielx9ADISF4Q2IgYmpaWaWadi0ZgEA3hnNtQ%22&timeout=10000 200 ok 10042
[notice] 2017-02-21T23:08:39.773000Z couchdb@localhost <0.18522.1766> b10f69b2d2 servername:5984 149.223.224.36 undefined GET /dbname/_changes?feed=continuous&style=all_docs&since=%22482896-g1AAAAJ7eJzLYWBg4MhgTmEQTM4vTc5ISXLIyU9OzMnILy7JAUoxJTIkyf___z8rgzmJgeF1by5QjD3NwDjFPNkEmx48JiUpAMkke4Rha8CGJVmkmSWam5NqmAPIsHiEYS0QlxlbpiUakmxYAsiweoRh-8CGWSSaWliaGpBoWB4LkGRoAFJA8-ZDDTwHNtDYzCI1McWCLAMXQAzcDzHwTRAk7IzNU5LSUsky8ADEwPtQF94FG5honmRsaUielx9ADISF4Q2IgYmpaWaWadi0ZgEA3hnNtQ%22&timeout=10000 200 ok 10424
[notice] 2017-02-21T23:08:50.255000Z couchdb@localhost <0.9952.4075> bdb771bc58 servername:5984 149.223.224.36 undefined GET /dbname/_changes?since=%22482896-g1AAAAJ7eJzLYWBg4MhgTmEQTM4vTc5ISXLIyU9OzMnILy7JAUoxJTIkyf___z8rgzmJgeF1by5QjD3NwDjFPNkEmx48JiUpAMkke4Rha8CGJVmkmSWam5NqmAPIsHiEYS0QlxlbpiUakmxYAsiweoRh-8CGWSSaWliaGpBoWB4LkGRoAFJA8-ZDDTwHNtDYzCI1McWCLAMXQAzcDzHwTRAk7IzNU5LSUsky8ADEwPtQF94FG5honmRsaUielx9ADISF4Q2IgYmpaWaWadi0ZgEA3hnNtQ%22&limit=0 200 ok 2757
[notice] 2017-02-21T23:08:51.292000Z couchdb@localhost <0.18522.1766> 18e9dee4d3 servername:5984 149.223.224.36 undefined GET /dbname/_changes?feed=continuous&style=all_docs&since=%22482896-g1AAAAJ7eJzLYWBg4MhgTmEQTM4vTc5ISXLIyU9OzMnILy7JAUoxJTIkyf___z8rgzmJgeF1by5QjD3NwDjFPNkEmx48JiUpAMkke4Rha8CGJVmkmSWam5NqmAPIsHiEYS0QlxlbpiUakmxYAsiweoRh-8CGWSSaWliaGpBoWB4LkGRoAFJA8-ZDDTwHNtDYzCI1McWCLAMXQAzcDzHwTRAk7IzNU5LSUsky8ADEwPtQF94FG5honmRsaUielx9ADISF4Q2IgYmpaWaWadi0ZgEA3hnNtQ%22&timeout=10000 200 ok 11504
[notice] 2017-02-21T23:09:02.404000Z couchdb@localhost <0.18522.1766> 2ae1bf4f82 servername:5984 149.223.224.36 undefined GET /dbname/_changes?feed=continuous&style=all_docs&since=%22482896-g1AAAAJ7eJzLYWBg4MhgTmEQTM4vTc5ISXLIyU9OzMnILy7JAUoxJTIkyf___z8rgzmJgeF1by5QjD3NwDjFPNkEmx48JiUpAMkke4Rha8CGJVmkmSWam5NqmAPIsHiEYS0QlxlbpiUakmxYAsiweoRh-8CGWSSaWliaGpBoWB4LkGRoAFJA8-ZDDTwHNtDYzCI1McWCLAMXQAzcDzHwTRAk7IzNU5LSUsky8ADEwPtQF94FG5honmRsaUielx9ADISF4Q2IgYmpaWaWadi0ZgEA3hnNtQ%22&timeout=10000 200 ok 11109
[info] 2017-02-21T23:09:14.252000Z couchdb@localhost <0.212.0> -------- couch_proc_manager <0.9345.4064> died normal
[error] 2017-02-21T23:09:14.253000Z couchdb@localhost <0.11021.4075> -------- OS Process Error <0.9345.4064> :: {os_process_error,"OS process timed out."}
[error] 2017-02-21T23:09:14.376000Z couchdb@localhost emulator -------- Error in process <0.11021.4075> on node 'couchdb@localhost' with exit value: {{nocatch,{os_process_error,"OS process timed out."}},[{couch_os_process,prompt,2,[{file,"src/couch_os_process.erl"},{line,59}]},{couch_query_servers,map_doc_raw,2,[{file,"src/couch_query_servers.erl"},{line,67}]},{couch_mrview_updater...
[notice] 2017-02-21T23:09:14.705000Z couchdb@localhost <0.18522.1766> fe12433fb6 servername:5984 149.223.224.36 undefined GET /dbname/_changes?feed=continuous&style=all_docs&since=%22482896-g1AAAAJ7eJzLYWBg4MhgTmEQTM4vTc5ISXLIyU9OzMnILy7JAUoxJTIkyf___z8rgzmJgeF1by5QjD3NwDjFPNkEmx48JiUpAMkke4Rha8CGJVmkmSWam5NqmAPIsHiEYS0QlxlbpiUakmxYAsiweoRh-8CGWSSaWliaGpBoWB4LkGRoAFJA8-ZDDTwHNtDYzCI1McWCLAMXQAzcDzHwTRAk7IzNU5LSUsky8ADEwPtQF94FG5honmRsaUielx9ADISF4Q2IgYmpaWaWadi0ZgEA3hnNtQ%22&timeout=10000 200 ok 12295
[error] 2017-02-21T23:09:15.374000Z couchdb@localhost <0.31759.4074> 3f0fdef0dc rexi_server throw:{os_process_error,"OS process timed out."} [{couch_mrview_util,get_view,4,[{file,"src/couch_mrview_util.erl"},{line,56}]},{couch_mrview,query_view,6,[{file,"src/couch_mrview.erl"},{line,244}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,139}]}]
[error] 2017-02-21T23:09:15.753000Z couchdb@localhost <0.4603.4075> 3f0fdef0dc req_err(3183461804) unknown_error : function_clause
[<<"couch_mrview_show:list_cb/2 L212">>,<<"fabric_view_map:go/7 L54">>,<<"couch_query_servers:with_ddoc_proc/2 L421">>,<<"chttpd:process_request/1 L293">>,<<"chttpd:handle_request_int/1 L229">>,<<"mochiweb_http:headers/6 L122">>,<<"proc_lib:init_p_do_apply/3 L237">>]
[notice] 2017-02-21T23:09:15.753000Z couchdb@localhost <0.4603.4075> 3f0fdef0dc farmi-9w4vns1:5984 10.217.46.47 undefined GET /dbname/_design/adag/_list/signals/trw_id?start_key=%22VDP480001%22;end_key=%22VDP480001\u9999%22; 500 ok 49629
[notice] 2017-02-21T23:09:26.761000Z couchdb@localhost <0.18522.1766> a69a211a53 servername:5984 149.223.224.36 undefined GET /dbname/_changes?feed=continuous&style=all_docs&since=%22482896-g1AAAAJ7eJzLYWBg4MhgTmEQTM4vTc5ISXLIyU9OzMnILy7JAUoxJTIkyf___z8rgzmJgeF1by5QjD3NwDjFPNkEmx48JiUpAMkke4Rha8CGJVmkmSWam5NqmAPIsHiEYS0QlxlbpiUakmxYAsiweoRh-8CGWSSaWliaGpBoWB4LkGRoAFJA8-ZDDTwHNtDYzCI1McWCLAMXQAzcDzHwTRAk7IzNU5LSUsky8ADEwPtQF94FG5honmRsaUielx9ADISF4Q2IgYmpaWaWadi0ZgEA3hnNtQ%22&timeout=10000 200 ok 11137
[notice] 2017-02-21T23:09:27.476000Z couchdb@localhost <0.10851.4075> 7b10680917 servername:5984 149.223.224.36 undefined GET /dbname/_changes?since=%22482896-g1AAAAJ7eJzLYWBg4MhgTmEQTM4vTc5ISXLIyU9OzMnILy7JAUoxJTIkyf___z8rgzmJgeF1by5QjD3NwDjFPNkEmx48JiUpAMkke4Rha8CGJVmkmSWam5NqmAPIsHiEYS0QlxlbpiUakmxYAsiweoRh-8CGWSSaWliaGpBoWB4LkGRoAFJA8-ZDDTwHNtDYzCI1McWCLAMXQAzcDzHwTRAk7IzNU5LSUsky8ADEwPtQF94FG5honmRsaUielx9ADISF4Q2IgYmpaWaWadi0ZgEA3hnNtQ%22&limit=0 200 ok 4564
[error] 2017-02-21T23:09:34.145000Z couchdb@localhost <0.9743.4075> -------- OS Process Error <0.10358.4064> :: {os_process_error,"OS process timed out."}
[info] 2017-02-21T23:09:34.336000Z couchdb@localhost <0.212.0> -------- couch_proc_manager <0.10358.4064> died normal
[error] 2017-02-21T23:09:34.336000Z couchdb@localhost emulator -------- Error in process <0.9743.4075> on node 'couchdb@localhost' with exit value: {{nocatch,{os_process_error,"OS process timed out."}},[{couch_os_process,prompt,2,[{file,"src/couch_os_process.erl"},{line,59}]},{couch_query_servers,map_doc_raw,2,[{file,"src/couch_query_servers.erl"},{line,67}]},{couch_mrview_updater...
[error] 2017-02-21T23:09:34.468000Z couchdb@localhost <0.32198.4074> 3f0fdef0dc rexi_server throw:{os_process_error,"OS process timed out."} [{couch_mrview_util,get_view,4,[{file,"src/couch_mrview_util.erl"},{line,56}]},{couch_mrview,query_view,6,[{file,"src/couch_mrview.erl"},{line,244}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,139}]}]
[notice] 2017-02-21T23:09:38.300000Z couchdb@localhost <0.18522.1766> ae291841ad servername:5984 149.223.224.36 undefined GET /dbname/_changes?feed=continuous&style=all_docs&since=%22482896-g1AAAAJ7eJzLYWBg4MhgTmEQTM4vTc5ISXLIyU9OzMnILy7JAUoxJTIkyf___z8rgzmJgeF1by5QjD3NwDjFPNkEmx48JiUpAMkke4Rha8CGJVmkmSWam5NqmAPIsHiEYS0QlxlbpiUakmxYAsiweoRh-8CGWSSaWliaGpBoWB4LkGRoAFJA8-ZDDTwHNtDYzCI1McWCLAMXQAzcDzHwTRAk7IzNU5LSUsky8ADEwPtQF94FG5honmRsaUielx9ADISF4Q2IgYmpaWaWadi0ZgEA3hnNtQ%22&timeout=10000 200 ok 10824
[info] 2017-02-21T23:09:50.162000Z couchdb@localhost <0.212.0> -------- couch_proc_manager <0.9644.4075> died normal
[notice] 2017-02-21T23:09:50.229000Z couchdb@localhost <0.18522.1766> 8033bbafb0 servername:5984 149.223.224.36 undefined GET /dbname/_changes?feed=continuous&style=all_docs&since=%22482896-g1AAAAJ7eJzLYWBg4MhgTmEQTM4vTc5ISXLIyU9OzMnILy7JAUoxJTIkyf___z8rgzmJgeF1by5QjD3NwDjFPNkEmx48JiUpAMkke4Rha8CGJVmkmSWam5NqmAPIsHiEYS0QlxlbpiUakmxYAsiweoRh-8CGWSSaWliaGpBoWB4LkGRoAFJA8-ZDDTwHNtDYzCI1McWCLAMXQAzcDzHwTRAk7IzNU5LSUsky8ADEwPtQF94FG5honmRsaUielx9ADISF4Q2IgYmpaWaWadi0ZgEA3hnNtQ%22&timeout=10000 200 ok 11907
[error] 2017-02-21T23:09:50.644000Z couchdb@localhost <0.1803.4075> -------- OS Process Error <0.9644.4075> :: {os_process_error,"OS process timed out."}
[error] 2017-02-21T23:09:50.718000Z couchdb@localhost emulator -------- Error in process <0.1803.4075> on node 'couchdb@localhost' with exit value: {{nocatch,{os_process_error,"OS process timed out."}},[{couch_os_process,prompt,2,[{file,"src/couch_os_process.erl"},{line,59}]},{couch_query_servers,map_doc_raw,2,[{file,"src/couch_query_servers.erl"},{line,67}]},{couch_mrview_updater...
[error] 2017-02-21T23:09:50.889000Z couchdb@localhost <0.6778.4075> 3f0fdef0dc rexi_server throw:{os_process_error,"OS process timed out."} [{couch_mrview_util,get_view,4,[{file,"src/couch_mrview_util.erl"},{line,56}]},{couch_mrview,query_view,6,[{file,"src/couch_mrview.erl"},{line,244}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,139}]}]
[info] 2017-02-21T23:09:53.518000Z couchdb@localhost <0.212.0> -------- couch_proc_manager <0.9477.4075> died normal
[error] 2017-02-21T23:09:53.970000Z couchdb@localhost <0.4641.4075> -------- OS Process Error <0.9477.4075> :: {os_process_error,"OS process timed out."}
[error] 2017-02-21T23:09:54.029000Z couchdb@localhost emulator -------- Error in process <0.4641.4075> on node 'couchdb@localhost' with exit value: {{nocatch,{os_process_error,"OS process timed out."}},[{couch_os_process,prompt,2,[{file,"src/couch_os_process.erl"},{line,59}]},{couch_query_servers,map_doc_raw,2,[{file,"src/couch_query_servers.erl"},{line,67}]},{couch_mrview_updater...
[error] 2017-02-21T23:09:54.143000Z couchdb@localhost <0.5079.4075> 3f0fdef0dc rexi_server throw:{os_process_error,"OS process timed out."} [{couch_mrview_util,get_view,4,[{file,"src/couch_mrview_util.erl"},{line,56}]},{couch_mrview,query_view,6,[{file,"src/couch_mrview.erl"},{line,244}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,139}]}]
[error] 2017-02-21T23:09:55.888000Z couchdb@localhost <0.7766.4075> -------- OS Process Error <0.9307.4075> :: {os_process_error,"OS process timed out."}
[info] 2017-02-21T23:09:55.888000Z couchdb@localhost <0.212.0> -------- couch_proc_manager <0.9307.4075> died normal
[error] 2017-02-21T23:09:55.888000Z couchdb@localhost emulator -------- Error in process <0.7766.4075> on node 'couchdb@localhost' with exit value: {{nocatch,{os_process_error,"OS process timed out."}},[{couch_os_process,prompt,2,[{file,"src/couch_os_process.erl"},{line,59}]},{couch_query_servers,map_doc_raw,2,[{file,"src/couch_query_servers.erl"},{line,67}]},{couch_mrview_updater...
[error] 2017-02-21T23:09:55.992000Z couchdb@localhost <0.5686.4075> 3f0fdef0dc rexi_server throw:{os_process_error,"OS process timed out."} [{couch_mrview_util,get_view,4,[{file,"src/couch_mrview_util.erl"},{line,56}]},{couch_mrview,query_view,6,[{file,"src/couch_mrview.erl"},{line,244}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,139}]}]
[notice] 2017-02-21T23:09:59.153000Z couchdb@localhost <0.7323.4075> 0afe317a73 servername:5984 149.223.224.36 undefined GET /dbname/_changes?since=%22482896-g1AAAAJ7eJzLYWBg4MhgTmEQTM4vTc5ISXLIyU9OzMnILy7JAUoxJTIkyf___z8rgzmJgeF1by5QjD3NwDjFPNkEmx48JiUpAMkke4Rha8CGJVmkmSWam5NqmAPIsHiEYS0QlxlbpiUakmxYAsiweoRh-8CGWSSaWliaGpBoWB4LkGRoAFJA8-ZDDTwHNtDYzCI1McWCLAMXQAzcDzHwTRAk7IzNU5LSUsky8ADEwPtQF94FG5honmRsaUielx9ADISF4Q2IgYmpaWaWadi0ZgEA3hnNtQ%22&limit=0 200 ok 756
[notice] 2017-02-21T23:10:01.630000Z couchdb@localhost <0.18522.1766> 4bf75dae18 servername:5984 149.223.224.36 undefined GET /dbname/_changes?feed=continuous&style=all_docs&since=%22482896-g1AAAAJ7eJzLYWBg4MhgTmEQTM4vTc5ISXLIyU9OzMnILy7JAUoxJTIkyf___z8rgzmJgeF1by5QjD3NwDjFPNkEmx48JiUpAMkke4Rha8CGJVmkmSWam5NqmAPIsHiEYS0QlxlbpiUakmxYAsiweoRh-8CGWSSaWliaGpBoWB4LkGRoAFJA8-ZDDTwHNtDYzCI1McWCLAMXQAzcDzHwTRAk7IzNU5LSUsky8ADEwPtQF94FG5honmRsaUielx9ADISF4Q2IgYmpaWaWadi0ZgEA3hnNtQ%22&timeout=10000 200 ok 10912
[info] 2017-02-21T23:10:05.671000Z couchdb@localhost <0.212.0> -------- couch_proc_manager <0.29924.3320> died normal
[error] 2017-02-21T23:10:05.671000Z couchdb@localhost <0.9423.4075> -------- OS Process Error <0.29924.3320> :: {os_process_error,"OS process timed out."}
[error] 2017-02-21T23:10:05.672000Z couchdb@localhost emulator -------- Error in process <0.9423.4075> on node 'couchdb@localhost' with exit value: {{nocatch,{os_process_error,"OS process timed out."}},[{couch_os_process,prompt,2,[{file,"src/couch_os_process.erl"},{line,59}]},{couch_query_servers,map_doc_raw,2,[{file,"src/couch_query_servers.erl"},{line,67}]},{couch_mrview_updater...
[error] 2017-02-21T23:10:05.672000Z couchdb@localhost <0.3293.4075> 3f0fdef0dc rexi_server throw:{os_process_error,"OS process timed out."} [{couch_mrview_util,get_view,4,[{file,"src/couch_mrview_util.erl"},{line,56}]},{couch_mrview,query_view,6,[{file,"src/couch_mrview.erl"},{line,244}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,139}]}]

Any idea of what could be going on? I am running CouchDB 2.0.0.1 under Windows 7 with a single node. I have not modified most of the default CouchDB settings.

Regards,

Gustavo Delfino

[Adam Kocoloski]

Hi Gustavo, there are a couple of things going on here. Let’s address them individually:

> On Feb 21, 2017, at 6:17 PM, Gustavo Delfino <Gustavo...@zf.com> wrote:
>
> Hi, I am evaluating using CouchDB and all worked well with a small test database. Now I am trying to use it with a much larger database and I am having an issue creating views. My view map function is very simple:
>
> function (doc) {
> var trw_id;
> if(doc.customer_id){
> emit(doc.customer_id, doc._id);
> }
> }
>
> With a few hundred documents it works well but not as the size of the db grows (or maybe I have an issue with the function above).
>
> I can see in the log how the shards start working:
>
> [info] 2017-02-21T22:38:58.786000Z couchdb@localhost <0.29209.6> -------- Starting index update for db: shards/20000000-3fffffff/vw.1487715840 idx: _design/appname
> [info] 2017-02-21T22:38:58.786000Z couchdb@localhost <0.29194.6> -------- Starting index update for db: shards/00000000-1fffffff/vw.1487715840 idx: _design/appname
> [info] 2017-02-21T22:38:58.786000Z couchdb@localhost <0.29191.6> -------- Starting index update for db: shards/60000000-7fffffff/vw.1487715840 idx: _design/appname
> [info] 2017-02-21T22:38:58.786000Z couchdb@localhost <0.29205.6> -------- Starting index update for db: shards/80000000-9fffffff/vw.1487715840 idx: _design/appname
> [info] 2017-02-21T22:38:58.786000Z couchdb@localhost <0.29218.6> -------- Starting index update for db: shards/40000000-5fffffff/vw.1487715840 idx: _design/appname
> [info] 2017-02-21T22:38:58.786000Z couchdb@localhost <0.29228.6> -------- Starting index update for db: shards/a0000000-bfffffff/vw.1487715840 idx: _design/appname
> [info] 2017-02-21T22:38:58.786000Z couchdb@localhost <0.29225.6> -------- Starting index update for db: shards/c0000000-dfffffff/vw.1487715840 idx: _design/appname
> [info] 2017-02-21T22:38:58.788000Z couchdb@localhost <0.29208.6> -------- Starting index update for db: shards/e0000000-ffffffff/vw.1487715840 idx: _design/appname
>
> I see high CPU activity signaling that the index is being created and suddenly it stops:
>
> [error] 2017-02-21T22:39:58.931000Z couchdb@localhost <0.19734.6> d4985a33d1 fabric_worker_timeout map_view,couchdb@localhost,<<"shards/00000000-1fffffff/vw.1487715840">>
> [error] 2017-02-21T22:39:58.931000Z couchdb@localhost <0.19734.6> d4985a33d1 fabric_worker_timeout map_view,couchdb@localhost,<<"shards/20000000-3fffffff/vw.1487715840">>
> [error] 2017-02-21T22:39:58.931000Z couchdb@localhost <0.19734.6> d4985a33d1 fabric_worker_timeout map_view,couchdb@localhost,<<"shards/40000000-5fffffff/vw.1487715840">>
> [error] 2017-02-21T22:39:58.931000Z couchdb@localhost <0.19734.6> d4985a33d1 fabric_worker_timeout map_view,couchdb@localhost,<<"shards/60000000-7fffffff/vw.1487715840">>
> [error] 2017-02-21T22:39:58.931000Z couchdb@localhost <0.19734.6> d4985a33d1 fabric_worker_timeout map_view,couchdb@localhost,<<"shards/80000000-9fffffff/vw.1487715840">>
> [error] 2017-02-21T22:39:58.931000Z couchdb@localhost <0.19734.6> d4985a33d1 fabric_worker_timeout map_view,couchdb@localhost,<<"shards/a0000000-bfffffff/vw.1487715840">>
> [error] 2017-02-21T22:39:58.931000Z couchdb@localhost <0.19734.6> d4985a33d1 fabric_worker_timeout map_view,couchdb@localhost,<<"shards/c0000000-dfffffff/vw.1487715840">>

> [error] 2017-02-21T22:39:58.931000Z couchdb@localhost <0.19734.6> d4985a33d1 fabric_worker_timeout map_view,couchdb@localhost,<<"shards/e0000000-ffffffff/vw.1487715840”>>

These timeouts are the expected behavior in 2.0 when a request for a view hits a configurable limit. The default timeout is 60 seconds. I believe this may have been a change from 1.x where the socket would sit open as long as necessary. If you need to recover that behavior you can set

[fabric]
request_timeout = infinity

You could also configure some other number in milliseconds:

; Give up after 10 seconds
[fabric]
request_timeout = 10000

In any case the indexing jobs should have continued even after this timeout. I think that’s why the request worked when you reloaded the page.

> [error] 2017-02-21T22:39:59.000000Z couchdb@localhost <0.19734.6> d4985a33d1 req_err(1329706011) unknown_error : function_clause
> [<<"couch_mrview_show:list_cb/2 L212">>,<<"fabric_view_map:go/7 L52">>,<<"couch_query_servers:with_ddoc_proc/2 L421">>,<<"chttpd:process_request/1 L293">>,<<"chttpd:handle_request_int/1 L229">>,<<"mochiweb_http:headers/6 L122">>,<<"proc_lib:init_p_do_apply/3 L237">>]
> [notice] 2017-02-21T22:39:59.002000Z couchdb@localhost <0.19734.6> d4985a33d1 127.0.0.1:5984 127.0.0.1 undefined GET /dbname/_design/appname/_list/data/customer_id?key=%22PRIV-SE270_FC_AZT10L16_016%22 500 ok 60218

Here the “60218” number is the response time in milliseconds, which confirms that you bumped into the default timeout. However, you should have gotten something more informative than this function_clause error response. That’s a bug in our error handling; if you like I’d encourage you to file a bug report:

https://issues.apache.org/jira/browse/COUCHDB <https://issues.apache.org/jira/browse/COUCHDB>

You’ll need an account first if you don’t already have one:

https://issues.apache.org/jira/secure/Signup!default.jspa <https://issues.apache.org/jira/secure/Signup!default.jspa>

> In the web browser, I get this:
>
> {"error":"unknown_error","reason":"function_clause","ref":1329706011}
>
> When the error happened, also I was replicating from another CouchDB server that has a large number of documents. I was running a test requesting the views as the db was getting filled in to see at what point I started to get the issue. So I started seeing the issue with about 17k documents (0.7GB).
>
> I have just reloaded the page and it now works, but I have not been able to make the view work on another machine with my complete DB which is much bigger (1/2 million docs, 22GB)
>
> This is what I see in the log in the machine with the large database:
>
> [info] 2017-02-21T23:08:26.127000Z couchdb@localhost <0.4854.995> -------- Starting index update for db: shards/00000000-1fffffff/vw.1481754819 idx: _design/adag

<snip>

> [info] 2017-02-21T23:09:14.252000Z couchdb@localhost <0.212.0> -------- couch_proc_manager <0.9345.4064> died normal
> [error] 2017-02-21T23:09:14.253000Z couchdb@localhost <0.11021.4075> -------- OS Process Error <0.9345.4064> :: {os_process_error,"OS process timed out."}

> [error] 2017-02-21T23:09:14.376000Z couchdb@localhost emulator -------- Error in process <0.11021.4075> on node 'couchdb@localhost' with exit value: {{nocatch,{os_process_error,"OS process timed out."}},[{couch_os_process,prompt,2,[{file,"src/couch_os_process.erl"},{line,59}]},{couch_query_servers,map_doc_raw,2,[{file,"src/couch_query_servers.erl"},{line,67}]},{couch_mrview_updater…

Now *this* is a separate issue. The “OS process timed out” error can be caused by a lot of things, but one of the most common is a large JSON document. I’ve seen documents around, say, 10 MB cause this timeout. Any chance you’ve got some of those hanging around? Again this is a configurable value which defaults to 5 seconds:

; Allow the system 20 seconds to process a document in a view
[couchdb]
os_process_timeout = 20000

At some point though this is a losing battle. Better to keep the documents under 1 MB if you have that flexibility.

> Any idea of what could be going on? I am running CouchDB 2.0.0.1 under Windows 7 with a single node. I have not modified most of the default CouchDB settings.

I haven’t kept pace with the current state of the art in the Windows build, so there may be other platform-specific issues at work with this “OS process timed out”. Cheers,

Adam

>
> Regards,
>
> Gustavo Delfino
>

[Joan Touzet]

> > Any idea of what could be going on? I am running CouchDB 2.0.0.1
> > under Windows 7 with a single node. I have not modified most of
> > the default CouchDB settings.
>
> I haven’t kept pace with the current state of the art in the Windows
> build, so there may be other platform-specific issues at work with
> this “OS process timed out”. Cheers,

Not to my knowledge, other than somewhat poorer performance than the
same machine running Linux.

Bump the limits that Adam describes, or get smaller documents; if
neither of those help please let us know.

-Joan

[Gustavo Delfino]

Adam,

Thank you for your help. Setting [fabric] request_timeout to infinity solved the first problem.

In regards to the second problem, it remains. I did a study about the size of my documents. I designed the db with documents as small as possible but a few of them are large. By doing a HEAD request for all my documents, I extracted the content-length for all of them. So my documents are like this:

Median size: 52k
Average size: 170k
3/4 of the documents are smaller than 229k
9/10 of the documents are smaller than 436k
Max size: 14000k

I have set [couchdb] os_process_timeout to 1000000 and still fails like this:

[info] 2017-02-22T23:16:22.533000Z couchdb@localhost <0.4854.995> -------- Starting index update for db: shards/00000000-1fffffff/vw.1481754819 idx: _design/appname
[info] 2017-02-22T23:16:22.533000Z couchdb@localhost <0.1391.995> -------- Starting index update for db: shards/40000000-5fffffff/vw.1481754819 idx: _design/appname
[info] 2017-02-22T23:16:22.533000Z couchdb@localhost <0.4588.995> -------- Starting index update for db: shards/60000000-7fffffff/vw.1481754819 idx: _design/appname
[info] 2017-02-22T23:16:22.533000Z couchdb@localhost <0.3463.995> -------- Starting index update for db: shards/80000000-9fffffff/vw.1481754819 idx: _design/appname
[info] 2017-02-22T23:16:22.533000Z couchdb@localhost <0.5295.995> -------- Starting index update for db: shards/c0000000-dfffffff/vw.1481754819 idx: _design/appname
[info] 2017-02-22T23:16:22.533000Z couchdb@localhost <0.5037.995> -------- Starting index update for db: shards/a0000000-bfffffff/vw.1481754819 idx: _design/appname
[info] 2017-02-22T23:16:22.533000Z couchdb@localhost <0.4141.984> -------- Starting index update for db: shards/20000000-3fffffff/vw.1481754819 idx: _design/appname
(...)
[info] 2017-02-22T23:17:02.547000Z couchdb@localhost <0.212.0> -------- couch_proc_manager <0.9875.4224> died normal
[error] 2017-02-22T23:17:03.069000Z couchdb@localhost <0.3010.4224> -------- OS Process Error <0.9875.4224> :: {os_process_error,"OS process timed out."}
[error] 2017-02-22T23:17:03.078000Z couchdb@localhost emulator -------- Error in process <0.3010.4224> on node 'couchdb@localhost' with exit value: {{nocatch,{os_process_error,"OS process timed out."}},[{couch_os_process,prompt,2,[{file,"src/couch_os_process.erl"},{line,59}]},{couch_query_servers,map_doc_raw,2,[{file,"src/couch_query_servers.erl"},{line,67}]},{couch_mrview_updater...
[error] 2017-02-22T23:17:03.078000Z couchdb@localhost <0.5772.4224> 2fa00b7084 rexi_server throw:{os_process_error,"OS process timed out."} [{couch_mrview_util,get_view,4,[{file,"src/couch_mrview_util.erl"},{line,56}]},{couch_mrview,query_view,6,[{file,"src/couch_mrview.erl"},{line,244}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,139}]}]
[error] 2017-02-22T23:17:03.079000Z couchdb@localhost <0.7330.4224> 2fa00b7084 req_err(3183461804) unknown_error : function_clause
[<<"couch_mrview_show:list_cb/2 L212">>,<<"fabric_view_map:go/7 L54">>,<<"couch_query_servers:with_ddoc_proc/2 L421">>,<<"chttpd:process_request/1 L293">>,<<"chttpd:handle_request_int/1 L229">>,<<"mochiweb_http:headers/6 L122">>,<<"proc_lib:init_p_do_apply/3 L237">>]
[notice] 2017-02-22T23:17:03.079000Z couchdb@localhost <0.7330.4224> 2fa00b7084 servername:5984 10.217.46.47 undefined GET /vw/_design/appname/_list/signals/trw_id?start_key=%22VDP480001%22;end_key=%22VDP480001\u9999%22; 500 ok 40560
[info] 2017-02-22T23:17:03.592000Z couchdb@localhost <0.212.0> -------- couch_proc_manager <0.29176.4223> died normal
[error] 2017-02-22T23:17:03.894000Z couchdb@localhost <0.6096.4224> -------- OS Process Error <0.29176.4223> :: {os_process_error,"OS process timed out."}
[error] 2017-02-22T23:17:03.895000Z couchdb@localhost <0.6780.4224> 2fa00b7084 rexi_server throw:{os_process_error,"OS process timed out."} [{couch_mrview_util,get_view,4,[{file,"src/couch_mrview_util.erl"},{line,56}]},{couch_mrview,query_view,6,[{file,"src/couch_mrview.erl"},{line,244}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,139}]}]
[error] 2017-02-22T23:17:03.901000Z couchdb@localhost emulator -------- Error in process <0.6096.4224> on node 'couchdb@localhost' with exit value: {{nocatch,{os_process_error,"OS process timed out."}},[{couch_os_process,prompt,2,[{file,"src/couch_os_process.erl"},{line,59}]},{couch_query_servers,map_doc_raw,2,[{file,"src/couch_query_servers.erl"},{line,67}]},{couch_mrview_updater...
[info] 2017-02-22T23:17:06.366000Z couchdb@localhost <0.212.0> -------- couch_proc_manager <0.7753.4224> died normal
[error] 2017-02-22T23:17:06.366000Z couchdb@localhost <0.4459.4224> -------- OS Process Error <0.7753.4224> :: {os_process_error,"OS process timed out."}
[error] 2017-02-22T23:17:06.367000Z couchdb@localhost emulator -------- Error in process <0.4459.4224> on node 'couchdb@localhost' with exit value: {{nocatch,{os_process_error,"OS process timed out."}},[{couch_os_process,prompt,2,[{file,"src/couch_os_process.erl"},{line,59}]},{couch_query_servers,map_doc_raw,2,[{file,"src/couch_query_servers.erl"},{line,67}]},{couch_mrview_updater...
[error] 2017-02-22T23:17:06.367000Z couchdb@localhost <0.11272.4224> 2fa00b7084 rexi_server throw:{os_process_error,"OS process timed out."} [{couch_mrview_util,get_view,4,[{file,"src/couch_mrview_util.erl"},{line,56}]},{couch_mrview,query_view,6,[{file,"src/couch_mrview.erl"},{line,244}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,139}]}]
[info] 2017-02-22T23:17:08.245000Z couchdb@localhost <0.212.0> -------- couch_proc_manager <0.8188.4224> died normal
[error] 2017-02-22T23:17:08.245000Z couchdb@localhost <0.28430.4223> -------- OS Process Error <0.8188.4224> :: {os_process_error,"OS process timed out."}
[error] 2017-02-22T23:17:08.245000Z couchdb@localhost emulator -------- Error in process <0.28430.4223> on node 'couchdb@localhost' with exit value: {{nocatch,{os_process_error,"OS process timed out."}},[{couch_os_process,prompt,2,[{file,"src/couch_os_process.erl"},{line,59}]},{couch_query_servers,map_doc_raw,2,[{file,"src/couch_query_servers.erl"},{line,67}]},{couch_mrview_updater...
[error] 2017-02-22T23:17:08.319000Z couchdb@localhost <0.5275.4224> 2fa00b7084 rexi_server throw:{os_process_error,"OS process timed out."} [{couch_mrview_util,get_view,4,[{file,"src/couch_mrview_util.erl"},{line,56}]},{couch_mrview,query_view,6,[{file,"src/couch_mrview.erl"},{line,244}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,139}]}]
[notice] 2017-02-22T23:17:11.735000Z couchdb@localhost <0.18522.1766> 39ca2df11d servername:5984 149.223.224.36 undefined GET /vw/_changes?feed=continuous&style=all_docs&since=%22482896-g1AAAAJ7eJzLYWBg4MhgTmEQTM4vTc5ISXLIyU9OzMnILy7JAUoxJTIkyf___z8rgzmJgeF1by5QjD3NwDjFPNkEmx48JiUpAMkke4Rha8CGJVmkmSWam5NqmAPIsHiEYS0QlxlbpiUakmxYAsiweoRh-8CGWSSaWliaGpBoWB4LkGRoAFJA8-ZDDTwHNtDYzCI1McWCLAMXQAzcDzHwTRAk7IzNU5LSUsky8ADEwPtQF94FG5honmRsaUielx9ADISF4Q2IgYmpaWaWadi0ZgEA3hnNtQ%22&timeout=10000 200 ok 10090
[info] 2017-02-22T23:17:22.138000Z couchdb@localhost <0.212.0> -------- couch_proc_manager <0.31614.4116> died normal
[error] 2017-02-22T23:17:22.139000Z couchdb@localhost <0.5831.4224> -------- OS Process Error <0.31614.4116> :: {os_process_error,"OS process timed out."}
[error] 2017-02-22T23:17:22.148000Z couchdb@localhost emulator -------- Error in process <0.5831.4224> on node 'couchdb@localhost' with exit value: {{nocatch,{os_process_error,"OS process timed out."}},[{couch_os_process,prompt,2,[{file,"src/couch_os_process.erl"},{line,59}]},{couch_query_servers,map_doc_raw,2,[{file,"src/couch_query_servers.erl"},{line,67}]},{couch_mrview_updater...
[error] 2017-02-22T23:17:22.148000Z couchdb@localhost <0.4214.4224> 2fa00b7084 rexi_server throw:{os_process_error,"OS process timed out."} [{couch_mrview_util,get_view,4,[{file,"src/couch_mrview_util.erl"},{line,56}]},{couch_mrview,query_view,6,[{file,"src/couch_mrview.erl"},{line,244}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,139}]}]
[notice] 2017-02-22T23:17:22.226000Z couchdb@localhost <0.18522.1766> db25906ce1 servername:5984 149.223.224.36 undefined GET /vw/_changes?feed=continuous&style=all_docs&since=%22482896-g1AAAAJ7eJzLYWBg4MhgTmEQTM4vTc5ISXLIyU9OzMnILy7JAUoxJTIkyf___z8rgzmJgeF1by5QjD3NwDjFPNkEmx48JiUpAMkke4Rha8CGJVmkmSWam5NqmAPIsHiEYS0QlxlbpiUakmxYAsiweoRh-8CGWSSaWliaGpBoWB4LkGRoAFJA8-ZDDTwHNtDYzCI1McWCLAMXQAzcDzHwTRAk7IzNU5LSUsky8ADEwPtQF94FG5honmRsaUielx9ADISF4Q2IgYmpaWaWadi0ZgEA3hnNtQ%22&timeout=10000 200 ok 10482
[info] 2017-02-22T23:17:24.372000Z couchdb@localhost <0.212.0> -------- couch_proc_manager <0.9283.4224> died normal
[error] 2017-02-22T23:17:24.372000Z couchdb@localhost <0.3599.4224> -------- OS Process Error <0.9283.4224> :: {os_process_error,"OS process timed out."}
[error] 2017-02-22T23:17:24.391000Z couchdb@localhost emulator -------- Error in process <0.3599.4224> on node 'couchdb@localhost' with exit value: {{nocatch,{os_process_error,"OS process timed out."}},[{couch_os_process,prompt,2,[{file,"src/couch_os_process.erl"},{line,59}]},{couch_query_servers,map_doc_raw,2,[{file,"src/couch_query_servers.erl"},{line,67}]},{couch_mrview_updater...
[error] 2017-02-22T23:17:24.488000Z couchdb@localhost <0.8575.4224> 2fa00b7084 rexi_server throw:{os_process_error,"OS process timed out."} [{couch_mrview_util,get_view,4,[{file,"src/couch_mrview_util.erl"},{line,56}]},{couch_mrview,query_view,6,[{file,"src/couch_mrview.erl"},{line,244}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,139}]}]
[notice] 2017-02-22T23:17:24.964000Z couchdb@localhost <0.3439.4224> 21b9175dee servername:5984 149.223.224.36 undefined GET /vw/_changes?since=%22482896-g1AAAAJ7eJzLYWBg4MhgTmEQTM4vTc5ISXLIyU9OzMnILy7JAUoxJTIkyf___z8rgzmJgeF1by5QjD3NwDjFPNkEmx48JiUpAMkke4Rha8CGJVmkmSWam5NqmAPIsHiEYS0QlxlbpiUakmxYAsiweoRh-8CGWSSaWliaGpBoWB4LkGRoAFJA8-ZDDTwHNtDYzCI1McWCLAMXQAzcDzHwTRAk7IzNU5LSUsky8ADEwPtQF94FG5honmRsaUielx9ADISF4Q2IgYmpaWaWadi0ZgEA3hnNtQ%22&limit=0 200 ok 4
[info] 2017-02-22T23:17:25.307000Z couchdb@localhost <0.212.0> -------- couch_proc_manager <0.4265.4224> died normal
[error] 2017-02-22T23:17:25.307000Z couchdb@localhost <0.25135.4223> -------- OS Process Error <0.4265.4224> :: {os_process_error,"OS process timed out."}
[error] 2017-02-22T23:17:25.326000Z couchdb@localhost emulator -------- Error in process <0.25135.4223> on node 'couchdb@localhost' with exit value: {{nocatch,{os_process_error,"OS process timed out."}},[{couch_os_process,prompt,2,[{file,"src/couch_os_process.erl"},{line,59}]},{couch_query_servers,map_doc_raw,2,[{file,"src/couch_query_servers.erl"},{line,67}]},{couch_mrview_updater...
[error] 2017-02-22T23:17:25.328000Z couchdb@localhost <0.7179.4224> 2fa00b7084 rexi_server throw:{os_process_error,"OS process timed out."} [{couch_mrview_util,get_view,4,[{file,"src/couch_mrview_util.erl"},{line,56}]},{couch_mrview,query_view,6,[{file,"src/couch_mrview.erl"},{line,244}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,139}]}]

Any order idea on what could be wrong?

Regards,

Gustavo Delfino

-----Original Message-----
From: Joan Touzet [mailto:woh...@apache.org]
Sent: Wednesday, February 22, 2017 1:00 PM
To: us...@couchdb.apache.org
Subject: Re: views failing due to fabric_worker_timeout and OS process timed out

> > Any idea of what could be going on? I am running CouchDB 2.0.0.1
> > under Windows 7 with a single node. I have not modified most of the
> > default CouchDB settings.
>
> I haven’t kept pace with the current state of the art in the Windows
> build, so there may be other platform-specific issues at work with
> this “OS process timed out”. Cheers,

Not to my knowledge, other than somewhat poorer performance than the same machine running Linux.

Bump the limits that Adam describes, or get smaller documents; if neither of those help please let us know.

-Joan

[Joan Touzet]

I recommend keeping document size below 1MB. It's almost certainly
your 14MB document that is causing issues with the view timing out.

Is your view particularly complex?

[Gustavo Delfino]

I installed CouchDB 1.6.1 on a separate machine and copied all the data. Initially it failed too, but then I read about increasing the memory to couchjs.exe and now it seems to be working (the view is 50% ready after a few hours):

http://stackoverflow.com/questions/21273736/couchdb-views-os-process-error-big-documents

I changed [query_servers] javascript from

./couchjs.exe ../share/couchdb/server/main.js

to

./couchjs.exe -S 268435456 ../share/couchdb/server/main.js

This is four times the default amount of memory.

Unfortunately this doesn't work with CouchDB 2.0.0 (see https://github.com/apache/couchdb-couch/pull/216 )

Do you think that switching to Erlang view server would better for handling big documents?

> 9}]}] [error] 2017-02-22T23:17:03.901000Z couchdb@localhost emulator
> -------- Error in process <0.6096.4224> on node 'couchdb@localhost'
> with exit value: {{nocatch,{os_process_error,"OS process timed
> out."}},[{couch_os_process,prompt,2,[{file,"src/couch_os_process.erl"},{line,59}]},{couch_query_servers,map_doc_raw,2,[{file,"src/couch_query_servers.erl"},{line,67}]},{couch_mrview_updater...
> [info] 2017-02-22T23:17:06.366000Z couchdb@localhost <0.212.0>
> -------- couch_proc_manager <0.7753.4224> died normal [error]
> 2017-02-22T23:17:06.366000Z couchdb@localhost <0.4459.4224>
> -------- OS Process Error <0.7753.4224> :: {os_process_error,"OS
> process timed out."} [error] 2017-02-22T23:17:06.367000Z
> couchdb@localhost emulator
> -------- Error in process <0.4459.4224> on node 'couchdb@localhost'
> with exit value: {{nocatch,{os_process_error,"OS process timed
> out."}},[{couch_os_process,prompt,2,[{file,"src/couch_os_process.erl"},{line,59}]},{couch_query_servers,map_doc_raw,2,[{file,"src/couch_query_servers.erl"},{line,67}]},{couch_mrview_updater...
> [error] 2017-02-22T23:17:06.367000Z couchdb@localhost <0.11272.4224>
> 2fa00b7084 rexi_server throw:{os_process_error,"OS process timed
> out."}
> [{couch_mrview_util,get_view,4,[{file,"src/couch_mrview_util.erl"},{li

> ne,56}]},{couch_mrview,query_view,6,[{file,"src/couch_mrview.erl"},{li
> ne,244}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,13

> 9}]}] [info] 2017-02-22T23:17:08.245000Z couchdb@localhost <0.212.0>
> -------- couch_proc_manager <0.8188.4224> died normal [error]
> 2017-02-22T23:17:08.245000Z couchdb@localhost <0.28430.4223>
> -------- OS Process Error <0.8188.4224> :: {os_process_error,"OS
> process timed out."} [error] 2017-02-22T23:17:08.245000Z
> couchdb@localhost emulator
> -------- Error in process <0.28430.4223> on node 'couchdb@localhost'
> with exit value: {{nocatch,{os_process_error,"OS process timed
> out."}},[{couch_os_process,prompt,2,[{file,"src/couch_os_process.erl"},{line,59}]},{couch_query_servers,map_doc_raw,2,[{file,"src/couch_query_servers.erl"},{line,67}]},{couch_mrview_updater...
> [error] 2017-02-22T23:17:08.319000Z couchdb@localhost <0.5275.4224>
> 2fa00b7084 rexi_server throw:{os_process_error,"OS process timed
> out."}
> [{couch_mrview_util,get_view,4,[{file,"src/couch_mrview_util.erl"},{li

> ne,56}]},{couch_mrview,query_view,6,[{file,"src/couch_mrview.erl"},{li
> ne,244}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,13

> 9}]}] [notice] 2017-02-22T23:17:11.735000Z couchdb@localhost
> <0.18522.1766> 39ca2df11d servername:5984 149.223.224.36 undefined GET
> /vw/_changes?feed=continuous&style=all_docs&since=%22482896-g1AAAAJ7eJ
> zLYWBg4MhgTmEQTM4vTc5ISXLIyU9OzMnILy7JAUoxJTIkyf___z8rgzmJgeF1by5QjD3N
> wDjFPNkEmx48JiUpAMkke4Rha8CGJVmkmSWam5NqmAPIsHiEYS0QlxlbpiUakmxYAsiweo
> Rh-8CGWSSaWliaGpBoWB4LkGRoAFJA8-ZDDTwHNtDYzCI1McWCLAMXQAzcDzHwTRAk7IzN
> U5LSUsky8ADEwPtQF94FG5honmRsaUielx9ADISF4Q2IgYmpaWaWadi0ZgEA3hnNtQ%22&
> timeout=10000
> 200 ok 10090
> [info] 2017-02-22T23:17:22.138000Z couchdb@localhost <0.212.0>
> -------- couch_proc_manager <0.31614.4116> died normal [error]
> 2017-02-22T23:17:22.139000Z couchdb@localhost <0.5831.4224>
> -------- OS Process Error <0.31614.4116> :: {os_process_error,"OS
> process timed out."} [error] 2017-02-22T23:17:22.148000Z
> couchdb@localhost emulator
> -------- Error in process <0.5831.4224> on node 'couchdb@localhost'
> with exit value: {{nocatch,{os_process_error,"OS process timed
> out."}},[{couch_os_process,prompt,2,[{file,"src/couch_os_process.erl"},{line,59}]},{couch_query_servers,map_doc_raw,2,[{file,"src/couch_query_servers.erl"},{line,67}]},{couch_mrview_updater...
> [error] 2017-02-22T23:17:22.148000Z couchdb@localhost <0.4214.4224>
> 2fa00b7084 rexi_server throw:{os_process_error,"OS process timed
> out."}
> [{couch_mrview_util,get_view,4,[{file,"src/couch_mrview_util.erl"},{li

> ne,56}]},{couch_mrview,query_view,6,[{file,"src/couch_mrview.erl"},{li
> ne,244}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,13

> 9}]}] [notice] 2017-02-22T23:17:22.226000Z couchdb@localhost
> <0.18522.1766>
> db25906ce1 servername:5984 149.223.224.36 undefined GET
> /vw/_changes?feed=continuous&style=all_docs&since=%22482896-g1AAAAJ7eJ
> zLYWBg4MhgTmEQTM4vTc5ISXLIyU9OzMnILy7JAUoxJTIkyf___z8rgzmJgeF1by5QjD3N
> wDjFPNkEmx48JiUpAMkke4Rha8CGJVmkmSWam5NqmAPIsHiEYS0QlxlbpiUakmxYAsiweo
> Rh-8CGWSSaWliaGpBoWB4LkGRoAFJA8-ZDDTwHNtDYzCI1McWCLAMXQAzcDzHwTRAk7IzN
> U5LSUsky8ADEwPtQF94FG5honmRsaUielx9ADISF4Q2IgYmpaWaWadi0ZgEA3hnNtQ%22&
> timeout=10000
> 200 ok 10482
> [info] 2017-02-22T23:17:24.372000Z couchdb@localhost <0.212.0>
> -------- couch_proc_manager <0.9283.4224> died normal [error]
> 2017-02-22T23:17:24.372000Z couchdb@localhost <0.3599.4224>
> -------- OS Process Error <0.9283.4224> :: {os_process_error,"OS
> process timed out."} [error] 2017-02-22T23:17:24.391000Z
> couchdb@localhost emulator
> -------- Error in process <0.3599.4224> on node 'couchdb@localhost'
> with exit value: {{nocatch,{os_process_error,"OS process timed
> out."}},[{couch_os_process,prompt,2,[{file,"src/couch_os_process.erl"},{line,59}]},{couch_query_servers,map_doc_raw,2,[{file,"src/couch_query_servers.erl"},{line,67}]},{couch_mrview_updater...
> [error] 2017-02-22T23:17:24.488000Z couchdb@localhost <0.8575.4224>
> 2fa00b7084 rexi_server throw:{os_process_error,"OS process timed
> out."}
> [{couch_mrview_util,get_view,4,[{file,"src/couch_mrview_util.erl"},{li

> ne,56}]},{couch_mrview,query_view,6,[{file,"src/couch_mrview.erl"},{li
> ne,244}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,13

> 9}]}] [notice] 2017-02-22T23:17:24.964000Z couchdb@localhost
> <0.3439.4224> 21b9175dee servername:5984 149.223.224.36 undefined GET
> /vw/_changes?since=%22482896-g1AAAAJ7eJzLYWBg4MhgTmEQTM4vTc5ISXLIyU9Oz
> MnILy7JAUoxJTIkyf___z8rgzmJgeF1by5QjD3NwDjFPNkEmx48JiUpAMkke4Rha8CGJVm
> kmSWam5NqmAPIsHiEYS0QlxlbpiUakmxYAsiweoRh-8CGWSSaWliaGpBoWB4LkGRoAFJA8
> -ZDDTwHNtDYzCI1McWCLAMXQAzcDzHwTRAk7IzNU5LSUsky8ADEwPtQF94FG5honmRsaUi
> elx9ADISF4Q2IgYmpaWaWadi0ZgEA3hnNtQ%22&limit=0
> 200 ok 4
> [info] 2017-02-22T23:17:25.307000Z couchdb@localhost <0.212.0>
> -------- couch_proc_manager <0.4265.4224> died normal [error]
> 2017-02-22T23:17:25.307000Z couchdb@localhost <0.25135.4223>
> -------- OS Process Error <0.4265.4224> :: {os_process_error,"OS
> process timed out."} [error] 2017-02-22T23:17:25.326000Z
> couchdb@localhost emulator
> -------- Error in process <0.25135.4223> on node 'couchdb@localhost'
> with exit value: {{nocatch,{os_process_error,"OS process timed
> out."}},[{couch_os_process,prompt,2,[{file,"src/couch_os_process.erl"},{line,59}]},{couch_query_servers,map_doc_raw,2,[{file,"src/couch_query_servers.erl"},{line,67}]},{couch_mrview_updater...
> [error] 2017-02-22T23:17:25.328000Z couchdb@localhost <0.7179.4224>
> 2fa00b7084 rexi_server throw:{os_process_error,"OS process timed
> out."}
> [{couch_mrview_util,get_view,4,[{file,"src/couch_mrview_util.erl"},{li

> ne,56}]},{couch_mrview,query_view,6,[{file,"src/couch_mrview.erl"},{li
> ne,244}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,13
> 9}]}]
>

[Joan Touzet]

Switching to Mango would be best. :)

[Gustavo Delfino]

I tried using Mango and the index builds without errors and much faster. This is great, however I am trying to understand how this fits with the rest of CouchDB. In particular, with Mango it looks like we no longer "You query your view to retrieve the view result" and this would imply that converting couchapps to Mango is not trivial. Is this correct?

[Joan Touzet]

So glad Mango worked well for you! In effect, Mango uses the same
mapreduce functionality under the covers as a traditional JS view,
but the implementation is optimized and the domain-specific language
is intended to be convenient and friendly to use.

Correct. couchapps are, in effect, deprecated functionality. There
should be ways to enable that functionality in future versions of
CouchDB, but it most likely won't be the default configuration, and
performance will not be optimized for couchapps.

-Joan

[Aurélien Bénel]

Hi Joan,

> couchapps are, in effect, deprecated functionality.

What do you mean exactly by saying this?
That CouchDB is now just a DB and no more an application server?
That all the time and code invested by my team and my partners since 2010 are now wasted?
That I should stop immediately teaching CouchDB in my university as a perfect example on how to implement a RESTful API and Web application?

Which features needed for « couchapps » are going to disappear in future releases?

Is it a definitive decision? Or is it still possible to discuss it?


Regards,

Aurélien

P.S. Sorry if my e-mail looks aggressive, but I’m really really worried…
I’m surprised too to discover such an important news on a nearly unrelated topic, whereas the last time the need for « couchapps » were discussed on the « user » list, a few years ago, there was a consensus that it was a really important feature for a significant part of the user base.

[Joan Touzet]

HI Aurélien,

Your email is aggressive, and your apology is not accepted. I disagree that a "significant part of the user base" needs this functionality, as do the overwhelming majority of the PMC and the commercial entities on this mailing list.

This topic has been discussed to death on the mailing lists and I am not going to be pulled into a retread of this argument.

For the latest developer discussions on this topic, search for the words "Deprecation Wishlist" in this message:

http://mail-archives.apache.org/mod_mbox/couchdb-dev/201702.mbox/%3CB6DB98EC-42B1-4960-9E43-257F040238F1%40apache.org%3E

You may also want to search on the word "NoJS".

-Joan

----- Original Message -----
> From: "Aurélien Bénel" <aurelie...@utt.fr>
> To: us...@couchdb.apache.org
> Sent: Saturday, February 25, 2017 9:33:38 AM
> Subject: Re: views failing due to fabric_worker_timeout and OS process timed out
>

[Aurélien Bénel]

Hi Joan,

> Your email is aggressive, and your apology is not accepted.

I didn’t want it to be. I beg you for your pardon then.
My frustration was real, but I can assure you that I am not an aggressive person.
There would not have been any ambiguity in my mother language :
discussing technologies in a foreign language is one thing, expressing your feelings is another.

> This topic has been discussed to death on the mailing lists and I am not going to be pulled into a retread of this argument.

> http://mail-archives.apache.org/mod_mbox/couchdb-dev/201702.mbox/%3CB6DB98EC-42B1-4960-9E43-257F040238F1%40apache.org%3E

I’m just a « user »… a very dedicated and passionated user (I’m in the top 10% on StackOverflow about CouchDB and I taught CouchDB to more than 150 french software engineers), but a user. That’s why I never subscribed to the « dev » mailing list (or for a very short period of time). I now understand that I should have, but it’s too late.

My frustration is as high as has been my passion for six years for this incredibly interesting project.
I respect the board decisions but now I will have a hard time finding money (which is sparse in academic research) to move all of our software to a different technology stack and arguments to explain to all of my collaborators that I bet on a technology stack that got rapidly deprecated.

Thank you for your understanding.


Best regards,

Aurélien

[Joan Touzet]

----- Original Message -----
> From: "Aurélien Bénel" <aurelie...@utt.fr>

> I’m just a « user »… a very dedicated and passionated user (I’m in
> the top 10% on StackOverflow about CouchDB and I taught CouchDB to
> more than 150 french software engineers), but a user. That’s why I
> never subscribed to the « dev » mailing list (or for a very short
> period of time). I now understand that I should have, but it’s too
> late.

At Apache, all people with an opinion are entitled to express that
opinion and, where appropriate, have it considered by the community.

Naturally we'd like to have consensus on all decisions we make, but
ultimately some more controversial decisions have to be voted upon.
Only acknowledged and accredited project committers can vote in those
decisions. The policy for this is summarized in our project bylaws at

https://couchdb.apache.org/bylaws.html

and clear guidelines are presented.

Of course, it is best for us to listen to all users who use the
software. But many of us who develop and manage Apache CouchDB with
commercial interest have more experience with CouchDB than you do, have
dealt with orders of magnitude more users and developers than you have,
and thus have a more intimate understanding of the poor design decisions
made nearly 10 years ago now.

10 years is an epoch, given the advancement of technology. The project
needs to evolve, not just because the world has moved on from CGI-style
web server implementations, but because we have found more fruitful
design choices that play to our strengths. Ultimately, this means some
API endpoints will be deprecated, and some new endpoints will be
introduced - just like other long-lived projects that have had to make
similar decisions. That exact list will be hammered out on the dev@
mailing list over the next several months and will be widely
communicated.

> My frustration is as high as has been my passion for six years for
> this incredibly interesting project.
> I respect the board decisions but now I will have a hard time finding
> money (which is sparse in academic research) to move all of our
> software to a different technology stack and arguments to explain to
> all of my collaborators that I bet on a technology stack that got
> rapidly deprecated.

CouchDB 1.x and 2.x will forever support everything you see in the API
today, though deprecation announcements may be forthcoming on certain
calls for the 3.x and future releases. I'm positive there will be people
continuing to use 1.x/2.x for another 10 years; there is absolutely no
reason to migrate off of those versions if your needs are satisfied
there. As examples, I still know people using perl 4, python 2, Oracle
10.x, Linux 2.x. The community will continue to embrace and support
older CouchDB versions as long as it is practical to do so.

> Thank you for your understanding.

And you for yours.

-Joan

[Jan Lehnardt]

Aurélien,

I see that at least at some point you were subscribed and participating on the couc...@couchdb.apache.org mailing list. From the stated goal of the list (find a new technical foundation for CouchApp) and the lack of significant engagement (users and devs alike) there, it should have been clear where this is headed.

And just to reiterate:

1. CouchApp was an attempt to revolutionise web development as we know it. — It failed, in like 2011.

2. It was designed in a world before Node.js. Most folks who want to do JavaScript and CouchDB have moved on.

3. There are SEVERE technical limitations, most of which aren’t as bad as a view index generator, but VERY bad for anything OLTP (think CGI from 90s).

4. The features are unmaintained at this point, future refactorings might make the unavailable (e.g. in a http layer rewrite). The last significant work on the relevant code is 5-6 years in the past.

5.We invited the CouchApp community to step up and build a future-ready version of CouchApps, complete with a design direction and own mailing list.. Nobody stepped up, and at the end of the day, a project goes where developers can spend time.

6. and to be clear, we are talking about: 1. _show & _list 2. _update funs, 3. rewrites // for the time being, we’ll keep validate_doc_update and filter functions, but plan to replace them with per-doc access control and Mango schema enforcement. The idea of design docs, or attachments on documents are not going away.

In terms of ease of building web apps: a Node.js process running next to CouchDB is only minimally more setup hassle and gives you:

1. The same baseline features, plus a lot more.
2. A simple app building model.
3. A RICH ecosystem of third party libraries.
4. WAAAAAAAY better performance and scalability.
5. A future for you to do just the things you are already doing without moving to another platform.

Best
Jan
--

--
Professional Support for Apache CouchDB:
https://neighbourhood.ie/couchdb-support/

[Martin Broerse]

We use the hosting from couchapp for many projects via
https://www.npmjs.com/package/ember-cli-deploy-couchdb so keep it in
couchdb. To replace excel sheets in businesses it is super you don't need a
separate hosting stack. An example couchapp hosted only on Cloudant:
https://bloggr.exmer.com

- Martin

[Jan Lehnardt]

Hi Martin,

thanks for your comment.

> On 27 Feb 2017, at 07:52, Martin Broerse <in...@martinbroerse.com> wrote:
>
> We use the hosting from couchapp for many projects via
> https://www.npmjs.com/package/ember-cli-deploy-couchdb so keep it in
> couchdb. To replace excel sheets in businesses it is super you don't need a
> separate hosting stack. An example couchapp hosted only on Cloudant:
> https://bloggr.exmer.com

Existing versions of CouchDB that support CouchApps aren’t going away,
and I’m sure Cloudant will keep things around for a while, too.

This is about the future of CouchDB and the non-existent developer
time that is required to maintain these features as CouchDB evolves.

Best
Jan
--

[Martin Broerse]

Hi Jan,

If we split thinking about CouchApp in the hosting part and the backend
coding part it would not hurt our usage if we lose the coding part. The
coding part we need on the backend like password resetmails and other
scheduled tasks are not there so the coding part needs to be more powerful
before we can use it. We can solve this tasks with OpenWhisk so perhaps
keep the hosting feature and lose the rest?

The Ember guys at LinkedIn found it is faster to eval javascript loaded as
strings than loading the javascript from the backend. We have not tested
this yet but if this is true we can perhaps bootstrap javascript apps from
strings hosted in CouchDB but we still need the CouchDB hosting part for
the bootstrap code.

So in the future we are for keeping the hosting and lose the rest.

- Martin

[Jan Lehnardt]

Not sure what you mean with `the hosing feature`, but attachments are not
going away.

FWIW, I’d be in favour of having a modern CouchApp platform hooked into
CouchDB, but in the past 5 years we haven’t found anyone who’d be willing
to put in the work.

Best
Jan
--

[Martin Broerse]

Hi Jan,

By `the hosting feature` I mean the rewrite rules and the vhost. Good to
hear that is not going away.

- Martin

[Jan Lehnardt]

> On 27 Feb 2017, at 11:23, Martin Broerse <in...@martinbroerse.com> wrote:
>
> Hi Jan,
>

> By `the hosting feature` I mean the rewrite rules and the vhost. Good to
> hear that is not going away.

I said attachments are not going away. rewrites and vhosts are probably
not going to be re-implemented in a new http layer (unless we find
someone who wants to write the code and maintain it, *hint* *hint*)

[Aurélien Bénel]

Hi Jan,

> 6. and to be clear, we are talking about: 1. _show & _list 2. _update funs, 3. rewrites

Thank you very much for the clarification.
As you can imagine, it is greatly needed to plan our own developments.

> 5.We invited the CouchApp community to step up and build a future-ready version of CouchApps, complete with a design direction and own mailing list.. Nobody stepped up, and at the end of the day, a project goes where developers can spend time.

If I may say so, I doubt a participative process could have work this way with no facilitator in charge from the very beginning. But be sure that I’m not here to blame anyone. And since we don’t have a time machine, this would be useless. So, let’s focus on the future.

> In terms of ease of building web apps: a Node.js process running next to CouchDB is only minimally more setup hassle and gives you (…)

Right. In several of our applications, we already have an ExpressJS layer in front of CouchDB in order to circumvent its limitations/constraints.

Transforming `_show` and `_list` into ExpressJS middleware shouldn’t be that hard… except for the lack of `userCtx`.
How do you plan to manage users in this new architecture?
- Where will be the right place to handle session cookies? in CouchDB or in NodeJS?
- If it is in NodeJS:
- Will `proxy_auth` become the only authentication mode?
- Will `_users` database become obsolete?
- How per-doc access-control will be designed? Only with users defined elsewhere?

On a more theoretic side, I am a bit sad of the disappearance of `_show` and `_list` since, by providing the building blocks to content negotiation (when used with rewrites.json « hacks »), they made CouchDB the ultimate illustration of HTTP and REST concepts (even better than Atom Publication Protocol!). I was so impressed by this… And I was so excited when I first provided CSV, FreeMind, BibTeX (depending on the domain) alternate representations… May I hope that in the future content negotiation could be made first class citizen in CouchDB again (even in a better – less hacky – way)?

As for the disappearance of rewrites… I was just discovering the potential (and the bugs) of the brand new JavaScript rewrite… As soon as it is born, it has to die. Wow.

> // for the time being, we’ll keep validate_doc_update and filter functions, but plan to replace them with per-doc access control and Mango schema enforcement. The idea of design docs, or attachments on documents are not going away.

OK. Just to be sure (because I’m not sure to understand what « NoJS mode » means exactly):
- Will JavaScript views still exist in CouchDB 3?
- If the answer is no, will Mango be the only option or will it be still possible to implement Erlang map/reduce views?

I ask this because we put much love into our CouchDB views and succeeded in getting quite efficient implementations for things that did not scale on PostgreSQL. These views are clearly out of the scope of Mango, and would probably need good skills in Erlang to be ported…


Regards,

Aurélien

[Martin Broerse]

Hi Jan,

We use the vhost and rewrite rules everywhere via
https://www.npmjs.com/package/ember-cli-deploy-couchdb . Sofa (
https://github.com/ampatspell/ember-cli-sofa) is a new project that started
using this. We are moving clients to Cloudant and because of the vhost it
is all they need to run their applications.

How is the voting for keeping this feature done?

- Martin