Understanding a couple replication errors

[Carlos Alonso]

Hi all!!

I've been observing logs for a while and trying to understand why a few
errors happen while (pull) replication is happening and also trying to
understand the consequences.

** connection_closed*

Log:
```
[error] <0.15968.1946> -------- Replicator, request GET to
"https://admin:*****@source.ip:443/my_db/_changes?feed=continuous&style=all_docs&since=%2210613426-g1AAAAOCeJyd0r1OwzAQB3CrRYIHYC8FCSZS2_kwEUM7svURiu1zVKqQSlUZYKFP0FeAV2DsBE_BCm-Suj4UKqRYIctZsv66n053OSHkaNoFcgpKzxdmBIrRQEt1lz1SygKdzx9AFsugMMvcZjuSqF5ZlrNpV5Hj8-t7-3eYZJyDzoD0qyZhTY8_lPBR6sRWNay0_pvTmJA8ScS-xhppPPJqo502qbSbwGlUUwNXrMVsiVe73WnPlXbx5DSQXBihW8xWF3NacWArWdnHgi8oXn7ifGHEeST_L_rPBMVXFN9RHORO5FowIeKm4m_KeywIfiD4heD6zIFpbCAL4xYrrIvtid8o_qxxvHGiiVKTQqM1zrad2hfx%22&timeout=10000"
failed due to error connection_closed
```

Cause: This one happens from time to time. It looks like the source is
closing connection, but why?
Consequences: None. I think replicator just retries the task.

* *changes_reader_died*

Log:
```
[error] <0.15968.1946> -------- Replicator, request GET to
"https://admin:*****@source.ip:443/my_db/_changes?feed=continuous&style=all_docs&since=%2235690238-g1...&timeout=10000"
failed due to error connection_closed

[error] <0.13952.1946> -------- ChangesReader process died with reason:
{http_request_failed,"GET","
https://admin:*****@source.ip:443/my_db/_changes?feed=continuous&style=all_docs&since=%2235690238-g1...&timeout=10000",{error,connection_closed
}}

[error] <0.13952.1946> -------- Replication
`363ed31e35361d024d22e0a990ca1ae5+continuous`
(`https://admin:*****@source.ip:443/my_db/`
-> `http://admin:*****@127.0.0.1:5984/my_db/`) failed: changes_reader_died

[error] <0.30404.6252> -------- Replicator, request GET to
"https://admin:*****@source.ip:443/my_db/b666ee7ec8ed5b82ae035fe5fa6a4a2c?revs=true&open_revs=%5B%221-5a7ba8a8658bd0ab7b3656aa089aa183%22%5D&latest=true"
failed due to error {error,closing_on_request}

[error] <0.352.0> -------- Error in replication
`363ed31e35361d024d22e0a990ca1ae5+continuous` (triggered by document
`28d973af08931f79fa423b54a8004b31`): changes_reader_died
Restarting replication in 5 seconds.

[error] <0.13952.1946> -------- gen_server <0.13952.1946> terminated with
reason: changes_reader_died
last msg:
{'EXIT',<0.15968.1946>,{http_request_failed,"GET","https://admin:
*****@source.ip:443/my_db/_changes?feed=continuous&style=all_docs&since=%2235690238-g1...&timeout=10000",{error,connection_closed}}}
state:
[{data,[{"State",{rep_state,{rep,{"363ed31e35361d024d22e0a990ca1ae5","+continuous"},{httpdb,"https://admin:
*****@source.ip:443/my_db/",nil,[{"Accept","application/json"},{"User-Agent","CouchDB-Replicator/2.0.0"}],30000,[{is_ssl,true},{socket_options,[{keepalive,true},{nodelay,false}]},{ssl_options,[{depth,3},{verify,0}]}],10,250,nil,20},{httpdb,"http://admin:
*****@127.0.0.1:5984/my_db/
",nil,[{"Accept","application/json"},{"User-Agent","CouchDB-Replicator/2.0.0"}],30000,[{socket_options,[{keepalive,true},{nodelay,false}]}],10,250,nil,20},[{checkpoint_interval,30000},{connection_timeout,30000},{continuous,true},{http_connections,20},{retries,10},{socket_options,[{keepalive,true},{nodelay,false}]},{use_checkpoints,true},{worker_batch_size,500},{worker_processes,4}],{user_ctx,null,[],undefined},db,nil,<<"28d973af08931f79fa423b54a8004b31">>,<<"shards/c0000000-dfffffff/_replicator.1501167373">>},"https://admin:
*****@source.ip:443/my_db/","http://admin:*****@127.0.0.1:5984/my_db/
",{httpdb,"https://admin:
*****@source.ip:443/my_db/",nil,[{"Accept","application/json"},{"User-Agent","CouchDB-Replicator/2.0.0"}],30000,[{is_ssl,true},{socket_options,[{keepalive,true},{nodelay,false}]},{ssl_options,[{depth,3},{verify,0}]}],10,250,<0.1722.1946>,20},{httpdb,"http://admin:
*****@127.0.0.1:5984/my_db/",nil,[{"Accept","application/json"},{"User-Agent","CouchDB-Replicator/2.0.0"}],30000,[{socket_options,[{keepalive,true},{nodelay,false}]}],10,250,<0.21751.1946>,20},[],{[{<<"session_id">>,<<"6f7aa01e879f5e1841cdb7100f40806b">>},{<<"source_last_seq">>,<<"36567936-g1...">>},{<<"replication_id_version">>,3},{<<"history">>,[{[{<<"session_id">>,<<"6f7aa01e879f5e1841cdb7100f40806b">>},{<<"start_time">>,<<"Fri,
28 Jul 2017 09:46:48 GMT">>},{<<"end_time">>,<<"Sun, 30 Jul 2017 02:28:59
GMT">>},{<<"start_last_seq">>,0},{<<"end_last_seq">>,<<"36567936-g1...">>},{<<"recorded_seq">>,<<"36567936-g1...">>},{<<"missing_checked">>,22991404},{<<"missing_found">>,22991404},{<<"docs_read">>,22991404},{<<"docs_written">>,22991404},{<<"doc_write_failures">>,0}]}]}]},{0,0},{46381,<<"36567936-g1...">>},{46388,<<"36574076-g1...">>},[{46389,<<"36575175-g1...">>},{46390,<<"36575938-g1...">>},{46391,<<"36576695-...">>},{46392,<<"36577391-g1...">>}],{46388,<<"36574076-g1...">>},{doc,<<"_local/363ed31e35361d024d22e0a990ca1ae5">>,{0,[<<"4647">>]},{[]},[],false,[]},{doc,<<"_local/363ed31e35361d024d22e0a990ca1ae5">>,{0,[<<"4647">>]},{[]},[],false,[]},"Fri,
28 Jul 2017 09:46:48
GMT",<<"0">>,<<"0">>,nil,<0.16590.1946>,<0.16953.1946>,<0.15968.1946>,[<0.18469.1946>,<0.10009.1946>,<0.16617.1946>,<0.16155.1946>],[{doc_write_failures,0},{docs_read,22994904},{docs_written,22994904},{missing_checked,22994904},{missing_found,22994904}],<<"6f7aa01e879f5e1841cdb7100f40806b">>,nil,nil,nil,nil,<<"255587270-g1...">>,true,30000,db,nil}}]}]

[error] <0.13952.1946> -------- CRASH REPORT Process (<0.13952.1946>) with
2 neighbors exited with reason: changes_reader_died at
gen_server:terminate/6(line:737) <= proc_lib:init_p_do_apply/3(line:237);
initial_call: {couch_replicator,init,['Argument__1']}, ancestors:
[couch_replicator_job_sup,couch_replicator_sup,<0.349.0>], messages: [],
links: [<0.16590.1946>,<0.16953.1946>,<0.18469.1946>,<0.16617.1946>,...],
dictionary:
[{task_status_props,[{changes_pending,127404519},{checkpoint_interval,...},...]},...],
trap_exit: true, status: running, heap_size: 4185, stack_size: 27,
reductions: 26502159

[error] <0.364.0> -------- Supervisor couch_replicator_job_sup had child
"363ed31e35361d024d22e0a990ca1ae5+continuous" started with
{gen_server,start_link,undefined} at <0.13952.1946> exit with reason
changes_reader_died in context child_terminated

[error] <0.16617.1946> -------- gen_server <0.16617.1946> terminated with
reason: changes_reader_died
last msg: {'EXIT',<0.13952.1946>,changes_reader_died}
state: {state,<0.13952.1946>,<0.15004.1946>,20,{httpdb,"
https://admin:******@source.ip:443/my_db/",nil,[{"Accept","application/json"},{"User-Agent","CouchDB-Replicator/2.0.0"}],30000,[{is_ssl,true},{socket_options,[{keepalive,true},{nodelay,false}]},{ssl_options,[{depth,3},{verify,0}]}],10,250,<0.1722.1946>,20},{httpdb,"http://admin:******@127.0.0.1:5984/my_db/",nil,[{"Accept","application/json"},{"User-Agent","CouchDB-Replicator/2.0.0"}],30000,[{socket_options,[{keepalive,true},{nodelay,false}]}],10,250,<0.21751.1946>,20},[<0.2293.6254>,<0.19041.6253>,<0.19785.6253>,<0.7470.6253>,<0.23117.6253>,<0.31026.6250>,<0.11251.6253>,<0.27821.6253>,<0.8477.6253>,<0.28980.6253>,<0.22848.6253>,<0.16549.6251>,<0.26208.6253>,<0.27365.6253>,<0.15830.6253>,<0.24330.6253>,<0.20562.6253>,<0.20869.6253>,<0.21799.6253>,<0.16311.6253>],nil,{{<0.15004.1946>,#Ref<0.0.6636.248998>},{<<"07597cb4d450c2b6ff674b675b9d5168">>,[{1,<<93,162,57,7,69,142,28,226,77,41,199,156,216,131,208,152>>}],[]}},nil,[{docs_read,199},{docs_written,25},{missing_checked,500},{missing_found,500}],nil,nil,{batch,[<<"{\"_id\":\"b65rnw8pcm0jbt7jdpxvgmo1r\",\"_rev\":\"2-4d5283c4d125a0da73b95cc80b92abd6\
",...}}

[error] <0.16155.1946> -------- gen_server <0.16155.1946> terminated with
reason: changes_reader_died
last msg: {'EXIT',<0.13952.1946>,changes_reader_died}
state: {state,<0.13952.1946>,<0.21653.1946>,20,{httpdb,"
https://admin:******@source.ip:443/my_db/",nil,[{"Accept","application/json"},{"User-Agent","CouchDB-Replicator/2.0.0"}],30000,[{is_ssl,true},{socket_options,[{keepalive,true},{nodelay,false}]},{ssl_options,[{depth,3},{verify,0}]}],10,250,<0.1722.1946>,20},{httpdb,"http://admin:******@127.0.0.1:5984/my_db/",nil,[{"Accept","application/json"},{"User-Agent","CouchDB-Replicator/2.0.0"}],30000,[{socket_options,[{keepalive,true},{nodelay,false}]}],10,250,<0.21751.1946>,20},[<0.27802.6253>,<0.27692.6253>,<0.10519.6253>,<0.5584.6253>,<0.9485.6253>,<0.2597.6253>,<0.12901.6253>,<0.21982.6253>,<0.20758.6253>,<0.12367.6253>,<0.30450.6253>,<0.15900.6253>,<0.17133.6253>,<0.22945.6253>,<0.8889.6253>,<0.20333.6253>,<0.12924.6253>,<0.32204.6252>,<0.32140.6253>,<0.5556.6253>],nil,{{<0.21653.1946>,#Ref<0.0.6636.251466>},{<<"9lcen85eu5ztrg8cs287sacoy">>,[{2,<<58,122,97,121,57,131,34,12,88,18,7,165,20,247,0,70>>}],[]}},nil,[{docs_read,436},{docs_written,57},{missing_checked,500},{missing_found,500}],nil,nil,{batch,[<<"{\"_id\":\"1802v33hahs2ql9951vitjbih\",\"_rev\":\"5-138167bbafe87da23205b457c3918bd4\
",...],...}}

[error] <0.16617.1946> -------- CRASH REPORT Process (<0.16617.1946>)
with 15 neighbors exited with reason: changes_reader_died at
gen_server:terminate/6(line:737) <= proc_lib:init_p_do_apply/3(line:237);
initial_call: {couch_replicator_worker,init,['Argument__1']}, ancestors:
[<0.13952.1946>,couch_replicator_job_sup,couch_replicator_sup,...],
messages: [], links:
[<0.21799.6253>,<0.24330.6253>,<0.27365.6253>,<0.28980.6253>,...],
dictionary:
[{ibrowse_stream_status,init},{last_stats_report,{1501,381757,...}}],
trap_exit: true, status: running, heap_size: 4185, stack_size: 27,
reductions: 6921530028

[error] <0.16155.1946> -------- CRASH REPORT Process (<0.16155.1946>)
with 15 neighbors exited with reason: changes_reader_died at
gen_server:terminate/6(line:737) <= proc_lib:init_p_do_apply/3(line:237);
initial_call: {couch_replicator_worker,init,['Argument__1']}, ancestors:
[<0.13952.1946>,couch_replicator_job_sup,couch_replicator_sup,...],
messages: [], links:
[<0.12924.6253>,<0.21982.6253>,<0.27692.6253>,<0.30450.6253>,...],
dictionary:
[{ibrowse_stream_status,init},{last_stats_report,{1501,381751,...}}],
trap_exit: true, status: running, heap_size: 6772, stack_size: 27,
reductions: 6914251593

[error] <0.18469.1946> -------- gen_server <0.18469.1946> terminated with
reason: changes_reader_died
last msg: {'EXIT',<0.13952.1946>,changes_reader_died}
state: {state,<0.13952.1946>,<0.21935.1946>,20,{httpdb,"
https://admin:******@source.ip:443/my_db/",nil,[{"Accept","application/json"},{"User-Agent","CouchDB-Replicator/2.0.0"}],30000,[{is_ssl,true},{socket_options,[{keepalive,true},{nodelay,false}]},{ssl_options,[{depth,3},{verify,0}]}],10,250,<0.1722.1946>,20},{httpdb,"http://admin:******@127.0.0.1:5984/my_db/",nil,[{"Accept","application/json"},{"User-Agent","CouchDB-Replicator/2.0.0"}],30000,[{socket_options,[{keepalive,true},{nodelay,false}]}],10,250,<0.21751.1946>,20},[<0.12414.6254>,<0.24987.6253>,<0.24685.6253>,<0.30475.6253>,<0.25412.6253>,<0.18441.6253>,<0.23627.6253>,<0.10529.6252>,<0.18425.6253>,<0.18994.6250>,<0.2214.6253>,<0.9544.6253>,<0.13738.6253>,<0.32462.6253>,<0.15946.6253>,<0.25304.6253>,<0.16690.6253>,<0.3468.6253>,<0.17659.6253>,<0.13537.6253>],nil,{{<0.21935.1946>,#Ref<0.0.6636.248486>},{<<"935cd70fbe4e02cb374180daa08a9fff">>,[{2,<<194,192,159,166,5,0,237,72,59,193,108,4,228,198,91,95>>}],[]}},nil,[{missing_checked,500},{missing_found,500}],nil,nil,{batch,[],0
}}

[error] <0.10009.1946> -------- gen_server <0.10009.1946> terminated with
reason: changes_reader_died
last msg: {'EXIT',<0.13952.1946>,changes_reader_died}
state: {state,<0.13952.1946>,<0.22823.1946>,20,{httpdb,"
https://admin:******@source.ip:443/my_db/",nil,[{"Accept","application/json"},{"User-Agent","CouchDB-Replicator/2.0.0"}],30000,[{is_ssl,true},{socket_options,[{keepalive,true},{nodelay,false}]},{ssl_options,[{depth,3},{verify,0}]}],10,250,<0.1722.1946>,20},{httpdb,"http://admin:******@source.ip:5984/my_db/",nil,[{"Accept","application/json"},{"User-Agent","CouchDB-Replicator/2.0.0"}],30000,[{socket_options,[{keepalive,true},{nodelay,false}]}],10,250,<0.21751.1946>,20},[<0.19696.6253>,<0.2504.6253>,<0.13314.6253>,<0.31996.6253>,<0.18686.6253>,<0.20563.6253>,<0.23938.6253>,<0.8048.6253>,<0.18021.6253>,<0.31079.6253>,<0.19418.6253>,<0.18053.6253>,<0.22230.6253>,<0.15505.6253>,<0.14240.6253>,<0.10981.6253>,<0.29957.6253>,<0.13571.6253>,<0.17024.6253>,<0.29113.6253>],nil,{{<0.22823.1946>,#Ref<0.0.6636.249314>},{<<"31wv3qb6ipiql29u3t1nbmh4p">>,[{2,<<129,194,93,245,75,131,59,18,8,235,62,169,28,55,201,79>>}],[]}},nil,[{docs_read,140},{docs_written,18},{missing_checked,500},{missing_found,500}],nil,nil,{batch,[<<"{\"_id\":\"20ab5179602db3c31abb60507b40ebe9\",\"_rev\":\"2-d774bbcec2927a9c2de7f24a21486765\
<https://admin:******@source.ip:443/maxi_billing/",nil,[{"Accept","application/json"},{"User-Agent","CouchDB-Replicator/2.0.0"}],30000,[{is_ssl,true},{socket_options,[{keepalive,true},{nodelay,false}]},{ssl_options,[{depth,3},{verify,0}]}],10,250,<0.1722.1946>,20},{httpdb,"http://admin:******@source.ip:5984/my_db/",nil,[{"Accept","application/json"},{"User-Agent","CouchDB-Replicator/2.0.0"}],30000,[{socket_options,[{keepalive,true},{nodelay,false}]}],10,250,<0.21751.1946>,20},[<0.19696.6253>,<0.2504.6253>,<0.13314.6253>,<0.31996.6253>,<0.18686.6253>,<0.20563.6253>,<0.23938.6253>,<0.8048.6253>,<0.18021.6253>,<0.31079.6253>,<0.19418.6253>,<0.18053.6253>,<0.22230.6253>,<0.15505.6253>,<0.14240.6253>,<0.10981.6253>,<0.29957.6253>,<0.13571.6253>,<0.17024.6253>,<0.29113.6253>],nil,{{<0.22823.1946>,#Ref<0.0.6636.249314>},{<<"31wv3qb6ipiql29u3t1nbmh4p">>,[{2,<<129,194,93,245,75,131,59,18,8,235,62,169,28,55,201,79>>}],[]}},nil,[{docs_read,140},{docs_written,18},{missing_checked,500},{missing_found,500}],nil,nil,{batch,[<<"{\"_id\":\"20ab5179602db3c31abb60507b40ebe9\",\"_rev\":\"2-d774bbcec2927a9c2de7f24a21486765\>
",,...],...}}

[error] <0.18469.1946> -------- CRASH REPORT Process (<0.18469.1946>) with
15 neighbors exited with reason: changes_reader_died at
gen_server:terminate/6(line:737) <= proc_lib:init_p_do_apply/3(line:237);
initial_call: {couch_replicator_worker,init,['Argument__1']}, ancestors:
[<0.13952.1946>,couch_replicator_job_sup,couch_replicator_sup,...],
messages: [], links:
[<0.18441.6253>,<0.25304.6253>,<0.30475.6253>,<0.32462.6253>,...],
dictionary:
[{ibrowse_stream_status,init},{last_stats_report,{1501,381762,...}}],
trap_exit: true, status: running, heap_size: 6772, stack_size: 27,
reductions: 6918077761

[error] <0.10009.1946> -------- CRASH REPORT Process (<0.10009.1946>)
with 15 neighbors exited with reason: changes_reader_died at
gen_server:terminate/6(line:737) <= proc_lib:init_p_do_apply/3(line:237);
initial_call: {couch_replicator_worker,init,['Argument__1']}, ancestors:
[<0.13952.1946>,couch_replicator_job_sup,couch_replicator_sup,...],
messages: [], links:
[<0.18053.6253>,<0.23938.6253>,<0.29957.6253>,<0.31079.6253>,...],
dictionary:
[{ibrowse_stream_status,init},{last_stats_report,{1501,381758,...}}],
trap_exit: true, status: running, heap_size: 6772, stack_size: 27,
reductions: 6919159795
```

Cause: This one looks similar to the one before. The trigger seems to be
that the source closes the connection for some reason, but the logs are
much more verbose this time.
Consequences: None. I think replicator just retries the task.

Furthermore, this last logs show the username and passwords plain text on
the state key.

Sorry for the wall of text.

Regards

--
[image: Cabify - Your private Driver] <http://www.cabify.com/>

*Carlos Alonso*
Data Engineer
Madrid, Spain

carlos...@cabify.com

Prueba gratis con este código
#CARLOSA6319 <https://cabify.com/i/carlosa6319>
[image: Facebook] <http://cbify.com/fb_ES>[image: Twitter]
<http://cbify.com/tw_ES>[image: Instagram] <http://cbify.com/in_ES>[image:
Linkedin] <https://www.linkedin.com/in/mrcalonso>

--
Este mensaje y cualquier archivo adjunto va dirigido exclusivamente a su
destinatario, pudiendo contener información confidencial sometida a secreto
profesional. No está permitida su reproducción o distribución sin la
autorización expresa de Cabify. Si usted no es el destinatario final por
favor elimínelo e infórmenos por esta vía.

This message and any attached file are intended exclusively for the
addressee, and it may be confidential. You are not allowed to copy or
disclose it without Cabify's prior written authorization. If you are not
the intended recipient please delete it from your system and notify us by
e-mail.

[Carlos Alonso]

On top of those, this one happens from time to time as well:

[error] 2017-07-31T13:41:50.258423Z couchdb@couch-2 emulator -------- Error
in process <0.16620.2434> on node 'couchdb@couch-2' with exit value:
{{nocatch,{mp_parser_died,noproc}},[{couch_att,'-foldl/4-fun-0-',3,[{file,"src/couch_att.erl"},{line,591}]},{couch_att,fold_streamed_data,4,[{file,"src/couch_att.erl"},{line,642}]},{couch_att,foldl,4,[{file,"src/couch_att.erl"},...

[error] 2017-07-31T13:41:50.260569Z couchdb@couch-2 <0.32053.2433> --------
Replicator, request PUT to "http://admin:*****@
127.0.0.1:5984/my_db/cba0aa16e347ac7d6b09732032828704?new_edits=false"
failed due to error {error,
{'EXIT',
{{{nocatch,{mp_parser_died,noproc}},
[{couch_att,'-foldl/4-fun-0-',3,
[{file,"src/couch_att.erl"},{line,591}]},
{couch_att,fold_streamed_data,4,
[{file,"src/couch_att.erl"},{line,642}]},
{couch_att,foldl,4,[{file,"src/couch_att.erl"},{line,595}]},
{couch_httpd_multipart,atts_to_mp,4,
[{file,"src/couch_httpd_multipart.erl"},{line,208}]}]},
{gen_server,call,
[<0.11924.2435>,
{send_req,
{{url,
"
http://admin:pass...@127.0.0.1:5984/my_db/cba0aa16e347ac7d6b09732032828704?new_edits=false
",
"127.0.0.1",5984,"admin","password",

"/my_db/cba0aa16e347ac7d6b09732032828704?new_edits=false",
http,ipv4_address},
[{"Accept","application/json"},
{"Content-Length",32845},
{"Content-Type",
"multipart/related;
boundary=\"46b37c9ba185ac67bf24ee40bcc38828\""},
{"User-Agent","CouchDB-Replicator/2.0.0"}],
put,
{#Fun<couch_replicator_api_wrap.11.79455979>,

{<<"{\"_id\":\"cba0aa16e347ac7d6b09732032828704\",\"_rev\":\"8-b90e6bd0b6281f303e587f4b9538a0af\",...\"_revisions\":{\"start\":8,\"ids\":[\"b90e6bd0b6281f303e587f4b9538a0af\",\"df66a380ea1258243bfd8a33506a18b4\",\"4cd932d1b71cebc4a55f91331d97f902\",\"a4dabd1dbf2b1de8987119a65cb44131\",\"82cc46c88ca768ff092de95c338bc0e9\",\"b4a7562ac8a831f46a194e97ea33e527\",\"d4fd4ddb998e4022e0335f609ed1ef50\",\"118cae6fc6864b3c48f788e78705c8cb\"]},\"_attachments\":{\"FF01-0020941.pdf\":{\"content_type\":\"application/pdf\",\"revpos\":8,\"digest\":\"md5-IT8Ms2HDG9EFLvfIxYTaVw==\",\"length\":21888,\"follows\":true},\"FF01-0020941.xml\":{\"content_type\":\"application/xml\",\"revpos\":8,\"digest\":\"md5-KJIXHDQbcJ3xeLaOUh11cw==\",\"length\":8526,\"follows\":true}}}">>,
[{att,<<"FF01-0020941.pdf">>,<<"application/pdf">>,
21888,21888,
<<33,63,12,179,97,195,27,209,5,46,247,200,197,132,
218,87>>,
8,
{follows,<0.31188.2433>,#Ref<0.0.12124.197441>},
identity},

{att,<<"FF01-0020941.xml">>,<<"application/xml">>,8526,
8526,

<<40,146,23,28,52,27,112,157,241,120,182,142,82,29,
117,115>>,
8,
{follows,<0.31188.2433>,#Ref<0.0.12124.197441>},
identity}],
<<"46b37c9ba185ac67bf24ee40bcc38828">>,32845}},
[{response_format,binary},
{inactivity_timeout,30000},

{socket_options,[{keepalive,true},{nodelay,false}]}],

infinity}},
infinity]}}}}


Thanks!



On Mon, Jul 31, 2017 at 11:52 AM Carlos Alonso <carlos...@cabify.com>
wrote:

> Hi all!!
>
> I've been observing logs for a while and trying to understand why a few
> errors happen while (pull) replication is happening and also trying to
> understand the consequences.
>
> ** connection_closed*
>
> Log:
> ```
> [error] <0.15968.1946> -------- Replicator, request GET to "https://admin:*****@source.ip:443/my_db/_changes?feed=continuous&style=all_docs&since=%2210613426-g1AAAAOCeJyd0r1OwzAQB3CrRYIHYC8FCSZS2_kwEUM7svURiu1zVKqQSlUZYKFP0FeAV2DsBE_BCm-Suj4UKqRYIctZsv66n053OSHkaNoFcgpKzxdmBIrRQEt1lz1SygKdzx9AFsugMMvcZjuSqF5ZlrNpV5Hj8-t7-3eYZJyDzoD0qyZhTY8_lPBR6sRWNay0_pvTmJA8ScS-xhppPPJqo502qbSbwGlUUwNXrMVsiVe73WnPlXbx5DSQXBihW8xWF3NacWArWdnHgi8oXn7ifGHEeST_L_rPBMVXFN9RHORO5FowIeKm4m_KeywIfiD4heD6zIFpbCAL4xYrrIvtid8o_qxxvHGiiVKTQqM1zrad2hfx%22&timeout=10000"
> failed due to error connection_closed
> ```
>
> Cause: This one happens from time to time. It looks like the source is
> closing connection, but why?
> Consequences: None. I think replicator just retries the task.
>
> * *changes_reader_died*
>
> Log:
> ```
> [error] <0.15968.1946> -------- Replicator, request GET to "https://admin:*****@source.ip:443/my_db/_changes?feed=continuous&style=all_docs&since=%2235690238-g1...&timeout=10000"
> failed due to error connection_closed
>
> [error] <0.13952.1946> -------- ChangesReader process died with reason:
> {http_request_failed,"GET","
> https://admin:*****@source.ip:443/my_db/_changes?feed=continuous&style=all_docs&since=%2235690238-g1...&timeout=10000",{error,connection_closed

> <https://admin:*****@source.ip:443/my_db/_changes?feed=continuous&style=all_docs&since=%2235690238-g1...&timeout=10000%22,%7Berror,connection_closed>

> }}
>
> [error] <0.13952.1946> -------- Replication
> `363ed31e35361d024d22e0a990ca1ae5+continuous` (`https://admin:*****@source.ip:443/my_db/`
> -> `http://admin:*****@127.0.0.1:5984/my_db/`

> <http://127.0.0.1:5984/my_db/>) failed: changes_reader_died

[Joan Touzet]

Hi Carlos,

This issue (the {{nocatch,{mp_parser_died,noproc}}) is recently
identified and patched!

https://github.com/apache/couchdb/issues/574

The other error just looks like a disconnection. You may want to
wait (just a couple more days!) for 2.1.0 which has all sorts of
new replication features and try again.

-Joan

[Carlos Alonso]

Hi Joan, many thanks for your reply

Why would the timeout and closed connections happen? I see it quite often
on the logs TBH but also, a slightly different one is a timeout against the
cluster itself. What may this mean?

[error] 2017-08-02T12:38:27.321697Z couchdb@couch-2 <0.7114.568> --------

Replicator, request PUT to "http://admin:*****@

127.0.0.1:5984/my_db/35e78fe9d421677c77dea34522de2b99?new_edits=false"
failed due to error {error,{conn_failed,{error,etimedout}}}

Regards

[Joan Touzet]

I haven't looked closely, but it could just be a side effect of
#574. I recommend you upgrade to 2.1.0 (when it releases, very
very very soon now) and see if the problem goes away first.

> -> `http://admin:*****@ 127.0.0.1:5984/my_db/`

> < http://127.0.0.1:5984/my_db/ >) failed: changes_reader_died
>
> [error] <0.30404.6252> -------- Replicator, request GET to "https://admin:*****@source.ip:443/my_db/b666ee7ec8ed5b82ae035fe5fa6a4a2c?revs=true&open_revs=%5B%221-5a7ba8a8658bd0ab7b3656aa089aa183%22%5D&latest=true"
> failed due to error {error,closing_on_request}
>
> [error] <0.352.0> -------- Error in replication
> `363ed31e35361d024d22e0a990ca1ae5+continuous` (triggered by document
> `28d973af08931f79fa423b54a8004b31`): changes_reader_died
> Restarting replication in 5 seconds.
>
> [error] <0.13952.1946> -------- gen_server <0.13952.1946> terminated with
> reason: changes_reader_died
> last msg:
> {'EXIT',<0.15968.1946>,{http_request_failed,"GET"," https://admin :
> *****@source.ip:443/my_db/_changes?feed=continuous&style=all_docs&since=%2235690238-g1...&timeout=10000",{error,connection_closed}}}
> state:
> [{data,[{"State",{rep_state,{rep,{"363ed31e35361d024d22e0a990ca1ae5","+continuous"},{httpdb," https://admin :
> *****@source.ip:443/my_db/",nil,[{"Accept","application/json"},{"User-Agent","CouchDB-Replicator/2.0.0"}],30000,[{is_ssl,true},{socket_options,[{keepalive,true},{nodelay,false}]},{ssl_options,[{depth,3},{verify,0}]}],10,250,nil,20},{httpdb," http://admin :

> *****@ 127.0.0.1:5984/my_db/

> ",nil,[{"Accept","application/json"},{"User-Agent","CouchDB-Replicator/2.0.0"}],30000,[{socket_options,[{keepalive,true},{nodelay,false}]}],10,250,nil,20},[{checkpoint_interval,30000},{connection_timeout,30000},{continuous,true},{http_connections,20},{retries,10},{socket_options,[{keepalive,true},{nodelay,false}]},{use_checkpoints,true},{worker_batch_size,500},{worker_processes,4}],{user_ctx,null,[],undefined},db,nil,<<"28d973af08931f79fa423b54a8004b31">>,<<"shards/c0000000-dfffffff/_replicator.1501167373">>}," https://admin :

> *****@source.ip:443/my_db/","http://admin:*****@ 127.0.0.1:5984/my_db/

> ",{httpdb," https://admin :
> *****@source.ip:443/my_db/",nil,[{"Accept","application/json"},{"User-Agent","CouchDB-Replicator/2.0.0"}],30000,[{is_ssl,true},{socket_options,[{keepalive,true},{nodelay,false}]},{ssl_options,[{depth,3},{verify,0}]}],10,250,<0.1722.1946>,20},{httpdb," http://admin :

> *****@ 127.0.0.1:5984/my_db/ ",nil,[{"Accept","application/json"},{"User-Agent","CouchDB-Replicator/2.0.0"}],30000,[{socket_options,[{keepalive,true},{nodelay,false}]}],10,250,<0.21751.1946>,20},[],{[{<<"session_id">>,<<"6f7aa01e879f5e1841cdb7100f40806b">>},{<<"source_last_seq">>,<<"36567936-g1...">>},{<<"replication_id_version">>,3},{<<"history">>,[{[{<<"session_id">>,<<"6f7aa01e879f5e1841cdb7100f40806b">>},{<<"start_time">>,<<"Fri,

> 28 Jul 2017 09:46:48 GMT">>},{<<"end_time">>,<<"Sun, 30 Jul 2017 02:28:59
> GMT">>},{<<"start_last_seq">>,0},{<<"end_last_seq">>,<<"36567936-g1...">>},{<<"recorded_seq">>,<<"36567936-g1...">>},{<<"missing_checked">>,22991404},{<<"missing_found">>,22991404},{<<"docs_read">>,22991404},{<<"docs_written">>,22991404},{<<"doc_write_failures">>,0}]}]}]},{0,0},{46381,<<"36567936-g1...">>},{46388,<<"36574076-g1...">>},[{46389,<<"36575175-g1...">>},{46390,<<"36575938-g1...">>},{46391,<<"36576695-...">>},{46392,<<"36577391-g1...">>}],{46388,<<"36574076-g1...">>},{doc,<<"_local/363ed31e35361d024d22e0a990ca1ae5">>,{0,[<<"4647">>]},{[]},[],false,[]},{doc,<<"_local/363ed31e35361d024d22e0a990ca1ae5">>,{0,[<<"4647">>]},{[]},[],false,[]},"Fri,
> 28 Jul 2017 09:46:48
> GMT",<<"0">>,<<"0">>,nil,<0.16590.1946>,<0.16953.1946>,<0.15968.1946>,[<0.18469.1946>,<0.10009.1946>,<0.16617.1946>,<0.16155.1946>],[{doc_write_failures,0},{docs_read,22994904},{docs_written,22994904},{missing_checked,22994904},{missing_found,22994904}],<<"6f7aa01e879f5e1841cdb7100f40806b">>,nil,nil,nil,nil,<<"255587270-g1...">>,true,30000,db,nil}}]}]
>
> [error] <0.13952.1946> -------- CRASH REPORT Process (<0.13952.1946>)
> with 2 neighbors exited with reason: changes_reader_died at
> gen_server:terminate/6(line:737) <= proc_lib:init_p_do_apply/3(line:237);
> initial_call: {couch_replicator,init,['Argument__1']}, ancestors:
> [couch_replicator_job_sup,couch_replicator_sup,<0.349.0>], messages: [],
> links: [<0.16590.1946>,<0.16953.1946>,<0.18469.1946>,<0.16617.1946>,...],
> dictionary:
> [{task_status_props,[{changes_pending,127404519},{checkpoint_interval,...},...]},...],
> trap_exit: true, status: running, heap_size: 4185, stack_size: 27,
> reductions: 26502159
>
> [error] <0.364.0> -------- Supervisor couch_replicator_job_sup had child
> "363ed31e35361d024d22e0a990ca1ae5+continuous" started with
> {gen_server,start_link,undefined} at <0.13952.1946> exit with reason
> changes_reader_died in context child_terminated
>
> [error] <0.16617.1946> -------- gen_server <0.16617.1946> terminated with
> reason: changes_reader_died
> last msg: {'EXIT',<0.13952.1946>,changes_reader_died}
> state: {state,<0.13952.1946>,<0.15004.1946>,20,{httpdb,"

> https://admin:******@source.ip:443/my_db/",nil,[{"Accept","application/json"},{"User-Agent","CouchDB-Replicator/2.0.0"}],30000,[{is_ssl,true},{socket_options,[{keepalive,true},{nodelay,false}]},{ssl_options,[{depth,3},{verify,0}]}],10,250,<0.1722.1946>,20},{httpdb,"http://admin:******@ 127.0.0.1:5984/my_db/ ",nil,[{"Accept","application/json"},{"User-Agent","CouchDB-Replicator/2.0.0"}],30000,[{socket_options,[{keepalive,true},{nodelay,false}]}],10,250,<0.21751.1946>,20},[<0.2293.6254>,<0.19041.6253>,<0.19785.6253>,<0.7470.6253>,<0.23117.6253>,<0.31026.6250>,<0.11251.6253>,<0.27821.6253>,<0.8477.6253>,<0.28980.6253>,<0.22848.6253>,<0.16549.6251>,<0.26208.6253>,<0.27365.6253>,<0.15830.6253>,<0.24330.6253>,<0.20562.6253>,<0.20869.6253>,<0.21799.6253>,<0.16311.6253>],nil,{{<0.15004.1946>,#Ref<0.0.6636.248998>},{<<"07597cb4d450c2b6ff674b675b9d5168">>,[{1,<<93,162,57,7,69,142,28,226,77,41,199,156,216,131,208,152>>}],[]}},nil,[{docs_read,199},{docs_written,25},{missing_checked,500},{missing_found,500}],nil,nil,{batch,[<<"{\"_id\":\"b65rnw8pcm0jbt7jdpxvgmo1r\",\"_rev\":\"2-4d5283c4d125a0da73b95cc80b92abd6\

> ",...}}
>
> [error] <0.16155.1946> -------- gen_server <0.16155.1946> terminated with
> reason: changes_reader_died
> last msg: {'EXIT',<0.13952.1946>,changes_reader_died}
> state: {state,<0.13952.1946>,<0.21653.1946>,20,{httpdb,"

> https://admin:******@source.ip:443/my_db/",nil,[{"Accept","application/json"},{"User-Agent","CouchDB-Replicator/2.0.0"}],30000,[{is_ssl,true},{socket_options,[{keepalive,true},{nodelay,false}]},{ssl_options,[{depth,3},{verify,0}]}],10,250,<0.1722.1946>,20},{httpdb,"http://admin:******@ 127.0.0.1:5984/my_db/ ",nil,[{"Accept","application/json"},{"User-Agent","CouchDB-Replicator/2.0.0"}],30000,[{socket_options,[{keepalive,true},{nodelay,false}]}],10,250,<0.21751.1946>,20},[<0.27802.6253>,<0.27692.6253>,<0.10519.6253>,<0.5584.6253>,<0.9485.6253>,<0.2597.6253>,<0.12901.6253>,<0.21982.6253>,<0.20758.6253>,<0.12367.6253>,<0.30450.6253>,<0.15900.6253>,<0.17133.6253>,<0.22945.6253>,<0.8889.6253>,<0.20333.6253>,<0.12924.6253>,<0.32204.6252>,<0.32140.6253>,<0.5556.6253>],nil,{{<0.21653.1946>,#Ref<0.0.6636.251466>},{<<"9lcen85eu5ztrg8cs287sacoy">>,[{2,<<58,122,97,121,57,131,34,12,88,18,7,165,20,247,0,70>>}],[]}},nil,[{docs_read,436},{docs_written,57},{missing_checked,500},{missing_found,500}],nil,nil,{batch,[<<"{\"_id\":\"1802v33hahs2ql9951vitjbih\",\"_rev\":\"5-138167bbafe87da23205b457c3918bd4\

> https://admin:******@source.ip:443/my_db/",nil,[{"Accept","application/json"},{"User-Agent","CouchDB-Replicator/2.0.0"}],30000,[{is_ssl,true},{socket_options,[{keepalive,true},{nodelay,false}]},{ssl_options,[{depth,3},{verify,0}]}],10,250,<0.1722.1946>,20},{httpdb,"http://admin:******@ 127.0.0.1:5984/my_db/ ",nil,[{"Accept","application/json"},{"User-Agent","CouchDB-Replicator/2.0.0"}],30000,[{socket_options,[{keepalive,true},{nodelay,false}]}],10,250,<0.21751.1946>,20},[<0.12414.6254>,<0.24987.6253>,<0.24685.6253>,<0.30475.6253>,<0.25412.6253>,<0.18441.6253>,<0.23627.6253>,<0.10529.6252>,<0.18425.6253>,<0.18994.6250>,<0.2214.6253>,<0.9544.6253>,<0.13738.6253>,<0.32462.6253>,<0.15946.6253>,<0.25304.6253>,<0.16690.6253>,<0.3468.6253>,<0.17659.6253>,<0.13537.6253>],nil,{{<0.21935.1946>,#Ref<0.0.6636.248486>},{<<"935cd70fbe4e02cb374180daa08a9fff">>,[{2,<<194,192,159,166,5,0,237,72,59,193,108,4,228,198,91,95>>}],[]}},nil,[{missing_checked,500},{missing_found,500}],nil,nil,{batch,[],0

Carlos Alonso

Data Engineer
Madrid, Spain

carlos...@cabify.com



Prueba gratis con este código
#CARLOSA6319

FacebookTwitterInstagramLinkedin

[Carlos Alonso]

Cool, thanks Joan!!

Is there any upgrade procedure available to get ready for it?

Regards