Help understanding crash log
189 views
Skip to first unread message
Herman Chan
Apr 30, 2014, 11:47:07 PM4/30/14
to us...@couchdb.apache.org
hi all,
Our couchdb server (1.2) just died and had to be restarted, can someone explain to me the crash log below?
Thu, 01 May 2014 03:30:02 GMT] [error] [<0.4723.2131>] {error_report,<0.31.0>,
{<0.4723.2131>,crash_report,
[[{initial_call,
{couch_index_server,init,['Argument__1']}},
{pid,<0.4723.2131>},
{registered_name,[]},
{error_info,
{exit,
{timeout,
{gen_server,call,
[couch_config,
{register,
#Fun<couch_index_server.config_change.2>,
<0.4723.2131>}]}},
[{gen_server,init_it,6},
{proc_lib,init_p_do_apply,3}]}},
{ancestors,
[couch_secondary_services,couch_server_sup,
<0.32.0>]},
{messages,[]},
{links,[<0.27000.2602>]},
{dictionary,[]},
{trap_exit,true},
{status,running},
{heap_size,377},
{stack_size,24},
{reductions,144}],
[]]}}
[Thu, 01 May 2014 03:30:02 GMT] [error] [<0.27000.2602>] {error_report,<0.31.0>,
{<0.27000.2602>,supervisor_report,
[{supervisor,{local,couch_secondary_services}},
{errorContext,start_error},
{reason,
{timeout,
{gen_server,call,
[couch_config,
{register,
#Fun<couch_index_server.config_change.2>,
<0.4723.2131>}]}}},
{offender,
[{pid,<0.27013.2602>},
{name,index_server},
{mfargs,{couch_index_server,start_link,[]}},
{restart_type,permanent},
{shutdown,brutal_kill},
{child_type,worker}]}]}}
[Thu, 01 May 2014 03:30:02 GMT] [error] [<0.27000.2602>] {error_report,<0.31.0>,
{<0.27000.2602>,supervisor_report,
[{supervisor,{local,couch_secondary_services}},
{errorContext,start_error},
{reason,
{timeout,
{gen_server,call,
[couch_config,
{register,
#Fun<couch_index_server.config_change.2>,
<0.4723.2131>}]}}},
{offender,
[{pid,<0.27013.2602>},
{name,index_server},
{mfargs,{couch_index_server,start_link,[]}},
{restart_type,permanent},
{shutdown,brutal_kill},
{child_type,worker}]}]}}
[Thu, 01 May 2014 03:30:02 GMT] [error] [<0.27000.2602>] {error_report,<0.31.0>,
{<0.27000.2602>,supervisor_report,
[{supervisor,{local,couch_secondary_services}},
{errorContext,shutdown},
{reason,reached_max_restart_intensity},
{offender,
[{pid,<0.27013.2602>},
{name,index_server},
{mfargs,{couch_index_server,start_link,[]}},
{restart_type,permanent},
{shutdown,brutal_kill},
{child_type,worker}]}]}}
[Thu, 01 May 2014 03:30:02 GMT] [error] [<0.83.0>] {error_report,<0.31.0>,
{<0.83.0>,supervisor_report,
[{supervisor,{local,couch_server_sup}},
{errorContext,child_terminated},
{reason,shutdown},
{offender,
[{pid,<0.27000.2602>},
{name,couch_secondary_services},
{mfargs,{couch_secondary_sup,start_link,[]}},
{restart_type,permanent},
{shutdown,infinity},
{child_type,supervisor}]}]}}
Our couchdb server (1.2) just died and had to be restarted, can someone explain to me the crash log below?
Thu, 01 May 2014 03:30:02 GMT] [error] [<0.4723.2131>] {error_report,<0.31.0>,
{<0.4723.2131>,crash_report,
[[{initial_call,
{couch_index_server,init,['Argument__1']}},
{pid,<0.4723.2131>},
{registered_name,[]},
{error_info,
{exit,
{timeout,
{gen_server,call,
[couch_config,
{register,
#Fun<couch_index_server.config_change.2>,
<0.4723.2131>}]}},
[{gen_server,init_it,6},
{proc_lib,init_p_do_apply,3}]}},
{ancestors,
[couch_secondary_services,couch_server_sup,
<0.32.0>]},
{messages,[]},
{links,[<0.27000.2602>]},
{dictionary,[]},
{trap_exit,true},
{status,running},
{heap_size,377},
{stack_size,24},
{reductions,144}],
[]]}}
[Thu, 01 May 2014 03:30:02 GMT] [error] [<0.27000.2602>] {error_report,<0.31.0>,
{<0.27000.2602>,supervisor_report,
[{supervisor,{local,couch_secondary_services}},
{errorContext,start_error},
{reason,
{timeout,
{gen_server,call,
[couch_config,
{register,
#Fun<couch_index_server.config_change.2>,
<0.4723.2131>}]}}},
{offender,
[{pid,<0.27013.2602>},
{name,index_server},
{mfargs,{couch_index_server,start_link,[]}},
{restart_type,permanent},
{shutdown,brutal_kill},
{child_type,worker}]}]}}
[Thu, 01 May 2014 03:30:02 GMT] [error] [<0.27000.2602>] {error_report,<0.31.0>,
{<0.27000.2602>,supervisor_report,
[{supervisor,{local,couch_secondary_services}},
{errorContext,start_error},
{reason,
{timeout,
{gen_server,call,
[couch_config,
{register,
#Fun<couch_index_server.config_change.2>,
<0.4723.2131>}]}}},
{offender,
[{pid,<0.27013.2602>},
{name,index_server},
{mfargs,{couch_index_server,start_link,[]}},
{restart_type,permanent},
{shutdown,brutal_kill},
{child_type,worker}]}]}}
[Thu, 01 May 2014 03:30:02 GMT] [error] [<0.27000.2602>] {error_report,<0.31.0>,
{<0.27000.2602>,supervisor_report,
[{supervisor,{local,couch_secondary_services}},
{errorContext,shutdown},
{reason,reached_max_restart_intensity},
{offender,
[{pid,<0.27013.2602>},
{name,index_server},
{mfargs,{couch_index_server,start_link,[]}},
{restart_type,permanent},
{shutdown,brutal_kill},
{child_type,worker}]}]}}
[Thu, 01 May 2014 03:30:02 GMT] [error] [<0.83.0>] {error_report,<0.31.0>,
{<0.83.0>,supervisor_report,
[{supervisor,{local,couch_server_sup}},
{errorContext,child_terminated},
{reason,shutdown},
{offender,
[{pid,<0.27000.2602>},
{name,couch_secondary_services},
{mfargs,{couch_secondary_sup,start_link,[]}},
{restart_type,permanent},
{shutdown,infinity},
{child_type,supervisor}]}]}}
Adam Kocoloski
May 1, 2014, 12:11:18 AM5/1/14
to us...@couchdb.apache.org
The process that manages the .ini configuration information inside the server was taking longer than 5 seconds to respond to requests. This caused other processes to crash. Eventually the death rate for processes in the supervision tree grew large enough that the VM decided to nuke the entire Erlang application.
Of course, this raises the question -- why was couch_config timing out on register requests? There's nothing in that log which can help us figure that out.
Adam
Of course, this raises the question -- why was couch_config timing out on register requests? There's nothing in that log which can help us figure that out.
Adam
Herman Chan
May 1, 2014, 7:14:27 AM5/1/14
to us...@couchdb.apache.org
Thanks Adam,
It seems like it is happening again, with more info this time. It looks like I am hitting some sort of system limit, can anyone point out where to look next?
[Thu, 01 May 2014 08:32:57 GMT] [error] [<0.4862.1232>] {error_report,<0.31.0>,
{<0.4862.1232>,crash_report,
[[{initial_call,
{mochiweb_socket_server,init,['Argument__1']}},
{pid,<0.4862.1232>},
{registered_name,couch_httpd},
{error_info,
{exit,
{system_limit,
[{erlang,spawn_link,
[proc_lib,init_p,
[couch_httpd,
[couch_secondary_services,couch_server_sup,
<0.32.0>],
mochiweb_acceptor,init,
[<0.4862.1232>,#Port<0.1019424>,
#Fun<mochiweb_http.0.65037361>]]]},
{proc_lib,spawn_link,3},
{mochiweb_socket_server,recycle_acceptor,2},
{mochiweb_socket_server,handle_cast,2},
{gen_server,handle_msg,5},
{proc_lib,init_p_do_apply,3}]},
[{gen_server,terminate,6},
{'EXIT',<0.32257.1924>,normal},
{'EXIT',<0.10859.1927>,normal},
{'EXIT',<0.10108.1927>,normal},
{'EXIT',<0.14640.1927>,normal},
{'EXIT',<0.29939.1985>,normal},
{'EXIT',<0.7931.1984>,normal},
{'EXIT',<0.4449.1985>,normal},
{'EXIT',<0.25731.1985>,normal},
{'EXIT',<0.9721.1987>,normal},
{'EXIT',<0.10558.1987>,normal},
{'EXIT',<0.28101.1987>,normal},
{'EXIT',<0.31758.1987>,normal},
{'EXIT',<0.6565.2050>,normal},
{'EXIT',<0.9376.2050>,normal}]},
{links,
[<0.774.1349>,<0.5904.1475>,<0.7251.1600>,
<0.30241.1666>,<0.7102.1793>,<0.24040.1857>,
<0.3099.1923>,<0.685.1924>,<0.13290.1924>,
<0.16551.1924>,<0.18397.1924>,<0.15623.1924>,
<0.6609.1924>,<0.8631.1924>,<0.11356.1924>,
<0.7864.1924>,<0.2600.1924>,<0.1248.1924>,
<0.26762.1923>,<0.654.1924>,<0.14429.1923>,
<0.21428.1923>,<0.12438.1923>,<0.17675.1862>,
<0.27305.1863>,<0.6859.1922>,<0.21700.1922>,
<0.15868.1922>,<0.5406.1921>,<0.10183.1921>,
<0.4209.1863>,<0.22336.1863>,<0.34.1863>,
<0.10899.1858>,<0.28185.1858>,<0.25943.1861>,
<0.11942.1862>,<0.16801.1861>,<0.15068.1858>,
<0.28187.1857>,<0.9003.1858>,<0.9009.1858>,
<0.6787.1858>,<0.25901.1857>,<0.10955.1798>,
<0.7361.1856>,<0.3933.1857>,<0.21045.1857>,
<0.22194.1857>,<0.16436.1857>,<0.17729.1857>,
<0.18985.1856>,<0.25162.1856>,<0.30460.1856>,
<0.20116.1856>,<0.11971.1856>,<0.13515.1856>,
<0.11335.1856>,<0.4659.1799>,<0.11134.1804>,
<0.16714.1804>,<0.6153.1801>,<0.12446.1801>,
<0.22422.1799>,<0.21336.1798>,<0.31470.1798>,
<0.17373.1798>,<0.8167.1796>,<0.12866.1797>,
<0.1220.1798>,<0.10395.1798>,<0.16510.1797>,
<0.106.1798>,<0.2541.1797>,<0.4215.1797>,
<0.7722.1797>,<0.3045.1797>,<0.16605.1796>,
<0.27758.1796>,<0.11818.1794>,<0.25467.1794>,
<0.27660.1794>,<0.29591.1795>,<0.26899.1794>,
<0.21140.1794>,<0.24245.1794>,<0.13941.1794>,
<0.13009.1793>,<0.9970.1793>,<0.5728.1732>,
<0.31532.1734>,<0.20101.1742>,<0.15183.1792>,
<0.29006.1792>,<0.4300.1793>,<0.6785.1793>,
<0.31530.1792>,<0.19338.1792>,<0.25167.1792>,
<0.28653.1792>,<0.18982.1792>,<0.7255.1792>,
<0.7445.1792>,<0.13516.1792>,<0.14592.1792>,
<0.7706.1792>,<0.7312.1792>,<0.15718.1747>,
<0.137.1792>,<0.141.1792>,<0.12800.1757>,
<0.2679.1743>,<0.22873.1745>,<0.32384.1735>,
<0.8253.1737>,<0.28419.1737>,<0.11164.1737>,
<0.23579.1736>,<0.2043.1737>,<0.24505.1736>,
<0.7226.1736>,<0.1954.1736>,<0.15836.1735>,
<0.17749.1735>,<0.24425.1735>,<0.29928.1735>,
<0.22608.1735>,<0.16792.1735>,<0.8048.1735>,
<0.9565.1735>,<0.13482.1735>,<0.9162.1735>,
<0.4542.1735>,<0.7932.1735>,<0.80.1735>,
<0.27204.1733>,<0.12399.1734>,<0.19006.1734>,
<0.23319.1734>,<0.30827.1734>,<0.30826.1734>,
<0.19606.1734>,<0.23257.1734>,<0.16768.1734>,
<0.18675.1734>,<0.14808.1734>,<0.15843.1734>,
<0.13853.1734>,<0.102.1734>,<0.4349.1734>,
<0.11895.1734>,<0.3130.1734>,<0.2803.1734>,
<0.31505.1733>,<0.31736.1733>,<0.31729.1733>,
<0.27510.1733>,<0.27108.1732>,<0.21084.1733>,
<0.24389.1733>,<0.25944.1733>,<0.26050.1733>,
<0.24084.1733>,<0.22915.1733>,<0.8186.1733>,
<0.18381.1733>,<0.18382.1733>,<0.30292.1732>,
<0.28981.1732>,<0.15608.1732>,<0.19139.1732>,
<0.23884.1732>,<0.24099.1732>,<0.21688.1732>,
<0.17326.1732>,<0.17763.1732>,<0.7495.1732>,
<0.11357.1732>,<0.13257.1732>,<0.9109.1732>,
<0.5732.1732>,<0.31583.1670>,<0.19683.1673>,
<0.1485.1730>,<0.10926.1731>,<0.28468.1731>,
<0.3573.1732>,<0.5141.1732>,<0.14511.1731>,
<0.19499.1731>,<0.25843.1730>,<0.1102.1731>,
<0.31659.1730>,<0.18511.1730>,<0.10347.1730>,
<0.6727.1729>,<0.9902.1729>,<0.19448.1729>,
<0.7704.1729>,<0.7651.1729>,<0.12980.1728>,
<0.14949.1728>,<0.27554.1678>,<0.1867.1672>,
<0.10706.1673>,<0.10776.1673>,<0.17424.1673>,
<0.22263.1672>,<0.17651.1671>,<0.24624.1671>,
<0.1356.1672>,<0.17813.1671>,<0.14169.1671>,
<0.26739.1668>,<0.17849.1669>,<0.21292.1670>,
<0.31476.1670>,<0.22843.1670>,<0.20265.1669>,
<0.10089.1669>,<0.12590.1669>,<0.15345.1669>,
<0.12879.1669>,<0.11554.1669>,<0.1307.1669>,
<0.6734.1669>,<0.31213.1668>,<0.26881.1668>,
<0.17243.1667>,<0.4739.1668>,<0.14390.1668>,
<0.16608.1668>,<0.13975.1668>,<0.3916.1668>,
<0.4366.1668>,<0.29566.1667>,<0.656.1668>,
<0.6036.1667>,<0.7620.1667>,<0.15362.1667>,
<0.2154.1667>,<0.3685.1667>,<0.4986.1667>,
<0.3546.1667>,<0.587.1667>,<0.15187.1606>,
<0.10288.1664>,<0.20792.1665>,<0.5071.1666>,
<0.25743.1666>,<0.27388.1666>,<0.29174.1666>,
<0.26103.1666>,<0.26574.1666>,<0.21647.1666>,
<0.22370.1666>,<0.22262.1666>,<0.13801.1666>,
<0.19361.1666>,<0.25474.1665>,<0.29539.1665>,
<0.3562.1666>,<0.28624.1665>,<0.22727.1665>,
<0.24582.1665>,<0.24583.1665>,<0.23223.1665>,
<0.21197.1665>,<0.21693.1665>,<0.20885.1665>,
<0.4360.1665>,<0.6802.1665>,<0.15519.1665>,
<0.19258.1665>,<0.20552.1665>,<0.18502.1665>,
<0.14356.1665>,<0.6777.1665>,<0.6784.1665>,
<0.6505.1665>,<0.6509.1665>,<0.5320.1665>,
<0.25160.1664>,<0.30000.1664>,<0.31791.1664>,
<0.29782.1664>,<0.18265.1664>,<0.18593.1664>,
<0.19348.1664>,<0.18590.1664>,<0.14961.1664>,
<0.14962.1664>,<0.14590.1664>,<0.16758.1607>,
<0.8782.1609>,<0.7601.1664>,<0.7713.1664>,
<0.7698.1664>,<0.12729.1630>,<0.7447.1664>,
<0.24667.1614>,<0.6061.1608>,<0.32476.1608>,
<0.3392.1609>,<0.6804.1608>,<0.30233.1608>,
<0.20030.1607>,<0.5990.1608>,<0.31903.1606>,
<0.6625.1607>,<0.10804.1607>,<0.15683.1607>,
<0.8024.1607>,<0.9165.1607>,<0.4669.1607>,
<0.5649.1607>,<0.5782.1607>,<0.4785.1607>,
<0.4146.1607>,<0.4303.1607>,<0.4031.1607>,
<0.17378.1606>,<0.19026.1606>,<0.29306.1606>,
<0.18402.1606>,<0.17343.1606>,<0.16785.1606>,
<0.14744.1603>,<0.8644.1605>,<0.30312.1605>,
<0.10720.1606>,<0.12515.1606>,<0.12954.1606>,
<0.14517.1606>,<0.12546.1606>,<0.12497.1606>,
<0.12504.1606>,<0.12487.1606>,<0.7966.1606>,
<0.9591.1606>,<0.8233.1606>,<0.67.1606>,
<0.17590.1605>,<0.23871.1605>,<0.26644.1605>,
<0.19398.1605>,<0.17778.1605>,<0.16511.1605>,
<0.17441.1605>,<0.15428.1604>,<0.27519.1604>,
<0.778.1605>,<0.1828.1605>,<0.31210.1604>,
<0.702.1605>,<0.20493.1604>,<0.24296.1604>,
<0.20056.1604>,<0.17995.1604>,<0.25927.1603>,
<0.4140.1604>,<0.10193.1604>,<0.29951.1603>,
<0.25818.1603>,<0.25858.1603>,<0.17842.1603>,
<0.25679.1603>,<0.28190.1601>,<0.20733.1602>,
<0.29484.1602>,<0.4592.1603>,<0.5423.1603>,
<0.3647.1603>,<0.22412.1602>,<0.22369.1602>,
<0.14006.1602>,<0.14897.1602>,<0.15392.1602>,
<0.14929.1602>,<0.14808.1602>,<0.12401.1602>,
<0.13956.1602>,<0.10896.1602>,<0.32219.1600>,
<0.19259.1601>,<0.25904.1601>,<0.25531.1601>,
<0.10195.1601>,<0.16185.1601>,<0.6787.1601>,
<0.8483.1601>,<0.6743.1601>,<0.16154.1600>,
<0.19335.1600>,<0.31800.1600>,<0.31799.1600>,
<0.17706.1600>,<0.14956.1600>,<0.15843.1600>,
<0.9227.1600>,<0.10289.1600>,<0.15898.1481>,
<0.25930.1539>,<0.15536.1541>,<0.9167.1543>,
<0.21017.1543>,<0.1365.1544>,<0.24637.1581>,
<0.7160.1600>,<0.21914.1580>,<0.418.1544>,
<0.10190.1543>,<0.17772.1543>,<0.17860.1543>,
<0.14601.1543>,<0.10105.1543>,<0.10004.1543>,
<0.4605.1543>,<0.6629.1543>,<0.9085.1543>,
<0.9160.1543>,<0.9157.1543>,<0.7587.1543>,
<0.9081.1543>,<0.4611.1543>,<0.6003.1543>,
<0.6628.1543>,<0.4609.1543>,<0.25109.1542>,
<0.32659.1542>,<0.59.1543>,<0.12390.1542>,
<0.13055.1542>,<0.7970.1542>,<0.11888.1542>,
<0.6722.1542>,<0.17760.1540>,<0.25926.1540>,
<0.735.1541>,<0.4483.1541>,<0.14391.1541>,
<0.3046.1541>,<0.32256.1540>,<0.32219.1540>,
<0.22916.1540>,<0.24084.1540>,<0.20018.1540>,
<0.19578.1540>,<0.2707.1540>,<0.14275.1540>,
<0.14348.1540>,<0.13536.1540>,<0.14260.1540>,
<0.10013.1540>,<0.31959.1539>,<0.686.1540>,
<0.1428.1540>,<0.28666.1539>,<0.30385.1539>,
<0.27850.1539>,<0.24039.1537>,<0.13556.1538>,
<0.7870.1539>,<0.14428.1539>,<0.18524.1539>,
<0.14745.1539>,<0.12957.1539>,<0.12982.1539>,
<0.14161.1539>,<0.12964.1539>,<0.10214.1539>,
<0.5424.1539>,<0.5577.1539>,<0.25745.1538>,
<0.3150.1539>,<0.23421.1538>,<0.9024.1538>,
<0.12379.1538>,<0.12758.1538>,<0.12638.1538>,
<0.11111.1538>,<0.10931.1538>,<0.32176.1537>,
<0.7854.1538>,<0.25289.1537>,<0.24621.1537>,
<0.16581.1536>,<0.11492.1537>,<0.20791.1537>,
<0.22891.1537>,<0.22724.1537>,<0.20695.1537>,
<0.1274.1537>,<0.6800.1537>,<0.7713.1537>,
<0.5457.1537>,<0.21443.1536>,<0.16901.1536>,
<0.7050.1536>,<0.12101.1536>,<0.13514.1536>,
<0.11025.1536>,<0.26205.1486>,<0.30895.1486>,
<0.22532.1486>,<0.19896.1486>,<0.20688.1477>,
<0.4108.1478>,<0.20466.1478>,<0.5935.1480>,
<0.22574.1480>,<0.8858.1481>,<0.7430.1481>,
<0.21994.1480>,<0.17758.1479>,<0.22318.1479>,
<0.460.1480>,<0.12113.1479>,<0.32611.1478>,
<0.13875.1478>,<0.16816.1478>,<0.16856.1478>,
<0.16748.1478>,<0.16749.1478>,<0.16412.1478>,
<0.12509.1478>,<0.12516.1478>,<0.13148.1478>,
<0.5355.1478>,<0.12483.1478>,<0.28514.1477>,
<0.32710.1477>,<0.1360.1478>,<0.3127.1478>,
<0.2193.1478>,<0.173.1478>,<0.443.1478>,
<0.31507.1477>,<0.31734.1477>,<0.31733.1477>,
<0.28900.1477>,<0.30797.1477>,<0.28655.1477>,
<0.25350.1477>,<0.25651.1477>,<0.26237.1477>,
<0.23380.1477>,<0.23649.1477>,<0.22916.1477>,
<0.23314.1477>,<0.22882.1477>,<0.8632.1476>,
<0.3843.1477>,<0.8675.1477>,<0.17816.1477>,
<0.19747.1477>,<0.19345.1477>,<0.15517.1477>,
<0.6700.1477>,<0.8017.1477>,<0.7583.1477>,
<0.6569.1477>,<0.28509.1476>,<0.32520.1476>,
<0.32539.1476>,<0.2545.1477>,<0.30543.1476>,
<0.20468.1476>,<0.26391.1476>,<0.27458.1476>,
<0.23928.1476>,<0.20205.1476>,<0.20414.1476>,
<0.17965.1476>,<0.1592.1476>,<0.7447.1476>,
<0.8587.1476>,<0.8590.1476>,<0.8417.1476>,
<0.5133.1476>,<0.6912.1476>,<0.4741.1476>,
<0.13787.1475>,<0.25570.1475>,<0.18242.1475>,
<0.10093.1475>,<0.11572.1475>,<0.7169.1475>,
<0.6318.1475>,<0.13958.1409>,<0.26839.1413>,
<0.7412.1472>,<0.1677.1473>,<0.29932.1473>,
<0.31275.1474>,<0.4594.1475>,<0.5020.1475>,
<0.3605.1475>,<0.3689.1475>,<0.2157.1475>,
<0.21093.1474>,<0.23502.1474>,<0.32584.1473>,
<0.16779.1473>,<0.21745.1473>,<0.28328.1473>,
<0.29775.1473>,<0.21662.1473>,<0.7879.1473>,
<0.9745.1473>,<0.15699.1473>,<0.9446.1473>,
<0.8774.1473>,<0.5839.1473>,<0.6809.1473>,
<0.17106.1472>,<0.24812.1472>,<0.30831.1472>,
<0.32317.1472>,<0.29028.1472>,<0.30245.1472>,
<0.19932.1472>,<0.24163.1472>,<0.18587.1472>,
<0.19340.1472>,<0.18254.1472>,<0.12189.1472>,
<0.14602.1472>,<0.15182.1472>,<0.16259.1472>,
<0.15117.1472>,<0.12982.1472>,<0.14597.1472>,
<0.14594.1472>,<0.12895.1472>,<0.7474.1472>,
<0.9043.1472>,<0.12146.1472>,<0.7838.1472>,
<0.9004.1472>,<0.7430.1472>,<0.7415.1472>,
<0.21416.1415>,<0.16208.1419>,<0.32590.1422>,
<0.6718.1472>,<0.7159.1472>,<0.7147.1472>,
<0.519.1472>,<0.520.1472>,<0.27578.1422>,
<0.21987.1422>,<0.13591.1416>,<0.6215.1417>,
<0.12938.1417>,<0.12505.1417>,<0.30868.1416>,
<0.32460.1416>,<0.32087.1416>,<0.23129.1416>,
<0.8141.1416>,<0.10598.1416>,<0.8447.1416>,
<0.5603.1416>,<0.25544.1415>,<0.14499.1414>,
<0.6630.1415>,<0.15702.1415>,<0.17820.1415>,
<0.30828.1414>,<0.11892.1414>,<0.13120.1414>,
<0.14056.1414>,<0.12511.1414>,<0.12527.1414>,
<0.38.1414>,<0.8584.1414>,<0.7971.1414>,
<0.28528.1413>,<0.27944.1410>,<0.23952.1412>,
<0.8341.1413>,<0.11228.1413>,<0.17494.1413>,
<0.24086.1413>,<0.13067.1413>,<0.10060.1413>,
<0.9751.1413>,<0.4242.1413>,<0.6760.1413>,
<0.6730.1413>,<0.1784.1413>,<0.3867.1413>,
<0.27845.1412>,<0.28122.1411>,<0.9088.1412>,
<0.16614.1412>,<0.14339.1412>,<0.29610.1411>,
<0.25819.1411>,<0.25851.1411>,<0.26470.1411>,
<0.25825.1411>,<0.16885.1411>,<0.25576.1411>,
<0.14509.1411>,<0.14717.1411>,<0.9557.1411>,
<0.8997.1410>,<0.18050.1410>,<0.24792.1410>,
<0.26566.1410>,<0.26841.1410>,<0.26289.1410>,
<0.25680.1410>,<0.21445.1410>,<0.23416.1410>,
<0.24348.1410>,<0.20241.1410>,<0.13565.1410>,
<0.14799.1410>,<0.15941.1410>,<0.17944.1410>,
<0.17960.1410>,<0.15869.1410>,<0.14425.1410>,
<0.14741.1410>,<0.14004.1410>,<0.10928.1410>,
<0.12596.1410>,<0.12639.1410>,<0.11826.1410>,
<0.9160.1410>,<0.24580.1409>,<0.25985.1409>,
<0.31982.1409>,<0.8964.1410>,<0.31525.1409>,
<0.25897.1409>,<0.25964.1409>,<0.25880.1409>,
<0.21195.1409>,<0.23985.1409>,<0.24041.1409>,
<0.22463.1409>,<0.22763.1409>,<0.20212.1409>,
<0.20215.1409>,<0.17686.1409>,<0.16435.1409>,
<0.32658.1350>,<0.23018.1358>,<0.17661.1408>,
<0.6598.1409>,<0.7096.1409>,<0.9851.1409>,
<0.12903.1409>,<0.7744.1409>,<0.6746.1409>,
<0.6806.1409>,<0.6786.1409>,<0.6602.1409>,
<0.6728.1409>,<0.22142.1408>,<0.3910.1409>,
<0.6506.1409>,<0.31801.1408>,<0.2368.1409>,
<0.30469.1408>,<0.21575.1408>,<0.21718.1408>,
<0.20913.1408>,<0.21444.1408>,<0.8734.1408>,
<0.14958.1408>,<0.15544.1408>,<0.16148.1408>,
<0.14959.1408>,<0.14431.1408>,<0.14599.1408>,
<0.13251.1408>,<0.14207.1408>,<0.12964.1408>,
<0.13584.1378>,<0.7699.1408>,<0.8687.1408>,
<0.7074.1408>,<0.7619.1408>,<0.27596.1358>,
<0.24661.1358>,<0.13640.1351>,<0.5951.1352>,
<0.1226.1353>,<0.7418.1353>,<0.16157.1355>,
<0.20895.1358>,<0.1476.1353>,<0.6478.1353>,
<0.29403.1352>,<0.30867.1352>,<0.6360.1352>,
<0.23459.1351>,<0.1361.1352>,<0.2660.1352>,
<0.5584.1352>,<0.30738.1351>,<0.14641.1351>,
<0.16793.1351>,<0.4172.1351>,<0.4546.1351>,
<0.6636.1351>,<0.9084.1351>,<0.6633.1351>,
<0.4544.1351>,<0.4543.1351>,<0.3098.1351>,
<0.3431.1351>,<0.3400.1351>,<0.86.1351>,
<0.30883.1349>,<0.12473.1350>,<0.21335.1350>,
<0.26997.1350>,<0.31529.1350>,<0.31535.1350>,
<0.22897.1350>,<0.16784.1350>,<0.21293.1350>,
<0.16756.1350>,<0.13483.1350>,<0.3125.1350>,
<0.6850.1350>,<0.12405.1350>,<0.12463.1350>,
<0.10769.1350>,<0.11902.1350>,<0.7673.1350>,
<0.5447.1350>,<0.6793.1350>,<0.6801.1350>,
<0.5112.1350>,<0.716.1350>,<0.2232.1350>,
<0.3094.1350>,<0.1298.1350>,<0.104.1350>,
<0.469.1350>,<0.90.1350>,<0.17408.1349>,
<0.22382.1349>,<0.27508.1349>,<0.29007.1349>,
<0.25946.1349>,<0.27206.1349>,<0.19746.1349>,
<0.20916.1349>,<0.22380.1349>,<0.19748.1349>,
<0.18385.1349>,<0.19419.1349>,<0.9230.1349>,
<0.16528.1349>,<0.16930.1349>,<0.16539.1349>,
<0.13200.1349>,<0.16507.1349>,<0.11950.1349>,
<0.8674.1349>,<0.9207.1349>,<0.8942.1349>,
<0.6979.1349>,<0.7705.1349>,<0.8643.1349>,
<0.4646.1349>,<0.26815.1306>,<0.21644.1318>,
<0.5358.1345>,<0.31990.1346>,<0.31625.1347>,
<0.16554.1348>,<0.20020.1348>,<0.21728.1348>,
<0.30120.1348>,<0.30314.1348>,<0.27737.1348>,
<0.20575.1348>,<0.21725.1348>,<0.20057.1348>,
<0.18368.1348>,<0.19164.1348>,<0.19505.1348>,
<0.16610.1348>,<0.5729.1348>,<0.6539.1348>,
<0.8648.1348>,<0.11316.1348>,<0.8586.1348>,
<0.5777.1348>,<0.6254.1348>,<0.4784.1348>,
<0.5482.1348>,<0.1128.1348>,<0.4740.1348>,
<0.15361.1347>,<0.25672.1347>,<0.25855.1347>,
<0.29723.1347>,<0.28664.1347>,<0.25678.1347>,
<0.25680.1347>,<0.22033.1347>,<0.25573.1347>,
<0.25671.1347>,<0.23158.1347>,<0.22922.1347>,
<0.17527.1347>,<0.21194.1347>,<0.21356.1347>,
<0.17286.1347>,<0.17091.1347>,<0.7712.1347>,
<0.12973.1347>,<0.14776.1347>,<0.14755.1347>,
<0.9803.1347>,<0.11666.1347>,<0.9786.1347>,
<0.9800.1347>,<0.9559.1347>,<0.2379.1347>,
<0.5005.1347>,<0.4201.1347>,<0.1681.1347>,
<0.2216.1347>,<0.29938.1345>,<0.23381.1346>,
<0.26558.1346>,<0.28974.1346>,<0.30542.1346>,
<0.31657.1346>,<0.30339.1346>,<0.26850.1346>,
<0.28939.1346>,<0.26560.1346>,<0.26172.1346>,
<0.26546.1346>,<0.26505.1346>,<0.24790.1346>,
<0.23538.1346>,<0.9569.1346>,<0.18646.1346>,
<0.21952.1346>,<0.18269.1346>,<0.17945.1346>,
<0.6536.1346>,<0.6642.1346>,<0.7088.1346>,
<0.9010.1346>,<0.6896.1346>,<0.6593.1346>,
<0.4494.1346>,<0.5078.1346>,<0.176.1346>,
<0.1411.1346>,<0.32055.1345>,<0.20832.1345>,
<0.28137.1345>,<0.29882.1345>,<0.29916.1345>,
<0.29897.1345>,<0.29039.1345>,<0.29012.1345>,
<0.24623.1345>,<0.25973.1345>,<0.27201.1345>,
<0.27878.1345>,<0.25898.1345>,<0.25966.1345>,
<0.23991.1345>,<0.24622.1345>,<0.21973.1345>,
<snip>….
<0.7980.1287>,<0.28152.1286>,<0.6526.1287>,
<0.6359.1232>,<0.17993.1284>,<0.93.0>]},
{stack_size,24},
{reductions,348611}],
[{neighbour,
[{pid,<0.11356.1924>},
{registered_name,[]},
{initial_call,
{mochiweb_acceptor,init,
['Argument__1','Argument__2','Argument__3']}},
{current_function,
{couch_changes,wait_db_updated,3}},
{ancestors,
[couch_httpd,couch_secondary_services,
couch_server_sup,<0.32.0>]},
{messages,[]},
{links,
[<0.4862.1232>,<0.28215.1987>,
#Port<0.1026167>]},
{dictionary,
[{mochiweb_request_qs,
[{"feed","longpoll"},
{"limit","50"},
{"heartbeat","300000"},
{"style","all_docs"},
{"since","8226"}]},
{couch_rewrite_count,0},
{jsonp,no_jsonp},
{mochiweb_request_cookie,[]},
{last_changes_heartbeat,
{1398,933177,291407}}]},
{trap_exit,false},
{status,waiting},
{heap_size,6765},
{stack_size,56},
{reductions,5878}]},
{neighbour,
[{pid,<0.8631.1924>},
{registered_name,[]},
{initial_call,
{mochiweb_acceptor,init,
['Argument__1','Argument__2','Argument__3']}},
{current_function,{mochiweb_http,request,2}},
{ancestors,
[couch_httpd,couch_secondary_services,
couch_server_sup,<0.32.0>]},
{messages,[]},
{links,[<0.4862.1232>,#Port<0.1026166>]},
{dictionary,[{couch_rewrite_count,0}]},
{trap_exit,false},
{status,waiting},
{heap_size,4181},
{stack_size,8},
{reductions,5839}]},
{neighbour,
[{pid,<0.6609.1924>},
{registered_name,[]},
{initial_call,
{mochiweb_acceptor,init,
['Argument__1','Argument__2','Argument__3']}},
{current_function,
{couch_changes,wait_db_updated,3}},
{ancestors,
[couch_httpd,couch_secondary_services,
couch_server_sup,<0.32.0>]},
{messages,[]},
{links,
[<0.4862.1232>,<0.5022.1987>,#Port<0.1026164>]},
{dictionary,
[{mochiweb_request_qs,
[{"feed","longpoll"},
{"limit","50"},
{"heartbeat","300000"},
{"style","all_docs"},
{"since","295"}]},
{couch_rewrite_count,0},
{jsonp,no_jsonp},
{mochiweb_request_cookie,[]},
{last_changes_heartbeat,{1398,933177,46188}}]},
{trap_exit,false},
{status,waiting},
{heap_size,6765},
{stack_size,56},
……
It seems like it is happening again, with more info this time. It looks like I am hitting some sort of system limit, can anyone point out where to look next?
[Thu, 01 May 2014 08:32:57 GMT] [error] [<0.4862.1232>] {error_report,<0.31.0>,
{<0.4862.1232>,crash_report,
[[{initial_call,
{mochiweb_socket_server,init,['Argument__1']}},
{pid,<0.4862.1232>},
{registered_name,couch_httpd},
{error_info,
{exit,
{system_limit,
[{erlang,spawn_link,
[proc_lib,init_p,
[couch_httpd,
[couch_secondary_services,couch_server_sup,
<0.32.0>],
mochiweb_acceptor,init,
[<0.4862.1232>,#Port<0.1019424>,
#Fun<mochiweb_http.0.65037361>]]]},
{proc_lib,spawn_link,3},
{mochiweb_socket_server,recycle_acceptor,2},
{mochiweb_socket_server,handle_cast,2},
{gen_server,handle_msg,5},
{proc_lib,init_p_do_apply,3}]},
[{gen_server,terminate,6},
{proc_lib,init_p_do_apply,3}]}},
{ancestors,
[couch_secondary_services,couch_server_sup,
<0.32.0>]},
{messages,
[{'EXIT',<0.21726.1924>,normal},
{ancestors,
[couch_secondary_services,couch_server_sup,
<0.32.0>]},
{messages,
{'EXIT',<0.32257.1924>,normal},
{'EXIT',<0.10859.1927>,normal},
{'EXIT',<0.10108.1927>,normal},
{'EXIT',<0.14640.1927>,normal},
{'EXIT',<0.29939.1985>,normal},
{'EXIT',<0.7931.1984>,normal},
{'EXIT',<0.4449.1985>,normal},
{'EXIT',<0.25731.1985>,normal},
{'EXIT',<0.9721.1987>,normal},
{'EXIT',<0.10558.1987>,normal},
{'EXIT',<0.28101.1987>,normal},
{'EXIT',<0.31758.1987>,normal},
{'EXIT',<0.6565.2050>,normal},
{'EXIT',<0.9376.2050>,normal}]},
{links,
[<0.774.1349>,<0.5904.1475>,<0.7251.1600>,
<0.30241.1666>,<0.7102.1793>,<0.24040.1857>,
<0.3099.1923>,<0.685.1924>,<0.13290.1924>,
<0.16551.1924>,<0.18397.1924>,<0.15623.1924>,
<0.6609.1924>,<0.8631.1924>,<0.11356.1924>,
<0.7864.1924>,<0.2600.1924>,<0.1248.1924>,
<0.26762.1923>,<0.654.1924>,<0.14429.1923>,
<0.21428.1923>,<0.12438.1923>,<0.17675.1862>,
<0.27305.1863>,<0.6859.1922>,<0.21700.1922>,
<0.15868.1922>,<0.5406.1921>,<0.10183.1921>,
<0.4209.1863>,<0.22336.1863>,<0.34.1863>,
<0.10899.1858>,<0.28185.1858>,<0.25943.1861>,
<0.11942.1862>,<0.16801.1861>,<0.15068.1858>,
<0.28187.1857>,<0.9003.1858>,<0.9009.1858>,
<0.6787.1858>,<0.25901.1857>,<0.10955.1798>,
<0.7361.1856>,<0.3933.1857>,<0.21045.1857>,
<0.22194.1857>,<0.16436.1857>,<0.17729.1857>,
<0.18985.1856>,<0.25162.1856>,<0.30460.1856>,
<0.20116.1856>,<0.11971.1856>,<0.13515.1856>,
<0.11335.1856>,<0.4659.1799>,<0.11134.1804>,
<0.16714.1804>,<0.6153.1801>,<0.12446.1801>,
<0.22422.1799>,<0.21336.1798>,<0.31470.1798>,
<0.17373.1798>,<0.8167.1796>,<0.12866.1797>,
<0.1220.1798>,<0.10395.1798>,<0.16510.1797>,
<0.106.1798>,<0.2541.1797>,<0.4215.1797>,
<0.7722.1797>,<0.3045.1797>,<0.16605.1796>,
<0.27758.1796>,<0.11818.1794>,<0.25467.1794>,
<0.27660.1794>,<0.29591.1795>,<0.26899.1794>,
<0.21140.1794>,<0.24245.1794>,<0.13941.1794>,
<0.13009.1793>,<0.9970.1793>,<0.5728.1732>,
<0.31532.1734>,<0.20101.1742>,<0.15183.1792>,
<0.29006.1792>,<0.4300.1793>,<0.6785.1793>,
<0.31530.1792>,<0.19338.1792>,<0.25167.1792>,
<0.28653.1792>,<0.18982.1792>,<0.7255.1792>,
<0.7445.1792>,<0.13516.1792>,<0.14592.1792>,
<0.7706.1792>,<0.7312.1792>,<0.15718.1747>,
<0.137.1792>,<0.141.1792>,<0.12800.1757>,
<0.2679.1743>,<0.22873.1745>,<0.32384.1735>,
<0.8253.1737>,<0.28419.1737>,<0.11164.1737>,
<0.23579.1736>,<0.2043.1737>,<0.24505.1736>,
<0.7226.1736>,<0.1954.1736>,<0.15836.1735>,
<0.17749.1735>,<0.24425.1735>,<0.29928.1735>,
<0.22608.1735>,<0.16792.1735>,<0.8048.1735>,
<0.9565.1735>,<0.13482.1735>,<0.9162.1735>,
<0.4542.1735>,<0.7932.1735>,<0.80.1735>,
<0.27204.1733>,<0.12399.1734>,<0.19006.1734>,
<0.23319.1734>,<0.30827.1734>,<0.30826.1734>,
<0.19606.1734>,<0.23257.1734>,<0.16768.1734>,
<0.18675.1734>,<0.14808.1734>,<0.15843.1734>,
<0.13853.1734>,<0.102.1734>,<0.4349.1734>,
<0.11895.1734>,<0.3130.1734>,<0.2803.1734>,
<0.31505.1733>,<0.31736.1733>,<0.31729.1733>,
<0.27510.1733>,<0.27108.1732>,<0.21084.1733>,
<0.24389.1733>,<0.25944.1733>,<0.26050.1733>,
<0.24084.1733>,<0.22915.1733>,<0.8186.1733>,
<0.18381.1733>,<0.18382.1733>,<0.30292.1732>,
<0.28981.1732>,<0.15608.1732>,<0.19139.1732>,
<0.23884.1732>,<0.24099.1732>,<0.21688.1732>,
<0.17326.1732>,<0.17763.1732>,<0.7495.1732>,
<0.11357.1732>,<0.13257.1732>,<0.9109.1732>,
<0.5732.1732>,<0.31583.1670>,<0.19683.1673>,
<0.1485.1730>,<0.10926.1731>,<0.28468.1731>,
<0.3573.1732>,<0.5141.1732>,<0.14511.1731>,
<0.19499.1731>,<0.25843.1730>,<0.1102.1731>,
<0.31659.1730>,<0.18511.1730>,<0.10347.1730>,
<0.6727.1729>,<0.9902.1729>,<0.19448.1729>,
<0.7704.1729>,<0.7651.1729>,<0.12980.1728>,
<0.14949.1728>,<0.27554.1678>,<0.1867.1672>,
<0.10706.1673>,<0.10776.1673>,<0.17424.1673>,
<0.22263.1672>,<0.17651.1671>,<0.24624.1671>,
<0.1356.1672>,<0.17813.1671>,<0.14169.1671>,
<0.26739.1668>,<0.17849.1669>,<0.21292.1670>,
<0.31476.1670>,<0.22843.1670>,<0.20265.1669>,
<0.10089.1669>,<0.12590.1669>,<0.15345.1669>,
<0.12879.1669>,<0.11554.1669>,<0.1307.1669>,
<0.6734.1669>,<0.31213.1668>,<0.26881.1668>,
<0.17243.1667>,<0.4739.1668>,<0.14390.1668>,
<0.16608.1668>,<0.13975.1668>,<0.3916.1668>,
<0.4366.1668>,<0.29566.1667>,<0.656.1668>,
<0.6036.1667>,<0.7620.1667>,<0.15362.1667>,
<0.2154.1667>,<0.3685.1667>,<0.4986.1667>,
<0.3546.1667>,<0.587.1667>,<0.15187.1606>,
<0.10288.1664>,<0.20792.1665>,<0.5071.1666>,
<0.25743.1666>,<0.27388.1666>,<0.29174.1666>,
<0.26103.1666>,<0.26574.1666>,<0.21647.1666>,
<0.22370.1666>,<0.22262.1666>,<0.13801.1666>,
<0.19361.1666>,<0.25474.1665>,<0.29539.1665>,
<0.3562.1666>,<0.28624.1665>,<0.22727.1665>,
<0.24582.1665>,<0.24583.1665>,<0.23223.1665>,
<0.21197.1665>,<0.21693.1665>,<0.20885.1665>,
<0.4360.1665>,<0.6802.1665>,<0.15519.1665>,
<0.19258.1665>,<0.20552.1665>,<0.18502.1665>,
<0.14356.1665>,<0.6777.1665>,<0.6784.1665>,
<0.6505.1665>,<0.6509.1665>,<0.5320.1665>,
<0.25160.1664>,<0.30000.1664>,<0.31791.1664>,
<0.29782.1664>,<0.18265.1664>,<0.18593.1664>,
<0.19348.1664>,<0.18590.1664>,<0.14961.1664>,
<0.14962.1664>,<0.14590.1664>,<0.16758.1607>,
<0.8782.1609>,<0.7601.1664>,<0.7713.1664>,
<0.7698.1664>,<0.12729.1630>,<0.7447.1664>,
<0.24667.1614>,<0.6061.1608>,<0.32476.1608>,
<0.3392.1609>,<0.6804.1608>,<0.30233.1608>,
<0.20030.1607>,<0.5990.1608>,<0.31903.1606>,
<0.6625.1607>,<0.10804.1607>,<0.15683.1607>,
<0.8024.1607>,<0.9165.1607>,<0.4669.1607>,
<0.5649.1607>,<0.5782.1607>,<0.4785.1607>,
<0.4146.1607>,<0.4303.1607>,<0.4031.1607>,
<0.17378.1606>,<0.19026.1606>,<0.29306.1606>,
<0.18402.1606>,<0.17343.1606>,<0.16785.1606>,
<0.14744.1603>,<0.8644.1605>,<0.30312.1605>,
<0.10720.1606>,<0.12515.1606>,<0.12954.1606>,
<0.14517.1606>,<0.12546.1606>,<0.12497.1606>,
<0.12504.1606>,<0.12487.1606>,<0.7966.1606>,
<0.9591.1606>,<0.8233.1606>,<0.67.1606>,
<0.17590.1605>,<0.23871.1605>,<0.26644.1605>,
<0.19398.1605>,<0.17778.1605>,<0.16511.1605>,
<0.17441.1605>,<0.15428.1604>,<0.27519.1604>,
<0.778.1605>,<0.1828.1605>,<0.31210.1604>,
<0.702.1605>,<0.20493.1604>,<0.24296.1604>,
<0.20056.1604>,<0.17995.1604>,<0.25927.1603>,
<0.4140.1604>,<0.10193.1604>,<0.29951.1603>,
<0.25818.1603>,<0.25858.1603>,<0.17842.1603>,
<0.25679.1603>,<0.28190.1601>,<0.20733.1602>,
<0.29484.1602>,<0.4592.1603>,<0.5423.1603>,
<0.3647.1603>,<0.22412.1602>,<0.22369.1602>,
<0.14006.1602>,<0.14897.1602>,<0.15392.1602>,
<0.14929.1602>,<0.14808.1602>,<0.12401.1602>,
<0.13956.1602>,<0.10896.1602>,<0.32219.1600>,
<0.19259.1601>,<0.25904.1601>,<0.25531.1601>,
<0.10195.1601>,<0.16185.1601>,<0.6787.1601>,
<0.8483.1601>,<0.6743.1601>,<0.16154.1600>,
<0.19335.1600>,<0.31800.1600>,<0.31799.1600>,
<0.17706.1600>,<0.14956.1600>,<0.15843.1600>,
<0.9227.1600>,<0.10289.1600>,<0.15898.1481>,
<0.25930.1539>,<0.15536.1541>,<0.9167.1543>,
<0.21017.1543>,<0.1365.1544>,<0.24637.1581>,
<0.7160.1600>,<0.21914.1580>,<0.418.1544>,
<0.10190.1543>,<0.17772.1543>,<0.17860.1543>,
<0.14601.1543>,<0.10105.1543>,<0.10004.1543>,
<0.4605.1543>,<0.6629.1543>,<0.9085.1543>,
<0.9160.1543>,<0.9157.1543>,<0.7587.1543>,
<0.9081.1543>,<0.4611.1543>,<0.6003.1543>,
<0.6628.1543>,<0.4609.1543>,<0.25109.1542>,
<0.32659.1542>,<0.59.1543>,<0.12390.1542>,
<0.13055.1542>,<0.7970.1542>,<0.11888.1542>,
<0.6722.1542>,<0.17760.1540>,<0.25926.1540>,
<0.735.1541>,<0.4483.1541>,<0.14391.1541>,
<0.3046.1541>,<0.32256.1540>,<0.32219.1540>,
<0.22916.1540>,<0.24084.1540>,<0.20018.1540>,
<0.19578.1540>,<0.2707.1540>,<0.14275.1540>,
<0.14348.1540>,<0.13536.1540>,<0.14260.1540>,
<0.10013.1540>,<0.31959.1539>,<0.686.1540>,
<0.1428.1540>,<0.28666.1539>,<0.30385.1539>,
<0.27850.1539>,<0.24039.1537>,<0.13556.1538>,
<0.7870.1539>,<0.14428.1539>,<0.18524.1539>,
<0.14745.1539>,<0.12957.1539>,<0.12982.1539>,
<0.14161.1539>,<0.12964.1539>,<0.10214.1539>,
<0.5424.1539>,<0.5577.1539>,<0.25745.1538>,
<0.3150.1539>,<0.23421.1538>,<0.9024.1538>,
<0.12379.1538>,<0.12758.1538>,<0.12638.1538>,
<0.11111.1538>,<0.10931.1538>,<0.32176.1537>,
<0.7854.1538>,<0.25289.1537>,<0.24621.1537>,
<0.16581.1536>,<0.11492.1537>,<0.20791.1537>,
<0.22891.1537>,<0.22724.1537>,<0.20695.1537>,
<0.1274.1537>,<0.6800.1537>,<0.7713.1537>,
<0.5457.1537>,<0.21443.1536>,<0.16901.1536>,
<0.7050.1536>,<0.12101.1536>,<0.13514.1536>,
<0.11025.1536>,<0.26205.1486>,<0.30895.1486>,
<0.22532.1486>,<0.19896.1486>,<0.20688.1477>,
<0.4108.1478>,<0.20466.1478>,<0.5935.1480>,
<0.22574.1480>,<0.8858.1481>,<0.7430.1481>,
<0.21994.1480>,<0.17758.1479>,<0.22318.1479>,
<0.460.1480>,<0.12113.1479>,<0.32611.1478>,
<0.13875.1478>,<0.16816.1478>,<0.16856.1478>,
<0.16748.1478>,<0.16749.1478>,<0.16412.1478>,
<0.12509.1478>,<0.12516.1478>,<0.13148.1478>,
<0.5355.1478>,<0.12483.1478>,<0.28514.1477>,
<0.32710.1477>,<0.1360.1478>,<0.3127.1478>,
<0.2193.1478>,<0.173.1478>,<0.443.1478>,
<0.31507.1477>,<0.31734.1477>,<0.31733.1477>,
<0.28900.1477>,<0.30797.1477>,<0.28655.1477>,
<0.25350.1477>,<0.25651.1477>,<0.26237.1477>,
<0.23380.1477>,<0.23649.1477>,<0.22916.1477>,
<0.23314.1477>,<0.22882.1477>,<0.8632.1476>,
<0.3843.1477>,<0.8675.1477>,<0.17816.1477>,
<0.19747.1477>,<0.19345.1477>,<0.15517.1477>,
<0.6700.1477>,<0.8017.1477>,<0.7583.1477>,
<0.6569.1477>,<0.28509.1476>,<0.32520.1476>,
<0.32539.1476>,<0.2545.1477>,<0.30543.1476>,
<0.20468.1476>,<0.26391.1476>,<0.27458.1476>,
<0.23928.1476>,<0.20205.1476>,<0.20414.1476>,
<0.17965.1476>,<0.1592.1476>,<0.7447.1476>,
<0.8587.1476>,<0.8590.1476>,<0.8417.1476>,
<0.5133.1476>,<0.6912.1476>,<0.4741.1476>,
<0.13787.1475>,<0.25570.1475>,<0.18242.1475>,
<0.10093.1475>,<0.11572.1475>,<0.7169.1475>,
<0.6318.1475>,<0.13958.1409>,<0.26839.1413>,
<0.7412.1472>,<0.1677.1473>,<0.29932.1473>,
<0.31275.1474>,<0.4594.1475>,<0.5020.1475>,
<0.3605.1475>,<0.3689.1475>,<0.2157.1475>,
<0.21093.1474>,<0.23502.1474>,<0.32584.1473>,
<0.16779.1473>,<0.21745.1473>,<0.28328.1473>,
<0.29775.1473>,<0.21662.1473>,<0.7879.1473>,
<0.9745.1473>,<0.15699.1473>,<0.9446.1473>,
<0.8774.1473>,<0.5839.1473>,<0.6809.1473>,
<0.17106.1472>,<0.24812.1472>,<0.30831.1472>,
<0.32317.1472>,<0.29028.1472>,<0.30245.1472>,
<0.19932.1472>,<0.24163.1472>,<0.18587.1472>,
<0.19340.1472>,<0.18254.1472>,<0.12189.1472>,
<0.14602.1472>,<0.15182.1472>,<0.16259.1472>,
<0.15117.1472>,<0.12982.1472>,<0.14597.1472>,
<0.14594.1472>,<0.12895.1472>,<0.7474.1472>,
<0.9043.1472>,<0.12146.1472>,<0.7838.1472>,
<0.9004.1472>,<0.7430.1472>,<0.7415.1472>,
<0.21416.1415>,<0.16208.1419>,<0.32590.1422>,
<0.6718.1472>,<0.7159.1472>,<0.7147.1472>,
<0.519.1472>,<0.520.1472>,<0.27578.1422>,
<0.21987.1422>,<0.13591.1416>,<0.6215.1417>,
<0.12938.1417>,<0.12505.1417>,<0.30868.1416>,
<0.32460.1416>,<0.32087.1416>,<0.23129.1416>,
<0.8141.1416>,<0.10598.1416>,<0.8447.1416>,
<0.5603.1416>,<0.25544.1415>,<0.14499.1414>,
<0.6630.1415>,<0.15702.1415>,<0.17820.1415>,
<0.30828.1414>,<0.11892.1414>,<0.13120.1414>,
<0.14056.1414>,<0.12511.1414>,<0.12527.1414>,
<0.38.1414>,<0.8584.1414>,<0.7971.1414>,
<0.28528.1413>,<0.27944.1410>,<0.23952.1412>,
<0.8341.1413>,<0.11228.1413>,<0.17494.1413>,
<0.24086.1413>,<0.13067.1413>,<0.10060.1413>,
<0.9751.1413>,<0.4242.1413>,<0.6760.1413>,
<0.6730.1413>,<0.1784.1413>,<0.3867.1413>,
<0.27845.1412>,<0.28122.1411>,<0.9088.1412>,
<0.16614.1412>,<0.14339.1412>,<0.29610.1411>,
<0.25819.1411>,<0.25851.1411>,<0.26470.1411>,
<0.25825.1411>,<0.16885.1411>,<0.25576.1411>,
<0.14509.1411>,<0.14717.1411>,<0.9557.1411>,
<0.8997.1410>,<0.18050.1410>,<0.24792.1410>,
<0.26566.1410>,<0.26841.1410>,<0.26289.1410>,
<0.25680.1410>,<0.21445.1410>,<0.23416.1410>,
<0.24348.1410>,<0.20241.1410>,<0.13565.1410>,
<0.14799.1410>,<0.15941.1410>,<0.17944.1410>,
<0.17960.1410>,<0.15869.1410>,<0.14425.1410>,
<0.14741.1410>,<0.14004.1410>,<0.10928.1410>,
<0.12596.1410>,<0.12639.1410>,<0.11826.1410>,
<0.9160.1410>,<0.24580.1409>,<0.25985.1409>,
<0.31982.1409>,<0.8964.1410>,<0.31525.1409>,
<0.25897.1409>,<0.25964.1409>,<0.25880.1409>,
<0.21195.1409>,<0.23985.1409>,<0.24041.1409>,
<0.22463.1409>,<0.22763.1409>,<0.20212.1409>,
<0.20215.1409>,<0.17686.1409>,<0.16435.1409>,
<0.32658.1350>,<0.23018.1358>,<0.17661.1408>,
<0.6598.1409>,<0.7096.1409>,<0.9851.1409>,
<0.12903.1409>,<0.7744.1409>,<0.6746.1409>,
<0.6806.1409>,<0.6786.1409>,<0.6602.1409>,
<0.6728.1409>,<0.22142.1408>,<0.3910.1409>,
<0.6506.1409>,<0.31801.1408>,<0.2368.1409>,
<0.30469.1408>,<0.21575.1408>,<0.21718.1408>,
<0.20913.1408>,<0.21444.1408>,<0.8734.1408>,
<0.14958.1408>,<0.15544.1408>,<0.16148.1408>,
<0.14959.1408>,<0.14431.1408>,<0.14599.1408>,
<0.13251.1408>,<0.14207.1408>,<0.12964.1408>,
<0.13584.1378>,<0.7699.1408>,<0.8687.1408>,
<0.7074.1408>,<0.7619.1408>,<0.27596.1358>,
<0.24661.1358>,<0.13640.1351>,<0.5951.1352>,
<0.1226.1353>,<0.7418.1353>,<0.16157.1355>,
<0.20895.1358>,<0.1476.1353>,<0.6478.1353>,
<0.29403.1352>,<0.30867.1352>,<0.6360.1352>,
<0.23459.1351>,<0.1361.1352>,<0.2660.1352>,
<0.5584.1352>,<0.30738.1351>,<0.14641.1351>,
<0.16793.1351>,<0.4172.1351>,<0.4546.1351>,
<0.6636.1351>,<0.9084.1351>,<0.6633.1351>,
<0.4544.1351>,<0.4543.1351>,<0.3098.1351>,
<0.3431.1351>,<0.3400.1351>,<0.86.1351>,
<0.30883.1349>,<0.12473.1350>,<0.21335.1350>,
<0.26997.1350>,<0.31529.1350>,<0.31535.1350>,
<0.22897.1350>,<0.16784.1350>,<0.21293.1350>,
<0.16756.1350>,<0.13483.1350>,<0.3125.1350>,
<0.6850.1350>,<0.12405.1350>,<0.12463.1350>,
<0.10769.1350>,<0.11902.1350>,<0.7673.1350>,
<0.5447.1350>,<0.6793.1350>,<0.6801.1350>,
<0.5112.1350>,<0.716.1350>,<0.2232.1350>,
<0.3094.1350>,<0.1298.1350>,<0.104.1350>,
<0.469.1350>,<0.90.1350>,<0.17408.1349>,
<0.22382.1349>,<0.27508.1349>,<0.29007.1349>,
<0.25946.1349>,<0.27206.1349>,<0.19746.1349>,
<0.20916.1349>,<0.22380.1349>,<0.19748.1349>,
<0.18385.1349>,<0.19419.1349>,<0.9230.1349>,
<0.16528.1349>,<0.16930.1349>,<0.16539.1349>,
<0.13200.1349>,<0.16507.1349>,<0.11950.1349>,
<0.8674.1349>,<0.9207.1349>,<0.8942.1349>,
<0.6979.1349>,<0.7705.1349>,<0.8643.1349>,
<0.4646.1349>,<0.26815.1306>,<0.21644.1318>,
<0.5358.1345>,<0.31990.1346>,<0.31625.1347>,
<0.16554.1348>,<0.20020.1348>,<0.21728.1348>,
<0.30120.1348>,<0.30314.1348>,<0.27737.1348>,
<0.20575.1348>,<0.21725.1348>,<0.20057.1348>,
<0.18368.1348>,<0.19164.1348>,<0.19505.1348>,
<0.16610.1348>,<0.5729.1348>,<0.6539.1348>,
<0.8648.1348>,<0.11316.1348>,<0.8586.1348>,
<0.5777.1348>,<0.6254.1348>,<0.4784.1348>,
<0.5482.1348>,<0.1128.1348>,<0.4740.1348>,
<0.15361.1347>,<0.25672.1347>,<0.25855.1347>,
<0.29723.1347>,<0.28664.1347>,<0.25678.1347>,
<0.25680.1347>,<0.22033.1347>,<0.25573.1347>,
<0.25671.1347>,<0.23158.1347>,<0.22922.1347>,
<0.17527.1347>,<0.21194.1347>,<0.21356.1347>,
<0.17286.1347>,<0.17091.1347>,<0.7712.1347>,
<0.12973.1347>,<0.14776.1347>,<0.14755.1347>,
<0.9803.1347>,<0.11666.1347>,<0.9786.1347>,
<0.9800.1347>,<0.9559.1347>,<0.2379.1347>,
<0.5005.1347>,<0.4201.1347>,<0.1681.1347>,
<0.2216.1347>,<0.29938.1345>,<0.23381.1346>,
<0.26558.1346>,<0.28974.1346>,<0.30542.1346>,
<0.31657.1346>,<0.30339.1346>,<0.26850.1346>,
<0.28939.1346>,<0.26560.1346>,<0.26172.1346>,
<0.26546.1346>,<0.26505.1346>,<0.24790.1346>,
<0.23538.1346>,<0.9569.1346>,<0.18646.1346>,
<0.21952.1346>,<0.18269.1346>,<0.17945.1346>,
<0.6536.1346>,<0.6642.1346>,<0.7088.1346>,
<0.9010.1346>,<0.6896.1346>,<0.6593.1346>,
<0.4494.1346>,<0.5078.1346>,<0.176.1346>,
<0.1411.1346>,<0.32055.1345>,<0.20832.1345>,
<0.28137.1345>,<0.29882.1345>,<0.29916.1345>,
<0.29897.1345>,<0.29039.1345>,<0.29012.1345>,
<0.24623.1345>,<0.25973.1345>,<0.27201.1345>,
<0.27878.1345>,<0.25898.1345>,<0.25966.1345>,
<0.23991.1345>,<0.24622.1345>,<0.21973.1345>,
<snip>….
<0.7980.1287>,<0.28152.1286>,<0.6526.1287>,
<0.6359.1232>,<0.17993.1284>,<0.93.0>]},
{dictionary,[]},
{trap_exit,true},
{status,running},
{heap_size,17711},
{trap_exit,true},
{status,running},
{stack_size,24},
{reductions,348611}],
[{neighbour,
[{pid,<0.11356.1924>},
{registered_name,[]},
{initial_call,
{mochiweb_acceptor,init,
['Argument__1','Argument__2','Argument__3']}},
{current_function,
{couch_changes,wait_db_updated,3}},
{ancestors,
[couch_httpd,couch_secondary_services,
couch_server_sup,<0.32.0>]},
{messages,[]},
{links,
[<0.4862.1232>,<0.28215.1987>,
#Port<0.1026167>]},
{dictionary,
[{mochiweb_request_qs,
[{"feed","longpoll"},
{"limit","50"},
{"heartbeat","300000"},
{"style","all_docs"},
{"since","8226"}]},
{couch_rewrite_count,0},
{jsonp,no_jsonp},
{mochiweb_request_cookie,[]},
{last_changes_heartbeat,
{1398,933177,291407}}]},
{trap_exit,false},
{status,waiting},
{heap_size,6765},
{stack_size,56},
{reductions,5878}]},
{neighbour,
[{pid,<0.8631.1924>},
{registered_name,[]},
{initial_call,
{mochiweb_acceptor,init,
['Argument__1','Argument__2','Argument__3']}},
{current_function,{mochiweb_http,request,2}},
{ancestors,
[couch_httpd,couch_secondary_services,
couch_server_sup,<0.32.0>]},
{messages,[]},
{links,[<0.4862.1232>,#Port<0.1026166>]},
{dictionary,[{couch_rewrite_count,0}]},
{trap_exit,false},
{status,waiting},
{heap_size,4181},
{stack_size,8},
{reductions,5839}]},
{neighbour,
[{pid,<0.6609.1924>},
{registered_name,[]},
{initial_call,
{mochiweb_acceptor,init,
['Argument__1','Argument__2','Argument__3']}},
{current_function,
{couch_changes,wait_db_updated,3}},
{ancestors,
[couch_httpd,couch_secondary_services,
couch_server_sup,<0.32.0>]},
{messages,[]},
{links,
[<0.4862.1232>,<0.5022.1987>,#Port<0.1026164>]},
{dictionary,
[{mochiweb_request_qs,
[{"feed","longpoll"},
{"limit","50"},
{"heartbeat","300000"},
{"style","all_docs"},
{"since","295"}]},
{couch_rewrite_count,0},
{jsonp,no_jsonp},
{mochiweb_request_cookie,[]},
{last_changes_heartbeat,{1398,933177,46188}}]},
{trap_exit,false},
{status,waiting},
{heap_size,6765},
{stack_size,56},
……
Interactive Blueprints
May 1, 2014, 8:47:27 AM5/1/14
to us...@couchdb.apache.org
2014-05-01 13:14 GMT+02:00 Herman Chan <herm...@gmail.com>:
> Thanks Adam,
>
> It seems like it is happening again, with more info this time. It looks like I am hitting some sort of system limit, can anyone point out where to look next?
Just guessing here..
> Thanks Adam,
>
> It seems like it is happening again, with more info this time. It looks like I am hitting some sort of system limit, can anyone point out where to look next?
What could be is that you hit the max open file limit of your system.
With "ulimit -a" you can see the limits on your system.
Usually the max open file limit is somewhere around 1024.
I noticed that couchdb loves to have a lot of files open simultaneously.
Iin the same shell you start couchdb, right before you start couchdb,
you can do a "ulimit -a 4096" (or another large value), this should
give coudhb the ability to open more files.
Hope this helps.
Pieter van der Eems
Interactive Blueprints
Adam Kocoloski
May 1, 2014, 9:18:30 AM5/1/14
to us...@couchdb.apache.org
Adam
Herman Chan
May 1, 2014, 10:43:31 AM5/1/14
to us...@couchdb.apache.org
We do have 1000+ connection to the db, which we are trying to dial down. However, even with lower connection, we hit the crash again, this time I was able to get a better log. You are right that we are hitting some limit,
before the crash, the log shows that couch is still trying to open up index from a reboot that we did. Once it crash, the log start print out with "Index shutdown by monitor". Is there any limit parameter that we can increase?
[Thu, 01 May 2014 14:28:04 GMT] [error] [emulator] Too many processes
[Thu, 01 May 2014 14:28:04 GMT] [error] [emulator] Error in process <0.3672.477> with exit value: {system_limit,[{erlang,spawn_opt,[proc_lib,init_p,[<0.3672.477>,[],gen,init_it,[
gen_server,<0.3672.477>,<0.3672.477>,couch_db,{<<42 bytes>>,"/usr/local/var/lib/couchdb/group_370c0635-e593-45ed-ac96-75e6b318cb35.couch",<0.21556.480>,[{user_ctx,{user_ctx,null,
[<<6 bytes>>],undefined...
[Thu, 01 May 2014 14:28:04 GMT] [error] [<0.21556.480>] ** Generic server <0.21556.480> terminating
** Last message in was {'EXIT',<0.3672.477>,
{system_limit,
[{erlang,spawn_opt,
[proc_lib,init_p,
[<0.3672.477>,[],gen,init_it,
[gen_server,<0.3672.477>,<0.3672.477>,couch_db,
{<<"group_370c0635-e593-45ed-ac96-75e6b318cb35">>,
"/usr/local/var/lib/couchdb/group_370c0635-e593-45ed-ac96-75e6b318cb35.couch",
<0.21556.480>,
[{user_ctx,
{user_ctx,null,[<<"_admin">>],undefined}}]},
[]]],
[link]]},
{proc_lib,start_link,5},
{couch_db,start_link,3},
{couch_server,'-open_async/5-fun-0-',4}]}}
** When Server state == {file,
{file_descriptor,prim_file,
{#Port<0.898531>,307709}},
1261681}
** Reason for termination ==
** {system_limit,
[{erlang,spawn_opt,
[proc_lib,init_p,
[<0.3672.477>,[],gen,init_it,
[gen_server,<0.3672.477>,<0.3672.477>,couch_db,
{<<"group_370c0635-e593-45ed-ac96-75e6b318cb35">>,
"/usr/local/var/lib/couchdb/group_370c0635-e593-45ed-ac96-75e6b318cb35.couch",
<0.21556.480>,
[{user_ctx,{user_ctx,null,[<<"_admin">>],undefined}}]},
[]]],
[link]]},
{proc_lib,start_link,5},
{couch_db,start_link,3},
{couch_server,'-open_async/5-fun-0-',4}]}
[Thu, 01 May 2014 14:28:04 GMT] [error] [<0.21556.480>] {error_report,<0.31.0>,
{<0.21556.480>,crash_report,
[[{initial_call,{couch_file,init,['Argument__1']}},
{pid,<0.21556.480>},
{registered_name,[]},
{error_info,
{exit,
{system_limit,
[{erlang,spawn_opt,
[proc_lib,init_p,
[<0.3672.477>,[],gen,init_it,
[gen_server,<0.3672.477>,<0.3672.477>,
couch_db,
{<<"group_370c0635-e593-45ed-ac96-75e6b318cb35">>,
"/usr/local/var/lib/couchdb/group_370c0635-e593-45ed-ac96-75e6b318cb35.couch",
<0.21556.480>,
[{user_ctx,
{user_ctx,null,
[<<"_admin">>],
undefined}}]},
[]]],
[link]]},
{proc_lib,start_link,5},
{couch_db,start_link,3},
{couch_server,'-open_async/5-fun-0-',4}]},
[{gen_server,terminate,6},
{proc_lib,init_p_do_apply,3}]}},
{ancestors,[<0.3672.477>]},
{messages,[]},
{links,[]},
{stack_size,24},
{reductions,973}],
[]]}}
[Thu, 01 May 2014 14:28:04 GMT] [info] [<0.20971.87>] Index shutdown by monitor notice for db: group_5747d16f-4b3b-4522-af10-1dc7d0d644aa idx: _design/filters
[Thu, 01 May 2014 14:28:04 GMT] [info] [<0.4883.35>] Index shutdown by monitor notice for db: group_15ccf331-257d-4b54-b457-997d342816b9 idx: _design/hub
[Thu, 01 May 2014 14:28:04 GMT] [info] [<0.4892.35>] Index shutdown by monitor notice for db: group_15ccf331-257d-4b54-b457-997d342816b9 idx: _design/filters
[Thu, 01 May 2014 14:28:04 GMT] [info] [<0.12040.33>] Index shutdown by monitor notice for db: group_d006a71d-b0de-4d71-b2f7-06abeeb34e00 idx: _design/filters
[Thu, 01 May 2014 14:28:04 GMT] [info] [<0.20971.87>] Closing index for db: group_5747d16f-4b3b-4522-af10-1dc7d0d644aa idx: _design/filters sig: "3e823c2a4383ac0c18d4e574135a5b08"
reason: normal
[Thu, 01 May 2014 14:28:04 GMT] [info] [<0.12032.33>] Index shutdown by monitor notice for db: group_d006a71d-b0de-4d71-b2f7-06abeeb34e00 idx: _design/hub
[Thu, 01 May 2014 14:28:04 GMT] [info] [<0.4892.35>] Closing index for db: group_15ccf331-257d-4b54-b457-997d342816b9 idx: _design/filters sig: "3e823c2a4383ac0c18d4e574135a5b08"
reason: normal
[Thu, 01 May 2014 14:28:04 GMT] [info] [<0.4292.4>] Index shutdown by monitor notice for db: group_ae50933f-de22-4879-9624-b760106060b3 idx: _design/filters
[Thu, 01 May 2014 14:28:04 GMT] [info] [<0.4285.4>] Index shutdown by monitor notice for db: group_ae50933f-de22-4879-9624-b760106060b3 idx: _design/hub
[Thu, 01 May 2014 14:28:04 GMT] [info] [<0.20956.87>] Index shutdown by monitor notice for db: group_5747d16f-4b3b-4522-af10-1dc7d0d644aa idx: _design/hub
[Thu, 01 May 2014 14:28:04 GMT] [info] [<0.4883.35>] Closing index for db: group_15ccf331-257d-4b54-b457-997d342816b9 idx: _design/hub sig: "4f6edcabc4b7a6357b714e1391ed93ac"
reason: normal
[Thu, 01 May 2014 14:28:04 GMT] [info] [<0.12040.33>] Closing index for db: group_d006a71d-b0de-4d71-b2f7-06abeeb34e00 idx: _design/filters sig: "3e823c2a4383ac0c18d4e574135a5b08"
reason: normal
[Thu, 01 May 2014 14:28:04 GMT] [info] [<0.18850.44>] Index shutdown by monitor notice for db: group_721d99a3-2257-48d0-8a1e-89294874d06e idx: _design/filters
[Thu, 01 May 2014 14:28:04 GMT] [info] [<0.18842.44>] Index shutdown by monitor notice for db: group_721d99a3-2257-48d0-8a1e-89294874d06e idx: _design/hub
[Thu, 01 May 2014 14:28:04 GMT] [info] [<0.12032.33>] Closing index for db: group_d006a71d-b0de-4d71-b2f7-06abeeb34e00 idx: _design/hub sig: "4f6edcabc4b7a6357b714e1391ed93ac"
reason: normal
[Thu, 01 May 2014 14:28:04 GMT] [info] [<0.27768.43>] Index shutdown by monitor notice for db: group_56a0df90-c79e-4863-ae71-2bde3cb0d801 idx: _design/hub
[Thu, 01 May 2014 14:28:04 GMT] [info] [<0.27775.43>] Index shutdown by monitor notice for db: group_56a0df90-c79e-4863-ae71-2bde3cb0d801 idx: _design/filters
[Thu, 01 May 2014 14:28:04 GMT] [info] [<0.4292.4>] Closing index for db: group_ae50933f-de22-4879-9624-b760106060b3 idx: _design/filters sig: "3e823c2a4383ac0c18d4e574135a5b08"
reason: normal
[Thu, 01 May 2014 14:28:04 GMT] [info] [<0.4285.4>] Closing index for db: group_ae50933f-de22-4879-9624-b760106060b3 idx: _design/hub sig: "4f6edcabc4b7a6357b714e1391ed93ac"
reason: normal
[Thu, 01 May 2014 14:28:04 GMT] [info] [<0.6010.43>] Index shutdown by monitor notice for db: group_7f082ae6-f41d-4a14-a836-2360303b2e9a idx: _design/filters
[Thu, 01 May 2014 14:28:04 GMT] [info] [<0.6003.43>] Index shutdown by monitor notice for db: group_7f082ae6-f41d-4a14-a836-2360303b2e9a idx: _design/hub
[Thu, 01 May 2014 14:28:04 GMT] [info] [<0.20956.87>] Closing index for db: group_5747d16f-4b3b-4522-af10-1dc7d0d644aa idx: _design/hub sig: "4f6edcabc4b7a6357b714e1391ed93ac"
reason: normal
[Thu, 01 May 2014 14:28:04 GMT] [info] [<0.5933.42>] Index shutdown by monitor notice for db: group_8c49d7e8-b61e-41e5-a220-11df59b9cce4 idx: _design/hub
[Thu, 01 May 2014 14:28:04 GMT] [info] [<0.5940.42>] Index shutdown by monitor notice for db: group_8c49d7e8-b61e-41e5-a220-11df59b9cce4 idx: _design/filters
[Thu, 01 May 2014 14:28:04 GMT] [info] [<0.18842.44>] Closing index for db: group_721d99a3-2257-48d0-8a1e-89294874d06e idx: _design/hub sig: "4f6edcabc4b7a6357b714e1391ed93ac"
reason: normal
[Thu, 01 May 2014 14:28:04 GMT] [info] [<0.17529.33>] Index shutdown by monitor notice for db: group_98ff493c-63e8-4714-9940-ccea514d4b1d idx: _design/hub
[Thu, 01 May 2014 14:28:04 GMT] [info] [<0.18850.44>] Closing index for db: group_721d99a3-2257-48d0-8a1e-89294874d06e idx: _design/filters sig: "3e823c2a4383ac0c18d4e574135a5b08"
reason: normal
[Thu, 01 May 2014 14:28:04 GMT] [info] [<0.17536.33>] Index shutdown by monitor notice for db: group_98ff493c-63e8-4714-9940-ccea514d4b1d idx: _design/filters
[Thu, 01 May 2014 14:28:04 GMT] [info] [<0.27768.43>] Closing index for db: group_56a0df90-c79e-4863-ae71-2bde3cb0d801 idx: _design/hub sig: "4f6edcabc4b7a6357b714e1391ed93ac"
before the crash, the log shows that couch is still trying to open up index from a reboot that we did. Once it crash, the log start print out with "Index shutdown by monitor". Is there any limit parameter that we can increase?
[Thu, 01 May 2014 14:28:04 GMT] [error] [emulator] Too many processes
[Thu, 01 May 2014 14:28:04 GMT] [error] [emulator] Error in process <0.3672.477> with exit value: {system_limit,[{erlang,spawn_opt,[proc_lib,init_p,[<0.3672.477>,[],gen,init_it,[
gen_server,<0.3672.477>,<0.3672.477>,couch_db,{<<42 bytes>>,"/usr/local/var/lib/couchdb/group_370c0635-e593-45ed-ac96-75e6b318cb35.couch",<0.21556.480>,[{user_ctx,{user_ctx,null,
[<<6 bytes>>],undefined...
[Thu, 01 May 2014 14:28:04 GMT] [error] [<0.21556.480>] ** Generic server <0.21556.480> terminating
** Last message in was {'EXIT',<0.3672.477>,
{system_limit,
[{erlang,spawn_opt,
[proc_lib,init_p,
[<0.3672.477>,[],gen,init_it,
[gen_server,<0.3672.477>,<0.3672.477>,couch_db,
{<<"group_370c0635-e593-45ed-ac96-75e6b318cb35">>,
"/usr/local/var/lib/couchdb/group_370c0635-e593-45ed-ac96-75e6b318cb35.couch",
<0.21556.480>,
[{user_ctx,
{user_ctx,null,[<<"_admin">>],undefined}}]},
[]]],
[link]]},
{proc_lib,start_link,5},
{couch_db,start_link,3},
{couch_server,'-open_async/5-fun-0-',4}]}}
** When Server state == {file,
{file_descriptor,prim_file,
{#Port<0.898531>,307709}},
1261681}
** Reason for termination ==
** {system_limit,
[{erlang,spawn_opt,
[proc_lib,init_p,
[<0.3672.477>,[],gen,init_it,
[gen_server,<0.3672.477>,<0.3672.477>,couch_db,
{<<"group_370c0635-e593-45ed-ac96-75e6b318cb35">>,
"/usr/local/var/lib/couchdb/group_370c0635-e593-45ed-ac96-75e6b318cb35.couch",
<0.21556.480>,
[{user_ctx,{user_ctx,null,[<<"_admin">>],undefined}}]},
[]]],
[link]]},
{proc_lib,start_link,5},
{couch_db,start_link,3},
{couch_server,'-open_async/5-fun-0-',4}]}
[Thu, 01 May 2014 14:28:04 GMT] [error] [<0.21556.480>] {error_report,<0.31.0>,
{<0.21556.480>,crash_report,
[[{initial_call,{couch_file,init,['Argument__1']}},
{pid,<0.21556.480>},
{registered_name,[]},
{error_info,
{exit,
{system_limit,
[{erlang,spawn_opt,
[proc_lib,init_p,
[<0.3672.477>,[],gen,init_it,
[gen_server,<0.3672.477>,<0.3672.477>,
couch_db,
{<<"group_370c0635-e593-45ed-ac96-75e6b318cb35">>,
"/usr/local/var/lib/couchdb/group_370c0635-e593-45ed-ac96-75e6b318cb35.couch",
<0.21556.480>,
[{user_ctx,
{user_ctx,null,
[<<"_admin">>],
undefined}}]},
[]]],
[link]]},
{proc_lib,start_link,5},
{couch_db,start_link,3},
{couch_server,'-open_async/5-fun-0-',4}]},
[{gen_server,terminate,6},
{proc_lib,init_p_do_apply,3}]}},
{ancestors,[<0.3672.477>]},
{messages,[]},
{links,[]},
{dictionary,[]},
{trap_exit,true},
{status,running},
{heap_size,610},
{trap_exit,true},
{status,running},
{stack_size,24},
{reductions,973}],
[]]}}
[Thu, 01 May 2014 14:28:04 GMT] [info] [<0.20971.87>] Index shutdown by monitor notice for db: group_5747d16f-4b3b-4522-af10-1dc7d0d644aa idx: _design/filters
[Thu, 01 May 2014 14:28:04 GMT] [info] [<0.4883.35>] Index shutdown by monitor notice for db: group_15ccf331-257d-4b54-b457-997d342816b9 idx: _design/hub
[Thu, 01 May 2014 14:28:04 GMT] [info] [<0.4892.35>] Index shutdown by monitor notice for db: group_15ccf331-257d-4b54-b457-997d342816b9 idx: _design/filters
[Thu, 01 May 2014 14:28:04 GMT] [info] [<0.12040.33>] Index shutdown by monitor notice for db: group_d006a71d-b0de-4d71-b2f7-06abeeb34e00 idx: _design/filters
[Thu, 01 May 2014 14:28:04 GMT] [info] [<0.20971.87>] Closing index for db: group_5747d16f-4b3b-4522-af10-1dc7d0d644aa idx: _design/filters sig: "3e823c2a4383ac0c18d4e574135a5b08"
reason: normal
[Thu, 01 May 2014 14:28:04 GMT] [info] [<0.12032.33>] Index shutdown by monitor notice for db: group_d006a71d-b0de-4d71-b2f7-06abeeb34e00 idx: _design/hub
[Thu, 01 May 2014 14:28:04 GMT] [info] [<0.4892.35>] Closing index for db: group_15ccf331-257d-4b54-b457-997d342816b9 idx: _design/filters sig: "3e823c2a4383ac0c18d4e574135a5b08"
reason: normal
[Thu, 01 May 2014 14:28:04 GMT] [info] [<0.4292.4>] Index shutdown by monitor notice for db: group_ae50933f-de22-4879-9624-b760106060b3 idx: _design/filters
[Thu, 01 May 2014 14:28:04 GMT] [info] [<0.4285.4>] Index shutdown by monitor notice for db: group_ae50933f-de22-4879-9624-b760106060b3 idx: _design/hub
[Thu, 01 May 2014 14:28:04 GMT] [info] [<0.20956.87>] Index shutdown by monitor notice for db: group_5747d16f-4b3b-4522-af10-1dc7d0d644aa idx: _design/hub
[Thu, 01 May 2014 14:28:04 GMT] [info] [<0.4883.35>] Closing index for db: group_15ccf331-257d-4b54-b457-997d342816b9 idx: _design/hub sig: "4f6edcabc4b7a6357b714e1391ed93ac"
reason: normal
[Thu, 01 May 2014 14:28:04 GMT] [info] [<0.12040.33>] Closing index for db: group_d006a71d-b0de-4d71-b2f7-06abeeb34e00 idx: _design/filters sig: "3e823c2a4383ac0c18d4e574135a5b08"
reason: normal
[Thu, 01 May 2014 14:28:04 GMT] [info] [<0.18850.44>] Index shutdown by monitor notice for db: group_721d99a3-2257-48d0-8a1e-89294874d06e idx: _design/filters
[Thu, 01 May 2014 14:28:04 GMT] [info] [<0.18842.44>] Index shutdown by monitor notice for db: group_721d99a3-2257-48d0-8a1e-89294874d06e idx: _design/hub
[Thu, 01 May 2014 14:28:04 GMT] [info] [<0.12032.33>] Closing index for db: group_d006a71d-b0de-4d71-b2f7-06abeeb34e00 idx: _design/hub sig: "4f6edcabc4b7a6357b714e1391ed93ac"
reason: normal
[Thu, 01 May 2014 14:28:04 GMT] [info] [<0.27768.43>] Index shutdown by monitor notice for db: group_56a0df90-c79e-4863-ae71-2bde3cb0d801 idx: _design/hub
[Thu, 01 May 2014 14:28:04 GMT] [info] [<0.27775.43>] Index shutdown by monitor notice for db: group_56a0df90-c79e-4863-ae71-2bde3cb0d801 idx: _design/filters
[Thu, 01 May 2014 14:28:04 GMT] [info] [<0.4292.4>] Closing index for db: group_ae50933f-de22-4879-9624-b760106060b3 idx: _design/filters sig: "3e823c2a4383ac0c18d4e574135a5b08"
reason: normal
[Thu, 01 May 2014 14:28:04 GMT] [info] [<0.4285.4>] Closing index for db: group_ae50933f-de22-4879-9624-b760106060b3 idx: _design/hub sig: "4f6edcabc4b7a6357b714e1391ed93ac"
reason: normal
[Thu, 01 May 2014 14:28:04 GMT] [info] [<0.6010.43>] Index shutdown by monitor notice for db: group_7f082ae6-f41d-4a14-a836-2360303b2e9a idx: _design/filters
[Thu, 01 May 2014 14:28:04 GMT] [info] [<0.6003.43>] Index shutdown by monitor notice for db: group_7f082ae6-f41d-4a14-a836-2360303b2e9a idx: _design/hub
[Thu, 01 May 2014 14:28:04 GMT] [info] [<0.20956.87>] Closing index for db: group_5747d16f-4b3b-4522-af10-1dc7d0d644aa idx: _design/hub sig: "4f6edcabc4b7a6357b714e1391ed93ac"
reason: normal
[Thu, 01 May 2014 14:28:04 GMT] [info] [<0.5933.42>] Index shutdown by monitor notice for db: group_8c49d7e8-b61e-41e5-a220-11df59b9cce4 idx: _design/hub
[Thu, 01 May 2014 14:28:04 GMT] [info] [<0.5940.42>] Index shutdown by monitor notice for db: group_8c49d7e8-b61e-41e5-a220-11df59b9cce4 idx: _design/filters
[Thu, 01 May 2014 14:28:04 GMT] [info] [<0.18842.44>] Closing index for db: group_721d99a3-2257-48d0-8a1e-89294874d06e idx: _design/hub sig: "4f6edcabc4b7a6357b714e1391ed93ac"
reason: normal
[Thu, 01 May 2014 14:28:04 GMT] [info] [<0.17529.33>] Index shutdown by monitor notice for db: group_98ff493c-63e8-4714-9940-ccea514d4b1d idx: _design/hub
[Thu, 01 May 2014 14:28:04 GMT] [info] [<0.18850.44>] Closing index for db: group_721d99a3-2257-48d0-8a1e-89294874d06e idx: _design/filters sig: "3e823c2a4383ac0c18d4e574135a5b08"
reason: normal
[Thu, 01 May 2014 14:28:04 GMT] [info] [<0.17536.33>] Index shutdown by monitor notice for db: group_98ff493c-63e8-4714-9940-ccea514d4b1d idx: _design/filters
[Thu, 01 May 2014 14:28:04 GMT] [info] [<0.27768.43>] Closing index for db: group_56a0df90-c79e-4863-ae71-2bde3cb0d801 idx: _design/hub sig: "4f6edcabc4b7a6357b714e1391ed93ac"
Adam Kocoloski
May 1, 2014, 10:51:49 AM5/1/14
to us...@couchdb.apache.org
Hi Herman, I think those are just the view groups shutting down after the parent DB crashed because you ran out of processes.
You can increase the maximum number of processes via the ERL_FLAGS environment variable, e.g.
> $ ERL_FLAGS="+P 512000" erl
> Erlang R14B01 (erts-5.8.2) [source] [64-bit] [smp:4:4] [rq:4] [async-threads:0] [hipe] [kernel-poll:false]
>
> Eshell V5.8.2 (abort with ^G)
> 1> erlang:system_info(process_limit).
> 512000
The default is 256k, assuming you've got enough RAM you can bump that up to 1M with impunity. Regards,
Adam
You can increase the maximum number of processes via the ERL_FLAGS environment variable, e.g.
> $ ERL_FLAGS="+P 512000" erl
> Erlang R14B01 (erts-5.8.2) [source] [64-bit] [smp:4:4] [rq:4] [async-threads:0] [hipe] [kernel-poll:false]
>
> Eshell V5.8.2 (abort with ^G)
> 1> erlang:system_info(process_limit).
> 512000
The default is 256k, assuming you've got enough RAM you can bump that up to 1M with impunity. Regards,
Adam
Herman Chan
May 1, 2014, 12:24:57 PM5/1/14
to us...@couchdb.apache.org
Thanks Adam,
We just tried that and it seems to hold up. Just wondering if there is some kind of formula on what to set ERL_FLAGS to?
Herman
We just tried that and it seems to hold up. Just wondering if there is some kind of formula on what to set ERL_FLAGS to?
Herman
Adam Kocoloski
May 1, 2014, 2:31:32 PM5/1/14
to us...@couchdb.apache.org
Sure, here are a few rules of thumb:
* 1 process per inbound TCP connection
* 4 processes per open DB (up to [couchdb] max_dbs_open DBs will be kept open simultaneously)
* 3 processes per open view group (I might be off by one or two here)
More Erlang processes require more RAM, so don't go crazy.
Adam
* 1 process per inbound TCP connection
* 4 processes per open DB (up to [couchdb] max_dbs_open DBs will be kept open simultaneously)
* 3 processes per open view group (I might be off by one or two here)
More Erlang processes require more RAM, so don't go crazy.
Adam
Herman Chan
May 1, 2014, 2:47:26 PM5/1/14
to us...@couchdb.apache.org
Thanks Adam,
It make sense now why we crashed, we've set a very high number on max_dbs_open (something like 100000) and with the formula you described, it'll create something like 800,000 processes (we have around 800,000 db on this box), which is higher than what we set on ERL_FLAGS.
Thanks for your help!
Herman
It make sense now why we crashed, we've set a very high number on max_dbs_open (something like 100000) and with the formula you described, it'll create something like 800,000 processes (we have around 800,000 db on this box), which is higher than what we set on ERL_FLAGS.
Thanks for your help!
Herman
Adam Kocoloski
May 1, 2014, 3:56:26 PM5/1/14
to us...@couchdb.apache.org
Great, glad we got that sorted out.
On the topic of max_dbs_open - it can be a funny optimization dance, because the data structures that manage the list of open databases will become less efficient as the number of open databases increases. In particular, it becomes expensive to find the least recently used (LRU) DB in order to close it.
In my experience, increasing max_dbs_open makes a lot of sense if e.g. you have 1000 active databases. By the time you reach 100k databases it's often better to match the max_dbs_open to the number of DBs that are queried over some short timespan, so that it's cheap to find the right one to close. Best,
Adam
On the topic of max_dbs_open - it can be a funny optimization dance, because the data structures that manage the list of open databases will become less efficient as the number of open databases increases. In particular, it becomes expensive to find the least recently used (LRU) DB in order to close it.
In my experience, increasing max_dbs_open makes a lot of sense if e.g. you have 1000 active databases. By the time you reach 100k databases it's often better to match the max_dbs_open to the number of DBs that are queried over some short timespan, so that it's cheap to find the right one to close. Best,
Adam
Reply all
Reply to author
Forward
0 new messages