Unable to open CouchDb database

[Tito Ciuro]

Hello,

We’ve had our database working for a few months without an incident. I just started seeing our app fail due to CouchDB. The CouchDB log states the following:

> SyntaxError: JSON.parse: unterminated string
> Stacktrace:
> ()@share/couchdb/server/main.js:1556
> @share/couchdb/server/main.js:1573
> @:0
> Failed to execute script.
> Apache CouchDB 1.6.1 (LogLevel=warning) is starting.
> Apache CouchDB has started. Time to relax.
> [error] [<0.148.0>] Could not open file /Library/Developer/Database/xcs.couch: file already exists
> [Sat, 04 Apr 2015 19:00:37 GMT] [error] [<0.148.0>] Could not open file /Library/Developer/Database/xcs.couch: file already exists
> [error] [emulator] Error in process <0.277.0> with exit value: {function_clause,[{couch_compress,decompress,[<<0 bytes>>],[{file,"/SourceCache/XCSCouchDB/XCSCouchDB-2/dependencies/couchdb/src/couchdb/couch_compress.erl"},{line,67}]},{couch_file,pread_term,2,[{file,"/SourceCache/XCSCo…


Why is it stating "Could not open file <…> file already exists”? If the database exists (expected), why cannot be opened? And why is share/couchdb/server/main.js giving an error? AFAIK, our CouchDB installation hasn’t been touched in months. Any ideas?

Thanks,

— Tito

[Alexander Shorin]

Hi Tito,

It looks like database file is corrupted. When this error happens? On
CouchDB start? On database or it document query? The JavaScript
stacktrace looks as it failed on view or other ddoc function request.
--
,,,^..^,,,

[Tito Ciuro]

Hi Alexander,

This error happens when the database is first opened. Any query that follows returns an error:

> [error] [emulator] Error in process <0.7431.0> with exit value: {function_clause,[{couch_compress,decompress,[<<0 bytes>>],[{file,"/SourceCache/XCSCouchDB/XCSCouchDB-2/dependencies/couchdb/src/couchdb/couch_compress.erl"},{line,67}]},{couch_file,pread_term,2,[{file,"/SourceCache/XCSCo...
>
>
> [Sat, 04 Apr 2015 19:40:54 GMT] [error] [emulator] Error in process <0.7431.0> with exit value: {function_clause,[{couch_compress,decompress,[<<0 bytes>>],[{file,"/SourceCache/XCSCouchDB/XCSCouchDB-2/dependencies/couchdb/src/couchdb/couch_compress.erl"},{line,67}]},{couch_file,pread_term,2,[{file,"/SourceCache/XCSCo...
>
>
>
> =ERROR REPORT==== 4-Apr-2015::12:40:54 ===
> Error in process <0.7431.0> with exit value: {function_clause,[{couch_compress,decompress,[<<0 bytes>>],[{file,"/SourceCache/XCSCouchDB/XCSCouchDB-2/dependencies/couchdb/src/couchdb/couch_compress.erl"},{line,67}]},{couch_file,pread_term,2,[{file,"/SourceCache/XCSCo...
>
> [error] [<0.7364.0>] {error_report,<0.30.0>,
> {<0.7364.0>,crash_report,
> [[{initial_call,
> {mochiweb_acceptor,init,
> ['Argument__1','Argument__2','Argument__3']}},
> {pid,<0.7364.0>},
> {registered_name,[]},
> {error_info,
> {error,badarg,
> [{erlang,list_to_binary,
> [[{couch_compress,decompress,
> [<<>>],

> [{file,
> "/SourceCache/XCSCouchDB/XCSCouchDB-2/dependencies/couchdb/src/couchdb/couch_compress.erl"},
> {line,67}]},
> {couch_file,pread_term,2,
> [{file,

> "/SourceCache/XCSCouchDB/XCSCouchDB-2/dependencies/couchdb/src/couchdb/couch_file.erl"},
> {line,135}]},
> {couch_btree,get_node,2,
> [{file,
> "/SourceCache/XCSCouchDB/XCSCouchDB-2/dependencies/couchdb/src/couchdb/couch_btree.erl"},
> {line,349}]},
> {couch_btree,modify_node,4,
> [{file,
> "/SourceCache/XCSCouchDB/XCSCouchDB-2/dependencies/couchdb/src/couchdb/couch_btree.erl"},
> {line,309}]},
> {couch_btree,query_modify,4,
> [{file,
> "/SourceCache/XCSCouchDB/XCSCouchDB-2/dependencies/couchdb/src/couchdb/couch_btree.erl"},
> {line,196}]},
> {couch_mrview_updater,write_kvs,4,
> [{file,
> "/SourceCache/XCSCouchDB/XCSCouchDB-2/dependencies/couchdb/src/couch_mrview/src/couch_mrview_updater.erl"},
> {line,225}]},
> {couch_mrview_updater,write_results,2,
> [{file,
> "/SourceCache/XCSCouchDB/XCSCouchDB-2/dependencies/couchdb/src/couch_mrview/src/couch_mrview_updater.erl"},
> {line,166}]}]],
> []},
> {couch_httpd,error_info,1,
> [{file,
> "/SourceCache/XCSCouchDB/XCSCouchDB-2/dependencies/couchdb/src/couchdb/couch_httpd.erl"},
> {line,818}]},
> {couch_httpd,send_error,2,
> [{file,
> "/SourceCache/XCSCouchDB/XCSCouchDB-2/dependencies/couchdb/src/couchdb/couch_httpd.erl"},
> {line,925}]},
> {couch_httpd,handle_request_int,5,
> [{file,
> "/SourceCache/XCSCouchDB/XCSCouchDB-2/dependencies/couchdb/src/couchdb/couch_httpd.erl"},
> {line,353}]},
> {mochiweb_http,headers,5,
> [{file,
> "/SourceCache/XCSCouchDB/XCSCouchDB-2/dependencies/couchdb/src/mochiweb/mochiweb_http.erl"},
> {line,94}]},
> {proc_lib,init_p_do_apply,3,
> [{file,"proc_lib.erl"},{line,239}]}]}},
> {ancestors,
> [couch_httpd,couch_secondary_services,
> couch_server_sup,<0.31.0>]},
> {messages,[]},
> {links,[<0.104.0>,#Port<0.7351>]},
> {dictionary,
> [{mochiweb_request_qs,
> [{"include_docs","true"},
> {"startkey","[\"device\"]"},
> {"endkey","[\"device\",{}]"}]},
> {couch_rewrite_count,0},
> {mochiweb_request_cookie,[]}]},
> {trap_exit,false},
> {status,running},
> {heap_size,6772},
> {stack_size,27},
> {reductions,5327}],
> []]}}
> [Sat, 04 Apr 2015 19:40:54 GMT] [error] [<0.7364.0>] {error_report,<0.30.0>,
> {<0.7364.0>,crash_report,
> [[{initial_call,
> {mochiweb_acceptor,init,
> ['Argument__1','Argument__2','Argument__3']}},
> {pid,<0.7364.0>},
> {registered_name,[]},
> {error_info,
> {error,badarg,
> [{erlang,list_to_binary,
> [[{couch_compress,decompress,
> [<<>>],

> [{file,
> "/SourceCache/XCSCouchDB/XCSCouchDB-2/dependencies/couchdb/src/couchdb/couch_compress.erl"},
> {line,67}]},
> {couch_file,pread_term,2,
> [{file,

> "/SourceCache/XCSCouchDB/XCSCouchDB-2/dependencies/couchdb/src/couchdb/couch_file.erl"},
> {line,135}]},
> {couch_btree,get_node,2,
> [{file,
> "/SourceCache/XCSCouchDB/XCSCouchDB-2/dependencies/couchdb/src/couchdb/couch_btree.erl"},
> {line,349}]},
> {couch_btree,modify_node,4,
> [{file,
> "/SourceCache/XCSCouchDB/XCSCouchDB-2/dependencies/couchdb/src/couchdb/couch_btree.erl"},
> {line,309}]},
> {couch_btree,query_modify,4,
> [{file,
> "/SourceCache/XCSCouchDB/XCSCouchDB-2/dependencies/couchdb/src/couchdb/couch_btree.erl"},
> {line,196}]},
> {couch_mrview_updater,write_kvs,4,
> [{file,
> "/SourceCache/XCSCouchDB/XCSCouchDB-2/dependencies/couchdb/src/couch_mrview/src/couch_mrview_updater.erl"},
> {line,225}]},
> {couch_mrview_updater,write_results,2,
> [{file,
> "/SourceCache/XCSCouchDB/XCSCouchDB-2/dependencies/couchdb/src/couch_mrview/src/couch_mrview_updater.erl"},
> {line,166}]}]],
> []},
> {couch_httpd,error_info,1,
> [{file,
> "/SourceCache/XCSCouchDB/XCSCouchDB-2/dependencies/couchdb/src/couchdb/couch_httpd.erl"},
> {line,818}]},
> {couch_httpd,send_error,2,
> [{file,
> "/SourceCache/XCSCouchDB/XCSCouchDB-2/dependencies/couchdb/src/couchdb/couch_httpd.erl"},
> {line,925}]},
> {couch_httpd,handle_request_int,5,
> [{file,
> "/SourceCache/XCSCouchDB/XCSCouchDB-2/dependencies/couchdb/src/couchdb/couch_httpd.erl"},
> {line,353}]},
> {mochiweb_http,headers,5,
> [{file,
> "/SourceCache/XCSCouchDB/XCSCouchDB-2/dependencies/couchdb/src/mochiweb/mochiweb_http.erl"},
> {line,94}]},
> {proc_lib,init_p_do_apply,3,
> [{file,"proc_lib.erl"},{line,239}]}]}},
> {ancestors,
> [couch_httpd,couch_secondary_services,
> couch_server_sup,<0.31.0>]},
> {messages,[]},
> {links,[<0.104.0>,#Port<0.7351>]},
> {dictionary,
> [{mochiweb_request_qs,
> [{"include_docs","true"},
> {"startkey","[\"device\"]"},
> {"endkey","[\"device\",{}]"}]},
> {couch_rewrite_count,0},
> {mochiweb_request_cookie,[]}]},
> {trap_exit,false},
> {status,running},
> {heap_size,6772},
> {stack_size,27},
> {reductions,5327}],
> []]}}
>
> =CRASH REPORT==== 4-Apr-2015::12:40:54 ===
> crasher:
> initial call: mochiweb_acceptor:init/3
> pid: <0.7364.0>
> registered_name: []
> exception error: bad argument
> in function list_to_binary/1
> called as list_to_binary([{couch_compress,decompress,
> [<<>>],

> [{file,
> "/SourceCache/XCSCouchDB/XCSCouchDB-2/dependencies/couchdb/src/couchdb/couch_compress.erl"},
> {line,67}]},
> {couch_file,pread_term,2,
> [{file,

> "/SourceCache/XCSCouchDB/XCSCouchDB-2/dependencies/couchdb/src/couchdb/couch_file.erl"},
> {line,135}]},
> {couch_btree,get_node,2,
> [{file,
> "/SourceCache/XCSCouchDB/XCSCouchDB-2/dependencies/couchdb/src/couchdb/couch_btree.erl"},
> {line,349}]},
> {couch_btree,modify_node,4,
> [{file,
> "/SourceCache/XCSCouchDB/XCSCouchDB-2/dependencies/couchdb/src/couchdb/couch_btree.erl"},
> {line,309}]},
> {couch_btree,query_modify,4,
> [{file,
> "/SourceCache/XCSCouchDB/XCSCouchDB-2/dependencies/couchdb/src/couchdb/couch_btree.erl"},
> {line,196}]},
> {couch_mrview_updater,write_kvs,4,
> [{file,
> "/SourceCache/XCSCouchDB/XCSCouchDB-2/dependencies/couchdb/src/couch_mrview/src/couch_mrview_updater.erl"},
> {line,225}]},
> {couch_mrview_updater,write_results,2,
> [{file,
> "/SourceCache/XCSCouchDB/XCSCouchDB-2/dependencies/couchdb/src/couch_mrview/src/couch_mrview_updater.erl"},
> {line,166}]}])
> in call from couch_httpd:error_info/1 (/SourceCache/XCSCouchDB/XCSCouchDB-2/dependencies/couchdb/src/couchdb/couch_httpd.erl, line 818)
> in call from couch_httpd:send_error/2 (/SourceCache/XCSCouchDB/XCSCouchDB-2/dependencies/couchdb/src/couchdb/couch_httpd.erl, line 925)
> in call from couch_httpd:handle_request_int/5 (/SourceCache/XCSCouchDB/XCSCouchDB-2/dependencies/couchdb/src/couchdb/couch_httpd.erl, line 353)
> in call from mochiweb_http:headers/5 (/SourceCache/XCSCouchDB/XCSCouchDB-2/dependencies/couchdb/src/mochiweb/mochiweb_http.erl, line 94)
> ancestors: [couch_httpd,couch_secondary_services,couch_server_sup,
> <0.31.0>]
> messages: []
> links: [<0.104.0>,#Port<0.7351>]
> dictionary: [{mochiweb_request_qs,
> [{"include_docs","true"},
> {"startkey","[\"device\"]"},
> {"endkey","[\"device\",{}]"}]},
> {couch_rewrite_count,0},
> {mochiweb_request_cookie,[]}]
> trap_exit: false
> status: running
> heap_size: 6772
> stack_size: 27
> reductions: 5327
> neighbours:
> ^C


By the way, Futon > Diagnostics > Verify installation returns "Installation.Your installation looks fine. Time to Relax”. It’s interesting because I can switch through the many views and browse the documents, so if it was really corrupted I’d see some sort of error or timeout trying to read it, no?… :-/

Thanks for the help,

— Tito

[Tito Ciuro]

I wonder if the only alternative is to export all docs and re-import them to a fresh, new db? My worry though is why it got corrupted in the first place. I thought CouchDB was pretty good at avoiding these kind of things. I have verified that the drive is OK, so a priori it seems a database file related issue. I’m not claiming to be an expert or degrading CouchDB! I’m just trying to understand what could have caused this problem.

— Tito

[Tito Ciuro]

Hello again,

After trying to unsuccessfully export the documents, I decided to obliterate the indexes and reindex the entire database. I’ve seen in the past similar issues with other RDBMs where the index (not the database itself) became corrupted. After the indexing completed, the database issue seemed to be fixed; everything returning back to normal. Inspecting the couchdb log revealed that the previous reported errors where now gone. However, one error statement persists:

> [Mon, 06 Apr 2015 23:08:58 GMT] [error] [<0.148.0>] Could not open file /Library/Developer/XcodeServer/Database/xcs.couch: file already exists

> Apache CouchDB 1.6.1 (LogLevel=warning) is starting.
> Apache CouchDB has started. Time to relax.

> [error] [<0.148.0>] Could not open file /Library/Developer/XcodeServer/Database/xcs.couch: file already exists


Is this something I should worry about? The message is a bit ambiguous, making it sound that it could not open the file because it already exists? What does that mean? What is CouchDB trying to tell me here?

Thanks,

— Tito

> On Apr 4, 2015, at 2:39 PM, Tito Ciuro <tci...@mac.com> wrote:
>
> I wonder if the only alternative is to export all docs and re-import them to a fresh, new db? My worry though is why it got corrupted in the first place. I thought CouchDB was pretty good at avoiding these kind of things. I have verified that the drive is OK, so a priori it seems a database file related issue. I’m not claiming to be an expert or degrading CouchDB! I’m just trying to understand what could have caused this problem.
>
> — Tito
>

[Alexander Shorin]

Hi Tito,

Sorry for silence, I had no good idea what you could do in this
situation. But seems you handled by pretty trivial way (:
As for new error, it seems that XCode CI server tries to create a
database again for some unknown reason. Could you try to play with
XCode CI server configuration[1]. For instance, check Cleaning option.

Suddenly, I don't have OSX to be sure that this will work, it's just a
guess which worth to try. Please, share with us what you find! Or may
be someone else experienced with XCode CI server it could help you.

[1]: https://developer.apple.com/library/ios/documentation/IDEs/Conceptual/xcode_guide-continuous_integration/ConfigureBots.html#//apple_ref/doc/uid/TP40013292-CH9-SW1

--
,,,^..^,,,

[Tito Ciuro]

Hi Alexander,

The problem is innocuous: it turns out to be an error that results from attempting to issue a PUT /<database> when that database already exists. Nothing more. So the solution is to obtain the list of databases (GET /_all_dbs) and issue the PUT if the database is not in the list.

Thanks for the help,

— Tito

> On Apr 7, 2015, at 12:44 PM, Alexander Shorin <kxe...@gmail.com> wrote:
>
> Hi Tito,
>
> Sorry for silence, I had no good idea what you could do in this
> situation. But seems you handled by pretty trivial way (:
> As for new error, it seems that XCode CI server tries to create a
> database again for some unknown reason. Could you try to play with
> XCode CI server configuration[1]. For instance, check Cleaning option.
>
> Suddenly, I don't have OSX to be sure that this will work, it's just a
> guess which worth to try. Please, share with us what you find! Or may
> be someone else experienced with XCode CI server it could help you.
>

> [1]: https://developer.apple.com/library/ios/documentation/IDEs/Conceptual/xcode_guide-continuous_integration/ConfigureBots.html#//apple_ref/doc/uid/TP40013292-CH9-SW1 <https://developer.apple.com/library/ios/documentation/IDEs/Conceptual/xcode_guide-continuous_integration/ConfigureBots.html#//apple_ref/doc/uid/TP40013292-CH9-SW1>