[DuraSpace JIRA] (FCREPO-994) Purging more than 200 files concurrently depletes the DB Pool and causes inconsistencies.
frank asseg (Created) (DuraSpace JIRA)
-----------------------------------------------------------------------------------------
Key: FCREPO-994
URL: https://jira.duraspace.org/browse/FCREPO-994
Project: Fedora Repository Project
Issue Type: Bug
Components: Fedora
Affects Versions: Fedora 3.4.2
Environment: SMP PREEMPT Sat Aug 6 16:18:35 CEST 2011 x86_64 Intel(R) Core(TM)2 Quad CPU Q9550 @ 2.83GHz GenuineIntel GNU/Linux
java version "1.6.0_26"
Java(TM) SE Runtime Environment (build 1.6.0_26-b03)
Java HotSpot(TM) 64-Bit Server VM (build 20.1-b02, mixed mode)
Reporter: frank asseg
When purging Objects rapidly the DB Thread Pool gets exhausted
first a log message like "Max DB Threads (200) reached..." get printed before
the system then says something along the lines "could'nt delete object
from Database. but this should be ok". After that the system is in an inconsistent state
and e.g. search requests end in a HTTP 500 Server Error.
I will add a test case soon...
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://jira.duraspace.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
frank asseg (Commented) (DuraSpace JIRA)
[ https://jira.duraspace.org/browse/FCREPO-994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=22194#comment-22194 ]
frank asseg commented on FCREPO-994:
------------------------------------
these are fedora's warnings:
INFO: Maximum number of threads (200) created for connector with address null and port 8080
11:06:21.744 [http-8080-10] WARN o.f.server.storage.DefaultDOManager - Object couldn't be removed from registry, but that might be ok; continuing with purge
11:06:21.807 [http-8080-50] WARN o.f.server.storage.DefaultDOManager - Object couldn't be removed from registry, but that might be ok; continuing with purge
11:06:21.807 [http-8080-27] WARN o.f.server.storage.DefaultDOManager - Object couldn't be removed from registry, but that might be ok; continuing with purge
11:06:21.807 [http-8080-166] WARN o.f.server.storage.DefaultDOManager - Object couldn't be removed from registry, but that might be ok; continuing with purge
11:06:21.806 [http-8080-24] WARN o.f.server.storage.DefaultDOManager - Object couldn't be removed from registry, but that might be ok; continuing with purge
11:06:35.557 [http-8080-180] WARN o.f.server.storage.DefaultDOManager - Object couldn't be removed from registry, but that might be ok; continuing with purge
11:06:35.559 [http-8080-180] WARN o.f.server.storage.DefaultDOManager - Object couldn't be removed from FieldSearch index (Error attempting delete of object with pid 'fcrepo994:3161': FATAL: remaining connection slots are reserved for non-replication superuser connections), but that might be ok; continuing with purge
11:06:35.593 [http-8080-180] WARN o.f.server.storage.DefaultDOManager - Object couldn't be removed from registry, but that might be ok; continuing with purge
11:06:35.594 [http-8080-180] WARN o.f.server.storage.DefaultDOManager - Object couldn't be removed from FieldSearch index (Error attempting delete of object with pid 'fcrepo994:3447': FATAL: remaining connection slots are reserved for non-replication superuser connections), but that might be ok; continuing with purge
frank asseg (Updated) (DuraSpace JIRA)
[ https://jira.duraspace.org/browse/FCREPO-994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
frank asseg updated FCREPO-994:
-------------------------------
Attachment: fcrepo-994.tar.gz
DON'T USE ON PRODUCTION -> this will leave the repo in a n inconsitent state.
this maven project is for reproducing FCREPO-994. and has a single class with a main method called FCREPO994Reproducer to run.
you might have to tweak the setting OBJECT_COUNT to actually reproduce the failure depending on machine perfomance.
> Purging more than 200 files concurrently depletes the DB Pool and causes inconsistencies.
> -----------------------------------------------------------------------------------------
>
> Key: FCREPO-994
> URL: https://jira.duraspace.org/browse/FCREPO-994
> Project: Fedora Repository Project
> Issue Type: Bug
> Components: Fedora
> Affects Versions: Fedora 3.4.2
> Environment: SMP PREEMPT Sat Aug 6 16:18:35 CEST 2011 x86_64 Intel(R) Core(TM)2 Quad CPU Q9550 @ 2.83GHz GenuineIntel GNU/Linux
> java version "1.6.0_26"
> Java(TM) SE Runtime Environment (build 1.6.0_26-b03)
> Java HotSpot(TM) 64-Bit Server VM (build 20.1-b02, mixed mode)
> Reporter: frank asseg
> Attachments: fcrepo-994.tar.gz
Benjamin Armintor (Commented) (DuraSpace JIRA)
[ https://jira.duraspace.org/browse/FCREPO-994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=22209#comment-22209 ]
Benjamin Armintor commented on FCREPO-994:
------------------------------------------
Can you post the configuration for your connection pool manager? there is a parameter that indicates the behavior on pool exhaustion- "whenExhaustedAction" - that takes 3 values:
0 - fail
1 - wait for freed resource
2 - grow the pool
The default value is 0; you might try 1 (2 will probably swamp your Db).
> Purging more than 200 files concurrently depletes the DB Pool and causes inconsistencies.
> -----------------------------------------------------------------------------------------
>
> Key: FCREPO-994
> URL: https://jira.duraspace.org/browse/FCREPO-994
> Project: Fedora Repository Project
> Issue Type: Bug
> Components: Fedora
> Affects Versions: Fedora 3.4.2
> Environment: SMP PREEMPT Sat Aug 6 16:18:35 CEST 2011 x86_64 Intel(R) Core(TM)2 Quad CPU Q9550 @ 2.83GHz GenuineIntel GNU/Linux
> java version "1.6.0_26"
> Java(TM) SE Runtime Environment (build 1.6.0_26-b03)
> Java HotSpot(TM) 64-Bit Server VM (build 20.1-b02, mixed mode)
> Reporter: frank asseg
> Attachments: fcrepo-994.tar.gz
frank asseg (Commented) (DuraSpace JIRA)
[ https://jira.duraspace.org/browse/FCREPO-994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=22222#comment-22222 ]
frank asseg commented on FCREPO-994:
------------------------------------
"<param name="whenExhaustedAction" value="1">...</param>"
this is from my fcrepo installation, which is unchanged since installation
> Purging more than 200 files concurrently depletes the DB Pool and causes inconsistencies.
> -----------------------------------------------------------------------------------------
>
> Key: FCREPO-994
> URL: https://jira.duraspace.org/browse/FCREPO-994
> Project: Fedora Repository Project
> Issue Type: Bug
> Components: Fedora
> Affects Versions: Fedora 3.4.2
> Environment: SMP PREEMPT Sat Aug 6 16:18:35 CEST 2011 x86_64 Intel(R) Core(TM)2 Quad CPU Q9550 @ 2.83GHz GenuineIntel GNU/Linux
> java version "1.6.0_26"
> Java(TM) SE Runtime Environment (build 1.6.0_26-b03)
> Java HotSpot(TM) 64-Bit Server VM (build 20.1-b02, mixed mode)
> Reporter: frank asseg
> Attachments: fcrepo-994.tar.gz
Chris Wilper (Commented) (DuraSpace JIRA)
[ https://jira.duraspace.org/browse/FCREPO-994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=22299#comment-22299 ]
Chris Wilper commented on FCREPO-994:
-------------------------------------
Talked about this on Sept 6 2011 call. Agreed to send a 503 up the stack in case of exhaustion.
> Purging more than 200 files concurrently depletes the DB Pool and causes inconsistencies.
> -----------------------------------------------------------------------------------------
>
> Key: FCREPO-994
> URL: https://jira.duraspace.org/browse/FCREPO-994
> Project: Fedora Repository Project
> Issue Type: Bug
> Components: Fedora
> Affects Versions: Fedora 3.4.2
> Environment: SMP PREEMPT Sat Aug 6 16:18:35 CEST 2011 x86_64 Intel(R) Core(TM)2 Quad CPU Q9550 @ 2.83GHz GenuineIntel GNU/Linux
> java version "1.6.0_26"
> Java(TM) SE Runtime Environment (build 1.6.0_26-b03)
> Java HotSpot(TM) 64-Bit Server VM (build 20.1-b02, mixed mode)
> Reporter: frank asseg
> Attachments: fcrepo-994.tar.gz
Benjamin Armintor (Updated) (DuraSpace JIRA)
[ https://jira.duraspace.org/browse/FCREPO-994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Benjamin Armintor updated FCREPO-994:
-------------------------------------
Status: Open (was: Received)
> Purging more than 200 files concurrently depletes the DB Pool and causes inconsistencies.
> -----------------------------------------------------------------------------------------
>
> Key: FCREPO-994
> URL: https://jira.duraspace.org/browse/FCREPO-994
> Project: Fedora Repository Project
> Issue Type: Bug
> Components: Fedora
> Affects Versions: Fedora 3.4.2
> Environment: SMP PREEMPT Sat Aug 6 16:18:35 CEST 2011 x86_64 Intel(R) Core(TM)2 Quad CPU Q9550 @ 2.83GHz GenuineIntel GNU/Linux
> java version "1.6.0_26"
> Java(TM) SE Runtime Environment (build 1.6.0_26-b03)
> Java HotSpot(TM) 64-Bit Server VM (build 20.1-b02, mixed mode)
> Reporter: frank asseg
> Attachments: fcrepo-994.tar.gz