bareos-dir can't connect to database after several weeks of running

254 views

Skip to first unread message

jan.po...@gmail.com

unread,

Jan 27, 2016, 4:28:50 AM1/27/16

to bareos-users

Hi,
I have a strange problem with bareos-dir (15.2.2-37.1.el7.x86_64) on CentOS 7. I use local postgresql (9.4.5-1PGDG.rhel7.x86_64) for catalog and everything runs fine. But after some time (few weeks), bareos-dir cannot connect to database anymore and all backups start to fail. This can be repaired only by restarting bareos-dir and then everything runs fine again for serveral weeks. And then the process repeats. The postgresql server has not been restarted for several months and it is working normally.

Is there any way to find out what's going on?

Last time it happened, I've added option "-d 100" for debug.
I've got this:

/var/log/messages:
Jan 27 09:49:53 bareos bareos-dir: bareos-dir (50): postgresql.c:665-0 Result status fatal: SELECT PoolId,Name FROM Pool WHERE Name='Incremental'
Jan 27 09:49:56 bareos bareos-dir: bareos-dir (20): ua_status.c:146-0 status:stat dir:
Jan 27 09:49:56 bareos bareos-dir: bareos-dir (100): job.c:1734-0 wstorage=BareosStorage1
Jan 27 09:49:56 bareos bareos-dir: bareos-dir (100): job.c:1743-0 wstore=BareosStorage1 where=Job resource
Jan 27 09:49:56 bareos bareos-dir: bareos-dir (100): ua_output.c:1159-0 complete_jcr open db
Jan 27 09:49:56 bareos bareos-dir: bareos-dir (100): sql_pooling.c:60-0 db_sql_get_non_pooled_connection allocating 1 new non pooled database connection to database bareos, backend type postgresql
Jan 27 09:49:56 bareos bareos-dir: bareos-dir (100): cats_backends.c:63-0 db_init_database: Trying to find mapping of given interfacename postgresql to mapping interfacename dbi, partly_compare = true
Jan 27 09:49:56 bareos bareos-dir: bareos-dir (100): cats_backends.c:63-0 db_init_database: Trying to find mapping of given interfacename postgresql to mapping interfacename mysql, partly_compare = false
Jan 27 09:49:56 bareos bareos-dir: bareos-dir (100): cats_backends.c:63-0 db_init_database: Trying to find mapping of given interfacename postgresql to mapping interfacename postgresql, partly_compare = false
Jan 27 09:49:56 bareos bareos-dir: bareos-dir (100): postgresql.c:1223-0 DB REopen bareos
Jan 27 09:50:43 bareos bareos-dir: bareos-dir (50): postgresql.c:665-0 Result status fatal: INSERT INTO Pool (Name,NumVols,MaxVols,UseOnce,UseCatalog,AcceptAnyVolume,AutoPrune,Recycle,VolRetention,VolUseDuration,MaxVolJobs,MaxVolFiles,MaxVolBytes,PoolType,LabelType,LabelFormat,RecyclePoolId,ScratchPoolId,ActionOnPurge,MinBlocksize,MaxBlocksize) VALUES ('Incremental',0,50,0,1,0,1,1,5184000,0,0,0,5368709120,'Backup',0,'Incremental-',0,0,0,0,0)
Jan 27 09:50:43 bareos bareos-dir: bareos-dir (100): job.c:1734-0 wstorage=BareosStorage1
Jan 27 09:50:43 bareos bareos-dir: bareos-dir (100): job.c:1743-0 wstore=BareosStorage1 where=Job resource
Jan 27 09:50:43 bareos bareos-dir: bareos-dir (100): ua_output.c:1159-0 complete_jcr open db
Jan 27 09:50:43 bareos bareos-dir: bareos-dir (100): sql_pooling.c:60-0 db_sql_get_non_pooled_connection allocating 1 new non pooled database connection to database bareos, backend type postgresql
Jan 27 09:50:43 bareos bareos-dir: bareos-dir (100): cats_backends.c:63-0 db_init_database: Trying to find mapping of given interfacename postgresql to mapping interfacename dbi, partly_compare = true
Jan 27 09:50:43 bareos bareos-dir: bareos-dir (100): cats_backends.c:63-0 db_init_database: Trying to find mapping of given interfacename postgresql to mapping interfacename mysql, partly_compare = false
Jan 27 09:50:43 bareos bareos-dir: bareos-dir (100): cats_backends.c:63-0 db_init_database: Trying to find mapping of given interfacename postgresql to mapping interfacename postgresql, partly_compare = false
Jan 27 09:50:43 bareos bareos-dir: bareos-dir (100): postgresql.c:1223-0 DB REopen bareos
Jan 27 09:51:33 bareos bareos-dir: bareos-dir (50): postgresql.c:665-0 Result status fatal: SELECT PoolId,Name,NumVols,MaxVols,UseOnce,UseCatalog,AcceptAnyVolume,AutoPrune,Recycle,VolRetention,VolUseDuration,MaxVolJobs,MaxVolFiles,MaxVolBytes,PoolType,LabelType,LabelFormat,RecyclePoolId,ScratchPoolId,ActionOnPurge,MinBlocksize,MaxBlocksize FROM Pool WHERE Pool.Name='Incremental'
Jan 27 09:52:23 bareos bareos-dir: bareos-dir (50): postgresql.c:665-0 Result status fatal: SELECT PoolId,Name,NumVols,MaxVols,UseOnce,UseCatalog,AcceptAnyVolume,AutoPrune,Recycle,VolRetention,VolUseDuration,MaxVolJobs,MaxVolFiles,MaxVolBytes,PoolType,LabelType,LabelFormat,RecyclePoolId,ScratchPoolId,ActionOnPurge,MinBlocksize,MaxBlocksize FROM Pool WHERE Pool.Name='Incremental'
Jan 27 09:53:13 bareos bareos-dir: bareos-dir (50): postgresql.c:665-0 Result status fatal: SELECT PoolId,Name,NumVols,MaxVols,UseOnce,UseCatalog,AcceptAnyVolume,AutoPrune,Recycle,VolRetention,VolUseDuration,MaxVolJobs,MaxVolFiles,MaxVolBytes,PoolType,LabelType,LabelFormat,RecyclePoolId,ScratchPoolId,ActionOnPurge,MinBlocksize,MaxBlocksize FROM Pool WHERE Pool.Name='Incremental'
Jan 27 09:54:03 bareos bareos-dir: bareos-dir (50): postgresql.c:665-0 Result status fatal: SELECT PoolId,Name,NumVols,MaxVols,UseOnce,UseCatalog,AcceptAnyVolume,AutoPrune,Recycle,VolRetention,VolUseDuration,MaxVolJobs,MaxVolFiles,MaxVolBytes,PoolType,LabelType,LabelFormat,RecyclePoolId,ScratchPoolId,ActionOnPurge,MinBlocksize,MaxBlocksize FROM Pool WHERE Pool.Name='Incremental'
Jan 27 09:54:53 bareos bareos-dir: bareos-dir (50): postgresql.c:665-0 Result status fatal: SELECT PoolId,Name FROM Pool WHERE Name='Incremental'
Jan 27 09:55:43 bareos bareos-dir: bareos-dir (50): postgresql.c:665-0 Result status fatal: INSERT INTO Pool (Name,NumVols,MaxVols,UseOnce,UseCatalog,AcceptAnyVolume,AutoPrune,Recycle,VolRetention,VolUseDuration,MaxVolJobs,MaxVolFiles,MaxVolBytes,PoolType,LabelType,LabelFormat,RecyclePoolId,ScratchPoolId,ActionOnPurge,MinBlocksize,MaxBlocksize) VALUES ('Incremental',0,50,0,1,0,1,1,5184000,0,0,0,5368709120,'Backup',0,'Incremental-',0,0,0,0,0)
Jan 27 09:55:43 bareos bareos-dir: bsock_tcp.c:422 Write error sending 77 bytes to client:10.0.4.160:9101: ERR=Broken pipe

/var/log/bareos/bareos.log:
27-Jan 10:14 bareos-dir JobId 0: Fatal error: sql_get.c:621 sql_get.c:621 query SELECT PoolId,Name,NumVols,MaxVols,UseOnce,UseCatalog,AcceptAnyVolume,AutoPrune,Recycle,VolRetention,VolUseDuration,MaxVolJobs,MaxVolFiles,MaxVolBytes,PoolTyp
e,LabelType,LabelFormat,RecyclePoolId,ScratchPoolId,ActionOnPurge,MinBlocksize,MaxBlocksize FROM Pool WHERE Pool.Name='Incremental' failed:
no connection to the server

strace:
[pid 13262] nanosleep({5, 0}, NULL) = 0
[pid 13262] nanosleep({5, 0}, <unfinished ...>
[pid 21809] <... futex resumed> ) = -1 ETIMEDOUT (Connection timed out)
[pid 21809] futex(0x6a7ce0, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 21809] futex(0x6a7ca4, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 26337, {1453884936, 188996000}, ffffffff <unfinished ...>
[pid 13262] <... nanosleep resumed> NULL) = 0
[pid 13262] nanosleep({5, 0}, <unfinished ...>
[pid 21808] <... futex resumed> ) = -1 ETIMEDOUT (Connection timed out)
[pid 21808] futex(0x7f2638c7c760, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 21808] futex(0x7f2638c7c724, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 75619, {1453884849, 996788000}, ffffffff <unfinished ...>

Catalog config:
Catalog {
Name = Catalog
dbdriver = "postgresql"
dbname = "bareos"
dbuser = "bareos"
dbpassword = "" # local connection
}

Even bconsole stat dir freezes with DB connection errors.

Thank you for your help.

Jan

Reply all

Reply to author

Forward

0 new messages