Fatal error: No Job status returned from FD

[Jason Bailey]

Hi all,

I have a few servers whose backup jobs aren't completing, and I haven't the slightest idea why. Basically, the backup job begins, but immediately goes nowhere. The rest of my servers are backed up correctly.

My Bareos director/storage server (CentOS 7) initiates the backup of several related CentOS 6.6 servers. BAT shows:

bareos-dir Using Device "Device::server3-Metadata" to write.
Start Backup JobId 20274, Job=Backup::server3-Metadata.2015-04-01_15.00.00_32
bareos-sd Ready to append to end of Volume "Server_server3-Metadata_1444" size=322
Volume "Server_server3-Metadata_1444" previously written, moving to end of data.

It'll stay that way before the job finally fails with the following:

Fatal error: No Job status returned from FD.
 Job Backup::server3-Metadata.2015-04-01_14.00.00_21 waiting 1800 seconds for scheduled start time.
bareos-sd Fatal error: append.c:156 Error reading data header from FD. ERR=No data available
bareos-dir Fatal error: Network error with FD during Backup: ERR=No data available
bareos-sd Elapsed time=01:35:40, Transfer rate=0  Bytes/second
bareos-dir Rescheduled Job Backup::server3-Metadata.2015-04-01_14.00.00_21 at 01-Apr-2015 17:07 to re-run in 1800 seconds (01-Apr-2015 17:37).
 AfterJob: sent: 1; skipped: 0; total: 1
 AfterJob: info from server: "processed: 1; failed: 0; total: 1; seconds spent: 0.000069"

I have verified the network connection is solid, and that firewalls aren't the issue (they've been shut off completely while troubleshooting this issue). In fact, the file daemon on the CentOS 6.6 server I'm trying to back up is running, and is listening on port 9102.

I configured the CentOS 6.6 servers (server3, etc) to log with verbose logging, and this is what I see when the backup starts:

server3: cram-md5.c:68-0 send: auth cram-md5 <1740083343.1428050470@server3> ssl=0
server3: cram-md5.c:143-0 sending resp to challenge: q15S568wOT+pM5QSa//2sA
server3: dir_cmd.c:1430-0 set sd auth key
server3: dir_cmd.c:1265-0 level_cmd: level = full  mtime_only=0
server3: cram-md5.c:143-0 sending resp to challenge: TC+zahNH74/kuX/O+6+aaA
server3: cram-md5.c:75-0 send: auth cram-md5 <816177969.1428050470@server3> ssl=0
server3: cram-md5.c:94-0 Authenticate OK kGxv64+jMD+u29o73goI8B
server3: find.c:161-0 Verify=<V> Accurate=<Cmcs> BaseJob=<Jspug5> flags=<1006639904>
server3: cram-md5.c:68-0 send: auth cram-md5 <1286187644.1428071256@server3> ssl=0
server3: cram-md5.c:143-0 sending resp to challenge: Q9AR08JAFw+Go69Aa//g4B
server3: dir_cmd.c:1430-0 set sd auth key
server3: cram-md5.c:68-0 send: auth cram-md5 <120451322.1428071286@server3> ssl=0
server3: cram-md5.c:143-0 sending resp to challenge: m9+zRntS099xS+xKVF+idA
server3: dir_cmd.c:1430-0 set sd auth key
server3: cram-md5.c:68-0 send: auth cram-md5 <1083732140.1428071316@server3> ssl=0
server3: cram-md5.c:143-0 sending resp to challenge: 42FQK//Kq5s+cEBZQVwSBA
server3: dir_cmd.c:1430-0 set sd auth key
server3: cram-md5.c:68-0 send: auth cram-md5 <1198484833.1428071346@server3> ssl=0
server3: cram-md5.c:143-0 sending resp to challenge: Fh4iaXliB7llBUMu9A+pxB
server3: dir_cmd.c:1430-0 set sd auth key
server3: cram-md5.c:68-0 send: auth cram-md5 <909821655.1428071376@server3> ssl=0
server3: cram-md5.c:143-0 sending resp to challenge: G+/wN5/b8/YwYW/aw9McJA
server3: dir_cmd.c:1430-0 set sd auth key
server3: cram-md5.c:68-0 send: auth cram-md5 <876879294.1428071406@server3> ssl=0
server3: cram-md5.c:143-0 sending resp to challenge: K9UPck//d/+VO4//V7h9dA
server3: dir_cmd.c:1430-0 set sd auth key
server3: cram-md5.c:68-0 send: auth cram-md5 <1165299654.1428071436@server3> ssl=0
server3: cram-md5.c:143-0 sending resp to challenge: HHB9u++h7E5auQ+mS6/K5D
server3: dir_cmd.c:1430-0 set sd auth key
server3: cram-md5.c:68-0 send: auth cram-md5 <894739885.1428071466@server3> ssl=0
server3: cram-md5.c:143-0 sending resp to challenge: H/IWS8EB3i/dUC+Jx/FoLA
server3: dir_cmd.c:1430-0 set sd auth key
server3: cram-md5.c:68-0 send: auth cram-md5 <1627460710.1428071496@server3> ssl=0
server3: cram-md5.c:143-0 sending resp to challenge: c4+dd9+So65orW/aFw+vcA
server3: dir_cmd.c:1430-0 set sd auth key
server3: cram-md5.c:68-0 send: auth cram-md5 <336370686.1428071526@server3> ssl=0
server3: cram-md5.c:143-0 sending resp to challenge: e+/D1XpBj5+D2m+lz7+JYD
server3: dir_cmd.c:1430-0 set sd auth key
server3: cram-md5.c:68-0 send: auth cram-md5 <1612948752.1428071556@server3> ssl=0
server3: cram-md5.c:143-0 sending resp to challenge: Y+/kMEpZd3/iP6VppiobfD
server3: dir_cmd.c:1430-0 set sd auth key
server3: cram-md5.c:68-0 send: auth cram-md5 <588451083.1428071586@server3> ssl=0
server3: cram-md5.c:143-0 sending resp to challenge: GBcgs+/fa5+jDDV1M3Y2yB
server3: dir_cmd.c:1430-0 set sd auth key
server3: cram-md5.c:68-0 send: auth cram-md5 <387210219.1428071616@server3> ssl=0
server3: cram-md5.c:143-0 sending resp to challenge: 29+k18ct9+gQQE/C9X/VcB
server3: dir_cmd.c:1430-0 set sd auth key
server3: cram-md5.c:68-0 send: auth cram-md5 <1309805109.1428071646@server3> ssl=0
server3: cram-md5.c:143-0 sending resp to challenge: 44+L0U/OT9/jqF0/U7s/PC
server3: dir_cmd.c:1430-0 set sd auth key
server3: cram-md5.c:68-0 send: auth cram-md5 <169372452.1428071676@server3> ssl=0
server3: cram-md5.c:143-0 sending resp to challenge: A+/xM6+8L5oNzCgoz/I9XA
server3: dir_cmd.c:1430-0 set sd auth key
server3: cram-md5.c:68-0 send: auth cram-md5 <703272097.1428071706@server3> ssl=0
server3: cram-md5.c:143-0 sending resp to challenge: w0+1r4+cg/MY30+rQ/QaRB
server3: dir_cmd.c:1430-0 set sd auth key
server3: cram-md5.c:68-0 send: auth cram-md5 <1606619603.1428071736@server3> ssl=0
server3: cram-md5.c:143-0 sending resp to challenge: u3UCvB/gQh+a4mQApiwqpC
server3: dir_cmd.c:1430-0 set sd auth key
server3: cram-md5.c:68-0 send: auth cram-md5 <1413810102.1428071766@server3> ssl=0
server3: cram-md5.c:143-0 sending resp to challenge: Vh/hLT+PT9Q+g6+v0+dnPB
server3: dir_cmd.c:1430-0 set sd auth key
server3: cram-md5.c:68-0 send: auth cram-md5 <89579650.1428071796@server3> ssl=0
server3: cram-md5.c:143-0 sending resp to challenge: cl/yzzA/vAsqq1lUvElNAA
server3: dir_cmd.c:1430-0 set sd auth key
server3: cram-md5.c:68-0 send: auth cram-md5 <149501157.1428071826@server3> ssl=0
server3: cram-md5.c:143-0 sending resp to challenge: 7F/Ou4+gl7/TbC+amx/4lB
server3: dir_cmd.c:1430-0 set sd auth key
server3: cram-md5.c:68-0 send: auth cram-md5 <998289109.1428071856@server3> ssl=0
server3: cram-md5.c:143-0 sending resp to challenge: 7npUo+8ik7RtI5Uff493JC
server3: dir_cmd.c:1430-0 set sd auth key
server3: cram-md5.c:68-0 send: auth cram-md5 <768486526.1428071886@server3> ssl=0
server3: cram-md5.c:143-0 sending resp to challenge: Y7dMlkNsJ7+ukjga87AvgC
server3: dir_cmd.c:1430-0 set sd auth key
server3: cram-md5.c:68-0 send: auth cram-md5 <405127777.1428071916@server3> ssl=0
server3: cram-md5.c:143-0 sending resp to challenge: cX+Oa5kNG5+EdE/jf7/ljA
server3: dir_cmd.c:1430-0 set sd auth key
server3: cram-md5.c:68-0 send: auth cram-md5 <82136936.1428071946@server3> ssl=0
server3: cram-md5.c:143-0 sending resp to challenge: SB+yd7+gmSAqX5Usx6/vYC
server3: dir_cmd.c:1430-0 set sd auth key
server3: cram-md5.c:68-0 send: auth cram-md5 <467987065.1428071976@server3> ssl=0
server3: cram-md5.c:143-0 sending resp to challenge: /8+rr9+sN3/zFW//tH/OCA
server3: dir_cmd.c:1430-0 set sd auth key
server3: cram-md5.c:68-0 send: auth cram-md5 <492892072.1428072006@server3> ssl=0
server3: cram-md5.c:143-0 sending resp to challenge: D8+2K9R768Rrtl+955AKgB
server3: dir_cmd.c:1430-0 set sd auth key

I've verified the authorization key is correct and verified DNS is configured correctly (the backup server is reachable). I'm not sure what else to try.

Bareos server is version 13.2.2. The server(s) I'm trying to back up are running Bareos file daemon version 14.2.2.

Thanks in advance for any help you can provide!

Jason

[Philipp Storz]

Hello Jason,

I would suggest you run the file daemons that do not do what you expect in debug mode;
the output will hopefully show you what the problem is.

(e.g. something like bareos-fd -d 200 -f)


Are you using accurate mode? If yes, please try to deactivate it as it might need too much memory.


best regards,

Philipp

> --
> You received this message because you are subscribed to the Google Groups "bareos-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to
> bareos-users...@googlegroups.com <mailto:bareos-users...@googlegroups.com>.
> To post to this group, send email to bareos...@googlegroups.com
> <mailto:bareos...@googlegroups.com>.
> For more options, visit https://groups.google.com/d/optout.


--
Mit freundlichen Grüßen

Philipp Storz philip...@bareos.com
Bareos GmbH & Co. KG Phone: +49 221 63 06 93-92
http://www.bareos.com Fax: +49 221 63 06 93-10

Sitz der Gesellschaft: Köln | Amtsgericht Köln: HRA 29646
Geschäftsführer: Stephan Dühr, M. Außendorf,
J. Steffens, P. Storz, M. v. Wieringen