Confusing fatal error: directors line to sd dropped ..

[rivim...@gmail.com]

Folks,

I am repeatedly getting an issue with multiple clients where the backup starts ok and gets to the point of writing to tape and then fails, as shown in the log attached. Snippet below:

16-Feb 18:07 greyarea-bareos-dir JobId 15775: Connected Client: greyarea-dns-fd at greyarea-dns.cam.ivimey.org:9102, encryption: TLS_CHACHA20_POLY1305_SHA256 TLSv1.3

...

16-Feb 18:07 greyarea-dns-fd JobId 15775: Connected Storage daemon at helva.cam.ivimey.org:9103, encryption: TLS_CHACHA20_POLY1305_SHA256 TLSv1.3

...

16-Feb 18:07 helva-sd JobId 15775: Spooling data ...

16-Feb 18:07 greyarea-dns-fd JobId 15775: Fatal error: TLS read/write failure.: ERR=error:0A000126:SSL routines::unexpected eof while reading

16-Feb 18:07 greyarea-bareos-dir JobId 15775: Fatal error: lib/tls_openssl_private.cc:357 TLS read/write failure.: ERR=error:0A000126:SSL routines::unexpected eof while reading 

16-Feb 18:07 greyarea-bareos-dir JobId 15775: Fatal error: Director's comm line to SD dropped.

It has been happening for a while, with multiple clients and jobs, and has me utterly baffled especially as that there are jobs for some clients that successfully complete. Can anyone help?

The director, storage and client are all running apt-installed community builds tagged 24.0.0~pre1546.c16dbcf30-8, which was from 14th Dec 2024. The OS is Ubuntu 24.04 'noble' running on amd64 cpus, and all but 'storage' are qemu VMs.

Possibly relevant is that I also have some clients which the director seems unable to contact, even though 'ss -lt' shows the process is listening, 'tcpdump' shows connection requests reach the client system on the right port, and 'ping' works fine in both directions. an 'strace' of the bareos-fd process shows it is not even attempting to respond to the connection request from the director. This may or may not be related... ^o^.

[Bruno Friedmann (bruno-at-bareos)]

Can be sure at 100% 

but you certainly have to investigate in connection drop, 

Maybe you need to check if you can setup `heartbeat interval` https://docs.bareos.org/TasksAndConcepts/NetworkSetup.html#heartbeat-interval-tcp-keepalive, set client as passive https://docs.bareos.org/TasksAndConcepts/NetworkSetup.html#passive-clients

check any intrusive firewall that might cut idle tcp connection too agressivly.

Hope this help you a bit.

[rivim...@gmail.com]

Bruno, thanks for reply.

I have already checked the firewall - there is nothing set when looking either using iptables nor nftables and all interfaces involved are on the same subnet.

Are you suggesting adding the heartbeat or ensure it is not added... unsure what you mean. I normally don't have it enabled and have not needed it so far.

I can try passive client, certainly. It would be irritating to need it but hey ho.

The SD is running directly on (Debian-12 based) proxmox (to avoid issues getting access to the tape drives), while the director and the client are both in VMs. I am aware proxmox does some network stuff... might this be getting in the way? I don't understand how, if so, but it is the only thing I can think of.

Ruth

[Bruno Friedmann]

On lundi, 17 février 2025 12.23:19 h heure normale d’Europe centrale rivim...@gmail.com wrote:

> Bruno, thanks for reply.

>

> I have already checked the firewall - there is nothing set when looking

> either using iptables nor nftables and all interfaces involved are on the

> same subnet.

>

> Are you suggesting adding the heartbeat or ensure it is not added... unsure

> what you mean. I normally don't have it enabled and have not needed it so

> far.

Adding it, in case it help

 

> I can try passive client, certainly. It would be irritating to need it but

> hey ho.

I don't know what you find irritating using passive = yes will become the default sooner or later.

Maybe you get confused with Client initiated connection, which is completely another topic :-)

 

> The SD is running directly on (Debian-12 based) proxmox (to avoid issues

> getting access to the tape drives), while the director and the client are

> both in VMs. I am aware proxmox does some network stuff... might this be

> getting in the way? I don't understand how, if so, but it is the only thing

> I can think of.

>

> Ruth

>

Ok that's an indication, which might help other to help.

Beware of something nasty debian is often doing

Remove the 127.0.1.1 fdqn line from yours /etc/hosts file, instead adding fixes it might confused some daemon in some cases.

> On Monday, 17 February 2025 at 10:57:22 UTC Bruno Friedmann

>

--

  Bruno Friedmann             

[rivim...@gmail.com]

Yes, I was thinking of passive as being client-initiated.

Ruth