Unable to connect to Storage daemon on host:9103. ERR=Connection refused

Dat Nguyen

unread,

Apr 9, 2019, 4:37:49 AM4/9/19

to bareos-users

Hello!

I have updated Bareos to version 18.2.
Everything works, but for time reasons I deactivated the new TLS encryption from the beginning. This will be a topic for the next months.

We have over 130 clients in Bareos, and everyone can connect to Bareos-Director and Storage-Daemon. Except one.

The error message of the web interface is as follows:

JobId 199189: Fatal error: lib/bsock_tcp.cc:139 Unable to connect to Storage daemon on *STORAGE_DAEMON*:9103. ERR=Connection refused
JobId 199189: Fatal error: Failed to connect to Storage daemon: *STORAGE_DAEMON*:9103
JobId 199189: Fatal error: Bad response to Storage command: wanted 2000 OK storage
, got 2902 Bad storage
JobId 199189: Fatal error: lib/bsock_tcp.cc:139 Unable to connect to Storage daemon on *STORAGE_DAEMON*:9103. ERR=Connection refused

I used ping to test general connection, it works.

I used telnet for testing connection from client to server >>telnet 192.168.72.24 9103<<, it works.

Telnet from server to client on same port >>telnet 192.168.72.24 9103<<, Connection refused. But it shouldn't be important anyway. With the other servers it's the same game and it works there.

I flew through the logs from bareos-server.

>>tail -f /var/log/bareos/bareos.log<<

Unfortunately only the same messages as on the webui.

So I don't have enough information.

To get more information I start the Director, as well as the Storage-Daemon and the File-Daemon in Debug-Mode.

On Server:

>>/usr/sbin/bareos-sd -f -d 1000<<

>>/usr/sbin/bareos-dir-f -d 1000<<

On Client:

>>/usr/sbin/bareos-fd -f -d 1000<<

Then i rerun the backup job via webui.
I will add log snippets as files

I didn't add the director log, unfortunately it ran too fast and the information didn't seem to be relevant for this case.

Despite all that information, I have no idea what the problem is.

I had already had problems with this client on version 17.2, because of the size of the pidfile that is transferred. I had never been able to solve this problem.

Would someone be so kind as to help me out?
If more information is needed for this interesting case, then I will submit it later.

bareos-sd_debuglog

bareos-fd_debuglog

Erik Šolc

unread,

Apr 9, 2019, 4:43:09 AM4/9/19

to bareos-users

Hi Dat,
first step comes to my mind, did you check configuration? Is the password correct in all necessary configuration files?

Erik

Erik Šolc

unread,

Apr 9, 2019, 4:44:14 AM4/9/19

to bareos-users

And also check permissions on these files on client.

Dat Nguyen

unread,

Apr 9, 2019, 4:57:14 AM4/9/19

to bareos-users

Hi, i rechecked all configurations password is correct in all necessary configurations. It worked some months before too, i didn't change the configuration after all.

I will add a snippet of director and filedaemon configuration.

There is one more thing.
I have several jobs running for this client, none of these jobs works anymore, they always return the same error message.

bareos-client_director

Dat Nguyen

unread,

Apr 9, 2019, 4:58:05 AM4/9/19

to bareos-users

I did it the hard way and give all config files 777 and owner bareos:bareos, so this shouldn't be the problem i guess

Erik Šolc

unread,

Apr 9, 2019, 7:00:06 AM4/9/19

to bareos-users

>
> I did it the hard way and give all config files 777 and owner bareos:bareos, so this shouldn't be the problem i guess

So the versions are the same? 18.2 also on client?
what does director shows in bconsole when you run status client=your_client ?

Dat Nguyen

unread,

Apr 9, 2019, 7:59:51 AM4/9/19

to bareos-users

Hi, server and client are on version 18.2.

Bconsole with status client=my_client:

*status client=*CLIENT*.local
Connecting to Client *CLIENT*.local at 10.100.0.73:9102
Handshake: Cleartext, Encryption: None

*CLIENT*.local Version: 18.2.5 (30 January 2019) Linux-4.4.92-6.18-default redhat Red Hat Enterprise Linux Server release 7.0 (Maipo)
Daemon started 09-Apr-19 10:46. Jobs: run=0 running=0, bareos.org build binary
Heap: heap=135,168 smbytes=22,940 max_bytes=23,329 bufs=65 max_bufs=68
Sizeof: boffset_t=8 size_t=8 debug=0 trace=0 bwlimit=0kB/s

Running Jobs:
bareos-dir (director) connected at: 09-Apr-19 13:54
No Jobs running.
====

Terminated Jobs:
JobId Level Files Bytes Status Finished Name
======================================================================
174061 Incr 686 12.37 G OK 11-Oct-18 21:21 *CLIENT*
174127 Incr 0 0 OK 11-Oct-18 22:03 *CLIENT*-database
174233 Incr 723 13.84 G OK 12-Oct-18 21:57 *CLIENT*
174299 Incr 0 0 OK 13-Oct-18 00:12 *CLIENT*-database
174405 Incr 666 11.43 G OK 13-Oct-18 21:07 *CLIENT*
174471 Full 0 0 OK 13-Oct-18 23:38 *CLIENT*-database
174577 Incr 746 13.97 G OK 14-Oct-18 22:01 *CLIENT*
174643 Incr 0 0 OK 15-Oct-18 00:22 *CLIENT*-database
174841 Full 66,868 58.53 G OK 16-Oct-18 06:32 *CLIENT*
174842 Incr 817 19.39 G OK 16-Oct-18 07:42 *CLIENT*
====

As we can see, there were jobs that went well. a while a go...
Never changed config files.
Communication between SD and DIR , DIR and FD is good.
Is it possible to test communication between FD and SD?

Erik Šolc

unread,

Apr 9, 2019, 8:16:50 AM4/9/19

to bareos-users

One more thing, I dont see any plugin info (python-fd.so). Could you try to install package bareos-vmware-plugin-compat?
In documentation "Since Bareos Version >= 17.2.4 the module_path is without vmware_plugin directory."
Nothing else comes to my mind.

Erik

Dat Nguyen

unread,

Apr 9, 2019, 8:32:48 AM4/9/19

to bareos-users

I installed bareos-vmware-plugin-compat restarted DIR and rerun the job.

Bareos Log:

09-Apr 10:39 bareosdir JobId 199361: Fatal error: Socket error on Storage command: ERR=No data available
09-Apr 10:39 bareosdir JobId 199361: Fatal error: Network error with FD during Backup: ERR=No data available
09-Apr 10:39 bareosdir JobId 199361: Fatal error: No Job status returned from FD.

Socket error on Storage is new to me

Erik Šolc

unread,

Apr 9, 2019, 8:40:14 AM4/9/19

to bareos-users

> I installed bareos-vmware-plugin-compat restarted DIR and rerun the job.
>
> Bareos Log:
>
> 09-Apr 10:39 bareosdir JobId 199361: Fatal error: Socket error on Storage command: ERR=No data available
> 09-Apr 10:39 bareosdir JobId 199361: Fatal error: Network error with FD during Backup: ERR=No data available
> 09-Apr 10:39 bareosdir JobId 199361: Fatal error: No Job status returned from FD.
>
>
> Socket error on Storage is new to me

1. Restart also bareos-fd on director.
2. Install compat plugin on client and restart.
3. If not, add two lines in director in /etc/bareos/bareos-fd.d/client/myself.conf
Plugin Directory = /usr/lib64/bareos/plugins
Plugin Names = python
4. Try again telnet and check in bconsole status client

This is configuration for vmware plugin, but it may help to fd plugin.

Dat Nguyen

unread,

Apr 9, 2019, 8:48:24 AM4/9/19

to bareos-users

I did install the plugin on client server and activated plugin directory and names in config.
Error is still there Job canceled in error.

I have the feeling that this has something to do with the socket of SD

Dat Nguyen

unread,

Apr 10, 2019, 3:39:06 AM4/10/19

to bareos-users

I GOT IT!

This is not a misconfiguration, but the maximum size of the spooling data is too small for the storage, so it issued an error message.
The log didn't give me any clues as I said, but I worked out this configuration point by opening it up.

Spooling the data is advantageous, because the data is stored directly on the disk, therefore the Filedaemon has less load, the actual backup into the database happens then only on the server itself.

Erik Šolc

unread,

Apr 10, 2019, 3:59:42 AM4/10/19

to bareos-users

Good job!

Stefano Guidobaldi

unread,

Apr 10, 2019, 11:01:39 AM4/10/19

to bareos-users

Hi Dat,

I'd really appreciate if you could post the configuration you changed to make this work. I just opened a similar issue (though I'm trying to send data to S3 instead of local volumes) but I didn't exactly get where you configured different pool size.
Thank you,
Stefano

Dat Nguyen

unread,

May 9, 2019, 4:44:03 AM5/9/19

to bareos-users

Sorry! Late answere...

The config was /etc/bareos/bareos-sd.d/device/FileStorage.conf

had to change "Maximum Spool Size" from 1024GB to 2048GB for the device i needed

Yeah, we got a lot of data.

Reply all

Reply to author

Forward