Is lsyncd the best tool for my job?

[Giorgio Pintaudi]

Hello!

I am an experimental physicist and I am currently developing the DAQ software for our experiment.

We need to continuously backup our data to a remote server. I was wondering if lsyncd is the right tool for this job.

In the following, I will refer to the remote server to backup TO as "the server" and to the local DAQ PC to backup FROM as the "DAQ PC".

Here are my requirements:

• The data rate is quite low (some MBps tops)
• I need only to backup the data one way: from the DAQ PC to the server and not the other way around, i.e. if I delete a file on the DAQ PC I do NOT want the file to be deleted on the server. The DAQ PC storage is limited so we need to continuously push the data from the DAQ PC to the server for long-term storage.
• The OS on the DAQ PC is CentOS 7. I have root access.
• The OS on the server is Scientific Linux 6.10. I do NOT have root access.
• The server can be accessed by SSH connection
• The rsync version installed on the server is 3.0.6

If you think that a different tool would be more appropriate, please let me know.

Thank you

Giorgio

[Frank Jonsson]

My experience from lsyncd is good but not without challenges, I wanted to make a chain of backups: from the first backup server lsyncd on the backup server pushed a backup to the second backup (geographically apart) server in order to put a minimum strain on the production server. The servers I got delivered were different Linux flavors and the different  version of lsyncd on CentOS 7 and Ubuntu 18.04 coursed different syntax in the config file.

Compared to a rsync daemon combined with cronjob, lsyncd put lesser strain on the server and the sync are more live.

The two can be combinated to avoid incidents with data corruption some where in the backup chain. Depends on how safe.

Remember to setup some check/alerts on log files and have some measures to secure your backup eg. an ordinary rsync.

Permissions may be a challenge 

Cheers

--
You received this message because you are subscribed to the Google Groups "lsyncd" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lsyncd+un...@googlegroups.com.
To post to this group, send email to lsy...@googlegroups.com.
Visit this group at https://groups.google.com/group/lsyncd.
For more options, visit https://groups.google.com/d/optout.

--

Best Regards

Frank Jonsson
Mail: fr...@unity3d.com

http://unity3d.com/
DD:   55°40'48.773"N 12°34'36.898"E

DDS: 55.680215 12.576916

Altitude 5 meters

Unity Technologies ApS
Løvstræde 5, 2nd floor
DK-1152 Copenhagen K

[Giorgio Pintaudi]

Thank you very much for the hints. I will keep them in mind!

On Monday, April 1, 2019 at 3:09:53 PM UTC+9, Frank Jonsson wrote:

My experience from lsyncd is good but not without challenges, I wanted to make a chain of backups: from the first backup server lsyncd on the backup server pushed a backup to the second backup (geographically apart) server in order to put a minimum strain on the production server. The servers I got delivered were different Linux flavors and the different  version of lsyncd on CentOS 7 and Ubuntu 18.04 coursed different syntax in the config file.

Compared to a rsync daemon combined with cronjob, lsyncd put lesser strain on the server and the sync are more live.

The two can be combinated to avoid incidents with data corruption some where in the backup chain. Depends on how safe.

Remember to setup some check/alerts on log files and have some measures to secure your backup eg. an ordinary rsync.

Permissions may be a challenge 

Cheers

On Sat, Mar 30, 2019 at 1:39 PM Giorgio Pintaudi <giorgi...@gmail.com> wrote:

Hello!

I am an experimental physicist and I am currently developing the DAQ software for our experiment.

We need to continuously backup our data to a remote server. I was wondering if lsyncd is the right tool for this job.

In the following, I will refer to the remote server to backup TO as "the server" and to the local DAQ PC to backup FROM as the "DAQ PC".

Here are my requirements:

• The data rate is quite low (some MBps tops)
• I need only to backup the data one way: from the DAQ PC to the server and not the other way around, i.e. if I delete a file on the DAQ PC I do NOT want the file to be deleted on the server. The DAQ PC storage is limited so we need to continuously push the data from the DAQ PC to the server for long-term storage.
• The OS on the DAQ PC is CentOS 7. I have root access.
• The OS on the server is Scientific Linux 6.10. I do NOT have root access.
• The server can be accessed by SSH connection
• The rsync version installed on the server is 3.0.6

If you think that a different tool would be more appropriate, please let me know.

Thank you

Giorgio

--
You received this message because you are subscribed to the Google Groups "lsyncd" group.

To unsubscribe from this group and stop receiving emails from it, send an email to lsy...@googlegroups.com.

To post to this group, send email to lsy...@googlegroups.com.
Visit this group at https://groups.google.com/group/lsyncd.
For more options, visit https://groups.google.com/d/optout.

[Axel Kittenberger]

In my opinion it depends very much on how your data is structured. Are you creating a new file every n minutes? Then it should work fine. Are you continuously changing one big file? Then not so much... or it depends on how effective rsync is to detect and transfer the changed parts. At the end of the day, Lsyncd just a tool that automates the calling of rsync while trying to help it, to be as effective as possible (via the filter lists)  

Kind regards, Axel

On Sat, Mar 30, 2019 at 1:39 PM Giorgio Pintaudi <giorgi...@gmail.com> wrote:

--

You received this message because you are subscribed to the Google Groups "lsyncd" group.

To unsubscribe from this group and stop receiving emails from it, send an email to lsyncd+un...@googlegroups.com.