Data integrity

[Neil Fletcher-Baggaley]

Hi Folks,

I have a quick question for you all, How do you handle data integrity with ESOS?, as from what I can see it's RAID stack is mdadm which in my personal opinion performs it's task very well for what it was designed for (RAID stack only) 

Is there anything in ESOS that can check the data integrity of the data on the array at all? 

I am aware that ZFS can do this on the fly, but not really a huge fan of it - again just my personal opinion.

Is there a recommended tool that provides this functionality, or am I just being paranoid about silent data corruption :) 

[Marc Smith]

There is the whole data integrity block layer in Linux which might be
what you're looking for. Most (or all) of that is not enabled in ESOS
currently. If you wanted to investigate this, it might be interesting
to share with the group what you discover, and what we would need to
enable it (make optional) in ESOS.

--Marc


>
> --
> You received this message because you are subscribed to the Google Groups "esos-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to esos-users+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/esos-users/4ec331f0-0345-42a9-ba39-cd2f5357a181n%40googlegroups.com.

[Neil Fletcher-Baggaley]

Thanks for this Marc, It is food for though..

I have been looking on and off to see just how to achieve this, It appears that there is a lvm-integrity command that is part of LVM2 that appears to perform this function. I have never used this, as I usually create am mdadm array, and go from there, but I have found out that lmv2 can also create software array, but is done through lvm2 and does not use mdadm.

I have set up a test box with 4 ssd's in and just created a lmv-raid after creating a partition on each of the drives, and then running the following commands.

1. PV_SIZE=`lsblk |grep sdb |grep disk | awk '{print $4}' `
   
2. VG_NAME="vg1"
   
3. LV_NAME="lvm_raid5"
   
4. sudo vgcreate $VG_NAME /dev/sdb1 /dev/sdc1 /dev/sdd1 /dev/sde1
   
5. sudo lvcreate --type raid5 --raidintegrity y -n $LV_NAME -L $PV_SIZE $VG_NAME

Please note, I did this on a ubuntu 22.04 server, and that was installed on a /dev/sda1 (on LVM too), the other 4 drives were added afterwards, and the array created then.

It then looks like the array does a sync by the looks of things - just like mdadm does.

root@test:~# lvs
  LV        VG        Attr       LSize    Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  ubuntu-lv ubuntu-vg -wi-ao---- <221.57g
  lvm_raid5 vg1       rwi-a-r---  447.10g                                    1.06

Note the 1.06 is the percent of sync progress.

From what I have seen on various pages, lvm-integrity and lvm-raid is quite new, and there are pro's and cons of using such method, I have not had chance to do anything destructive on this box yet to see just how it behaves, but I will keep you all informed.

Does anyone know if ESOS use LVM2? as I may just start on ESOS rather than ubuntu.

[Marc Smith]

On Fri, May 5, 2023 at 11:26 AM 'Neil Fletcher-Baggaley' via
esos-users <esos-...@googlegroups.com> wrote:
>
> Thanks for this Marc, It is food for though..
>
> I have been looking on and off to see just how to achieve this, It appears that there is a lvm-integrity command that is part of LVM2 that appears to perform this function. I have never used this, as I usually create am mdadm array, and go from there, but I have found out that lmv2 can also create software array, but is done through lvm2 and does not use mdadm.
>
> I have set up a test box with 4 ssd's in and just created a lmv-raid after creating a partition on each of the drives, and then running the following commands.
>
> PV_SIZE=`lsblk |grep sdb |grep disk | awk '{print $4}' `
> VG_NAME="vg1"
> LV_NAME="lvm_raid5"
> sudo vgcreate $VG_NAME /dev/sdb1 /dev/sdc1 /dev/sdd1 /dev/sde1
> sudo lvcreate --type raid5 --raidintegrity y -n $LV_NAME -L $PV_SIZE $VG_NAME
>
> Please note, I did this on a ubuntu 22.04 server, and that was installed on a /dev/sda1 (on LVM too), the other 4 drives were added afterwards, and the array created then.
>
> It then looks like the array does a sync by the looks of things - just like mdadm does.
>
> root@test:~# lvs
> LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
> ubuntu-lv ubuntu-vg -wi-ao---- <221.57g
> lvm_raid5 vg1 rwi-a-r--- 447.10g 1.06
>
>
> Note the 1.06 is the percent of sync progress.
>
>
> From what I have seen on various pages, lvm-integrity and lvm-raid is quite new, and there are pro's and cons of using such method, I have not had chance to do anything destructive on this box yet to see just how it behaves, but I will keep you all informed.
>
> Does anyone know if ESOS use LVM2? as I may just start on ESOS rather than ubuntu.

Yes, LVM2.

--Marc

>
> On Friday, 28 April 2023 at 14:08:42 UTC+1 Marc Smith wrote:
>>
>> On Thu, Apr 6, 2023 at 5:01 AM 'Neil Fletcher-Baggaley' via esos-users
>> <esos-...@googlegroups.com> wrote:
>> >
>> > Hi Folks,
>> >
>> > I have a quick question for you all, How do you handle data integrity with ESOS?, as from what I can see it's RAID stack is mdadm which in my personal opinion performs it's task very well for what it was designed for (RAID stack only)
>> >
>> > Is there anything in ESOS that can check the data integrity of the data on the array at all?
>> > I am aware that ZFS can do this on the fly, but not really a huge fan of it - again just my personal opinion.
>> >
>> > Is there a recommended tool that provides this functionality, or am I just being paranoid about silent data corruption :)
>>
>> There is the whole data integrity block layer in Linux which might be
>> what you're looking for. Most (or all) of that is not enabled in ESOS
>> currently. If you wanted to investigate this, it might be interesting
>> to share with the group what you discover, and what we would need to
>> enable it (make optional) in ESOS.
>>
>> --Marc
>>
>>
>> >
>> > --
>> > You received this message because you are subscribed to the Google Groups "esos-users" group.
>> > To unsubscribe from this group and stop receiving emails from it, send an email to esos-users+...@googlegroups.com.
>> > To view this discussion on the web visit https://groups.google.com/d/msgid/esos-users/4ec331f0-0345-42a9-ba39-cd2f5357a181n%40googlegroups.com.
>
> --
> You received this message because you are subscribed to the Google Groups "esos-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to esos-users+...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/esos-users/3e794fe9-a1b0-4570-9603-7458c87e28fbn%40googlegroups.com.