8 TB of File removal = 4 hours

[John L. Poole]

I had Moonfire-nvr running on a Raspberry Pi 4 with 8 TB of external storage for video.  My Pi crashed in March and I ended up having to re-install on the SD HC card a new operating system.  Left over from the previous install was the 8 TBs at 100% capacity with old video files.

It took me over 4 hours to delete 4 directories having a combined size of 8 TBs.  Here are two subdirectories I removed, I had allocated 2 TB for each camera.  I used root because my experience has been root can quickly delete whereas lesser empowered accounts can take longer.  Still... 1 hour to delete 2 TBs of approximately 45-50k files??  That's a lot of time!

    root@raspberrypi:/videos# ls peck_east |wc
      48281   48281  820765
    root@raspberrypi:/videos# ls peck_west |wc
      47246   47246  803170
    root@raspberrypi:/videos# time rm -rf peck_east

    real    61m35.872s
    user    0m0.367s
    sys     1m15.157s
    root@raspberrypi:/videos# time rm -rf peck_west

    real    60m31.124s
    user    0m0.267s
    sys     1m13.934s
    root@raspberrypi:/videos#

My hard drive is a 12 TB Western Digital Purple (optimal and recommended for video storage).  I partitioned 8 TB for video storage for 4 cameras. 

Using LVM:

pi@raspberrypi:/usr/local/src/moonfire-nvr $ sudo lvdisplay
  --- Logical volume ---
  LV Path                /dev/vgpurple/cameras
  LV Name                cameras
  VG Name                vgpurple
  LV UUID                swI29O-uPH0-cxJN-KXVt-6oBn-TZdM-1aLgZr
  LV Write Access        read/write
  LV Creation host, time raspberrypi, 2020-06-10 13:43:34 -0700
  LV Status              NOT available
  LV Size                8.00 TiB
  Current LE             2097152
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto

File system ext4:

pi@raspberrypi:/videos $ sudo file -sL /dev/mapper/vgpurple-cameras
/dev/mapper/vgpurple-cameras: Linux rev 1.0 ext4 filesystem data, UUID=7dc1f64f-0c02-43de-9a7d-089dc606099e (needs journal recovery) (errors) (extents) (64bit) (large files) (huge files)
pi@raspberrypi:/videos $

Scott recommend "mkfs.ext4 -T largefile" per https://github.com/scottlamb/moonfire-nvr/wiki/System-setup#sample-directory-filesystems-on-hard-disks  I did not preserve what I used, but the "(large files)" in the file output suggest I followed his suggestion.

In hindsight, I did not need to preserve anything on that partition and it might have just been quicker to re-install ext4 on that partition and saved the wear & tear of the heads and 4 hours.

Thoughts?

John

--

John Laurence Poole
1566 Court ST NE
Salem OR 97301-4241
707-812-1323 office

[Scott Lamb]

On Wed, Aug 11, 2021 at 3:16 PM John L. Poole <jlpo...@gmail.com> wrote:

I had Moonfire-nvr running on a Raspberry Pi 4 with 8 TB of external storage for video.  My Pi crashed in March and I ended up having to re-install on the SD HC card a new operating system.  Left over from the previous install was the 8 TBs at 100% capacity with old video files.

It took me over 4 hours to delete 4 directories having a combined size of 8 TBs.  Here are two subdirectories I removed, I had allocated 2 TB for each camera.  I used root because my experience has been root can quickly delete whereas lesser empowered accounts can take longer.  Still... 1 hour to delete 2 TBs of approximately 45-50k files??  That's a lot of time!

    root@raspberrypi:/videos# ls peck_east |wc
      48281   48281  820765
    root@raspberrypi:/videos# ls peck_west |wc
      47246   47246  803170
    root@raspberrypi:/videos# time rm -rf peck_east

    real    61m35.872s
    user    0m0.367s
    sys     1m15.157s
    root@raspberrypi:/videos# time rm -rf peck_west

    real    60m31.124s
    user    0m0.267s
    sys     1m13.934s
    root@raspberrypi:/videos#

Huh. Yeah, that feels slow to me too but I've never tried this before. I think for each file it's touching the inode (somewhat spread out on the disk), the directory entry (which should all be together), and something that says what blocks are in use (maybe also spread out). If it flushes in big batchs, it should write occasional journal entries (insignificant) and do maybe 2 seeks per file. Totally random seeks take ~10 ms. So I'd expect it to take maybe 1,000 seconds (~20 minutes), or faster if it's smart enough to arrange into a more sequential access pattern.

I just tried creating 50,000 empty files and deleting them. Deletion was fast. So the size of the file is probably a factor, although I wouldn't expect it to have to touch all the bytes of the file.

[slamb@slamb-workstation /media/8tb/stuff]$ mkdir files
[slamb@slamb-workstation /media/8tb/stuff]$ cd files
[slamb@slamb-workstation /media/8tb/stuff/files]$ seq 1 50000 | xargs touch
[slamb@slamb-workstation /media/8tb/stuff/files]$ cd ..
[slamb@slamb-workstation /media/8tb/stuff]$ echo 2 | sudo tee /proc/sys/vm/drop_caches
2
[slamb@slamb-workstation /media/8tb/stuff]$ time rm -r files

real    0m0.306s
user    0m0.016s
sys     0m0.262s

I wonder if it would have been faster to make many unlink calls in parallel, maybe via: find . -type f -print0 | xargs -P64 -0 rm

I might experiment with this more sometime (with realistically sized files), as I expect reducing a camera's capacity below current usage in "moonfire-nvr config" would have similar performance. It'd be useful to speed that up if it's also quite slow.

My hard drive is a 12 TB Western Digital Purple (optimal and recommended for video storage).  I partitioned 8 TB for video storage for 4 cameras. 

Using LVM:

pi@raspberrypi:/usr/local/src/moonfire-nvr $ sudo lvdisplay
  --- Logical volume ---
  LV Path                /dev/vgpurple/cameras
  LV Name                cameras
  VG Name                vgpurple
  LV UUID                swI29O-uPH0-cxJN-KXVt-6oBn-TZdM-1aLgZr
  LV Write Access        read/write
  LV Creation host, time raspberrypi, 2020-06-10 13:43:34 -0700
  LV Status              NOT available
  LV Size                8.00 TiB
  Current LE             2097152
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto

File system ext4:

pi@raspberrypi:/videos $ sudo file -sL /dev/mapper/vgpurple-cameras
/dev/mapper/vgpurple-cameras: Linux rev 1.0 ext4 filesystem data, UUID=7dc1f64f-0c02-43de-9a7d-089dc606099e (needs journal recovery) (errors) (extents) (64bit) (large files) (huge files)
pi@raspberrypi:/videos $

Scott recommend "mkfs.ext4 -T largefile" per https://github.com/scottlamb/moonfire-nvr/wiki/System-setup#sample-directory-filesystems-on-hard-disks  I did not preserve what I used, but the "(large files)" in the file output suggest I followed his suggestion.

I don't think I'd made that suggestion yet, so you probably didn't.

"(large files)" is (confusingly) describing something different: the ability to store files larger than 2 GiB. I imagine it's been the default for 20+ years.

The "-T largefile" changes the ratio of inodes to blocks (by referencing an entry in /etc/e2fs.conf). It shouldn't affect how quickly files are deleted; it just preserves more of the space for actual data at the expense of a lower maximum number of files. You could tell if you'd done it by comparing the "Inode count" with "Block count" in dumpe2fs output.

In hindsight, I did not need to preserve anything on that partition and it might have just been quicker to re-install ext4 on that partition and saved the wear & tear of the heads and 4 hours.

Yeah, I'd expect that to be very quick. At least it always has been for me.

 

Thoughts?

John

--

John Laurence Poole
1566 Court ST NE
Salem OR 97301-4241
707-812-1323 office

--
You received this message because you are subscribed to the Google Groups "moonfire-nvr-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to moonfire-nvr-us...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/moonfire-nvr-users/9112f0f8-7410-49d6-b59a-022e105b177f%40gmail.com.

--

Scott Lamb <https://www.slamb.org/>