Qnap Duplicate File Finder

0 views

Skip to first unread message

Francisca Noggles

unread,

Aug 4, 2024, 5:33:07 PM8/4/24

to spywazsato

Theres quite of number of freeware duplicate file finder programs available. I used one that would find duplicates based on metadata but I've moved to a new computer and can't remember the name (sorry). With a single click you can select the duplicates and then delete them.

Greetings from France all.

I bought a second-hand QNAP TS-253A (8GB RAM,Intel Celeron CPU N3160 @ 1.60GHz; 2 bays, USB3 ports) which is compatible with Linux NAS distributions.

Initially, my plan was to go with OpenMediaVault 6 but the encryption plugin is not yet ported 6 - I found it after configuring!

Disk encryption is a requirement for me - for privacy reasons (if your disk needs to be replaced, don't you mind someone potentially reading its content ?)

Therefore, I decided to give Unraid a try - with release 6.10.1.

It was not easy but in the end it works.

1. I tried to use the USB Creator with 2 different USB2 keys but it failed.

The program displayed an error - and even some weird messages in Chinese or other foreign language.

I tried a 3rd USB2 key (ENUODA brand) - I used just once. This time the USB Creator was successful.

2. I installed Unraid and tried to configure it. As a new user, I had to read forum posts, to

understand what is required for a mirroring/RAID1 configuration (1 disk + 1 parity drives).

The array creation took more than 11 hours (6GB drives). I found it hard to understand

how encryption can be configured. Based on forum reading, I understood that the whole

disk has to be encrypted - contrary to classical Linux which is partition based, and Synology/QNAP which supports volume based encryption. I decided to go with xfs-encryption based on

forums suggestions. I was really happy with Brtfs on Fedora and Synology but looking further after forum readings I noticed a Phoronix article which scared me a bit : =news_item&px=Linux-5.16-Better-Btrfs

errors. I tried to check the content of /boot/config and this directory was empty. It's weird as this USB stick was close to brand-new and running badblocks -w -s -o /tmp/usbstick.log I saw no errors in the generated log.

4. I decided to use a 4th USB stick - SanDisk Ultra 64 GB Dual Drive. Windows refused to format it as FAT so I had to format it on Linux - I created a 2GB partition. This time I used 6.10.2 which got released on that day and I copied the content of the zip file - manual USB disk set-up.

5. I booted the new USB stick and everything went well. I had to do the Array configuration again (11 hours, 30 minutes again), and format the disk so I can create Shared - it took me some time to figure out - this was missing. THis time I backuped the configuration quickly just in case I face some new issues - I don't want to wait for close to 12 hours for the array creation.

To summarize:

- the USB Creator really needs some fixing making it easier to install the set-up program on a USB drive. I'm pretty sure many people will not give Unraid a try because of that.

- mirroring configuration required having a look at forum post. Mirroring or even any configuration could be simplified by an assistant, asking the number of disks, the need for redundancy or not and explaining what is to be done.

- disk encryption configuration is a bit limited and confusing at first (I was expecting the passphrase on the Disk settings). I really hope this is something that will be improved in the future. I still wonder if some people go with brtfs-encrypted which was my initial idea - but as the kernel is still 5.15 based it might not be a good idea

- I really wonder if there is not some weird thing going when using USB2 sticks on USB3 ports

I'm using a 64 USB3 key for the time being - a bit overkill but my 3 USB2 sticks were not ok for Unraid USB disk creation or usage.

- The web interface is fine but could be improved (modernized look, it is not as polished as Synology, and help pointers could be added on each menu parts to clarify further)

I will continue to get familiar with Unraid (disk tuning - I've a feeling the disks don't spin down after 1 hours) and play with the Apps.

- I really wonder if there is not some weird thing going when using USB2 sticks on USB3 ports

I'm using a 64 USB3 key for the time being - a bit overkill but my 3 USB2 sticks were not ok for Unraid USB disk creation or usage.

Issues with booting from usb3 ports has been a somewhat common issue in the past. It was never clear to me if it was due to linux support for booting from usb3 controllers or if it was issues with some of the earlier usb3 controller hardware. It has been less of an issue recently but still crops up from time to time. Not sure if the qnap platform has issues with this. The recommendation for unraid is typically a usb3 drive in a usb2 port when possible for the best compatibility.

While mirroring is supported in cache pools it is not really intended to be the primary method of redundancy. Unraid is built around the unraid array (which in my opinion is its star feature). The unraid array is parity protected with single or dual redundancy options while still allowing each data disk to be read independently.

Many thanks for your post.

Unfortunately my device has no USB 2 port - it comes with 4 x USB 3.2 Gen1 ports. Anyway, the issue got solved and full USB3 combination works fine - even with 64 Gb sticks.

I didn't see the Help global button - thanks for highlighting it.

Thanks for your explanations about Unraid advantages over systems.

I installed the Nerdpack so I can using rmlint to find duplicate files. I will install yt-dlp manually so I can download some videos without resorting to a Docker image - so sad Nerdpack does not propose it.

Not sure, if it is a 6.10.2 bug, but plugging an external USB disk, I see no disk listed and I had to manually mount it through CLI. I was expecting to have a graphical way to mount as in QNAP/Synlogy/OMV.

Weird as the disk was recognized by system - as I can see through Tools/SysDevs URL :

May be more difficult than just using a docker. Unraid is a stripped down custom slackware NAS OS which hosts dockers and VMs. Probably doesn't include many things to make manual install of other things easy. Also, docker would be more secure than putting more code in the same space as the OS (which is in RAM in case you didn't know).

Docker is the recommended way to install/use applications in unRAID. Its another one of the particulars of the unRAID system. Docker application containers (especially those available in community applications) allow applications to be somewhat isolated from the system so they can only access intended data and are isolated from compatibility issues with unRAIDs stripped down OS. Plugins on the other hand are more for things that require direct integration with the OS or webui.

I don't use that, but you need to understand how user shares work. User shares are just the combined top level folders on array and pools. If that tool tells you a file is on a disk, and also on a user share, that is not a duplicate, it is just another view of the same file.

Probably better off if you don't try to leverage your linux skills with Unraid until you get a better idea how it is different. It is not a general purpose, multiuser linux. Only root user has access to the command line and webUI. Other users are strictly for network access of NAS files.

The OS installs fresh from the archives on flash, into RAM, at each boot, and runs completely in RAM. Think of it as firmware. All settings from the webUI are also stored on flash so they can be reapplied at boot.

Thanks primeval_god & trurl for your valuable information.

Initially,I looked for mount in All Apps, and didn't think about looking for Unassigned / use that keyword.

I installed the Unassigned Applications ! I thought such feature would be made available by default.

I tried to use the rmlint Docker image but it provides the Graphical interface / I prefer to use the CLI version.

I use it so far through the terminal in the Web admin interface - as root to identify and remove duplicate files

on external disk or a specific share. Detecting duplicate files on NAS is a frequent requirement.

I now understand the Unraid approach/philosophy way better and will create a Docker image based on Alpine Linux

with rmlint and yt-dlp for my specific needs - seems like I will need to create a Github repository.

Out of curiosity, did I miss to read an Unraid primer or other Introduction document that would have saved meyou some time ?

Other topic : I upgraded to 10.6.3 - smooth upgrade and will buy an Unraid license for sure ?

I had very high hopes for the option of setting the date to which to restore my files. But I was disappointed to discover both files and libraries that were deleted (as a part of my testing I-Drive) still showing up.

How would I do a recovery should my hard drive crash? all the hundresds of files and many libraries which were deleted during the weeks and months preceding the crash would all come back? I hope that I am doing something wrong and that this can be fixed by a simple setting change, otherwise, I would not be able to use the system.

I am running on a PC with Win-10.

My current backup (Memopal) has problems but does address this issue: on the initial restore screen it only shows current libraries and files. Of course it also provides access to older versions. In my opinion, this is a critical requirement for any backup system.