Project: Building A Cheap NAS, Part 4
Matt Dreher
I find it helpful to decouple data storage from my other systems. I upgrade my main workstation and laptop every two to three years, and migrating my data between computers was always a pain. A dedicated storage server eliminates most data migrations and facilitates sharing files between my systems.
Fortunately, my old Synology recovered after I cleaned it out and reseated the disks, but it was an important wake-up call. I decided to switch to TrueNAS, as it offers an open-source implementation of an open storage format.
When I bought my Synology NAS, I initially installed three 4 TB drives and left the fourth slot empty. That gave me a total of 7 TB of usable space with Synology Hybrid Raid. Three years later, I was running out of space, so I added a fourth drive, bringing my total usable space to 10 TB.
I decided to apply the same strategy for my new build. I wanted a system that met my current needs with room to grow. My rough target was to start with 20 TB of usable storage and extra headroom for up to 30 TB if I add disks later.
ZFS offers different options for redundancy: raidz1, raidz2, and raidz3. The main difference is in robustness. raidz1 can survive one disk failure without losing data. raidz2 can survive two simultaneous disk failures, and raidz3 can survive three.
Further, rebuilding a ZFS pool puts an unusual amount of strain on all of the surviving disks. A disk that would have lasted a few more months under normal usage might die under the additional load of a pool rebuild.
From what I had read, ZFS is not very CPU-intensive. I ran a basic test by installing TrueNAS on a cheap Dell OptiPlex 7040 mini PC. It barely used the CPU, so it seemed safe to go with a low-powered option.
With six drive bays available in the case, I decided to start with four 8 TB disks, which translates to 22.5 TB of usable storage under raidz1. When I need to expand in the future, a fifth disk will bring me to 30.9 TB, and a sixth would get me 37 TB.
In researching different TrueNAS builds, I saw several posts claiming that ECC RAM (error correction code RAM) is a must-have to prevent data corruption. I ultimately decided against ECC RAM and just used standard, consumer-grade RAM.
The idea is that writing to an SSD is orders of magnitude faster than writing to multiple spinning disks. When an application writes data, ZFS can quickly write it to the SSD, tell the application that the write succeeded, then asynchronously move the data from the SSD to the storage pool. The SLOG improves write speeds significantly.
Strangely, even after I got the system to boot with borrowed parts, the motherboard reported that it was running BIOS version 2203, which ASUS claims is compatible with the AMD Athlon 3000G CPU. But I updated to the latest BIOS, which was 5862.
Performance topped out at 111 MiB/s (931 Mbps), which is suspiciously close to 1 Gbps. This suggests that the limiting factor is my networking hardware, as my switch, my desktop, and the NAS servers all have 1 Gbps Ethernet ports.
Although my old Synology managed to outshine TrueNAS on reads, this was not the case for writes. Even on an unencrypted volume, TrueNAS was 77% faster on small files, and the two systems performed similarly on 1 GiB files.
I found third-party apps much harder to install on TrueNAS. I use Plex Media Server to stream my movie and TV collection, and Plex is a pre-configured plugin on TrueNAS. It should be one of the easiest apps to install, but it took me an hour of fiddling and searching through documentation. By comparison, installing Plex on Synology takes about two minutes of clicking through a wizard.
Every minute of 4K ProRes LT footage (which is a very lightweight format, compared to RAW) is 3 GB of space. A typical video I produce has between 30-60 minutes of raw footage (which brings the total project size up to around 100-200 GB).
To edit footage well, the data not only needs to move fast (1 Gbps barely cuts it for a single stream), it also needs to have very low latency, otherwise Final Cut Pro (my editor of choice) lags quite a bit while scrubbing over dozens of video clips.
Therefore, I always used to edit videos off my local SSD drive. And sometimes over the network using macOS's built-in file sharing. But as my video workflow matures, I find myself needing a central storage solution disconnected from my main workstation.
I had five 8TB Samsung QVO SSDs from my insane $5000 Raspberry Pi server build. Until now, I had them inside my 2.5 Gbps NAS. But I wanted to build my own NAS capable of saturating a 10 Gbps connection, and allowing extremely low latency data access over the network to my two Macs, both of which are connected to my wired 10 Gbps home network.
When looking at the price discrepancy, you have to realize (a) I already had the 8TB SSDs, from some projects I tested last year... most people don't have those things laying around, and (b) I will actually use that much low-latency storage... most people probably don't and would be better off with less SSD storage and more spinning disks (which are much cheaper per TB).
The motivation for this build came from finding this Supermicro X10SDV-4C-TLN2F Mini ITX motherboard. The price ($270 shipped) was low enough I could consider building with it, and it already included an older-but-not-too-power-hungry Xeon D SoC, two 10 Gbps Ethernet ports, and 5 SATA-III connectors, the basic components I needed for the build.
ServeTheHome gave this motherboard a good review when it came out, and as long as things were in working order, it should still be a good choice, though less efficient and performant than a more expensive 2020s-era board.
I needed to find a Mini ITX rackmount enclosure, and luckily I'd been in talks with MyElectronics after using their prototype 'blue' enclosure for a remote Pi cluster installation I was testing. They are working on a new 2U mini ITX short-depth enclosure, and they sent me an early revision to use in this build:
I put all the hardware together, and installed TrueNAS from a USB stick to the NVMe drive. If you're wondering why I chose TrueNAS core, check out the full video where I speak to Wendell from Level1Techs about performance considerations and the storage layout for this NAS.
But I had a problem: I couldn't get the Supermicro motherboard to boot off UEFI. I was getting errors about "CPU with APIC ID 0 is not enabled", and I spent an hour messing around in the BIOS before (temporarily) giving up and throwing in a SATA boot SSD.
This motherboard is from around 2015, an era when NVMe boot on servers wasn't commonplace. Couple that with the M.2 slot accepting either SATA or NVMe drives, and most people preferring to boot off a SuperDOM instead, and I knew I was kind of on my own debugging it.
After some more consulting with Wendell and talking to Patrick from ServeTheHome, I was able to figure out both the boot issue (just had to mess with the UEFI boot settings a bit more), and the performance issue (it... ended up going away after I replugged everything).
The nightly snapshots (and indeed, ZFS/RAID itself) is NOT at all a substitution for a rigorous backup plan. For now I'll be rsyncing the volume to my main NAS nightly, and that's backed up offsite to Amazon Glacier weekly. Eventually I might use ZFS's sync functionality to synchronize snapshots to my backup server. We'll see.
HDDs don't provide the low latency access times that I need for scrubbing through video projects with a few thousand (often large file-size) assets, though. There are ways you can mitigate that if editing off hard drives, but it involves rendering proxy footage or using other compromises that slow down or hinder modern editing workflows (which I'd rather not do).
Been doin this kinda thing with hdds since the 90s. This just tells me you don't know much about video editing. Hdds get the job done just fine despite what you read or what your sponsor told you to say. I can build the same machine for $500, but you spent 5k AND called it a budget box. Dude reconsider.
Please don't start throwing around bigot accusations over this kind of stuff. He's literally using the old thing to prove the point that age DOESN'T matter, just because you worked with HDDs in the 90s, technology changes and has no real bearing on the reality of performance requirements today.
I've been a victim of racism, I lean left politically, but focusing on micro-aggressions or how it makes you feel or wtv is just minimizing the seriousness of actual bigotry that deeply affects people. It doesn't make sense to do that when the problems are still serious today. It fuels the other side, gives them plenty of material to work with when you conflate someone talking about age to make a point to bigotry. If 19 out of 20 bigotry claims are these kind of things, do you really think bigots see the error of their ways?
I had to fudge around with the UEFI settings in the BIOS boot menu, and add an option again after I reinstalled TrueNAS using the UEFI setting in the installer. To do that I had to pull the NVMe drive, put it in a USB adapter, plug it into my Mac, erase it with Disk Utility (so it wouldn't try booting off the broken half-UEFI/half-BIOS install), and then install TrueNAS from scratch again.
An 3rd part HBA has an extremely high power consumption....the HBA properly use more power than than the whole server!
You should go a long way to prevent that. The build-in Intel SATA controller is very efficient power-wise......cheaper, product less heat, doesn't take up a PCIe slot, add an additional point-of-failure.
A really bad idea to use.
I found today this article published by you and I don't understand anything anymore.
A couple of years ago you criticized the problems with True Nas!
I followed your projects and Dave Stephens with Ansible-NAs !
What to think now!?
What is your opinion about Ansible-Nas and True Nas now !?
Best Regards,
Peter
First, why build a NAS rather than buy a prebuilt one like the Synology DS1522+? There are pros and cons to each approach, but the DIY was attractive to me due to the better expansion, cost effectiveness, and subjectively I just have fun building PCs.
7fc3f7cf58