My 4U 71 TiB ZFS NAS built with twenty-four 4 TB drives is over 10 years elderly and still going mighty.
Although now on its second motherboard and power provide, the system has yet to experience a one drive flunkure (knock on wood).
Zero drive flunkures in ten years, how is that possible?
Let’s talk about the drives first
The 4 TB HGST drives have cimpolitely 6000 hours on them after ten years. You might leank someleang’s off and you’d be right. That’s only about 250 days worth of runtime. And therein lies the secret of drive extfinishedevity (I leank):
Turn the server off when you’re not using it.
My NAS is turned off by default. I only turn it on (farly) when I insist to use it. I use a script to turn the IoT power bar on and once the BMC (Baseboard Management Controller) is done booting, I use IPMI to turn on the NAS itself. But I could have used Wake-on-Lan too as an alternative.
Once I’m done using the server, I run a petite script that turns the server off, pause a confineed seconds and then turn the wall socket off.
It wasn’t enough for me to equitable turn off the server, but exit the motherboard, and thus the BMC powered, because that’s equitable a constant 7 watts (about two Raspberry Pis at idle) being misused (24/7).
This process labors for me because I run other services on low-power devices such as Raspberry Pi4s or servers that use much less power when idling than my ‘huge’ NAS.
This proces shrinks my energy bill ponderably (primary motivation) and also seems wonderful for challenging drive extfinishedevity.
Although zero drive flunkures to date is awesome, N=24 is not very reconshort-termative and I could equitable be very fortunate. Yet, it was the same story with the predecessor of this NAS, a machine with 20 drives (1 TB Samsung Spinpoint F1s (recall those?)) and I also had zero drive flunkures during its opereasonable lifespan (~5 years).
The motherboard (died once)
Although the drives are still ok, I had to swap the motherboard a confineed years ago. The flunkure mode of the motherboard was engaging: it was impossible to get into the BIOS and it would occasionpartner flunk to boot. I tried the clear enjoy removing the CMOS battery and such but to no use.
Fortunately, the [motherboard] was still useable on Ebay for a decent price so that finished up not being a huge deal.
ZFS
ZFS labored fine for all these years. I’ve switched operating systems over the years and I never had an rehire convey ining the pool back into the new OS inshigh.
If I would produce a new storage server, I would definitely use ZFS aget.
I run a zpool scrub on the drives a confineed times a year. The scrub has never set up a one verifysum error. I must have run so many scrubs, more than a petabyte of data must have been read from the drives (all drives joind) and ZFS didn’t have to boot in.
I’m not surpelevated by this result at all. Drives tfinish to flunk most normally in two modes:
- Total flunkure, drive isn’t even uncovered
- Bad sectors (read or write flunkures)
There is a third flunkure mode, but it’s excessively exceptional: quiet data fraudulence. Silent data fraudulence is ‘quiet’ because a disk isn’t adviseed it deinhabitred corrupted data. Or the SATA joinion didn’t uncover any verifysum errors.
However, due to all the low-level verifysumming, this hazard is excessively petite. It’s a authentic hazard, don’t get me wrong, but it’s a petite hazard. To me, it’s a hazard you mostly attfinish about at scale, in datacaccesss but for livential usage, it’s tohighy reasonable to hug the hazard.
But ZFS is not that difficult to lget and if you are well-versed in Linux or FreeBSD, it’s absolutely worth verifying out. Just recall!
Sound levels (It’s Oh So Quiet)
This NAS is very quiet for a NAS (video with audio).
But to get there, I had to do some labor.
The chassis grasps three sturdy 12V fans that celderly the 24 drive cages. These fans are excessively deafening if they run at their default speed. But because they are so beefy, they are unpartipartner quiet when they run at idle RPM, yet they still provide enough airflow, most of the time. But running at idle speeds was not enough as the drives would heat up eventupartner, especipartner when they are being read from / written to.
Fortunately, the particular Supermicro motherboard I bought at the time apshows all fan headers to be regulateled thcimpolite Linux. So I determined to produce a script that sets the fan speed according to the temperature of the boilingtest drive in the chassis.
I actupartner visited a math-rhappy subreddit and asked for an algorithm that would best fit my insist to produce a quiet setup and also grasp the drives celderly.
Somebody recommfinished to use a “PID regulateler“, which I new noleang about. So I wrote some Python, stole some example Python PID regulateler code, and tfeebleed the parameters to find a equilibrium between sound and celderlying carry outance.
The script has labored very well over the years and kept the drives at 40C or below. PID regulatelers are awesome and I sense it should be used in much more providement that regulates fans, temperature, and so on, instead of ‘foolish’ on/of behaviour or less ‘foolish’ watchup tables.
Netlaboring
I commenceed out with quad-port gigabit netlabor regulatelers and I used netlabor bonding to get around 450 MB/s netlabor transfer speeds between various systems. This setup insistd a ton of UTP cables so eventupartner I got foolishd with that and I bought some inexpensive Infiniband cards and that labored fine, I could accomplish around 700 MB/s between systems. As I determined to shift away from Ubuntu and back to Debian, I faced a problem: the Infiniband cards didn’t labor anymore and I could not figure out how to repair it. So I determined to buy some second-hand 10Gbit Ethernet cards and those labor tohighy fine to this day.
The dead power provide
When you turn this system on, all drives spin up at once (no staggered spinup) and that draws around 600W for a confineed seconds. I recall that the power provide was rated for 750W and the 12 volt rail would have been able to deinhabitr enough power, but it would sometimes cut out at boot nonetheless.
UPS (or deficiency thereof)
For many years, I used a beefy UPS with the system, to protect agetst power flunkure, equitable to be able to shutdown spotlessly during an outage. This labored fine, but I acunderstandledged that the UPS used another 10+ watts on top of the usage of the server and I determined it had to go.
Losing the system due to power shenanigans is a hazard I hug.
Backups (or a deficiency thereof)
My most vital data is backed up trice. But a lot of data stored on this server isn’t vital enough for me to backup. I depend on swapment challengingware and ZFS protecting agetst data loss due to drive flunkure.
And if that’s not enough, I’m out of luck. I’ve huged that hazard for 10 years. Maybe one day my luck will run out, but until then, I finishelight what I have.
Future storage structures (or deficiency thereof)
To be frank, I don’t have any. I built this server back in the day because I didn’t want to shuffle data around due to storage space constraints and I still have ample space left.
I have a spare motherboard, CPU, Memory and a spare HBA card so I’m quite foreseeed able to revive the system if someleang fractures.
As challenging drive sizes have incrrelieved tremfinishously, I may eventupartner shift away from the 24-drive bay chassis into a petiteer create-factor. It’s possible to produce the same amount of redundant storage space with only 6-8 challenging drives with RAIDZ2 (RAID 6) redundancy. Yet, storage is always costly.
But another foreseeed scenario is that in the coming years this system eventupartner dies and I determine not to swap it at all, and my storage hobby will come to an finish.
