Proxmox Backup Server with high-capacity Toshiba hard drives

However, a common criticism of large HDD pools—especially in connection with ZFS under PBS—is performance. The downside: PBS stores all data in 4 MB chunks. The Proxmox backup solution prioritizes high IOPS for storage access rather than sequential read/write operations, as is usual in the backup sector. 

So good for SSDs and bad for HDDs.

This is precisely where the metadata cache —also known as a special device or special vdev — comes into play. This cache consists of NVMe disks, which are known to achieve impressive IOPS values. Once created, they take over the entire storage of the ZFS pool's metadata. Normally, this data is located on the data vdevs, but it is outsourced with this cache. 

This significantly increases access times and thus the verify, prune, and garbage collection jobs of the PBS. As a general rule, a metadata cache is a must for PBS when using HDDs with several terabytes or more! As a result, however, backup jobs for very large data sets remain high-performance, while the actual user data is stored cost-effectively on the Toshiba HDDs.

However, the metadata cache also poses a risk. Since all of the pool's metadata is stored there, the entire pool would be lost if the special vdev failed. For this reason, Starline always backs it up with a 3-way mirror. Ideally, this special vdev consists of NVMes, regardless of whether they are M.2 or U.2 enclosure variants.

To further maximize reliability, different manufacturers should be used for the 3-way mirror. For example, 2x Kioxia SSDs and 1x Samsung SSD. AOC adapter cards have proven to be very space-saving for M.2 models.

The rule of thumb for scaling is:

Approximately 3 GB special vdev per terabyte of storage.

Of course, the hard drives must also be redundantly secured among themselves. Two variants are suitable for the project with the 36-bay enclosure:

RAIDZ2: Min 4, Max 10 ⇾ Capacity: TB of the smallest hard drive * (number - 2)

RAIDZ3: Min 5, Max 12 ⇾ Capacity: TB of the smallest hard drive * (number - 3)

Example – Data vdev under RAIDZ3 with 12 HDDs, each with a capacity of 20 TB, results in:

20 TB * (12-3) = 20 TB * 9 = 180 TB usable capacity per vdev

We generally recommend creating several small vdevs instead of a few large ones. We do not use RAIDZ1 at all in this case. After all, in the event of a failure, you are left without any redundancy. So if something goes wrong during resilvering, we have data loss: the entire vdev's data stock is then lost.

The limit per vdev is set as a “soft” criterion, but should still not be exceeded. The reasoning behind this concerns the resilvering process: when a disk is replaced, all data is read from the other disks in order to distribute the data across the new disks. And it is precisely here that, if disks of the same age, runtime, or utilization are affected, it is quite likely that other disks could fail. The risk therefore increases with the number of disks in a vdev.