RAID, Backplane and Chassis Choices for OpenZFS - Starline Computer: Storage und Server Lösungen von erfahrenen Experten

RAID controller, Backplane and Chassis under OpenZFS

About forbidden fruits and a lot of hardware choices in ZFS systems.

How to use RAID controller and HBA with OpenZFS

In a previous article, we talked about how OpenZFS is both a filesystem and a disk/RAID manager at the same time. This usually imposes the question of whether or not we could use RAID controllers with our OpenZFS builds.

Can we use RAID controllers with OpenZFS/TrueNAS?

The short answer is: Yes. But, the more accurate question would be: Do you really need a RAID controller in your build or would a much cheaper, and less complicating, HBA adapter suffice?

There is a common myth amongst the OpenZFS community that RAID controllers are a forbidden fruit that should not be implemented in any OpenZFS build. This myth might have originated from the fact that ZFS can compose and handle its RAIDZ configurations just fine – without the need for extra hardware pieces. It might have also stemmed from FreeBSD’s poor support for some (actually most) of the available RAID controllers. After all, FreeBSD has always been on the conservative side when it comes to adopting newer technologies and driver availability for FreeBSD is much less than that of Linux.

If you factor in OpenZFS’s ability to utilise ARC, L2ARC, and SSD-based SLOG caches, then the need for a RAID controller diminishes even more.

What are the potential benefits of RAID controllers?

In general, RAID controllers can improve a server’s performance, and usable capacity, by:

  1. CPU offloading: by performing the RAID computations on-card, the system CPU will have more cycles to spend on other tasks, such as data encryption and deduplication.
  2. Caching reads and writes: this can improve HDD-based servers’ performance.
  3. SAS/JBOD expansion: Supporting an increased number of disks and/or external JBODs.

What to avoid when using a RAID controller with OpenZFS?

Based on my experience, one can use a RAID controller in an OpenZFS build on the following conditions:

  1. The RAID card should be one that is supported by FreeBSD/TrueNAS.
  2. The RAID card must support Standard RAID Levels and not use an uncommon proprietary RAID or partitioning schemes.
  3. Should the RAID card fail, you must be able to replace it with any available card – or directly plug in the disk into your mainboard/backplane to continue working.
  4. Do not use a controller with caching capability unless the cache is Battery-Backed.
  5. If the RAID controller offers on-chip encryption do NOT enable that feature.
  6. Avoid the controller’s CPU overheating.

CPU offloading is a feature available for RAID controllers, network adapters, cryptographic modules, etc. This is basically a form of Computation Offloading – sometimes also called Hardware Acceleration – where the RAID chip takes care of the computational burden of assembling files from the data shards saved on different disks, and vice versa. If you are not enabling CPU-intensive features in your build, such as encryption or deduplication, then maybe there is not much help a dedicated RAID controller with CPU offloading can offer in this regard.

When you connect your disks to a RAID controller you need to wipe and “initialise” them in order to create a RAID array from those disks. During this process the controller will add specific identifying data to a hidden portion of each disk to help mark it as a member of its RAID. Some RAID controllers use uncommon, proprietary, techniques to mark those disks and this would make it almost impossible for you to replace that card with another from a different manufacturer and, sometimes, you wouldn’t be able to replace that card with something from the same manufacturer.

For instance, I found that LSI SAS9341-4i and SAS 9361-4i cards are not interchangeable. If you initialise the disks on the 9341 you will not be able to move them to the 9361 without reformatting.

So, make sure any RAID card you opt for can be easily replaced by another.

The easiest test, is to remove the card and plug your disks into an HBA, or directly into the mainboard. If the OpenZFS/TrueNAS system recognises the drives you should be fine using that card.

Caching reads and writes is where a good RAID controller can improve the performance – because RAID controllers use ECC RAM to cache disk IO. As mentioned before, losing read cache will not corrupt your data since the original data still resides on the pool. What can really cause problems is a power interruption or hardware failure of the card during a write operation.  To shield yourself against that select battery-backed controller. Just like the PFP mechanism in an SSD disk, the battery’s function is to keep the cache circuitry (RAM chips) of the card active long enough for the writes to happen.

If your RAID controller offers on-chip encryption of the RAID member disks do not activate that feature. If you are storing sensitive data, or if you just want to enjoy the additional security data encryption can offer you, use the built-in encryption feature in TrueNAS Core. It is important to note that:

  • Pool encryption can only be activated during pool creation.
  • You must backup your encryption keys. 
    No one can help you recover your data if you lose them!
Enabling_Encryption_in_TrueNAS-Core-1024x521
Enabling pool encryption on TrueNAS core. Notice the warning about possible data-loss?

It could happen that your selected chassis can physically host a larger number of disks than that supported by your mainboard and/or backplane. Also, some RAID controllers come with external SAS ports that could be used to attach a JBOD enclosure to your server, allowing you to significantly increase your usable storage capacity without having to purchase additional complete servers.

Finally, avoid overheating. RAID (and HBA) controllers have their own CPUs and most of them come with just a basic heatsink attached. OpenZFS generates a lot of IO that will whip your RAID/HBA card CPU to its top clocks most of the time – especially during pool scrubbing and disk resilvering. This can introduce unnecessary downtime – in addition to the cost of a replacement card. I have witnessed the demise of some expensive cards that were deployed in poorly ventilated cases.

Use a well-ventilated chassis. If you have free PCI slots on your mainboard, try to space your cards apart for better airflow and to avoid creating heat traps between them. Sometimes, removing a couple PCI slot metal covers will help. If your card has only a heatsink, try to position the card right into the air stream of one of the chassis fans. It would be even better if you can attach a dedicated fan to the card.

Should I use a RAID controller with OpenZFS?

No. The fact that you can use one does not necessitate that you should use one. If you are not sure whether or not you should use a RAID card, the chances are you do not need one.

Most of the time you will be better off with a good SAS backplane or a passive, and reliable, HBA. If you already have a RAID card laying around, that can help you increase the number of disks you attach to your system, set it into JBOD mode, pass-through mode, or create multiple RAID0 containing a single disk each – provided the you have read and followed the “What to avoid” tips above. Do not use other on-chip RAID functions.

The reason is: You do not want to stack different RAIDs on top of each other, because none of them will be fully aware of what the other is doing. Using both the on-chip RAID and the software RAIDZ of OpenZFS will only complicate things for you, especially when troubleshooting problems.

Also, some RAID cards tend to supress S.M.A.R.T messages, which TrueNAS relies on to probe the disks and send you early system alerts notifying you of disk issues and possible disk failures.

What is IT-mode?

IT-mode is a short term for Initiator Target mode. When your RAID/HBA card is in IT-mode, it directly passes individual drives to the host system so OpenZFS can construct and manage its own RAIDZ arrays – without any interference from the on-card RAID firmware.

If your card does not come pre-flashed with an IT-mode firmware there is a good chance you can find an IT-mode firmware for it – especially if it is an LSI card. Cross-flashing LSI firmware is a relatively a simple procedure. Search the TrueNAS forums for guidance regarding your specific card.

What is a recommended RAID/HBA for a TrueNAS build?

LSI adapters are the go-to for OpenZFS. They are proven to be robust with a brilliant driver and firmware support.

Depending on the type of disks you will use in your server you will most likely be choosing between two types of interfaces:

SFF-8087 connector | Older interface | supports SAS1/SAS2 (3.0 and 6.0 Gbps speeds)
SFF-8643 Connector | Newer interface | Supports SAS3 (12 Gbps)

Some modern LSI cards with the SFF-8643 Connector can also support NVMe/U.2 SSD disks, which makes them quite useful in builds where you intend to use L2ARC and SLOG devices.

Example model numbers:

For PCIe-2.0 boards, the LSI SAS 9211-8i/LSI 9220-8i
For PCIe-3.0 boards, the LSI SAS 9207-8i
For 12 Gbit SAS, the LSI 9305-8i/16i/24i SAS

Avoid older adapters such as the LSI 1068. Those adapters cannot support disks larger than 2 TB.

SFF-8087.vs_.SFF-8643.Connectors--1024x550
The older SFF-8087 connector (right) vs the new SFF-8643 connector (left).
SFF-8087_Card-1024x576
An HBA with the older SFF-8087 connector
SFF-8643.Card_-1024x576
An HBA with the new SFF-8643 connector
1/3
Why are we talking about older PCI-2.0 boards and adapters?

Because, for home and small office builds, those boards are still very usable and quite price economical.

The same applies for older PCI-2.0 adapters when using HDD SATA disks. For instance, the LSI 9211-8i can be had for as little at €50 and supports up to 8x drives. This card can handle anything your new 14TB SATA 3.0 HDD can push without a hiccup.

What does the “i” and “e” suffix, in the HBA name, mean?

Those suffixes indicate the number of internal and/or external devices supported by the card.
For instance, the LSI 9305-8i card can support up to 8 internal devices, while the LSI 9305-8i-4e card can support up to 8 internal devices + 4 external ones and the LSI 9305-4e can 4 external devices. 
So,if you want an HBA that could support both internal drives plus an external JBOD, then make sure you choose accordingly.

What is a recommended SAS adapter for attaching external JBODs and tape libraries?

Again, LSI is your best choice for an HBA. Next comes ATTO HBA adapters. They have stable drivers and work well with external SAS JBODs and tape libraries.

The ATTO HBA driver installation is as easy as it gets.

ATTO.Driver.Installer-1024x249

The ATTO External SAS HBA

ATTO_ExpressSAS_H_1280GT

How to differentiate between a RAID card and an HBA?

There are some visual differences that can help you identify if a given card is an HBA or a RAID controller.

  • One major identifier is the existence of RAM chips on the card. 
    RAID cards do have multiple RAM chips and/or RAM banks on them. HBAs don’t.
     
  • If there is a tiny Piezoelectric speaker, it is a RAID card.
    RAID controllers use those tiny buzzers to alert you to possible drive failures or RAID malfunctions.
     
  • If it has a RAID-key or a RAID-key slot, it is a RAID card.
    Some RAID cards require hardware keys (in the form of a tiny pluggable circuit) to unlock advanced RAID configurations. Yes, you might have to pay extra to be able to do RAID10 with your drives – which is something you could do for free with OpenZFS.

How to spot a RAID card

How.to_.identify.a.RAID_.card_II-1024x719

What is a JBOD mode/enclosure?

JBOD is an abbreviation of the term Just a Bunch of Disks and it is commonly used when you group a number of disks of varying specs and capacities into a RAID set.

When your controller is set to JBOD mode it acts as an HBA, passing the drives directly to OpenZFS. 
A JBOD enclosure (also called SAS expander) on the other hand is a disk enclosure that contains no PC mainboard, CPU, or RAM. It has only a backplane and external connector ports that you can be used to attach it to an actual server. This means that JBODs can help increase your usable capacity at a reasonably low cost, because you don’t have to purchase a mainboard, a CPU, or RAM for it.

JBODs come in varying sizes and can support up to tens of disks per JBOD chassis so, they are quite useful if you want to run low IO pools, such as backups and archives. They would be much of much less use if you want to increase your storage capacity while maintaining high IO. You will be better off buying a new server, or even upgrading your current server’s hardware.

Because a JBOD has no mainboard, CPU, or RAM, you will not be able to install and/or boot TrueNAS on it, and it cannot serve as a failover clone. You will always have to attach it to an OpenZFS/TrueNAS server to be able to use it.

Mixing old/shelved disks into a JBOD is not always useful. Keep in mind that, within a VDEV, the capacity and the speed of that VDEV is tied to its smallest and slowest member. 

  • If you group 3x 8 TB disks + 1x 2 TB disk into the same VDEV you are practically using 4x 2 TB disks and you have just wasted away 18 TB of disk space. 
  • Also, if you group 3x 2 TB SAS disks + 1x SATA 2 TB into the same VDEV, you have just downgraded the SAS disks to SATA performance. 

You will be better off unifying your drives.

Warning: When using external JBODs, it is advised to construct independent pools on them, instead of letting your pools span both the server and the JBOD. This is because if your JBOD shuts off or disconnects and your server stays on there is a high risk of corrupting the data on your pool.

 

If you use JBODs, make sure they have redundant PSUs and that both your server and your JBOD are sitting in a rack that can supply them with enough power, and that the rack is well protected by a good UPS.

Front side, backside, and the inside of a Supermicro 4-bays JBOD. Note the tiny SAS controller inside, and the absence of a mainboard.

Supermicro-JBOD-Combo-1024x862
What is the difference between SAS2 and SAS3?

SAS2 supports a throughput of up to 6 Gbps, while SAS 3 supports a throughput of up to 12 Gbps.
SAS3 chassis are faster and more suitable for higher IOPs applications. If you plan on using SSDs you should go with a SAS3 setup.

What are Backplanes?

A backplane is the board inside your server chassis that sets behind the drives bays and serving as a connector between those bays and your serve’s mainboard or your HBA.

There are different types of backplanes. Some backplanes are a dummy 1-to-1 passthrough connector board. This type requires one cable per disk. This means that if you want to plug in 24 drives into one of them, you will have to run 24 SAS/SATA cables from that backplane to your mainboard or HBA. This could become a cable management nightmare, especially for compact chassis, unless you use newer and thinner cables.

Other backplanes have an expander chip in them. For example, the older supermicro 846TQ and the recent 825TQ backplanes, are a passthrough SATA board with no expander chip.

The Supermicro 825TQ SAS Backplane.

825-Backplanem-1024x576
What are SAS Expander Backplanes?

A SAS expander backplane is a more intelligent backplane board, with a number of SAS connectors (i.e SFF-8640) that you can use to Expand the amount of disks you can connect to your HBA/RAID controller. It functions in a way similar to a network switch, by giving you more SAS ports to use, albeit it uses SAS as the communication protocol.

With a SAS expander you can connect more disks to a single HBA than normally attach to that HBA using direct cabling. It saves you money because you do not have to buy multiple HBAs or controllers to plug all your disks into the same box.

One downside of SAS expanders is that they have their own on-board I/O chips which, under high loads, can introduce a minute amount of latency to your disk I/O and slightly affect your server’s performance. This is worth mentioning, although it will not significantly impact the server performance.

Some SAS expanders, such as Supermicro’s SAS2-846EL2 have two expander chips, which allow for dual-port connectivity – thus offering a bit of protection against single-chip/single-port failure.

You should avoid SATA-only expanders, unless you absolutely know you will never want a performance upgrade. They offer you no chance of future upgrade to SAS drives, thus requiring the purchase of new chassis. Spend a little bit more and enjoy both better performance (remember: SAS is 12 Gbps) and future savings.

Not all backplanes have expanders in them. For budget builds, older backplanes such as the 846TQ should be OK. They are dummy passthrough boards and require a lot of cables to wire the drives to your HBAs.
Also, avoid the older 846EL2 SAS expanders, because they do not support disks larger than 2 TB.

What constitutes a good server chassis for my build?

A good chassis should:

  • Be able to host your planned capacity
  • Has redundant power supplies with enough power for the components and the drives
  • Has good ventilation to keep your drives cool
  • The backplane has an expander that supports your disk connector type (SATA/SAS/U.2 etc.)

For home servers, and where you can afford some downtime, you can get away with a single power supply. 
For anything serious, use a chassis with redundant PSUs with a wattage rating that can cover your components consumption – plus some more.

Backplane_Multi_Cable-1024x721
A Supermicro server with a 825TQ backplane. Note the 8x SAS cables running between the backplane and the HBA.
Backplane_Single_Cable-1024x707
A Supermicro server with a SAS3-216EL expander. Note the single SAS cables running between the backplane and the HBA.
1/2

TL;DR

  • Yes, you can use RAID controllers in an OpenZFS/TrueNAS build
  • No, you do not have to. Most of the time a simple HBA is the better solution.
  • OpenZFS’s built in ARC, L2ARC, SLOG, and RAIDZ abilities
  • Only use controllers that do not lock you and your disks in.
  • If your controller has a cache, make sure it is battery-backed.
  • A simple test for the controller is to remove it and plug the disk cables into a different card or directly into your mainboard. If you cannot easily access the disks, without re-initialising them, dump that card.
  • LSI controllers are the ones best supported by FreeBSD/TrueNAS.
  • Do NOT use on-card disk encryption. Use the in-built OpenZFS encryption instead.
  • Avoid the controller CPU overheating. Use well-ventilated server cases.
  • There are two main different types of backplanes. Basic backplanes and SAS expander backplanes.
  • Basic backplanes are 1-t-1 connector boards. They are simple, cheap and efficient.
  • SAS expander backplanes have dedicated onboard I/O chips and can support a larger number of drives. They can also uses compact SAS cabling.
  • SAS expander backplanes may introduce a slight amount of latency, caused by their onboard chip.
  • Avoid older 846EL2 backplanes. They do not support drives larger than 2 TB.
  • If you want good performance and future budget savings, get a SAS expander.
  • A good chassis should be able to host your planned capacity, have good ventilation and redundant PSUs.
starline_logo_kontur_300
Open Source Team
Technik

Our experts for Linux, Ceph and ZFS.