Intel Patsburg and Software RAID

I just got done reading the “Intel eats crow on software RAID” writeup from the The UK Register. On one side I’m really happy to see that server based software RAID (or Virtual RAID Adapters, VRAs, as we called them at Ciprico and DotHill) coming into the spotlight again. Performance, especially now with SSD usage on the rise, is definitely one of the strengths of a software RAID solution which has the ability to scale to much faster rates than a hardware RAID adapter in terms of raw IOPs or MB/s. After all, it’s using the power of a 2-3GHz multi-core Intel or AMD CPU coupled to a very fast memory and I/O bus, versus some fixed function, 800M-1.2GHz embedded RAID CPU hanging off the PCIe bus.

On the other hand, asking if software RAID is faster than or can replace hardware RAID is not really the right question to be asking here. Sure, software RAID with persistent storage like SSDs is changing the landscape as far as making a pure host based software RAID viable, but for traditional hard disk drives not much has changed. There’s a lot of volatile elements (i.e. gone if the lights go out) type storage stages used all the way from the application that wrote the data, through the storage IO device (be it a hardware accelerated RAID adapter or simple IO device), through the 32+MBytes of cache on the drive if you left it enabled, until you actually arrive at the persistent media storage platter. Oh, and then there is VMware ESX which can’t support a conventional software RAID stack yet.

So let’s get some perspective here.

First, as any good RAID vendor will tell you, it’s not so much about software vs hardware RAID, it’s about who is providing your RAID stack, how many “RAID bytes served so far”, how good service and support is and essentially how much you trust the vendor offering the software RAID stack. This is where a RAID stack’s “age” and pedigree is important regardless of its implementation. Being a good software RAID provider goes well beyond making it fast. It’s how robust your solution is and also how great your support is when things don’t work right and you need help fixing it. Hard disks (and SSDs are no exception) throw all sorts of curve balls at you and only the robustness of your RAID vendor’s test and compatibility labs can really filter a lot of this out. It often takes a knowledgeable RAID systems engineer to figure out that it either was, or was not, the fault of the RAID stack in the first place. My deepest respect is for those folks that have to spend their Sundays way into the wee hours of the morning figuring these sort of things out when the fault defies conventional logic.

Second, on the technology side, RAID is always implemented in software in IT application regardless if host or hardware based. It either runs on the host CPU (software, chipset or host RAID) or on a dedicated CPU on the RAID adapter (hardware RAID), sometimes in host software with some assistance from the hardware (e.g. XOR calculations). Granted, one runs in an unpredictable OS environment and the other in a more closed and predictable embedded one, but they end up doing the same thing in software on different CPUs. While there are cases where software RAID may be sufficient and more affordable as it eliminates much of the hardware cost, there are probably just as many cases where it just doesn’t work well at all. Case in point being VMware ESX (see earlier post on this topic here) where there are no commercially available, bootable software RAID solutions available, plus there are less general CPU cycles available anyhow. So hardware RAID tends to win out here. Also, software RAID doesn’t protect your data fully from a system power loss unless you are protecting the whole server with a dedicated UPS which can do an orderly shutdown of the system in the event of a plant power loss. Then there are the video editing crowd that maybe use their host CPUs for video compression, another case where software RAID often fails due to lack of enough available CPU cycles.

So, the key questions to be asking about software RAID in my mind are not how fast it can go, but:

  • How robust is the RAID stack in question i.e. how many “bytes were served” before you got to it, who else is using it a mission critical environment?
  • How would my business be impacted by a server power loss running software RAID? Can I live with a UPS to protect the whole server as I have a fast means of getting back to a fully operational level?
  • Who’s going to support it when it goes wrong and how good is this support when it comes to knowing both the RAID stack strengths and limitations?
  • Are you comfortable buying a RAID solution a chip vendor or storage vendor, the latter who makes their livelihood from creating highly robust disk array systems? You may be perfectly ok with the former.

All of these will depend on just how important your data is and more importantly, how quickly you can restore the system to full operation in the event of a hardware failure.

VMware ESXi at Home

What started out as a simple experiment to help me learn more about VMware ESX has now turned into a full blown experiment running my current Windows Home Server setup, along with two Linux servers used for an online FEAR Combat gaming server (for the kids of course) and a private WordPress website development environment running on a single Dell T110 server.

I am now able to create and tear down “server sandboxes” right next to my “leave alone” server setups (i.e. Windows Home Server) given the relative ease with which I can now create new virtual servers. More experimentation is necessary to get to streaming high definition videos which seem to struggle, but standard resolution and audio seem to work fine so far. This certainly seems like virtual servers are now well within the reach of the tekinerd and small home office type setups.

In addition to the Dell server, I also created a low cost iSCSI box using the free OpenFiler software and the VIA Artigo A2000 shoebox sized computer to expand the storage capabilities of ESX and primarily the virtual Windows Home Server which required additional storage to handle my total home PC backup requirements. Details of the final setup are included below, with a more detailed writeup on the Tekinerd Server Pages at http://tekinerd.com/server-pages/at-home-with-vmware-esxi/.

Hardware:

  • Dell T110 server (2x 160G drives in my particular setup), 2G DRAM (~$399 special at Dell)
  • VIA Artigo A2000 for the iSCSI storage box with 2x WD 500G drives (~$350 all in)

Software

  • Dell: VMware ESXi v4 (free download from VMware)
  • Dell: Client OS#1: Microsoft Windows Home Server ($99)
  • Dell: Client OS#2: OpenSUSE 11.2 ($0) setup as a FEAR Combat Server ($0)
  • Dell: Client OS#3: OpenSUSE 11.2 ($0) setup as a WordPress Development Server ($0)
  • VIA Artigo A2000: Openfiler v2.3 ($0) configured with 2 iSCSI targets (317G + 465G available)
  • Laptop: VMware vSphere Client software ($0)

PCIe Flash versus SATA or SAS Based SSD

The impressive results being presented by the new PCIe based server or workstation add-in card flash memory products hitting the market from the likes of FusionIO and others are certainly pushing up the performance envelope of many applications, especially in transactional database applications where the number of user requests is directionally proportional to the storage IOPs or data throughput capabilities.

In just about all cases, general purpose off the shelf PCIe SSD devices all present themselves as a regular storage device to the server e.g. in Windows, they appear as a SCSI like device that can be configured in the disk manager as regular disk volume (e.g. E: or F:). The biggest advantage PCIe SSDs have over standalone SATA or SAS SSD drives is that they can handle greater data traffic throughput and I/Os as they use the much faster PCIe bus to connect directly to multiple channels of flash memory, often using a built in RAID capability to stripe data across multiple channels of flash mounted directly on board the add-in card.

To help clear up confusion for some of the readers, the primary differences between PCIe Flash memory and conventional SSDs can be summarized as follows:

Where PCIe Flash Works Well

The current generation of PCIe flash SSDs are best suited to applications that require the absolute highest performance with less of an emphasis on long term serviceability as you have to take the computer offline to replace defective or worn out SSDs. They also tend to work best when the total storage requirements for the application can live on the flash drive. Today’s capacities of up to 320G (SLC) or 640G (MLC) are more than ample for many database applications, so placing the entire SQL database on the drive is not uncommon. Host software RAID 1 is typically used to make the setup more robust but starts to get expensive as high capacity PCIe SSD cards run well north of $10,000 retail, the high price typically a result of the extensive reliability and redundancy capability of the card’s on-board flash controller. As the number of PCIe flash adapter offerings grow and the market segments into the more traditional low-mid-high product categories and features, expect the average price of these types of products to come down relatively fast.

Where SSDs Work Well

SATA or SAS based SSDs, by design, work pretty much anywhere a conventional hard drive does. For that reason we see laptops, desktops, servers and external disk arrays adopting them relatively quickly. Depending on the PCIe flash being compared to, it can take anywhere from 5-8 SSDs to match the performance of a PCIe version using a hardware RAID adapter which tends to push the overall price higher when using the more expensive SLC based SSDs. So SATA or SAS SSDs tend to be best suited to applications that can use them as a form of cache in combination with a traditional SATA or SAS disk array setup. For instance, it is possible to achieve a similar performance and significantly lower system and running costs using 1-4 enterprise class SSDs and SATA drive in a SAN disk array versus a Fibre Channel or SAS 15K SAN disk array setup. Most disk array vendors are now offering SSD versions of their Fibre Channel, iSCSI or SAS based RAID offerings.

Enterprise Flash Memory Industry Direction

At the Flash Summit we learned that between SSDs and DRAM a new class of storage will appear for computing, referred to as SCM, or storage class memory. Defined as something broader than just ultra fast flash based storage, it does require that the storage be persistent and appear more like conventional DRAM does to the host i.e. linear memory versus a storage I/O controller with mass storage and a SCSI host driver. SCM is expected to enter mainstream servers by 2013.