Category Archives: Servers

SSD Tiering versus Caching: Part 2

A while back I wrote about some of the differences between caching and tiering when using solid state disk (SSD) drives in a PC or server.

Having just returned from the 2011 Flash Memory Summit in Santa Clara, I feel compelled to add some additional color around the topic given the level of confusion clearly evident at the show. Also I’d like to blatantly plug an upcoming evolution in tiering, called MicroTiering from our own company, Enmotus which emerged from stealth at the show.

The simplest high level clarification that emerged from the show I’m glad to say matched what we described in our earlier blog (SSD Caching versus Tiering): caching makes a copy of frequently accessed data from a hard drive and places it in the SSD for future reads, whereas tiering moves the data permanently to the SSD and it’s no longer stored on the hard drive. Caching speeds up reads only at this point with a modified caching algorithm to account for SSD behavior versus RAM based schemes, whereas tiering simply maps the host reads and writes to appropriate storage tier with no additional processing overhead. So in teiring, you get the write advantage and of lesser benefit, the incremental capacity of the SSD which becomes available to the host as usable storage (minus some minor overheads to keep track of the mapping tables).

Why the confusion? One RAID vendor in particular, along with several caching companies, are calling their direct attached storage (or DAS) caching solution “tiering”, even though they are only caching the data to speed up reads and data isn’t moved. Sure write based caching is coming, but it’s still fundamentally a copy of the data that is on the hard drive not a move and SSD caching algorithms apply.

Where Caching is Deployed

SSD caching has a strong and viable place in the world of storage and computing at many levels so it’s not a case of tiering versus caching, but more when to use either or both. Also, caching is relatively inexpensive and will most likely end up bundling for free in PC desktop applications with the SSD you are purchasing for Windows applications for example, simply because this is how all caching ends up i.e. “free” with some piece of hardware, an SSD in this case. Case in point is Intel and Matrix RAID, which has now been enhanced with it’s own caching scheme called Smart Response Technology (SRT) currently available for Z68 flavor motherboards and systems.

In the broader sense, we are now seeing SSD caching deployed in a number of environments:

  • Desktops (eventually notebooks with both SSD and hard drives) bundled with SSDs or as standalone software e.g. Intel SRT and Nvelo (typically Windows only)
  • Server host software based caching e.g. FusionIO, IOturbine, Velobit (Windows and VMware)
  • Hardware PCIe adapter based server RAID SSD caching e.g. LSI’s CacheCade (most operating systems)
  • SAN based SSD caching software, appliances or modules within disk arrays e.g. Oracle’s ZFS caching schemes (disk arrays) or specialist appliances that transparently cache data into SSDs in the SAN network.

Where Data Tiering is Deployed

Tiering is still fundamentally a shared SAN based storage technology used in large data sets. In its current form, it’s really an automated way to move data to and from slow, inexpensive bulk storage (e.g. SATA drives, possibly even tape drives) to fast, expensive storage based on its frequency of access or “demand”. Why? So data managers can keep expensive storage costs to a minimum by taking advantage of the fact that typically less than 20% of data is being accessed over any specific period of time. Youtube is a perfect example. You don’t want to store a newly uploaded video and keep it stored on a large SSD disk array just in case it becomes highly popular versus the other numerous uploads. Tiering automatically identifies that the file (or more correctly a file’s assocatied low level storage ‘blocks’) is starting to increase in popularity, and moves it up to the fast storage for you automatically. Once on the higher performance storage, it can handle a significantly higher level of hits without causing excessive end user delays and the infamous video box ‘spinning wheel’. Once it dies down, it moves it back making way for other content that may be on the popularity rise.

Tiering Operates Like A Human Brain

The thing I like about teiring is that it’s more like how we think as humans i.e. pattern recognition over a large data set, with an almost automated and instant response to a trend rather than looking at independent and much smaller slices of data as with caching. A tiering algorithm observes data access patterns on the fly and determines how often and more importantly, what type of access is going on and adapts accordingly. For example, it can determine if an access pattern is random or sequential and allocate storage to the right type of storage media based on it’s characteristics. A great “big iron” example solution is EMC’s FAST, or the now defunct Atrato.

Tiering can also scale better to multiple levels of storage types. Whereas caching is limited to either RAM, single SSDs or tied to a RAID adapter, tiering can operating on multiple tiers of storage from a much broader set up to and including cloud storage (i.e. a very slow tier) for example.

MicroTeiring

At the show, I introduced the term MicroTiering, one of the solutions our company Enmotus will be providing in the near future. MicroTiering is essentially a direct attach storage version of its SAN cousin but applied on the much smaller subset of storage that is inside the server itself. It’s essentially a hardware accelerated approach to teiring at DAS level that doesn’t tax the host CPU and facilitates a much broader set of operating systems and hypervisor support versus the narrow host SSD caching only offerings we see today that are confined to just a few environments.

Tiering and Caching Together

The two technologies are not mutually exclusive. In fact, it is more than likely that tiering and caching involving SSDs will be deployed together as they both provide different benefits. For example, caching tends to favor the less expensive MLC SSDs as the data is only copied and handles the highly read only transient or none critical data, so loss of the SSD cache itself is none critical. It’s also the easiest way to add a very fast, direct attached SSD cache to your sever provided your operating system or VM environment can handle it.

On the other hand, as tiering relocates the data to the SSD, SLC is preferable for it’s higher performance on reads and writes, higher resilience and data retention characteristics. In the case of DAS based tiering solutions like MicroTiering, it is expected that tiering may also be better suited to virtual machine environments and databases due to it’s inherent and simpler write advantage, low to zero host software layers and VMware’s tendencies to shift the read-write balance more toward 50/50.

What’s for sure, lots of innovation and exciting things still going on this space with lots more to come.

Snapshot and Full Image Backup for Windows Home Server

One of the short comings of the original 32 bit Windows Home Server (WHS) for me was the lack of any built in tools to backup the primary WHS boot drive. While using a RAID 1 boot drive would protect me from a disk drive crash, it didn’t offer the capability to “rewind” back to a former backup copy to fix a system drive corruption issue. To compound the problem, my favorite Windows disk image backup utility doesn’t support server based operating systems, presumably because they have a higher cost enterprise class version they sell into the classic server markets.

So for most of us, the problem remains on how to easily backup and restore the primary boot volume of the WHS server or upgrading the entire server hardware without losing the current WHS configuration. The good news is that backup and restore for a primary boot drive becomes significantly easier when you are running as a virtual machine. Better still, these come for free if implementing WHS on VMware’s ESXi hypervisor as illustrated in At Home with ESXi posted earlier.

Additional Backup Options Provided by a WHS ESXi Setup

With VMware ESXi, you automatically get two ways to create backups of your primary home server boot drive via the vSphere utility run from your regular PC:

  • Snapshot – takes a point in time copy of the complete WHS virtual machine on the same physical drive as the primary WHS boot image. Total time around 3 mins and take as many as you have room for on the disk.
  • Full Image Copy – the complete WHS virtual machine is copied via the network to your local PC or a network drive. Total time will be several hours depending on your network speed (70GBytes take a while to copy across home network)

Read more at System Drive Backup Options for Windows Home Server on VMware ESXi.

Cloudy Server Growth

Infoworld recently posted an interesting article (The numbers don’t lie — cloud computing boosts server sales) about how server market revenues have increased by around 11 percent in the second quarter of 2010, with quarterly revenues coming in around $10.9 billion for the quarter. In fact, if you look at IDC’s 2Q10 press release, unit shipments were up 23.8% year over year which is impressive, reportedly the fastest quarter over quarter growth in 5 years. Much of this can certainly be attributed to recovery of the server market segment, but as David Linthicum discusses, shouldn’t overall server growth start declining in the face of server virtualization and cloud computing? Hm… good point.

In many ways, growth in the face of a technology shift caused by more efficient cloud and utility based computing models should not be that surprising given we’ve seen a similar phenomenon before. We lived through it in the early broadband communications days where the concept of an overlay network was used to introduce new services non-disruptively i.e. rather than replace the old network, a new one was built along side hence the term “overlay”. So it is not an unlikely scenario in the cloud age where both corporate and public computer-storage networks are effectively overlaying new systems on top of their existing systems to minimize disruptions and in many cases, testing this stuff out incrementally before they turn it all over to cloud and virtualization – be it internal or externally hosted.

This definitely seemed to be the case when I recently called my hosting provider to ask why FrontPage extensions could no longer be enabled for a website I was putting together as a temporary placeholder for another project. Turned out that all new customers were being steered to the new “grid computer system” which doesn’t support FrontPage extensions anymore, so I had to be moved back to the “legacy” systems. Bingo…. two networks, one for old and one for new. Reminded me of a tour of a central office in Austin, Texas we were given when developing the early broadband network technologies. We were there to review how they planned to roll out DSL when it eventually arrived. I was surprised then to see a lone Siemens digital switch in the corner providing ISDN services purely for data as an overlay service instead of using the already huge installed base of AT&T switches they’d been using for mainline customers and installing ISDN line cards. The technology and switch could handle it, but for service and disruption reasons it was not the preferred way to roll out an infant service.

Bottom line. Traditional models always stick around much longer than us technologists think for usually non-technical reasons, even when there are many advantages to ditching the old stuff for the new. Another reminder of the hype curve age we live in.

Windows Home Server to the Rescue

I have now been running WHS since Dec 2007 on a small self-built VIA based ITX system, chosen because it was small and ran on much lower power and generated less heat and noise than a conventional PC or low cost server allowing me to leave it on all the time. It’s worked like a charm since it was turned on in 2007 and not crashed once! Since then I’ve added a second WHS using the VIA Artigo shoebox PC which I use for projects like the Earthquake monitoring project referred to on this site a few months back.

We have a pretty active PC home. I have gamer kids (I’m one of them) in the house with 3 dedicated gaming PCs mixed in with 3 laptops (2 college, 1 work), a digital audio workstation for audio and midi recording, plus a home built media center PC in the family room. Operating systems are a mix of Windows XP, Vista 32 bit, Vista 64 bit and Windows 7. All of the PCs have the WHS Connector software, though only 2 of them (my home desktop and home laptop) wake up automatically to do backup all the time as I found that the laptop would wake up in my hotel room looking for my WHS when travelling! The rest, I tend to run manual backups or turn on the automated backup only when I know the work laptop is going to be stationary for a while or I have a lot of new content I’m creating.

Having lost data from my pre-WHS writing days, I’d already developed a healthy habit of making sure I had multiple copies of important data (e.g. photos, audio/midi projects) in multiple locations so I’ve managed to avoid catastrophic loss of personal data with a little careful management so far. WHS really helped automate and simplify this previously manual process. On occasion however, there are situations where a recovery of a complete PC or file is still necessary beyond the normal “copy off your backup USB drive” scenario.

(Read the full article on MSWHS.COM here.)

Intel Patsburg and Software RAID

I just got done reading the “Intel eats crow on software RAID” writeup from the The UK Register. On one side I’m really happy to see that server based software RAID (or Virtual RAID Adapters, VRAs, as we called them at Ciprico and DotHill) coming into the spotlight again. Performance, especially now with SSD usage on the rise, is definitely one of the strengths of a software RAID solution which has the ability to scale to much faster rates than a hardware RAID adapter in terms of raw IOPs or MB/s. After all, it’s using the power of a 2-3GHz multi-core Intel or AMD CPU coupled to a very fast memory and I/O bus, versus some fixed function, 800M-1.2GHz embedded RAID CPU hanging off the PCIe bus.

On the other hand, asking if software RAID is faster than or can replace hardware RAID is not really the right question to be asking here. Sure, software RAID with persistent storage like SSDs is changing the landscape as far as making a pure host based software RAID viable, but for traditional hard disk drives not much has changed. There’s a lot of volatile elements (i.e. gone if the lights go out) type storage stages used all the way from the application that wrote the data, through the storage IO device (be it a hardware accelerated RAID adapter or simple IO device), through the 32+MBytes of cache on the drive if you left it enabled, until you actually arrive at the persistent media storage platter. Oh, and then there is VMware ESX which can’t support a conventional software RAID stack yet.

So let’s get some perspective here.

First, as any good RAID vendor will tell you, it’s not so much about software vs hardware RAID, it’s about who is providing your RAID stack, how many “RAID bytes served so far”, how good service and support is and essentially how much you trust the vendor offering the software RAID stack. This is where a RAID stack’s “age” and pedigree is important regardless of its implementation. Being a good software RAID provider goes well beyond making it fast. It’s how robust your solution is and also how great your support is when things don’t work right and you need help fixing it. Hard disks (and SSDs are no exception) throw all sorts of curve balls at you and only the robustness of your RAID vendor’s test and compatibility labs can really filter a lot of this out. It often takes a knowledgeable RAID systems engineer to figure out that it either was, or was not, the fault of the RAID stack in the first place. My deepest respect is for those folks that have to spend their Sundays way into the wee hours of the morning figuring these sort of things out when the fault defies conventional logic.

Second, on the technology side, RAID is always implemented in software in IT application regardless if host or hardware based. It either runs on the host CPU (software, chipset or host RAID) or on a dedicated CPU on the RAID adapter (hardware RAID), sometimes in host software with some assistance from the hardware (e.g. XOR calculations). Granted, one runs in an unpredictable OS environment and the other in a more closed and predictable embedded one, but they end up doing the same thing in software on different CPUs. While there are cases where software RAID may be sufficient and more affordable as it eliminates much of the hardware cost, there are probably just as many cases where it just doesn’t work well at all. Case in point being VMware ESX (see earlier post on this topic here) where there are no commercially available, bootable software RAID solutions available, plus there are less general CPU cycles available anyhow. So hardware RAID tends to win out here. Also, software RAID doesn’t protect your data fully from a system power loss unless you are protecting the whole server with a dedicated UPS which can do an orderly shutdown of the system in the event of a plant power loss. Then there are the video editing crowd that maybe use their host CPUs for video compression, another case where software RAID often fails due to lack of enough available CPU cycles.

So, the key questions to be asking about software RAID in my mind are not how fast it can go, but:

  • How robust is the RAID stack in question i.e. how many “bytes were served” before you got to it, who else is using it a mission critical environment?
  • How would my business be impacted by a server power loss running software RAID? Can I live with a UPS to protect the whole server as I have a fast means of getting back to a fully operational level?
  • Who’s going to support it when it goes wrong and how good is this support when it comes to knowing both the RAID stack strengths and limitations?
  • Are you comfortable buying a RAID solution a chip vendor or storage vendor, the latter who makes their livelihood from creating highly robust disk array systems? You may be perfectly ok with the former.

All of these will depend on just how important your data is and more importantly, how quickly you can restore the system to full operation in the event of a hardware failure.

VMware ESXi at Home

What started out as a simple experiment to help me learn more about VMware ESX has now turned into a full blown experiment running my current Windows Home Server setup, along with two Linux servers used for an online FEAR Combat gaming server (for the kids of course) and a private WordPress website development environment running on a single Dell T110 server.

I am now able to create and tear down “server sandboxes” right next to my “leave alone” server setups (i.e. Windows Home Server) given the relative ease with which I can now create new virtual servers. More experimentation is necessary to get to streaming high definition videos which seem to struggle, but standard resolution and audio seem to work fine so far. This certainly seems like virtual servers are now well within the reach of the tekinerd and small home office type setups.

In addition to the Dell server, I also created a low cost iSCSI box using the free OpenFiler software and the VIA Artigo A2000 shoebox sized computer to expand the storage capabilities of ESX and primarily the virtual Windows Home Server which required additional storage to handle my total home PC backup requirements. Details of the final setup are included below, with a more detailed writeup on the Tekinerd Server Pages at http://tekinerd.com/server-pages/at-home-with-vmware-esxi/.

Hardware:

  • Dell T110 server (2x 160G drives in my particular setup), 2G DRAM (~$399 special at Dell)
  • VIA Artigo A2000 for the iSCSI storage box with 2x WD 500G drives (~$350 all in)

Software

  • Dell: VMware ESXi v4 (free download from VMware)
  • Dell: Client OS#1: Microsoft Windows Home Server ($99)
  • Dell: Client OS#2: OpenSUSE 11.2 ($0) setup as a FEAR Combat Server ($0)
  • Dell: Client OS#3: OpenSUSE 11.2 ($0) setup as a WordPress Development Server ($0)
  • VIA Artigo A2000: Openfiler v2.3 ($0) configured with 2 iSCSI targets (317G + 465G available)
  • Laptop: VMware vSphere Client software ($0)

PCIe Flash versus SATA or SAS Based SSD

The impressive results being presented by the new PCIe based server or workstation add-in card flash memory products hitting the market from the likes of FusionIO and others are certainly pushing up the performance envelope of many applications, especially in transactional database applications where the number of user requests is directionally proportional to the storage IOPs or data throughput capabilities.

In just about all cases, general purpose off the shelf PCIe SSD devices all present themselves as a regular storage device to the server e.g. in Windows, they appear as a SCSI like device that can be configured in the disk manager as regular disk volume (e.g. E: or F:). The biggest advantage PCIe SSDs have over standalone SATA or SAS SSD drives is that they can handle greater data traffic throughput and I/Os as they use the much faster PCIe bus to connect directly to multiple channels of flash memory, often using a built in RAID capability to stripe data across multiple channels of flash mounted directly on board the add-in card.

To help clear up confusion for some of the readers, the primary differences between PCIe Flash memory and conventional SSDs can be summarized as follows:

Where PCIe Flash Works Well

The current generation of PCIe flash SSDs are best suited to applications that require the absolute highest performance with less of an emphasis on long term serviceability as you have to take the computer offline to replace defective or worn out SSDs. They also tend to work best when the total storage requirements for the application can live on the flash drive. Today’s capacities of up to 320G (SLC) or 640G (MLC) are more than ample for many database applications, so placing the entire SQL database on the drive is not uncommon. Host software RAID 1 is typically used to make the setup more robust but starts to get expensive as high capacity PCIe SSD cards run well north of $10,000 retail, the high price typically a result of the extensive reliability and redundancy capability of the card’s on-board flash controller. As the number of PCIe flash adapter offerings grow and the market segments into the more traditional low-mid-high product categories and features, expect the average price of these types of products to come down relatively fast.

Where SSDs Work Well

SATA or SAS based SSDs, by design, work pretty much anywhere a conventional hard drive does. For that reason we see laptops, desktops, servers and external disk arrays adopting them relatively quickly. Depending on the PCIe flash being compared to, it can take anywhere from 5-8 SSDs to match the performance of a PCIe version using a hardware RAID adapter which tends to push the overall price higher when using the more expensive SLC based SSDs. So SATA or SAS SSDs tend to be best suited to applications that can use them as a form of cache in combination with a traditional SATA or SAS disk array setup. For instance, it is possible to achieve a similar performance and significantly lower system and running costs using 1-4 enterprise class SSDs and SATA drive in a SAN disk array versus a Fibre Channel or SAS 15K SAN disk array setup. Most disk array vendors are now offering SSD versions of their Fibre Channel, iSCSI or SAS based RAID offerings.

Enterprise Flash Memory Industry Direction

At the Flash Summit we learned that between SSDs and DRAM a new class of storage will appear for computing, referred to as SCM, or storage class memory. Defined as something broader than just ultra fast flash based storage, it does require that the storage be persistent and appear more like conventional DRAM does to the host i.e. linear memory versus a storage I/O controller with mass storage and a SCSI host driver. SCM is expected to enter mainstream servers by 2013.

Hardware RAID Adapters Making A Comeback?

We are constantly reading about how the number of cores being offered by Intel and AMD will eventually make hardware acceleration devices in a server mostly obsolete. But hang on, I recall hearing something similar back in the 90s when MIPS – meaningless indication of performance provided by sales people – was the in metric. Having just been involved in a key OEM software RAID project and watching the steady replacement of server based RAID adapter unit shipments start to shift downward as more CPU cycles come on line, I recently saw some recent market numbers from Gartner that showed the reverse trend. Hardware RAID adapter cards, which were supposed to be dying slowly, are starting to see a resurgence in servers.

So why is this?

Not surprisingly, one of the key contributing suspects is looking like increased virtual server adoption, in particular VMware ESX. Unlike the conventional operating system environment, a dedicated hypervisor environment like ESX doesn’t have the same “luxurious” methods for developing a broad range of device drivers for starters, let alone the lack of RAM space to load up complex device drivers. The whole point of virtual servers is to drive the hardware to a minimal number of absolutely necessary interface types, so no surprises that there is a rather slim pickings of storage adapters in the standard distribution of ESX. This is specially so with the new breed of skinny hypervisors capable of operating on minimal ROM and RAM footprints (e.g. inside the system BIOS itself) that require skinny device drivers versus something like a fat fully featured software RAID driver that could required up to 1Gbyte RAM minimum in a single-OS setup. Then there is the user aspect of loading custom drivers via the vSphere command line if VMware doesn’t bundle the one you need with their standard distribution. I’m still coming up my learning curve on ESX, but there is a definate and significant learning curve as I attempt to explore the capabilities of my newfound experimental VMware ESXi system to make sure I get it right (i.e. don’t kill it) versus the Microsoft easy-to-install approach for standard apps and drivers we’ve all become used to seeing.

In the case of hardware RAID versus software RAID, the initial problem is that conventional software RAID just doesn’t fit well into a VMware ESX hypervisor environment because most of today’s solutions were built with a single-OS in mind. Apart from the fact you can’t get hold of a software RAID stack for ESX, even if you could, it is likely to take up significant – possibly excessive – system RAM resources as a percentage of the overall hypervisor functions. Re-enter hardware RAID as it doesn’t really care about what OS you are running above it. It can use a simpler, skinny host driver without impacting the system resources and ports over easier to the ESX environment as the RAID stuff runs down on a dedicated hardware accelerated engine or storage processor.

One of the other contributing factors to hardware RAID’s incline again could also be the increased usage and focus on external SAS disk arrays. DAS is certainly making a comeback, especially given that the latest generation SAS disk arrays can operate at up to 24Gbps rates over a single cable to the host (four SAS channels running at 6Gbps). Having a hardware RAID adapter as the primary connection to a SAS JBOD ensures that there is always a consistent virtual interface to the hypervisor layers and that the CPU and RAM resources are not being overtaxed. Sure a software RAID stack can do the same given enough system resources and CPU cycles, but concerns about scaleability as you add drives over time is something that makes software RAID more difficult to manage in an ESX environment as it impacts the RAM usage when adding more drives for starters. Again, not a problem with hardware RAID. Same resources consistently presented to the hypervisor layers without significant CPU-RAM impact as drive changes and capacity increases are made.

So for hardware RAID, and many other traditional functions that were on the track to oblivion if you read the multi-core CPU tea leaves of late, maybe things aren’t so bad after all for storage in particular. VMware creates a new lease of life for performance or IO functions that really need to operate at maximum performance and not steal CPU cycles or RAM resources from the hypervisor and/or applications.

If hardware RAID vendors can continue to add enhanced functions such as the SSD caching algorithms and other storage virtualization functions behind an external SAS switched storage setup, then there is definitely some life left in ye old RAID engines, at least in the opinion of this blogger.

While software RAID has a solid place in the future, dare I say “long live hardware RAID” (again)?