Tag Archives: ssd

SSDs and World of Warcraft

For enthusiasts and gamers, over the past several years, combining several hard drives in a striped RAID configuration has provided reasonable benefits over and above a single hard drive. With my new SSD insight however, it’s now apparent that RAID never really provided sufficient improvements that justified the additional cost, time and effort involved other than for a few applications. It was really the HD video editing users that benefited the most as they were able to increase the size and performance of a single storage volume and support merging of two or more HD video streams in real time as a single disk drive on it’s own just wasn’t fast enough i.e. they were forced to RAID. Since then, compression techniques have improved to the point where you don’t need to RAID disks even for video in many situations (except for the high end professional rigs).

Why were there no real improvements observed from RAID in gaming applications? Talking to a few gamer developers it’s pretty apparent that given the slow response times of hard disks, a lot of development hours have been spent ensuring that the software avoided hitting that slow rotating media at all costs, relying instead on RAM based caching. Then along came games like Blizzard’s World of Warcraft and Starcraft 2(ok  – full disclosure, two of my favorites) that had to load a significant amount of data from the hard drives as you loaded or transitioned to new areas of the online world. This load time becomes worse as customizations in the form of add-on utilities are added and as the expansion packs were introduced with richer media content.

Then Along Came Enthusiast Class SSDs

SSDs changed the situation considerably. We are now able to get a simple add-on component that operates just like a regular hard drive, but with 5x or more the data streaming capability directly off the drive and 100x+ (more for higher end) the random IO performance versus a regular hard drive. They are also small, compact and lower power just like a laptop drive, so no upgrades to your computer case and PSU as was typical in the RAID days.

We did a little experiment of our own to see how well SSDs behaved using a Monster Digital LeMans series SSD, a SATA 6Gbps model based on the newer Sandforce chipsets. The same Intel core i5 system was setup with a regular hard drive for one configuration and the same system then installed onto an SSD. In both cases, the entire operating system and World of Warcraft game was loaded onto the single storage devices with no other applications or antivirus installed other than Fraps for capturing video.

 

Game load time World of Warcraft on Hard Disk followed by SSD

 

Configured with a 7200 RPM 500G disk drive, the time taken to load World of Warcraft with a significant number of add-on helper applications was nearly 2 minutes 30s, including the time taken for all the neighboring characters and players to load. The exact same configuration on the SSD took a little over 7s. To see how much this was just the SSD versus the 6G SATA interface (the Intel motherboard we used had 2 AHCI SATA 6G ports) we repeated the test with a a 3G SATA device and it only took around 8 seconds i.e. unless your application really needs the extra few seconds, 3G SATA works pretty well also.

The system configuration we used for this test was:

  • Intel Core i5 3.1GHz
  • Asus P8 Z68-V/Gen3 Motherboard
  • 4G RAM
  • AMD Radeon HD 6770 1G Graphics Card
  • WD 500G 7200 RPM SATA drive for HDD based testing
  • Monster Digital LeMans 480G SSD SATA 6G drive for SSD based testing

Conclusion

If you can afford it, take a serious look at an SSD. They speed up your system without a doubt in gaming applications and the system just works better overall. Professional gamers have already made the transition. The only thing to be concerned about is reinstalling your operating system and applications to get the maximum possible performance which is fine for most enthusiasts, but for the adventurous, there are solutions coming to help address this also or you can try migrating your existing system over using some of the software tools available.

Overall, very happy with the change to SSD.

SSD Caching versus Tiering

In some recent discussions, I sensed there is some confusion around solid state device (SSD) storage used as a storage tier vs, a cache. While there are some similarities and both are intended to achieve the same end result i.e. acceleration of data accesses from slower storage, there are some definite differences which I thought I’d try to clarify here. This is from my working viewpoint here, so please do post comments if  you feel differently.

Firstly, SSD caching is temporary storage of data in an SSD cache whereas true data tiering is classed as a semi-permanent movement of data to or from an SSD storage tier. Both are based on algorithms or policies that ultimately result in data being copied to, or removed from, SSDs. To clarify further, if you were to unplug or remove your SSDs, for the caching case, the user data is still stored in the primary storage behind the SSD cache and is still from the original source (albeit slower) whereas in a data tier environment, the user data (and capacity) is no longer available if the SSD tier were removed as the data was physically moved to the SSDs and most likely removed from the original storage tier.

Another subtle difference between caching and teiring is if the SSD capacity is visible or not. In a cached mode, the SSD capacity is totally invisible i.e. the end application simply sees the data accessed much faster if it has been previously accessed and is still in cache store (i.e. a cache hit). So if a 100G SSD cache exists in a system with say 4TB of hard disk drive (HDD) storage, the total capacity is still only 4TB i.e. that of the hard disk array, with 100% of the data always existing on the 4TB with copies only of the data in the SSD cache based on the caching algorithm used. In a true data tiering setup using SSDs, the total storage is 4.1TB and though this may be presented to a host computer as one large virtual storage device, part of the data exists on the SSD and the remainder on the hard disk storage. Typically, such small amounts of SSD would not be implemented as a dedicated tier, but you get the idea if say 1TB of SSD storage was being used in a storage area network system of 400TB of hard drive based storage creating 401TB of usable capacity.

So how does data make it into a cache versus a tier? Cache and block level automated data tiering controllers monitor and operate on statistics gathered from the stream of storage commands and in particular the addresses of the storage blocks being accessed.

SSD Caching Simplified

Caching models typically employ a lookup table method based on the block level address (or range of blocks) to establish if the data the host is requesting has been accessed before and potentially exists in the SSD cache. Data is typically moved more quickly into an SSD cache versus say tiering where more analysis of the longer term trend is typically employed which can span hours if not days in some cases. Unlike DRAM based caches however where it is possible to cache all reads, a little more care and time is taken with SSDs to ensure that excessive writing to the cache is avoided given the finite number of writes an SSD can tolerate. Most engines use some form of “hot-spot” detection algorithm to identify frequently accessed regions of storage and move data into the cache area once it has been established there is a definite frequent access trend.

Traditional caching involves one of several classic caching algorithms which result in either read-only or read and write caching. Cache algorithms and approaches vary by vendor and dictate how a read from the HDD storage results in a copy of the original data entering the cache table and how long it ”lives” in the cache itself. Subsequent reads to that same data who’s original location was on the hard drive can now be sent from the SSD cache instead of the slower HDD i.e. a cache hit (determined using a address lookup in the cache tables). If this is the first time data is being accessed from a specific location on the hard drive(s), then the data must first be accessed from the slower drives and a copy made in the SSD cache if the hot spot checking algorithms deems necessary (triggered by the cache miss).

Caching algorithms often try to use more sophisticated models to pre-fetch data based on a trend and store it in the cache if it thinks there a high probability it may be accessed soon e.g. in sequential video streaming or VMware virtual machine migrations where it is beneficial to cache data from the next sequential addresses and pull them into the cache at the same time as the initial access. After some period of time or when new data needs to displace older or stale data in the cache, a cache flush cleans out the old data. This may also be triggered by the hot spot detection logic determining that the data is now “cold”.

The measure of a good cache is how many hits it gets versus misses. If data is very random and scattered over the entire addressable range of storage with infrequent accesses back to the same locations, then the effectiveness is significantly lower and sometimes detrimental to overall performance as there is an overhead in attempting to locate data in the cache on every data access.

SSD Auto Tiering Basics

An automated data tiering controller treats the SSD and HDDs as two separate physical islands of storage, even if presented to the host application (and hence the user) as one large contiguous storage pool (a virtual disk). A statistics gathering or scanning engine collects data over time and looks for data access patterns and trends that match a pre-defined set of policies or conditions. These engines use a mix of algorithms and rules that indicate how and when a particular block (or group of blocks) of storage is to be migrated or moved.

The simplest “caching like” approach used by a data tiering controller is based on frequency of access. For example, it may monitor data blocks being accessed from the hard drives and if it passes a pre-defined number of accesses per hour “N” for a period of time “T”, then a rule may be employed that says when N>1000 AND T>60 minutes, move the data up to the next logical tier. So if data being accessed a lot from the hard drive and there are only two tiered defined, SSD being the faster of the two, the data will be copied to the SSD tier (i.e. promoted) and the virtual address map that converts real time host addresses to the physical updated to point data to the new location in SSD storage. All of this of course happens behind a virtual interface to the host itself who has no idea the storage just moved to a new physical location. Depending on the tiering algorithm and vendor, the data may be discarded on the old tier to free up capacity. The converse is also true. If data is infrequently accessed and lives on the SSD tier, it may be demoted to the HDD tier based on similar algorithms.

More sophisticated tiering models exist of course, some that work at file layers and look at the specific data or file metadata to make more intelligent decisions about what to do with data.

Where is SSD Caching or Tiering Applied?

Typically, SSD caching is implemented as a single SATA or PCIe flash storage device along with an operating system driver layer software in a direct attached storage (DAS) environment to speed up Windows or other operating system accesses. In much larger data center storage area networks (SAN) and cloud server-storage environments, there are an increasing number of dedicated rackmount SSD storage units that can act as a transparent cache at LUN level where the caching is all done in the storage area network layer, again invisible to the host computer. The benefit of cache based systems are that they can be added transparently and often non-disruptively (other than the initial install). Unlike with tiering, there is no need to setup dedicated pools or tiers of storage i.e. they can be overlaid on top of an existing storage setup.

Tiering is more often found in larger storage area network based environments with several disk array and storage appliance vendors offering the capability to tier between different disk arrays based on their media type or configuration. Larger tiered systems often also use other backup storage media such as tape or virtual tape systems. Automated tiering can substantially reduce the management overhead associated with backup and archival of large amounts of data by fully automating the movement process, or helping meet data accessibility requirements of government regulations. In many cases, it is possible to tier data transparently between different media types within the same physical disk array e.g. a few SSD drives in RAID 1 or 10, 4-6 SAS drives in a RAID 10 and 6-12 SATA drives in a RAID  i.e. 3 distinct tiers of storage. Distributed or virtualizaed storage environments also offer either manual or automated tiering mechanisms that work within their proprietary environments. At the other end of the spectrum, file volume manager and storage virtualization solutions running on the host or in a dedicated appliance can allow IT managers to organize existing disk array devices of different types and vendors and sort them into tiers. This is typically a process that requires a reasonable amount of planning and often disruption, but can yield tremendous benefits once deployed.

PCIe Flash versus SATA or SAS Based SSD

The impressive results being presented by the new PCIe based server or workstation add-in card flash memory products hitting the market from the likes of FusionIO and others are certainly pushing up the performance envelope of many applications, especially in transactional database applications where the number of user requests is directionally proportional to the storage IOPs or data throughput capabilities.

In just about all cases, general purpose off the shelf PCIe SSD devices all present themselves as a regular storage device to the server e.g. in Windows, they appear as a SCSI like device that can be configured in the disk manager as regular disk volume (e.g. E: or F:). The biggest advantage PCIe SSDs have over standalone SATA or SAS SSD drives is that they can handle greater data traffic throughput and I/Os as they use the much faster PCIe bus to connect directly to multiple channels of flash memory, often using a built in RAID capability to stripe data across multiple channels of flash mounted directly on board the add-in card.

To help clear up confusion for some of the readers, the primary differences between PCIe Flash memory and conventional SSDs can be summarized as follows:

Where PCIe Flash Works Well

The current generation of PCIe flash SSDs are best suited to applications that require the absolute highest performance with less of an emphasis on long term serviceability as you have to take the computer offline to replace defective or worn out SSDs. They also tend to work best when the total storage requirements for the application can live on the flash drive. Today’s capacities of up to 320G (SLC) or 640G (MLC) are more than ample for many database applications, so placing the entire SQL database on the drive is not uncommon. Host software RAID 1 is typically used to make the setup more robust but starts to get expensive as high capacity PCIe SSD cards run well north of $10,000 retail, the high price typically a result of the extensive reliability and redundancy capability of the card’s on-board flash controller. As the number of PCIe flash adapter offerings grow and the market segments into the more traditional low-mid-high product categories and features, expect the average price of these types of products to come down relatively fast.

Where SSDs Work Well

SATA or SAS based SSDs, by design, work pretty much anywhere a conventional hard drive does. For that reason we see laptops, desktops, servers and external disk arrays adopting them relatively quickly. Depending on the PCIe flash being compared to, it can take anywhere from 5-8 SSDs to match the performance of a PCIe version using a hardware RAID adapter which tends to push the overall price higher when using the more expensive SLC based SSDs. So SATA or SAS SSDs tend to be best suited to applications that can use them as a form of cache in combination with a traditional SATA or SAS disk array setup. For instance, it is possible to achieve a similar performance and significantly lower system and running costs using 1-4 enterprise class SSDs and SATA drive in a SAN disk array versus a Fibre Channel or SAS 15K SAN disk array setup. Most disk array vendors are now offering SSD versions of their Fibre Channel, iSCSI or SAS based RAID offerings.

Enterprise Flash Memory Industry Direction

At the Flash Summit we learned that between SSDs and DRAM a new class of storage will appear for computing, referred to as SCM, or storage class memory. Defined as something broader than just ultra fast flash based storage, it does require that the storage be persistent and appear more like conventional DRAM does to the host i.e. linear memory versus a storage I/O controller with mass storage and a SCSI host driver. SCM is expected to enter mainstream servers by 2013.

Kingston SSDnow VSeries and RAID

I’ve been watching the price of SSDs fall for a little while now and was intrigued by the sub $100 pricing being offered by Kingston and their MLC based 30G SSDnow Vseries, a 3G SATA II based SSD. Once they reached $80 at NewEgg.com, I dove straight in. Actually, I dove right in at the deep end and bought 6 of them as I wanted to see just how well they performed with an old 3-port NetCell RAID SATA I (1.5Gbps) card I had (some of you may remember them), along with a more recent 3ware (pre LSI) 8 port hardware RAID SATA II (3Gbps) card to see just how fast they were capable of going.

I definitely saw some interesting results. I should say upfront that this was a quick and dirty test to see if you can use these low cost devices in any RAID configuration, which they were not really designed for, so please take that into consideration as you read this as this was more an academic exercise to prove a point when mixing old with new.

Inside the Kingston 30G SSDs

For fun, I disassembled one of the SSDs at the end of my tests (don’t worry Kingston; not going to try and return it) and I was surprised to see lots of fresh air inside. These things are tiny (duh! I hear a few folks saying). They kept the cost low by removing much of the length of the electronic circuit card, with the Toshiba T6UG1XBG controller on one side with a Micron 9TB17-D9JVD DRAM memory, and four 8GB Toshiba flash devices on the underside. This combination according to the published specs should result in streaming speeds of up to 180MB/s read, and 50MB/s write, so definately more on the entry level side versus what you’d see in some of the more expensive versions.

Testing with Legacy RAID Cards

I picked two old legacy cards to work with: a PNY NetCell 3-port SATA RAID 3 and a 3ware 9650SE 8 port SATA II RAID card. I was really pleased that the SSD worked just fine with my old PNY NetCell RAID card in one of the older PCI test rigs. The Kingston looked just like a regular SATA drive as advertised and the NetCell card auto-configured it nicely into a single drive before booting into Windows. No surprises there.

However, given my primary performance test system was a SuperMicro X8DT6 PCIe setup which had no PCI slots, I quickly moved onto testing with the PCIe x4 Gen1 3ware 9650SE SATA2 RAID adapter as I wanted to see just how well traditional RAID adapters work with SSDs. Of course, having just attended the Flash Summit and was told 100 times over that most of today’s controllers and applications don’t take advantage of SSDs the way they should be doing, I really had to see it for myself. I was also curious why most benchmarks used host based software RAID, and following this am beginning to see why.

Quick Test RAID Performance

Low cost SSDs don’t really work that well in a traditional hardware RAID setup. On the positive side, using the 3ware RAID adapter, a single drive outperformed against it’s published spec of 180MB/s coming in at 200MB/s on streaming reads. The performance test results were based on IOmeter 2006 running under Windows XP and were all performed using raw volumes, Q size of 1 and only two block sizes; 1M for streaming reads and writes, and 512 bytes for random I/Os. The drives were set up in simple RAID0 configurations and with the write caches left on.

While the Kingston SSD’s I used were not really designed for RAID applications, I saw no reason why I shouldn’t at least see linear performance increases as I added drives from the RAID controller. The quick results published here in the two charts did convince me as an novice SSD user that throwing these into a system without consideration of all the various moving parts doesn’t automatically make for a high performance system. In fact, it could even slow it down. While 1-2 drives in a RAID0 configuration did reasonably well on streaming reads and significantly higher that ye old hard drives, this quickly hit a ceiling at around 550MB/s with not much of a benefit in performance past 4 SSDs. Not bad for an 3-4 year old RAID card that wasn’t really optimized for low cost flash memory. However, the writes were a different story, barely making it to 50MB/s (with some very high max latencies >4 seconds in some cases). IOPs were a real mixed bag with decent results at around 4500 IOPs, but never really seeing the increase as more SSDs were added. IOPS writes were again a mixed bag and not really related to the number of drives at all, with the performance actually falling off as more drives were added beyond 3 SSDs. Nothing conclusive here of course, but definitely illustrates that the caching algorithms of the older RAID cards were not optimized for a 4000+ IOPs device hanging off each port!

Conclusion

This exercise with legacy RAID components for me confirms what we saw at the Flash Summit i.e. even with all the excellent work going on in the industry on making SSDs perform to their true potential, it will be a while before the whole PC eco-system will be able to take full advantage of them. While these devices work well as a low cost SSD for entry level single drive systems, it’s unlikely they can offer much when used with legacy RAID adapter cards. A lot of effort has gone into making a flash drive look just like a regular hard disk drive which has fooled a few of us end users into thinking it’s just a faster hard drive, whereas it’s a very different beast. Lesson learned here is that applications, software drivers and of course RAID firmware all need to be re-written to ensure that they are SSD literate. Bottom line is you need to use a modern RAID card that is SSD literate. LSI and Adaptec are moving in this direction with their built in SSD caching technology, but general concensus is that hardware RAID engines in all types of equipment are going to need an overhaul given the quantum leap in performance of SSDs over traditional hard drives.

Flash Memory Summit 2010

The Flash Memory Summit was well attended this year with attendance up over 50%. A total of around 2000 people showed up versus the planned 1200 according to the show organizers.

It was first time for me, but I definitely thought the show was well worth attending (Note: I’m an independent and not associated with the show in any way). Not only did it bring folks like me up to speed quickly on what was happening, but it also provided a forum for discussing where this industry is heading and more importantly, what the various vendors and researchers are up to in this space. We even had Steve Wozniak (now at FusionIO) as a keynote speaker, entertaining us with his various brushes with memory from the first DRAM chips from his computer club days to the hilarious pranks he used to play on folks, including Steve Jobs in the early years.

In summary… things are definitely looking up for the flash industry.

Technology tidbits of interest:

  • PCM (Phase Change Memory) as a future technology looked very interesting, especially given it’s significantly higher write endurance (1M+) though not meeting density expectations yet
  • MLC 3 -bits per cell using 25nm semiconductor process was announced by Intel and Micron; time for higher densities (albeit at worse write cycle levels)
  • Most SSD manufacturers appear close to having a PCIe Flash add-in card solution
  • Automatic tiered storage and auto load balancing are on the rise and badly needed wherever flash shows up in general computing
  • The term “short-stroking SSDs” came up which was really another way to say only use 75% of the capacity of an MLC-SSD in order to provide the optimum performance
  • The concept of SCM – storage class memory – due in 2013 onwards also came up on a number of occasions as being a new class of memory mapped flash versus the current storage I/O model commonly used today (i.e. flash becomes an extension of DRAM space in this case)
  • FusionIO were giving away nice looking capes, monkeys and T-shirts. They also had a very cool looking display with a huge number of video streams.

One point of interest on the SSD adoption side; Intel claim almost all datacenters want MLC as the TCO for SLC solutions doesn’t compute for most end IT managers (lots of discussion around that one). It was pretty clear throughout the show that an enterprise class MLC solution was badly needed, tied of course to the basic cost advantages over SLC.

Other new products of interest introduced at the show included FusionIO’s hybrid flash card with external Infiniband (IB) I/O (40Gbps) connectivity, the ioSAN, which was simply a Infiniband PCIe controller integrated with a Fusion Duo card using a PCIe switch . While there is no onboard bridging offered, the system software driver does allow the card to transfer data to and from the flash memory as a target IB device, offering a very fast method to move data onto or off the card, and potentially offer support for virtual machine migration down the road. Future versions will offer 10G iSCSI support also.

Another cool little device was the SANDisk® 64GB iSSD SATA-flash module in the form of a silicon BGA package for embedded applications, no bigger than the size of a US postage stamp. Given the increased availability of SATA ports appearing in embedded chipsets, this makes for a very nice component for many applications beyond just the mobile space.

The only concern that continues to come up consistently is the write endurance problem, along with how end users and applications still need to understand the differences between SSD and traditional hard drives to achieve the most out of them. As noted in one of the keynotes, it has taken almost 30 years to go from a few 10s of IOPS on a HDD to approximately 280+ IOPs. All applications and operating systems have conditioned themselves around this level of performance, so it’s not surprising that a sudden jump by 10-20x is not being seem in anything other than raw benchmarks. RethinkDB™ was one company that had gone to the trouble of rewriting a SQL database application to take advantage of SSDs, and managed to push operations per second up from 200 to over 1200 for the same hardware.

There is still the serviceability issue however for enterprise class and data centers whose IT managers have been well trained by the hard drive industry in ways not always well suited to SSDs. While there are tremendous gains from technologies such as PCIe Flash cards from the likes of FusionIO and OCZ, there is also the question of how to replace them when they go bad without turning off the server and migrating data for example. This will continue to drive new ideas and solutions for a while in the enterprise space.

All in all, a great show and well worth attending.