Government agencies almost by definition are at the forefront of the big data challenges facing all large enterprises. How to access large amounts of data, make sense of it and act on it in a timely manner in dynamic environments is as much as an issue for intelligence agencies, homeland security, first responders, and health care agencies as it is for the financial services industry and Amazon.com. In the middle of the big data debate is whether solid state flash technologies live up to their hype as the answer to every big data problem. In fact, some have said, “Flash changes everything.” What government CIOs and their teams need to realize is that flash--in the right circumstances--is indeed a game changer. However, it is not always the most cost-effective technology for all workloads. Let’s look at some examples where flash does shine:
In hybrid storage arrays (which include both traditional spinning disks and some amount of flash storage), flash has already proven its value, particularly in server virtualization and desktop virtualization environments helping with small block, random reads and to some extent, random writes. The most effective hybrid array vendors have figured out how to incorporate flash into their ecosystems, without forcing you to manage another tier of storage. Lately, however, we are witnessing the emergence of dedicated flash vendors, whose appliances are being used to handle the more demanding enterprise workloads.
The most mission critical enterprise application workflows depend heavily on high transaction rate, low latency online transaction processing databases. These applications and databases are also dominated by small block (8 Kbyte or 16 Kbyte) random input/output, generally mostly reads vs. writes. In the past, database administrators had to go to great lengths to keep these databases working properly. It was common practice to run these databases in raw partitions (without a file system); they would have to pay close attention to the RAID (redundant array of independent discs) configurations used, often giving up storage efficiency for performance; they would reorganize databases on regular basis to remove “chaining” (which is disruptive); and the more desperate would resort to short-stroking disks, forcing them to pay for far more disk capacity than they wanted, just to get enough disk spindles to provide the performance they needed.
In this type of environment flash can be transformational. It is now possible to get an all-flash appliance that holds several Terabytes of capacity and which delivers almost 400,000 input/output per second. Think about it: This small device replaces almost 2,000 spinning 15K RPM disk drives, along with the power, cooling and rackspace required to support them. Furthermore, because these devices deliver transactions at near sub-millisecond latencies, database administrators no longer need heroic actions (especially the time consuming reorgs) to make their databases perform. Using these types of devices for the more demanding tables and indexes, and then using slower speed 7.2K RPM disks for colder data provides the best of both worlds: the best $/IOPS AND the best $/GB, lowering the overall system cost.
Another area where flash can be transformational is in tactical and first responder vehicles. In those environments, spinning disks are generally not an option. In fact, in the past these vehicles had to park to process large amounts of data, because spinning disk drives could not take the shock and vibration. Flash storage technologies now allow the command and control applications running in these vehicles to process data on the move, even in harsh environments and over rugged terrain. Flash is not only valuable in combat vehicles. In fact, anywhere space, temperature and vibration are issues, flash offers a better solution. That includes aircraft, smaller watercraft, helicopters and submarines. The same is true for first responder vehicles here at home, to improve the situational awareness of police officers and help them do their jobs more safely and effectively.
For streaming data, flash may not be a good fit. If the data is large block, with some randomness, there may be some benefit, but it is not nearly as cost-effective as in the small block, random I/O world. Large amounts of streaming data quickly overrun the capacities of flash devices. Generally, SATA or near-line SAS disks are better suited for these environments and far more cost-effective.
So, while flash doesn’t “change everything” where it does fit its impact is powerful. In the environments mentioned above, flash technology adoption is beginning to kick the fast (10K and 15K RPM) disk market to the curb. If that trend continues, we may witness the demise ofthat market, particularly as newer, more capable flash technologies come on the scene and price and latency continue to fall.