Virtualizing High Performance Storage
Written by kammo on January 4th, 2011Virtualization has really shined in the ability to minimize the physical footprint in the datacenter by sharing physical hardware resources. Not too long ago, the common practice was to virtualize the servers with a low workload, while keeping the heavy hitters physical. One of the greatest reasons for not virtualizing a server is the applications I/O requirement. Once virtualized, the virtualization layer simply couldn’t accommodate high I/O loads. When attempting to do so, the VM would claim a high amount of overhead to process the I/O queues, hindering the overall performance of the specific VM. This hindrance sometimes causes a bottleneck in a dense virtualization environment, impacting the neighboring guests utilizing the same hardware. There are several options now that allow you to virtualize those heavy workloads that demand a lot of hardware.
In VMware, the default storage controller will be one of the virtual LSI Logic or Bus Logic adapters. This can be changed during the deploy wizard, or after virtual machine guest creation. Let’s go over our options for virtual machine storage adapter options.
LSI Logic
The LSI Logic virtual storage adapter is tried and true and is the most common when installing a VM. This adapter provides the necessities for VM storage, but does not fit the bill for high workloads.
There are two LSI Logic options: Parallel and SAS. Parallel is the default and recommended, while SAS is intended for Microsoft Windows Server 2008 in conjunction with clustering.
Buslogic
Typically not recommended for VMware guests, however, when necessary, use the driver from VMware, not to OS default. Like LSI Logic, provides the standard use for VM Storage.
Raw Device Mapping (RDM)
Allows access and management of raw SCSI devices as VMFS files. RDM’s act as a proxy between the VM guest and the raw device. The RDM files contain metadata that manages the access to the physical device, as well as the redirection.
The use of RDM’s does not seem to gain a performance benefit as much as it does a “features” benefit. Your features will come in the form of SAN replication, running SAN aware applications on VM disks, clustering, and one that’s not so practical: attaching a LUN larger than the maximums allow for a VMDK.
VMware Paravirtual SCSI (PVSCSI)
For high I/O scenario’s (greater than 2,000 IOPS and issuing greater than 4 outstanding I/O’s), PVSCSI offers significant reduction in CPU utilization by bypassing the virtualization layer and going strait to one of the physical storage adapters shared by the hypervisor. The ability to share the physical storage adapters is one of the greatest reasons for choosing PVSCSI since all of the virtual machines can share the same storage adapter. Keep in mind that the more sharing that occurs on one adapter, the more latency you will see.
There are excellent benefits to using PVSCSI; however, there are a few drawbacks as well. Fault Tolerance is unavailable for any guest that utilizes a PVSCSI adapter. For guests using Microsoft Cluster Service (MSCS), you will need to use another adapter. It is also not advisable to use PVSCSI with Direct Attached Storage (DAS).
The driver for PVSCSI is included in the VMware Tools installation and in a floppy image. If you are converting a guest to use PVSCSI, then you should be able to add the hardware with no problem. Keep in mind, though, that you will need to install the PVSCSI adapter and boot the OS so the driver will install, then shut it down and move the disks to the new adapter. If you move the disks to the new adapter before loading the driver, your mileage may vary when booting the OS since Windows does not include a native driver out of the box. For new VM’s, you may create the VM with the PVSCSI adapter, then load the driver when it prompts for storage using the provided floppy image.
VMDirectPath
Although the concept of virtualization is to be hardware independent and share the physical layer amongst a number of guests, VMDirectPath has come to us as a new feature in vSphere. This feature will allow you to utilize up to two PCI(e) devices on the host server for connecting to a specific VM. This eliminates sharing of the device and gives its undivided attention to one virtual machine, eliminating the latency caused by sharing the hardware resources. VMDirectPath applies to more than just storage adapters, including the ability to assign a 10Gb NIC to one VM. Even though pretty much any PCI(e) device can be utilized, only a handful of devices are supported for this feature. Be sure to check VMware’s HCL, otherwise your support issue may not get resolved when you call in.
Before diving into VMDirectPath, consider the requirements. VMDirectPath requires the host server to have Intel’s VT-d or AMD’s IOMMU technology enabled. Also remember that this feature connects your VM to a specific hardware device on the host. vMotion, Storage vMotion, Fault Tolerance, Snapshots, VM Suspend, and Device Hot Add will no longer be available to your VM. Because of this, it becomes an administration headache when performing normal maintenance. For example, when patching a vSphere host, the reboot will require a shutdown of any guests using the VMDirectPath feature, since you will not be able to move it off to another host when entering maintenance mode. This also does not provide any redundancy during a hardware failure for the same guests. The guests using the feature will have to wait for the hardware to be fixed before it can come back online, the same as with a physical server.
When I have a VM guest that has a high I/O load, my first step is typically changing to the PVSCSI adapter for the VM guest. In my opinion it provides the most “bang for your buck” without losing too many virtualization features. If I had to use VMDirectPath, I would utilize a single host, not in a cluster, with only a few guests loaded on it for the purpose of only assigning specific hardware to each one. For normal workloads, use the default storage adapter for the guest you are installing. When working with templates, I would not recommend using anything but the default when configuring your base templates, unless you are creating a specific template for a high workload. If needed, you can always change your storage adapter post-deployment.




