Kammo’s Tech Blog

Virtualization has really shined in the ability to minimize the physical footprint in the datacenter by sharing physical hardware resources. Not too long ago, the common practice was to virtualize the servers with a low workload, while keeping the heavy hitters physical. One of the greatest reasons for not virtualizing a server is the applications I/O requirement. Once virtualized, the virtualization layer simply couldn’t accommodate high I/O loads. When attempting to do so, the VM would claim a high amount of overhead to process the I/O queues, hindering the overall performance of the specific VM. This hindrance sometimes causes a bottleneck in a dense virtualization environment, impacting the neighboring guests utilizing the same hardware. There are several options now that allow you to virtualize those heavy workloads that demand a lot of hardware.

In VMware, the default storage controller will be one of the virtual LSI Logic or Bus Logic adapters. This can be changed during the deploy wizard, or after virtual machine guest creation. Let’s go over our options for virtual machine storage adapter options.

LSI Logic

The LSI Logic virtual storage adapter is tried and true and is the most common when installing a VM. This adapter provides the necessities for VM storage, but does not fit the bill for high workloads.

There are two LSI Logic options: Parallel and SAS. Parallel is the default and recommended, while SAS is intended for Microsoft Windows Server 2008 in conjunction with clustering.

Buslogic

Typically not recommended for VMware guests, however, when necessary, use the driver from VMware, not to OS default. Like LSI Logic, provides the standard use for VM Storage.

Raw Device Mapping (RDM)

Allows access and management of raw SCSI devices as VMFS files. RDM’s act as a proxy between the VM guest and the raw device. The RDM files contain metadata that manages the access to the physical device, as well as the redirection.

The use of RDM’s does not seem to gain a performance benefit as much as it does a “features” benefit. Your features will come in the form of SAN replication, running SAN aware applications on VM disks, clustering, and one that’s not so practical: attaching a LUN larger than the maximums allow for a VMDK.

VMware Paravirtual SCSI (PVSCSI)

For high I/O scenario’s (greater than 2,000 IOPS and issuing greater than 4 outstanding I/O’s), PVSCSI offers significant reduction in CPU utilization by bypassing the virtualization layer and going strait to one of the physical storage adapters shared by the hypervisor. The ability to share the physical storage adapters is one of the greatest reasons for choosing PVSCSI since all of the virtual machines can share the same storage adapter. Keep in mind that the more sharing that occurs on one adapter, the more latency you will see.

There are excellent benefits to using PVSCSI; however, there are a few drawbacks as well. Fault Tolerance is unavailable for any guest that utilizes a PVSCSI adapter. For guests using Microsoft Cluster Service (MSCS), you will need to use another adapter. It is also not advisable to use PVSCSI with Direct Attached Storage (DAS).

The driver for PVSCSI is included in the VMware Tools installation and in a floppy image. If you are converting a guest to use PVSCSI, then you should be able to add the hardware with no problem. Keep in mind, though, that you will need to install the PVSCSI adapter and boot the OS so the driver will install, then shut it down and move the disks to the new adapter. If you move the disks to the new adapter before loading the driver, your mileage may vary when booting the OS since Windows does not include a native driver out of the box. For new VM’s, you may create the VM with the PVSCSI adapter, then load the driver when it prompts for storage using the provided floppy image.

VMDirectPath

Although the concept of virtualization is to be hardware independent and share the physical layer amongst a number of guests, VMDirectPath has come to us as a new feature in vSphere. This feature will allow you to utilize up to two PCI(e) devices on the host server for connecting to a specific VM. This eliminates sharing of the device and gives its undivided attention to one virtual machine, eliminating the latency caused by sharing the hardware resources. VMDirectPath applies to more than just storage adapters, including the ability to assign a 10Gb NIC to one VM. Even though pretty much any PCI(e) device can be utilized, only a handful of devices are supported for this feature. Be sure to check VMware’s HCL, otherwise your support issue may not get resolved when you call in.

Before diving into VMDirectPath, consider the requirements. VMDirectPath requires the host server to have Intel’s VT-d or AMD’s IOMMU technology enabled. Also remember that this feature connects your VM to a specific hardware device on the host. vMotion, Storage vMotion, Fault Tolerance, Snapshots, VM Suspend, and Device Hot Add will no longer be available to your VM. Because of this, it becomes an administration headache when performing normal maintenance. For example, when patching a vSphere host, the reboot will require a shutdown of any guests using the VMDirectPath feature, since you will not be able to move it off to another host when entering maintenance mode. This also does not provide any redundancy during a hardware failure for the same guests. The guests using the feature will have to wait for the hardware to be fixed before it can come back online, the same as with a physical server.

When I have a VM guest that has a high I/O load, my first step is typically changing to the PVSCSI adapter for the VM guest. In my opinion it provides the most “bang for your buck” without losing too many virtualization features. If I had to use VMDirectPath, I would utilize a single host, not in a cluster, with only a few guests loaded on it for the purpose of only assigning specific hardware to each one. For normal workloads, use the default storage adapter for the guest you are installing. When working with templates, I would not recommend using anything but the default when configuring your base templates, unless you are creating a specific template for a high workload. If needed, you can always change your storage adapter post-deployment.

The usual mount -o ro,loop image.iso /mnt/iso/, which you’re used to on Linux, can’t be used on AIX. On AIX you dd the ISO file into a raw Logical Volume, then mount the LV as a filesystem. Here are the steps for copying the ISO named image.iso into the folder /mnt/iso/ (a JFS filesystem):

1. Create a filesystem with size slightly bigger than the size of the ISO image. Do NOT mount the filesystem yet. Here I assume we’re dealing with a DVD ISO image ~ 4.7G:

# crfs -v jfs -g rootvg -a size=5000M -m/mnt/iso -Ano -pro -tno
Based on the parameters chosen, the new /mnt/iso JFS file system
is limited to a maximum size of 134217728 (512 byte blocks)

New File System size is 10354688

2. Get the logical volume name associated with the new filesystem:

# lsfs | grep iso
/dev/lv01 -- /mnt/iso jfs 10354688 ro no no

(we see that the logical volume name is lv01)

3. dd the ISO image into rlv01 (raw lv01):

# dd if=/path/to/image.iso of=/dev/rlv01 bs=100M
43+1 records in.
43+1 records out.

(Note the r before lv01! This is important)

4. Mount the file system:

# mount -v cdrfs /mnt/iso

(Note the filesystem type! Unlike the crfs command above, where we used -v jfs, now we use -v cdrfs)

5. Do whatever you do with the files in /mnt/iso.

6. When finished, unmount the filesystem:

# umount /mnt/iso

7. And finally, remove the filesystem:

# rmfs /mnt/iso
rmlv: Logical volume lv01 is removed.

Notes:
*) The above was tested on AIX 5.3

-b filename Block special file
-c filename Special character file
-d directoryname Check for directory existence
-e filename Check for file existence
-f filename Check for regular file existence not a directory
-G filename Check if file exists and is owned by effective group ID.
-g filename true if file exists and is set-group-id.
-k filename Sticky bit
-L filename Symbolic link
-O filename True if file exists and is owned by the effective user id.
-r filename Check if file is a readable
-S filename Check if file is socket
-s filename Check if file is nonzero size
-u filename Check if file set-ser-id bit is set
-w filename Check if file is writable
-x filename Check if file is executable

A sample script to check for the existence of a directory

The variable set called directory contains a folder in the present working directory defined by the ./. You could define it from the root of the server as well. EX: directory="/var/scripts/folder"

The script:
#!/bin/bash
directory="./folder"
#
if [ -d $directory ]; then
echo “Directory exists”
else
echo “Directory does not exists”
fi

Since the release of Windows 7, VMware Virtual Infrastructure Client needs special attention for installation on this platform.  Keep in mind that VMware VI Client is not supported on Windows 7; however, you can follow this guide to get it installed and functioning correctly.

1. The following ’system.dll’ file is required. This can be downloaded via the hyperlink below.

system.dll (this file has been zipped)

2. Once downloaded, either open the zip file or unzip it and place the ‘system.dll’ file into one of the following directories depending on the version of Windows 7 being used, which would be either 32 or 64 bit.

x86 Directory Path = C:\Program Files\VMware\Infrastructure\Virtual Infrastructure Client\Launcher\Lib

x64 Directory Path = C:\Program Files (x86)\VMware\Infrastructure\Virtual Infrastructure Client\Launcher\Lib

Note: If the ‘Lib directory does not exist, create it within the ‘Launcher’ directory.

3. Edit the VpxClient.exe.config file which is located in the ‘Launcher’ directory, making the changes as outlined below and as seen in the image.

<runtime>
<developmentMode developerInstallation=”true”/>
</runtime>

Once the file has been edited, save the changes and close the file.

Note: If you receive a warning stating you cannot save the file, either reboot the computer or ensure that any VMware related software is closed and try again. If this doesn’t work then copy the file out of the “Launcher” directory, edit it, save it and then copy the file over the original one.

4. Next, edit the system properties of your machine. This can be achieved by right clicking on ’Computer’ (was usually known as ‘My Computer’ in older Microsoft OS versions) and selecting ’Properties’. Select ‘Advanced System Settings’

5. Select the ‘Environment Variables’ button

6. Add a new System variable. To do this, click ‘New…’ and add the following details.

Note: Depending on the Windows 7 OS version will depend on which Variable is required. Ensure the same path is used as per step 2.

Variable name = DEVPATH
Variable value (x86) = C:\Program Files\VMware\Infrastructure\Virtual Infrastructure Client\Launcher\Lib
Variable value (x64) = C:\Program Files (x86)\VMware\Infrastructure\Virtual Infrastructure Client\Launcher\Lib

Once this text is entered into the required fields, click OK, OK, OK and then close the View basic information about your computer properties window.

7. Launch the VI Client.

Note: You should run the VI Client with administrator privileges.

Follow me on Twitter! - http://twitter.com/kameronkenny

Finally today I had implemented NIC bonding (bind two NICs so that it works as a single device for load balancing and/or fault tolerance).  The idea here is to improve performance by pumping out more data from both NICs via NIC bonding/teaming

This box act as datastore server for my test VMware environment.  Each virtual machine is run on top of an ESXi box, and the storage is stored on this server.  Each virtual disk is anywhere between 15GB and 50GB.  Since the virtual disks are stored on a separate piece of hardware from the virtualizing hardware, we must have a high level of throughput for our network traffic.  I am using CentOS 5.3 for the purpose of this document.  This process should readily work for most if not all Red Hat based distro’s.  Click to continue »

Need to know the release version (Update 3, Update 4, etc.) of the ESX 3.5 servers in an environment, but don’t want to consult a chart of build numbers? Just execute this command as root:

esxupdate query | grep -o 'ESX Server 3.5.0 Update.*' | sort | tail -n 1

If the server is a build of ESX prior to Update 1, the command will return nothing. It’s nice to have the update version in this familiar format for documentation.

Okay, you probably already know the ol’ Command-Shift-3 shortcut for taking a screen capture of your entire screen, and you may even know about Command-Shift-4, which gives you a crosshair cursor so you can choose which area of the screen you want to capture. But perhaps the coolest, most-secret hidden capture shortcut is Control-Command-Shift-3 (or 4), which, instead of creating a file on your desktop, copies the capture into your Clipboard memory, so you can paste it where you want.

I’m excited to be participating in the 2009 Outrun the Sun Race Against Melanoma in Indianapolis on Saturday, June 6. You can share in the excitement by helping support my involvement in this important initiative. All you need to do is log on to my personal fundraising page to learn more about melanoma education and research. It’s easy!

Please forward this email to anyone in your address book that you feel would want to help in such a generous cause!

Did you know:
Melanoma affects people of every age and every ethnicity?
Melanoma is the number one cancer in people ages 25-29?
One American dies of melanoma nearly every hour?

All gifts are important, welcome and appreciated! Thanks very much for your help.

Protect your skin. Protect yourself. Protect your life.

Follow This Link to visit my personal web page and help me in my efforts to support Outrun the Sun, Inc.

******************************************************************************
Some email systems do not support the use of links and therefore this link may not appear to work. If so, copy and paste the following into your browser:
http://raceagainstmelanoma.kintera.org/faf/r.asp?t=4&i=310059&u=310059-252960692
******************************************************************************