Vmware Log Analyzer

[Peppin Kishore]

ThanksI currently have that product as well. I have reporting available from a multitude of products I just cant seem to get over 25,000 IOPS during my tests. I am trying to figure out if its an issue with the testing product VMware I/O analyzer.

The CBRC uses a ramdisk only, the flash cache can make use of an SSD in Enterprise Plus installs. Keep in mind that if your testing tool is generating random I/O then you will not see a performance improvement, as the Filesystem Device Switch module in the vmkernal looks to see if the data has been cached before, before determining routing to either the Virtual Flash Filesystem or a disk based datastore.

I will take a look at the server and check the specs on the slots in my server. The total theoretical bandwidth that I would need for this test would be 95000 IOPS x 512 bytes/s which comes up to about 50 MB/s. With the understanding that I probably will not reach 95000 but I think I should get more that 25000.

I/O Analyzer is a wrapper around IOmeter where the predefined profiles have particular values to them. I made an effort to parse through each profile for a past project and found a few interesting points:

If the ACK is not coming fast enough, then you are looking at a re-transmit which would account for the slow down as it is waiting for an ACK for each packet before sending the next instead of sending them four or more at a time.

The overall goal is to test for an existing project that requires around 40000 IO which is about 93 percent write at 7.5 KB. My peak bandwidth is around 600 MB/s both read and write which is were the bottleneck will be. I only have 4 1 Gb NICS on my backend storage. These IOP requirements only happen about 3 times a day and typically last an hour or less. This is why I am looking at server side cache to help offload my storage. The other alternative is to stay physical :(, buy another SAN that supports 10GB, or cache. The good news is that I am in the process of building up a test VM that will mimic the production physical server. I can then use some real world workloads for testing.

To use the Virtual USB Analyzer, you first need to capture a log of some USB protocol traffic. This tutorial will show you how to use the logging built in to VMware's virtual USB stack. You'll need either VMware Workstation, VMware Fusion, or the free VMware Player as well as a virtual machine and USB device you want to capture data from.

Note that you can capture the log file using any host operating system, and your virtual machine can be running any operating system that supports USB. However, you'll probably want to analyze the resulting logs on a Linux machine- the vusb-analyzer tool may be difficult to run on Windows or Mac OS. See the system requirements.

For this step, you'll need to edit your virtual machine's configuration file. This is the text file with a ".vmx" extension, which specifies all of a virtual machine's devices, shared folders, debug options, and other settings. On Windows, you'll want to use Wordpad or any programmer's editor. On Linux or Mac OS, any plain text editor should work.

This .vmx file is the same file you open in VMware Workstation or Player in order to load the virtual machine. In VMware Fusion, the VM may be a bundle, a directory which appears as a single file in the Finder. In this case, you'll need to open the bundle by right-clicking it and selecting "Show Package Contents."

Enable debugging for this VM. This switches to a different version of the VMware virtual machine runtime which is compiled with debug checks enabled. This is the same as setting the "Gather debugging information" advanced option in VMware Workstation to "Full." Note that this option only takes effect the next time you power on the virtual machine.

This is optional, but we'll increase the length of the payload data that we log for each USB packet. By default, only 80 bytes (5 hex dump lines) of data are logged for each packet. To see the entire contents, we'll increase this to 8192. This will make your log files significantly bigger, so if you don't need the full packet contents you can set this to a small number to decrease logging overhead.

This is also optional. By default, some virtual machines (Windows NT and later) will get a virtual USB mouse device by default. From the guest's point of view it looks like a normal USB device, but it is emulated entirely in software. This device will show up in the USB log output. If the extra output gets in the way, you can use this option to disable the virtual USB mouse. Note that with the virtual USB mouse disabled, mouse functionality in the VM may be reduced.

Also note that this option only tells the virtual USB subsystem when to attach a new virtual mouse. If the mouse is already attached, you'll need to remove the corresponding "usb:" lines. See the example below.

So, to enable debugging, enable the USB analyzer, and set the line length, we'll add these three lines to the VMX file. Note that the VMX file must not contain duplicate values for a single key. Make sure these config options don't already exist before adding them, or your VM will fail to start.

When you power on or resume this VM, you should start to see USBIO log lines in vmware.log for each USB packet that traverses the virtual USB stack. On most Windows VMs, you will immediately see some USBIO log lines for the two virtual peripherals that automatically attach to every VM: The virtual hub and virtual mouse. (If you did not elect to disable the virtual mouse, that is.) Some example USBIO log entries:

They are designed to be somewhat human readable even without the Virtual USB Analyzer tool. Some very basic Chapter 9 protocol decoding is performed in the virtual USB stack, and packet data is dumped in hex and in ASCII. The "Down" lines indicate a request that is on its way from the guest OS to the (virtual) hardware, and an "Up" is a response or completion which is returning from the hardware back to the guest OS.

For this tutorial, we'll capture the USB bus traffic that results after plugging in a USB flash drive. We'll capture logs from both Windows and Linux virtual machines, so we can compare the two. We'd like to know what, if anything, is different about how Windows and Linux access USB storage devices.

So, let's start by giving it a single log file. The UI should appear immediately along with the first section of the log file. If you're viewing a large log file, it will continue to load in the background. The progress bar at the bottom-right corner of the window indicates loading progress. It disappears when the file is fully loaded.

The transactions we've been talking about so far are also known as USB Request Blocks (URBs). This is the typical unit of communication used by USB drivers. URBs are distinct from the actual packets that appear on the physical USB wire.

When a USB driver submits a URB to the operating system's USB stack, the USB stack converts this URB into a set of lower-level primitives (Transfer Descriptors) and sends it off to the USB host controller. The TDs tell the host controller to poll the device. The host controller continues to poll the device, in hardware, until the transfer is complete. When the last TD finishes, the host controller fires an interrupt, and the operating system completes the URB.

Since the Virtual USB analyzer only sees URBs, not wire-level packets or TDs, it is a higher level analyzer than a typical hardware analyzer like those made by Ellisys or LeCroy. It is more similar to USB Snoopy, which captures URBs as they traverse the Windows driver stack.

Each URB always has two entries in the list: One showing when the driver submitted it, and one showing when it completed. Most URBs complete pretty quickly, and their duration is a good measure of the latency we're experiencing from the device and the host machine's USB stack. This isn't always the case, though, as we'll see later.

The rightmost column contains a summary of the decoded high-level protocol information in that transfer. Here's another example: the completion of a GET_DESCRIPTOR request for reading the configuration descriptor on the VMware Virtual Hub device.

On the left side you'll find a full hex dump of the transfer data, on the right is the annotation provided by a protocol decoder plugin. In this case, it's disassembling a few of the standard USB Chapter 9 descriptors. You can copy and paste text from either side of the window.

The timing diagram is a graphical visualization of the time, quantity, and size of URBs. The horizontal axis is time. The vertical axis is arbitrary, but transactions are first grouped by device and endpoint, then stacked tightly for the best space utilization.

Let's take a closer look at the beginning of the Linux USB storage log we captured above. If you skipped the capturing a log file section above, download one of the sample log files. The log file we're looking at here is linux-storage-read.

This is a typical diagram showing an enumeration sequence. In the "Dev 0, EP0" column, you can see the SET_ADDRESS URBs which give two other devices addresses 3 and 4. Immediately afterward, many small control requests are sent as the device's driver loads and initializes. All of these packets are very quick, and have very little data content.

You'll also notice a few red stripes on the graph. These are errors- in this case, the USB storage driver sent an unsupported command, so the device reported a stall. The red stripe on EP0 is an unsupported class-specific request, which also reported a stall.

Well, the timing diagram always places the left and right edges of a box according to the submission and completion timestamps of a URB. If multiple URBs are pending, the timing diagram will stack them vertically. The vertical stacking order is arbitrary, it just tries to fill space as efficiently as possible. As a general rule, the taller a stack is the more URBs are being run concurrently by the device's driver.

3a8082e126