Virtualization is an increasingly popular way to increase power efficiency in the datacenter by giving each individual server more work to do, so that the machine wastes less power on idle cycles. Processors from Intel and AMD both offer hardware-level support for virtualization, with the result that virtualization is moving down-market while commodity x86 processors continue to march up-market. So what’s a datacenter manager to do if she’s trying to chose from among x86-based virtualization offerings from Microsoft, VMWare, and XenSource?
Industry benchmark consortium SPEC plans to help buyers pick the highest performance virtualization option by eventually offering a set of standard benchmarks for evaluating virtualization solutions. The idea is to see how well each solution performs with different numbers of virtual machines running different mixes of workloads, so that buyers can get a feel for the relative performance of each virtualization offering on the same workloads. To this end, SPEC has formed a new working group to begin developing such a benchmark suite.
The current roster of subgroup members includes AMD, Dell, Fujitsu Siemens, Hewlett-Packard, Intel, IBM, Sun Microsystems, and VMware, but SPEC is soliciting more members who may be able to contribute (Microsoft?).
Currently, VMWare’s relatively new benchmark, dubbed VMmark, is the SPEC-approved option for comparing virtualization solutions. VMmark works by firing up multiple virtual machines and then running a different workload in each one. In contrast to a normal benchmark workload, which seeks to max out some aspect of the system, each individual VMmark workload is run at less than full utilization. The stress on the system comes from all of those only moderately heavy workloads running simultaneously.
As you can probably guess from even this brief description, benchmarking virtual machine performance is something of a challenge. It’s important that the mix of workloads on the different virtual machines faithfully reflect the workload mix that the datacenter will be using in a production environment. This is because different VM solutions will have different types of bottlenecks at different layers of the hardware/software stack, and those bottlenecks will affect different types of workloads in different ways.
Further readingVMWare, VMmark: A Scalable Benchmark for Virtualized Systems [PDF]SPEC.orgBroad alliance backing VMware desktop visionVirtual machine shootout: VMware vs. Virtual PC