Let me explain the problem, as some customers find it, and some do not. Here’s the issue: You start your virtual infrastructure by purchasing 3 hosts. Let’s say that the 3 hosts have Intel E5310 processors in them. You start to load up your VM’s on that cluster over the next 6 months. When the cluster is full you go back to your OEM vendor of choice, only to find that they no longer make that server model with a E5310 processor. The same model from the OEM now contains a Intel E5420 CPU. There are some differences in this CPU from those in the original hosts , specifically the instruction set (SSE4). If you try to add that new host to the existing cluster you will quickly find that you cannot vMotion to it.
Here’s why: when you boot a virtual machine, the operating system queries the CPU with a CPUID instruction . The response of that instruction tells the operating system what features and instructions the CPU is capable of. Intel and AMD typically add functionality to the CPUs in newer generations. This CPUID instruction lets the OS and Apps know if newer functions are available. For instance, if I were to start a virtual machine on that new host from my example above, the OS and apps would know (via the CPUID instruction) that that new node is using a SSE4 instruction set. Suppose I start an app in the VM that uses some of the new functions in the SSE4 set. If I were to live migrate (or quick migrate) that VM to a host from the existing 3-host cluster, the application would crash because the new SSE4 functions that were available a second ago, are no longer available. If you’re wondering what this looks like, here’s a good video on what happens when you don’t check . ESX protects users from this by checking for CPU differences and preventing these kinds of vMotions by default. New users to virtualization don’t typically know the functionality of each and every CPU so default safeguards must be put in place. Frankly, I didn’t know the differences until I started working at VMware. You would not want customers who don’t understand the differences to start live-migrating VM’s and have apps crashing due to CPU incompatibilities. Who would those users blame? The virtualization vendor probably, or even worse, virtualization as a whole.
Whew! Now for the solution: Beginning in ESX 3.5 Update 2, VMware added Enhanced vMotion Compatibility (EVC). Additionally, on the hardware side, beginning in Intel’s Penryn and AMD’s Rev. E or F Opterons, the ability for a hypervisor to change the response to the CPUID instruction was added. Intel calls theirs FlexMigration, AMD’s is called AMD-V Extended Migration. Here’s how it works: Back to my original example. You have 3 hosts with E5310 processors in them. Now the administrator goes to the cluster setting in Virtual Center and enables EVC for the cluster. When the new host is added, it’s CPUID instruction is set to respond with the same set of instructions that the existing CPU’s have. Now that we have the new host in this EVC cluster, we can vMotion between all of the nodes of the cluster. One thing to remember, the new CPUs have to be capable of EVC (either the same instruction level as the existing hosts or is capable of changing what the CPUID reports.) Here’s a list of the CPU’s that will work from VMware’s KB. The good news is when you go to the EVC setting for the cluster in Virtual Center, it will tell you if all of the necessary requirements are met. Additionally, this does not impact any of the performance of the new CPU hardware. The hardware on the newly added hosts run at their full speed.
One important factor to note. In order to enable EVC for a cluster, you cannot have any virtual machines running on that cluster. All of the virtual machines must be powered down (or a clean cluster with no VMs.) This protects the stability of the existing VMs. If you had a VM running on a host and it changed it’s CPUID while a VM was running, it would appear to the OS and app that the CPU changed while the VM was running, thus adding instability just like our original problem above. Once EVC is enabled for a cluster, there are no more VM outages required to add additional hosts to that cluster. The only additional VM outage required would be to raise the compatibility level. As you remove the older hosts from the cluster, you can raise the compatibility to the next highest level available to take advantage of some of the new instructions in the CPUs.
Another thing to be aware of: the FlexMigration (Intel) and Extended Migration (AMD) features do not block the actual instructions themselves. They only mask what the CPUID instruction reports. Back in the 486 days, application programmers would figure out what CPU they were on by trying different instructions. Intel formally added the CPUID instruction in the first Pentium processors in the early 1990′s. They also recommended that developers use this standard method going forward to determine CPU functionality. This is a much better method as it is much more consistent and reliable. The point is that it is possible that there could be code in use today, from the early 1990′s, that has not been upgraded and does not use the standard CPU checking with CPUID and tries to determine CPU capability by "trying functions." These applications would circumvent EVC, as they are using outdated methods to determine the CPU. Keep in mind that if applications exist today that use the old methods, they are bugged versions of software that do not adhere to current standards and should be patched or upgraded. If I were an admin, I would definitely want to test any apps that I have not updated in the last 10 years. I know that sounds almost comical, but you would be amazed at what I run into in customer’s environments.
In summary, it took a combination of hardware and software capability to make it happen. This is not the full solution to retrofit every piece of legacy hardware in datacenters today. However, it is a problem that has been addressed and will continue to fade in time as we refresh our hardware going forward.