This KB article (http://kb.vmware.com/kb/1008027) reads:

For VMware FT to be supported, the servers that host the virtual machines must each use a supported processor from the same category as documented below:

Intel Xeon based on 45nm Core 2 Microarchitecture Category:
3100 Series
3300 Series
5200 Series (DP)
5400 Series
7400 Series

Intel Xeon based on Core i7 Microarchitecture Category:
3500 Series
5500 Series

AMD 3rd Generation Opteron Category:
1300 and 1400 Series
2300 and 2400 Series (DP)
8300 and 8400 Series (MP)

Please note the requirement “same category.”  As an example, if you have a server with a 54xx series Intel Processor and a Intel 55xx series processor (both have the technology for FT), you can vMotion and DRS between them (via EVC) but you cannot run a Fault Tolerant pair across them.  The Lockstep technology from Intel changed in the 35xx and 55xx CPUs and is not compatible with the previous generations of lockstep.

You can read all of the details in the white paper here .

Here’s why: when you boot a virtual machine, the operating system queries the CPU with a CPUID instruction .  The response of that instruction tells the operating system what features and instructions the CPU is capable of.  Intel and AMD typically add functionality to the CPUs in newer generations.  This CPUID instruction lets the OS and Apps know if newer functions are available.  For instance, if I were to start a virtual machine on that new host from my example above, the OS and apps would know (via the CPUID instruction) that that new node is using a SSE4 instruction set.  Suppose I start an app in the VM that uses some of the new functions in the SSE4 set.  If I were to live migrate (or quick migrate) that VM to a host from the existing 3-host cluster, the application would crash because the new SSE4 functions that were available a second ago, are no longer available.  If you’re wondering what this looks like, here’s a good video on what happens when you don’t check .  ESX protects users from this by checking for CPU differences and preventing these kinds of vMotions by default. New users to virtualization don’t typically know the functionality of each and every CPU so default safeguards must be put in place.   Frankly, I didn’t know the differences until I started working at VMware.  You would not want customers who don’t understand the differences to start live-migrating VM’s and have apps crashing due to CPU incompatibilities.  Who would those users blame?  The virtualization vendor probably, or even worse, virtualization as a whole.

Whew!  Now for the solution:  Beginning in ESX 3.5 Update 2, VMware added Enhanced vMotion Compatibility (EVC).  Additionally, on the hardware side, beginning in Intel’s Penryn and AMD’s Rev. E or F Opterons, the ability for a hypervisor to change the response to the CPUID instruction was added.  Intel calls theirs FlexMigration, AMD’s is called AMD-V Extended Migration.  Here’s how it works:  Back to my original example.  You have 3 hosts with E5310 processors in them.  Now the administrator goes to the cluster setting in Virtual Center and enables EVC for the cluster.  When the new host is added, it’s CPUID instruction is set to respond with the same set of instructions that the existing CPU’s have.  Now that we have the new host in this EVC cluster, we can vMotion between all of the nodes of the cluster.  One thing to remember, the new CPUs have to be capable of EVC (either the same instruction level as the existing hosts or is capable of changing what the CPUID reports.)  Here’s a list of the CPU’s that will work from VMware’s KB.  The good news is when you go to the EVC setting for the cluster in Virtual Center, it will tell you if all of the necessary requirements are met.  Additionally, this does not impact any of the performance of the new CPU hardware.  The hardware on the newly added hosts run at their full speed.

One important factor to note.  In order to enable EVC for a cluster, you cannot have any virtual machines running on that cluster.  All of the virtual machines must be powered down (or a clean cluster with no VMs.)  This protects the stability of the existing VMs.  If you had a VM running on a host and it changed it’s CPUID while a VM was running, it would appear to the OS and app that the CPU changed while the VM was running, thus adding instability just like our original problem above.  Once EVC is enabled for a cluster, there are no more VM outages required to add additional hosts to that cluster.  The only additional VM outage required would be to raise the compatibility level.  As you remove the older hosts from the cluster, you can raise the compatibility to the next highest level available to take advantage of some of the new instructions in the CPUs.

Another thing to be aware of: the FlexMigration (Intel) and Extended Migration (AMD) features do not block the actual instructions themselves.  They only mask what the CPUID instruction reports.  Back in the 486 days, application programmers would figure out what CPU they were on by trying different instructions.  Intel formally added the CPUID instruction in the first Pentium processors in the early 1990′s.  They also recommended that developers use this standard method going forward to determine CPU functionality.  This is a much better method as it is much more consistent and reliable.  The point is that it is possible that there could be code in use today, from the early 1990′s, that has not been upgraded and does not use the standard CPU checking with CPUID and tries to determine CPU capability by "trying functions."  These applications would circumvent EVC, as they are using outdated methods to determine the CPU.  Keep in mind that if applications exist today that use the old methods, they are bugged versions of software that do not adhere to current standards and should be patched or upgraded.  If I were an admin, I would definitely want to test any apps that I have not updated in the last 10 years.  I know that sounds almost comical, but you would be amazed at what I run into in customer’s environments.

In summary, it took a combination of hardware and software capability to make it happen.  This is not the full solution to retrofit every piece of legacy hardware in datacenters today.  However, it is a problem that has been addressed and will continue to fade in time as we refresh our hardware going forward.

1. EVC (Enhanced VMotion Compatibility) This allows you to live migrate your VM’s between different generations of CPUs from the same vendor. Basically you need Intel’s Merom (or newer) processor or AMD’s Rev.E (or newer) Based Processor.  The official list can be found here .
2. New guest operating system support: Windows Server 2008, Solaris 10 U5, SUSE Linux Enterprise Server 10 SP2, and Ubuntu 8.04.  Customers are always happy when we expand the supported OS list, some don’t even wait for us to try a new OS out.  However, if you call in for support, the guest OS needs to be on the supported list .
3. Storage VMotion to iSCSI and FC.  You can now live migrate VM’s on storage (move where the VM’s are stored) from one datastore to another on either (or both) of the storage platforms.
4. Hot virtual disk extend.  You can now entend a virtual disk drive while the VM is running from the GUI.  Just edit the VM’s hard disk config while the VM is running to the max allowed by the datastore.
5. VSS quescing support.  The VCB (VMware Consolidated Backup) agents now allow for VSS integration so when you take a backup of a Server 2003 VM, VCB will quiesce the data from the app (like SQL and Exchange) and then have ESX quiesce the disks and then take a nice clean VM snapshot.  Which then can be backup by the backup agents on the VCB backup proxy.  This give a more consistant snapshot image to work with.

Those are some of the main features added in U2.  If you are not ready to go up to ESX 3.5 Update 3, Update 2 should be on your radar.  The licensing bug is no more and, in my opinion, the list of features warrants considering it during your next patch cycle.