The VMguy

You may remember from a recent article that I wrote about the VMware licensing dilemmas, that one of the scenarios I mentioned was SRM licensing when a customer wants to protect only a small percentage of VMs.  In the per-CPU licensing model, a customer would have to license all of the CPUs in a cluster even if they wanted to protect only 10% of the VMs.  VMware has announced that Per-VM licensing will be available on September 1, 2010.  Customers will now be able to license SRM on a Per-VM basis.  Customers who like their per-CPU model will be able to continue that purchasing method until December 15, 2010.  After that, it’s per-VM only.

There are a few things to think about with regard to licensing  first, vSphere 4.1 now allows for DRS affinity so that VMs only move between certain hosts of a cluster.  I’m still waiting for a definite answer from my VMware friends but that should allow you to protect some VM’s and set their DRS Affinity to only the hosts that you own SRM CPUs for and still keep the full cluster for the unprotected VMs. Previously, VMware would recommend that you create a separate cluster for your “protected” VMs if they were a small subset of the whole.  Now with DRS Affinity, you can dictate that certain “protected” VMs only move between a subset of a cluster.  We’ll still have to wait and see the final ruling from VMware but I’m thinking that would work in the short-term for those in the per-CPU dilemma.

The second feature of the new licensing that I really like is the rolling average of VMs over the last twelve months.  What that translates to is that now I need to buy what my daily average of VMs protected would be over a 12 month period.  If I have certain points of the year where my VM count spikes, this average would be monitored by vCenter and alarm if I am going over my licensing limits.  However, I would only need the average number of protected VMs over the past year.  The system will continue to run after going over your limit but that’s definitely not something I would condone (Famous VMware SE saying: ethics don’t ship in the box people).

The per-vm licenses are sold in blocks of 25 and range from $1,250 to $11,250 depending on the product.  Per-vm licensing will be available for Chargeback, Appspeed, SRM, and, later this year, CapacityIQ.  You can find more information on VMware’s website here.

The last question I had was, “How do I know what my rolling average is for those licenses?”  The good news is that once you enter in a license key, the new license reporting manager in vSphere 4.1 will tell you what your rolling average is year-to-date.  Looks like someone was planning ahead.

The way I see it, there are two dilemmas that VMware has in the way their licensing is designed today.  One of them works against VMware and one works against VMware customers (or at least makes it harder for them).  The former is definitely the bigger of the two so lets discuss that one first.  This topic comes up frequently when new versions of ESX are coming out.  We’ve already heard that an update is coming this year so I figured that since today is the half-way point in the year, this was a good time to bring up the topic again.

You probably noticed by now that there is a limitation in Standard and Enterprise editions of vSphere to a maximum of 6-cores per CPU.  The Advanced and Enterprise-Plus editions of vSphere have a licensed limit of 12-cores per CPU.  Now that Intel’s 8-core CPUs and AMD’s 12-cores are out, what’s next?  Intel and AMD are sure to develop a proc with more than 12 cores (and probably sooner than we all think).  What will happen to VMware’s licensing then?  You have to remember that from a revenue standpoint, when a 24 core proc comes out, customers will be able to run twice as many workloads on that proc (or at least 50% more).  Moore’s Law states that processing performance of CPUs will double every two years.  With the processors doubling in power so quickly, customers are typically not doubling their number of VMs in the same time period.  The result is that customers tend to have a diminishing need to increase their ESX per-CPU licensing.  I know that there are exceptions to this rule, but in the SMB space the majority are not growing that fast (at least not in this economy).  The increase in processor performance actually works against VMware’s current licensing model. It not good to have a direct connection between your main revenue stream and someone else’s CPU release schedule.  What will happen?  What’s the right answer?  Your guess is as good as mine.  Will they go to a per-vm model?  Increase their current limits?  Find some middle-ground between the two?  Will they “grandfather” their customers like AT&T did with the iPad data plans?  Only VMware knows.  My opinion is that this is an issue that has to be dealt with eventually.  Maybe this will be the year, maybe next.

The second licensing dilemma that I run into is in Site Recovery Manager.  It’s no secret that SRM is my favorite non-ESX product from VMware.  As you probably know, SRM is licensed by the physical CPU where the protected virtual machines reside or could reside.  Here’s where that model breaks down:  let’s say I have a smaller customer who’s policy is only to have a DR plan for 5 of their most critical Virtual Machines.  Those five VMs run in a cluster comprise of 5 dual CPU hosts with HA and DRS enabled.  According to the SRM licensing model, I need 10 CPUs of SRM for those 5 VMs.  That does not fly well.  The solution I’ve heard some engineers mention is to create a separate smaller cluster for just the protected VMs.  I’m not fond of that idea because it goes against the consolidation principal.  I’ve never felt that lowering your consolidation ratio was justified because it did not fit a licensing model.

I know there are people much smarter than me at work trying to find a solution to both of these scenarios.  I’m hopeful that they will get resolved in a way that’s fair to both sides.  Maybe this is the year, maybe it is not.  Either way, we’ve made it thru half of 2010, perhaps the answers lie in the last 6 months of the year.

I was assisting a customer this week in upgrading to vSphere and installing and running vReplicator from Vizioncore.  vReplicator is not a complex product but works well for what it does: replicate VMs.  During the install of vReplicator, we setup replication for a few VMs.  The product has a few options for how to determine what to replicate.  Since we were now on ESX4 on source and target, I suggested we use Changed Block Tracking mode (CBT) for replication.

When I suggested CBT to the customer they asked, “Why that one?” and how it worked.  So I explained:  When we replicate from source to target, the first copy is a full copy of the data (the “seed” it is often called).  When we go to replicate the next time, we don’t want to replicate the whole thing again, just what has changed since the last time we replicated (often called a “differential”).  The replication software needs to determine what’s changed.  Prior to ESX 4, there was not a built in method to do this.  The software would have to find another method, such as compare snapshot information and determine which blocks are new.  That uses CPU cycles on the ESX hosts and takes time (differential mode in vReplicator takes  roughly 1 minute per GB of VM data).  On the other hand, CBT is a feature in ESX4 that tracks the block changes that have occurred since a point in time.  It does not keep a copy of the changed data in a separate location, just a log that the blocks in question have changed.  This is a huge help to backup and replication technologies who typically have to determine what has changed on the disks via their own methods.  Now, ESX can tell them directly what has changed and they can get right to copying those changed blocks.  This makes the overall replication and backup jobs much quicker.

Now for a few lessons learned in using it.  First, it requires hardware version 7 VM’s (HW7) and ESX4.  VM’s need to have their VMtools upgraded to the latest version and then you can upgrade the VMs to HW7 when they are powered off via right clicking them (this updates the virtual hardware presented to the VMs and will require another reboot in Windows after powering it on when the OS discovers the new virtual HW and loads the drivers – thanks Microsoft!).  Second, CBT it is not on by default.  It is set per VM and is an advanced option you can set in the VM’s config.  Some software have the capability to change the CBT setting for you.  In our case, vReplicator has this option on the CBT options page.  On that page, it will check every VM that it can see and if they are HW7.  If they are HW7, they will show as supported.  On that screen, you will also see a checkbox for the “enabled” field.  When you click the enabled box on your HW7 VMs, vReplicator makes the change for you in the VM’s configuration.  However, as mentioned earlier, you must completely power down that VM and power it back on.  The reason for this is that, to start using it, ESX needs to create the tracking log for each disk (the log is about .5MB for ever GB of VMDK or Virtual Mapped RDM and it’s stored with the VM) and ESX only does this setup process at VM boot time.  So make note, a restart won’t work.  It has to be a VM power down and VM power back on.  There is a great article that taught me a few things on CBT by Eric Siebert that goes into a little more technical detail and you can find it here.

Once we got this process completed, my customer’s replication jobs ran MUCH faster.  The data being copied from the source to the target was the same, but the time it took vReplicator to determine what to replicate went from minutes to seconds.  Great news too was that we were able to change the replication method on the fly (from Differential to CBT, if you’re using hybrid, I think you need to re-seed).

My final advice, is make sure you understand if your backup/replication software can use CBT and what you need to enable it.  It does take a bit of work to upgrade the tools and virtual hardware (use Update Manager!).  However it’s well worth it in the long run.

By now, most of you know how Site Recovery Manager works.  SRM requires a SRA (Storage Replication Adapter) which is basically a translator to allow SRM talk directly to the storage arrays.  6 of the SRAs were updated on VMware’s download site for SRM in January.  They were:

Dell EqualLogics, Version 1.0.2, Released 01/20/2010

EMC Celerra, Version 4.0.17, Released 01/29/2010

Fujitsu ETERNUS SF AdvancedCopy Manager, Version 1.3| Released 01/29/2010

IBM DS4000/5000, Version 01.01.35.05, Released 01/15/2010

IBM N-series SAN Adapter, Version 1.4.2, Released 01/07/2010

LSI, Version 1.01.30.05, Released 01/21/2010

You can find the updated SRAs on the SRM download site here.  As always, please check to make sure these updates are necessary for your environment.  I would have included the release notes but unfortunately the vendors are not keep track of them.  Upgrade at your own risk, and if you do, please test your recovery plans out fully with the new adapters.

At long last, Site Recovery Manager 4.0 has been released.  Here’s the what’s new section from the release notes:

This release of Site Recovery Manager introduces several new features:

• Full compatibility with vCenter 4.
• Full support for NFS-based arrays.
• Support for shared recovery sites.
  Enables many-to-one pairings of protected sites with a recovery site. For more information, see the technical note Installing, Configuring, and Using Shared Recovery Site Support, which is available at http://www.vmware.com/support/pubs/srm_pubs.html.
• Resilience in the face of vCenter unavailability during a test recovery.
  Placeholder virtual machines can be quickly repaired after the protected site vCenter becomes available again.
• New repair-mode installation features.
  You can run the SRM installer in repair mode if you need to change configuration parameters such as vCenter credentials, database connection information or credentials, and certificate details.
• Graphical interface to advanced settings.
  Eliminates most requirements to edit the XML configuration file
• Support for DB2 as an SRM database server.
• New licensing options.
• Improved scalability.
  A single protection group can now include up to 1000 virtual machines.
• Full Compatibility With DPM (Distributed Power Management)
  SRM recovery plans can now power-on or power-off a host that is in standby mode.
• New Option to dr-ip-customizer Utility
  The dr-ip-customizer utility now logs less verbose diagnostic output by default. To force dr-ip-customizer to log the same level of diagnostic output that it produced in earlier releases, use the -verbose option.
• Change in Certificate Validation
  When you select certificate authentication, the SRM installation validates the certificate you supply before continuing. Certificates signed with an MD5 key are no longer allowed.
• Support for Protecting Fault-Tolerant Virtual Machines.
  SRM can now protect virtual machines that have been configured for fault-tolerant operation. When recovered, these virtual machines lose their fault tolerance, and must be manually reconfigured after recovery to restore fault tolerance.
• Improved context-sensitive Help.
• PDF documents available on release media
  Current versions of the PDF documents for this release are available in the docs folder at the root of the SRM 4.0 CD. Updated versions of these documents may be available at http://www.vmware.com/support/pubs/srm_pubs.html.