There are a few things to think about with regard to licensing  first, vSphere 4.1 now allows for DRS affinity so that VMs only move between certain hosts of a cluster.  I’m still waiting for a definite answer from my VMware friends but that should allow you to protect some VM’s and set their DRS Affinity to only the hosts that you own SRM CPUs for and still keep the full cluster for the unprotected VMs. Previously, VMware would recommend that you create a separate cluster for your “protected” VMs if they were a small subset of the whole.  Now with DRS Affinity, you can dictate that certain “protected” VMs only move between a subset of a cluster.  We’ll still have to wait and see the final ruling from VMware but I’m thinking that would work in the short-term for those in the per-CPU dilemma.

The second feature of the new licensing that I really like is the rolling average of VMs over the last twelve months.  What that translates to is that now I need to buy what my daily average of VMs protected would be over a 12 month period.  If I have certain points of the year where my VM count spikes, this average would be monitored by vCenter and alarm if I am going over my licensing limits.  However, I would only need the average number of protected VMs over the past year.  The system will continue to run after going over your limit but that’s definitely not something I would condone (Famous VMware SE saying: ethics don’t ship in the box people).

The per-vm licenses are sold in blocks of 25 and range from $1,250 to $11,250 depending on the product.  Per-vm licensing will be available for Chargeback, Appspeed, SRM, and, later this year, CapacityIQ.  You can find more information on VMware’s website here.

The last question I had was, “How do I know what my rolling average is for those licenses?”  The good news is that once you enter in a license key, the new license reporting manager in vSphere 4.1 will tell you what your rolling average is year-to-date.  Looks like someone was planning ahead.

You probably noticed by now that there is a limitation in Standard and Enterprise editions of vSphere to a maximum of 6-cores per CPU.  The Advanced and Enterprise-Plus editions of vSphere have a licensed limit of 12-cores per CPU.  Now that Intel’s 8-core CPUs and AMD’s 12-cores are out, what’s next?  Intel and AMD are sure to develop a proc with more than 12 cores (and probably sooner than we all think).  What will happen to VMware’s licensing then?  You have to remember that from a revenue standpoint, when a 24 core proc comes out, customers will be able to run twice as many workloads on that proc (or at least 50% more).  Moore’s Law states that processing performance of CPUs will double every two years.  With the processors doubling in power so quickly, customers are typically not doubling their number of VMs in the same time period.  The result is that customers tend to have a diminishing need to increase their ESX per-CPU licensing.  I know that there are exceptions to this rule, but in the SMB space the majority are not growing that fast (at least not in this economy).  The increase in processor performance actually works against VMware’s current licensing model. It not good to have a direct connection between your main revenue stream and someone else’s CPU release schedule.  What will happen?  What’s the right answer?  Your guess is as good as mine.  Will they go to a per-vm model?  Increase their current limits?  Find some middle-ground between the two?  Will they “grandfather” their customers like AT&T did with the iPad data plans?  Only VMware knows.  My opinion is that this is an issue that has to be dealt with eventually.  Maybe this will be the year, maybe next.

The second licensing dilemma that I run into is in Site Recovery Manager.  It’s no secret that SRM is my favorite non-ESX product from VMware.  As you probably know, SRM is licensed by the physical CPU where the protected virtual machines reside or could reside.  Here’s where that model breaks down:  let’s say I have a smaller customer who’s policy is only to have a DR plan for 5 of their most critical Virtual Machines.  Those five VMs run in a cluster comprise of 5 dual CPU hosts with HA and DRS enabled.  According to the SRM licensing model, I need 10 CPUs of SRM for those 5 VMs.  That does not fly well.  The solution I’ve heard some engineers mention is to create a separate smaller cluster for just the protected VMs.  I’m not fond of that idea because it goes against the consolidation principal.  I’ve never felt that lowering your consolidation ratio was justified because it did not fit a licensing model.

I know there are people much smarter than me at work trying to find a solution to both of these scenarios.  I’m hopeful that they will get resolved in a way that’s fair to both sides.  Maybe this is the year, maybe it is not.  Either way, we’ve made it thru half of 2010, perhaps the answers lie in the last 6 months of the year.

When I suggested CBT to the customer they asked, “Why that one?” and how it worked.  So I explained:  When we replicate from source to target, the first copy is a full copy of the data (the “seed” it is often called).  When we go to replicate the next time, we don’t want to replicate the whole thing again, just what has changed since the last time we replicated (often called a “differential”).  The replication software needs to determine what’s changed.  Prior to ESX 4, there was not a built in method to do this.  The software would have to find another method, such as compare snapshot information and determine which blocks are new.  That uses CPU cycles on the ESX hosts and takes time (differential mode in vReplicator takes  roughly 1 minute per GB of VM data).  On the other hand, CBT is a feature in ESX4 that tracks the block changes that have occurred since a point in time.  It does not keep a copy of the changed data in a separate location, just a log that the blocks in question have changed.  This is a huge help to backup and replication technologies who typically have to determine what has changed on the disks via their own methods.  Now, ESX can tell them directly what has changed and they can get right to copying those changed blocks.  This makes the overall replication and backup jobs much quicker.

Now for a few lessons learned in using it.  First, it requires hardware version 7 VM’s (HW7) and ESX4.  VM’s need to have their VMtools upgraded to the latest version and then you can upgrade the VMs to HW7 when they are powered off via right clicking them (this updates the virtual hardware presented to the VMs and will require another reboot in Windows after powering it on when the OS discovers the new virtual HW and loads the drivers – thanks Microsoft!).  Second, CBT it is not on by default.  It is set per VM and is an advanced option you can set in the VM’s config.  Some software have the capability to change the CBT setting for you.  In our case, vReplicator has this option on the CBT options page.  On that page, it will check every VM that it can see and if they are HW7.  If they are HW7, they will show as supported.  On that screen, you will also see a checkbox for the “enabled” field.  When you click the enabled box on your HW7 VMs, vReplicator makes the change for you in the VM’s configuration.  However, as mentioned earlier, you must completely power down that VM and power it back on.  The reason for this is that, to start using it, ESX needs to create the tracking log for each disk (the log is about .5MB for ever GB of VMDK or Virtual Mapped RDM and it’s stored with the VM) and ESX only does this setup process at VM boot time.  So make note, a restart won’t work.  It has to be a VM power down and VM power back on.  There is a great article that taught me a few things on CBT by Eric Siebert that goes into a little more technical detail and you can find it here.

Once we got this process completed, my customer’s replication jobs ran MUCH faster.  The data being copied from the source to the target was the same, but the time it took vReplicator to determine what to replicate went from minutes to seconds.  Great news too was that we were able to change the replication method on the fly (from Differential to CBT, if you’re using hybrid, I think you need to re-seed).

My final advice, is make sure you understand if your backup/replication software can use CBT and what you need to enable it.  It does take a bit of work to upgrade the tools and virtual hardware (use Update Manager!).  However it’s well worth it in the long run.

Dell EqualLogics, Version 1.0.2, Released 01/20/2010

EMC Celerra, Version 4.0.17, Released 01/29/2010

Fujitsu ETERNUS SF AdvancedCopy Manager, Version 1.3| Released 01/29/2010

IBM DS4000/5000, Version 01.01.35.05, Released 01/15/2010

IBM N-series SAN Adapter, Version 1.4.2, Released 01/07/2010

LSI, Version 1.01.30.05, Released 01/21/2010

You can find the updated SRAs on the SRM download site here.  As always, please check to make sure these updates are necessary for your environment.  I would have included the release notes but unfortunately the vendors are not keep track of them.  Upgrade at your own risk, and if you do, please test your recovery plans out fully with the new adapters.

This release of Site Recovery Manager introduces several new features:

• Full compatibility with vCenter 4.
• Full support for NFS-based arrays.
• Support for shared recovery sites.
  Enables many-to-one pairings of protected sites with a recovery site. For more information, see the technical note Installing, Configuring, and Using Shared Recovery Site Support, which is available at http://www.vmware.com/support/pubs/srm_pubs.html.
• Resilience in the face of vCenter unavailability during a test recovery.
  Placeholder virtual machines can be quickly repaired after the protected site vCenter becomes available again.
• New repair-mode installation features.
  You can run the SRM installer in repair mode if you need to change configuration parameters such as vCenter credentials, database connection information or credentials, and certificate details.
• Graphical interface to advanced settings.
  Eliminates most requirements to edit the XML configuration file
• Support for DB2 as an SRM database server.
• New licensing options.
• Improved scalability.
  A single protection group can now include up to 1000 virtual machines.
• Full Compatibility With DPM (Distributed Power Management)
  SRM recovery plans can now power-on or power-off a host that is in standby mode.
• New Option to dr-ip-customizer Utility
  The dr-ip-customizer utility now logs less verbose diagnostic output by default. To force dr-ip-customizer to log the same level of diagnostic output that it produced in earlier releases, use the -verbose option.
• Change in Certificate Validation
  When you select certificate authentication, the SRM installation validates the certificate you supply before continuing. Certificates signed with an MD5 key are no longer allowed.
• Support for Protecting Fault-Tolerant Virtual Machines.
  SRM can now protect virtual machines that have been configured for fault-tolerant operation. When recovered, these virtual machines lose their fault tolerance, and must be manually reconfigured after recovery to restore fault tolerance.
• Improved context-sensitive Help.
• PDF documents available on release media
  Current versions of the PDF documents for this release are available in the docs folder at the root of the SRM 4.0 CD. Updated versions of these documents may be available at http://www.vmware.com/support/pubs/srm_pubs.html.

The first is wondering how an admin can do more advanced things with the View Manager Server.  For instance: assign a default desktop out of a pool to a specific user or  listing orphaned desktops (users that no longer exist or have changed permissions and no longer have access to a desktop in a pool but still have one assigned to them) or cleaning up after removing a secondary View Manager server.  All of these items can be performed by a little-known command line utility included in View called vdmadmin.exe. vdmadmin.exe can do all of the things mentioned.  The tool is installed by default with view in the  C:\Program Files\VMware\VMware View\Server\bin directory.  You can read all of the details about the tool and all of the command options here .  "Why is this a command-line tool?" you ask.  When features are requested for the GUI, a programmer has to create and test the GUI options which takes a lot of time.  The product manager has to decide what features will make in into the released product and still keep it on time for release.  Sometime features or controls don’t make it but are still important.  A developer can write a command line executable and they are much easier to test as the command line either works or it does not.  The Quality Assurance process is much easier for command line utilities.

Another tool that’s good to know is sviconfig.exe.  It’s located on the vCenter server where you install composer in the C:\Program Files\VMware\VMware View Composer directory.  This tool allows you to unlock protected entities in vCenter.  For instance, if a user checks out a VM offline, it gets protected in vCenter so that an admin cannot accidentally delete it or power it on and make changes.  If the admin needs to make changes to it for some reason they can use the sviconfig tool to unlock that VM.

There’s also a few other topics that come up frequently.  Here are a few:

Can you protect View Desktops with Site Recovery Manager? Unfortunately, SRM is not "View aware" as yet.  The work is underway however.  Tommy Walker from VMware (one of the best @ VMware) has written a blueprint with EMC on how to accomplish this task (although experimental at this point).  It’s pretty interesting reading and I’m very hopeful that we’ll see native integration coming.  You can read the very detailed blueprint here (registration required).

What features do I get with thin client "x"? With the different number of options available for View Thin Clients there’s also a number of different thin client OS options.  Here is a great matrix of what features you get with each View client that is available.  Good information to know when picking out thin clients

How can I make location-aware desktops? This is a common question.  How can I make a specific thin client map a specific printer, etc based on the thin client location?  The answer: Client Information Variables.  Here’s a nice writeup on some basic scripting that can determine the variables and make appropriate decisions based on their values.

Can I use group policy to control the client or desktop agent? You bet.  You can find the administrative templates for group policy in the C:\Program Files\VMware\View Manager\Server\Extras\GroupPolicyFiles directory.  If you’re not too sure about using adm files, here’s a kb article from Microsoft on how to use and manage them.  There are 4 included with View: vdm-agent.adm to control the agent in the guest OS, vdm_client.adm to control the properties of the View client, vdm_server.adm to control the connection broker and vdm_common.adm for common View components across all of View.  A very detailed list of the options of each can be found in the View admin guide starting on page 176.

This release is all bug fixes, here’s the list from the release notes:

The following issues have been resolved since the last release of Data Recovery. For a full list of known existing issues, see VMware Data Recovery 1.0 Release Notes. The list of resolved issues below pertains to this release of Data Recovery only.

• Large Temporary Files Removed as ExpectedData Recovery modifies virtual machines’ vmdk files’ settings so a snapshot can be created for backup purposes. In the past, after the backup has been created, the vmdk file’s settings was sometimes left configured for snapshots even after the backup was complete. This led to these virtual machines being left in snapshot mode while accumulating snapshots that were undetected by vSphere Client. This process has been redesigned so that these temporary files are no longer be left behind. In previous versions of Data Recovery, this issue can be resolved by following the process described in the knowledge base article titled “Delete ddb.delete entries and snapshots left behind by Vmware Data Recovery”.
• Backups Can Be Completed While Integrity Checks Are RunningData Recovery can complete backup operations at the same time that an integrity check is running. In the past, when an integrity check was running, backups could not be completed.
• Improved Integrity Check Backup SpeedIntegrity check has been optimized for faster performance. In the past, comparable integrity checks took longer to complete.
• Improved VMotion Licensing SupportVirtual machines can be moved between hosts using VMotion without producing licensing issues. In the past, if a virtual machine was moved between hosts using VMotion, licensing checks sometimes produced errors.
• Reduced Data Recovery Backup Appliance Shutdown TimeThe Data Recovery Backup appliance now shuts down more quickly than it did before. In the past, the appliance often took 15 minutes to shutdown.
• Improved Support for Different Time ZonesIn the past, Data Recovery did not consistently handle time zones with positive offsets relative to GMT. For example, Data Recovery could encounter issues with data associated with the Paris time zone, which has an offset of +1, whereas data associated with the New York time zone, which has an offset of -5 was handled as expected. These issues no longer occur.
• Data Recovery Supported with Essentials Plus LicensesData Recovery is included in Essentials Plus licenses. In the past, using Data Recovery with Essentials Plus licenses failed. Backup jobs created with Essentials Plus licenses failed with the error, License not available to perform operation. Feature hotplug not licensed....
• Integrity Check Optimized to Run During Idle TimesBefore running regularly scheduled integrity checks, the Backup Appliance determines if the current time is during a backup window. If the current time is not during a backup window, the integrity check runs. If the current time is during a backup window, the backup appliance checks the backup schedule to determine if there will be a time in the next 24 hours that will not be during a backup window. If there is a time in the next 24 hours that is not during a backup window, the Backup Appliance waits for that time. If there is no time that is not during a backup window in the next 24 hours, the Backup Appliance completes the integrity check.

What does Fault Tolerance do to prevent a split brain if both Primary and Secondary VMs become isolated?

Fault Tolerance (FT) uses an on-disk generation number file.  When FT is enabled the primary VM creates a file on shared storage called generation.N where N is a counter number.  The secondary VM is started and when it connects to the primary, the primary tells the secondary what the generation number is.  Once the Primary or secondary detects that there is a failure in the other half of the VM pair, it will try to rename the generation.N file to generation.N+1.  If the rename succeeds, the VM takes over as being the Primary (or remains the primary if it already was) and takes corrective action to rebuild a secondary and become protected again.  If the rename of the generation.N file fails, that means that the other VM in the pair already renamed the file and took over and the current VM shuts down.

There you have it, the disk subsystem prevents both VM’s from becoming the primary at the same time and creating a split brain.