Performance is definitely the harder of the two to determine.  Not every storage frame is created equal.  I’ve held Engineering positions at EMC and NetApp.  I can tell you that there’s a lot of marketing you are going to have to weed through to find out what you need from a performance standpoint.  In addition, no two customer’s environments require the same storage performance.  What I often suggest is to track the disk I/O on your physical machines.  You can use the standard OS Performance counters to do this.  Take a note on the physical hosts and gather the average MB/second of storage reads and writes and the average numbers of transactions per second.  You will also want the peak of these counters as well.  You can provide an aggregate of these to your storage vendor’s engineer to confirm their frame can deliver your minimum performance requirements.  Clearly you would want to pad these numbers to anticipate future growth.  If you do not have the expertise (or time) in-house, check with your storage vendor or Partner to see if they offer any virtualization/storage assessments.  Some offer free for basic results and some have paid assessments which can get extremely detailed and leave "no stone unturned".  To answer the performance question, the more detail you have on your own performance requirements, the better off you will be.  VMware has a very high-level protocol performance whitepaper which describes how the protocols perform from ESX server.  Remember that there are different price points for the protocols with Fiber Channel typically being the most expensive.  Price is always another factor to consider but every customer weighs price differently in the list of priorities.  Look for a balance of performance, price and functionality.  Know the priority of each factor for your company and review it often during the evaluation process, it may change.

The second factor in the protocol decision is functionality.  While I know that performance can be a difficult factor to clarify, functionality is pretty cut and dry. These are listed by protocols with the most functionality to the least.

Fiber Channel

Fiber Channel (FC) has most complete set of functionality.  If you go with a FC solution, there are no suprises with functionality limitations.  All of product functionality is supported (vMotion, HA, DRS, Lifecycle Manager, Lab Manager, Boot from SAN, etc.).  The other protocols run into a few limitations however.

iSCSI

From a VMware perspective booting the ESX hypervisor from SAN is only supported on iSCSI if you use hardware-based iSCSI initators.  The included software-based initatior does not support ESX boot-from-SAN.  Please remember, I’m referring to booting the ESX hypervisor itself from iSCSI is not supported with the software initiator, booting the VM’s from iSCSI is fully supported. Second, Microsoft Cluster Services is not supported on iSCSI, only Fiber Channel.  If you wanted to run MSCS nodes inside virtual machines, iSCSI will not work for you.

NFS

You cannot boot ESX from NFS.  That seems obvious but I’ll list it for the newcomers.  Second, you cannot do Raw Device Mappings on NFS.  Raw Device Mappings are presenting a LUN directly to a VM’s operating system.  No LUNs here on NFS.  Third, Storage vMotion is not supported going to or from NFS.  Storage vMotion is migrating where the VMs are stored while they are running.  If you want to migrate any VM’s to or from NFS today, you have to shut down the VMs and do a cold migration to move them.  Fourth, Site Recovery Manager currently does not support NFS.  If you want to use SRM to manage your DR plan today, NFS will not get you there.

One additional point I want to make is two pieces of functionality that NFS does support that all the other protocols do not.  They are thin provisioning and dynamic expandability.  NFS will allow you to thin provision VM’s and it’s actually the default behavior.  Thin provisioning is creating a virtual disk for a virtual machine and the only space taken on physical disk is the amount of data stored in that virtual disk.  For example, if I create a VM with a virtual disk that is 10GB.  I then load an OS in that VM that takes up 5GB of that virtual disk.  On local storage, iSCSI, and Fiber Channel this scenario will use 10GB of physical storage (the full size of the virtual disk, even if I’ve stored anything in it or not).  In the same scenario, NFS only takes up 5GB and it’s virtual disk file would grow dynamically as I put more in the VM.  VMware has announced that thin provisioning is coming to VMFS datastores in 2009 , NFS does it today.  Additionally, some storage frames do this behind the scenes at a storage frame level.  Check with your vendor’s capabilities.

The last benefit that NFS has over the others is dynamic expandibility.  ESX uses datastores to store it’s VMs.  Datastores can be comprised of one or more LUNs, or in the case of NFS, a mountpoint.  With an NFS datastore, the filesystem is maintained by the storage frame.  You can dynamically expand the NFS datastore by increasing it’s size on the storage frame and the ESX servers will recognize the increased storage with no additional effort.  With the VMFS datastores, you have to add extents to increase the datastore’s size.  In the case of VMFS, there’s a bit of administrative work to make this happen.

Local Storage

My least favorite and only recommended to customers just getting into virtualization who cannot afford a SAN or for small Remote Offices with no shared storage available.  Local does allow you to boot ESX from it (obviously).  However vMotion, DRS and HA are not permitted.  You cannot use it for a MSCS cluster.  It is not supported with SRM or Storage vMotion.

Here’s a quick summary chart of the functionality differences:

In summary, know what you require for performance and make sure your storage vendor can meet your demands.  Then make sure the protocol has the functionality that you need at a price you are willing to accept.  You’ll be off and running in no time, with a infrastructure that has the elasticity and versatility to respond to whatever your business can ask of it.