The VI3 product suite includes several products that make up the full feature set of enterprise virtualization. The products in the VI3 suite include:

♦ VMware ESX Server

♦ VMware Virtual SMP

♦ VMware VirtualCenter

♦ Virtual Infrastructure Client

♦ VMware VMotion

♦ VMware Distributed Resource Scheduler (DRS)

♦ VMware High Availability (HA)

♦ VMware Consolidated Backup (VCB)

Rather than wait to introduce the individual products in their own chapters, I'll introduce each product so I can refer to the products and explain how they affect each piece of the design, installation, and configuration of your virtual infrastructure. Once you understand the basic functions and features of each product in the suite, you'll have a better grasp of how that product fits into the big picture of virtualization, and you'll more clearly understand how each of the products fits into the design.

VMware ESX Server 3.5 and ESXi are the core of the VI3 product suite. They function as the hypervisor, or virtualization layer, that serves as the foundation for the whole VI3 package. Unlike some virtualization products that require a host operating system, ESX Server is a bare metal installation, which means no host operating system (Windows or Linux) is required. ESX Server is a leaner installation than products requiring a host operating system, which allows more of its hardware resources to be utilized by virtual machines rather than by processes required to run the host. The installation process for ESX Server installs two components that interact with each other to provide a dynamic and robust virtualization environment: the Service Console and the VMkernel.

The Service Console, for all intents and purposes, is the operating system used to manage ESX Server and the virtual machines that run on the server. The console includes services found in other operating systems, such as a firewall, Simple Network Management Protocol (SNMP) agents, and a web server. At the same time, the Service Console lacks many of the features and benefits that other operating systems offer. This deficiency, however, serves as a true advantage in making the Service Console a lean, mean, virtualization machine.

The other installed component is the VMkernel. While the Service Console gives you access to the VMkernel, it is the VMkernel that is the real foundation of the virtualization process. The VMkernel manages the virtual machines' access to the underlying physical hardware by providing CPU scheduling, memory management, and virtual switch data processing. Figure 1.1 shows the structure of ESX Server.

Figure 1.1 Installing ESX Server installs two interoperable components: 1) the Linux-derived Service Console, and 2) the virtual machine-managing VMkernel.

The VMware Virtual Symmetric Multi-Processing (SMP) product allows virtual infrastructure administrators to construct virtual machines with multiple virtual processors. VMware Virtual SMP is not the licensing product that allows ESX Server to be installed on servers with multiple processors; it is the configuration of multiple processors inside a virtual machine. Figure 1.2 identifies the differences between multiple processors in the ESX Server host system and multiple virtual processors.

Figure 1.2 VMware Virtual SMP allows virtual machines to be created with two or four processors.

Stop for a moment and think about your current Windows network. Does it include Active Directory? There is a good chance it does. Now imagine your Windows network without Active Directory, without the ease of a centralized management database, without the single sign-on capabilities, and without the simplicity of groups. That is what managing ESX Server computers would be like without using VMware VirtualCenter 2.0. Now calm yourself down, take a deep breath, and know that VirtualCenter, like Active Directory, is meant to provide a centralized management utility for all ESX Server hosts and their respective virtual machines. VirtualCenter is a Windows-based, database-driven application that allows IT administrators to deploy, manage, monitor, automate, and secure a virtual infrastructure in an almost effortless fashion. The back-end database (SQL or Oracle) used by VirtualCenter stores all the data about the hosts and virtual machines. In addition to its configuration and management capabilities, VirtualCenter provides the tools for the more advanced features of VMware VMotion, VMware DRS, and VMware HA. Figure 1.3 details the VirtualCenter features provided for the ESX Server hosts it manages.

In Chapter 5, you'll learn the details of the VirtualCenter implementation, configuration, and management, as well as look at ways to ensure its availability.

The Virtual Infrastructure (VI) Client is a Windows-based application that allows you to connect to and manage an ESX Server or a VirtualCenter Server. You can install the VI Client by browsing to the URL of an ESX Server or VirtualCenter and selecting the appropriate installation link. The VI Client is a graphical user interface (GUI) used for all the day-to-day management tasks and for the advanced configuration of a virtual infrastructure. Using the client to connect directly to an ESX Server requires that you use a user account residing in the Service Console (a Linux account), while using the client to connect to a VirtualCenter Server requires you to use a Windows account. Figure 1.4 shows the account authentication for each connection type.

Figure 1.3 VirtualCenter 2.0 is a Windows-based application used for the centralization of authentication, accounting, and management of ESX Server hosts and their corresponding virtual machines.

Figure 1.4 The Virtual Infrastructure Client can be used to manage an individual ESX Server by authenticating with a Linux account that resides in the Service Console; however, it can also be used to manage an entire enterprise by authenticating to a VirtualCenter Server using a Windows account.

Almost all the management tasks available when you're connected directly to an ESX Server are available when you're connected to a VirtualCenter Server, but the opposite is not true. The management capabilities available through VirtualCenter Server are more significant and outnumber the capabilities of connecting directly to an ESX Server.

If you have read anything about VMware, you have most likely read about the extremely unique and innovative feature called VMotion. VMotion is a feature of ESX Server and VirtualCenter that allows a running virtual machine to be moved from one ESX Server host to another without having to power off the virtual machine. Figure 1.5 illustrates the VMotion feature of VirtualCenter.

Figure 1.5 The VMotion feature of VirtualCenter allows a running virtual machine to be transitioned from one ESX Server host to another.

Now that I've piqued your interest with the introduction of VMotion, let me introduce VMware Distributed Resource Scheduler (DRS). If you think that VMotion sounds exciting, your anticipation will only grow after learning about DRS. DRS, simply put, is a feature that aims to provide automatic distribution of resource utilization across multiple ESX hosts that are configured in a cluster. An ESX Server cluster is a new feature in VMware Infrastructure 3. The use of the term cluster often draws IT professionals into thoughts of Microsoft Windows Server clusters. However, ESX Server clusters are not the same. The underlying concept of aggregating physical hardware to serve a common goal is the same, but the technology, configuration, and feature sets are very different between ESX Server clusters and Windows Server clusters.

An ESX Server cluster is an implicit aggregation of the CPU power and memory of all hosts involved in the cluster. Once two or more hosts have been assigned to a cluster, they work in unison to provide CPU and memory to the virtual machines assigned to the cluster. The goal of DRS is to provide virtual machines with the required hardware resources while minimizing the amount of contention for those resources in an effort to maintain good performance levels.

DRS has the ability to move running virtual machines from one ESX Server host to another when resources from another host can enhance a virtual machine's performance. Does that sound familiar? It should, because the behind-the-scenes technology for DRS is VMware VMotion. DRS can be configured to automate the placement of each virtual machine as it is powered on as well as to manage the virtual machine's location once it is running. For example, let's say three servers have been configured in an ESX Server cluster with DRS enabled. When one of those servers begins to experience a high contention for CPU utilization, DRS will use an internal algorithm to determine which virtual machine(s) will experience the greatest performance boost by being moved to another server with less CPU contention. Figure 1.6 outlines the automated feature of DRS.

Figure 1.6 VMware Distributed Resource Scheduler (DRS) aims to maintain balance and fairness of resource utilization for virtual machines running within an ESX Server cluster.

With the introduction of the ESX Server cluster, VMware has also introduced a new feature called VMware High Availability (HA). Once again, by nature of the naming conventions (clusters, high availability), many traditional Windows administrators will have preconceived notions about this feature. Those notions, however, are premature in that VMware HA does not function like a high-availability configuration in Windows. The VMware HA feature provides an automated process for restarting virtual machines that were running on an ESX Server at a time of complete server failure. Figure 1.7 depicts the virtual machine migration that occurs when an ESX Server that is part of an HA-enabled cluster experiences failure.

Figure 1.7 The VMware High Availability (HA) feature will power on any virtual machines that were previously running on an ESX Server that has experienced server failure.

One of the most critical aspects to any network, not just a virtualized infrastructure, is a solid backup strategy as defined by a company's disaster recovery and business continuity plan. VMware Consolidated Backup (VCB) is a Windows application that provides a LAN-free Fibre Channel or iSCSI-based backup solution that offloads the backup processing to a dedicated physical server. VCB takes advantage of the snapshot functionality in ESX Server to mount the snapshots into the file system of the dedicated VCB server. Once the respective virtual machine files are mounted, entire virtual machines or individual files can be backed up using third-party backup tools. VCB scripts integrate with several major third-party backup solutions to provide a means of automating the backup process. Figure 1.8 details a VCB implementation.

Figure 1.8 VMware Consolidated Backup (VCB) is a LAN-free online backup solution that uses a Fibre Channel or iSCSI connection to expedite and simplify the backup process.

In the world of information technology management, there are many models that reflect the project management lifecycle. In each of the various models, it is almost guaranteed that you'll find a step that involves planning. Though these models might stress this stage of the lifecycle, the reality is that planning is often passed over very quickly if not avoided altogether. However, a VI3 project requires careful planning due to hardware constraints for the ESX Server software. In addition, the server planning can have a significant financial impact when calculating the return on investment for a VI3 deployment.

VMware ESX Server includes stringent hardware restrictions. Though these hardware restrictions provide a limited environment for deploying a supported virtual infrastructure, they also ensure the hardware has been tested and will function as expected as a platform for VMware's VMkernel hypervisor. Although not every vendor or whitebox configuration can play host to ESX Server, the list of supported hardware platforms will continue to change as newer models and more vendors are tested by VMware. The official VMware Systems Compatibility guide can be found on VMware's website at http://www.vmware.com/pdf/vi3_systems_guide.pdf. With a quick glance at the systems compatibility guide, you will notice Dell, HP, and IBM among a dozen or so lesser-known vendors. Within the big three, you will find different server models that provide a tested and supported platform for ESX Server.

A deeper look into a specific vendor, like Dell, will reveal that the compatibility guide identifies server models of all sizes (see Figure 2.1) as valid ESX Server hosts, including:�

♦ The 1U PowerEdge 1950

♦ The 2U PowerEdge 2950 and 2970

♦ The 4U PowerEdge R900

♦ The 6U PowerEdge 6850 and 6950

♦ The PowerEdge 1955 Blade Server

Figure 2.1 Servers on the compatibility list come in various sizes and models.

In today's world, every company is anxious and hoping for the opportunity for growth. Expansion is often a sign that a company is fiscally successful and in a position to take on the new challenges that come with an increasing product line or customer base. For the IT managers, expansion means planning and budgeting for human capital, computing power, and spatial constraints.

As many organizations are figuring out, virtualization is a means of reducing the costs and overall headaches involved with either consistent or rapid growth. Virtualization offers solutions that help IT managers address the human, computer, and spatial challenges that accompany corporate demands.

Let's look at a common scenario facing many successful medium-to-large business environments. Take the fictitious company Learn2Virtualize (L2V) Inc. L2V currently has 40 physical servers and an EMC fibre channel storage device in a datacenter in St. Petersburg, Florida. During the coming fiscal year, through acquisitions, new products, and new markets L2V expects to grow to more than 100 servers. If L2V continues to grow using the traditional information systems model, they will buy close to 100 physical servers during their rapid expansion. This will allow them to continue minimizing services on hosts in an effort to harden the operating systems. This practice is not uncommon for many IT shops. As a proven security technique, it is best to minimize the number of services provided by a given server to reduce the exposure to vulnerability across different services. Using physical server deployment will force L2V to look at their existing and future power and datacenter space consumption. In addition, they will need to consider the additional personnel that might be required. With physical server implementations, L2V might be looking at expenses of more than $150,000 in hardware costs alone. And while that might be on the low side, consider that power costs will rise and that server CPU utilization, if it is consistent with industry norms, might sit somewhere between 5 and 10 percent. The return on investment just doesn't seem worth it.

Now let's consider the path to virtualization. Let's look at several options L2V might have if they move in the direction of server consolidation using the VI3 platform. Since L2V already owns a storage device, we'll refrain from including that as part of the return on investment (ROI) calculation for their virtual infrastructure. L2V is interested in the enterprise features of VMotion, DRS, and HA, and therefore they are included in each of the ROI calculations.

Each of the following three ROI calculations identifies various levels of availability, including single server failure, two-server failure, or no consideration for failover. All of the required software licenses have been included as part of the calculation; however, annual licensing fees have not been included since there are several options and they are recurring annual charges.�

In addition to the choice of server vendor, model, and hardware specification, the planning process involves a decision between using ESX Server 3.5 versus ESXi 3.5. This chapter will cover the installation of ESX Server 3.5, while Chapter 13 will examine the specifics of ESXi 3.5.

Installing ESX Server 3.5 can be done in a graphical mode or a text-based installation, which limits the intricacy of the screen configuration during the installation. The graphical mode is the more common of the two installation modes. The text mode is reserved for remote installation scenarios where the wide area network is not strong enough to support the graphical nature of the graphical installation mode.

Before we offer step-by-step instructions for installing ESX Server, it is important to review some of the functional components of the disk architecture upon which ESX Server will be installed. Because of its roots in Red Hat Linux, ESX Server does not use drive letters to represent the partitioning of the physical disks. Instead, like Linux, ESX Server uses mount points to represent the various partitions. Mount points involve the association of a directory with a partition on the physical disk. Using mount points for various directories under the root file system protects the root file system by not allowing a directory to consume so much space that the root becomes full. Since most folks are familiar with the Microsoft Windows operating system, think of the following example. Suppose you have a server that runs Windows using a standard C: system volume label. What happens when the C drive runs out of space? Without going into detail let's just leave the answer as a simple one: bad things. Yes, bad things happen when the C drive of a Windows computer runs out of space. In ESX Server, as noted, there is no C drive. The root of the operating system file structure is called exactly that: the root. The root is noted with the / character. Like Windows, if the / (root) runs out of space, bad things happen. Figure 2.2 compares Windows disk partitioning and notation against the Linux disk partitioning and notation methods.

Figure 2.2 Windows and Linux represent disk partitions in different ways. Windows, by default uses drive letters, while Linux uses mount points.

Readers who have already done ESX Server installs are probably wondering what we could be talking about in this section given that the installation can be completed by simply clicking Next until the Finish button shows up. And though this is true, there are some significant decisions to be made through the installation � decisions that affect the future of the ESX Server deployment as well as decisions that could cause severe damage to company data. For this reason, it is important for the experienced administrator and the installation newbie to read this section carefully and understand how best to install ESX Server to support the current and future demands of the VI3 deployment.

Perform the following steps to install ESX Server 3.5 from a CD:

1. Configure the server to boot from the CD, insert the VMware ESX Server 3.5 CD, and reboot the computer.

2. Select the graphical installation mode by pressing the Enter key at the boot options screen, shown in Figure 2.3.

Figure 2.3 ESX Server 3.5 includes a graphical installation mode, which includes an enhanced GUI and a text-based installation mode better suited for installing over a wide area network.

3.�At the CD Media Test screen, shown in Figure 2.4, click the Skip button to continue with the installation. Click the Test button to identify any errors in the installation media.

Figure 2.4 To prevent installation errors due to bad media, the CD can be tested early in the install procedure.

4. Click the Next button on the Welcome to the ESX Server 3.5 Installer screen.

5. Select the U.S. English keyboard layout, or whichever is appropriate for your installation, as shown in Figure 2.5. Then click the Next button.

Figure 2.5 ESX Server 3.5 offers support for numerous keyboard layouts.

6.�Select the Wheel Mouse (PS/2), shown in Figure 2.6. Or if you choose to match your mouse model exactly, select the appropriate option.

Figure 2.6 ESX Server 3.5 offers support for numerous models of mouse devices.

7.�Select the Yes button to initialize any device to be used for storing the ESX Server 3.5 installation partitions, shown in Figure 2.7.

Figure 2.7 Unknown devices must be initialized for ESX Server 3.5 to be installed.

Warning! You Could Lose Data if You Don't Read This�

If SAN storage has already been presented to the server being installed, it could be possible to initialize SAN LUNs with production data. As a precaution, it is an excellent idea to disconnect the server from the SAN or ensure LUN masking has been performed to prevent the server from accessing LUNs.

Access to the SAN is only required during installation if a boot from SAN configuration is required.�

8.�As shown in Figure 2.8, select the check box labeled I Accept the Terms of the License Agreement and click the Next button.

Figure 2.8 The ESX Server 3.5 license agreement must be accepted; however no licenses are configured during the installation wizard.

9.�As shown in Figure 2.9, select the Recommended radio button option to allow the installation wizard to automatically partition the local disk. Ensure that the local disk option is selected in the Install ESX Server on the drop-down list. To protect any existing VMFS data, ensure that the Keep Virtual Machines and the VMFS (Virtual Machine File System) That Contains Them option is selected.

Figure�2.9 The ESX Server 3.5 installation wizard offers automatic partitioning of the selected disk and protection for any existing data that resides in a VMFS-formatted partition.

10. Click the Yes button on the partition removal warning, shown in Figure 2.10.

11. Review the partitioning strategy, as shown in Figure 2.11, and click the Next button to continue the installation.

Figure 2.10 The ESX Server 3.5 installation wizard offers a warning before removing all partitions on the selected disk.

Figure 2.11 ESX Server 3.5 default partitioning provides a configuration that offers successful installation and system operation.

Stray from the Norm

As discussed in the previous section, it might be necessary to alter the default partitioning strategy. This does not mean that all partitions must be built from scratch. To change the default partition strategy, select the partition to change and click the Edit button.

Start the partition customization by reducing the space allocated to the local partition with a type of VMFS 3. Once this partition is reduced, the other partitions � /boot, /swap, and /var/log � can be edited. After these partitions have been reconfigured, any leftover space can be given back to the local VMFS 3 partition and the installation can proceed.�

12.�Ensure that the ESX Server 3.5 installation wizard has selected to boot from the same drive that was selected for partitioning. By default, the selection should be correct and should not be configurable without selecting the option to allow editing. As shown in Figure 2.12, this screen provides a default configuration consistent with the previous installation configuration. This avoids misconfiguration in which the installation is performed on a local disk but the server is booted from a SAN LUN, or vice versa.

Figure 2.12 An ESX Server 3.5 host should be booted from the same device where the installation partitions have been configured.

13.�As shown in Figure 2.13, select the network interface card (NIC) through which the Service Console should communicate. Assign a valid IP address, as well as subnet mask, default gateway, DNS servers, and host name for the ESX Server 3.5 host.

Figure 2.13 A NIC must be selected and configured for Service Console communication over the appropriate physical network.

If the Service Console must communicate over a virtual LAN (VLAN), enter the appropriate VLAN ID in the VLAN Settings text box.

If virtual machines must communicate over the same physical subnet as the Service Console, leave the Create a Default Network for Virtual Machines option selected. The outcome of this option can always be modified during postinstallation configuration. Once the Network Configuration page is configured correctly, click the Next button.

Do I Have to Memorize the PCI Addresses of My NICs?

Although the configuration screen for the Service Console is not very user friendly with respect to identifying the physical NICs in the computer, it is not a big deal to fix the NIC association should the wrong NIC be selected during the installation wizard. As part of the PostInstallation Configuration section of this chapter, we will detail how to recover if the wrong NIC is selected during the installation wizard.

The bright side is that if your hardware remains consistent, then the PCI addresses would also remain consistent. Therefore, company policy could document the PCI address to be selected during any new ESX Server deployments.

Keep in mind that since the NIC was incorrect, access to the server via SSH, web page, or VI Client will fail. The fix to be detailed later in the chapter requires direct access to the console or an out-of-band management tool like Dell's Remote Access Controller, which provides console access from a dedicated Ethernet port.

�14.�Select the appropriate time zone for the ESX Server host and then click the Next button, as shown in Figure 2.14.

Figure 2.14 ESX Server 3.5 can be configured with one of many time zones from around the world.

15.�Set and confirm a root password, as shown in Figure 2.15.

Figure 2.15 Each ESX Server 3.5 host maintains its own root user and password configuration. The password must be at least six characters.

16.�Review the installation configuration parameters, as shown in Figure 2.16.

Figure 2.16 The installation review offers a final chance to double-check the server configuration.

If everything looks correct, click the Next button to begin the installation procedure.

17. As shown in Figure 2.17, the installation will begin.

18. As shown in Figure 2.18, click the Finish button to reboot the computer once the installation�is complete.

19. During the reboot, the GRUB boot loader will show the boot options, as shown in Figure 2.19. Ensure that the VMware ESX Server option is selected and press the Enter key (or select nothing and the option will be selected by default).

20.�Upon completion of the server reboot, the console session will display the information for accessing the server from a remote computer, as shown in Figure 2.20.

Figure 2.17 Installing ESX Server 3.5

Figure 2.18 The new ESX Server 3.5 host must be rebooted to finalize the installation.

Despite the ease with which ESX Server 3.5 can be installed, it is still not preferable to perform manual, attended installations of a bulk number of servers. Nor is it preferable to perform manual, attended installation in environments that are rapidly deploying new ESX Server hosts. To support large numbers or rapid-deployment scenarios, ESX Server 3.5 can be installed in an unattended fashion.

Installing an ESX Server 3.5 host in an unattended fashion can be done using third-party imaging tools or using the native VMware tools. Using the native tools requires several network-accessible components, including:

♦ An existing ESX Server 3.5 installation

♦ An NFS server accessible by the host to be installed

♦ A copy of the ESX Server 3.5 installation media

♦ An installation script with the appropriate configuration parameters

Figure 2.19 ESX Server 3.5 uses the GRUB boot loader.

Figure 2.20 After a reboot, the console offers the data necessary for accessing the server from a remote computer.

Figure 2.21 details the infrastructure components needed to complete an unattended ESX Server 3.5 installation using the tools built into ESX Server 3.5.

The unattended installation procedure involves booting the computer and reading the installation files, and reading the unattended installation script. The destination host can be booted from CD, floppy, or PXE boot and then directed to the location of the installation files and answer files. The installation files and/or answer script can be stored and accessed from any of the following locations:

♦ An HTTP URL

♦ A shared NFS directory

♦ An FTP directory

♦ A CD (install files only)

♦ A floppy disk (answer files only)

Table 2.4 outlines the various methods and the boot options required for each option set. The boot option is typed at the boot prompt on ESX Server 3.5 graphical versus text mode selection screen.

Figure 2.21 Performing an unattended ESX Server 3.5 installation requires the proper network servers and services.

Table 2.4: Unattended Installation Methods

If the computer boots from	And the media is stored on a	And the answer file is stored on a	Then the boot option is
PXE	(Media) URL	(Answer) URL	esx ks=<answer URL> method=<media URL> ksdevice=<NIC>
CD	CD	URL	esx ks=<answer URL> ksdevice=<NIC>
CD	CD	Floppy	esx ks=<floppy drive>
Floppy	URL	Floppy	esx ks=<floppy drive>

The kickstart answer file is created from a web-based wizard accessible from the default home page of an ESX Server host, as shown in Figure 2.22.

By default, an ESX Server 3.5 host is not configured to allow access to the scripted installer. An error, shown in Figure 2.23, identifies clearly that a given host has not been configured.

Figure 2.22 The home page for an ESX Server host provides access to the Scripted Installer, which generates an answer file through a web-based wizard.�

Figure 2.23 Access to the scripted installer must be enabled on an ESX Server 3.5 host.

Perform the following steps to enable the Scripted Installer on an ESX Server 3.5 host and to create a kickstart file:

1. Establish a console session with an ESX Server 3.5 host.

2. Type the following command:

cd /usr/lib/vmware/webAccess/tomcat/apache-tomcat-5.5.17/webapps/ui/WEB-INF

3.�Type the following command to get a list of all files and folders in the current directory:

ls

4. Type the following command:

nano -w struts-config.xml

5. Comment out the following line by adding <!-- to the front of the line and --> to the end of the line, as shown here and in Figure 2.24:

<!-- action path = "/scriptedInstall"�type="org.apache.struts.actions.

ForwardAction"�parameter="/WEB-INF/jsp/scriptedInstall/disabled.jsp"�-->

Figure 2.24 Enabling the Scripted Installer requires minor editing of the struts-config.xml file.

6.�Uncomment the following lines by removing the <!-- and --> that precede and conclude�the series of lines, as shown here and in Figure 2.24:

<action path="scriptedInstall" type="com.vmware.webcenter.scripted.ProcessAction">

�<forward name="scriptedInstall.form1"�path="/Web-INF/jsp/scriptedInstall/form1.jsp" />

�<forward name="scriptedInstall.form2"�path="/Web-INF/jsp/scriptedInstall/form2.jsp" />

�<forward name="scriptedInstall.form3"�path="/Web-INF/jsp/scriptedInstall/form3.jsp" />

�<forward name="scriptedInstall.form4"�path="/Web-INF/jsp/scriptedInstall/form4.jsp" />

�<forward name="scriptedInstall.form5"�path="/Web-INF/jsp/scriptedInstall/form5.jsp" />

�<forward name="scriptedInstall.form6"�path="/Web-INF/jsp/scriptedInstall/form6.jsp" />

�<forward name="scriptedInstall.form7"�path="/Web-INF/jsp/scriptedInstall/form7.jsp" />

</action>

7. Type the following command:

service vmware-webAccess restart

8. Return to the ESX Server 3.5 home page and click the link labeled Log In to the Scripted Installer.

9. The Scripted Install web-based wizard will begin. Select the appropriate options for the unattended installation, as shown in Figure 2.25, and then click the Next button. The options include:

♦�Installation Type: Initial Installation | Upgrade

♦�Installation Method: CD-ROM | Remote | NFS

�♦�This selection identifies where the installation files are located.

♦ Remote Server URL:<URL of remote server when installation method is set to Remote>

♦ Network Method: DHCP | Static IP

�♦�Static will be the more common selection.

♦�Create a default network for VMs: Yes | No

�♦�This option is negligible. If Yes is selected, the VM network can be deleted later. If No is selected, the VM network can be created later.

♦�VLAN:<VLAN ID if Service Console should be on a VLAN>

�♦�Provide a VLAN ID for Service Console only if you know that a VLAN is configured on the physical switch to which the network adapter is connected.

♦ Time Zone

♦ Reboot After Installation: Yes | No

♦ Root password

Figure 2.25 The Scripted Installer wizard defines the installation type and method as well as the Service Console configuration information.

10. As shown in Figure 2.26, set the hostname and IP address information of the server to be installed with the answer file and then click the Next button.

11. Select the check box labeled I Have Read and Accept the Terms of the License Agreement and then click the Next button.

12. As shown in Figure 2.27, configure the disk partitioning strategy and licensing mode for the target server and then click the Next button. Apply any necessary customizations to the partitions. Licensing options include: Post Install | Use License Server | Use Host License File.

13.�Configure the licensing options, as shown in Figure 2.28, and then click the Next button.

Figure 2.26 The Scripted Installer defines the hostname and IP address configuration for the target ESX Server.�

Figure 2.27 The Scripted Installer allows disk partitioning customizations and licensing mode.

14.�Click the Download Kickstart File from the final page of the Scripted Installer, as shown in Figure 2.29.

Since a Windows file share is not an option for the location of a kickstart file, the file must be copied to an appropriate location, which is most commonly an NFS directory. Use WordPad to review the kickstart file created by the Scripted Installer wizard. A sample default file is shown in Figure 2.30.

Figure 2.28 The Scripted Installer automates the configuration of pointing an ESX Server to a license server.

Figure 2.29 The finished kickstart file can be saved to the local computer accessing the Scripted Installer web-based wizard.

Using free tools like Veeam FastSCP (http://www.veeam.com) or WinSCP (http://www.winscp.com), the kickstart file can be copied to an NFS directory that is accessible to the target ESX Server. Once the file is in place on the NFS directory, the unattended installation can be launched from the target server.

Perform the following steps to perform an unattended installation using a CD for the installation files and a remote NFS directory for the kickstart file:

1. Boot the target computer from the ESX Server 3.5 CD.

2. At the installation mode selection screen, type the following command, as shown in Figure 2.31:

esx ks=nfs:<IP address or name of NFS server>:<path to kickstart file> ksdevice=<NIC to use>

3.�The installation will begin and continue until the final reboot.

Kickstart files can be edited directly through WordPad so that the wizard does not have to be executed for each new installation. Unfortunately, the kickstart file does not provide a way of generating unique information for each installation and therefore each install will require a manually created (or adjusted) kickstart file that is specific to that installation � particularly configuration of static information that must be unique like IP address and hostname.

Figure 2.30 A sample kickstart file created by the Scripted Installer wizard, viewed through WordPad�

Figure 2.31 Using a local installation media with a kickstart file on an NFS directory

Kickstart Customizations

You may have noticed the kickstart file creation wizard did not allow for configuration of Service Console NIC or any virtual networking or storage configuration. That's because by default it doesn't. The lack of Service Console NIC configuration can cause access problems because the kickstart installation automatically selects the NIC with the lowest PCI address. If your Service Console is not to be associated with the NIC that has the lowest PCI address, a postinstallation configuration will be required to unlink the current NIC and link the correct NIC. We will cover how to do this in the next section of this chapter.

Kickstart files can be edited to configure postinstallation configuration. These configurations can include Service Console NIC corrections, creation of virtual switches and port groups, storage configuration, and even customizations of Service Console config files for setting up external time servers.

The command-line syntax for virtual networking and storage will be covered in Chapters 3 and 4. The following kickstart file makes many postinstallation changes:

# Advanced Kickstart file with postinstallation configuration.

# Installation Method

cdrom

# root Password

rootpw --iscrypted a6fh$/hkQQrCaeuc0mAe38$.captvmyeT4

# Authconfig

auth --enableshadow --enablemd5

# BootLoader (Grub has to be the default boot loader for ESX Server 3)

bootloader --driveorder=sda --location=mbr

# Timezone (set this time zone to fit your company policy)

timezone --utc UTC

# Do not install the X windowing System

skipx

# Clean Installation or upgrade an existing installation

install

# Text Mode

text

# Network install type (this server will have a static IP address of 172.30.0.105

with a subnet mask of 2555.255.255.0, a gateway of 172.30.0.1, a DNS server

# of 172.30.0.2 and a hostname of silo3505.vdc.local. It will not be configured on a vlan network --bootproto static --ip 172.30.0.105 --netmask 255.255.255.0 --gateway 172.30.0.1 --nameserver 172.30.0.2 --hostname silo3505.vdc.local --vlanid=0

# Language

lang en_US

# Language Support

langsupport --default en_US

# Keyboard

keyboard us

# Mouse

mouse none

# Force a reboot after the install

reboot

# Firewall settings

firewall --disabled

# Clear all Partitions on the local disk sda

clearpart --all --initlabel --drives=sda

# Partitioning strategy for ESX Server host

part /boot --fstype ext3 --size 200 --ondisk sda

part / --fstype ext3 --size 25000 --ondisk sda

part swap --size 1600 --ondisk sda

part None --fstype vmfs3 --size 1 --grow --ondisk sda part None --fstype vmkcore --size 100 --ondisk sda

part /var --fstype ext3 --size 12000 --ondisk sda part /tmp --fstype ext3 --size 2000 --ondisk sda

# VMware Specific Commands for accepting the license agreement, configuring a

license server at 172.30.0.2 on port 270000, and a full license

vmaccepteula

vmlicense --mode=server --server=27000@172.30.0.2 --edition=esxFull

%packages

@base

@ everything

%post

# Create a new file named S11PostInstallConfig that will become an executable

that is run during the first reboot of the ESX Server

cat > /etc/rc.d/rc3.d/S11PostInstallConfig << EOF #!/bin/bash

# Overwrite the resolv.conf file to create primary and secondary DNS entries

cat > /etc/resolv.conf << DNS

nameserver 172.30.0.2 nameserver 172.30.0.3 DNS

# Link vSwitch0 used for Service Console communication to vmnic2 if the vmnic0 was not correct

/usr/sbin/esxcfg-vswitch -U vmnic0 vSwitch0 /usr/sbin/esxcfg-vswitch -L vmnic1 vSwitch0

# Add a vmkernel�port for NAS access named NFSPort, with IP address of 172.30.0.101, and a default gateway of 172.30.0.1 (if required for routing)

/usr/sbin/esxcfg-vswitch -A NFSAccess vSwitch0

/usr/sbin/esxcfg-vmknic -a -i�172.30.0.101 -n 255.255.255.0 NFSport /usr/sbin/esxcfg-route 172.30.0.1

# Add an NFS datastore named NFSDatastore01 with an NFS server at 172.30.0.100 and a shared directory of ISOImages

/usr/sbin/esxcfg-nas -a -o 172.30.0.100 -s /ISOImages NFSDatastore01

#�Enable the Service Console firewall�to allow ntp and iSCSI client firewall�ports

/usr/sbin/esxcfg-firewall�-e ntpClient

/usr/sbin/esxcfg-firewall�-e swISCSIClient

# Add a vmkernel�port named VMotion on a virtual�switch named vSwitch1. The VMkernel

port will�have an IP address of 172.29.0.105

# and a subnet mask of 255.255.255.0

/usr/sbin/esxcfg-vswitch -a vSwitch1

/usr/sbin/esxcfg-vswitch -A VMotion vSwitch1

/usr/sbin/esxcfg-vswitch -L vmnic0 vSwitch1

/usr/sbin/esxcfg-vmknic -a -i�172.29.0.105 -n 255.255.255.0 VMotion

# Add a vswitch named vSwitch2 with a virtual machine port group named ProductionLAN

/usr/sbin/esxcfg-vswitch -a vSwitch2

/usr/sbin/esxcfg-vswitch -L vmnic2 vSwitch2 /usr/sbin/esxcfg-vswitch -A ProductionLAN vSwitch2

# Set up time synchronization for ESX Server

cat > /etc/ntp.conf << NTP

restrict default kod nomodify notap noquery nopeer

restrict 173.30.0.111

172.30.0.111

fudge 127.127.1.0 stratum 10

driftfile /etc/ntp/drift

broadcastdelay 0.008

authenticate yes

keys /etc/ntp/keys

NTP

cat > /etc/ntp/step-tickers << STEP

172.30.0.111

STEP

/sbin/service ntpd start

/sbin/chkconfig --level�3 ntpd on

# Update system clock

/sbin/hwclock --systohc --utc

# The --utc setting in the "timezone" command above eliminates the need for updating

the clock file

#cat > /etc/sysconfig/clock << CLOCK

#ZONE="UTC"

#UTC=true

#ARC=false

#CLOCK

#�Allow incoming/outgoing communications on the Service Console via SSH.

esxcfg-firewall�-e sshServer

esxcfg-firewall�-e sshClient

#�Rename the S11Post_Install_Config file to S11Post_Install_Complete after first execution.�Since file name will�now be incorrect it will

# not be triggered in subsequent ESX Server boot sequences.�EOF dictates end of file.

mv /etc/rc.d/rc3.d/S11Post_Install_Config/etc/rc.d/rc3.d/S11Post_Install_complete

EOF

# Make the S11servercfg file an executable

/bin/chmod +x /etc/rc.d/rc3.d/S11Post_Install_Config

Once the installation of ESX Server is complete, there are several postinstallation changes that either must be set or are just strongly recommended. Among these configurations are adjusting the amount of RAM allocated to the Service Console, changing the physical NIC used by the Service Console, and configuring the ESX Server host to synchronize with an external Network Time Protocol (NTP) server.

During the installation of ESX Server, the NIC selection screen creates a virtual switch bound to the selected physical NIC. The tricky part, as noted earlier, is choosing the correct PCI address that corresponds to the physical NIC connected to the physical switch that makes up the logical�IP subnet from which the ESX Server will be managed. The problem often arises when the wrong PCI address is selected, resulting in the inability to access the Service Console. Figure 2.32 shows the structure of the virtual networking when the wrong NIC is selected and when the correct NIC is selected.

Figure 2.32 The virtual switch used by the Service Console must be associated with the physical switch that makes up the logical subnet from which the Service Console will be managed.

Should the incorrect PCI address be selected, the result is an inability to reach the ESX Server Web Access page after the installation is complete. The simplest fix for this problem is to unplug the network cable from the current Ethernet port and continue trying the remaining ports until the web page is accessible. The problem with this solution is that it puts a quick end to any type of documented standard that dictates the physical connectivity of the ESX Server hosts in a virtual environment.

So what then is the better fix? Is a reinstallation in order? If you like installations, go for it, but there is something much better. A quick visit to the command line and this problem is solved:

Sometimes It's All About the Case�

Remember that ESX Server holds its roots in Linux and therefore any type of command-line management or configuration will always be case sensitive.

1. Log in to the console of the ESX Server using the root user account.�

2. Review the PCI addresses of the physical NICs in the server by typing the following command:

esxcfg-nics -l

3.�The results, as shown in Figure 2.33, will list identifying information for each NIC. Note the PCI addresses and names of each adapter.

Figure 2.33 The esxcfg-nics command provides detailed information about each adapter in an ESX Server host.

4. Review the existing Service Console configuration by typing the following command:

esxcfg-vswitch -l

5. The results, as shown in Figure 2.34, will display the current configuration of the Service Console port association.

Figure 2.34 The esxcfg-vswitch command provides information about the current Service Console configuration.

6.�To change the NIC association, the existing NIC must be unlinked by typing the following command:

esxcfg-vswitch -U vmnic# vSwitch#

In this example the appropriate command would be:

esxcfg-vswitch -U vmnic0 vSwitch0

7.�Use the following command to associate a new NIC with the vSwitch0 used by the Service Console:

esxcfg-vswitch -L vmnic# vSwitch#

If still unsure of the correct NIC, try each NIC listed in the output from step 2. For this example, to associate vmnic1 with a PCI address of 08:07:00, the appropriate command would be:

esxcfg-vswitch -L vmnic1 vSwitch0

8.�Repeat steps 6 and 7 until a successful connection is made to the Web Access page of the ESX Server host.

Adjusting the amount of memory given to the Service Console is not mandatory but is strongly recommended if you have to install third-party applications into the console operating system. These third-party applications will consume memory available to the Service Console. As noted earlier, the Service Console is only granted 272MB of RAM by default, as shown in Figure�2.35, with a hard-coded maximum of 800MB.

Figure 2.35 The Service Console is allocated 272 MB of RAM by default.

The difference of 528MB is, and should be, negligible in relation to the amount of memory in the ESX Server host. Certainly an ESX Server host in a production network would not have less than 8GB of memory. Even that would be the low end. So adding 528MB of memory for use by the Service Console does not place a significant restriction on the number of virtual machines a host is capable of running due to lack of available memory.

Perform the following steps to increase the amount of memory allocated to the Service Console:

1. Use the VI Client to connect to an ESX Server host or VirtualCenter Server installation.

2. Select the appropriate host from the inventory tree on the left and then select the Configuration tab from the details pane on the right.

3. Select Memory from the Hardware menu.

4. Click the Properties link.

5. As shown in Figure 2.36, enter the amount of memory to be allocated to the Service Console in the text box and then click the OK button. The value entered must be between 256 and 800.

Figure 2.36 The amount of memory allocated to the Service Console can be increased to a maximum of 800 MB.

6.�Reboot the ESX Server host. As shown in Figure 2.37, the Configuration tab now reflects the current memory allocated and the new amount of memory to be allocated after a reboot.

Figure 2.37 Altering the amount of memory allocated to the Service Console requires a reboot of the ESX Server host.

Time synchronization in ESX Server is an important configuration because the ramifications of incorrect time run deep. Time synchronization issues can affect things like performance charting, SSH key expirations, NFS access, backup jobs, authentication, and more. After the installation of ESX Server (or in a kickstart script), the host should be configured to perform time synchronization with a reliable time source. This source could be another server on your network or an Internet time source. For the sake of managing time synchronization, it is easiest to synchronize all your servers against one reliable internal time server and then synchronize the internal time server with a reliable Internet time server.

Configuring time synchronization for an ESX Server requires several steps, including Service Console firewall configuration and edits to several configuration files.

Perform the following steps to enable the NTP Client in the Service Console firewall:

1. Use the VI Client to connect directly to the ESX Server host or to a VirtualCenter installation.

2. Select the hostname from the inventory tree on the left and then click the Configuration tab in the details pane on the right.

3. Select Security Profile from the Software menu.

4. As shown in Figure 2.38, enable the NTP Client option in the Firewall Properties dialog box.

5. Alternatively the NTP client could be enabled using the following command:

esxcfg-firewall�-e ntpClient

Type the following command to apply the changes made to the Service Console Firewall: service mgmt-vmware restart

Perform the following steps to configure the ntp.conf and step-tickers files for NTP time synchronization on an ESX Server host:

1.�Log in to a console or SSH session with root privileges. If SSH has not been enabled for the host, log in with a standard user account and use the su - command to elevate to the root user privileges and environment.

Figure 2.38 The NTP Client can be enabled through the Security Profile of an ESX Server host configuration.

In the previous section, we looked at some good examples of virtual network architectures needed to support the physical networking components. Now that you have some design and configuration basics under your belt, let's move on to extending the virtual networking beyond just establishing communication. A NIC team can support any of the connection types discussed in the previous section. Using NIC teams provides redundancy and load balancing of network communications to Service Console, VMkernel, and virtual machines.

A NIC team, shown in Figure 3.20 and Figure 3.21, is defined as a vSwitch configured with an association to multiple physical network adapters (uplinks). As mentioned in the previous section, the ESX Server host can either have a maximum of 32 uplinks spread across multiple vSwitches or be configured as a NIC team on one vSwitch.

Successful NIC teaming requires that all uplinks be connected to physical switches that belong to the same broadcast domain. As shown in Figure 3.22, all of the physical network adapters in the NIC team should be connected to the same physical switch or to physical network adapters connected to physical switches that are connected to one another.

Figure 3.20 Virtual switches, like vSwitch1, with multiple uplinks offer redundancy and load balancing.

Figure 3.21 A NIC team is identified by the association of multiple physical network adapters assigned to a vSwitch.

Figure 3.22 All of the physical network adapters that make up a NIC team must belong to same Layer 2 broadcast domain.

Constructing NIC Teams

NIC teams should be built on physical network adapters located on separate bus architectures. For example, if an ESX Server host contains two on-board network adapters and a PCI-based quad-port network adapter, a NIC team should be constructed using one on-board network adapter and one network adapter on the PCI bus. This design eliminates a single point of failure.

Perform the following steps to create a NIC team using the VI Client:�

1. Use the VI Client to establish a connection to a VirtualCenter server or an ESX Server host.

2. Click the hostname in the inventory panel on the left, select the Configuration tab from the details pane on the right, and then select Networking from the Hardware menu list.

3. Click the Properties for the virtual switch that will be assigned a NIC team and select the Network Adapters tab.

4. Click Add and select the appropriate adapter from the Unclaimed Adapters list, as shown in Figure 3.23.

Figure 3.23 Create a NIC team using unclaimed network adapters that belong to the same Layer 2 broadcast domain as the original adapter.

5. Adjust the Policy Failover Order as needed to support an Active/Standby configuration.

6. Review the summary of the virtual switch configuration, click Next, and then click Finish.

The load-balancing feature of NIC teaming does not function like the load-balancing feature of advanced routing protocols. Load balancing across a NIC team is not a product of identifying the amount of traffic transmitted through a network adapter and shifting traffic to equalize data flow through all available adapters. The load-balancing algorithm for NIC teams in a vSwitch is a balance of the number of connections � not the amount of traffic. NIC teams on a VI vSwitch can be configured with one of the following three load-balancing policies:

♦ vSwitch port-based load balancing (default)

♦ Source MAC-based load balancing

♦ IP hash-based load balancing

Outbound Load Balancing

The load-balancing feature of NIC teams on a vSwitch only applies to the outbound traffic.�

The vSwitch port-based load-balancing policy that is used by default uses an algorithm that ties each virtual switch port to a specific uplink associated with the vSwitch. The algorithm will maintain an equal number of port-to-uplink assignments across all uplinks to achieve load balancing. As shown in Figure 3.24, this policy setting ensures that traffic from a specific virtual network adapter connected to a virtual switch port will consistently use the same physical network adapter. In the event that one of the uplinks fails, the traffic from the failed uplink will failover to another physical network adapter.

Figure 3.24�The vSwitch port-based load-balancing policy assigns each virtual switch port to a specific uplink. Failover to another uplink occurs when one of the physical network adapters experiences failure.

The second load-balancing policy available for a NIC team is the source MAC-based policy, shown in Figure 3.25. This policy is susceptible to the same pitfalls as the vSwitch port-based policy simply because the static nature of the source MAC address is the same as the static nature of a vSwitch port assignment. Like the vSwitch port-based policy, the source MAC-based policy is best used when the number of virtual network adapters exceeds the number of physical network adapters. In addition, virtual machines are still not capable of using multiple physical adapters unless configured with multiple virtual network adapters. Multiple virtual network adapters inside the guest operating system of a virtual machine will provide multiple source MAC addresses and therefore offer an opportunity to use multiple physical network adapters.�

The third load-balancing policy available for NIC teams is the IP hash-based policy, also called the out-IP policy. This policy, shown in Figure 3.26, addresses the limitation of the other two policies that prevents a virtual machine from accessing two physical network adapters without having two virtual network adapters. The IP hash-based policy uses the source and destination IP addresses to determine the physical network adapter for communication. This algorithm then allows a single virtual machine to communicate over different physical network adapters when communicating with different destinations.

Figure 3.25 The source MAC-based load-balancing policy, as the name suggests, ties a virtual network adapter to a physical network adapter based on the MAC address.

Balancing for Large Data Transfers

Although the IP hash-based load-balancing policy can more evenly spread the transfer traffic for a single virtual machine, it does not provide a benefit for large data transfers occurring between the same source and destination systems. Since the source-destination hash will be the same for the duration of the data load, it will only flow through a single physical network adapter.�

A vSwitch with a NIC team set to use the IP hash-based load-balancing policy should have all physical network adapters connected to the same physical switch to support link aggregation. ESX Server supports standard 802.3ad teaming in static (manual) mode, but does not support the Link Aggregation Control Protocol (LACP) or Port Aggregation Protocol (PAgP) commonly found on switch devices. Link aggregation will increase throughput by combining the bandwidth of multiple physical network adapters for use by a single virtual network adapter of a virtual machine.�

Follow these steps to alter the load-balancing policy of a vSwitch with a NIC team:

1.�Use the VI Client to establish a connection to a VirtualCenter server or an ESX Server host.

2.�Click the hostname in the inventory panel on the left, select the Configuration tab from the details pane on the right, and then select Networking from the Hardware menu list.�

3. Click the Properties for the virtual switch, select the name of virtual switch from the Configuration list, and then click the Edit button.

4. Select the NIC Teaming tab and then select the desired load-balancing strategy from the Load Balancing drop-down list.

5. Click OK and then click Close.

Figure 3.26 The IP hash-based policy is a more scalable load-balancing policy that allows virtual machines to use more than one physical network adapter when communicating with multiple destination hosts.

Now that the load-balancing policies are explained, let's take a deeper look at the failover and failback of uplinks in a NIC team. Failover detection in a NIC team can be configured to use either a link status method or a link status and beacon probing method.

The link status failover detection method works just as the name suggests. Failure of an uplink is identified by the link status provided by the physical network adapter. In this case, failure is identified for events like removed cables or power failures on a physical switch. The downside to the link status failover detection setting is its inability to identify misconfigurations or pulled cables that connect the switch to other networking devices (i.e., a cable connecting one switch to another switch.)

The beacon probing failover detection setting, which includes link status as well, sends Ethernet broadcast frames across all physical network adapters in the NIC team. These broadcast frames allow the vSwitch to detect upstream network connection failures and will force failover when ports are blocked by a spanning tree, are configured with the wrong vLAN, or a switch-to-switch connection has failed. When a beacon is not returned on a physical network adapter, the vSwitch triggers the failover notice and reroutes the traffic from the failed network adapter through another available network adapter based on the failover policy. Consider a vSwitch with a NIC team consisting of four physical network adapters, where each adapter is connected to a different physical switch and each physical switch is connected to a single physical switch, which is then connected to a router, as shown in Figure 3.27. When the NIC team is set to the beacon probing failover detection method, a beacon will be sent out over all four uplinks.

Figure 3.27 The beacon probing failover detection policy sends beacons out across the physical network adapters of a NIC team to identify upstream network failures or switch misconfigurations.

Once a failure has been detected, the vSwitch will use either a Rolling Failover or a Failover Order policy setting to continue passing traffic through another physical network adapter. By default, a NIC team is set to use a Rolling Failover policy of No, as shown in Figure 3.28. When the Rolling Failover policy is set to No, the NIC team not only provides a failover policy but also applies a fail-back policy. This means that if a failed physical network adapter is repaired and brought back online, the NIC team will reroute the traffic back to the adapter and relieve the standby adapter that assumed its role during failover. When the Rolling Failover policy is set to Yes, the NIC team will not perform a failback. In this case, the physical network adapter that assumed responsibility for the failed adapter will continue to process the traffic. Meanwhile, the failed adapter that is not back online will function as a standby until another adapter experiences failure.

Figure 3.28 By default, a NIC team is configured with a Rolling Failover policy setting of No.

As an alternative to the Rolling Failover policy, a Failover Order policy can be manually configured. The Failover Order policy involves an administrative assignment of priority to each of the physical network adapters in the NIC team. As shown in Figure 3.29, a vSwitch with a NIC team set to use a Failover Order policy can have physical network adapters designated as active and standby.

Perform the following steps to configure the Failover Order policy for a NIC team:

1. Use the VI Client to establish a connection to a VirtualCenter server or an ESX Server host.

2. Click the hostname in the inventory panel on the left, select the Configuration tab from the details pane on the right, and then select Networking from the Hardware menu list.

3. Click the Properties for the virtual switch, select the name of virtual switch from the Configuration list, and then click the Edit button.

4. Select the NIC Teaming tab.

5. Use the Move Up and Move Down buttons to adjust the order of the network adapters and their location within the Active Adapters, Standby Adapters, and Unused Adapters lists, as shown in Figure 3.30.

6. Click OK and then click Close.

Figure 3.29 The Failover Order policy allows for the designation of active, standby, and unused network adapters for a NIC team.

Figure 3.30 Failover order for a NIC team is determined by the order of network adapters as listed in the Active Adapters, Standby Adapters, and Unused Adapters lists.

When a failover event occurs on a vSwitch with a NIC team, the vSwitch is obviously aware of the event. The physical switch that the vSwitch is connected to, however, will not know immediately. As shown in Figure 3.31, a NIC Team includes a Notify Switches configuration setting which, when set to Yes, will allow the physical switch to immediately learn of any of the following changes:

♦ A virtual machine is powered on (or any other time a client registers itself with the vSwitch)

♦ A VMotion occurs

♦ A MAC address is changed

♦ A NIC team failover or failback has occurred

Figure 3.31 The Notify Switches option allows physical switches to be notified of changes in NIC teaming configurations.

In any of these events, the physical switch is notified of the change using the Reverse Address Resolution Protocol (RARP). RARP updates the lookup tables on the physical switches and offers the shortest latency when a failover event occurs.

Although the VMkernel works proactively to keep traffic flowing from the virtual networking components to the physical networking components, VMware recommends taking the following actions to minimize networking delays:

♦�Disable Port Aggregation Protocol (PAgP) and Link Aggregation Control Protocol (LACP) on the physical switches

♦ Disable Dynamic Trunking Protocol (DTP) or trunk negotiation

♦ Disable Spanning Tree Protocol (STP)

Virtual Switches with Cisco Switches

VMware recommends configuring Cisco devices to use PortFast mode for access interfaces or PortFast trunk mode for trunking interfaces.�

Even though the vSwitches created in the VMkernel are considered to be �dumb switches�, they can be configured with vSwitch security policies to enhance or ensure Layer 2 security. Security policies can be applied at the vSwitch or at the lower-level connection types configured on a vSwitch and include the following three security options:

♦ Promiscuous Mode

♦ MAC Address Changes

♦ Forged Transmits

Applying a security policy to the vSwitch is effective, by default, for all connection types within the switch. However, if a connection type, or port group, is configured with a competing security policy, it will override the policy set at the vSwitch. As in the example in Figure 3.35, if a vSwitch is configured with a security policy that rejects the use of MAC address changes but a virtual machine port group on the switch is configured to accept MAC address changes, then any virtual machines connected to that port group will be allowed to communicate even though it is using a MAC address that differs from what is configured in its VMX file.

Figure 3.35�Security policies at the switch level are effective by default for all connection types on the switch. Security policies at the connection type (port group) level override the policy set at the virtual switch.

The Promiscuous Mode option is set to Reject by default to prevent virtual network adapters from observing any of the traffic submitted through the vSwitch. For enhanced security, allowing�Promiscuous mode is not recommended because it is an insecure mode of operation that allows virtual adapters to access traffic other than its own. Despite the security concerns, there are valid reasons for permitting a switch to operate in Promiscuous mode. An intrusion detection system (IDS) requires the ability to identify all traffic to scan for anomalies and malicious patterns of traffic. To support the use of the IDS without overextending the reduced security of Promiscuous mode, you can create a dedicated virtual machine port group for use with the IDS. As shown in Figure 3.37, the virtual switch security policy will remain at the default setting of Reject for the Promiscuous Mode option, while the virtual machine port group for the IDS will be set to Accept. This setting will override the virtual switch, allowing the IDS to monitor all switch traffic.

Figure 3.36 The default security profile for a virtual switch prevents Promiscuous Mode but allows MAC Address Changes and Forged transmits.

When a virtual machine is created with one or more virtual network adapters, a MAC address is generated for each virtual adapter. Just as Intel, Broadcom, and others manufacture network adapters and include unique MAC address strings, VMware is also a network adapter manufacturer that has its own MAC prefix to ensure uniqueness. Of course, VMware doesn't actually manufacture anything, since the product exists as a virtual NIC in a virtual machine. The six-byte, randomly generated MAC addresses for a virtual machine can be seen in the configuration file�(.vmx) of the virtual machine, as shown in Figure 3.38. A VMware-assigned MAC address begins with the prefix 00:50:56 or 00:0C:29. The value of the fourth set (XX) cannot exceed 3F to prevent conflicts with other VMware products, while the fifth and sixth sets (YY:ZZ) are generated randomly based on the Universally Unique Identifier (UUID) of the virtual machine that is tied to the location of the virtual machine. For this reason, when a virtual machine location is changed a prompt will appear prior to successful boot. The prompt will inquire about keeping the UUID or generating a new UUID, which helps prevent MAC address conflicts.

Figure 3.37 Promiscuous mode, though a reduction in security, is required when using an intrusion detection system.

Figure 3.38 A virtual machine's initial MAC address is automatically generated and listed in the configuration file for the virtual machine.�

Manually Setting a MAC�

Manually configuring a MAC address in the configuration file of a virtual machine will not work unless the first three bytes are VMware-provided prefixes and the last three bytes are unique. If a non-VMware MAC prefix is entered in the configuration file, the virtual machine will not power on.

All virtual machines have two MAC addresses: the initial MAC and the effective MAC. The initial MAC address is the MAC discussed in the previous paragraph that is generated automatically and that resides in the configuration file. The guest operating system has no control over the initial MAC address. The effective MAC address is the MAC address configured by the guest operating system that is used during communication with other systems. The effective MAC address is included in network communication as the source MAC of the virtual machine. By default, these two addresses are identical. To force a non-VMware-assigned MAC address to a guest operating system, change the effective MAC address from within the guest operating system, as shown in Figure�3.39.

Figure 3.39 A virtual machine's source MAC address is the effective MAC address, which by default matches the initial MAC address configured in the VMX file. The effective MAC, however, can be changed in the guest operating system.�

The ability to alter the effective MAC address cannot be removed from the guest operating system. However, the ability to let the system function with this altered MAC address is easily addressable through the security policy of a vSwitch. The remaining two settings of a virtual switch security policy are MAC Address Changes and Forged Transmits. Both of these security policies are concerned with allowing or denying differences between the initial MAC address in the configuration file and the effective MAC address in the guest operating system. As noted earlier, the default virtual switch security is to accept the differences and process traffic as needed.

The difference between the MAC Address Changes and Forged Transmits security settings involves the direction of the traffic. MAC Address Changes is concerned with the integrity of incoming traffic, while Forged Transmits oversees the integrity of outgoing traffic. If the MAC Address Changes option is set to Reject, traffic will not be passed through the vSwitch to the virtual machine (incoming) if the initial and the effective MAC addresses do not match. If the Forged Transmits option is set to Reject, traffic will not be passed from the virtual machine to the vSwitch (outgoing) if the initial and the effective MAC addresses do not match. Figure 3.40 highlights the security restrictions implemented when MAC Address Changes and Forged Transmits are set to Reject.

Figure 3.40 The MAC Address Changes and Forged Transmits security options deal with incoming and outgoing traffic respectively.�

For the highest level of security, VMware recommends setting MAC Address Changes, Forged Transmits, and Promiscuous Mode on each vSwitch to Reject. When warranted or necessary, use port groups to loosen the security for a subset of virtual machines to connect to the port group.�

Real World Scenario

Virtual Switch Policies for Microsoft Network Load Balancing

As with anything, there are, of course, exceptions. For virtual machines that will be configured as part of a Microsoft network load balancing (NLB) cluster set in Unicast mode, the virtual machine port�group must allow MAC Address Changes and Forged Transmits. Systems that are part of an NLB cluster will share a common IP address and virtual MAC address, as shown here:

The shared virtual MAC address is generated by using an algorithm that includes a static component based on the NLB cluster's configuration of Unicast or Multicast mode plus a hexadecimal representation of the four octets that make up the IP address. This shared MAC address will certainly differ from the MAC address defined in the VMX file of the virtual machine. If the virtual machine port group does not allow for differences between the MAC addresses in the VMX and guest operating system, NLB will not function as expected. VMware recommends running NLB clusters in Multicast mode due to these issues with NLB clusters in Unicast mode.�

Perform the following steps to edit the security profile of a vSwitch:�

1. Use the VI Client to establish a connection to a VirtualCenter server or an ESX Server host.

2. Click the hostname in the inventory panel on the left, select the Configuration tab from the details pane on the right, and then select Networking from the Hardware menu list.

3. Click the Properties link for the virtual switch.

4. Click the name of the virtual switch under the Configuration list and then click the Edit button.

5. Click the Security tab and make the necessary adjustments.

6. Click OK and then click Close.�

Follow these steps to edit the security profile of a port group:

1. Use the VI Client to establish a connection to a VirtualCenter server or an ESX Server host.

2. Click the hostname in the inventory panel on the left, select the Configuration tab from the details pane on the right, and then select Networking from the Hardware menu list.

3. Click the Properties link for the virtual switch.

4. Click the name of the port group under the Configuration list and then click the Edit button.

5. Click the Security tab and make the necessary adjustments.

6. Click OK and then click Close.

Managing the security of a virtual network architecture is much the same as managing the security for any other portion of your information systems. Security policy should dictate that settings be configured as secure as possible to err on the side of caution. Only with proper authorization, documentation, and change management processes should security be reduced. In addition, the reduction in security should be as controlled as possible to affect the least number of systems if not just the systems requiring the adjustments.

After you finish your debate on fibre channel versus iSCSI and you purchase one or the other, you will then have to spend some time devising the proper procedure for managing and implementing LUNs. We are discussing LUN creation and management separately from the fibre channel and iSCSI sections because of its independence from the actual storage architecture. Details on the configuration of fibre channel and iSCSI will follow.

A logical unit number (LUN) is a logical configuration of disk space carved from an underlying set of physical disks. The physical disks on which LUNs are configured are most often arranged as a Redundant Array of Independent Disks (RAID) to support performance and/or redundancy for the data to be stored on the LUN. This section will look at RAID architectures, LUN addressing, and the age-old question of many little LUNs versus fewer big LUNs.

No matter your storage device, fibre channel or iSCSI, you will need to create LUNs, or at least work closely with someone who will create LUNs for you. Virtual machine performance can, in some cases, come down to a matter of having a solid LUN strategy in place for the activity level�of that VM. Choosing the right RAID level for a LUN is therefore an integral part of your VI3 implementation. The most common types of RAID configurations are:

RAID 0�Disks configured in RAID 0 do not offer any type of redundancy or protection against drive failure. RAID 0 does, however, provide the fastest performance times because data is written simultaneously to all drives involved. A RAID 0 volume, also commonly referred to as a stripe, can have two or more disks as part of the array. Figure 4.4 outlines the structure of a RAID 0 configuration.

Figure 4.4 A RAID 0 disk configuration provides high-speed performance for data stored across a series of disks.

RAID 1�A RAID 1 configuration puts two identically sized allocations of space from two drives together for the provisioning of a backup strategy that allows for either of the drives to fail and still maintain data. A RAID 1 volume, also commonly referred to as a mirrored array, loses 50 percent of the available drive space. For example, two 500GB LUNs configured as a RAID 1 array will only provide 500GB of storage. Figure 4.5 outlines the structure of a RAID 1 configuration.

Figure 4.5 A RAID 1 disk array, or mirrored volume, provides redundancy in the event of a single drive failure. A RAID 1 array is the most expensive disk type because it incurs a 50 percent loss in the amount of storage.

RAID 5�A RAID 5 array writes data and parity simultaneously to all drives involved in the array. RAID 5 arrays provide redundancy in the event of a single drive failure by writing parity in equal increments across all drives. Parity is a mathematical calculation that allows N-1 drives the ability to make up the data on any other drive. A RAID 5 is the most efficient array�when looking at disk space loss. The RAID 5 array only loses one drive's worth of space. For example, a RAID 5 array made up of four 250GB hard drives will have approximately 750GB of storage space available. Figure 4.6 outlines the structure of a RAID 5 configuration.

Figure 4.6 A RAID 5 array is commonly used because of its data protection and limited loss of space. Both data and parity are written equally across all drives in the array.

RAID 1+0/RAID 0+1�For a more advanced disk array configuration, a RAID 1+0 or RAID 0+1 might be used. These structures combine the use of RAID 0 and RAID 1 technologies. RAID 1+0 involves mirroring a stripe, while RAID 0+1 involves striping several mirrors.

One of the most common challenges facing VI3 administrators is the process of sizing LUNs. Administrators can quite easily determine the RAID levels of a LUN (as explained in the preceding paragraphs), but sizing the LUN is an entirely different challenge. To determine the size of a LUN, administrators must have a game plan for testing virtual machine performance or have a solid understanding of the functions of the virtual machine(s) to be located on a LUN.�

How Much Space Does a Virtual Machine Consume?�

There is no definitive answer to this question, simply because administrators can choose to build virtual machines with virtual hard drives of varying sizes. There is, however, a generic but effective way of determining size requirements for a virtual machine. For each virtual machine, there is a set of associated files that have a direct influence on storage requirements, including the virtual machine hard disk, the suspended state, and the virtual machine swap file. Use the following formula to calculate the storage requirements for a virtual machine:

<size of the virtual machine hard disks> + <size of suspended state for virtual machine> + <memory limit - memory reservation> = minimum storage requirement for a virtual machine

For example, if a virtual machine consisted of a 25GB Virtual Machine Disk Format (VMDK) file, a memory limit of 4GB, and a memory reservation of 2GB, the minimum storage requirement could be calculated as follows:

25GB (virtual machine) + 25GB (suspended state) + 2GB (limit - reservation) = 52GB

In this case, a 55GB LUN would suffice for the virtual machine. However, keep in mind that if you decide to increase the RAM limit to 8GB or to reduce the RAM reservation to 0, you will then be jeopardizing the accuracy of the minimum storage requirement calculation.

Storage space for snapshots should also be considered, even if only for the temporary duration of the snapshot process (see Chapter 6). Luckily for SAN and VI3 administrators, the placement of a virtual machine's files is not a permanent decision. Moving a virtual machine to different storage locations is a simple but offline process.�

When it comes to LUN design and management, VMware defines two common philosophies:�

♦ The adaptive scheme

♦ The predictive scheme

Each scheme offers its own set of advantages and disadvantages to VI3 administrators. Undoubtedly, you will find that neither option is the appropriate solution in every situation. It is safe to say that most administrators will find themselves incorporating a blend of both philosophies as a means of compromise and earning the best of both worlds.

We'll start by introducing the adaptive scheme because of its simplicity. The adaptive scheme involves creating a small number of larger LUNs for the storage of virtual machines. The adaptive scheme results in fewer requirements on the part of the SAN administrator, less effort when performing LUN masking, fewer datastores to manage, and better opportunities for virtual disk resizing.

The downside to the adaptive scheme is the increased contention for LUN access across all of the virtual machines in the datastore. For example, if a 500GB LUN holds the virtual machine disk files for 10 virtual machines, then there will be contention among all of the virtual machines for access to the LUN. This might not be an issue, as the virtual machines' disk files residing on the LUN may be for virtual machines that are not disk intensive � that is, they do not rely heavily on hard disk input/output (I/O). For the adaptive scheme to be a plausible and manageable solution, VI3 administrators must be proactive in monitoring the virtual machines stored together on a LUN. When the performance of the virtual machines begins to reach unacceptable levels, administrators must look to creating more LUNs to be made available for new or existing virtual machines. Figure 4.7 shows an implementation of the adaptive scheme for storing virtual machines.

The predictive scheme overcomes the limitations of the adaptive scheme but introduces administrative challenges of its own. The predictive involves the additional administrative effort of customizing LUNs to be specific for individual virtual machines. Take the following example: When administrators deploy a new server to play host to a database application, it is a common practice to enhance database performance by implementing multiple disks with characteristics specific to the data stored on the disk. On a database server, this often means a RAID 1 (mirror) volume for the operating system, a RAID 5 volume for the database files, and another RAID 1 volume for the database logs. Using the predictive scheme to architect a LUN solution for this database server would result in three SAN LUNs built on RAID arrays as needed by the database server. The sizes of the LUNs would depend on the estimated sizes of the operating system, database, and log files. Figure 4.8 shows this type of predictive approach to LUN design. Table 4.2 outlines all of the pros and cons for each of the LUN design strategies.

Figure 4.7 The adaptive scheme involves creating a small number of LUNs that are larger in size and play host to virtual machine disk files for multiple virtual machines.

Figure 4.8 The predictive scheme, though administratively more involved, offers better performance measures for critical virtual machines.�

Table 4.2:�Adaptive and Predictive Scheme Comparisons�

Type of Scheme	Pros	Cons
Adaptive	Less need for SAN administrator Easy resizing of virtual disks Easy snapshot management Less volume management	Possible undersizing of LUN, resulting in greater administrative effort to create new LUNs Possible oversizing of LUN, resulting in wasted storage space
Predictive	Less contention on each VMFS More flexible share allocation and management Less wasted space on SAN storage RAID specificity for VMs Greater multipathing capability Support for Microsoft clusters Greater backup policy flexibility	Greater administrative overhead for LUN masking Greater administrative effort involved in VMotion, DRS, and HA planning

As we noted earlier in this section, the most appropriate solution will most likely involve a combination of the two design schemes. You may find a handful of virtual machines where performance is unaffected by storing all the virtual machine disk files on the same LUN, and at the same time you will find those virtual machines that require a strict nonsharing approach for the virtual machine disk files. But in between the two extremes, you will find the virtual machines that require specific RAID characteristics but, at the same time, can share LUN access with multiple virtual machines. Figure 4.9 shows a LUN design strategy that incorporates both the adaptive and predictive schemes as well as a hybrid approach.

Figure 4.9 Neither the adaptive nor the predictive scheme will be the most appropriate solution in all cases, which means most environments will be built on a hybrid solution that involves both philosophies.

With all of the effort that will be put into designing the appropriate LUN structures, you will undoubtedly run into situations in which the design will require change. Luckily for the VI3 administrative community, the product is very flexible in the way virtual machine disk files are managed. In just a few short steps, a virtual machine's disk files can be moved from one LUN to another. The simplified nature of relocating disk files means that if you begin with one approach and discover it does not fit your environment, you can easily transition to a more suitable LUN structure. In Chapter 6, we'll detail the steps required to move a virtual machine from one datastore to another.

VMware Infrastructure 3 offers three shared storage options for locating virtual disk files, ISO files, and templates. Each storage technology presents its own benefits and challenges and requires careful attention. Despite their differences, there is often room for two or even all three of the technologies within a single virtualized environment.

Despite its high cost, many companies rely on fibre channel storage as the backbone for critical data storage and management. The speed and security of the dedicated fibre channel storage network are attractive assets to companies looking for reliable and efficient storage solutions.

Fibre channel SANs can run at either 2GFC or 4GFC speeds and can be constructed in three different topologies: point-to-point, arbitrated loop, or switched fabric. The point-to-point fibre channel architecture involves a direct connection between the server and the fibre channel storage device. The arbitrated loop, as the name suggests, involves a loop created between the storage device and the connected servers. In either of these cases, a fibre channel switch is not required. Each of these topologies places limitations on the scalability of the fibre channel architecture by limiting the number of nodes that can connect to the storage device. The switched fabric architecture is the most common and offers the most functionality, so we will focus on it for the duration of this chapter and throughout the book. The fibre channel switched fabric includes a fibre channel switch that manages the flow of the SCSI communication over fibre channel traffic between servers and the storage device. Figure 4.10 displays the point-to-point and arbitrated loop architectures.

Figure 4.10 Fibre channel SANs can be constructed as point-to-point or arbitrated loop architectures.

The switched fabric architecture is more common because of its scalability and increased reliability. A fibre channel SAN is made up of several different components, including:

Logical unit numbers (LUNs)�A logical configuration of disk space created from one or more underlying physical disks. LUNs are most commonly created on multiple disks in a RAID configuration appropriate for the disk usage. LUN design considerations and methodologies will be covered later in this chapter.

Storage device�The storage device houses the disk subsystem from which the storage pools or LUNs are created.

Storage Processor (SP)�One or more storage processors (SPs) provide connectivity between the storage device and the host bus adapters in the hosts. SPs can be connected directly or through a fibre channel switch.

Fibre channel switch�A hardware device that manages the storage traffic between servers and the storage device. Although devices can be directly connected over fibre channel networks, it is more common to use a fibre channel switched network. The term fibre channel fabric refers to the network created by using fibre-optic cables to connect the fibre channel switches to the HBAs and SPs on the hosts and storage devices, respectively.

Host bus adapters (HBAs)�A hardware device that resides inside a server that provides connectivity to the fibre channel network through a fibre-optic cable.

These SAN components make up the infrastructure that processes storage requests and manages the flow of traffic among the nodes on the network. Figure 4.11 shows a commonly configured fibre channel storage area network with two ESX Servers, redundant fibre channel switches, and a storage device.

Figure 4.11 Most storage area networks consist of hosts, switches, and a storage device interconnected to provide servers with reliable and redundant access to storage pools residing in the storage device.

A SAN can be an expensive investment, predominantly because of the redundant hardware built into each of the segments of the SAN architecture. As shown in Figure 4.11, the hosts were outfitted with multiple HBAs connected to the fibre channel fabric, which consisted of multiple fibre channel switches connected to multiple storage processors in the storage device. The trade-off for the higher cost is less downtime in the event of a single piece of hardware failing in the SAN structure.

Now that we have covered the hardware components of the storage area network, it is important that, before moving into ESX specifics, we discuss how the different SAN components communicate with one another.

Each node in a SAN is identified by a globally unique 64-bit hexadecimal World Wide Name (WWN) or World Wide Port Name (WWPN) assigned to it. A WWN will look something like this:

22:00:00:60:01:B9:A7:D2

The WWN for a fibre channel node is discovered by the switch and is then assigned a port address upon login to the fabric. The WWN assigned to a fibre channel node is the equivalent of the globally unique Media Access Control (MAC) address assigned to network adapters on Ethernet networks.

Once the nodes are logged in and have been provided addresses they are free to begin communication across the fibre channel network as determined by the zoning configuration on the fibre channel switches. The process of zoning involves the configuration of a set of access control parameters that determine which nodes in the SAN architecture can communicate with other nodes on the network. Zoning establishes a definition of communication between storage processors in the storage device and HBAs installed on the ESX Server hosts. Figure 4.12 shows a fibre channel zoning configuration.

Figure 4.12 Zoning a fibre channel network at the switch level provides a security boundary that ensures that host devices do not see specific storage devices.

Zoning is a highly effective means of preventing non-ESX hosts from discovering storage volumes that are formatted as VMFS. This process effectively creates a security boundary between fibre channel nodes that simplifies management in large SAN environments. The nodes within a zone, or segment, of the network can communicate with one another but not with other nodes outside their zone. The zoning configuration on the fibre channel switches dictates the number of targets available to an ESX Server host. By controlling and isolating the paths within the switch fabric, the switch zoning can establish strong boundaries of fibre channel communication.

In most VI3 deployments, only one zone will be created since the VMotion, DRS, and HA features require all nodes to have access to the same storage. That is not to say that larger, enterprise VI3 deployments cannot realize a security and management advantage by configuring multiple zones to establish a segregation of departments, projects, or roles among the nodes. For example, a large enterprise with a storage area network that supports multiple corporate departments (i.e., marketing, sales, finance, and research) might have ESX Server hosts and LUNs for each respective department. In an effort to prevent any kind of cross-departmental LUN access, the switches can establish a zone for each department ensuring only the appropriate LUN access. Proper fibre channel switch zoning is a critical tool for separating a test or development environment from a production environment.

In addition to configuring zoning at the fibre channel switches, LUNs must be presented, or not presented, to an ESX Server. This process of LUN masking, or hiding LUNs from a fibre channel node, is another means of ensuring that a server does not have access to a LUN. As the name implies, this is done at the LUN level inside the storage device and not on the fibre channel switch. More specifically, the storage processor (SP) on the storage device allows for LUNs to be made visible or invisible to the fibre channel nodes that are available based on the zoning configuration. The hosts with LUNs that have been masked are not allowed to store or retrieve data from those LUNs.

Zoning provides security at a higher, more global level, whereas LUN masking is a more granular approach to LUN security and access control. The zoning and LUN masking strategies of your fibre channel network will have a significant impact on the functionality of your virtual infrastructure. You will learn in Chapter 9 that LUN access is critical to the advanced VMotion, DRS, and HA features of VirtualCenter.

Figure 4.13 shows a fibre channel switch fabric with multiple storage devices and LUNs configured on each storage device. Table 4.3 describes a LUN access matrix that could help a storage administrator and VI3 administrator work collaboratively on planning the zoning and LUN masking strategies.

Figure 4.13 A fibre channel network consists of multiple hosts, multiple storage devices, and LUNs across each storage device. Every host does not always need access to every storage device or every LUN, so zoning and masking are a critical part of SAN design and configuration.

Fibre channel storage networks are synonymous with �high performance� storage systems. Arguably, this is in large part due to the efficient manner in which communication is managed by the fibre channel switches. Fibre channel switches work intelligently to reduce, if not eliminate, oversubscription problems in which multiple links are funnelled into a single link. Oversubscription results in information being dropped. With less loss of data on fibre channel networks, there is reduced need for retransmission of data and, in turn, processing power becomes available to process new storage requests instead of retransmitting old requests.

Configuring ESX for Fibre Channel Storage

Since fibre channel storage is currently the most efficient SAN technology, it is a common back-end to a VI3 environment. ESX has native support for connecting to fibre channel networks through the host bus adapter. However, ESX Server has limited support for the available storage devices and host bus adapters. Before investing in a SAN, make sure it is compatible and supported by VMware. Even if the SAN is capable of "working" with ESX, it does not mean VMware is going to provide support. VMware is very stringent with the hardware support for VI3; therefore, you should always implement hardware that has been tested by VMware.

Table 4.3:�LUN Access Matrix

Host	SD1	SD2	LUN1	LUN2	LUN3
ESX1	Yes	No	Yes	Yes	No
ESX2	Yes	No	No	Yes	Yes
ESX3	Yes	No	Yes	Yes	Yes
ESX4	No	Yes	Yes	Yes	Yes
ESX5	No	Yes	Yes	No	Yes
ESX6	No	Yes	Yes	No	Yes

Note: The processes of zoning and masking can be facilitated by generating a matrix that defines which hosts should have access to which storage devices and which LUNs.

Always check the compatibility guides before adding new servers, new hardware, or new storage devices to your virtual infrastructure.

Since VMware is the only company (at this time) that provides drivers for hardware supported by ESX, you must be cautious when adding new hardware like host bus adapters. The bright side, however, is that so long as you opt for a VMware-supported HBA, you can be certain it will work without incurring any of the driver conflicts or misconfiguration common in other operating systems.

VMware Fibre Channel SAN Compatibility�

You can find a complete list of compatible SAN devices online on VMware's website at http://www.vmware.com/pdf/vi3_san_guide.pdf. Be sure to check the guides regularly as they are consistently updated. When testing a fibre channel SAN against ESX, VMware identifies compatibility in all of the following areas:

♦ Basic connectivity to the device.

♦ Multipathing capability for allowing access to storage via different paths.

♦ Host bus adapter (HBA) failover support for eliminating single point of failure at the HBA.

♦ Storage port failover capability for eliminating single point of failure on the storage device.

♦ Support for Microsoft Clustering Services (MSCS) for building server clusters when the guest operating system is Windows 2000 Service Pack 4 or Windows 2003.

♦ Boot-from-SAN capability for booting an ESX server from a SAN LUN.

♦ Point-to-point connectivity support for nonswitch-based fibre channel network configurations.

Naturally, since VMware is owned by EMC Corporation you can find a great deal of compatibility between ESX Server and the EMC line of fibre channel storage products (also sold by Dell). Each of the following vendors provides storage products that have been tested by VMware:

♦ 3PAR: http://www.3par.com

♦ Bull: http://www.bull.com

♦ Compellent: http://www.compellent.com

♦ Dell: http://www.dell.com

♦ EMC: http://www.emc.com

♦ Fujitsu/Fujitsu Siemens: http://www.fujitsu.com and http://www.fujitsu-siemens.com

♦ HP: http://www.hp.com

♦ Hitachi/Hitachi Data Systems (HDS): http://www.hitachi.com and http://www.hds.com

♦ IBM: http://www.ibm.com

♦ NEC: http://www.nec.com

♦ Network Appliance (NetApp): http://www.netapp.com

♦ Nihon Unisys: http://www.unisys.com

♦ Pillar Data: http://www.pillardata.com

♦ Sun Microsystems: http://www.sun.com

♦ Xiotech: http://www.xiotech.com

�Although the nuances, software, and practices for managing storage devices across different vendors will most certainly differ, the concepts of SAN storage covered in this book transcend the vendor boundaries and can be used across various platforms.�

Currently, ESX Server supports many different QLogic 236x and 246x fibre channel HBAs for connecting to fibre channel storage devices. However, because the list can change over time, you should always check the compatibility guides before purchasing and installing a new HBA.

It certainly does not make sense to make a significant financial investment in a fibre channel storage device and still have a single point of failure at each server in the infrastructure. We recommend that you build redundancy into the infrastructure at each point of potential failure. As shown in the diagrams earlier in the chapter, each ESX Server host should be equipped with a minimum of two fibre channel HBAs to provide redundant path capabilities in the event of HBA failure. ESX Server 3 supports a maximum of 16 HBAs per system and a maximum of 15 targets per HBA. The 16-HBA maximum can be achieved with four quad-port HBAs or eight dual-port HBAs provided that the server casing has the expansion capability.

Adding a new HBA requires that the physical server be turned off since ESX Server does not support adding hardware while the server is running, otherwise known as a ��hot add�� of hardware. Figure 4.14 displays the redundant HBA and storage processor (SP) configuration of a VI3 environment.

Figure 4.14 An ESX Server configured through Vir-tualCenter with two QLogic 236x fibre channel HBAs and multiple SCSI targets or storage processors (SPs) in the storage device.

Once fibre channel storage is presented to a server and the server recognizes the pools of storage, then the administrator can create datastores. A datastore is a storage pool on an ESX Server host that can be a local disk, fibre channel LUN, iSCSI LUN, or NFS share. A datastore provides a location for placing virtual machine files, ISO images, and templates.

For the VI3 administrator, the configuration of datastores on fibre channel storage is straightforward. It is the LUN masking, LUN design, and LUN management that incur significant administrative overhead (or more to the point, brainpower!). For VI3 administrators who are not responsible for SAN management and configuration, it is essential to work closely with the storage personnel to ensure performance and security of the storage pools used by the ESX Server hosts.

Later in this chapter we'll discuss LUN design in greater detail, but for now let's assume that LUNs have been created and masking has been performed. With those assumptions in place, the work required by the VI3 administrator is quick and easy. Figure 4.15 identifies five LUNs that are available to silo105.vdc.local through its redundant connection to the storage device. The ESX Server silo105.vdc.local has two HBAs connecting to a storage device, with two SPs creating redundant paths to the available LUNs. Although there are six LUNs in the targets list, the LUN with ID 0 is disregarded since it is not available to the ESX Server for storage.

A portion of the ESX Server boot process includes LUN discovery. An ESX Server, at boot-up and by default, will attempt to enumerate LUNs with LUN IDs between 1 and 255.

Even though silo105.vdc.local is presented with five LUNs, it does not mean that all five LUNs are currently being used to store data for the server. Figure 4.16 shows that silo105.vdc.local has three datastores, only two of which are LUNs presented by the fibre channel storage device. With two fibre channel SAN LUNs already in use, silo105.vdc.local has three more LUNs available when needed. Later in this chapter you'll learn how to use the LUNs as VMFS volumes.

Figure 4.15 An ESX Server discovers its available LUNs and displays them under each available SCSI target. Here, five LUNs are available to the ESX Server for storage.

Figure 4.16 An ESX Server host with a local datastore named storage1 (2) and two datastores ISOTemps (1) and LUN10 on a fibre channel storage device.

When an ESX Server host is powered on, it will process the first 256 LUNs (LUN 0 through LUN 255) on the storage devices to which it is given access. ESX will perform this enumeration at every boot, even if many of the LUNs have been masked out from the storage processor side. You can configure individual ESX Server hosts not to scan all the way up to LUN 255 by editing the Disk.MaxLUN configuration setting. Figure 4.17 shows the default configuration of the Disk.MaxLUN value that results in accessibility to the first 256 LUNs.

LUN Masking at the ESX Server

Despite the potential benefit of performing LUN masking at the ESX Server (to speed up the boot process), the work necessary to consistently manage LUNs on each ESX Server may offset that benefit. 1 suggest that you perform LUN masking at the SAN.

To change the Disk.MaxLUN setting, perform the following steps:�

1. Use the VI client to connect to a VirtualCenter Server or an individual ESX Server host.

2. Select the hostname in the inventory tree and select the Configuration tab in the details pane on the right.�

3. In the Software section, click the Advanced Settings link.

4. In the Advanced Settings for <hostname> window, select the Disk option from the selection tree.

5. In the Disk.MaxLUN text box, enter the desired integer value for the number of LUNs to scan.

Figure 4.17 Altering the Disk.MaxLUN value can result in a faster boot or rescan process for an ESX Server host. However, it may also require attention when new LUNs must be made available that exceed the custom configuration.

�You should alter the Disk.MaxLUN parameter only when you are certain that LUN IDs will never exceed the custom value. Otherwise, though a performance benefit might result, you will have to revisit the setting each time available LUN IDs must exceed the custom value.�

Although LUN masking is most commonly performed at the storage processor, as it should be, it is also possible to configure LUN masking on each individual ESX Server host to speed up the boot process.

Let's take an example where an administrator configures LUN masking at the storage processor. Once the masking at the storage processor is complete, the LUNs that have been presented to the hosts are the ones numbered 117 through 127. However, since the default configuration for ESX Server is set to enumerate the first 256 LUNs by default, it will move through each potential LUN even if the storage processor is preventing the LUN from being seen. In an effort to speed up the boot process, an ESX Server administrator can perform LUN masking at the server. In this example, if the administrator were to mask LUN 1 through LUN 116 and LUN 128 through LUN 256, then the server would only be enumerating the LUNs that it is allowed to see and, as a result, would boot quicker. To enable LUN masking on an ESX Server, you must edit the Disk.MaskLUN option (which you access by clicking the Advanced Settings link on the Configuration tab). The Disk.MaskLUN text box requires this format:

<adapter>: <target>: <LUN range lists separated by commas>;

For example, to mask the LUNs from the previous example (1 through 116 and 128 through 256) that are accessible through the first HBA and two different storage processors, you'd enter the following in the Disk.MaskLUN text box entry:

vmhba1:0:1-116,128-256;vmhba1:1:1-116,128-256;

The downside to configuring LUN masking on the ESX Server is the administrative overhead involved when a new LUN is presented to the server or servers. To continue with the previous example, if the VI3 administrator requests five new LUNs and the SAN administrator provisions LUNs with LUN IDs of 136 through 140, the VI3 administrator will have to edit all of the local masking configurations on each ESX Server host to read as follows:�

Vmhba1:0:1-116,128-135,141-254;vmhba1:1:1-116,128-135,141-256;

In theory, LUN masking on each ESX Server host sounds like it could be a benefit. But in practice, masking LUNs at the ESX Server in an attempt to speed up the boot process is not worth the effort. An ESX Server host should not require frequent reboots, and therefore the effect of masking LUNs on each server would seldom be felt. Additional administrative effort would be needed since each host would have to be revisited every time new LUNs are presented to the server.�

ESX LUN Maximums�

Be sure that storage administrators do not carve LUNs for an ESX Server that have ID numbers greater than 255. ESX hosts have a maximum capability of 256 LUNs, beginning with ID 1 and on through ID 255. Clicking the Rescan link located in the Storage Adapters node of the Configuration tab on a host will force the host to identify new LUNs or new VMFS volumes, with the exception that any LUNs with IDs greater than 255 will not be discoverable by an ESX host.�

Although adding a new HBA to an ESX Server host requires you to shut down the server, presenting and finding new LUNs only requires that you initiate a rescan from the ESX Server host.�

To identify new storage devices and/or new VMFS volumes that have been added since the last scan, click the Rescan link located in the Storage node of the Configuration tab. The host will launch an enumeration process beginning with the lowest possible LUN ID to the highest (1 to 255), which can be a slow process (unless LUN masking has been configured on the host as well as the storage processor).

You have probably seen by now, and hopefully agree, that VMware has done a great job of creating a graphical user interface (GUI) that is friendly, intuitive, and easy to use. Administrators also have the ability to manage LUNs from a Service Console command line on an ESX Server host.

The ability to scan for new storage is available in the VI Client using the Rescan link in the Storage Adapters node of the Configuration page, but it is also possible to rescan from a command line.

Establishing Console Access with Root Privileges�

The root user account does not have secure shell (SSH) capability by default. You must set the Permit-RootLogin entry in the /etc/ssh/sshdconfig file to Yes to allow access. Alternatively, you can log on to the console as a different user and use the #su - option to elevate the logon permissions. Opting to use the #su - option still requires that you know the root user's password but does not expose the system to allowing remote root logon via SSH.�

Use the following syntax to rescan vmhba1 from a Service Console command line:

1. Log on to a console session as a nonroot user.

2. Type su�� and then click Enter.

3. Type the root user password and then click Enter.

4. Type esxcfg-rescan vmhba1 at the # prompt.

When multiple vmhba devices are available to the ESX Server, repeat the command, replacing vmhba# with each device.

You can identify LUNs using the physical address (i.e., vmhba#:target#:lun:partition), but the Service Console references the LUNs using the device filename (i.e., sda, sdb, etc.). You can see the device filenames when installing an ESX Server that is connected to a SAN with accessible LUNs. By using an SSH tool (putty.exe) to establish a connection and then issuing the esxcfg commands, you can perform command-line LUN management.

To display a list of available LUNs with their associated paths, device names, and UUIDs, perform the following steps:

1. Log on to a console session as a nonroot user.

2. Type su�� and then click Enter.

3. Type the root user password and then click Enter.

4. Type esxcfg-vmhbadevs -m at the # prompt.

Figure 4.18 shows the resulting output for an ESX Server with an IP address of 172.30.0.106 and a nonroot user named roottoo.

Figure 4.18 The esxcfg commands offer parameters and switches for managing and identifying LUNs available to an ESX Server host.

The UUIDs displayed in the output are unique identifiers used by the Service Console and VMkernel. These values are also reflected in the Virtual Infrastructure Client; however, we do not commonly refer to them because using the friendly names or even the physical paths is much easier.

Fibre channel storage has a strong performance history and will continue to progress in the areas of performance, manageability, reliability, and scalability. Unfortunately, the large financial investment required to implement a fibre channel solution has scared off many organizations looking to deploy a virtual infrastructure that offers all the VMotion, DRS, and HA bells that VI3 provides. Luckily for the IT community, VMware now offers lower-cost (and potentially lower-performance) options in iSCSI and NAS/NFS.

As a response to the needs of not-so-deep-pocketed network administrators, Internet Small Computer Systems Interface (iSCSI) has become a strong alternative to fibre channel. The popularity of iSCSI storage, which offers both lower cost and increasing speeds, will continue to grow as it finds its place in virtualized networks.

iSCSI storage provides a block-level transfer of data using the SCSI communication protocol over a standard TCP/IP network. By using block-level transfer, as in a fibre channel solution, the storage device looks like a local device to the requesting host. With proper planning, an iSCSI SAN can perform nearly as well as a fibre channel SAN � or better. This depends on other factors, but we can dive into those in a moment. And before we make that dive into the configuration of iSCSI with ESX, let's first take a look at the components involved in an iSCSI SAN. Despite the fact that the goals and overall architecture of iSCSI are similar to fibre channel, when you dig into the configuration details, the communication architecture, and individual components of iSCSI, the differences are profound.

The components that make up an iSCSI SAN architecture, shown in Figure 4.19, include:

Hardware initiator�A hardware device referred to as an iSCSI host bus adapter (HBA) that resides in an ESX Server host and initiates storage communication to the storage processor (SP) of the iSCSI storage device.

Software initiator�A software-based storage driver initiator that does not require specific hardware and transmits over standard, supported Ethernet adapters.

Storage device�The physical device that houses the disk subsystem upon which LUNs are built.

Logical unit number (LUN)�A logical configuration of disk space created from one or more underlying physical disks. LUNs are most commonly created on multiple disks in a RAID configuration appropriate for the disk usage. LUN design considerations and methodologies will be covered later in this chapter.

Storage processor (SP)�A communication device in the storage device that receives storage requests from storage area network nodes.

Challenge Handshake Authentication Protocol (CHAP) An authentication protocol used by the iSCSI initiator and target that involves validating a single set of credentials provided by any of the connecting ESX Server hosts.

Ethernet switches Standard hardware devices used for managing the flow of traffic between ESX Server nodes and the storage device.

iSCSI qualified name (IQN)�The full name of an iSCSI node in the format of iqn.<year>-<month>.com.domain:alias. For example, iqn.1998-08.com.vmware:silo1-1 represents the registration of vmware.com on the Internet in August (08) of 1998. Nodes on an iSCSI deployment will have default IQNs that can be changed. However, changing an IQN requires a reboot of the ESX Server host.

iSCSI is thus a cheaper shared storage solution than fibre channel. Of course, the reduced cost does come at the expense of the better performance that fibre channel offers. Ultimately, the�question comes down to that difference in performance. The performance difference can, in large part, reflect the storage design and the disk intensity of the virtual machines stored on the iSCSI LUNs. Although this is true for fibre channel storage as well, it is less of a concern given the greater bandwidth available via a 4GB fibre channel architecture. In either case, it is the duty of the ESX Server administrator and the SAN administrator to regularly monitor the saturation level of the storage network.

Figure 4.19 An iSCSI SAN includes an overall architecture similar to fibre channel, but the individual components differ in their communication mechanisms.

When deploying an iSCSI storage network, you'll find that adhering to the following rules can help mitigate performance degradation or security concerns:

♦ Always deploy iSCSI storage on a dedicated network.

♦ Configure all nodes on the storage network with static IP addresses.

♦ Configure the network adapters to use full-duplex, gigabit autonegotiated recommended communication.

♦ Avoid funneling storage requests from multiple servers into a single link between the network switch and the storage device.

Deploying a dedicated iSCSI storage network reduces network bandwidth contention between the storage traffic and other common network traffic types such as e-mail, Internet, and file transfer. A dedicated network also offers administrators the luxury of isolating the SCSI communication protocol from ��prying eyes�� that have no legitimate need to access the data on the storage device.

iSCSI storage deployments should always utilize dedicated storage networks to minimize contention and increase security. Achieving this goal is a matter of implementing a dedicated switch or switches to isolate the storage traffic from the rest of the network. Figure 4.20 shows the differences and one that is integrated with the other network segments.

If a dedicated physical network is not possible, using a virtual local area network (VLAN) will segregate the traffic to ensure storage traffic security. Figure 4.21 shows iSCSI implemented over a VLAN to achieve better security. However, this type of configuration still forces the iSCSI communication to compete with other types of network traffic.

Figure 4.21 iSCSI can be implemented across VLANs to enhance security.

Figure 4.20 iSCSI should have a dedicated and isolated network.

Figure 4.21 iSCSI communication traffic can be isoloated from other network traffic by using vLANs.

Real World Scenario

A Common iSCSI Network Infrastructure Mistake

A common deployment error with iSCSI storage networks is the failure to provide enough connectivity between the Ethernet switches and the storage device to adequately handle the traffic requests from the ESX Server hosts. In the sample architecture shown here, four ESX Server hosts are configured with redundant connections to two Ethernet switches, which each have a connection to the iSCSI storage device. At first glance, it looks as if the infrastructure has been designed to support a redundant storage communication strategy. And perhaps it has. But what it has not done is maximize the efficiency of the storage traffic.

If each link between the ESX Server hosts and the Ethernet switches is a 1GB link, that means there is a total storage bandwidth of 8GB or 4GB per Ethernet switch. However, the connection between the Ethernet switches and the iSCSI storage device consists of a single 1GB link per switch. If each host maximizes the throughput from host to switch, the bandwidth needs will exceed the capabilities of the switch-to-storage link and will force packets to be dropped. Since TCP is a reliable transmission protocol, the dropped packets will be re-sent as needed until they have reached their destination. All of the new data processing, coupled with the persistent retries of dropped packets, consumes more and more resources and strains the communication, thus resulting in a degradation of server performance.

To protect against funneling too much data to the switch-to-storage link, the iSCSI storage network should be configured with multiple available links between the switches and the storage device. The image shown here represents an iSCSI storage network configuration that promotes redundancy and communication efficiency by increasing the available bandwidth between the switches and the storage device. This configuration will result in reduced resource usage as a result of less packet-dropping and less retrying.

To learn more about iSCSI, visit the Storage Networking Industry Association website at http://www.snia.org/tech_activities/ip_storage/iscsi.

Configuring ESX for iSCSI Storage

I can't go into the details of configuring the iSCSI storage side of things because each product has nuances that do not cross vendor boundaries, and companies don't typically carry an iSCSI SAN from each potential vendor. On the bright side, what I can and most certainly will cover in great detail is how to configure an ESX Server host to connect to an iSCSI storage device using both hardware and software iSCSI initiation.

As noted in the previous section, VMware is limited in its support for hardware device compatibility. As with fibre channel, you should always check VMware's website to review the latest SAN compatibility guide before purchasing any new storage devices. While software-initiated iSCSI has maintained full support since the release of ESX 3.0, hardware initiation with iSCSI devices did not garner full support until the ESX 3.0.1 release. The prior release, ESX 3.0, provided only experimental support for hardware-initiated iSCSI.

VMware iSCSI SAN Compatibility�

Each of the manufacturers listed here provides an iSCSI storage solution that has been tested and approved for use by VMware:

♦ 3PAR: http://www.3par.com

♦ Compellent: http://www.compellent.com

♦ Dell: http://www.dell.com

♦ EMC: http://www.emc.com

♦ EqualLogic: http://www.equallogic.com

♦ Fujitsu Siemens: http://www.fujitsu-siemens.com

♦ HP: http://www.hp.com

♦ IBM: http://www.ibm.com

♦ LeftHand Networks: http://www.lefthandnetworks.com

♦ Network Appliance (NetApp): http://www.netapp.com

♦ Sun Microsystems: http://www.sun.com

An ESX Server host can initiate communication with an iSCSI storage device by using a hardware device with dedicated iSCSI technology built into the device, or by using a software-based initiator that utilizes standard Ethernet hardware and is managed like normal network communication. Using a dedicated iSCSI HBA that understands the TCP/IP stack and the iSCSI communication protocol provides an advantage over software initiation. Hardware initiation eliminates some processing overhead in the Service Console and VMkernel by offloading the TCP/IP stack to the hardware device. This technology is often referred to as the TCP/IP Offload Engine (TOE). When you use an iSCSI HBA for hardware initiation, the VMkernel needs only the drivers for the HBA and the rest is handled by the device.�

For best performance, the iSCSI hardware-based initiation is the appropriate deployment. After you boot the server, the iSCSI HBA will display all its information in the Storage Adapters node of the Configuration tab, as shown in Figure 4.22. By default, as shown in Figure 4.23, iSCSI HBA devices will assign an IQN in the BIOS of the iSCSI HBA. Configuring the hardware iSCSI initiation with an HBA installed on the host is very similar to configuring a fibre channel HBA � the device will appear in the Storage Adapters node of the Configuration tab. The vmhba#'s for fibre channel and iSCSI HBAs will be enumerated numerically. For example, if an ESX Server host includes two fibre channel HBAs labeled vmhba1 and vmhba2, adding two iSCSI HBAs will result in labels of vmhba3 and vmhba4. The software iSCSI adapter in ESX Server 3.5 will always be labeled as vmhba32.. You can then configure the adapter(s) to support the storage infrastructure.

Figure 4.22 Supported iSCSI HBA devices will automatically be found and can be configured in the BIOS of the ESX host.�

Figure 4.23 The BIOS of the QLogic card provides an opportunity to configure the iSCSI HBA. If it's not configured in the BIOS, the defaults pass into the configuration display in the VI Client.

iSCSI Host bus adapters for ESX 3.0�

At the time this book was written, the only supported iSCSI HBA was a specific set of QLogic cards in the 4050 series. Although a card may work, if it is not on the compatibility list, obtaining support from VMware will be challenging. As with other hardware situations in VI3, always check the VMware compatibility guide prior to purchasing or installing an iSCSI HBA.�

To modify the setting of an iSCSI HBA, perform the following steps:�

1. In the Storage Adapters node on the Configuration tab, select the appropriate iSCSI HBA (i.e., vmhba2 or vmhba3) from the list and click the Properties link.

2. Click the Configure button.

3. For a custom iSCSI qualified name, enter a new iSCSI name and iSCSI alias in the respective text boxes.

4. If desired, entire the static IP address, subnet mask, default gateway, and DNS server for the iSCSI HBA.

5. Click OK. Do not click Close.

Once you've configured the iSCSI HBA with the appropriate IP information, you must configure it to accurately find the target available via iSCSI storage devices. ESX provides for the two types of target identification:

♦ Static discovery

♦ Dynamic discovery

As the names suggest, one method involves manual configuration of target information (static) while the other involves a less cumbersome, administratively easier means of finding storage (dynamic). The dynamic discovery method is also referred to as the SendTargets method in light of the SendTarget request made by the ESX host. To dynamically discover the available storage targets, you must configure the host manually with the IP address of at least one node. Ironically, when configuring a host to perform a SendTarget request (dynamic discovery), you configure a target on the Dynamic Discovery tab of the iSCSI initiator Properties box, and all of the dynamically identified targets appear on the Static Discovery tab. You perform static assignment as well on the Dynamic Discovery tab so that dynamic targets appear on the Static Discovery tab. Figure 4.24 details the SendTargets method of iSCSI LUN discovery.

The hardware-initiated iSCSI allows for either dynamic or static discovery of targets. The iSCSI software initiator built into ESX 3.0 only allows for the SendTargets discovery method.

To configure the iSCSI HBA for target discovery using the SendTargets method, perform the following steps:

1. In the iSCSI Initiator Properties dialog box, select the Dynamic Discovery tab and click the Add button.

2. Enter the IP address of the iSCSI device and the port number (if it has been changed from the default port of 3260).

3. Click Close.�

4. Click the Rescan link.

5. Review the Static Discovery tab of the iSCSI HBA properties.

Figure 4.24 The SendTargets iSCSI LUN discovery method requires that you manually configure at least one iSCSI target to issue a SendTargets request. The iSCSI device will then return information about all the targets available.

In this section I've hinted, or, better yet, blatantly stated, that iSCSI storage networks should be isolated from the other IP networks already configured in your infrastructure. However, this is not always a possibility due to such factors as budget constraints, IP addressing challenges, host limitations, and more. If you cannot isolate the iSCSI storage network, you can configure the storage device and the ESX nodes to use the Challenge Handshake Authentication Protocol (CHAP). CHAP provides a secure means of authenticating a user account without the need for exchanging the password over publicly accessible networks.

To configure an iSCSI HBA to authenticate using CHAP, follow these steps:

1. From the Storage Adapters node on the Configuration page, select the iSCSI HBA to be configured and click the Properties link.

2. Select the CHAP Authentication tab and click Configure.

3. Insert a custom name in the CHAP Name text box or select the Use Initiator Name checkbox.

4. Type a strong and secure string in the Chap Secret text box.

5. Click OK.

6. Click Close.

Software-initiated iSCSI is a cheaper solution than the iSCSI HBA hardware initiation because it does not require any special hardware. Software-based iSCSI initiation, as the name suggests, begins in the VMkernel and utilizes a normal Ethernet adapter installed on the ESX Server host. Unlike the iSCSI HBA solution, software initiation relies on a set of drivers and a TCP/IP stack that resides in the VMkernel. In addition, the iSCSI software initiator, of which there is only one, uses the name vmhba32 as opposed to being enumerated with the rest of the HBAs within a host. Figure 4.25 outlines the architectural differences between the hardware and software initiation mechanisms on ESX Server.

Figure 4.25 ESX Server supports using an iSCSI HBA hardware-based initiation, which reduces overhead on the VMkernel. For a cheaper solution, ESX Server also supports a software-based initiation that does not require specialized hardware.

Using the iSCSI software initiation built into ESX Server provides an easy means of configuring the host to communicate with the iSCSI storage device. The iSCSI software initiator uses the SendTargets method for obtaining information about target devices. The SendTargets request requires the manual entry of one of the IP addresses on the storage device. The software initiator will then query the provided IP address in search of all additional targets.

To enable iSCSI software initiation on an ESX Server, perform the following steps:

1.�Enable the Software iSCSI client in the firewall of the ESX Server host as shown in Figure 4.26:

�♦ On the Configuration tab of the ESX host, click the Security Profile link.

�♦ Click the Properties link.

�♦ Enable the Software iSCSI Client checkbox,

or

�♦ Open an SSH session with root privileges and type the following commands:

esxcfg-firewall�-r swISCSIClient

Figure 4.26 The iSCSI software must be enabled in the Service Console firewall.�

2.�Create a virtual switch with a VMkernel port and a Service Console (vswif) port. Bind the virtual switch to a physical network adapter connected to the dedicated storage network. Figure 4.27 shows a correctly configured switch for use in connecting to an iSCSI storage device.

Creating a VMkernel Port from a Command Line�

Log on to an ESX host using an SSH session and elevate the permissions using #su�� if necessary. Follow these steps:

1. Add a new port group named Storage to the virtual switch on the dedicated storage network:

esxcfg-vswitch -A Storage vSwitch2

2. Configure the VMkernel NIC with an IP address of 172.28.0.106 and a subnet mask of 255.255.255.0:

esxcfg-vmknic -a -i�172.28.0.106 -n 255.255.255.0 Storage

3.�Set the default gateway of the VMkernel to 172.28.0.1:

esxcfg-route 172.28.0.1

Figure 4.27 The VMkernel and Service Console must be able to communicate with the iSCSI storage device.

From the Storage Adapters node on the Configuration tab, shown in Figure 4.28, enable the iSCSI initiator. Alternatively, open an SSH session with root privileges and type the following command:

esxcfg-swiscsi�-e

Figure 4.28 Enabling the iSCSI will automatically populate the iSCSI name and alias for the software initiator.

4. Select the vmhba40 option beneath the iSCSI Software Adapter and click the Properties link.

5. Select the Dynamic Discovery tab and click the Add button.

6. Enter the IP address of the iSCSI device and the port number (if it has been changed from the default port of 3260).

7. Click OK. Click Close.

8. Select the Rescan link from the Storage Adapters node on the Configuration tab.

9. Click OK to scan for both new storage devices and new VMFS volumes.�

10.�As shown in Figure 4.29, any available iSCSI LUNs will now be reflected in the Details section of the vmhba40 option.

Figure 4.29 After configuring the iSCSI software adapter with the IP address of the iSCSI storage target, a rescan will identify the LUNs on the storage device that have been made available to the ESX host.�

The vmkiscsi-tool Command�

The vmki scsi -tool [options] vmhba## command allows command-line management of the iSCSI software initiator. The options for this command-line tool include:

♦ -I is used with -l or -a to display or add the iSCSI name.

♦ -k is used with -l or -a to display or add the iSCSI alias.

♦ -D is used with -a to perform discovery of a specified target device.

♦ -T is used with -l to list found targets.

Review the following examples:�

♦ To view the iSCSI name of the software initiator:

vmkiscsi-tool�-I -l

♦ To view the iSCSI alias of the software initiator:

vmkiscsi-tool�-k -l

♦ To discover additional iSCSI targets at 172.28.0.122:

vmkisci-tool�-D -a 172.28.0.122 vmhba40

♦ To list found targets:

vmkiscsi-tool -T -l vmhba40

Although Network Attached Storage (NAS) devices do not hold up to the performance and efficiency of fibre channel and iSCSI networks, they most certainly have a place on some networks. Virtual machines stored on NAS devices are still capable of the advanced VirtualCenter features of VMotion, DRS, and HA. With a significantly lower cost and simplified implementation, NAS devices can prove valuable in providing network storage in a VI3 environment.�

Unlike the block-level transfer of data performed by fibre channel and iSCSI networks, access to a NAS device happens at the file system level. You can access a NAS device by using Network File System (NFS) or Server Message Block (SMB), also referred to as Common Internet File System (CIFS). Windows administrators will be most familiar with SMB traffic, which occurs each time a user accesses a shared resource using a universal naming convention (UNC) like \\servername\sharename. Whereas Windows uses the SMB protocol for file transfer, Linux-based systems use NFS to accomplish the same thing.

Although you can configure the Service Console with a Samba client to allow communication with Windows-based computers, the VMkernel does not support using SMB and therefore lacks the ability to retrieve files from a computer running Windows. The VMkernel only supports NFS version 3 over TCP/IP.

Like the deployment of an iSCSI storage network, a NAS/NFS deployment can benefit greatly from being located on a dedicated IP network where traffic is isolated. Figure 4.30 shows a NAS/NFS deployment on a dedicated network.

Figure 4.30�An NAS Server deployed for shared storage among ESX Server hosts should be located on a dedicated network separated from the common intranet traffic.

Without competition from other types of network traffic (e-mail, Internet, instant messaging, etc.), the transfer of virtual machine data will be much more efficient and provide better performance.�

NFS Security�

NFS is unique because it does not force the user to enter a password when connecting to the shared directory. In the case of ESX, the connection to the NFS server happens under the context of root, thus making NFS a seamless process for the connecting client. However, you might be wondering about the inherent security. Security for NFS access is maintained by limiting access to only the specified or trusted hosts. In addition, the NFS server employs standard Linux file system permissions based on�user and group IDs. The user IDs (UIDs) and group IDs (GIDs) of users on a client system are mapped from the server to the client. If a user or a client has the same UID and GID as a user on the server, they are both granted access to files in the NFS share owned by that same UID and GID. As you have seen, ESX Server accesses the NFS server under the context of the root user and therefore has all the permissions assigned to the root user on the NFS server.�

When creating an NFS share on a Linux system, you must supply three pieces of information:�

♦ The path to the share (i.e., /nfs/ISOs).

♦ The hosts that are allowed to connect to the share, which can include:�

�♦ A single host identified by name or IP address.

�♦ Network Information Service (NIS) groups.

�♦ Wildcard characters such as * and ? (i.e., *.vdc.local).

�♦ An entire IP network (i.e., 172.30.0.0/24).

♦�Options for the share configuration, which can include:

�♦ root_squash, which maps the root user to the nobody user and thus prevents root access to the NFS share.

�♦ no_root_squash, which does not map the root user to the nobody user and thus provides the root user on the client system with full root privileges on the NFS server.

�♦ all_squash, which maps all UIDs and GIDs to the nobody user for enabling a simple anonymous access NFS share.

�♦ ro, for read-only access.

�♦ rw, for read-write access.

�♦ sync, which forces all data to be written to disk before servicing another request.

The configuration of the shared directories on an NFS server is managed through the /etc/exports file on the server. The following example shows a /etc/exports file configured to allow all hosts on the 172.30.0.0/24 network access to a shared directory named NFSShare:

root:�# cat /etc/exports/mnt/NFSShare 172.30.0.0/24 (rw,no_root_squash,sync)

The next section explores the configuration requirements for connecting an ESX Server host to a shared directory on an NFS server.

Configuring ESX to Use NAS/NFS Datastores

Before an ESX Server host can be connected to an NFS share, the NFS server must be configured properly to allow the host. Creating an NFS share on a Linux system that allows an ESX Server host to connect requires that you configure the share with the following three parameters:

♦ rw (read-write)

♦ no root squash

♦ sync�

To connect an ESX Server to an NAS/NFS datastore, you must create a virtual switch with a VMkernel port that has network access to the NFS server. As mentioned in the previous section, it would be ideal for the VMkernel port to be connected to the same physical network (the same IP subnet) as the NAS device. Unlike the iSCSI configuration, creating an NFS datastore does not require that the Service Console also have access to the NFS server. Figure 4.31 details the configuration of an ESX Server host connecting to a NAS device on a dedicated storage network.

Figure 4.31 Connecting an ESX Server to a NAS device with an NFS share requires the creation and configuration of a virtual switch with a VMkernel port.

To create a VMkernel port for connecting an ESX Server to a NAS device, perform these steps:

1. Use the VI Client to connect to VirtualCenter or an ESX Server host.

2. Select the hostname in the inventory panel and then click the Configuration tab.

3. Select Networking from the Hardware menu.

4. Select the virtual switch that is bound to a network adapter that connects to a physical network with access to the NAS device. (Create a new virtual switch if necessary.)

5. Click the Properties link of the virtual switch.

6. In the vSwitch# Properties box, click the Add button.

7. Select the radio button labeled VMkernel, as shown in Figure 4.32, and then click Next.

Figure 4.32 A VMkernel port is used for performing VMotion or communicating with an iSCSI or NFS storage device.�

8.�As shown in Figure 4.33, type a name for the port in the Network Label text box. Then provide an IP address and subnet mask appropriate for the physical network the virtual switch is bound to.

Figure 4.33 Connecting an ESX Host to an NFS server requires a VMkernel port with a valid IP address and subnet mask for the network on which the virtual switch is configured to communicate.

9.�Click Next, review the configuration, and then click Finish.�

VMkernel Default Gateway�

If you are prompted to enter a default gateway, choose No if one has already been assigned to a Service Console port on the same switch or if the VMkernel port is configured with an IP address on the same subnet as the NAS device. Select the Yes option if the VMkernel port is not on the same subnet as the NAS device.�

Unlike fibre channel and iSCSI storage, an NFS datastore cannot be formatted as VMFS. For this reason it is recommended that NFS datastores not be used for the storage of virtual machines in large enterprise environments. In non-business-critical situations such as test environments and small branch offices, or for storing ISO files and templates, NFS datastores are an excellent solution.�

Once you've configured the VMkernel port, the next step is to create a new NFS datastore. To create an NFS datastore on an ESX Server host, perform the following steps:

1. Use the VI Client to connect to a VirtualCenter or an ESX Server host.

2. Select the hostname in the inventory panel and then select the Configuration tab.

3. Select Storage (SCSI, SAN, and NFS) from the Hardware menu.

4. Click the Add Storage link.

5. Select the radio button labeled Network File System, as shown in Figure 4.34.

Figure 4.34 The option to create an NFS datastore is separated from the disk or LUN option that can be formatted as VMFS.

6. Type the name or IP address of the NFS server in the Server text box.�

7. Type the name of the shared directory on the NFS server in the Folder text box. Ensure that the folder listed here matches the entry of the /etc/hosts file on the NFS server. If the folder in /etc/hosts is listed as /ISOShare, then enter /ISOShare in this text box. If the folder is listed as /mnt/NFSShare then enter /mnt/NFSShare in the Folder text box of the Add Storage wizard.

8. Type a unique datastore name in the Datastore Name text box and click Next, as shown in Figure 4.35.

Figure 4.35 To create an NFS datastore, you must enter the name or IP address of the NFS server, the name of the directory that has been shared, and a unique datastore name.�

9.�Click Finish to view the NFS datastore in the list of Storage locations for the ESX Server host.

By now, you have a good understanding of the importance of VirtualCenter in a large enterprise environment. As noted in the introduction to this chapter, all things VirtualCenter are stored in a back-end database. So the truth is that the back-end database, not so much the front-end VirtualCenter, is the key component. Without the back-end database, you will find yourself rebuilding an entire infrastructure. Figure 5.4 illustrates the components of a typical Virtual Server infrastructure.

Figure 5.4 VirtualCenter acts as a proxy for managing ESX Server 3.0 hosts. All of the data for Virtual-Center is stored in an external database.

Working with Oracle as the back-end database naturally involves more effort than using a SQL Server back-end. I stress the naturally part only because VirtualCenter is a Windows-based application and therefore seems to have a tighter integration with SQL Server that facilitates the configuration. To use Oracle 9i or 10g, you will need to install Oracle and create a database for VirtualCenter to use. If your Oracle database resides on the same computer as VirtualCenter, perform the following steps to prepare Oracle for VirtualCenter:

1. Install the Oracle database driver to the VirtualCenter server.

2. Increase the number of open available cursors for the database by adding the following line:

open_cursors - 300 to the C:\Oracle\Admin\VPX\pfile\init.ora

3.�Create a new tablespace dedicated to the VirtualCenter:

CREATE TABLESPACE vc DATAFILE�'C:\Oracle\ORADATA\VPX\vpx.dat'�SIZE 250M;

4. Create a new Oracle user account (i.e., vdcdbuser) to use in the ODBC connection string:

CREATE USER vdcdbuser IDENTIFIED BY vdcdbuser DEFAULT TABLESPACE vc;

5. Ensure the database user has been given the CONNECT and DBA privileges.

6. Finally, create an ODBC connection string in the VirtualCenter server using Administrative Tools. Give the ODBC connection a meaningful name and utilize the account created for VirtualCenter access.�

For larger enterprise networks where the Oracle 9i or 10g database server is a separate computer, you will need to perform the following tasks on the computer running VirtualCenter:�

1. If necessary, download and install the Oracle client on the VirtualCenter server.

2. Download and install the Oracle ODBC driver on the VirtualCenter database.

3. Open the tnsnames.ora file in the C:\Oracle\Oraxx\network\admin directory, where xx is the version number of your ESX server.

4. Add the following text to the tnsnames.ora file:

VPX =

(Description =

(Description

(AddressList=

Address=(Protocol=TCP)(Host=vpxd-Oracle)(Port1521)))

Host =

In light of the existing widespread deployment of Microsoft SQL Server 2000 and Microsoft SQL Server 2005, it is most common to find SQL Server as the back-end database for VirtualCenter. This is not to say that Oracle does not perform as well or that there is any downside to using Oracle. Microsoft SQL Server just happens to be implemented more commonly than Oracle and therefore is a more common back-end for VirtualCenter.�

Connecting VirtualCenter to a Microsoft SQL Server database, like the Oracle implementation, requires some specific configuration steps. The SQL Server computer must be configured in Mixed Mode authentication, as shown in Figure 5.5. This setting allows authentication to be performed by either Windows or SQL Server (see the sidebar �Windows Authentication vs. SQL Server Authentication�). Once you configure the SQL Server to allow the appropriate authentication, you must create a new database for VirtualCenter. Finally, you must create a SQL Server user account that has full access to the database you created for VirtualCenter. You can easily set the appropriate permissions for the account by making the account a member of the db_owner database role, as shown in Figure 5.6.

Figure 5.5 Using SQL Server 2000 or SQL Server 2005 requires that the database server allow Windows and SQL Server Authentication.�

Take these steps prior to creating the ODBC connection to the SQL Server database. Using SQL Server 2005 requires not only that the account have dbo (db_owner) privileges, but that the account created actually own the database. In addition, the account used by VirtualCenter to access the SQL Server 2005 database must have membership in the db_owner database role in the msdb for the duration of the installation process. Figure 5.7 shows the creation of a new SQL Server 2005 database with the default owner changed to a custom SQL Server user account.�

Figure 5.6 The user account VirtualCenter uses to access the back-end SQL 2000 Server database requires database owner privileges to build the table structure and populate the tables with data.�

Figure 5.7 SQL Server 2005 databases used by VirtualCenter must be owned by the account Virtual-Center uses to connect to the database.

�SQL Server 2005 Permissions�

Not only will most database administrators cringe at the thought of over extending privileges to a SQL Server computer, it is not good practice to do so. As a best and strong security practice, it is best to minimize the permissions of each account that access the SQL Server computer. Therefore, in the case of the VirtualCenter 2.5 installation procedure, you will need to grant a SQL Server user account the db_owner membership on the MSDB database. However, once the installation is complete this role membership can be removed, and should be removed. Normal day-to-day operation of and access to the VirtualCenter database does not require this permission. It is a temporary requirement needed for the installation of VirtualCenter 2.5.�

If you have an existing SQL Server 2005 database that needs to be used as the back-end for VirtualCenter, you can use the sp_changedbowner stored procedure command to change the database ownership accordingly. For example, EXEC sp_changedbowner @loginame='vcdbuser', @map=' true' would change the database owner to a user account named vcdbuser.

Windows Authentication vs. SQL Server Authentication�

Microsoft SQL Server, both the 2000 and 2005 versions, supports two methods of authentication: Windows and SQL Server. While a default installation of SQL Server will allow only the Windows authentication method, this setting can be changed and in some cases, as with VirtualCenter, must be changed. Figure 5.8 shows a SQL Server 2005 server configured to allow Windows and SQL Server authentication, or what is often called Mixed Mode authentication.

Windows Authentication, as the name implies, involves authentication at the operating system level. The Windows operating system checks the username and password for a user attempting to use SQL Server. The SQL Server then only looks at the user's identity, including group memberships, to determine the level of access allowed to the SQL Server.

SQL Server authentication removes the Windows aspect of the authentication leaving the SQL Server to check the requesting user's username and password. SQL Server authentication is most commonly used when connecting non-Windows clients to a SQL Server database.

SQL Server 2000 and SQL Server 2005 can be configured to allow Windows-only authentication or to allow Windows and SQL Authentication (Mixed Mode), but there is no way to enforce a SQL Server authentication-only mode.�

The configuration necessary to allow VirtualCenter to communicate with SQL Server is not an uncommon one and should result in little resistance from seasoned database administrators. In some cases, however, company policy may prevent the use of SQL Server authentication on specific SQL Servers or perhaps entirely. In this case, administrators might have to install and configure a new SQL Server computer or SQL instance to host the VirtualCenter database. If your company policy does not allow SQL Authentication to be used anywhere on the network, you will have to install SQL Server on the same computer as VirtualCenter. Figure 5.8 illustrates the scenarios in which VirtualCenter and SQL Server can communicate.

Figure 5.8 Authentication from VirtualCenter to SQL Server must use SQL Authentication for a SQL Server user account; however, if VirtualCenter and SQL Server are on the same computer, a Windows user account can be used for authentication.

VirtualCenter Authentication to SQL Server�

VirtualCenter will only support the Windows Authentication method to a SQL Server if VirtualCenter and SQL Server are installed on the same computer.�

Once your database is setup you can create the ODBC connection to be used during the VirtualCenter Server installation wizard. If using SQL Server 2000, the ODBC connection can be created with the SQL Server driver. However, using SQL Server 2005 requires use of the SQL Native Client. Figure 5.9 shows both options available in the Create New Data Source wizard.

Figure 5.9 ODBC connections to SQL Server 2005 require the SQL Native Client driver.

If you do not find the SQL Native Client option during the creation of the ODBC Connection string you can download it from Microsoft's Web site or install it off of a SQL Server 2005 installation CD-Rom.

On the server where VirtualCenter will be installed perform the following steps to create an ODBC connection to a SQL Server 2005 database:

1. Open the Data Sources (ODBC) applet from the Administrative Tools menu.

2. Select the System DSN tab.

3. Click the Add button.

4. Select the SQL Native Client from the list of available drivers and click the Finish button. If the SQL Native Client is not in the list it can be downloaded from Microsoft's Web site.

5. The Create New Data Source to SQL Server dialog box will open. In the Name text box, type the name you want to use to reference the ODBC connection. This is the name you will give to VirtualCenter to establish the database connection.

6. In the Server drop down list, select the SQL Server 2005 computer where the database has been created.

7. Click the Next button.

8. Select the With SQL Server authentication using a login ID and password entered by the user radio button.

9. Enter the username and password for the SQL Server authenticated user account that has the appropriate permissions to the VirtualCenter database and the MSDB database.�

10. Click the Next button.

11. If the default database is listed as Master select the Change the default database to: check box and then select the name of the VirtualCenter database as the default. The appropriate database might be selected if the SQL Server user account was configured with the VirtualCenter database as the default.

12. Click the Next button.

13. Click the Finish button.

14. Click the Test Data Source button to test the ODBC connection.

15. Click the OK button twice.

Older versions of VirtualCenter (pre VC 2.5) were able to use the free Microsoft SQL Server Desktop Engine (MSDE) as the back-end database for a VirtualCenter installation, however doing�so would not earn support from VMware. The use of MSDE as a VirtualCenter database should be restricted to demonstration or development implementations where loss of data or excess downtime is not a major concern.

If the threat of nonsupport isn't enough to scare you away from relying on MSDE for production environments, perhaps knowing the limitations of MSDE will. The MSDE database, though robust enough to support development, test, and training environments for previous VirtualCenter versions, is hard-coded with a database size limit of 2GB and maximum RAM use of 2GB.

If you use an MSDE database simply to get your virtual infrastructure up and running without incurring the administrative or financial overhead of a full-blown SQL Server 2005 computer, it's possible to make the transition later. Use the following procedure to move an MSDE or SQL Server 2000 database to SQL Server 2005e:

1. Stop the SQL Server service on the existing VirtualCenter that runs the MSDE database.

2. Copy the *.mdf and *.ldf for the respective VirtualCenter database to a directory on the SQL Server 2005 computer.

3. Open the SQL Server 2005 Management Studio on the SQL Server computer. Authenticate to the local SQL Server installation and navigate through the tree structure to find the imported database.

4. Right-click the databases node and select the Attach option.

5. In the Attach Databases window, select the MDF file for the database you wish to attach.

6. Modify the database owner to match the name of the user account that VirtualCenter uses in its ODBC connection, as shown in Figure 5.10.

7. Perform a simple query against the VPXHOST table, as shown in Figure 5.11.

Figure 5.10 Attaching an MSDE database requires selecting the appropriate database files and assigning a new database owner to match the name on the user account used by VirtualCenter.

Figure 5.11 A simple query against the VPXHOST table is a good way to test that your data migration process is complete.

Microsoft Access�

Microsoft Access is�no�longer supported as the back-end database�for any installation of VirtualCenter.

You can migrate physical machines into virtual machines by performing one of the following:

♦ A hot (or live) migration � the conversion of a running physical computer.

♦ A cold migration � the conversion of a physical computer that is powered off.

Both types of migrations can be performed with the VMware Converter. The VMware Converter has traditionally been a stand-alone product that can be downloaded and installed for free. However, with the release of VirtualCenter 2.5, VMware has made the VMware Converter even more convenient by adding it into VirtualCenter as a plug-in accessible directly from the VirtualCenter 2.5 interface.

Perform the following steps on each VI Client that will manage physical-to-vertical (P2V) migration to enable the VMware Converter Enterprise feature of VirtualCenter 2.5:

1. Use the VI Client to connect to the VirtualCenter 2.5 Server.

2. Click the Plugins menu and select the Manage Plugins option.

3. In the Plugin Manager dialog box, shown in Figure 7.10, click the Download and Install button under VMware Converter Enterprise.

4. Click the Next button on the VMware Converter Enterprise Client installation wizard.

5. Click the Install button to begin the installation.

6. Click OK once the installation is complete.

7. As shown in Figure 7.11, click the Installed tab and enable the VMware Converter Enterprise Client plug-in by selecting the Enabled check box. You may need to exit the VI Client and log back in before the Installed tab displays the VMware Converter Enterprise Client plug-in.

Figure 7.10 The VMware Converter Enterprise feature must be installed on each VI Client that will manage conversions.�

Figure 7.11 Once installed, the VMware Converter Enterprise Client plug-in must be enabled.

VMware Converter Stand-alone Enterprise and Starter Editions

The VMware Converter is also available in a stand-alone product that can be installed and used outside the scope of VirtualCenter 2.5. The VMware website at http://www.vmware.com offers the VMware Converter Standalone Enterprise Edition and the VMware Converter Starter Edition.

The VMware Converter Enterprise edition found in VirtualCenter is equal in functionality to the Standalone Enterprise Edition, while the Starter Edition is less functional than both. The VMware Converter Starter Edition provides single-machine conversions by requiring an installation on each virtualization candidate. The end result of all the VMware Converter products is the ability to convert physical systems into virtual machines; the Enterprise Edition just makes it easier to accomplish this by providing a centralized management and operations interface.�

The new and improved VMware Converter is a marriage of two older tools: P2V Assistant and VMware Importer. Functionality is enhanced and the interface is simplified, but the result is an easier to use and easier to manage conversion process. All of the following virtual machine types can be imported with the VMware Converter tool:�

♦ VMware Workstation 4.x, 5.x, and 6.x

♦ VMware Ace 2.x

♦ VMware Fusion 1.x

♦ VMware Player 1.x and 2.x

♦ ESX Server 3.x

♦ ESX Server 2.5 if managed by VirtualCenter 2.x

♦ GSX Server 3.x

♦ VMware Server 1.x

♦ VirtualCenter 2.x

♦ Microsoft Virtual PC 7 and later

♦ Microsoft Virtual Server

In addition to these virtual machine formats, VMware Converter can import third-party system images created with the following products:

♦ Symantec Backup Exec System Recovery 7 (formerly LiveState Recovery)

♦ Norton Ghost

♦ Acronis True Image 9

VMware Converter and the VMware Converter agent can be run on any of the following operating systems:

♦ Windows Vista 32-bit (experimental on 64-bit)

♦ Windows Server 2003 32-bit and 64-bit

♦ Windows XP Professional 32-bit and 64-bit�

♦ Windows 2000

♦ Windows NT with Service Pack 6 and Internet Explorer 5 and later�

Greater Than or Equal To�

The operating system on which VMware Converter is running must be equal to or greater than the operating system being converted to a virtual machine. For example, if the system running the VMware Converter is Windows NT with Service Pack 6, then it can only handle migrations of source computers running Windows NT with Service Pack�6. If VMware Converter is installed on Windows Server 2003 or Windows Vista, then the source computer can be any of the listed operating systems.

Windows Activation

Servers that run Windows Server 2003 that are installed using retail media and keys that require activation will require a reactivation after the physical-to-virtual migration. The Windows Activation algorithm tracks changes to hardware specifications in a system, and when the amount of hardware change is deemed significant, the Windows Activation is once again required.

The migration of a physical system to a virtual machine incurs drastic changes to the hardware specifications of the system. Everything from the CPU to network card to video adapter to disk controllers is altered as part of the migration process. It is these changes that force the Windows Activation algorithm to require a reactivation. In addition, do not be surprised if the Internet-based activation procedure is not successful. You may find that you are forced into calling the toll-free Windows Activation phone line to explain that the old system is being decommissioned in favor of the new virtual machine and that you are not in breach of your licensing agreement with Microsoft.

Hot migration, or migration of a running physical computer, is ideal for those systems that cannot afford to be taken offline and do not perform consistent changes to local data. Systems like domain controllers, web servers, and print servers are excellent candidates for hot migrations because they generally have a static set of data. Hot migration allows administrators to convert the physical system to a virtual machine and then to cut over to the virtual machine and decommission the physical machine without losing data.

Perform the following steps to conduct a hot migration of a running computer:

1. Use the VI Client to connect to a VirtualCenter 2.5 server.

2. Right-click on a cluster or ESX Server host object in the VirtualCenter inventory and select the Import Machine option.

3. Click the Next button on the Source page of the Import Wizard.

4. As shown in Figure 7.12, select the radio button labeled Physical Computer and click the Next button.

5. Enter the target computer information, including name or IP address and the appropriate user account information, as shown in Figure 7.13, then click the Next button. The user account must have administrative privileges on the target computer being converted to a virtual machine.

Figure 7.12 The VMware Converter Enterprise converts physical computers into virtual machines.�

Figure 7.13 In addition to identifying the target computer by name or IP address, you must supply administrative credentials to perform a hot migration.

6.�Select to uninstall the VMware Converter Agent automatically or manually in the Remote Installation Required warning dialog box shown in Figure 7.14, then click the Yes button to continue.

Figure 7.14 The VMware Converter Agent can be uninstalled automatically or manually once the conversion is complete.

7.�On the Source Data page, shown in Figure 7.15, select the volumes that should be converted as part of the physical-to-virtual migration process. Volumes can be ignored by deselecting the respective check box.

Figure 7.15 The VMware Converter can migrate all, some, or only one of the volumes that exist on the physical server.

As part of the migration process, volumes can be resized. Notice in Figure 7.15 that the total size of volume C: on Disk0 is 42.33GB with 29.03GB used. The default value in the New Disk Space column is equal to the existing total size. Figure 7.16 shows the New Disk Space value has been increased by 10GB to give more space to the volume.

8.�Click Next once the Source Data page is configured appropriately for the migration.

Figure 7.16 The VMware Converter allows volumes to be resized (increased or decreased) as part of the migration process.

9.�Click Next to begin configuration of the destination details of the migration process.

10.�As shown in Figure 7.17, provide a virtual machine name and the location in the inventory where the virtual machine will reside, and then click Next.

Figure 7.17 The new virtual machine can be placed anywhere throughout the Virtual Machines & Templates inventory.

11.�As shown in Figure 7.18, select a datastore with ample storage for the new virtual machine disk files, and then click Next.

Figure 7.18 The conversion process details the storage location for the new virtual machine disk files.

Clicking the Advanced button lets you place each new virtual machine disk file in different datastores, as shown in Figure 7.19.

Figure 7.19 The Advanced configuration allows virtual machine files to be spread across different datastores.

12. On the Networks page, shown in Figure 7.20, select the number of network adapters for the new virtual machine and then select the network that each network adapter should be connected to. Then click Next.

Figure 7.20 The new virtual machine can be configured with up to four network adapters on any of the available virtual machine networks.

13.�On the Customization page, shown in Figure 7.21, select the check box labeled Install VMware tools, and then click Next.

Figure 7.21 The new virtual machine can be customized by installing VMware tools or by changing the entire identity of the virtual machine.

In the option labeled Customize, the identity of the virtual machine should only be selected if the physical machine and virtual machine will both remain part of the production network. Leave the box unchecked to maintain all existing settings of the physical computer. Selecting the option to customize the wizard includes the following additional pages:

♦ Computer Info

♦ Windows License

♦ Time Zone

♦ Network Settings

♦ Workgroup/Domain

14. On the Schedule Task page, select to perform the migration immediately or schedule it to take place at a later time, and then click Next.

15. To power on the virtual machine after the migration is complete, select the Power check box on the new virtual machine after creation. If you want to wait until the physical box is powered off before using the virtual machine, then leave this check box unselected. Click the Finish button to begin the conversion.

16. The Recent Tasks pane of VirtualCenter will detail the progress of the conversion process, as shown in Figure 7.22.

Figure 7.22 VirtualCenter's Recent Tasks pane will monitor the progress of the physical-to-virtual migration.

Eliminating Excess in a P2V

Often times physical servers are purchased with more hardware than is required simply because the budget is there to support it. For example, it is not uncommon to find simple domain controllers or print servers with hundreds of gigabytes of empty space. While this is not a huge administrative concern in a physical server environment, it is a red flag in the physical-to-virtual migration process. The process of converting a physical machine to a virtual machine should involve an inspection of the resources assigned to the physical server as it is migrated. In this case, when converting a domain controller or print server with hundreds of gigabytes of free storage, it is best to adjust the disk size as part of the conversion process. This ensures that expensive SAN storage is not aimlessly consumed by virtual machines that will never utilize the resource to its full capacity.

The same can be said for memory and CPU resource allocation. As part of the migration, RAM and CPU allocations should be revisited and adjusted to better suit the demands of the virtual machine. For example, a domain controller with two dual-core processors and 4GB of RAM is severely overallocated. While the conversion process does not directly allow these two resources to be adjusted as part of the conversion wizard, they can easily be adjusted after the virtual machine has been created, prior to the first power-on.

The process referred to as a cold migration involves the conversion of a physical machine to a virtual machine while the physical machine is not in a running state. This type of conversion is best for servers that have constantly changing data sets. For example, computers that run SQL Server or Exchange Server are good candidates for cold migrations. The downside is that the server must be taken offline as part of a planned outage. Cold migrations are preferable for database and mail servers because the hot migration does not continuously update as the conversion is in progress. Therefore, if a hot migration begins at 9:00 ~PM, the virtual machine would be consistent as of that time. Any updates since the 9:00 ~PM start time would not be captured in the virtual machine.

Naturally, a computer cannot be migrated while powered off completely, so the cold migration is not about migrating a physical computer that is powered off. Rather, it is the migration of a physical computer not booted into the standard default operating system. The cold clone process begins by booting the computer from the CD/DVD-ROM drive with the VMware Converter boot CD installed. You can download the VMware Converter boot CD from VMware's website and then burn it to CD or DVD media.

The target physical computer must be configured with more than 264 MB of RAM to perform a cold migration or an error will be thrown, as shown in Figure 7.23.

Figure 7.23 Target machines to be converted to virtual machines must be configured with more than 264 MB of RAM.

Perform the following steps to cold-migrate a physical computer into a virtual machine:

1. Boot the target computer from the VMware Converter boot CD.

2. Click the I Accept the Terms in the License Agreement radio button and then click OK.

3. If the target computer has less than 364 MB of RAM, a warning message, shown in Figure 7.24, will be thrown. If the target computer has more than 364 MB of RAM, skip to step 4.

Figure 7.24 The VMware Converter will suggest using a temporary directory for target machines with less than 364 MB of RAM.

Click the Yes button to edit the network configuration, as shown in Figure 7.25. This is just a suggestion by the VMware Converter and does not have to be done.

Figure 7.25 A mapped network drive can serve as the temporary directory used by the VMware Converter.

4. After accepting the license agreement, you will be prompted to update the network parameters. Click the Yes button to adjust the network properties shown in Figure 7.26.

5. Click OK once the network properties are adjusted as needed.

6. The VMware Converter window will open. Click the Import Machine button to launch the VMware Converter Import Wizard.

7.�Click Next.

Figure 7.26 The network properties of the target system can be adjusted as part of the cold clone process.

8.�On the Source Data page of the wizard, select either the Import all disks and maintain size option or the Select volumes and resize to save or add space option, as shown in Figure 7.27. Use the latter option to increase or decrease the amount of space on the volume. Figure 7.27 shows an increase in size on volume C: from 1.99GB to 10GB.

Figure 7.27 The cold clone process allows for increasing or decreasing the space allocated to converted volumes.

9.�Once the volumes have been resized as needed, click Next to continue.

10.�On the Destination Type page, select the destination for the import. For enterprise physical-to-virtual migrations, the VMware ESX Server or VirtualCenter virtual machine option, shown in Figure 7.28, will be the most appropriate. Click Next.

Figure 7.28 The VMware Converter Import Wizard offers destinations of ESX Server, VirtualCenter, Workstation, Fusion, and Server.

11. On the Destination Login page, provide the name or IP address of the ESX Server or Virtual-Center Server and the appropriate credentials to authenticate, as shown in Figure 7.29. Click Next.

Figure 7.29 Identification and authentication of the destination server require name resolution or the use of an IP address.

12.�On the Virtual Machine Name and Folder page, type a display name for the virtual machine and select the desired location in the VirtualCenter inventory. Click Next.

13.�On the Host or Cluster page, select the host, cluster, or resource pool under which the virtual machine will run, as shown in Figure 7.30. Click Next.

Figure 7.30 Migrated virtual machines can be placed under a host, cluster, or resource pool.

If a cluster is selected from the inventory list, the next page in the wizard will require the selection of a specific host. If a host is selected, the wizard will move on to the Datastore page.

14. On the Datastore page, select the datastore where the virtual machine disk files will be stored. Use the Advanced button to supply unique datastores for the different virtual machine disk files. Click Next.

15. On the Networks page, select the number of network adapters to include in the new virtual machine, and then select the virtual machine network that each network adapter should be connected to. Click Next.

16. On the Customization page, select the check box Install VMware Tools. If you want to change the identity of the virtual machine to be different from the target system, select the check box Customize the Identity of the Virtual Machine. Selecting this option adds the following pages to the VMware Converter Import Wizard:

♦ Computer Info

♦ Windows License

♦ Time Zone

♦ Network Settings

♦ Workgroup/Domain

17.�Click Next to continue.

18.�On the Ready to Complete page, select the check box Power On the Virtual Machine after Creation and then click the Finish button. The cold clone process will be detailed in the VMware Converter utility, as shown in Figure 7.31.

Figure 7.31 The VMware Converter provides detailed information about the progress of the cold migration process.

P2V or V2V One and the Same

While all of the samples here discuss the use of the VMware Converter to perform physical-to-virtual migrations, the tool can also be used to perform virtual-to-virtual migrations.

This can be a powerful utility for solving problems regarding the over- or under-allocation of disk space for a virtual machine. By using the VMware Converter, as you have seen in this chapter, the amount of space allocated to a volume can be decreased or increased as needed. This allows administrators to use a nondestructive process to easily manage volume sizes in Windows virtual machines. Typically, doing this requires costly third-party tools and can incur damage to the system if not done properly.

One of the most significant advantages of server virtualization is the ability to allocate resources to a virtual machine based on the machine's actual performance needs. In the traditional physical server environment, a server is often provided with more resources than it really needs because it was purchased with a specific budget in mind and the server specifications were maximized for the budget provided. For example, does a DHCP server really need dual processors, 4GB of RAM, and 146GB mirrored hard drives? In most situations, the DHCP server will most certainly under-utilize those resources. In the virtual world, we can create a virtual machine better suited for the DHCP services provided by the virtual machine. For this DHCP server, then, we would assemble a virtual machine with a more suitable 1GB of RAM, access to a single CPU, and 20GB of disk space, all of which are provided by the ESX host that the virtual machine is running on. Then, we can create additional virtual machines with the resources they need to operate effectively without wasting valuable memory, CPU cycles, and disk storage. As we add more virtual machines, each machine places additional demand on the ESX Server, and the host's resources are consumed to support the virtual machines. At a certain point, either the host will run out of resources or we will need to find an alternate way to share access to a limited resource.

Let's start with how memory is allocated to a virtual machine. Then, we'll discuss the mechanisms ESX will use to arbitrate access to the memory under contention and what you as administrator can do to change how virtual machines access memory.�

When you create a new virtual machine through the VI Client, the wizard will ask you how much memory the virtual machine should have, as shown in Figure 9.1.

The amount of memory you allocate on this screen is the amount the guest operating system will see � in this example, it is 1024MB. This is the same as when you build a system and put two 512MB memory sticks into the system board. If we install Windows 2003 in this virtual machine, Windows will report 1024MB of RAM installed. Ultimately this is the amount of memory the virtual machine "thinks" that it has.

Let's assume we have an ESX Server with 4GB of physical RAM available to run virtual machines (in other words, the Service Console and VMkernel are using some RAM and there's 4GB left over for the virtual machines). In the case of our new virtual machine, it will comfortably run, leaving approximately 3GB for other virtual machines (there is some additional overhead that we will discuss later, but for now let's assume that the 3GB is available to other virtual machines).

What happens when we run three more virtual machines each configured with 1GB of RAM? Each of the additional virtual machines will request 1GB of RAM from the ESX host. At this point, four virtual machines will be accessing the physical memory.

What happens when you launch a fifth virtual machine? Will it run? The short answer is yes, but the key to understanding why this is so is the mechanism that ESX Server employs � and it�is based on some default settings in the virtual machines' configuration that administrators have control over.

Figure 9.1 Initial Memory settings for a virtual machine indicate the amount of RAM the virtual machine "thinks" that it has.

In the advanced settings for a virtual machine, as shown in Figure 9.2, we can see there is a setting for a reservation, a limit, and shares. In this discussion, we will examine the limit and reservation settings and then come back later to deal with the shares.

Figure 9.2 Each virtual machine can be configured with a shares value, a reservation, and a limit.

To edit the reservation, limit, or shares of a virtual machine:

1. Use the VI Client to connect to a VirtualCenter Server or directly to an ESX Server host.

2. Drill down through the inventory to find the virtual machine to be edited.

3. Right-click the virtual machine and select the Edit Settings option.

4. Click the Resources tab.

5. On the Resources tab, select the CPU or Memory options from the Settings list on the left.

6. Adjust the Shares, Reservation, and Limit values as desired.

The following sections will detail the ramifications of setting custom Reservation, Limit, and Shares values.

The Reservation value is an optional setting you can set for each virtual machine � but note that the default value is 0MB, as this has a potential impact on virtual machine performance. The Reservation amount specified on the Resource tab of the virtual machine settings is the amount of actual, real physical memory that the ESX Server must provide to this virtual machine for the virtual machine to power on. A virtual machine with a reservation is guaranteed the amount of RAM configured in its Reservation setting. In our previous example, the virtual machine configured with 1GB of RAM and the default reservation of 0MB means the ESX Server does not have to provide the virtual machine with any physical memory. If the ESX Server is not required to provide actual RAM to the virtual machine, then where will the virtual machine get its memory? The answer is that it provides swap, or more specifically something called VMkernel swap.

VMkernel swap is a file created when a virtual machine is powered on with a .vswp extension. The per-virtual machine swap files created by the VMkernel reside by default in the same datastore location as the virtual machine's configuration file and virtual disk files. By default, this file will be equal to the size of the RAM that you configured the virtual machine with, and you will find it in the same folder where the rest of the virtual machines files are stored, as shown in Figure 9.3.

Figure 9.3 The VMkernel creates a per-virtual machine swap file stored in the same datastore as the other virtual machine files. The swap file has a .vswp extension.

In theory, this means a virtual machine can get its memory allocation entirely from VMkernel swap � or disk � resulting in virtual machine performance degradation. If the virtual machine is configured with a reservation or a limit, the VMkernel swap file could differ.

The Speed of RAM

How slow is VMkernel swap when compared to RAM? If we make some basic assumptions regarding RAM access times and disk seek times, we can see that both appear fairly fast in terms of a human but that in relation to each other RAM is faster:

RAM access time = 10 nanoseconds (for example) Disk seek time = 8 milliseconds (for example) The difference between these is calculated as follows:

0.008 � 0.000000010 = 800,000

RAM is accessed 800,000 times faster than disk. Or to put it another way, if RAM takes 1 second to access, then disk would take 800,000 seconds to access � or nine and a quarter days � ((800,000�� 60 seconds)�� 60 minutes)�� 24 hours).

As you can see, if virtual machine performance is your goal, it is prudent to spend your money on enough RAM to support the virtual machines you plan to run. There are other factors, but this is a very significant one.�

Does this mean that a virtual machine will get all of its memory from swap when ESX Server RAM is available? No. What this means is that if an ESX host doesn't have enough RAM available to provide all of the virtual machines currently running on the host with their memory allocation, the VMkernel will page some of each virtual machine's memory out to the individual virtual machine's VMkernel swap file (VSWP), as shown in Figure 9.4.

Figure 9.4 Memory allocation for a virtual machine with 1024MB of memory configured and no reservation

How do we control how much of an individual virtual machine's memory allocation can be provided by swap and how much must be provided by real physical RAM? This is where a memory reservation comes into play.

Let's look at what happens if we decide to set a memory reservation of 512MB for this virtual machine, shown in Figure 9.5. How does this change the way this virtual machine gets memory?

Figure 9.5 A virtual machine configured with a memory reservation of 512MB

In this example, when this virtual machine is started, the host must provide at least 512MB of real RAM to support this virtual machine's memory allocation, and the host could provide the remaining 512MB of RAM from VMkernel swap. See Figure 9.6.

This ensures that a virtual machine has at least some high-speed memory available to it if the ESX host is running more virtual machines than it has actual RAM to support, but there's also a downside. If we assume that each of the virtual machines we start on this host have a 512MB reservation and we have 4GB of available RAM in the host to run virtual machines, then we will only be able to launch eight virtual machines concurrently (8�512MB=4096MB). On a more positive note, if each virtual machine is configured with an initial RAM allocation of 1024MB, then we're now running virtual machines that would need 8GB of RAM on a host with only 4GB.

Figure 9.6 Memory allocation for a virtual machine with 1024MB of memory configured and a 512MB reservation

If we look back at Figure 9.5, you will also see a setting for a memory limit. By default, all new virtual machines are created without a limit, which means that the initial RAM you assigned to it in the wizard is its effective limit. So, what exactly is the purpose of the limit setting? The limit sets the actual limit. A virtual machine cannot be allocated more physical memory than is configured in the limit setting.

Let's now change the limit on this virtual machine from the unlimited default setting to 768MB, as shown in Figure 9.7.

Figure 9.7 A virtual machine configured with 1024MB of memory, a 512MB reservation, and a 768MB limit

This means that the top 256MB of RAM will always be provided by swap, as shown in Figure 9.8.

Figure 9.8 Memory allocation for a virtual machine with 1024MB of memory configured, a 512MB reservation, and a 768MB limit

Think about the server administrator who wants a new virtual machine with 16GB of RAM. You know his application doesn't need it, but you can't talk him out of his request � and worse than that, your supervisor has decided you need to build a virtual machine that actually has 16GB of RAM. Consider creating the virtual machine with an initial allocation of 16GB, and set a reservation of 1GB and a limit of 2GB. The operating system installed in the virtual machine will report 16GB of RAM (making that person happy and keeping your supervisor happy, too). The virtual machine will always consume 1GB of host memory. If your host has available RAM, then the virtual machine might consume up to 2GB of real physical memory with the top 14GB always being provided by VMkernel swap. Of course, the virtual machine performance should not be expected to perform as if it had all 16GB of memory from physical RAM.

Working together, an initial allocation of memory, a memory reservation, and a memory limit can be powerful tools in efficiently managing the memory available on an ESX Server host.

In Figure 9.5, there was a third setting called Shares that we have not discussed. The share system in VMware is a proportional share system that provides administrators with a means of assigning resource priority to virtual machines. For example, with memory settings, shares are a way of establishing a priority setting for a virtual machine requesting memory that is above the virtual machine's reservation but below its limit. In other words, if two virtual machines want more memory than their reservation limit, and the ESX host can't satisfy both of them using RAM, we can set share values on each virtual machine so that one gets higher-priority access to the RAM in the ESX host than the other. Some would say that you should just increase the reservation for that virtual machine. While that may be a valid technique, it may limit the total number of virtual machines that a host can run, as indicated earlier in this chapter. Increasing the limit also requires a reboot of the virtual machine to become effective, but shares can be dynamically adjusted while the virtual machine remains powered on.

For the sake of this discussion, let's assume we have two virtual machines (VM1 and VM2) each with a 512MB Reservation and a 1024MB Limit, and both running on an ESX host with less than 2GB of RAM available to the virtual machines. If the two virtual machines in question have an equal number of shares (let's assume it's 1000 each), then as each virtual machine requests memory above its reservation value, each virtual machine will receive an equal quantity of RAM from the ESX host and, because the host cannot supply all of the RAM to both virtual machines, each virtual machine will swap equally to disk (VMkernel pagefile VSWP).

If we change VM1's Shares setting to 2000, then VM1 now has twice the shares VM2 has assigned to it. This also means that when VM1 and VM2 are requesting the RAM above their respective reservation values, VM1 will swap one page to VMkernel pagefile for every two pages that VM2 swaps. Stated another way, VM1 gets two RAM pages for every one RAM page that VM2 gets. If VM1 has more shares, VM1 has a higher-priority access to available memory in the host.

It gets more difficult to predict the actual memory utilization and the amount of access each virtual machine gets as more virtual machines run on the same host. Later in this chapter we will discuss more sophisticated methods of assigning memory limits, reservations, and shares to a group of virtual machines using resource pools.

When creating a new virtual machine using the VI Client, the only question you are asked related to CPU is, �Number of virtual processors?�, as shown in Figure 9.9.

Figure 9.9 When a virtual machine is created, the wizard provides the opportunity to configure the virtual machine with one, two, or four virtual CPUs.

Like the memory settings discussed, the settings Shares, Reservation, and Limit can be configured for CPU. Figure 9.10 shows the default values for CPU Resource settings.

Figure 9.10 A virtual machine's CPU can be configured with Shares, Reservation, and Limit values.

As shown in Figure 9.10, the default CPU reservation for a new virtual machine starts at 0MHz. Therefore, by default a virtual machine is not guaranteed any CPU activity by the VMkernel. This means that when the virtual machine has work to be done, it places its CPU request into the CPU queue so that the VMkernel can handle the request in sequence along with all of the other virtual machines' requests. On a lightly loaded ESX host, it's unlikely the virtual machine will wait long for CPU time; however, on a heavily loaded host, the time this virtual machine may have to wait could be significant.

If we were to set a 300MHz reservation, as shown in Figure 9.11, this would effectively make that amount of CPU available instantly to this virtual machine if there is a need for CPU cycles.

Figure 9.11 A virtual machine configured with a 300 MHz reservation for CPU activity

Every virtual machine has an option that you can set to place a limit on the amount of CPU allocated. This will effectively limit the virtual machine's ability to see a maximum number of clock cycles per second, regardless of what the host has available. Keep in mind that a single virtual CPU virtual machine hosted on a 3GHz, quad-processor ESX Server will only see a single 3GHz core as its maximum, but as administrator you could alter the limit to hide the actual maximum core speed from the virtual machine. For instance, you could set a 500 MHz limit on that DHCP server so that when it reindexes the DHCP database, it won't try to take all of the 3GHz on the processor that it can see. The CPU limit provides you with the ability to show the virtual machine less processing power than is available on a core on the physical host. Not every virtual machine needs to have access to the entire processing capability of the physical processor core.

In a manner similar to memory allocation, you can assign CPU share values to a virtual machine. The shares for CPU will determine how much CPU is provided to a virtual machine in the face of contention with other virtual machines needing CPU activity. All virtual machines, by default, start with an equal number of shares, which means that if there is competition for CPU cycles on an ESX host, each virtual machine gets serviced with equal priority. Keep in mind that this share value only affects CPU cycles that are above the reservation set for the virtual machine. In other words, the virtual machine is granted access to its reservation cycles regardless of what else is happening on the host, but if the virtual machine needs more � and there's competition � then the share values come into play.

Several conditions have to be met for shares to even be considered for allocating CPU cycles. The best way to determine this is to consider several examples. For the examples to be covered, we will assume the following details about the environment:

♦ The ESX Server host includes dual single-core, 3GHz CPUs.

♦ The ESX Server host has one or more virtual machines.

Scenario 1 The ESX host has a single virtual machine running. The shares are set at default for the running virtual machines. Will the shares value have any effect in this scenario? No � there's no competition between virtual machines for CPU time.

Scenario 2 The ESX host has two idle virtual machines running. The shares are set at default for the running virtual machines. Will the shares values have any effect in this scenario? No � there's no competition between virtual machines for CPU time as both are idle.

Scenario 3�The ESX host has two equally busy virtual machines running (both requesting maximum CPU capabilities). The shares are set at default for the running virtual machines. Will the shares values have any effect in this scenario? No. Again, there's no competition between virtual machines for CPU time, this time because each virtual machine is serviced by a different core in the host.

Scenario 4�To force contention, both virtual machines are configured to use the same CPU by setting the CPU affinity, shown in Figure 9.12. The ESX host has two equally busy virtual machines running (both requesting maximum CPU capabilities). This ensures contention between the virtual machines.

Figure 9.12 CPU affinity can tie a virtual machine to physical CPU at the expense of eliminating VMotion capability.

The previously discussed settings for virtual machine resource allocation (memory and CPU reservations, limits, and shares) are methods used to designate the priority of an individual virtual machine compared to other virtual machines also seeking access to resources. In much the same way as we assign users to groups and then assign permissions to the groups, we can leverage resource pools to make the allocation of resources to collections of virtual machines a less tedious and more effective process.

A resource pool is a special type of container object, much like a folder, in the Hosts & Clusters view of inventory. We can create a resource pool on a stand-alone host or as a management object in a DRS cluster (discussed later in this chapter). Figure 9.13 shows the creation of a resource pool.

If you examine the properties of the resource pool, as shown in Figure 9.14, you'll see there are two sections: one for CPU settings (reservation, limit, and shares) and another section with similar settings for memory.

Figure 9.13 Resource pools can be created on individual hosts and within clusters. A resource pool provides a management and performance configuration layer in VirtualCenter inventory.

Figure 9.14 A resource pool is used for managing CPU and memory resources for multiple virtual machines contained within the resource pool.

To describe the function of resource pools, consider the following example. A company has two main classifications of servers: production and development. The goal of resource allocation in this scenario is to ensure that if there's competition for a particular resource, the virtual machines in production should be assigned higher-priority access to that resource. In addition to that goal, we need to ensure that the virtual machines in development cannot consume more than 4GB of physical memory with their running virtual machines. We don't care how many virtual machines run concurrently as part of the development group as long as they don't collectively consume more than 4GB of RAM.

The first step in creating the infrastructure to support the outlined goal is to create two resource pools: one called ProductionVMs and one called DevelopmentVMs, shown in Figure 9.15.

Figure 9.15 Two resource pools created on an ESX Server host

We will then modify the resource pool settings for each resource pool to reflect the goals of the organization, as shown in Figures 9.16 and 9.17.

Figure 9.16 The ProductionVMs resource pool is configured to be able to consume more resources in the face of contention.

As a final step in configuring the environment, the virtual machines must be moved into the appropriate resource pool by clicking on the virtual machine in the inventory panel and dragging it onto the appropriate resource pool. The result will be similar to that shown in Figure 9.18.

Now that we've got an example to work with, let's examine what the settings on each of these resource pools will do for the virtual machines contained in each of the resource pools.

In Figure 9.16, we set the ProductionVMs CPU shares to High (8000). In Figure 9.17, we set the DevelopmentVMs CPU shares to Low (2000). The effect of these two settings is similar to that of comparing two virtual machines' CPU share values � except in this case, if there is any competition for CPU resources between virtual machines in the ProductionVMs and the DevelopmentVMs resource pool, the ProductionVMs would have higher priority. To make this clearer, let's examine Figure 9.19 with the assumption that all the available virtual machines are competing for CPU cycles on the same physical CPU. Remember that share allocations only come into play when virtual machines are fighting one another for a resource. If an ESX Server host is only running four virtual machines on top of two dual-core processors, there won't be much contention to manage.

If there are two or more virtual machines in a resource pool that require different priority access to resources, we can still set individual Shares values on the virtual machines, or if we have groupings of virtual machines within a resource pool, we can also build a resource pool in a resource pool. A resource pool within a resource pool is called a child resource pool, and it contains its own shares, limits, and reservations separate from the parent resource pool.

Figure 9.17 The DevelopmentVMs resource pool is configured not to be able to consume more resources in the face of contention.

Figure 9.18 Virtual machines assigned to a resource pool consume resources allocated to the resource pool.

The next setting in the Resource Pool properties to evaluate is CPU Reservation (Figure 9.14). In the example, a CPU Reservation value of 1000 MHz has been set on the ProductionVMs resource pool. This will ensure that at least 1000 MHz of CPU time is available for all of the virtual machines located in that resource pool. This setting will have an additional effect: assuming that the ESX Server host has a total of 6000 MHz of total CPU, this means 5000 MHz of CPU time is available to other resource pools. If two more resource pools are created with a reservation value of 2500 MHz each, then the cumulative reservations on the system have reserved all available host CPU (1000+2500+2500). This essentially means the administrator will not be able to create additional resource pools with reservation values set.

The third setting on the Resource Pool is the CPU Limit field. This is similar to the individual virtual machines CPU Limit field, except in this case all virtual machines in the resource pool combined can consume up to this value. The limit applies to the collective sum of the virtual machines within the resource pool. In the example, the ProductionVMs resource pool has been configured with a CPU limit of 3000 MHz. In this case, no matter how many virtual machines are running in this resource pool, they can only consume a maximum of 3000 MHz of host CPU cycles.

Figure 9.19 Two resource pools with different Shares values will be allocated resources proportional to their percentage of share ownership.

Each resource pool also includes a setting to determine if the pool has an Expandable Reservation. The Expandable Reservation dictates whether a resource pool is allowed to ask the parent pool for more resources once it has consumed all its allocated resources. Consider a child who is given a weekly allowance of $10 by their parent. Suppose the child wants to make a purchase that costs $20. If the child's allowance is set to allow Expandable Reservations, then the child could ask the parent for the additional resource, and if the parent has the resource, it will be given. If the child's allowance is not set to allow Expandable Reservations, then they cannot ask the parent for additional resources.

Therefore, if the intent is to limit the total amount of CPU available to virtual machines in a resource pool, then the Expandable Reservation check box should be left empty. If left selected, a new virtual machine with a reservation configured that exceeds the capacity of the resource pool will be powered on if the parent is able to provide the necessary resource. In this case, the pool has not provided a hard cap on the amount of reserved resources.

Moving on to the Memory portion of the Resource Pool settings, the first setting is the Shares value. This setting works in much the same way as Memory Shares worked on individual virtual machines. It determines which pool of virtual machines will be the first to page in the face of contention. However, this setting is used to set a priority value for any virtual machine in the resource pool when competing for resources with virtual machines in other resource pools. Looking at the Memory Share settings in our example (ProductionVMs=Normal and DevelopmentVMs=Low), this would generally mean that if host memory was limited, virtual machines in the DevelopmentVMs area that need more memory than their reservation would use more pages in VMkernel swap than an equivalent virtual machine in the ProductionVMs resource pool.

The Memory Reservation value will reserve this amount of host RAM for virtual machines in this resource pool to run, which effectively ensures that there is some actual RAM that the virtual machines in this resource pool are guaranteed.

The Memory Limit value is an excellent way of setting a limit on how much host RAM a particular set of virtual machines can consume. If administrators have been given the �Create Virtual machines� permission in the DevelopmentVMs resource pool, then the Memory Limit value would prevent those administrators from running virtual machines that will consume more than that amount of actual host RAM. In our example, the Memory Limit value on the DevelopmentVMs resource pool is set to be 1024MB. How many virtual machines can administrators in Development create? They can create as many as they wish. But the number of virtual machines they will be able to run will be less. Unfortunately, this setting has nothing to do with the creation of virtual machines, but it will prevent administrators from running too many virtual machines at once. So how many can they run? The cap placed on memory use is not a per virtual machine setting but a cumulative setting. They might be able to run only one virtual machine with all the memory, or multiple virtual machines with lower memory configurations. Assuming that each virtual machine is created without an individual Memory Reservation value, the administrator can run as many virtual machines concurrently as she wants! The problem will be that once the virtual machines consume 1024MB of host RAM, anything above that amount will need to be provided by VMkernel swap. If she builds four virtual machines with 256MB as the initial memory amount, then all four virtual machines will consume 1024MB (assuming no overhead) and will run in real RAM. If she tries to run 20 of the same type of virtual machine, then all 20 virtual machines will share the 1024MB of RAM, even though their requirement is for 5120MB (20�256MB) � the remaining 4096MB would be provided by VMkernel swap. At this point performance might be noticeably slow.

The Unlimited check box should be cleared to set a limit value. The Expandable Reservation check box functions in the same way as the equivalent CPU setting. If you truly want to limit the resource pool's memory, then clear this check box.

Go Big if Just for a moment

An Expandable Reservation may not seem that useful given the comments in the text. However, think about temporarily allowing a resource pool to exceed its limit by using an expandable reservation. Consider a scenario where the Infrastructure resource pool needs more memory to deploy a new Windows 2003 virtual machine. They will use this new virtual machine to retire a Windows 2000 virtual machine and they will be doing this over the weekend. Simply select the Expandable Reservation option on Friday to effectively give them room to run both virtual machines at the same time to allow the data migration from the old virtual machine to the new one. After the weekend, verify that they have shut off the old virtual machine, clear the Expandable Reservation checkbox, and then everything is back to normal.

As they say, nothing in this world is free. There are several basic processes on an ESX host that will consume host memory. The VMkernel itself, the Service Console (272MB by default, 800MB maximum), and each virtual machine that is running will cause the VMkernel to allocate some memory to host the virtual machine above the initial amount that we assigned to it. The amount of RAM allocated to host each virtual machine depends on the configuration of each virtual machine, as shown in Table 9.1.

We've defined the VMware VMotion feature as the ability to perform a hot migration of a virtual machine from one ESX Server host to another without service interruption. This is an extremely effective tool for load-balancing virtual machines across ESX Server hosts. Additionally, if an ESX Server host needs to be powered off for hardware maintenance or some other function that would take it out of production, VMotion can be used to migrate all active virtual machines from the�host going offline to another host without waiting for a hardware maintenance window since the virtual machines will remain available to the users that need them.�

Table 9.1:�Virtual Machine Memory Overhead�

Virtual CPUs	Memory Assigned (MB)	Overhead for 32-bit	Overhead for 64-bit
		Virtual machine (MB)	Virtual machine (MB)
1	256	79	174
1	512	79	176
1	1024	84	180
1	2048	91	188
1	4096	107	204
1	8192	139	236
1	16384	203	300
2	256	97	288
2	512	101	292
2	1024	101	300
2	2048	125	316
2	4096	157	349
2	8192	221	413
2	16384	349	541
4	256	129	511
4	512	133	515
4	1024	141	523
4	2048	157	540
4	4096	189	572
4	8192	222	605
4	16384	350	734

VMotion works by copying the contents of virtual machine memory from one ESX host to another and then transferring control of the virtual machines' disk files to the target host.

VMotion operates in the following sequence of steps:

1.�An administrator initiates a migration of a running virtual machine (VM1) from one ESX Server (Silo104) to another (Silo105), shown in Figure 9.20.

Figure 9.20 Step 1 in a VMotion migration: invoking a migration while the virtual machine is powered on.

2.�The source hosts (Silo104) begins copying the active memory pages VM1 has in host memory to the destination host (Silo105). During this time, the virtual machine still services clients on the source (Silo104). As the memory is copied from the source host to the target, pages in memory could be changed. ESX Server handles this by keeping a log of changes that occur in the memory of the virtual machine on the source host after that memory address has been copied to the target host. This log is called a memory bitmap as shown in Figure 9.21.

The Memory Bitmap

The memory bitmap does not include the contents of the memory address that has changed; it simply includes the addresses of the memory that has changed � often referred to the �dirty memory.�

3. Once the entire contents of RAM for the virtual machine being migrated have been transferred to the target host (SILO105), then VM1 on the source ESX Server (SILO104) is quiesced. This means that it is still in memory but is no longer servicing client requests for� data. The memory bitmap file is then transferred to the target (Silo105). See Figure 9.22.

4. The target host (SILO105) reads the addresses in the memory bitmap file and requests the contents of those addresses from the source (SILO104). See Figure 9.23.

Figure 9.21 Step 2 in a VMotion migration: starting the memory copy and adding a memory bitmap

Figure 9.22 Step 3 in a VMotion migration: quiescing VM1 and transferring the memory bitmap file from the source ESX host to the destination ESX host

Figure 9.23 Step 4 in a VMotion migration: fetching the actual memory listed in the bitmap file from the source to the destination (dirty memory)

5.�Once the contents of the memory referred to in the memory bitmap file have been transferred to the target host, the virtual machine starts on that host. Note that this is not a reboot � the virtual machine's state is in RAM, so the host simply enables it. This will cause a Reverse Address Resolution Protocol (RARP) from the host to register its MAC address against the physical switch port the target ESX server is plugged into. This process enables the switch infrastructure to send network packets to the appropriate ESX host from the clients who are attached to the virtual machine that just moved. See Figure 9.24.

6.�Once the virtual machine is successfully operating on the target host, the memory the virtual machine was using on the source host is deleted. This memory becomes available to the VMkernel to use as appropriate. See Figure 9.25.

Try It with PING -t

Following the previous procedure carefully, you'll note there will be a time when the virtual machine being moved is not running on either the source host or the target host. This is typically a very short period of time. Testing has shown a continuous ping (ping -t) of the virtual machine being moved might, on a bad day, result in the loss of one ping packet. Most client-server applications are built to withstand the loss of more than a packet or two before the client is notified of a problem.

Figure 9.24 Step 5 in a VMotion migration: enabling the virtual machine on the target host and registering with the network infrastructure

Figure 9.25 Step 6 in a VMotion migration: deleting the virtual machine from the source ESX host

The VMotion migration is pretty amazing, and when they see it work for the first time in a live environment, most people are extremely impressed. However, detailed planning is necessary for this procedure to function properly. The hosts involved in the VMotion process have to meet certain requirements, along with the virtual machines being migrated.

Each of the ESX Server hosts that are involved in VMotion must meet the following requirements:

♦ Shared VMFS storage for the virtual machines files

♦ A gigabit network card with a VMkernel port defined and enabled for VMotion on each host (see Figure 9.26 through Figure 9.32)

Perform these steps to create a virtual switch with a VMotion-capable VMkernel port: 1.�Add a new switch that includes a VMkernel port, as shown in Figure 9.26.

Figure 9.26 A VMotion enabled VMkernel port is required to perform a hot migration of a virtual machine.

2.�Choose the network adapter that is connected to the VMotion network, as shown in Figure 9.27.

Figure 9.27 VMotion requires a virtual switch associated to a physical network adapter, preferably on a dedicated physical network.

3.�Enable the Use This Port Group for VMotion option and assign an IP address appropriate for the VMotion network, as shown in Figure 9.28.

Figure 9.28 The vSwitch with the VMkernel port group for VMotion must be VMotion capable.

4.�Click Finish. Add a default gateway if needed, as shown in Figures 9.29 and 9.30.

Figure 9.29 Finish the VMkernel port configuration to allow VMotion migrations.

Figure 9.30 A default gateway can be created for the VMkernel port if one is required. VMkernel ports with VMotion enabled should not require a default getaway.

If the VMotion network is nonroutable, leave the default gateway blank or simply use the default gateway assigned for the Service Console, as shown in Figure 9.31.

Figure 9.31 VMotion networks without a router do not need a default or the same default gateway as the Service Console can be entered.

A successful VMotion migration between two hosts relies on all of the following conditions being met:

♦�Both the source and destination hosts must be configured with identical virtual switches that have VMkernel port groups. The names of the switches must be the same as shown in Figure�9.32.

Figure 9.32 The two hosts involved in the VMotion migration must have similarly configured VMotion enabled virtual switches.

♦ All port groups to which the virtual machine being migrated is attached must exist on both ESX hosts. Port group naming is case sensitive, so create identical port groups on each host and make sure they plug into the same physical subnets or VLANs. A virtual switch named Production is not the same as a virtual switch named PRODUCTION. Remember that to prevent downtime the virtual machine is not going to change its network address as it is moved. The virtual machine will retain its MAC address and IP address so clients connected to it don't have to resolve any new information to reconnect.

♦ Processors in both hosts must be compatible. When a virtual machine is transferred between hosts, the virtual machine has already detected the type of processor it is running on when it booted. Since the virtual machine is not rebooted during a VMotion, the guest assumes the CPU instruction set on the target host is the same as on the source host. We can get away with slightly dissimilar processors, but in general the processors in two hosts that perform VMotion must meet the following requirements:

�♦ CPUs must be from the same vendor (Intel or AMD).

�♦ CPUs must be from the same CPU family (PIII, P4, Opteron).

�♦ CPUs must support the same features, such as the presence of SSE2, SSE3, and SSE4, and NX or XD (see the sidebar, �Processor Instruction�).

�♦ For 64-bit virtual machines, CPUs must have virtualization technology enabled (Intel VT or AMD-v).

Processor instruction

SSE2 (Streaming SIMD Extensions 2) was an enhancement to the original MMX instruction set found in the PIII processor. The enhancement was targeted at the floating-point calculation capabilities of the processor by providing 144 new instructions. SSE3 instruction sets are an enhancement to the SSE2 standard targeted at multimedia and graphics applications. The new SSE4 extensions target both the graphics and application server.

AMD's XD (eXecute Disable) and Intel's NX (NoExecute) are features of processors that mark memory pages as data only, which prevents a virus from running executable code at that address. The operating system needs to be written to take advantage of this feature, and in general, versions of Windows starting with Windows 2003 SP1 and Windows XP SP2 support this CPU feature.

The latest processors from Intel and AMD have specialized support for virtualization. The AMD-V and Intel Virtualization Technology (VT) must be enabled in the BIOS in order to create 64-bit virtual machines.

VMware includes a utility named cupid.iso.gz in the images subdirectory of the ESX Server installation CD. This tool can test a server to see what CPU features the host processors have. To perform the test, unzip it and make a bootable CD, then boot each ESX host from the cupid CD and compare the output. If they match, then VMotion is CPU compatible. If they don't match, then you need to determine what doesn't match.�

On a per-virtual machine basis, you'll find a setting that tells the virtual machine to show or mask the NX/XD bit in the host CPU. Masking the NX/XD bit from the virtual machine tells the virtual machine that there's no NX/XD bit present. This is useful if you have two otherwise compatible hosts with an NX/XD bit mismatch. If the virtual machine doesn't know there's an NX or XD bit on one of the hosts, it won't care if the target host has or doesn't have that bit if you migrate that virtual machine using VMotion. The greatest VMotion compatibility is achieved by masking the NX/XD bit. If the NX/XD bit is exposed to the virtual machine, as shown in Figure 9.33, the BIOS setting for NX/XD must match on both the source and destination ESX Server host.

What happens if you have SSE3 features on one host and not on the other? For mismatched SSE3 and SSE4 processors, you can change the masking by clicking the Advanced button, shown in Figure 9.33, and entering the CPU parameters you wish to mask, as shown in Figure 9.34.

Some administrators might recognize that this is a tedious task if you already have dozens of virtual machines built. The setting is changed on each virtual machine. However, if you know you have mismatched NX/XD bits or SSE3/SSE4 masks, you can change your template virtual machine and any virtual machine deployed from that template will also have the same setting.

Figure 9.33 Masking the NX/XD bit on a virtual machine

Figure 9.34 Masking SSE3 extensions for an Intel CPU

In addition to the VMotion requirements for the hosts involved, there are requirements that must be met by the virtual machine to be migrated:

♦�The virtual machine must not be connected to any device physically available to only one ESX host. This includes disk storage, CD-ROMs, floppy drives, serial ports, or parallel ports. If the virtual machine to be migrated has one of these mappings, simply clear the Connected check box beside the offending device, as shown in Figure 9.35.

Figure 9.35 Clear the Connected box for any locally mapped device prior to migrating with VMotion.

♦ The virtual machine must not be connected to an internal-only switch.

♦ The virtual machine must not have its CPU affinity set to a specific CPU.

♦ The virtual machine must not have a RAW disk mapping as part of a Microsoft Cluster Services (MSCS) configuration.

♦ The virtual machine must have all disk, configuration, log, and nonvolatile random access memory (NVRAM) files stored on a volume visible to both the source and the destination ESX Server host.

If you start a VMotion migration and VirtualCenter finds an issue that is considered a violation of the VMotion compatibility rules, you will see an error message. In some cases, a warning, not an error, will be issued. In the case of a warning, the VMotion migration will still succeed. For instance, if you have cleared the check box on the host-attached floppy drive, VirtualCenter will tell you there is a mapping to a host-only device that is not active. You'll see a prompt asking if the migration should take place anyway.

VMware states that you need a gigabit network card for VMotion; however, it does not have to be dedicated to VMotion. When you're designing the ESX Server host, dedicate a network adapter to VMotion if possible. You thus reduce the contention on the VMotion network, and the VMotion process can happen in a fast and efficient manner.

To perform a VMotion migration of a virtual machine:

1.�Select a powered-on virtual machine in your inventory, right-click the virtual machine, and select Migrate, as shown in Figure 9.36.

Figure 9.36 Starting the VMotion process

2.�Choose the target host, as shown in Figure 9.37.

Figure 9.37 Choose the target host.

3.�Choose the target resource pool (or cluster). Most of the time the same resource pool (or cluster) that the virtual machine currently resides in will suffice. Choosing a different resource pool might change that virtual machine's priority access to resources, as shown in Figure 9.38.

Figure 9.38 Choose a target resource pool for the virtual machine being migrated.

4. Select the priority that the VMotion migration needs to proceed with. Be aware that choosing high priority will cause CPU stress on the hosts involved in the migration, as described in the option shown in Figure 9.39.

Figure 9.39 Choosing a priority level for the migration

5. Click Finish once the validation has concluded and you have reviewed the information on the summary screen, as shown in Figure 9.40.

6.�The virtual machine should start to migrate. Often, the process will pause at about 10% in the progress dialog box, and then again at 90%. The 10% pause occurs while the hosts in question establish communications and gather the information for the pages in memory to be migrated; the 90% pause occurs when the source virtual machine is quiesced and the dirty memory pages are fetched from the source host, as shown in Figure 9.41.

Figure 9.40 Click Finish to start the actual migration.

VMotion is an invaluable tool for virtual administrators, and certainly the feature that put ESX Server on the map. But VMotion, in this latest version of ESX Server, has evolved into more than just a simple tool for moving virtual machines. VMotion is the backbone for the new Distributed Resource Scheduler (DRS) feature that can be enabled on ESX Server clusters. Before we get to the details of DRS, you need to understand clusters.

Figure 9.41 Progress of a VMotion migration

DRS is a feature of VirtualCenter on the properties of a cluster that balances load across multiple ESX hosts. It has two main functions: the first is to decide which node of a cluster should run a virtual machine when it's powered on, and the second is to evaluate the load on the cluster over time and either make recommendations for migrations or automatically move virtual machines to create a more balanced cluster workload using VMotion. Fortunately for those of us who like to retain control, there are parameters that set how aggressively DRS will automatically move virtual machines around the cluster.

If we start by looking at the DRS properties, as shown in Figure 9.48, there are three selections regarding the automation level of the DRS cluster: Manual, Partially Automated, and Fully Automated. The slider bar only affects the actions of the fully automated setting on the cluster. These settings control the initial placement of a virtual machine and the automatic movement of virtual machines between hosts.

Figure 9.48 A DRS cluster can be set to automate as much or as little as desired.

The Manual setting of the DRS cluster will prompt you every time you power on a virtual machine for the node that you want to launch that virtual machine on. The dialog box rates the available hosts according to suitability at that moment in time: the more stars, the better the choice, as shown in Figure 9.49.

The Manual setting will also suggest Migrations when DRS detects an imbalance between ESX hosts in the cluster. This is an averaging process that works over longer periods of time than many of us are used to in the information technology field. It is unusual to see DRS make any recommendations unless an imbalance has existed for longer than 5 minutes. The recommended list of migrations is available by selecting the cluster in Inventory and then selecting the Migrations tab, as shown in Figure 9.50.

Figure 9.50 shows that DRS rates both the SERVER1 virtual machine and the Win2008-02 virtual machine as very strong candidates to move from host Silo106 to Silo105 and Silo104, respectively. The number of stars a migration can have ranges from one to five: a one-star migration suggestion is a gentle recommendation and a four-star migration is a much stronger suggestion.

Figure 9.49 A DRS cluster set to Manual will let you specify where the virtual machine should be powered on.

Figure 9.50 Recommended migrations for a DRS cluster

To agree with DRS and start the migration, select the virtual machine you want to migrate on the Migrations tab and click the Apply Migration Recommendation button. VMotion will handle the migration automatically.

If you select the Partially Automated setting on the DRS properties, as shown in Figure 9.48, DRS will make an automatic decision about which host a virtual machine should run on when it is initially powered on (without prompting the admin who is performing the power on task), but it will still prompt for all migrations on the Migration tab.

The third setting for DRS is Fully Automated. This setting will make decisions for initial placement without prompting and will also make automatic VMotion decisions based on the selected automation level (the slider bar).

There are five positions for the slider bar on the Fully Automated setting of the DRS cluster. The values of the slider bar range from Conservative to Aggressive, as shown in Figure 9.48. Conservative automatically moves migrations evaluated with five stars. Any other migrations are listed on the Migrations tab and require administrator approval. If you move the slider bar from the most conservative setting one stop to the right, then all four- and five-star migrations will automatically be approved; three stars and less will wait for administrator approval. With the slider all the way over to the aggressive setting, any imbalance in the cluster that causes any recommendation will be automatically approved. Be aware that this can cause additional stress in your ESX host environment, as even a slight imbalance will trigger a migration.

DRS runs on an intelligent algorithm that is constantly being checked. Calculations for migrations can change regularly. Assume that during a period of high activity DRS makes a recommendation with three stars, and the automation level is set so three-star migrations need manual approval but the recommendation is not noticed (or an administrator is not even in the office).

An hour later, the virtual machines that caused the three-star migration in the first place have settled down and are now operating normally. At this point the Migrations tab no longer reflects the migration recommendation. The recommendation has since been withdrawn. This behavior occurs because if the migration was still listed, an administrator might approve it and cause an imbalance where one did not exist.

Earlier we mentioned that five-star migrations had little to do with load on the cluster. The first function that causes a five-star migration recommendation is when you put a host into Maintenance Mode, as shown in Figure 9.51.

Figure 9.51 An ESX Server host put into Maintenance Mode cannot power on new virtual machines or be a target for VMotion.

An administrator creates DRS rules to control how DRS decides which virtual machines should be combined with which other virtual machines, or which virtual machines should be kept separate by the DRS process. Consider the rules page shown in Figure 9.52.

Consider an environment with two mail server virtual machines. In all likelihood, administrators would not want both mail servers to reside on the same ESX Server host. At the same time, administrators would want a web application server and the back-end database server to reside on the same hosts. That might be a combination that should always be together to ensure rapid response times between them. These two scenarios could be serviced very well with DRS rules.

Figure 9.52 DRS affinity (Keep Virtual Machines Together) and anti-affinity rules (Separate Virtual Machines)

Perform the following steps to create a DRS anti-affinity rule:

1. Right-click the DRS cluster where the rules need to exist and select the Edit Settings option.

2. Click the Rules option.

3. Type a name for the rule and select the type of rule to create:

�♦ For anti-affinity rules, select the Separate Virtual Machines option.

�♦ For affinity rules, select the Keep Virtual Machines Together option.

4. Click the Add button to include the necessary virtual machines to the rule, as shown in Figure 9.53.

5. Click OK.

6. Review the new rule configuration, as shown in Figure 9.54.

7. Click OK.

With DRS rules, it is possible to create fallible rules, such as building a �Separate virtual machines� rule that has three virtual machines in it on a DRS cluster that only has two hosts. In this situation, VirtualCenter will generate report warnings because DRS cannot satisfy the requirements of the rule.

Rules can be temporarily disabled by clearing the check box next to the rule, as shown in Figure 9.55.

Although most virtual machines should be allowed to take advantage of the DRS balancing act, there will likely be enterprise-critical virtual machines that administrators are adamant about not being VMotion candidates. However, the virtual machines should remain in the cluster to take advantage of the High Availability (HA) feature. In other words, virtual machines will take part in HA but not DRS despite the fact that both features are enabled on the cluster. As shown in Figure 9.56, virtual machines in a cluster can be configured with individual DRS compatibility levels.

Figure 9.53 Adding virtual machines to a DRS rule

Figure 9.54 A completed DRS rule

Figure 9.55 Temporarily disabling a DRS rule

Figure 9.56 Virtual machine options for a DRS cluster

This dialog box lists the virtual machines that are part of the cluster and their default automation level. In this case, all virtual machines are set at Fully Automated because that's how the automation level of the cluster was set. The administrator can then selectively choose virtual machines that are not going to be acted upon by DRS in the same way as the rest in the cluster. The automation levels available include:

♦ Fully Automated

♦ Manual

♦ Partially Automated

♦ Disabled

♦ Default (inherited from the cluster setting)

The first three options work as discussed in this chapter. The Disabled option turns off DRS, including the automatic host selection at startup and the migration recommendations. The default option configures the virtual machine to accept the automation level set at the cluster.

At Least Be Open to Change

Even if a virtual machine or several virtual machines have been chosen not to participate in the automation of DRS, it is best not to set virtual machines to the Disabled option since recommendations will not be provided. It is possible that a four- or five-star recommendation could be provided that suggests moving a virtual machine an administrator thought was best on specific host. Yet the migration might suggest a different host. For this reason, the Manual option is better. At least be open to the possibility that a virtual machine might perform better on a different host.

VI3 provides a number of tools for administrators to make their lives easier as long as the tools are understood and set up properly. It might also be prudent to monitor the activities of these tools to see if a change to the configuration might be warranted over time as your environment grows.

When critical services and applications call for the highest levels of availability, many network administrators turn to Microsoft Cluster Services (MSCS). Microsoft Windows Server 2003 supports network load-balancing clusters and server clusters depending on the version of Windows Server 2003 installed on the server.

The network load-balancing (NLB) configuration involves an aggregation of servers that balances the requests for applications or services. In a typical NLB cluster, all nodes are active participants in the cluster and are consistently responding to requests for services. NLB clusters are most commonly deployed as a means of providing enhanced performance and availability. NLB clusters are best suited for scenarios involving Internet Information Services (IIS), virtual private networking, and Internet Security and Acceleration (ISA) Server, to name a few. Figure 10.1 details the architecture of an NLB cluster.

Figure 10.1 An NLB cluster can contain up to 32 active nodes that distribute traffic equally across each node. The NLB software allows the nodes to share a common name and IP address that is referenced by clients.

NLB Support from VMware

As of this writing, VMware does not support the use of NLB clusters across virtual machines. This is not to say it cannot be configured, or that it will not work; however, it is not a VMware-supported configuration.

Unlike NLB clusters, server clusters are used solely for the sake of availability. Server clusters do not provide performance enhancements outside of high availability. In a typical server cluster, multiple nodes are configured to be able to own a service or application resource, but only one node owns the resource at a given time. Server clusters are most often used for applications like Microsoft Exchange, Microsoft SQL Server, and DHCP services, which each share a need for a common datastore. The common datastore houses the information accessible by the node that is online and currently owns the resource, as well as the other possible owners that could assume ownership in the event of failure. Each node requires at least two network connections: one for the production network and one for the cluster service heartbeat between nodes. Figure 10.2 details the structure of a server cluster.�

The different versions of Windows Server 2003 offer various levels of support for NLB and server clusters. Table 10.1 outlines the cluster support available in each version of Windows Server 2003.�

Windows Clustering Storage Architectures�

Server clusters built on Windows Server 2003 can only support up to eight nodes when using a fibre channel-switched fabric. Storage architectures that use SCSI disks as direct attached storage or that use a fibre channel-arbitrated loop result in a maximum of only two nodes in a server cluster. Clustering virtual machines in ESX Server utilizes a simulated SCSI shared storage connection and is therefore limited to only two-node clustering. In addition, the clustered virtual machine solution uses only SCSI-2 reservations, not SCSI-3 reservations, and supports only the SCSI miniport drivers, not the STORPort drivers.�

Figure 10.2 Server clusters are best suited for applications and services like SQL Server, Exchange Server, DHCP, etc., that use a common data set.

Table 10.1:�Windows Server 2003 Clustering Support

Operating System	Network Load Balancing (NLB)	Server Cluster
Windows Server 2003 Web Edition	Yes (up to 32 nodes)	No
Windows Server 2003 Standard Edition	Yes (up to 32 nodes)	No
Windows Server 2003 Enterprise Edition	Yes (up to 32 nodes)	Yes (up to 8 nodes in fibre channel switch fabric)
Windows Server 2003 Datacenter Edition	Yes (up to 32 nodes)	Yes (up to 8 nodes in fibre channel switch fabric)

MSCS, when constructed properly, provides automatic failover of services and applications hosted across multiple cluster nodes. When multiple nodes are configured as a cluster for a service or application resource, only one node owns the resource at any given time. When the current resource owner experiences failure, causing a loss in the heartbeat between the cluster nodes, another node assumes ownership of the resource to allow continued access with minimal data loss. To configure multiple Windows Server 2003 nodes into a Microsoft cluster, the following requirements must be met:

♦ Nodes must be running either Windows Server 2003 Enterprise Edition or Datacenter Edition.

♦ All nodes should have access to the same storage device(s).

♦ All nodes should have two similarly connected and configured network adapters: one for the production network and one for the heartbeat network.

♦ All nodes should have Microsoft Cluster Services installed.

The clustering of Windows Server 2003 virtual machines using Microsoft Cluster Services (MSCS) can be done in one of three different configurations:

Cluster-in-a-Box�The clustering of two virtual machines that exist in the same ESX Server host.

Cluster-Across-Boxes�The clustering of two virtual machines that are running on different ESX Server hosts.

Physical-to-Virtual Clustering�The clustering of a physical server and a virtual machine that is running on an ESX Server host.

Clustering has long been considered an advanced technology implemented only by those with high technical skills in implementing and managing high-availability environments. While this might be more rumor than truth, it is certainly a more complex solution once the virtual machine solution is blended into the deployment.

While you may achieve results setting up clustered virtual machines, you may not receive support for your clustered solution if you violate any of the clustering restrictions put forth by VMware. The following list summarizes the dos and don'ts of clustering virtual machines as published by VMware:

♦ Only 32-bit virtual machines with a boot disk on local storage can be configured as nodes in a server cluster.

♦ Only two-node clustering is allowed.

♦ Virtual machines configured as cluster nodes must use the LSI Logic SCSI adapter and the vmxnet network adapter.

♦ Virtual machines in a clustered configuration cannot be connected to a switch configured with a NIC team.

♦ Virtual machines in a clustered configuration are not valid candidates for VMotion, nor can they be part of a DRS or HA cluster.

♦ ESX Servers hosts that run virtual machines that are part of a server cluster cannot be configured to perform a boot from SAN.

♦ ESX Server hosts that run virtual machines that are part of a server cluster cannot have both QLogic and Emulex HBAs.

The cluster-in-a-box scenario involves configuring two virtual machines hosted by the same ESX Server as nodes in a server cluster. The shared disks of the server cluster can exist as .vmdk files stored on local VMFS volumes. Figure 10.3 details the configuration of a cluster-in-a-box scenario.

After reviewing the diagram of a cluster-in-a-box configuration, you might wonder why you would want to deploy such a thing. The truth is, you wouldn't want to deploy cluster in a box because it still maintains a single point of failure. With both virtual machines running on the same host, if that host fails, both virtual machines fail. This architecture contradicts the very reason for creating failover clusters. A cluster-in-a-box still contains a single point of failure that can result in downtime of the clustered application. If the ESX Server hosting the two-node cluster-in-a-box fails, then both nodes are lost and a failover does not occur. This setup might, and I use might loosely, be used only to "play" with clustering services or to test clustering services and configurations. But ultimately, even for testing, it is best to use the cluster-across-boxes to get a better understanding of how this might be deployed in a production scenario.

Figure 10.3 A cluster-in-a-box configuration does not provide protection against a single point of failure. Therefore, it is not a common or suggested form of deploying Microsoft server clusters in virtual machines.

While the cluster-in-a-box scenario is more of an experimental or education tool for clustering, the cluster-across-boxes configuration provides a solid solution for critical virtual machines with stringent uptime requirements � for example, the enterprise-level servers and services like SQL Server and Exchange Server that are heavily relied on by the bulk of the end-user community. The cluster-across-boxes scenario, as the name applies, draws its high availability from the fact that the two nodes in the cluster are managed on different ESX Server hosts. In the event that one of the hosts fails, the second node of the cluster will assume ownership of the cluster group, and its resources and the service or application will continue responding to client requests.

The cluster-across-boxes configuration requires that virtual machines have access to the same shared storage, which must reside on a storage device external to the ESX Server hosts where the virtual machines run. The virtual hard drives that make up the operating system volume of the cluster nodes can be a standard VMDK implementation; however, the drives used as the shared storage must be set up as a special kind of drive called a raw device mapping. The raw device mapping is a feature that allows a virtual machine to establish direct access to a LUN on a SAN device.�

A cluster-across-boxes requires a more complex setup than the cluster-in-a-box scenario. When clustering across boxes, all proper communication between virtual machines and all proper communication from virtual machines and storage devices must be configured properly. Figure 10.4 provides details on the setup of a two-node virtual machine cluster-across-boxes using Windows Server 2003 guest operating systems.�

Perform the following steps to configure Microsoft Cluster Services across virtual machines on separate ESX Server hosts.�

Figure 10.4 A Microsoft cluster built on virtual machines residing on separate ESX hosts requires shared storage access from each virtual machine using a raw device mapping (RDM).

Creating the First Cluster Node

To create the first cluster node, follow these steps:

1. Create a virtual machine that is a member of a Windows Active Directory domain.

2. Right-click the new virtual machine and select the Edit Settings option.

3. Click the Add button and select the Hard Disk option.

4. Select the Raw Device Mappings radio button option, as shown in Figure 10.5, and then click the Next button.

Figure 10.5 Raw device mappings allow virtual machines to have direct LUN access.

5. Select the appropriate target LUN from the list of available targets, as shown in Figure 10.6.

6. Select the datastore location, shown in Figure 10.7, where the VMDK proxy file should be stored, and then click the Next button.

Figure 10.6 The list of available targets includes only the LUNs not formatted as VMFS.

Figure 10.7 By default the VMDK file that points to the LUN is stored in the same location as the existing virtual machine files.

�Select the Virtual radio button option to allow VMware snapshot functionality for the raw device mapping, as shown in Figure 10.8. Then click Next.

Figure 10.8 The Virtual Compatibility mode enables VMware snapshot functionality for RDMs. The physical mode allows raw LUN access but without VMware snapshots.

8.�Select the virtual device node to which the RDM should be connected, as shown in Figure 10.9. Then click Next.

Figure 10.9 The virtual device node for the additional RDMs in a cluster node must be on a different SCSI node.

SCSI nodes for RDMs

RDMs used for shared storage in a Microsoft server cluster must be configured on a SCSI node that is different from the SCSI to which the hard disk is connected that holds the operating system. For example, if the operating system's virtual hard drive is configured to use the SCSI0 node, then the RDM should use the SCSI1 node.�

9.�Click the Finish button.�

10. Right-click the virtual machine and select the Edit Settings option.

11. Select the new SCSI controller that was added as a result of adding the RDMs on a separate SCSI controller.

12. Select the Virtual radio button option under the SCSI Bus Sharing options, as shown in Figure 10.10.

13. Repeat steps 2 through 9 to configure additional RDMs for shared storage locations needed by nodes of a Microsoft server cluster.

14. Configure the virtual machine with two network adapters. Connect one network adapter to the production network and connect the other network adapter to the network used for heartbeat communications between nodes. Figure 10.11 shows a cluster node with two network adapters configured.�

NICs in a Cluster�

Because of PCI addressing issues, all RDMs should be added prior to configuring the additional network adapters. If the NICs are configured first, you may be required to revisit the network adapter configuration after the RDMs are added to the cluster node.�

Figure 10.10 The SCSI bus sharing for the new SCSI adapter must be set to Virtual to support running a virtual machine as a node in a Microsoft server cluster.

Figure 10.11 A node in a Microsoft server cluster requires at least two network adapters. One adapter must be able to communicate on the production network, and the second adapter is configured for internal cluster heartbeat communication.

15. Power on the first node of the cluster, and assign valid IP addresses to the network adapters configured for the production and heartbeat networks. Then format the additional drives and assign drive letters, as shown in Figure 10.12.

16. Shut down the first cluster node.

17. In the VirtualCenter inventory, select the ESX Server host where the first cluster node is configured and then select the Configuration tab.

18. Select Advanced Settings from the Software menu.

Figure 10.12 The RDMs presented to the first cluster node are formatted and assigned drive letters.

19.�In the Advanced Settings dialog box, configure the following options, as shown in Figure 10.13:

♦ Set the Disk.ResetOnFailure option to 1.

♦ Set the Disk.UseLunReset option to 1.

♦ Set the Disk.UseDeviceReset option to 0.

Figure 10.13 ESX Server hosts with virtual machines configured as cluster nodes require changes to be made to several advanced disk configuration settings.

20.�Proceed to the next section to configure the second cluster node and the respective ESX Server host.

Creating the Second Cluster Node

To create the second cluster node, follow these steps:

1. Create a second virtual machine that is a member of the same Active Directory domain as the first cluster node.

2. Add the same RDMs to the second cluster node using the same SCSI node values. For example, if the first node used SCSI 1:0 for the first RDM and SCSI 1:1 for the second RDM, then configure the second node to use the same configuration. As in the first cluster node configuration, add all RDMs to the virtual machine before moving on to step 3 to configure the network adapters. Don't forget to edit the SCSI bus sharing configuration for the new SCSI adapter.

3. Configure the second node with an identical network adapter configuration.

4. Verify that the hard drives corresponding to the RDMs can be seen in Disk Manager. At this point the drives will show as a status of Healthy but drive letters will not be assigned.

5. Power off the second node.

6. Edit the advanced disk settings for the ESX Server host with the second cluster node.

Creating the Management Cluster

To create the management cluster, follow these steps:

1. If you have the authority, create a new user account that belongs to the same Windows Active Directory domain as the two cluster nodes. The account does not need to be granted any special group memberships at this time.

2. Power on the first node of the cluster and log in as a user with administrative credentials.

3. Click Start→Programs→Administrative Tools, and select the Cluster Administrator console.

4. Select the Create new cluster option from the Open Connection to Cluster dialog box, as shown in Figure 10.14. Click OK.

Figure 10.14 The first cluster created is used to manage the nodes of the cluster.

5.�Provide a unique name for the name of the cluster, as shown in Figure 10.15. Ensure that it does not match the name of any existing computers on the network.

Figure 10.15 Configuring a Microsoft server cluster is heavily based on domain membership and the cluster name. The name provided to the cluster must be unique within the domain to which it belongs.

6.�As shown in Figure 10.16, the next step is to execute the cluster feasibility analysis to check for all cluster-capable resources. Then click Next.

Figure 10.16 The cluster analysis portion of the cluster configuration wizard identifies that all cluster-capable resources are available.

7.�Provide an IP address for cluster management. As shown in Figure 10.17, the IP address configured for cluster management should be an IP address that is accessible from the network adapters configured on the production network. Click Next.

Figure 10.17 The IP address provided for cluster management should be unique and accessible from the production network.

Cluster Management

To access and manage a Microsoft cluster, create a Host (A) record in the zone that corresponds to the domain to which the cluster nodes belong.�

8.�Provide the account information for the cluster service user account created in step 1 of the �Creating the Management Cluster� section. Note that the Cluster Service Account page of the New Server Cluster Wizard, shown in Figure 10.18, acknowledges that the account specified will be granted membership in the local administrators group on each cluster node. Therefore, do not share the cluster service password with users who should not have administrative capabilities. Click Next.�

Figure 10.18 The cluster service account must be a domain account and will be granted local administrator rights on each cluster node.

9.�At the completion of creating the cluster timeline, shown in Figure 10.19, click Next.

Figure 10.19 The cluster installation timeline provides a running report of the items configured as part of the installation process.

10.�Continue to review the Cluster Administrator snap-in and review the new management cluster that was created, shown in Figure 10.20.

Figure 10.20 The completion of the initial cluster management creation wizard will result in a Cluster Group and all associated cluster resources.

Adding the Second Node to the Management Cluster

To add the second node to the management cluster, follow these steps:

1. Leave the first node powered on and power on the second node.

2. Right-click the name of the cluster, select the New option, and then click the Node option, as shown in Figure 10.21.

Figure 10.21 Once the management cluster is complete, an additional node can be added.

3.�Specify the name of the node to be added to the cluster and then click Next, as shown in Figure 10.22.

Figure 10.22 You can type the name of the second node into the text box or find it using the Browse button.

4.�Once the cluster feasibility check has completed (see Figure 10.23), click the Next button.

Feasibility Stall

If the feasibility check stalls and reports a 0x00138f error stating that a cluster resource cannot be found, the installation will continue to run. This is a known issue with the Windows Server 2003 cluster configuration. If you allow the installation to continue, it will eventually complete and function as expected. For more information visit http://support.microsoft.com/kb/909968.

�5.�Proceed to review the Cluster Administrator identifying that two nodes now exist within the new cluster.�

Figure 10.23 A feasibility check is executed against each potential node to validate the hardware configuration that supports the appropriate shared resources and network configuration parameters.

At this point the management cluster is complete; from here, application and service clusters can be configured. Some applications like Microsoft SQL Server 2005 and Microsoft Exchange Server 2007 are not only cluster-aware applications but also allow for the creation of a server cluster as part of the standard installation wizard. Other cluster-aware applications and services can be configured into a cluster using the cluster administrator.

The last type of clustering scenario to discuss is the physical-to-virtual clustering configuration. As you may have guessed, this involves building a cluster with two nodes where one node is a physical machine and the other node is a virtual machine. Figure 10.24 details the setup of a two-node physical-to-virtual cluster.

Figure 10.24 Clustering physical machines with virtual machine counterparts can be a cost-effective way of providing high availability.

High availability has been an industry buzzword that has stood the test of time. The need and/or desire for high availability is often a significant component to network infrastructure design. Within the scope of ESX Server, VMware High Availability (HA) is a component of the VI3 Enterprise product that provides for automatic failover of virtual machines. But � and it's a big but at this point in time � HA does not provide high availability in the traditional sense of the term. Commonly, the term HA means automatic failover of a service or application to another server.

The VMware HA feature provides an automatic restart of the virtual machines that were running on an ESX Server host at the time it became unavailable, shown in Figure 10.25.

Figure 10.25 VMware HA provides an automatic restart of virtual machines that were running on an ESX Server host when it failed.

In the case of VMware HA, there is still a period of downtime when a server fails. Unfortunately, the duration of the downtime is not a value that can be calculated because it is unknown ahead of time how long it will take to boot a series of virtual machines. From this you can gather that, at this point in time, VMware HA does not provide the same level of high availability as found in a Microsoft server cluster solution. When a failover occurs between ESX Server hosts as a result of the HA feature, there is potential for data loss as a result of the virtual machine that was immediately powered off when the server failed and then brought back up minutes later on another server.

HA: Within, but Not Between, Sites

A requisite of HA is that each node in the HA cluster must have access to the same SAN LUNs. This requirement prevents HA from being able to failover between ESX Server hosts in different locations unless both locations have been configured to have access to the same storage devices. It is not acceptable just to have the data in LUNs the same due to SAN-replication software. Mirroring data from a LUN on a SAN in one location to a LUN on a SAN in a hot site is not conducive to allowing HA (VMotion or DRS).�

In the VMware HA scenario, two or more ESX Server hosts are configured in a cluster. Remember, a VMware cluster represents a logical aggregation of CPU and memory resources, as shown in Figure 10.26. By editing the cluster settings, the VMware HA feature can be enabled for a cluster. The HA cluster then determines the number of hosts failures it must support.�

Figure 10.26 A VMware ESX Server cluster logically aggregates the CPU and memory resources from all nodes in the cluster.

When ESX Server hosts are configured into a VMware HA cluster, they receive all the cluster information. VirtualCenter informs each node in the HA cluster about the cluster configuration.

HA and VirtualCenter

While VirtualCenter is most certainly required to enable and manage VMware HA, it is not required to execute HA. VirtualCenter is a tool that notifies each VMware HA-cluster node about the HA configuration. Once the nodes have been updated with the information about the cluster, VirtualCenter no longer maintains a persistent connection with each node. Each node continues to function as a member of the HA cluster independent of its communication status with VirtualCenter.

When an ESX Server host is added to a VMware HA cluster, a set of HA specific components are installed on the ESX Server. These components, shown in Figure 10.27, include:

♦ Automatic Availability Manager (AAM)

♦ VMap

♦ vpxa

Figure 10.27 Adding an ESX Server host to an HA cluster automatically installs the AAM, VMap, and possibly the vpxa components on the host.

The AAM, effectively the engine for HA, is a Legato-based component that keeps an internal database of the other nodes in the cluster. The AAM is responsible for the intracluster heartbeat used to identify available and unavailable nodes. Each node in the cluster establishes a heartbeat with each of the other nodes over the Service Console network. As a best practice, you should provide redundancy to the AAM heartbeat by establishing the Service Console port group on a virtual switch with an underlying NIC team. Though the Service Console could be multihomed and have an AAM heartbeat over two different networks, this configuration is not as reliable as the NIC team. The AAM is extremely sensitive to hostname resolution; the inability to resolve names will most certainly result in an inability to execute HA. When problems arise with HA functionality, look first at hostname resolution. Having said that, during HA troubleshooting, you should identify the answers to questions like:

♦ Is the DNS server configuration correct?

♦ Is the DNS server available?

♦ If DNS is on a remote subnet, is the default gateway correct and functional?

♦ Does the /etc/hosts file have bad entries in it?

♦ Does the /etc/resolv.conf have the right search suffix?

♦ Does the /etc/resolv.conf have the right DNS server?

Adding a Host to VirtualCenter

When a new host is added into the VirtualCenter inventory, the host must be added by its hostname or the HA will not function properly. As just noted, HA is heavily reliant on successful name resolution. ESX Server hosts should not be added to the VirtualCenter inventory using IP addresses.

The AAM on each ESX Server host keeps an internal database of the other hosts belonging to the cluster. All hosts in a cluster are considered either a primary host or a secondary host. However, only one ESX Server in the cluster is considered the primary host at a given time, with all others considered secondary hosts. The primary host functions as the source of information for all new hosts and defaults to the first host added to the cluster. If the primary host experiences failure, the HA cluster will continue to function. In fact, in the event of primary host failure, one of the secondary hosts will move up to the status of primary host. The process of promoting secondary hosts to primary is limited to four other hosts. Only five hosts could assume the role of primary host in an HA cluster.�

While the AAM is busy managing the intranode communications, the vpxa service manages the HA components. The vpxa service communicates to the AAM through a third component called the vMap.

Before we detail how to set up and configure the HA feature, let's review the requirements of HA. To implement HA, all of the following requirements should be met:

♦ All hosts in an HA cluster must have access to the same shared storage locations used by all virtual machines on the cluster. This includes any fibre channel, iSCSI, and NFS datas tores used by virtual machines.

♦ All hosts in an HA cluster should have an identical virtual networking configuration. If a new switch is added to one host, the same new switch should be added to all hosts in the cluster.

♦ All hosts in an HA cluster must resolve the other hosts using DNS names.�

First and foremost, to configure HA a cluster must be created. Once the cluster is created, you can enable and configure HA. Figure 10.28 shows a cluster enabled for HA.�

Figure 10.28 A cluster of ESX Server hosts can be configured with HA and DRS. The features are not mutually exclusive and can work together to provide availability and performance optimization.

Configuring an HA cluster revolves around three different settings:

♦ Host failures allowed

♦ Admission control

♦ Virtual machine options

The configuration option for the number of host failures to allow, shown in Figure 10.29, is a critical setting. It directly influences the number of virtual machines that can run in the cluster before the cluster is in jeopardy of being unable to support an unexpected host failure.

Figure 10.29 The number of host failures allowed dictates the amount of spare capacity that must be retained for use in recovering virtual machines after failure.

Real World Scenario

HA Configuration Failure

It is not uncommon for a host in a cluster to fail during the configuration of HA. Remember the stress we put on DNS earlier in the chapter. Well, if DNS is not set correctly, you will find that the host cannot be configured for HA. Take, for example, a cluster with three nodes being configured as an HA cluster to support two-node failure. Enabling HA forces a configuration of each node in the cluster. The image here shows an HA cluster where one of the nodes, Silo104, has thrown an error related to the HA agent and is unable to complete the HA configuration:

In this example, because the cluster was attempting to allow for two-node failure and there are only two nodes successfully configured, this would be impossible. The cluster in this case is now warning that there are insufficient resources to satisfy the HA failover level. Naturally, with only two nodes we cannot cover two-node failure. The image here shows an error on the cluster due to the failure in Silo104:

In the tasks pane of the graphic, you may have noticed that Silo105 and Silo106 both completed the HA configuration successfully. This provides evidence that the problem is probably isolated to Silo104. Reviewing the Tasks & Events tab to get more detail on the error reveals exactly that. The next image shows that the error was caused by an inability to resolve a name. This confirms the suspicion that the error is with DNS.

Perform the following steps to review or edit the DNS server for an ESX Server:�

1. Use the Virtual Infrastructure (VI) Client to connect to a VirtualCenter server.

2. Click the name of the host in the inventory tree on the left.

3. Click the Configuration tab in the details pane on the right.

4. Click DNS and Routing from the Advanced menu.

5. If needed, edit the DNS server, as shown in the image here, to a server that can resolve the other nodes in the HA cluster:�

Although they should not be edited on a regular basis, you can also check the /etc/hosts and /etc/resolv.conf files, which should contain static lists of hostnames to IP addresses or the DNS�search domain and name servers, respectively. This image offers a quick look at the information inside the /etc/hosts and /etc/resolv.conf files. These tools can be valuable for troubleshooting name resolution.

Once the DNS server, /etc/hosts, or /etc/resolv.conf has been corrected, the host with the failure can be reconfigured for HA. It's not necessary to remove the HA configuration from the cluster and then re-enable it. The image here shows the right-click context menu of Silo104, where it can be reconfigured for HA now that the name resolution problem has been fixed.

Upon completion of the configuration of the final node, the errors at the host and cluster levels will be removed, the cluster will be configured as desired, and the error regarding the inability to satisfy the failover level will disappear.

To explain the workings of HA and the differences in the configuration settings, let's look at implementation scenarios. For example, consider five ESX Server hosts named Silo101 through Silo105. All five hosts belong to an HA cluster configured to support single-host failure. Each node in the cluster is equally configured with 12GB of RAM. If each node runs eight virtual machines�with 1GB of memory allocated to each virtual machine, then 8GB of unused memory across four hosts is needed to support a single-host failure. The 12GB of memory on each host minus 8GB for virtual machines leaves 4GB of memory per host. Figure 10.30 shows our five-node cluster in normal operating mode.�

Figure 10.30 A five-node cluster configured to allow single-host failure.

Let's assume that service console and virtual machine overhead consume 1GB of memory, leaving 3GB of memory per host. If Silo101 fails, the remaining four hosts will each have 3GB of memory to contribute to running the virtual machines orphaned by the failure. The 8GB of virtual machines will then be powered on across the remaining four hosts that collectively have 12GB of memory to spare. In this case, the configuration supported the failover. Figure 10.31 shows our five-node cluster down to four after the failure of Silo101. Assume in this same scenario that Silo101 and Silo102 both experience failure. That leaves 16GB of virtual machines to cover across only three hosts with 3GB of memory to spare? In this case, the cluster is deficient and not all of the orphaned virtual machines will be restarted.

Figure 10.31 A five-node cluster configured to allow single-host failure is deficient in resources to support a second failed node.

Primary Host Limit

In the previous section introducing the HA feature, we mentioned that the Automated Availability Manager (AAM) caps the number of primary hosts at five. This limitation translates into a maximum of four host failures allowed in a cluster.�

The admission control setting goes hand in hand with the Number of host failures allowed setting. There are two possible settings for admission control:

♦ Do not power on virtual machines if they violate availability constraints (known as strict admission control).

♦ Allow virtual machines to be powered on even if they violate availability constraints (known as guaranteed admission control).

In the previous example, virtual machines would not power on when Silo102 experienced failure because by default an HA cluster is configured to use strict admission control. Figure 10.32 shows an HA cluster configured to use the default setting of strict admission control.�

Figure 10.32 Strict admission control for an HA cluster prioritizes resource balance and fairness over resource availability.

With strict admission control, the cluster will reach a point at which it will no longer start virtual machines. Figure 10.33 shows a cluster configured for two-node failover. A virtual machine with more than 3GB of memory reserved is powering on, and the resulting error is posted stating that insufficient resources are available to satisfy the configured HA level.

Figure 10.33 Strict admission control imposes a limit at which no more virtual machines can be powered on because the HA level would be jeopardized.

If the admission control setting of the cluster is changed from strict admission control to guaranteed admission control, then virtual machines will power on even in the event that the HA failover level is jeopardized. Figure 10.34 shows a cluster reconfigured to use guaranteed admission control.

Figure 10.34 Guaranteed admission control reflects the idea that when failure occurs, availability is more important than resource fairness and balance.

With that same cluster now configured with guaranteed admission control, the virtual machine with more than 3GB of memory can now successfully power on. In Figure 10.35, the virtual machine has successfully powered on despite the large memory use and lack of available unused resources to achieve the proper HA failover.

Figure 10.35 Guaranteed admission control allows resource consumption beyond the levels required to maintain spare resources for use in the event of a server failure.

Overcommitment in an HA cluster

When the admission control setting is set to allow virtual machines to be powered on even if they violate availability constraints, you could find yourself in a position where there is more physical memory allocated to virtual machines than actually exists. This situation, called overcommitment, can lead to poor performance on virtual machines that become forced to page information from fast RAM out to the slower disk based swap file.�

HA Restart Priority�

Not all virtual machines are equal. There are some that are more important or more critical and that require higher priority when ensuring availability. When an ESX Server host experiences failure and the remaining cluster nodes are tasked with bringing virtual machines back on line, they have a finite amount of resources to fill before there are no more resources to allocate to virtual machines that need to be powered on. Rather than leave the important virtual machines to chance, an HA cluster allows for the prioritization of virtual machines. The restart priority options for virtual machines in an HA cluster include Low, Medium, High, and Disabled. For those virtual machines that should be brought up first, the restart priority should be set to High. For those virtual machines that should be brought up if resources are available, the restart priority can be set to Medium and/or Low. For those virtual machines that will not be missed for a period of time and should not be brought on line during the period of reduced resource availability, the restart priority should be set to Disabled. Figure 10.36 shows an example of virtual machines with various restart priorities configured to reflect their importance. The diagram details a configuration where virtual machines like domain controllers, database servers, and cluster nodes are assigned higher restart priority.

Figure 10.36 Restart priorities help minimize the downtime for more important virtual machines.

The restart priority is only put into place for the virtual machines running on the ESX Server host that experienced an unexpected failure. Virtual machines running on hosts that have not failed are not affected by the restart priority. It is possible then that virtual machines configured with a restart priority of High may not be powered on by the HA feature due to limited resources, which are in part due to lower-priority virtual machines that continue to run. For example, as shown in Figure 10.37, Silo101 hosts five virtual machines with a priority of High and five other virtual machines with priority values of Medium and Low. Meanwhile, Silo102 and Silo103 each hold ten virtual machines, but of the 20 virtual machines between them, only four are considered of high priority. When Silo101 fails, Silo102 and Silo103 will begin powering the virtual machines with a high priority. However, assume there were only enough resources to power on four of the five virtual machines with high priority. That leaves a high-priority virtual machine powered off while all other virtual machines of medium and low priority continue to run on Silo102 and Silo103.

Figure 10.37 High-priority virtual machines from a failed ESX Server host may not be powered on because of a lack of resources � resources consumed by virtual machines with a lower priority that are running on the other hosts in an HA cluster.

At this point in the VI3 product suite, you can still manually remedy this imbalance. Any disaster-recovery plan in a virtual environment built on VI3 should include a contingency plan that identifies virtual machines to be powered off to make resources available for those virtual machines with higher priority as a result of the network services they provide. If the budget allows, construct the HA cluster to ensure that there are ample resources to cover the needs of the critical virtual machines, even in times of reduced computing capacity.

HA Isolation Response

Previously, we introduced the Automated Availability Manager (AAM) and its role in conducting the heartbeat that occurs among all the nodes in the HA cluster. The heartbeat among the nodes in the cluster identifies the presence of each node to the other nodes in the cluster. When a heartbeat is no longer presented from a node in the HA cluster, the other cluster nodes spring into action to power on all the virtual machines that the missing node was running.

But what if the node with the missing heartbeat was not really missing? What if the heartbeat was missing but the node was still running? And what if the node with the missing heartbeat is still locking the virtual machine files on SAN LUN, thereby preventing the other nodes from powering on the virtual machines?

Let's look at two particular examples of a situation VMware refers to as a split-brained HA cluster. Let's assume there are three nodes in an HA cluster: Silo101, Silo102, and Silo103. Each node is configured with a single virtual switch for VMotion, and a second virtual switch consisting of a Service Console port and a virtual machines port group, as shown in Figure 10.38.

Figure 10.38 ESX Server hosts in an HA cluster using a single virtual switch for Service Console and virtual machine communication.

To continue with the example, suppose that an administrator mistakenly unplugs the Silo101 Service Console network cable. When each of the nodes identifies a missing heartbeat from another node, the discovery process begins. After 15 seconds of missing heartbeats, each node then pings an address called the isolation response address. By default this address is the default gateway IP address configured for the Service Console. If the ping attempt receives a reply, the node considers itself valid and continues as normal. If a host does not receive a response, as Silo101 wouldn't, it considers itself in isolation mode. At this point, the node will identify the cluster's Isolation Response configuration, which will guide the host to either power off the existing virtual machines it is hosting or leave them powered on. This isolation response value, shown in Figure 10.39, is set on a per-virtual machine basis. So what should you do? Power off the existing virtual machine? Or leave it powered on?

The answer to this question is highly dependent on the virtual and physical network infrastructures in place. In our example, the Service Console and virtual machines are connected to the same virtual switch bound to a single network adapter. In this case, when the cable for the Service Console was unplugged, communication to the Service Console and every virtual machine on that computer was lost. The solution, then, should be to power off the virtual machines. By doing so, the other nodes in the cluster will identify the releases on the locks and begin to power on the virtual machines that were not otherwise included.

Figure 10.39 The Isolation Response identifies the action to occur when an ESX Server host determines it is offline but powered on.

In the next example, we have the same scenario but a different infrastructure, so we don't need to worry about powering off virtual machines in a split-brain situation. Figure 10.40 diagrams a virtual networking architecture in which the Service Console, VMotion, and virtual machines all communicate through individual virtual switches bound to different physical network adapters. In this case, if the network cable connecting the Service Console is removed, the heartbeat will once again be missing; however, the virtual machines will be unaffected since they reside on a different network that is still passing communications between the virtual and physical networks.

Figure 10.40 Redundancy in the physical infrastructure with isolation of virtual machines from the Service Console in the virtual infrastructure provides greater flexibility for isolation response.

Figure 10.41 shows the isolation response setting of Leave powered on, which would accompany an infrastructure built with redundancy at the virtual and physical network levels.

Figure 10.41 The option to leave virtual machines running when a host is isolated should only be set when the virtual and the physical networking infrastructures support high availability.

Real World Scenario

Configuring the Isolation Response Address

In some highly secure virtual environments, Service Console access is limited to a single, nonrouted management network. In some cases, the security plan calls for the elimination of a default gateway on the Service Console port configuration. The idea is to lock the Service Console onto the local subnet, thus preventing any type of remote network access. The disadvantage, as you might have guessed, is that without a default gateway IP address configured for the Service Console, there is no isolation address to ping as a determination of isolation status.

It is possible, however, to customize the isolation response address for scenarios just like this. The IP address can be any IP address, but should be an IP address that is not going to be unavailable or taken from the network at any time.

Perform the following steps to define a custom isolation response address:

1. Use the VI Client to connect to a VirtualCenter server.

2. Right-click on an existing cluster and select the Edit Settings option.

3. Click the VMware HA node.

4. Click the Advanced Options button.

5. Type das.isolationaddress in the Option column in the Advanced Options (HA) dialog box.

6. Type the IP address to be used as the isolation response address for ESX Server hosts that miss the AAM heartbeat. The following image shows a sample configuration in which the servers will ping the IP address 172.30.0.2:

7.�Click the OK button twice.�

This interface can also be configured with the following options:

♦ das.isolationaddress1: To specify the first address to try.

♦ das.isolationaddress2: To specify the second address to try.

♦ das.defaultfailoverhost: To identify the preferred host to fail over to.

♦ das.failuredetectiontime: Used to change the amount of time required for failover detection.�

To support a redundant HA architecture, it is best to ensure that the Service Console port is sitting atop a NIC team where each physical NIC bound to the virtual switch is connected to a different physical switch.

Virtual machines are no less likely to lose data, become corrupted, or fail the way their physical counterparts might. And though some may argue against that point, it is most certainly the best way for you to look at virtual machines. With this school of thought, you might be jeopardizing the infrastructure with overconfidence in virtual machine stability. It's better to be safe than sorry. When it comes to virtual machine backup planning, VMware suggests three different methods you can use to support your disaster recovery/business continuity plan. The three methods include:

♦ Using backup agents inside the virtual machine.

♦ Using VCB to perform virtual machine backups.

♦ Using VCB to perform file-level backups (Windows guests only).

VMware Consolidated Backup (VCB) is VMware's first entry into the backup space. (For those of you who have worked with ESX 2,I am not counting the vmsnap.pl and vmres.pl as attempts to provide a backup product). VCB is a framework for backing up that integrates easily into a handful of third-party products. Although VCB can be used on its own, it lacks some of the nice features third-party backup products bring to the table. These include features like cataloging backups, scheduling capability, and media management backups. For this reason, I recommend that you use the VCB framework in conjunction with third-party products that have been tested.

More than likely, none of the three methods listed will suffice if used alone. As this chapter provides more details about each of the methods, you'll see how a solid backup strategy is based on using several of these methods in a complementary fashion.

Oh so many years ago when virtualization was not even a spot on your IT roadmap, you were backing up your physical servers according to some kind of business need. For most organizations, the solution involved the purchase, installation, configuration, and execution of third-party backup agents on the operating systems running on physical hardware. Now that you have jumped onto the cutting edge of technology by leading the server consolation charge into a virtual IT infrastructure, you can still back up using the same traditional methods. Virtual machines like physical machines are targets for third-party backup tools. The downside to this time-tested model is the need to continue paying for the licenses needed to perform backups across all servers. As shown in Figure 10.42, you'll need a license for every virtual machine you wish to back up: 100 virtual machines = 100 licenses. Some vendors allow for a single ESX Server license that permits an unlimited number of agent licenses to be installed on virtual machines on that host.

In this case, virtualization has not lowered total ownership costs and the return on investment has not changed with regard to the fiscal accountability to the third-party backup company. So perhaps this is not the best avenue down which you should travel. With that being said, let's look�at other options that rely heavily on the virtualized aspect of the guest operating system. These options include:

♦ Using VCB for full virtual machine backups.

♦ Using VCB for single VMDK backups.

♦ Using VCB for file-level backups.

Figure 10.42 Using third-party backup agents inside a virtual machine does not take advantage of virtualization. Virtual machines are treated just like their physical counterparts for the sake of a disaster recovery or business continuity plan.

As we mentioned briefly in the opening section, VCB is a framework for backup that integrates with third-party backup software. It is a series of scripts that performs online, LAN-free backups of virtual machines or virtual machine files.

The requirements for VCB 1.1 include:�

♦�A physical server running Windows Server 2003 Service Pack 1.

�♦�If using Windows Server 2003 Standard Edition, the VCB server must be configured not to automatically assign drive letters using diskpart to execute automount disable and automount scrub.�

♦ Network connectivity for access to VirtualCenter.

♦ Fibre channel HBA with access to all SAN LUNs where virtual machine files are stored.

♦ Installation of the third-party software prior to installing and configuring VCB.�

Figure 10.43 looks at the VCB components and architecture.�

Figure 10.43 A VCB deployment relies on several communication mediums, including network access to VirtualCenter and fibre channel access to all necessary SAN LUNs.

VMware regularly tests third-party support for VCB, and, as a result, the list of supported backup products continues to change. As of this writing, the following products were listed as backup products for which VMware provides integration scripts:

♦ EMC NetWorker

♦ Symantec Backup Exec

♦ Tivoli Storage Manager

♦ Veritas NetBackup

In addition to these four products, VMware lists the following products as having created integrations to allow their products to capitalize on the VCB framework:

♦ Vizioncore vRanger Pro, formerly ESXRanger Pro

♦ Certificate Associates (CA) BrightStor ARCServe

♦ CommVault Galaxy

♦ EMC Avamar

♦ HP Data Protector versions 5.5 and 6

Be sure to regularly visit VMware's website to download and review the PDF at http://www.vmware.com/pdf/vi_3backupguide.pdf.

Although considered a framework for backup, VCB can actually be used as a backup product. However, it lacks the nice scheduling and graphical interface features of third-party products like Vizioncore vRanger Pro. Two of the more common VCB commands, shown in Figures 10.44 and 10.45, are:

♦�vcbVmName: This command enumerates the various ways a virtual machine can be referenced in the vcbMounter. Here's an example:

vcbVmName -h 172.30.0.120 -u administrator -p Sybex!! -s ipaddr:172.30.0.24

where

�♦ -h <VirtualCenter Server name or IP address>

�♦ -u <VirtualCenter username>

�♦ -p <VirtualCenter user password>

�♦ -s ipaddr: <IP address of virtual machine to backup>

♦�vcbMounter:

�♦ -h <VirtualCenter Server name or IP address>

�♦ -u <VirtualCenter username>

�♦ -p <VirtualCenter user password>

�♦ -a <name | ipaddr | moref | uuid>: <attribute value>

�♦ -t [fullvm | file]

�♦ -r <Backup directory on VCB proxy>

VCB Proxy Backup Directory

When specifying the VCB backup directory using the -r parameter, do not specify an existing folder. For example, if a backup directory E:\VCBBackups already exists and a new backup should be stored in a subdirectory named Server1, then specify the subdirectory without creating it first. In this case, the -r parameter might read as follows:

-r E:\VCBBackups\Server1

The vcbMounter will create the new directory as needed. If the directory is created first, an error will be thrown at the beginning of the backup process. The error will state that the directory already exists.�

Figure 10.44 The vcbVmName command enumerates the various virtual machine identifiers that can be used when running the VCB command.

Figure 10.45 The vcbMounter command can be used to perform full virtual machines backups or file-level backups for Windows virtual machines.

When VCB performs a full backup of a virtual machine, it engages the VMware snapshot functionality to quiesce the file system and perform the backup. Remember that snapshots are not complete copies of data. Instead, a snapshot is the creation of a differencing file (or redo log) to which all changes are written. When the vcbMounter command is used, a snapshot is taken of the virtual machine, as shown in Figure 10.46.

Figure 10.46 The snapshots created by VCB can be viewed in the snapshot manager of a virtual machine.

Any writes that occur during the backup are done to the differencing file. Meanwhile, VCB is busy making a copy of the VMDK, which is now read-only for the duration of the backup job. Figure 10.47 details the full virtual machine backup process. Once the backup job is completed, the snapshot is deleted, forcing all writes that occurred to the differencing file to be written to the virtual machine disk file.

Figure 10.47 Performing a full virtual machine backup utilizes the VMware snapshot functionality, which ensures that an online backup is correct as of that point in time.

Snapshots and VMFS Locking

Snapshots grow by default in 16MB increments, and for the duration of time it takes to grow a snapshot, a lock is held on the VMFS volume so the respective metadata can be updated to reflect the change in the snapshot. For this reason, do not instantiate a snapshot for many virtual machines at once. Although the lock is held only for the update to the metadata, the more virtual machines trying at the same time, the greater the chance of contention on the VMFS metadata. From an IT standpoint, this factor should drive your backup strategy to perform backups of many virtual machines only if the virtual machine files have been located on separate VMFS volumes.�

Perform the following steps to perform a full virtual machine backup using VCB:�

1. Log in to the backup proxy where VCB is installed.

2. Open a command prompt and change directories to the C:\Program Files\VMware\VMware Consolidated Backup Framework directory.

3. Use the vcbVmName tool to enumerate virtual machine identifiers. At the command prompt, type:

vcbVmName <IP or name of VirtualCenter>�-u�<username> -p <password> -s ipaddr:<IP address of virtual machine to backup>

4. From the results of running the vcbVmName tool, select which identifier to use (moref, name, uuid, or ipaddr) in the vcbMounter command.

5. At the command prompt, type:

VcbMounter -h <IP or name of VirtualCenter>�-u <username> -p <password> -a ipaddr:<IP address of virtual machine to backup> -t fullvm -r <VCB proxy backup directory>

Once the backup is complete, a list of the files can be reviewed in the directory provided in the backup script. Figure 10.48 shows the files created as part of the completed full backup of a virtual machine named Server1.�

Figure 10.48 A full virtual machine backup using VCB creates a directory of files that include a configuration file (VMX), log files, and virtual machine hard drives (VMDK), among others.

Redundant Paths

Let's look at an example of a VCB backup proxy with a single QLogic fibre channel HBA that is connected to a single fibre channel switch connected to two storage processors on the storage device. This configuration results in two different paths being available to the VCB server. The image here shows that a VCB server with a single HBA will find LUNs twice because of the redundancy at the storage-processor level:

When Disk Management shows this configuration for the older versions of VCB, it presents a problem that causes every backup attempt to fail. For the pre-VCB 1.0.3 versions, the LUNs identified as Unknown and Unreadable must be disabled in Disk Management. The option to disable is located on the properties of a LUN. The following image displays the General tab of LUN properties from Disk Management where a path to a LUN can be disabled. To remove the redundant unused paths from Computer Management, the Device Usage drop-down list should be set to Do not use this device (disable).

With redundant paths disabled, this will, of course, present a problem when a LUN trespasses to another storage processor. This requires a path to the LUN that is likely disabled.

Sometimes a full backup is just too much: too much data that hasn't changed, or too much data that is backed up more regularly and isn't needed again. For example, what if just the operating system drive needs to be backed up, not all the user data stored on other virtual machine disk files? A full backup would back up everything. For those situations, VCB provides a means of performing single virtual machine disk backups.

Perform the following steps for a single VMDK backup:

1. Log in to the backup proxy where VCB is installed.

2. Open a command prompt and change directories to the C:\Program Files\VMware\VMware Consolidated Backup Framework directory.

3. Use the vcbVmName tool to enumerate virtual machine identifiers. At the command prompt, type:

vcbVmName <IP or name of VirtualCenter>�-u�<username> -p <password> -s ipaddr:<IP address of virtual machine to backup>

4. From the results of running the vcbVmName command, note the moref value of the virtual� machine.

5. Use the following command to create a snapshot of the virtual machine:

vcbSnapshot -h <IP or name of VirtualCenter> -u <username> -p <password> -c <moref value of virtual�machine>�<name of snapshot>

6. Note the snapshot ID (SsiDd) from the results of step 5.

7. Enumerate the disks within the snapshot using the vcbSnapshot command:

vcbSnapshot -h <IP or name of VirtualCenter> -u <username> -p <password> -l <moref value of virtual�machine>�<snapshot ID>

8.�Change to the backup directory of the virtual machine and export the desired VMDK using the vcbExport command:

vcbExport -d <name of new VMDK copy>�-s <name of existing VMDK>�

9.�Remove the snapshot by once again using the vcbSnapshot command:

vcbSnapshot -h <IP or name of VirtualCenter> -u <username> -p <password> -d <moref value of virtual�machine>�<snapshot ID>

For Windows virtual machines, and only for Windows virtual machines, VCB offers file-level backups. A file-level backup is an excellent complement to the full virtual machine or the single VMDK backup discussed in the previous sections. For example, suppose you built a virtual machine using two virtual disks: one for the operating system and one for the custom user data. The operating system's virtual disk will not change often with the exception of the second Tuesday of each month when new patches are released. So that virtual disk does not need consistent and regular backups. On the other hand, the virtual disk that stores user data might be updated quite frequently. To get the best of both worlds and implement an efficient backup strategy, you need to do a single VMDK backup (for the OS) and file-level backup (for the data).

Perform the following steps to conduct a file-level backup using VCB:

1. Log in to the backup proxy where VCB is installed.

2. Open a command prompt and change directories to C:\Program Files\VMware\VMware Consolidated Backup Framework.

3. Use the vcbVmName tool to enumerate virtual machine identifiers. At the command prompt, type:

vcbVmName <IP or name of VirtualCenter>�-u�<username> -p <password> -s ipaddr:<IP address of virtual machine to backup>

4. From the results of running the vcbVmName tool, select which identifier to use (moref, name, uuid, or ipaddr) in the vcbMounter command.

5. As shown in Figure 10.49, type the following at the command prompt:

VcbMounter -h <IP or name of VirtualCenter>�-u <username> -p <password> -s ipaddr:<IP address of virtual�machine to backup> -t file -r <VCB proxy backup directory>

6. Browse to the mounted directory to back up the required files and folders.

7. After the file- or folder-level backup is complete, use the following command, shown in Figure 10.50, to remove the mount point:

mountvm -u�<path to mount point>

8.�Exit the command prompt.

Figure 10.49 A file- or folder-level backup begins with mounting the virtual machine drives as directories under a mount point on the VCB server.

Figure 10.50 After performing a file-or folder-level backup using the vcbMounter command, the mount point must be removed using the mountvm command.

Sticky Snapshots

If a snapshot refuses to delete when the mountvm -u command is issued, it can always be deleted from the snapshot manager user interface, which is accessible through the VI Client.�

Real World Scenario

VCB with Third-Party Products

Once you have mastered the VCB framework by understanding the VCB mounter commands and the way that VCB works, then working with VCB and third-party products is an easy transition. The third-party products simply call upon the VCB framework to perform the vcbmounter command. All the while the process is wrapped up nicely inside the GUI of the third-party product. This allows for scheduling the backups through backup jobs.

Let's look at an example with Symantec Backup Exec 11d. Once the 11d product is installed, followed by the installation of VCB, a set of integration scripts can be extracted from VCB to support the Backup Exec installation. When a backup job is created in Backup Exec 11d, a pre-backup script runs (which calls vcbmounter to create the snapshot and mount the VMDKs), and once the backup job completes, a post-backup script runs to unmount the VMDKs. During the period of time that VMDKs are mounted into the file system, the Backup Exec product has access to the mounted VMDKs in order to back them up to disk or tape, as specified in the backup job. See the following sample scripts, which perform a full virtual machine backup of a virtual machine with the IP address 192.168.4.1.

First, the pre-backup script example:

"C:\Program Files\VMware\VMware Consolidated Backup Framework\backupexec\pre-backup.bat"�Server1_FullVM 192.168.4.1-fullvm

Now, the post-backup script example:

"C:\Program Files\VMware\VMware Consolidated Backup Framework\backupexec\post-backup.bat"�Server1_FullVM 192.168.1.10-fullVM

Notice that there is no reference to vcbmounter or the parameters required to run the command. Behind the scenes the pre-backup. bat and post-backup.bat files are reading a configuration file named config.js to pull defaults for some of the parameters for vcbmounter and then using the information given in the lines shown earlier. When vcbmounter extracts the virtual machine files to the file system of the VCB proxy, the files will be found in a folder named 192.168.4.1-fullvm in a directory specified in the configuration file. A portion of the configuration file is shown here. Note that the file identifies the directory to mount the backups to (F:\\mnt) as well as the VirtualCenter server to connect to (vc01.vlearn.vmw) and the credentials to be used (administrator /Password1):

/*

* Generic configuration file for VMware Consolidated Backup (VCB).

*/

/*

* Directory where all the VM backup jobs are supposed to reside in.�

* For each backup job, a directory with a unique name derived from the

* backup type and the VM name will be created here.

* If omitted, BACKUPROOT defaults to c:\\mnt.

*

* Make sure this directory exists before attempting any VM backups.

*/

BACKUPROOT="F:\\mnt";/*

* URL that is used by "mountvm" to obtain the block list for a

* disk image that is to be mounted on the backup proxy.

*

* Specifying this option is mandatory. There is no default

* value.

*/

HOST="vc01.learn.vmw";

/*

* Port for communicating with all the VC SDK services.

* Defaults to 902

*/

// PORT="902";

/*

* Username/password used for authentication against the mountvm server.

* Specifying these options is mandatory.

*/

USERNAME="administrator"; PASSWORD="Password1";

The combination of the configuration file with the parameters passed at the time of execution results in a successful mount and copy of the virtual machine disk files followed by an unmount.

Depending on the size of the virtual machines to be backed up, it might be more feasible to back up to disk and then create a second backup job to take the virtual machine backups to a tape device.

Restoring data in a virtual environment can take many forms. If using VCB in combination with an approved third-party backup application, there are three specific types of restores that can be defined. These restore types include:

Centralized Restore�One backup agent on the VCB proxy.

Decentralized Restore�Several backup agents installed around the network, but not every system has one.

Self-Service Restore�Each virtual machine contains a backup agent.

Why are we discussing backup agents in the restore section? Remember, the number of backup agents purchased directly influences the virtual machines that can also be restored directly.�

No matter how you implant the whole backup/restore process, you must understand that it's either ��pay me now or pay me later.�� Something that is easier to back up is often more difficult to restore. On the flip side, something that is the most difficult to back up is often easier to restore. Figure 10.51 shows the difference between the centralized restore and the self-service restore.�

Figure 10.51 Backup agents are not just for backup. They also allow restore capability. The number of backup agents purchased and installed directly affects the recovery plan.

When bad things happen, such as the deletion or corruption of a virtual machine, a restore from a full virtual machine backup will return the environment to the point in time when the backup was taken.�

Figure 10.52 A server from the inventory is missing and a search through the data-stores does not locate the virtual machine disk files.

Perform the following steps on a virtual machine from a full virtual machine backup:

1.�Connect to the VCB proxy and use FastSCP, shown in Figure 10.53, or WinSCP to establish a secure copy protocol session with the remote host. Shown in Figure 10.53, the data from the E:\VCBBackups\Server1 folder can be copied into a temporary directory in the service console. The temporary directory houses all of the virtual machine files from the backup of the original virtual machine.

Figure 10.53 The FastSCP utility, as the name proclaims, offers a fast, secure copy protocol application to move files back and forth between Windows and ESX.

2. Upon completion of the restore to a temporary location process, verify the existence of the files by navigating to the shared site, as shown in Figure 10.54. Use Putty.exe to connect to the Service Console and navigate to the temporary directory where the backup files are stored. Then use the ls command to list all the files in the temporary directory.

Figure 10.54 Virtual machine files needed for the restore are located in the temporary directory specified in the command.

3.�From a command prompt, type the following:

vcbRestore -h 172.30.0.120 -u administrator -p Sybex123 -s <path to temp directory>

Figure 10.55 shows the virtual machine restore process.

4.�Upon completion of the restore from the temporary location process, verify the existence of the files by navigating to datastore or by quickly glancing at the tree view of VirtualCenter.

Figure 10.55 Virtual machine restore times are highly dependent on the size of the virtual machine and the number of writes to it.

Problems in the datacenter are not always as catastrophic as losing an entire virtual machine because of corruption or deletion. In fact, it is probably more common to experience minor issues like corrupted or deleted files. A full virtual machine backup does not have to be restored as a full virtual machine. Using the mountvm tool, it is possible to mount a virtual machine hard drive into the file system of the backup proxy (VCB) server. Once the hard drive is mounted, it can be browsed the same as any other directory on the server.

Let's say that a virtual machine named Server1 has a full backup that has been completed. An administrator deletes a file named FILE TO RECOVER.txt that was on the desktop of his/her profile on Server1 and now needs to recover the file. (No, it's not in the Recycle Bin anymore.) Using the mountvm command, the VMDK backup of Server1 can be mounted into the file system of the VCB proxy server and the file can be recovered. Figure 10.56 shows the mountvm command used to mount a backup VMDK named scsi0-0-0-server1.vmdk into a mount point named server1_restore_dir off the root of the E drive. Figure 10.56 also shows the Windows Explorer application drilled into the mounted VMDK.�

Figure 10.56 The mountvm command allows a VMDK backup file to be mounted into the VCB proxy server file system, where it can be browsed in search of files or folders to be recovered.

Perhaps one of the best new features of VirtualCenter 2.5 is the integration of the VMware Converter Enterprise. But to add to its benefit, VMware extended the functionality of the VMware Converter to allow it to perform restores of backups that were made using VMware Consolidated Backup. Figure 10.57 shows the VMware Converter Import Wizard option.

Figure 10.58 shows the files that are part of the VCB backup, including the required VMX file that must be referenced. During the import process, shown in Figure 10.59, you will need to provide the UNC path to the VMX file for the virtual machine to be restored.�

Figure 10.57 The VMware Converter Import Wizard greatly simplifies the procedure for restoring VCB backups.

Figure 10.58 The VCB backup files include a VMX file with all the data about the virtual machine.

Figure 10.59 The VMware Converter Import Wizard requires a UNC path that references the VMX file of the virtual machine to be restored.

The examples shown in the previous two figures show the configuration for a backup server named DR1 with a folder that has been shared as MNT. Therefore the appropriate path for the VMX file of the virtual machine to be restored would be \\DR1\MNT\192.168.168.8-fullVM\VAC-DC3.vmx. The remaining steps of the Import Wizard are identical to those outlined in Chapter 7.

This particular feature alone makes VirtualCenter 2.5 an invaluable tool for building a responsive disaster recovery and business continuity plan.

VirtualCenter has many new and updated features for creating and analyzing graphs. Without these graphs, analyzing the performance of a virtual machine would be nearly impossible. Installing agents inside a virtual machine will not provide accurate details about the server's behavior or resource consumption. The reason for this is elementary: a virtual machine is only configured with virtual devices. Only the VMkernel knows the exact amount of resource consumption for any of those devices since it acts as the translator between the virtual hardware and the physical hardware. In most virtual infrastructures, the virtual machines' virtual devices can outnumber the actual physical hardware devices, necessitating complex sharing and scheduling abilities in the VMkernel.

By clicking the Performance tab for a host or virtual machine, you can learn a wealth of information. The default view for either a host or a virtual machine is CPU consumption. But before we analyze the consumption, we need to get to know the performance graphs and legends.

Starting from the top and working our way down, on the top left you'll see the host or virtual machine being monitored. Just below the tabs, the type of chart and its interval appears. The graph above is a real-time graph that shows what has occurred in the last hour. It updates every 20 seconds. You can change the interval and resource being monitored by clicking the Change Chart Options link. At the top right, we see the Refresh icon, the Save icon, and the Tear-off icon. The Refresh icon is self-explanatory, but the next two will bear some explanation a little later.

On each side of the graph are units of measure. In Figure 11.12, the counters selected are measured in Percent and MHz. Depending on the counters chosen, there may be only one unit of measurement, but no more than two. Next, on the horizontal axis, is the Time interval. Below that, the Performance Chart Legend provides color-coded keys to help the user find a specific object or item of interest. This area also breaks down the graph into the object being measured, the measurement being used, the units of measure, and the Latest, Maximum, Minimum, and Average measurements recorded for that object.

Figure 11.12 A Performance graph for a single virtual machine.

Hovering the mouse over the graph at a particular recorded interval of interest displays the data points at that specific moment in time, as shown in Figure 11.13.

Another nice feature of the graphs is the ability to emphasize a specific object so that it is easier to pick out this object from the other objects. By clicking the specific key at the bottom, the key and its color representing a specific object will be emphasized while the other keys and their respective colors become lighter and less visible, as shown in Figure 11.14.

Figure 11.13 Hovering the mouse over a specific data point on the graph will display specific measurements for that particular point in time.

Figure 11.14 By clicking on a specific key in the Performance Chart Legend, that key will be emphasized while the other keys become less visible.

If the current graph does not reveal the data you were looking to find, click Change Chart Options at the top of the Performance chart to open a dialog box, shown in Figure 11.15, which lets you select from among many counters across the various physical hardware components.

On the left, you can choose which resource (CPU, Disk, Memory, Network, or System) to monitor or analyze. By selecting one of these options, you then have a choice of intervals to look at. Real-time will show you what has occurred in the last hour. The others are self-explanatory. For trend analysis, having all of these interval options allows you to choose exactly which interval you need. If these intervals are still not precise enough, you can create a custom interval c by selecting the Custom option under any of the available Chart Options, as shown in Figure 11.16.

Figure 11.15 The Customize Performance Chart dialog box has many options.

Figure 11.16 Setting a custom interval is easy. Here, the user only wants to look at the last six hours of data.

Many times, you want to look at what is happening now or what happened in the last hour for a host or a virtual machine. The Real-time interval gives the best view. This view also gives you access to certain counters for each resource type for a given host or virtual machine that are not available in the other views. If a particular counter is new to you, click on it to highlight the counter. At the bottom of the dialog box, in a section called Rollup, you'll see a description of the counter. For a host, the objects that can be monitored are the host as a whole and the individual physical devices. For the host chart option CPU, the objects and counters breakdown is shown in Figure 11.17.

Figure 11.17 A host's CPU objects and counters.

Viewing Objects and Counters

To get a clear view of all the objects and counters in the Customize Performance Chart dialog box, you have to stretch the dialog box at the top or the bottom to lengthen it. This makes all of the counters visible. Sadly, there is no way to increase the length of the Objects section to see all of them, so scrolling is necessary.�

For a host chart option Disk, the objects and counters are shown in Figure 11.18. The devices being monitored reflect the vmhba paths used by the VMkernel to access the disks associated with the virtual machine. Remember that the vmhba paths that are defined can use any of three types of controllers: a local SCSI controller, a fibre channel, or an iSCSI controller (if the controller is based on the iSCSI software initiator, the device name will be vmhba32).�

No NAS Monitoring Objects�

NAS cannot be monitored because the connection to NAS is not local SCSI, fibre channel, or iSCSI. Even if the ESX Server is configured to use NAS, that storage medium has no object listing or counters since all of the counters seem to be based on block storage only.�

Figure 11.18 A host's Disk objects and counters.

For a host chart option Memory, the objects and counters are shown in Figure 11.19. In this case, there is only one object, the host itself.

For a host chart option of Network, shown in Figure 11.20, notice the objects are the host itself and any vmnic devices used by the VMkernel. If you have a particular virtual switch you want to monitor, find out which vmnic(s) are associated with that vSwitch by switching your view in the VI Client to the Configuration tab of the ESX host and then selecting Networking in the Hardware section.

For a host chart option System, the objects and counters are shown in Figure 11.21. There are several objects for the host with this chart option. This allows the administrator to see how many physical CPU cycles are being consumed by certain processes like vmware-authd or drivers. Be aware that many of the objects may not have any data associated with them.

For a virtual machine, the objects that can be monitored are the virtual machine as a whole and the individual virtual devices. For a virtual machine chart option CPU, the objects and counters breakdown is shown in Figure 11.22. Notice that the virtual device is the virtual processor 0, not the physical processor 0. If the virtual machine was configured with more than one processor, they would increment by one � 0, 1, 2, 3, etc.

Figure 11.19 A host's Memory objects and counters.

Figure 11.20 A host's Network objects and counters.

Figure 11.21 A host's System objects and counters.

Figure 11.22 A virtual machine's CPU objects and counters.

For a virtual machine chart option Disk, the objects and counters breakdown is shown in Figure 11.23. Notice the virtual device is described as vmhba40:0:0, which corresponds to a VMk-ernel storage device, in this case, an iSCSI software initiator.

For a virtual machine chart option Memory, the objects and counters breakdown is shown in Figure 11.24. The only object is the virtual machine itself, but there are several counters.

Figure 11.23 A virtual machine's Disk objects and counters.

Figure 11.24 A virtual machine's Memory objects and counters.

For a virtual machine chart option Network, the objects and counters breakdown is shown in Figure 11.25. Notice that the virtual network device is described as 4000. If the virtual machine had two or more network adapters, they would be incremented by one for each � for example, 4001, 4002, 4003, 4004.

Figure 11.25 A virtual machine's Network objects and counters.

For a virtual machine chart option System, the object is the virtual machine itself, with counters of Heartbeat and Uptime, as shown in Figure 11.26.

Once you have selected the options you need for your graph, click OK.

Figure 11.26 A virtual machine's System objects and counters.

Now that we are back to the graph, the Save icon in the top-right corner saves the graph into an Excel-formatted file. In the Time section of the Export Performance dialog box shown in Figure 11.27, choose the currently displayed interval or a different time frame.

Next, in the Chart Options section, select the type of graph to be saved: Line Graph or Stacked Graph. Figure 11.28 shows an example of a line graph.

Figure 11.27 Saving a Performance graph. Be sure to select the interval and chart options you want. Also, you have a choice of the chart type and size.

Figure 11.28 An example of the line graph for a virtual machine.

A stacked graph of the same data looks as shown in Figure 11.29.

Figure 11.29 An example of a stacked graph for a virtual machine � same data, just a different view.

Stacked graphs are useful when looking at several virtual machines on the same host. When they are stacked on top of one another, it gives an aggregate usage, with colors to distinguish between them. This allows you to identify the busiest virtual machine or the one that represents the largest load on the server. Also, you can click the Advanced button to select or deselect specific counters to be saved as part of the Excel graph.

Here's another nice feature: click the Popup Chart icon to display the graph in a new window. This lets you easily compare this host or virtual machine with another host or virtual machine, as shown in Figure 11.30.

Figure 11.30 An example of the Popup Chart feature.

Virtual machine performance can also be monitored using a command-line tool named esxtop. A great reason to use esxtop is the immediate feedback it gives you after you've made an adjustment to a virtual machine. The esxtop command is not new, but it does include some new features and capabilities. All four resource types (CPU, Disk, Memory, and Network) can be monitored for a particular ESX host.

With the new and improved esxtop we can look at two counters, CPU Used and Ready Time. We can also see these counters in the virtual machine graphs, but with this tool they are calculated as percentages. If virtual machines are running on the same host, we can easily compare the virtual machines as they are identifiable by name, unlike in previous versions. If they run on two different hosts, then a direct comparison is harder without opening two different ssh client sessions. Even then, the virtual machines are not in direct competition, so an apples-to-apples comparison may be difficult. In Figure 11.31, esxtop is monitoring the CPU usage of several virtual machines running on the same host, which is the default view. The amount of processor time being used, %USED, and the amount of time not getting scheduled, %RDY, are listed for each virtual machine. If we increase a virtual machine's reservation or shares, we'll see a corresponding change in these two fields, as %USED will increase its percentage and %RDY will decrease. If you look at one of the other screens for a different resource, just press C to bring it back to the CPU resource screen.

Figure 11.31 The command-line tool esxtop showing CPU statistics for several virtual machines and VMkernel processes.

ESXTOP

Remember, esxtop only shows a single ESX host. In a virtual infrastructure where VMotion, DRS, and HA have been deployed, virtual machines may move around often. Making reservation or share changes while the virtual machine is currently on one ESX Server may not have the desired consequences if the virtual machine is moved to another server and the mix of virtual machines on that server represents different performance loads.�

To monitor memory usage, press M. This gives you real-time statistics about the ESX Server's memory usage in the top portion and the virtual machines' memory usage in the lower section, as shown in Figure 11.32.�

To monitor network statistics about the vmnics, individual virtual machines, or VMkernel ports used for iSCSI, VMotion, and NFS, press N. The columns showing network usage, as shown in Figure 11.33, include packets transmitted and received and megabytes transmitted and received for each vmnic or port. Also shown in the DNAME column are the vSwitches and, to the left,�what is plugged into them, including virtual machines, VMkernel, and service console ports. If a particular virtual machine is monopolizing the vSwitch, you can look at the amount of network traffic on a specific switch and the individual ports to see which virtual machine is the culprit.

Figure 11.32 Using esxtop to display memory usage on a single ESX host.

Figure 11.33 Using esxtop to display network statistics on an ESX Server.

To monitor disk I/O statistics about each of the SCSI controllers, press D. The output of disk I/O monitoring with esxtop is shown in Figure 11.34. The columns on the far left are most often used to determine disk loads. Those columns show loads based on reads and writes per second and megabytes read and written per second. Another important column is the NVMS, which shows how many virtual machines are sharing the same controller. If an application in one of the virtual machines is sluggish, it's easy to see using this column how many other virtual machines it may be competing with.

Figure 11.34 Using esxtop to display controller I/O statistics.

Another great feature of esxtop is the ability to capture performance data for a short period of time and then play back that data. Using the command vm-support, you can set an interval and duration for the capture.

Perform these steps to capture data to be played back on esxtop:

1.�While logged in as root or after switching to root user, change your working directory to /tmp by issuing the command cd /tmp.

2.�Issue the command vm-support -S -i 10 -d 180. This creates an esxtop snapshot, capturing data every ten seconds, for the duration of 180 seconds, as shown in Figure 11.35.

Figure 11.35 Capturing data for esxtop.

3. The resulting file is a tarball and is gzipped. It must be extracted with the command tar -xzf esx*. tgz. This will create a vm-support directory that will be called in the next command.

4. Run esxtop-R /vm-support* to replay the data for analysis, as shown in Figure 11.36.

Figure 11.36 Replaying the data in esxtop.