| Moab Workload Manager® |
Virtualization
Virtualization Overview
The types of workload in a cluster or grid environment are often complex. Jobs may not only perform intricate computations, but the required environment for each job may also be complex. In some cases different jobs may require different operating systems, libraries, applications, as well as different memory, disk, and processor resources. If there are even a handful of jobs with such variance in requirements, then an organization will quickly become overwhelmed with the monetary and temporal costs of maintaining compute resources to satisfy all possible requests. A solution to this dilemma is to use modern virtualization technologies.
Virtualization allows a compute node to run one or more virtual machines simultaneously, each with their own distinct operating systems, software, and hardware configurations. These virtual machines (also commonly called virtual domains) can be created and booted on-the-fly from pre-created system images. This is usually much more efficient than actually re-provisioning a node with new software or rebooting the node into a new operating system. Virtualized machines have recently made performance increases that make them only slightly slower than their physical counterparts.
Entire clusters using virtualization, or a subset of compute nodes, can therefore be switched to use a different system configuration/image in seconds, as opposed to minutes or hours. Moab can use this technology to automatically satisfy the complex environment requirements attached to different jobs. Before a job with special requirements starts, Moab can launch and configure virtual domains on the job's allocated resources. The job then runs without incident, oblivious that it is executing on virtualized nodes. When the job finishes the nodes are free to be used again by similar jobs or re-virtualized to another configuration.
Virtualization Prerequisites
Although several virtualization software packages are offered, Moab Workload Manager currently supports the Xen virtual machine monitor. Additionally, Moab currently supports virtualization of nodes running under TORQUE 2.1.0 or higher.
There are several prerequisites to Moab driving virtualization of compute resources in a cluster:
- Each compute node that you want to run virtual images on must have Xen installed and running.
- Each compute node that you want to run virtual images on must be controlled by TORQUE 2.1.0 or higher.
- Existing system images must be configured to use special start-up scripts provided by Cluster Resources.
- Each compute node must allow the cluster head node to log in as a Xen administrator (passwordless).
Configuring Virtualization
After the prerequisites have been met, you can now configure Moab to support virtualization. Moab comes with support scripts in its tools directory that are used to control the virtualization process. By associating these scripts with resource managers in the moab.cfg file, Moab can virtualize nodes owned by those resource managers and even migrate virtual nodes to another cluster using virtual migration.
First, examine the etc/config.xen.pl file. This file controls the configuration of the virtualization scripts; modify this file to fit your particular needs. If you need assistance with this task, please contact your support representative at Cluster Resources.
Next, we add an attribute to the moab.cfg file to activate the virtualization scripts. The RMCFG attribute used to configure the activation is NODEVIRTUALIZEURL. Add this attribute to every RMCFG defining resources managers that control virtualization capable nodes. The tools/node.virtualize.xen.pl script should be the value of this attribute.
Example
In addition to the NODEVIRTUALIZEURL, an additional resource manager should be defined that imports information about the available Xen images on each of the nodes. A simple script, tools/node.mon.xen.pl is bundled with Moab as an example of how one might discover available images using a native resource manager interface. (The bundled script does not scale well as the number of nodes in a cluster grows--it is only provided as an example. We recommend customizing this script to your site's needs in order to better increase its performance.) Using the node.mon.xen.pl script is straightforward. Simply change your moab.cfg as follows:
Example
When a virtual machine is created on an existing node, the virtual machine is given a unique hostname. The hostname is simply a variant of the physical node's name. How the virtual node is named is configured with two parameters: VIRTUALNODEPREFIX and VIRTUALNODESUFFIX. The values of these parameters is prepended or appended, respectively, to the physical node name in order to create the new hostname for the virtual node. For example, if you want all virtual nodes to have the prefix "v_," add a VIRTUALNODEPREFIX line to the moab.cfg file:
Note: The virtual node hostnames MUST be resolvable to valid IP addresses.
If you virtualize the physical compute node "node09," the new virtual machine (virtual child) running on "node09" would have the hostname "v_node09" and would have the IP address that resolves from "v_node09."
After making these changes to the moab.cfg file, restart/recycle Moab for the changes to take effect.
Virtualizing a Node
After meeting prerequisites and configuring Moab, you are ready to virtualize a node. The mnodectl command is used to create a new virtual node (virtual child) on one of the existing physical nodes. In addition to specifying which node the virtual machine should be started on, users also need to specify which image should be used to start the virtual machine. A list of the available images on a node can be determined by using the checknode -v command. Once the proper image has been located, you can virtualize a node using the command syntax mnodectl -V image=<IMAGE_NAME> <NODE_EXP>.
For example, if you want to create a new virtual child on the node "node09" using a Debian Sarge image, issue the following command:
You can see the progress of the node virtualization by running checknode -v on the physical node:
After the new virtual node has been created, its characteristics (such as memory and processor count) are reported through the physical node. So in our example, "node09" will take on the characteristics of its virtual child "v_node09." Moab recognizes that a reference to either "node09" or "j_node09" means the virtual node running on "node09." This allows users to submit jobs to either "node09" or "v_node09" and have confidence the jobs will successfully run on the new virtual node.
[an error occurred while processing this directive]See Also