Summary
The visual cluster gives an easy and concise way of viewing your entire cluster and the status of each node. The table and explanation below explain how to interpret the visual cluster:
Table 3-1. Visual Cluster Example
| Slot 1 | Slot 2 | Slot 3 | |
|---|---|---|---|
| Rack 1 | Node A | Node D | |
| Rack 2 | Node B | ||
| Rack 3 | Node C |
A rack is a physical frame that holds a node. The slot is the location of the node inside the rack. The racks make up the first column of the table. The slot locations increase from left to right. For example, Node A is located on Rack 1 in Slot 1. Node D is also located on Rack 1 but instead of Slot 1 it's located in Slot 3. In the visual cluster Node A through Node D are displayed as icons. The different icons can represent node state, node attributes, reservations, jobs, and/or nodes. The subpanel sections below describe these states in more detail. Further information can be gathered about nodes by hovering the mouse over any nodes.
It should be noted that the visual cluster is for display purposes only and the location of the node does not play any part in how Workload Manager schedules.
The node attribute selector gives the user the power to see various attributes of the nodes displayed in the Visual Cluster. This allows the user to compare and contrast attributes of interest. Node attributes include standard categories such as architecture, OS, hardware metrics (memory, disk, swap, etc.), as well as any metric read in through Moab as a generic metric (a node's GMETRIC). The default display for node attributes is the corresponding color of the outer rim of each node cell. This can be changed in the section titled "Node Display Preferences".
The "Clear Attribute" button will simply clear any selection and node attribute displayed currently. The "Graph Attributes" button will display each node attribute based on state, current load, or historical load. This is tied to the value currently selected in the "Node Usage Display Options".
Once a node attribute is selected, Moab Cluster Manager will determine the number of nodes and processors that describe each attribute and will display a corresponding key value that matches the Visual Cluster by color. Each attribute's display can be individually controlled via the check box next to each node attribute name and color.
If the node attribute is a numerical value - for example, a generic metric - then Moab Cluster Manager will attempt to place the values into a reasonable range as to effectively categorize the values.
This section provides a user with the option of highlighting resources in the visual cluster table. The three sections are divided into reservations, jobs, and nodes. Each section can be simultaneously displayed by having each border being a different color. The white box displays the names of the selected resources with the headers "Res.: ", "Job: ", and "Node: " respectively. The Select buttons open the lists of reservations, list jobs, or list nodes window depending on which resource the user has selected. The visual cluster window will appear with the desired resources highlighted. The Clear buttons remove the highlight from the visual cluster table and erases the names from the colored box. The Color button changes the highlight color for the specified resource. The new highlight color will be displayed in the colored box.
The Node Display Preferences window can be accessed from the Display > Preferences menu. There are a variety of options that can be used to change how the visual cluster displays its information. This information will be saved for future use.
There are different options depending on how the three checkboxes - "Hide Usage", "Hide Attributes", and "Auto Resize" - are set:
Usage unchecked, attributes unchecked (DEFAULT) - Usage will be displayed on the inside of the cell and attributes will be displayed on the outside border of the cell.
Usage checked, attributes unchecked - Usage will not be displayed, and attributes will take up the entire node cell.
Usage unchecked, attributes checked - Usage will take up the entire node cell, and attributes will not be displayed.
Usage checked, attributes checked - No information will be displayed, leaving each cell grayed out.
Auto resize checked - The table of nodes will try to fix to the size of the window given. If there are more nodes than can fit on the window, a minimum size will be set with the rest of the nodes dangling off the window.
Auto resize unchecked - The table of nodes will always be set to a minimum size. However, a horizontal scroll bar will be set if the nodes dangle off the side of the window.
The "Node Height" slider changes the height of each row to grow or shrink the nodes to fit the user's display needs. The "Numerical Range Count" specifies the number of groups to create when attempting to display node attributes' numerical values in sliced up ranges.
The highlighting of reservations, jobs, and nodes remain unaffected regardless of what node display options are set.
Note: This menu is also accessible by right clicking anywhere on the main window.
Actions Menu Options
Online Node - This option will change a node's status from unavailable to available. An online node is available for jobs to execute on it.
Offline Node - This option will change a node's status from available to unavailable. An offline node is unavailable for jobs.
Reserve Selected Nodes - This option will prepopulate the desired nodes in a create reservation window with the nodes that were selected using the mouse.
Reserve Highlighted Nodes - This option will prepopulate the desired nodes in a create reservation window with the nodes that were selected using the Node Attribute Selector.
Modify Nodes - This option will open a modify node(s) window that will allow the administrator to modify one selected node or perform group operations over numerous selected nodes.
Power On Nodes - This option will change the power status of the selected nodes to ON. To take advantage of this command, CLUSTERQUERYURL and NODEPOWERURL must be set up to handle xCAT or IPMI interfaces.
Power Off Nodes - This option will change the power status of the selected nodes to OFF. To take advantage of this command, CLUSTERQUERYURL and NODEPOWERURL must be set up to handle xCAT or IPMI interfaces.
Reboot Nodes - This option will change the power status of the selected nodes to REBOOT. To take advantage of this command, CLUSTERQUERYURL and NODEPOWERURL must be set up to handle xCAT or IPMI interfaces.
Highlight Menu Options
Highlight Jobs for Selected Nodes - This will get the name of the jobs from the selected node and highlight all the nodes that that job is on.
Highlight Reservations for Selected Nodes - This will get the name of the reservations from the selected node and highlight all the nodes the reservation is on.
Select Nodes with Credential - This will select each node running a job with the selected credential.
Display Menu Options
View Processor Usage - This will open a new window that displays processor usage.
The default display for usage breakdown is the inner core of the node cell. This can be changed in the section Node Display Options.
There are three options for displaying usage breakdown:
Display Node State
Display Current Load
Display Historical Load
Display node state displays the state the node is in according to the Workload Manager.
Down - The node is currently reporting a state of "Down" because of failure or administrative action.
Full Load - The node is currently reporting a state of "Busy".
Partial Load - The node is currently reporting a state of "Running".
Unused - This is currently unused by node state.
Offline - The node is currently reporting a state of "Offline." This is also the default sate when the state is not recognized.
Idle - The node is currently reporting a state of "Idle".
Display historical load displays the average percentage over time that the node has been used.
> 100 % - The node is currently executing more executables than it has processors
80% - 100% - The node is currently executing executables on between 80 and 100 percent of its processors.
60% - 80% - The node is currently executing executables on between 60 and 80 percent of its processors.
40% - 60% - The node is currently executing executables on between 40 and 60 percent of its processors.
20% - 40% - The node is currently executing executables on between 20 and 40 percent of its processors.
0% - 20% - The node is currently executing executables on between 0 and 20 percent of its processors.
Display historical load displays the average percentage over time that the node has been used.
> 100 % - The node has historically executed more executables than it has processors
80% - 100% - The node has historically executed executables on between 80 and 100 percent of its processors.
60% - 80% - The node has historically executed executables on between 60 and 80 percent of its processors.
40% - 60% - The node has historically executed executables on between 40 and 60 percent of its processors.
20% - 40% - The node has historically executed executables on between 20 and 40 percent of its processors.
0% - 20% - The node has historically executed executables on between 0 and 20 percent of its processors.
Summary
This graph displays how the cluster's processors are being used over time. The left bar, or y-axis, displays the number of processors. The bottom bar, or x-axis, displays time. The light yellow color displays the total available processors on the cluster. The dark yellow color displays the processors used by jobs and job reservations. The blue color displays the processors used by reservations other than job reservations.
The switch statstics option allows for "Available Processors" and "Jobs Reservations" colors to be switched.
Summary
The events calendar helps you view the events you are interested in while filtering out those you are not interested in. This can be much more convenient than searching through logs and event files. Once you have finished filtering, you can view the details of the remaining events and correlate this with your nodes' state history.
Filtering
The first filter you should probably apply is the time filter. After setting a start and end time, you can then select what type of events you wish to see. Use the tree control at the top left of this window to click the event types and/or subtypes you desire. For example, you might choose to see all events, all job events, or job end events only. After making your selection, click the "Apply Filter" button.
If you are looking for events for a specific object, and you know the object's name, you can filter out all other events by specifying the object type and id in the fields at the lower left. For example, you might wish to see only events relating to the job "moab.2". To do this, you would specify an object type of "Job" and an object ID of "moab.2".
Viewing Event Details
To view the details of desired events, simply click their colored icon representations in the timeline. Note that on busy schedulers, an icon may represent multiple events. You will be able to see the event type, id, time, object type, object id, and any messages from the scheduler pertaining to this event.
The Node State/Node Category Chart
Below the timeline is a colored line chart. Vertically below any point on the timeline one can see the percentage of nodes that were active, idle, down, etc. This can be useful in determining the events you might want to filter. For example, if you see that 30% of your nodes suddenly went idle, yet you know there was a large backlog during that time, you might want to view all reservation start events. This might help you find an unused reservation responsible for the idle nodes.
Summary
As the name suggests a resource manager manages compute resources. Different resource managers manage different resources. Possible resources are hardware, software licenses, storage, networks, or compute cycles.
Resource Manager Type - This field displays the type of resource manager interface that is being enabled.
Name - This field displays the unique resource manager name.
Description - This field displays a description of what the resource manager does.
Port - This field allows an administrator to select the port on which Workload Manager will communicate with this resource manager.
Server URL - This field allows an administrator to input the URL of the resource manager. A URL must be entered in one of the following formats:
File://[File Path] - This field requires a file that acts as a resource manager. For example, if a file called rmfile.txt were located in the tmp directory, then the format would be File://tmp/rmfile.txt
http://[address] = This field requires the web address of the resource manager. For example, if the resource manager were located at 10.10.10.100 then the format would be http://10.10.10.100
[PATH]/executable This field requires an executable. For example, if the resource manager were rm.sh, located in the tmp directory, then the format would be /tmp/rm.sh
Summary
As the name suggests a resource manager manages compute resources. Different resource managers manage different resources. Possible resources are hardware, software licenses, storage, networks, or compute cycles.
Resource Manager Name - This field displays the custom name given to the resource manager by the system administrator.
Resource Manager Type - This field displays the type of resource manager interface enabled.
Resource Manager State - This field displays the status of the resource manager. Possible states include active, idle, ordown.
Resource Manager Type - This field allows an adminstrator to change the resource manager interface.
Server URL - This field allows an administrator to input the URL of the resource manager. A URL must be entered in one of the following formats:
File://[File Path] - This field requires a file that acts as a resource manager. For example, if a file called rmfile.txt were located in the tmp directory, then the format would be File://tmp/rmfile.txt
http://[address] = This field requires the web address of the resource manager. For example, if the resource manager were located at 10.10.10.100 then the format would be http://10.10.10.100
[PATH]/executable This field requires an executable. For example, if the resource manager were rm.sh, located in the tmp directory, then the format would be /tmp/rm.sh
Name - This field allows an administrator to change the current resource manager name given to this resource manager interface.
Port - This field allows an administrator to select the port on which Workload Manager will communicate with this resource manager.
State - This field displays the current state of the resource manager interface.
Total Requests - This field displays the total number of communications that have occurred between Workload Manager and the resource manager.
Response Time (In Seconds) - This bar graph displays the average response time, as well as the maximum response time between Workload Manager and the resource manager. This information often provides valuable diagnostic information when resource manager errors are occurring.
Summary
Resource managers have the ability to report diagnostic messages and user specified messages. These messages can be used to gain further information or knowledge about a particular resource manager. This may be useful in trying to diagnose failures associated with the resource manager.
Resource managers' messages are divided into three categories: a diagnostic message, other messages, and peer service interface messages. All message types are described in greater detail below.
The first field in the resource manager messages frame is the diagnostic message. This diagnostic message reports any problems that Moab may see with the resource manager configuration. Examples include missing resource manager parameters or parameters that are malformed.
The second field is table of messages attached to the resource manager itself. These messages may be user specified messages that describe notes about the resource manager. They may also be generalized system messages Moab generates that summarize issues going on with the resource manager itself. The order that messages are appear are from oldest to newest.
The third field is also a table of messages, but it reports very specific information concerning the resource manager's peer service interface. This is the module inside the resource manager that is responsible for communicating with Moab and other resource managers. PSI messages consist of three parts:
Type - This is the type of failure reported by the message. Some types include "clusterquery", "workloadquery", or "rminitialize".
Time - This is the reported time of the message.
Message - This is the actual messsage text itself.
Summary
An allocation manager functions much like a bank in that it provides a form of currency which allows jobs to run on a cluster. Each job on the cluster requires a certain number of credits to be eligible to execute. An allocation manager tracks the used credits and notifies Workload Manager of any jobs that would exceed their credit limit.
Name - This field allows an administrator to define a name for the Allocation Manager.
Hostname - This field allows an administrator to input the URL of the resource manager. A URL must be entered in one of the following formats:
File://[File Path] - This field requires a file that acts as a resource manager. For example, if a file called rmfile.txt were located in the tmp directory, then the format would be File://tmp/rmfile.txt
http://[address] = This field requires the web address of the resource manager. For example, if the resource manager were located at 10.10.10.100 then the format would be http://10.10.10.100
[PATH]/executable This field requires an executable. For example, if the resource manager were rm.sh, located in the tmp directory, then the format would be /tmp/rm.sh
Port - This field allows an administrator to select the port on which Workload Manager will communicate with this allocation manager.
Timeout - This field allows an administrator to define how long Workload Manager will wait for the Allocation Manager to respond to messages.
Type - This field allows an administrator to define which allocation manager type is being used. The following options are available.
Gold
GGF
Qbank
ResD
File
Allocation Failure Job Action - This field allows an administrator to define what should happen to a job if an allocation manager failure is detected. The following options are available.
Log Failure
Reattempt
Wire Protocol - This field allows an administrator to define which wire protocol will be used by Workload Manager to communicate with the Allocation Manager. The following options are available.
Default
HTML
XML
SSS2
Socket Protocol - This field allows an administrator to define which socket protocol will be used by Workload Manager to communicate with the Allocation Manager. The following options are available.
HTTP
SSS-HALF
SSS-Challenge
Secret Key - This field allows an administrator to encrypt communication between the allocation manager and Cluster Manager using a secret key.
Append Machine Name - If this field is enabled, Cluster Manager will append the machine name to each account before submitting debits to the allocation manager. This will create unique charges per machine name.
Charge Rate Policy - This field allows an administrator to define how charging per job occurs. The following options are available.
DebitAllWC - This option will debit from the allocation manager according to the time used on the cluster.
DebitAllCPU - This option will debit from the allocation manager according to how many processors are used and for how long the processors are used.
DebitAllPE - This option will debit from the allocation manager according to processor equivalent [1] seconds.
DebitSuccessfulWC - This option will debit from the allocation manager when a job successfully completes execution according to the amount of time used on the cluster.
DebitSuccussfulCPU - This option will debit from the allocation manager when a job successfully completes execution according to how many processors are used and for how long the processors are used.
DebitAllPE - This option will debit from the allocation manager when a job successfully completes execution according to processor equivalent seconds.
Flush Interval - This field allows an administrator to define how long Workload Manager will wait before contacting the allocation manager.
Fall Back Account - This field allows an administrator to define a second account jobs can use if their allocation manager account doesn't have adequate resources to allow the job to start executing. If the second account isn't defined or doesn't have adequate resources, the job is then placed on hold.
Assign / Modify Fixed Allocations - This field opens the List Credentials window where throttling policies can be set. The throttling policies can be used to create fixed or unchanging restrictions on a credential.
Assign / Modify Rolling Allocations - This field opens the Fairshare window where fairshare targets can be set. The fairshare window can be used to create rolling or interval-based restrictions on a credential.
| [1] |
Processor equivalence is a relative measure of how much of a node is taken by a job, even if only one type of node resource is requested. For example, if a job requires 1 processor and 1GB of memory, and it is running on a 4 processor node with 1GB of memory, the PE of the job is 4. All of the processors are considered to be taken because the first job is using all of the memory, which prevents any other job from running on that node. |