| Moab Workload Manager® for Grids |
17.6 Resource Control and Access
- 17.6.1 Resource Control Overview
- 17.6.2 Controlling Resource Information
- 17.6.2.1 Direct Node View
- 17.6.2.2 Mapped Node View
- 17.6.2.3 Grid Resource View
- 17.6.2.4 Managing Queue Visibility over the Grid
- 17.6.3 Managing Resources with Grid Sandboxes
17.6.1 Resource Control Overview
Moab allows compute resources to be reported to peers in several formats:
- full resource information - nodes appear with complete remote hostnames and full attribute information
- remapped resource information - nodes appear with remapped local hostnames and full attribute information
- grid mode - information regarding nodes reported from a remote peer is aggregated and transformed into one or more SMP-like large pseudo nodes
The sections that follow describe how to configure how resource information is provided on both the source and destination side of the peer interface.
17.6.2 Controlling Resource Information
In a Moab peer-to-peer grid, resources can be viewed in one of three models:
- Direct - nodes are reported to remote clusters exactly as they appear in the local cluster
- Mapped - nodes are reported as individual nodes, but node names are mapped to a unique name when imported into the remote cluster
- Grid - node information is aggregated into a single large SMP-like pseudo-node before it is reported to the remote cluster
17.6.2.1 Direct Node View
Direct node import is the default resource information mode. No additional configuration is required to enable this mode.
17.6.2.2 Mapped Node View
In this mode, nodes are reported just as they appear locally by the exporting cluster. However, on the importing cluster side, Moab maps the specified node names using the resource manager object map. In an object map, node mapping is specified using the node keyword as in the following example:
In this example, all nodes reported by clusterB will be have the string 'b_' prepended to prevent node name space conflicts with nodes from other clusters. For example, if cluster clusterB reported the nodes node01, node02, and node03, cluster gridmaster would report them as b_node01, b_node02, and b_node03.
See object mapping for more information on creating an object map file.
17.6.2.3 Grid Resource View
The final method for displaying resources is to use the grid resource view. This view is enabled by setting the resource manager attribute FLAGS to grid. This flag will be enforced if set on either the source or destination side of a peer relationship.
If set, Moab will report all resources available to a peer in the form of a single SMP-like pseudo-node. This node will encompass the total configured, available, and dedicated resources of the reporting cluster and will be given the same name as the reporting cluster.
17.6.2.4 Managing Queue Visibility over the Grid
Queue information and access can be managed directly using the RMLIST attribute. This attribute can contain either a comma delimited list of resource managers which can view the queue or, if specified with a '!' (exclamation point) character, a list of resource managers which cannot view or access the queue. The example below highlights the use of RMLIST.
NOTE: If more advanced queue access/visibility management is required, consider using the resource manager object map feature.
17.6.3 Managing Resources with Grid Sandboxes
A cluster may wish to participate in a grid but may desire to dedicate only a set amount of resources to external grid workload or may only want certain peers to have access to particular sets of resources. With Moab, this can be achieved by way of a grid sandbox which must be configured at the destination cluster. Grid sandboxes can both constrain external resource access and limit which resources are reported to other peers. This allows a cluster to only report a defined subset of its total resources to source peers and restricts peer workload to the sandbox. The sandbox can be set aside for peer use exclusively, or can allow local workload to also run inside of it. Through the use of multiple, possibly overlapping grid sandboxes, a site may fully control resource availability on a per peer basis.
A grid sandbox is created by configuring a standing reservation on a destination peer and then specifying the ALLOWGRID flag on that reservation. This flag tells the Moab destination peer to treat the standing reservation as a grid sandbox, and, by default, only the resources in the sandbox are visible to grid peers. Also, the sandbox only allows workload from other peers to run on the contained resources.
Example 1: Dedicated Grid Sandbox
In the above example, the standing reservation sandbox1 creates a grid sandbox which always exists and contains the nodes node01, node02, and node03. This sandbox will only allow grid workload to run within it by default. This means that the scheduler will not consider the boxed resources for local workload.
Grid sandboxes inherit all of the same power and flexibility that standing reservations have. See Managing Reservations for additional information.
NOTE: The flag ALLOWGRID marks the reservation as a grid sandbox and as such, it precludes grid jobs from running anywhere else. However, it does not enable access to the reserved resources. The CLUSTERLIST attribute in the above example enables access to all remote jobs.
17.6.3.1 Controlling Access on a Per Cluster Basis
Often clusters may wish to control which peers are allowed to use certain sandboxes. For example, Cluster A may have a special contract with Cluster B and will let overflow workload from Cluster B run on 50% of its resources. A third peer in the grid, Cluster C, doesn't have the same contractual agreement, and is only allowed 10% of Cluster A at any given time. Thus two separate sandboxes must be made to accommodate the different policies.
The above sample configuration illustrates how cluster A could setup their sandboxes to follow a more complicated policy. In this policy, sandbox1 provides exclusive access to nodes 1 through 5 to jobs coming from peer ClusterB by including CLUSTERLIST=ClusterB in the definition. Reservation sandbox2 provides shared access to node6 to local jobs and to jobs from clusters B, C, and D through use of the CLUSTERLIST and USERLIST attributes.
With this setup, the following policies are enforced:
- local jobs may see all nodes and run anywhere except nodes 1 through 5
- jobs from cluster B may see and run only on nodes 1 through 6
- jobs from clusters C and D may see and run only on node 6
As shown in the example above, sandboxes can be shared across multiple peers by listing all sharing peers in the CLUSTERLIST attribute (comma delimited).
17.6.3.2 Access Control Lists/Granting Access to Local Jobs
It is not always desirable to have the grid sandbox reserve resources for grid consumption, exclusively. Many clusters may want to use the grid sandbox when local workload is high and demand from the grid is relatively low. Clusters may also wish to further restrict what kind of grid workload can run in a sandbox. This fine-grained control can be achieved by attaching access control lists (ACLs) to grid sandboxes.
Since sandboxes are basically special standing reservations, the syntax and rules for specifying an ACL is identical to those found in Managing Reservations.
Example
In the example above, a cluster decides to dedicate resources to a sandbox, but wishes local workload to also run within it. An additional ACL is then associated with the definition. The reservation 'sandbox2', takes advantage of this feature by allowing local jobs running with a QOS of 'high', or under the group 'engineer', to also run on the sandboxed nodes node04, node05, and node06.
17.6.3.3 Limiting Access To Peers (Source Side Limits)
In some cases, a site may want to constrain which users, accounts, queues or QOS's can utilize remote resources. Perhaps only certain users are trusted to participate in a special beta program or possibly only jobs in a certain queue will be able to find the needed applications or environment on the remote side.
Regardless of purpose, peer-to-peer job migration can be controlled on the source side using the RMCFG parameter's AUTHALIST, AUTHCLIST, AUTHGLIST, AUTHQLIST, and AUTHULIST attributes. These attributes are comma delimited and constrain who can utilize resources within the peer resource manager regardless of what authorization is allowed by the resource manager itself. Thus, even if a resource manager reports that it will accept jobs from any user, if the AUTHULIST parameter is set to steve,bob, then only jobs from those two users will be allowed to migrate to the peer cluster. If more than one authorized credential type is specified, jobs which satisfy any of the listed credentials will we allowed to use the resources.
17.6.3.4 Limiting Access From Peers (Destination Side Limits)
While source limits are set up on the source side of an interface and constrain which users can access remote resources, destination limits are set up on the destination side of an interface and constrain which remote workload requests will be accepted. These limits are useful when the remote peer is not under full local administrative control or can otherwise not be fully trusted. Often a remote source peer may allow unfettered access to peer resources while the destination may want to limit which jobs are allowed in locally.
Destination side credential limits are configured exactly like source side limits but are configured on the destination side of the interface. As with source side peer limits, these limits are enabled using the RMCFG parameter's AUTHALIST, AUTHCLIST, AUTHGLIST, AUTHQLIST, and AUTHULIST attributes. These attributes are comma delimited and constrain which remote peer jobs can utilize local resources within the peer resource manager regardless of what authorization is allowed by the remote source resource manager itself.