| Moab Workload Manager® |
17.16 Configuring a Grid with Globus
- 17.16.1 Peer-to-Peer Globus Interfacing Overview
- 17.16.2 Peer-to-Peer Globus Credential Authentication
- 17.16.3 Using Globus Pre-WS GRAM for Job Submission
- 17.16.4 Using Globus GridFTP for Data Staging
- 17.16.5 Interfacing with Globus
17.16.1 Peer-to-Peer Globus Interfacing Overview
Globus is a grid toolkit designed to simplify security authentication, job management, resource discovery, and data staging, among providing other services. Globus does this by using X.509 security credentials and a proxy delegation model to allow a "single sign-on" to access these grid services. (See the Globus homepage for more details about Globus technologies.) Moab may use Globus in the following ways:
- Globus authentication credentials are used to determine trust between Moab peers and/or grid users.
- Trusted peers/users may use the Globus Pre-WS GRAM service to submit grid workload.
- Trusted peers/users may use the Globus GridFTP service to stage data.
NOTE: To use Globus grid services, Moab must be specifically compiled with Globus support enabled. (You will need to use either a globus-enabled binary, or if using a source distribution, build Moab with -D__MGLOBUS and un-comment the necessary Globus libraries.)
NOTE: Before testing Moab with Globus, proper Globus configuration and operation should be verified using a simple acceptance test suite.
NOTE: Moab works with all major releases of Globus including Globus Toolkit 2.x, Globus Toolkit 3.x, and Globus Toolkit 4.x.
NOTE: If using Moab's bi-directional peer-to-peer communication configuration, Globus should be compiled threadsafe.
To Globus or Not To Globus
Moab can enable grids either with or without Globus services. When setting up a new grid that does not already have Globus installed, the question arises regarding whether Globus should be installed. Moab has the ability to enable most or all services provided by Globus using either internal capabilities or by taking advantage of other external services. Moab can enable secure peer-to-peer authentication, can handle job migration, and can use other mechanisms for data staging. However, Globus can also provide these services. The following table provides some of the pros and cons associated with this decision:
| Aspect | With-Globus | Without-Globus |
| Security | Globus security is very good and is widely accepted. It is based on X.509 certificate delegation and provides expiration of authorization, delegation of partial access rights, and other useful features. Globus based security is recommended for grids when connecting systems of widely diverse environments. Globus security allows both authentication and encryption of inter-server messages. | Moab's secret key based security is light-weight, fast, and secure. While not as widely accepted or used as Globus security, it is a good choice for new grids or grids within a controlled environment. Secret key based security allows both authentication and encryption of inter-server messages. |
| Performance | Globus based infrastructure imposes a fair amount of overhead on inter-server communication and job-staging requests. This overhead results in increased message latency. These latencies can be quite pronounced in Globus Toolkit 3.x but is acceptable for reasonably small environments (< 1000 transactions / hour). | Moab's peer-to-peer grid infrastructure imposes a measurably smaller latency hit on inter-server messages resulting in improved response time. |
| Grid Services Configuration/Maintenance | Most Globus installations require a significant investment in training and initial configuration. However, once configured, recent releases of Globus (GT4.x) have required only a small maintenance effort. | Moab peer-to-peer facilities are integrated into Moab and thus require no additional maintenance. |
| Moab Interface Configuration/Maintenance | For Globus based Moab peer-to-peer services, Moab configuration is minimal. | For non-Globus based Moab peer-to-peer services, Moab configuration is minimal. |
| Standards | Globus based standards are widely accepted and are recommended when multi-platform, multi-organizational grids are required and Moab usage is not widely established. | Non-Globus based standards are acceptable but do not have the broad support of Globus standards |
| Job Staging | Globus uses an X.509 certificate based protocol called GRAM which uses the gatekeeper service to enable authenticated remote execution. | Moab can enable direct job staging using internal services, but to do so requires that the Moab server runs as root. |
| Data Staging | Globus uses X.509 based data transfer services such as gridftp and GASS. These tools are reasonably robust, highly secure, and feature rich. | Moab can use non-Globus data transfer services such as SCP, NFS, or FTP. Moab's interface allows virtually any data transfer service to be encapsulated and used for inter-cluster data transfers. Authorization of these transfers typically require protocol-specific authorization methods such as the use of secret keys in SCP's .ssh/authorized_keys file. |
17.16.2 Peer-to-Peer Globus Credential Authentication
As mentioned, Moab peers may use Globus credentials to set up relationships of trust to allow public key based authentication. Globus credentials are used to replace the secret key model explained in Grid Security. Both models cannot be used when communicating with the same peer, but both can be used when a Moab server communicates with different peers.
Setting up Moab to use Globus credential authentication is straightforward, but before Moab can use this service the following conditions must be met:
- Moab must be compiled with Globus support.
- Globus libraries must be properly installed and configured on each Moab head node that will participate in the authentication (both destination and source peers).
- Moab needs ownership of either a valid host credential (if Moab is to run as root) or valid user proxy.
After conditions are met, Moab will automatically load in a Globus credential when started. It will attempt to acquire the credential owned by the user starting it. If running as root, Moab will use the host credential.
Moab must now be configured to use the credential for authentication purposes. If Moab is to use resources on a destination peer using Globus authentication it needs to specify this in the moab-private.cfg file. For example, if moab.cfg defines a destination peer resource manager named destpeer, then the moab-private.cfg could appear as follows:
This indicates that when communicating with the destination peer, the Globus credential loaded at startup will be used to authenticate. Notice also the credential's subject name /O=Grid/OU=GlobusTest/OU=simpleCA-localhost.localdomain/OU=localdomain/CN=DestHost is specified to notify Globus which credential it should accept communication from. (You can determine the credential's subject name by executing mdiag -S on the destination peer.)
Before the destination peer will accept Globus authentication from the source peer, it needs an entry in its moab-private.cfg file:
To view information about Globus credential usage, execute the mdiag -S command. To view information, including possible errors, with peers using Globus credential authentication use the mdiag -R command.
17.16.3 Using Globus Pre-WS GRAM for Job Submission
To enable inter-cluster job migration using Globus's Pre-WS GRAM protocol, the resource manager interface should be configured with the JOBSTAGEMETHOD attribute set to GLOBUS as in the following example:
The mdiag -R command can be used to display and diagnose this configuration.
Next, install the pbsnative Globus job manager bundled with Moab. This needs to be installed on every peer that will receive requests to submit jobs using the Globus GRAM service. The needed files can be found in the tools/globus directory. Edit the globus-job-manager-patch file so that the PBS and GLOBUS variables point to the correct locations. Save the file and then run it as root:
Those using Moab's client commands to access grid resources controlled under Globus authentication may find it necessary to set the Globus destination subject. This is done by setting the MGLOBUSDS environment variable to the appropriate destination subject.
17.16.4 Using Globus GridFTP for Data Staging
To enable inter-cluster data staging using Globus's GridFTP service, you will need to set up a storage manager that uses scripts that use the Globus services. Please see Data Staging for full details.
17.16.5 Interfacing with Globus
Moab can use the Globus gatekeeper service to stage and submit jobs if Moab is built with Globus support and can acquire a valid Globus credential. If gatekeeper is to be used, the source side resource manager job migration method must be set to globus as in the following example:
NOTE: Moab can use gatekeeper services even if non-Globus credential based peer-to-peer server authentication is being used, but will still need a Globus credential to authenticate against gatekeeper.
The mdiag -R -V job [RMID] command can be used to test and diagnose end-to-end job migration regardless of the job staging method configured.