[torqueusers] kerberos tickets

Mike Coyne Mike.Coyne at PACCAR.com
Thu Jun 10 08:36:38 MDT 2010

Attached is a compressed  diff between the gssapi svn archive at about
3651 and my working version of "torque_gss" I just finished updating my
copy of the repository and generated the patch against my local archive
using globus's makepatch utility. My update didn't go as well as I had
hoped so use a "weather eye". And I may have missed the most recent

The intent of the patch is to accept either globus jobs and credentials
via gram or directly via qsub and use passthough authentication to the
compute node or nodes Or < still working on the and > accept Kerberos
credentials via qsub and use passthough authentication to the compute
node or nodes. Also with consideration for the use of AFS home
directories on both the submit and compute nodes or workstations. 

I tried to implement the AFS setpag function within autoconfig options
so the correct afs libsys or equivalent libary  could be specified by
the user.   The configureation also checks for the GLOBUS_LOCATION
environment variable to be set for building against globus.

Below are part of some the notes I put together in contrib/gsiapi for
configuration options. This is still a work in progress for me, care and
feeding is still needed.  A better Clean up of old creds is needed after
mom sends stdout and stderr back to the user on the computenodes as well
as maybe reworking the naming scheme for Kerberos users to be the same a
globus/pbs_iff users so the gssapi-mech library could be used to select
the correct gssapi library gss/gsi based on the incoming credential ie
globus AND Kerberos. 

Thank you in advance for your time and patience in reviewing my humble

Licensing , 
	all patches are released under the license of the underling
packages , 
	any new material is release under the gnu gpl 2.0 or higher in
the hopes that it may be use.

mcoyne at paccar.com

Environment variables
kerberos specific

AKLOG points to exe location of openafs aklog 
PBS_DISABLE_STRICT_ACCEPTOR_CHECK set to accept any kerberos
host/tickets for mulitple interfaces

globus specific

MYPROXY_SERVER  host name for myproxy server to renew gsi tickets 

GKLOG_OPTS=-silent,-server,mygklogd.server.net comma seperated list of
gssklog options for grenew 
GKLOG points to exe location of gssklog

Mom options
$babysitter /opt/torque_alpha/sbin/grenew 
	program to exec the pbs jobshell and monitor and renew
credentials see krenew/grenew

$gssklog_prog /usr/bin/gssklog -silent -server mygklogd.server.net
	program to get a afs ticket from a globus credential

$aklog_prog /usr/bin/aklog 
	program to get a afs ticket from a kerberos credential

Configureation options 
you will need 
	when enabling either gssapi authenications other methods should
be disabled , ie pbs_iff 
works only for the pbs_server with the -t create option so root can
configure the queues, 
otherwise the credentials may not get forwarded to mom for the user and
the job will not start as server
authenicates to mom via the users credentials . 

for kerberos 
--with-afs-libsys=<full path to libsys.a> 
--with-gssapi=<path to head or gssapi install point>

for globus 
	when compiling for globus make shure to set GLOBUS_LOCATION
source in the globus-user-env.<sh/csh>
and have gsiscp in your seachpath.

--with-afs-libsys=<full path to libsys.a> 
--with-flavor=<globus flavor ie gcc64dbg>

additional sources

wget http://archives.eyrie.org/software/kerberos/kstart-3.16.tar.gz


Contrib patch notes

for Globus 5.0.1 the source patch will set the pag , get a afstoken and
redirect the Gatekeepers use of the user home directory to
/var/spool/globus/<username> for the purposes of launching the job, the
Gram-pbs job launcher which submits the job resets the home environment
variable to the users real home. As the gt5 gram will not work in with a
afs $HOME/.globus/jobs directory due to file locking requirements. The
gram fork module will not switch back to real home directory so it
launches  exec's under the /var/spool/globus/<username>. The
globus-gridftp-server also will set the uses pag and get a afstoken so
globus-url-copy etc can interoperate with afs home directorys.

for grenew and krenew, the patch creates a "special'ish" verison of
krenew ,and a globus api version called grenew. What is different is
both are setup to get the configureation options from the enviornment if
it called in a torque job. They also  add some randomness to the lenght
of time bewteen renew checks so large jobs with many parallel processes
wont all try to renew at the same time.  

-----Original Message-----
From: torqueusers-bounces at supercluster.org
[mailto:torqueusers-bounces at supercluster.org] On Behalf Of Garrick
Sent: Wednesday, June 09, 2010 4:14 PM
To: Torque Users Mailing List
Subject: Re: [torqueusers] kerberos tickets

Yup, here (or preferably torquedev) is a fine place for patches.

But I'm not sure we really have someone qualified to review gssapi
really need someone to stand up and take ownership of the branch again.

Who wants the job? Who wants the glory? I can't promise sex, money, or
but I can promise that kerberos tickets will flow like honey.

On Wed, Jun 09, 2010 at 04:01:40PM -0500, Mike Coyne alleged:
> Attached is a diff which I have used to deal with gssapi tickets on
> non-mother superior compute nodes for mpi runs using the TM interface.
> Basicly the pbs server uses the users credentials to auth to the
> superior compute node, mom then saves a copy of the credential on that
> node and starts the job. In order to use the TM interface on other
> you need a ticket to get you "afs token" and access you home directory
> (in my case)>I made use of Boings node check routing to inflict a
> authenaction check on each of the propective nodes to verify the user
> valid on the node and as a side effect to save a copy of there creds
> the other nodes. The other piece to the puzzle is for the other moms
> use the credential to set the users pag and get there token unlocking
> there home directory prior to running the requested script/command.
> the "new" alpha torque from what I understand the mom-mom rpp has
> changed from udp -> tcp? That would really make this much cleaner. 
> I would like to give back to the group the work I have put in thus far
> on this , my efforts have been revolving around getting either
> gssapi or Globus gssapi to function with torque.  Should I submit it
> this group of is there a better place / means to send the patch?
> Mike
> -----Original Message-----
> From: torqueusers-bounces at supercluster.org
> [mailto:torqueusers-bounces at supercluster.org] On Behalf Of Mike Coyne
> Sent: Wednesday, June 09, 2010 7:13 AM
> To: Torque Users Mailing List
> Subject: Re: [torqueusers] kerberos tickets
> The --with-gssapi path is to the install path for your gssapi , what
> is looking for is <gssapi-path>/bin/krb5-config , for instance on say
> redhat linux it would be /usr/kerberos ... 
> -----Original Message-----
> From: torqueusers-bounces at supercluster.org
> [mailto:torqueusers-bounces at supercluster.org] On Behalf Of Andreas
> Davour
> Sent: Wednesday, June 09, 2010 6:31 AM
> To: torqueusers at supercluster.org
> Subject: Re: [torqueusers] kerberos tickets
> On Saturday 05 June 2010 00:09:19 Garrick Staples wrote:
> > On Wed, Jun 02, 2010 at 04:42:16PM +0200, Andreas Davour alleged:
> > > Hi
> > >
> > > I wonder if someone here have set up torque to forward kerberos
> tickets
> > > to the submit host (after doing qmgr -c "set server submit_hosts =
> > > submithost"), and finally to the worker nodes?
> > >
> > > Is it done automatically via rsh -F or suchlike (and will that
> happen on
> > > submithost as well?) or do I have to tell torque about it?
> > >
> > > Feel free to point me to relevant sections of the documentation.
> > 
> > svn://svn.clusterresources.com/torque/branches/gssapi
> So what am I supposed to give as an argument to the configure flag
> --with-
> gssapi=PATH?? The path to what?
> /andreas
> -- 
> Systems Engineer
> PDC Center for High Performance Computing
> CSC School of Computer Science and Communication
> KTH Royal Institute of Technology
> SE-100 44 Stockholm, Sweden
> Phone: 087906658
> "A satellite, an earring, and a dust bunny are what made America
> _______________________________________________
> torqueusers mailing list
> torqueusers at supercluster.org
> http://www.supercluster.org/mailman/listinfo/torqueusers
> _______________________________________________
> torqueusers mailing list
> torqueusers at supercluster.org
> http://www.supercluster.org/mailman/listinfo/torqueusers

Garrick Staples, GNU/Linux HPCC SysAdmin
University of Southern California

Life is Good!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: gssapi_3651_torque_gss.makepatch.gz
Type: application/x-gzip
Size: 1315834 bytes
Desc: gssapi_3651_torque_gss.makepatch.gz
Url : http://www.supercluster.org/pipermail/torqueusers/attachments/20100610/d8f409fa/attachment-0001.gz 

More information about the torqueusers mailing list