[torquedev] [torqueusers] Performance of non-GPU codes on GPU nodes reduced by nvidia-smi overhead

[Doug Johnson]

Hi David,

I'm in the process of devising a cleaner way to do what I did in my
previous quick and dirty patch.  I'm working out of the torque-2.5
branch.  I'd be very interested in working with whoever gets assigned
this problem at Adaptive.

And as an update, I've executed many jobs on the GPU and non-GPU nodes
with the patch referenced below.  Same mom binary with nvml support on
both types of nodes with no apparent problems.

Doug


At Wed, 22 Feb 2012 13:53:16 -0700,
David Beer wrote:
> 
> [1  <multipart/alternative (7bit)>]
> [1.1  <text/plain; ISO-8859-1 (7bit)>]
> 
> [1.2  <text/html; ISO-8859-1 (quoted-printable)>]
> Doug,
> 
> This is now officially in the list of things to do, and I'll keep you updated on it. That may sound
> terrible, but it should be done quickly.
> 
> David
> 
> On Fri, Feb 17, 2012 at 3:12 PM, Doug Johnson <djohnson at osc.edu> wrote:
> 
>     At Fri, 17 Feb 2012 13:10:41 -0700,
>     David Beer wrote:
>     >
>     > [1  <multipart/alternative (7bit)>]
>     > [1.1  <text/plain; ISO-8859-1 (7bit)>]
>     >
>     > [1.2  <text/html; ISO-8859-1 (quoted-printable)>]
>     > Doug,
>     >
>     > I have created a ticket for our documentation team to note that the TDK is where nvml.h can be
>     found.
>     >
>     > We also thank you for the patch. I believe there is some more work that needs to be done beyond
>     just
>     > this change, but we will look to get those done very soon. I think it would be ideal to allow
>     people to
>     > use the same binary for both GPU enabled and non-GPU enabled nodes.
>     >
>    
>     Yeah, conceptual versus actually working.  There's no really proper
>     way to do this with how the gpu code is currently inline with many
>     ifdefs.  It's a surprisingly small number of entry points that need to
>     be modified.  See the attached patch, this allows an NVML enabled
>     build to run on either a GPU or non-GPU node.  If all the GPU routines
>     were moved into their own file this could be done cleanly and without
>     a lot of effort.
>    
>     Doug
>    
>     PS. caveat emptor with the patch, I've run two jobs on the nodes so
>     it's not exactly thought out.
>    
>     PPS. Should move thread to torque devel, sorry.
> 
>     _______________________________________________
>     torqueusers mailing list
>     torqueusers at supercluster.org
>     http://www.supercluster.org/mailman/listinfo/torqueusers
> 
> --
> David Beer | Software Engineer
> Adaptive Computing
> 
> 
> [2  <text/plain; us-ascii (7bit)>]
> _______________________________________________
> torquedev mailing list
> torquedev at supercluster.org
> http://www.supercluster.org/mailman/listinfo/torquedev