From Gareth.Williams at csiro.au Sat Oct 1 05:57:58 2011 From: Gareth.Williams at csiro.au (Gareth.Williams at csiro.au) Date: Sat, 1 Oct 2011 21:57:58 +1000 Subject: [Mauiusers] maui limits? looking for experience In-Reply-To: <20110930002231.2bb68438@amparo.bogus.net> References: <20110928164052.5a05183a@amarrosa.pic.es> <4E833A26.8070900@rqchp.qc.ca> <20110929115533.666d68a5@amarrosa.pic.es> <007DECE986B47F4EABF823C1FBB19C620102B6D6AE5A@exvic-mbx04.nexus.csiro.au> <20110930002231.2bb68438@amparo.bogus.net> Message-ID: <007DECE986B47F4EABF823C1FBB19C620102B6D6AE74@exvic-mbx04.nexus.csiro.au> > -----Original Message----- > From: Arnau Bria [mailto:arnaubria at pic.es] > Sent: Friday, 30 September 2011 8:23 AM > To: mauiusers at supercluster.org > Subject: Re: [Mauiusers] maui limits? looking for experience > > On Fri, 30 Sep 2011 07:23:52 +1000 > Gareth.Williams at csiro.au Gareth.Williams at csiro.au wrote: > > Hi Gareth, > > > > This really improves maui behaviour. But limiting idle queue was > the > > > last thing I wanted to do.... > > > > Idle limits are mostly good. This mostly limits the number of each > > users jobs that maui will consider in any scheduling cycle so it make > > the scheduling cycle shorter/faster. It also limits the priority > > accumulated by queued jobs and alleviates 'queue stuffing'. I'd > > recommend idle limits given that maui does not contain a better > > facility to handle such issues. > > Yep, a short queue reduces maui stress. I completely agree that. > Seting a limit of 100 jobs per user leaves a 1k idle queue in normal > behaviour, when many user are running jobs. That's the limit I'd use. > > > But, as I've never tried this before, let me ask how maui will behave > in this situation: > > > if the farm is 70%, and I have only two users who have submited jobs > (user A and B). User A has much more priority than user B, so let's say > that the 30% must be filled with 25% of jobs from user A and 5% jobs > from user B, if I have 1000 jobs in queue (500 from A and 500 from B) > IDLE queue will contain 100 jobs of each user, so each scheduling > cycle is going to schedule 200 jobs, is maui going to fill up the farm > respecting our policies (25/5)? or is it going to start 100 jobs from > each user on each scheduling cycle filling up the farm 15% and 15%? Hi Arnau, I have to guess more than I'd like to in order to answer. However, it might be good to start by noting that the question presumes that maui starts many jobs in one cycle. I think this is rare (but have no idea what the limit is if there is one). Most commonly, one might hope the cluster is busy so maui can only start a small number of jobs when other jobs finish. Jobs with highest priority reserve resources in the future to ensure they get started, other jobs may get backfilled. Only jobs that fit within the idle limits will be considered (for which they must be in an execution queue too). If the cluster is relatively idle, then it will probably fill up with work from whoever queues it first. If you have target shares, you can only meet them with a relatively consistent balance if all the shareholders submit plenty of jobs that don't have very different requirements. Otherwise you need to be content with reasonable balance over the long term and fair-share scheduling helping the priority to slosh back and forth usefully. So to your numbers. If the cluster is at 70% either a lot of work finished at once or nobody has been queuing work for a while. If the latter is true, probably one person will queue first and fill most of the 30% (if they have enough jobs). When someone else submits jobs, if enough time has passed, fair-share with kick in and their jobs will get the higher start priority. If it's the former, jobs at the top of the priority list should get started. If the main factor is fair-share then this probably means mostly one persons jobs so they can catch up to their target. If you have lost of very small jobs, well that is a challenge. Perhaps you can demonstrate to your users how to aggregate them into modest groups to get maybe 1-2hr jobs. Their overall throughput might increase... It might also be work that no-one wants to do. Gareth > > > > If I understand routing queues properly, they send jobs based on > job > > > required resources. our jobs do not require any special resource, > > > our users send jobs based on queue name that show time limits. So, > > > I think that routing queues can't help here. > > > > What is being proposed is that you have a routing queue setup with no > > special resources, just one routing queue per execution queue (but > > make it as fancy as you like - though simple is good). Put a limit > > on the number of (users) jobs in the execution queue(s) (enough to > > fill the cluster) but allow many jobs in the routing queue(s). Maui > > only need consider the execution queue so it's job becomes simpler > > and it can be faster. > > ok. now I understand. So, "hide" jobs to maui using routing queues. > > > Gareth (who used maui for some time but doesn't now) > > I've not said that. I'm just asking for other admin (which much > experience) experience. > > Many thanks for your reply, > Cheers, > Arnau From denismpa at gmail.com Sun Oct 2 19:49:01 2011 From: denismpa at gmail.com (Denis) Date: Sun, 2 Oct 2011 22:49:01 -0300 Subject: [Mauiusers] Maui, FairShare, and scheduling GPUs In-Reply-To: <4E85FED0.7070406@Jhu.edu> References: <4E81F238.8080403@Jhu.edu> <4E85FED0.7070406@Jhu.edu> Message-ID: Hi, Jason! 2011/9/30 Jason Williams > I didn't get anyone else replying to this, so I'm curious if anyone else > figured out a different way to make this work. > > Either way, I've been looking around the code today and playing with it, > and I think I have the easy part done. I have maui seeing the "gpus=" > attribute of the nodes and tracking it within the configured resources > and available resources (I think) within the MPBSNodeLoad() and > MPBSNodeUpdate() functions. I can create a new branch in the svn called > "3.3.2_gpu" I would be pleased to give it a spin here. > if anyone is interested in taking a look and/or helping to > implement the tracking code to update the available resources when > someone requests gpus to torque. It seems to me the trickier part will > be how do we want to track that usage within FairShare.... I haven't > started to think about that yet, but suggestions are welcome. > > -- > Jason Williams > Sr. Systems Administrator > Homewood HPC Cluster > Johns Hopkins University > > > > On 9/27/2011 8:08 PM, suraj prabhakaran wrote: > > Hello Jason and Denis, > > > > I too am starting to look into this and am at the beginner's stage trying > to understand how things work. I would be interested in working together on > this. If no one else replies to the main thread in 1-2 days, we can discuss > things and share knowledge. Please let me know if you are interested. > > > > Best regards, > > Suraj Prabhakaran > > > > > > On 09/27/11, Jason Williams wrote: > > > >> I'm curious if anyone has taken a look at getting Torque 2.5.x and Maui > >> working together to schedule GPUS and track the usage via FairShare. I > >> am pondering what would be needed to actually make that happen within > >> the Maui source, but if someone else has already started working on > >> this, it would be interesting to get their take on the situation. I've > >> noticed, via some googling and reading on the list here, that it seems > >> difficult to do without some mods to the source. If you've thought > >> about it or have started on it, please email me back. > >> > >> -- > >> Jason Williams > >> Sr. Systems Administrator > >> Homewood HPC Cluster > >> Johns Hopkins University > >> > >> > >> > >> -------------------------- > >> Suraj Prabhakaran > >> > >> German Research School for > >> Simulation Sciences GmbH > >> Laboratory for Parallel Progreamming > >> 52062 Aachen | Germany > >> > > _______________________________________________ > mauiusers mailing list > mauiusers at supercluster.org > http://www.supercluster.org/mailman/listinfo/mauiusers > -- Denis Anjos, www.versatushpc.com.br -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.supercluster.org/pipermail/mauiusers/attachments/20111002/98ec4d03/attachment.html From jayavant.patil82 at gmail.com Sun Oct 2 23:43:22 2011 From: jayavant.patil82 at gmail.com (Jayavant Patil) Date: Mon, 3 Oct 2011 11:13:22 +0530 Subject: [Mauiusers] Job priority with respect to queue Message-ID: Hi, Is it possible to change the priorities of jobs within the same queue? -- Thanks & Regards, Jayavant N. Patil -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.supercluster.org/pipermail/mauiusers/attachments/20111003/bcd2fd03/attachment.html From jasonw at Jhu.edu Mon Oct 3 07:55:01 2011 From: jasonw at Jhu.edu (Jason Williams) Date: Mon, 03 Oct 2011 09:55:01 -0400 Subject: [Mauiusers] Maui, FairShare, and scheduling GPUs In-Reply-To: References: <4E81F238.8080403@Jhu.edu> <4E85FED0.7070406@Jhu.edu> Message-ID: <4E89BEB5.9060909@Jhu.edu> Ok, I committed what I did on Friday to the svn server. It's in /branches/maui_jason_at_jhu/ and not in /trunk/ This is *HIGHLY* untested and doesn't actually track utilized resources yet. It merely reports them using checknode, but it shouldn't be too hard to actually get scheduling GPUs implemented (famous last words.) I'm pretty swamped today, but I'll try to do some more tomorrow or so. If anyone makes changes or starts on the implementation code, please fire me an email and let me know. On 10/2/2011 9:49 PM, Denis wrote: > Hi, Jason! > > 2011/9/30 Jason Williams > > > I didn't get anyone else replying to this, so I'm curious if > anyone else > figured out a different way to make this work. > > Either way, I've been looking around the code today and playing > with it, > and I think I have the easy part done. I have maui seeing the "gpus=" > attribute of the nodes and tracking it within the configured resources > and available resources (I think) within the MPBSNodeLoad() and > MPBSNodeUpdate() functions. I can create a new branch in the svn > called > "3.3.2_gpu" > > I would be pleased to give it a spin here. > > if anyone is interested in taking a look and/or helping to > implement the tracking code to update the available resources when > someone requests gpus to torque. It seems to me the trickier part > will > be how do we want to track that usage within FairShare.... I haven't > started to think about that yet, but suggestions are welcome. > > -- > Jason Williams > Sr. Systems Administrator > Homewood HPC Cluster > Johns Hopkins University > > > > On 9/27/2011 8:08 PM, suraj prabhakaran wrote: > > Hello Jason and Denis, > > > > I too am starting to look into this and am at the beginner's > stage trying to understand how things work. I would be interested > in working together on this. If no one else replies to the main > thread in 1-2 days, we can discuss things and share knowledge. > Please let me know if you are interested. > > > > Best regards, > > Suraj Prabhakaran > > > > > > On 09/27/11, Jason Williams wrote: > > > >> I'm curious if anyone has taken a look at getting Torque 2.5.x > and Maui > >> working together to schedule GPUS and track the usage via > FairShare. I > >> am pondering what would be needed to actually make that happen > within > >> the Maui source, but if someone else has already started working on > >> this, it would be interesting to get their take on the > situation. I've > >> noticed, via some googling and reading on the list here, that > it seems > >> difficult to do without some mods to the source. If you've thought > >> about it or have started on it, please email me back. > >> > >> -- > >> Jason Williams > >> Sr. Systems Administrator > >> Homewood HPC Cluster > >> Johns Hopkins University > >> > >> > >> > >> -------------------------- > >> Suraj Prabhakaran > >> > >> German Research School for > >> Simulation Sciences GmbH > >> Laboratory for Parallel Progreamming > >> 52062 Aachen | Germany > >> > > _______________________________________________ > mauiusers mailing list > mauiusers at supercluster.org > http://www.supercluster.org/mailman/listinfo/mauiusers > > > > > -- > Denis Anjos, > www.versatushpc.com.br -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.supercluster.org/pipermail/mauiusers/attachments/20111003/1d7ca5f0/attachment-0001.html From arnaubria at pic.es Wed Oct 5 08:04:36 2011 From: arnaubria at pic.es (Arnau Bria) Date: Wed, 5 Oct 2011 16:04:36 +0200 Subject: [Mauiusers] maui limits? looking for experience In-Reply-To: <007DECE986B47F4EABF823C1FBB19C620102B6D6AE74@exvic-mbx04.nexus.csiro.au> References: <20110928164052.5a05183a@amarrosa.pic.es> <4E833A26.8070900@rqchp.qc.ca> <20110929115533.666d68a5@amarrosa.pic.es> <007DECE986B47F4EABF823C1FBB19C620102B6D6AE5A@exvic-mbx04.nexus.csiro.au> <20110930002231.2bb68438@amparo.bogus.net> <007DECE986B47F4EABF823C1FBB19C620102B6D6AE74@exvic-mbx04.nexus.csiro.au> Message-ID: On Sat, Oct 1, 2011 at 1:57 PM, wrote: > > -----Original Message----- > > From: Arnau Bria [mailto:arnaubria at pic.es] > > Sent: Friday, 30 September 2011 8:23 AM > > To: mauiusers at supercluster.org > > Subject: Re: [Mauiusers] maui limits? looking for experience > > > > On Fri, 30 Sep 2011 07:23:52 +1000 > > Gareth.Williams at csiro.au Gareth.Williams at csiro.au wrote: > > > > Hi Arnau, > Hi Gareth, > I have to guess more than I'd like to in order to answer. However, it might > be good to start by noting that the question presumes that maui starts many > jobs in one cycle. I think this is rare (but have no idea what the limit is > if there is one). Most commonly, one might hope the cluster is busy so maui > can only start a small number of jobs when other jobs finish. Jobs with > highest priority reserve resources in the future to ensure they get started, > other jobs may get backfilled. Only jobs that fit within the idle limits > will be considered (for which they must be in an execution queue too). > Ideally it should start few jobs, but due to our "big" schedulling cycle (we need it because when maui schedules it does not respond to user commands) and the fact that maui stops schedulling jobs when it finds a "new" node busy (is a "bug" we are seeing since we increased our farm size, already reported here, maybe I should open a bug) some cycles could start more than 200 jobs. As the ideal IDLE limit should be around 100 jobs per user, we'll be facing the issue previosuly described. If the cluster is relatively idle, then it will probably fill up with work > from whoever queues it first. > > If you have target shares, you can only meet them with a relatively > consistent balance if all the shareholders submit plenty of jobs that don't > have very different requirements. Otherwise you need to be content with > reasonable balance over the long term and fair-share scheduling helping the > priority to slosh back and forth usefully. > Our case is the first you describe, and, after some test, we've met a good FS configuration and we're quite happy with FS targets.But as we have many users/groups we need a "long" IDLE queue in order to have enough jobs from any group / user. base 8.5% - lhcatlas 38.46% 37.42% lhccms 23.53% 22.16% lhclhcb 14.81% 14.39% lhtier2 14.48% 14.24% localat3 0.16% 1.9% magic 0.06% 3.8% pau 0.0% 3.8% So to your numbers. If the cluster is at 70% either a lot of work finished > at once or nobody has been queuing work for a while. If the latter is true, > probably one person will queue first and fill most of the 30% (if they have > enough jobs). When someone else submits jobs, if enough time has passed, > fair-share with kick in and their jobs will get the higher start priority. > If it's the former, jobs at the top of the priority list should get > started. If the main factor is fair-share then this probably means mostly > one persons jobs so they can catch up to their target. > This example you're talking about is a compressive scenario, but we're assuming that we have no queued jobs. But that IDLE 30% could be due to a drain of some blades of our farm, so, when we set all those nodes online again (with more than 1k jobs in queue) and maui not seeing the "real" queue, it's goint to fill up the farm without respecting FS. Maybe this situations isn't the most critical (I could control it by hand), but I'm really worried about the normal scheduling cycle we see in our farm. Last week we gave IDLE limit a new oportunity and we saw really bad results. > If you have lost of very small jobs, well that is a challenge. Perhaps you > can demonstrate to your users how to aggregate them into modest groups to > get maybe 1-2hr jobs. Their overall throughput might increase... It might > also be work that no-one wants to do. > Unfortunately 85% of our jobs are grid jobs, and changing "users" behaviour is really complicated. I'll play with NODEPOLLFREQUENCY and JOBAGGREGATIONTIME and see if I get an increase of maui performance. Gareth > Many thanks for your help Gareth, Cheers, Arnau -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.supercluster.org/pipermail/mauiusers/attachments/20111005/bda4fcde/attachment.html From vlad at cosy.sbg.ac.at Thu Oct 6 07:14:05 2011 From: vlad at cosy.sbg.ac.at (vlad at cosy.sbg.ac.at) Date: Thu, 6 Oct 2011 15:14:05 +0200 Subject: [Mauiusers] Jobs stuck in queues not being processed at all Message-ID: <8458d6716420ce16fbbe174e23847a1b.squirrel@webmail.cosy.sbg.ac.at> Greetings ! We are running Maui 3.3.1 combined with Torque Version 3.0.3-snap.201107121616 on a small cluster with mixed resources 7 GPU nodes (i7) and 14 Nodes with Opteron Magny-Cours-CPUs. The Maui and Torque service is running on our portal node called gpu We are able to submit simple jobs and OpenMPI jobs. They are being processed correctly until the resources get exhausted and the jobs get queued. After processing the current running jobs, the nodes get free again and stay that way, althogh the queue is still full. One should expect, that the jobs queued should then get executed, one by one, until the queue is empty again (as no further jobs are submitted). But, although pbsnodes states every single node as free (twisting thumbs..), the queue does not get processed. We have defined several queues, for each type of resouces one short (24h), medium (72h), one long (infinite), called gpushort. gpumedium, gpulong (for the GPU nodes) and respective optshort,optmedium, optlong (for the Opteron nodes). I have tried to set reservations, so that the Opterons get assigned to the Opteron queues (exclusively) and the GPU nodes to the GPU queues, so that one does not need to set the pbs requests (opteron or gpunode) in the submit script. I have failed in that . 1) How do I get the nodes assigned to their queues selected by their properties "opteron" and "gpunode" properly ? 2)More important: How can I fix this bad behaviour, that queued jobs are never been processed? I'd be grateful for any help, since I'm fairly new to this matter and I did not find my answers in the documentation. Greetings from Salzburg/Austria/Europe Vlad Popa University of Salzburg Computer Science/HPC Computing Jakob-Harringer-Str. 2 5020 Salzburg Austria PS: Below our configuration... pbsnodes: gpu01 state = free np = 8 properties = i7,i7-new,gpunode,16G ntype = cluster status = rectime=1317905387,varattr=,jobs=,state=free,netload=12779454,gres =,loadave=0.03,ncpus=8,physmem=16315316kb,availmem=48458408kb,totmem=49083308kb, idletime=16275,nusers=0,nsessions=0,uname=Linux gpu01 2.6.32-131.6.1.el6.x86_64 #1 SMP Fri Jul 15 09:29:38 EDT 2011 x86_64,opsys=linux mom_service_port = 15002 mom_manager_port = 15003 gpus = 2 gpu_status = gpu[1]=gpu_id=0000:06:00.0;,gpu[0]=gpu_id=0000:05:00.0;,driver _ver=280.13,timestamp=Thu Oct 6 14:53:52 201 .... and so on until gpu07: gpu07 state = free np = 8 properties = fermi,16G,gpunode,i7 ntype = cluster status = rectime=1317905499,varattr=,jobs=,state=free,netload=5695667,gres=,loadave=0.01,ncpus=8,physmem=16310908kb,availmem=48600784kb,totmem=49078900kb,idletime=14281,nusers=0,nsessions=0,uname=Linux gpu07 2.6.32-131.6.1.el6.x86_64 #1 SMP Fri Jul 15 09:29:38 EDT 2011 x86_64,opsys=linux mom_service_port = 15002 mom_manager_port = 15003 gpus = 1 gpu_status = gpu[0]=gpu_id=0000:01:00.0;,driver_ver=280.13,timestamp=Thu Oct 6 10:57:46 2011 ... followed by our Oteron Nodes called hex01-hex14, all staying free ..hex07 state = free np = 14 properties = opteron ntype = cluster status = rectime=1317905378,varattr=,jobs=,state=free,netload=70147843323,g res=,loadave=0.03,ncpus=16,physmem=32876308kb,availmem=84370548kb,totmem=9841230 0kb,idletime=172141,nusers=2,nsessions=15,sessions=2218 9895 10210 10378 10844 1 0964 11065 11150 11253 11338 11423 11508 11815 11902 12123,uname=Linux hex07 2.6 .32-131.12.1.el6.x86_64 #1 SMP Tue Aug 23 10:52:23 EDT 2011 x86_64,opsys=linux mom_service_port = 15002 mom_manager_port = 15003 gpus = 0 hex06 state = free np = 14 properties = opteron ntype = cluster status = rectime=1317905383,varattr=,jobs=,state=free,netload=1177038092,gr es=,loadave=0.05,ncpus=16,physmem=32877076kb,availmem=97417192kb,totmem=98413068 kb,idletime=884847,nusers=0,nsessions=0,uname=Linux hex06 2.6.32-131.6.1.el6.x86 _64 #1 SMP Fri Jul 15 09:29:38 EDT 2011 x86_64,opsys=linux mom_service_port = 15002 mom_manager_port = 15003 gpus = 0hex07 state = free np = 14 properties = opteron ntype = cluster status = rectime=1317905378,varattr=,jobs=,state=free,netload=70147843323,g res=,loadave=0.03,ncpus=16,physmem=32876308kb,availmem=84370548kb,totmem=9841230 0kb,idletime=172141,nusers=2,nsessions=15,sessions=2218 9895 10210 10378 10844 1 0964 11065 11150 11253 11338 11423 11508 11815 11902 12123,uname=Linux hex07 2.6 .32-131.12.1.el6.x86_64 #1 SMP Tue Aug 23 10:52:23 EDT 2011 x86_64,opsys=linux mom_service_port = 15002hex07 state = free np = 14 properties = opteron ntype = cluster status = rectime=1317905378,varattr=,jobs=,state=free,netload=70147843323,g res=,loadave=0.03,ncpus=16,physmem=32876308kb,availmem=84370548kb,totmem=9841230 0kb,idletime=172141,nusers=2,nsessions=15,sessions=2218 9895 10210 10378 10844 1 0964 11065 11150 11253 11338 11423 11508 11815 11902 12123,uname=Linux hex07 2.6 .32-131.12.1.el6.x86_64 #1 SMP Tue Aug 23 10:52:23 EDT 2011 x86_64,opsys=linux mom_service_port = 15002 mom_manager_port = 15003 gpus = 0 hex06 state = free np = 14 properties = opteron ntype = cluster status = rectime=1317905383,varattr=,jobs=,state=free,netload=1177038092,gr es=,loadave=0.05,ncpus=16,physmem=32877076kb,availmem=97417192kb,totmem=98413068 kb,idletime=884847,nusers=0,nsessions=0,uname=Linux hex06 2.6.32-131.6.1.el6.x86 _64 #1 SMP Fri Jul 15 09:29:38 EDT 2011 x86_64,opsys=linux mom_service_port = 15002 mom_manager_port = 15003 gpus = 0 mom_manager_port = 15003 gpus = 0 hex06 state = free np = 14 properties = opteron ntype = cluster status = rectime=1317905383,varattr=,jobs=,state=free,netload=1177038092,gr es=,loadave=0.05,ncpus=16,physmem=32877076kb,availmem=97417192kb,totmem=98413068 kb,idletime=884847,nusers=0,nsessions=0,uname=Linux hex06 2.6.32-131.6.1.el6.x86 _64 #1 SMP Fri Jul 15 09:29:38 EDT 2011 x86_64,opsys=linux mom_service_port = 15002 mom_manager_port = 15003 gpus = 0 [vlad at gpu ~]$ qmgr -c 'p s' # # Create queues and set their attributes. # # # Create and define queue gpushort # create queue gpushort set queue gpushort queue_type = Execution set queue gpushort resources_min.nodes = 1 set queue gpushort resources_default.neednodes = gpunode set queue gpushort resources_default.nodes = 1 set queue gpushort resources_default.walltime = 24:00:00 set queue gpushort enabled = True set queue gpushort started = True # # Create and define queue optlong # create queue optlong set queue optlong queue_type = Execution set queue optlong resources_default.neednodes = opteron set queue optlong resources_default.nodes = 1 set queue optlong enabled = True set queue optlong started = True # # Create and define queue gpumedium # create queue gpumedium set queue gpumedium queue_type = Execution set queue gpumedium resources_default.neednodes = gpunode set queue gpumedium resources_default.nodes = 1 set queue gpumedium resources_default.walltime = 72:00:00 set queue gpumedium enabled = True set queue gpumedium started = True # # Create and define queue gpulong # create queue gpulong set queue gpulong queue_type = Execution set queue gpulong resources_default.neednodes = gpunode set queue gpulong resources_default.nodes = 1 set queue gpulong enabled = True set queue gpulong started = True # # Create and define queue batch # create queue batch set queue batch queue_type = Execution set queue batch resources_default.nodes = 1 set queue batch resources_default.walltime = 01:00:00 set queue batch enabled = True set queue batch started = True # # Create and define queue optshort # create queue optshort set queue optshort queue_type = Execution set queue optshort resources_default.neednodes = opteron set queue optshort resources_default.nodes = 1 set queue optshort resources_default.walltime = 24:00:00 set queue optshort enabled = True set queue optshort started = True # # Create and define queue optmedium # create queue optmedium set queue optmedium queue_type = Execution set queue optmedium resources_default.neednodes = opteron set queue optmedium resources_default.nodes = 1 set queue optmedium resources_default.walltime = 72:00:00 set queue optmedium enabled = True set queue optmedium started = True # # Create and define queue short # create queue short set queue short queue_type = Execution set queue short resources_default.walltime = 24:00:00 set queue short enabled = True set queue short started = True # # Set server attributes.[vlad at gpu ~]$ qmgr -c 'p s' # # Create queues and set their attributes. # # # Create and define queue gpushort # create queue gpushort set queue gpushort queue_type = Execution set queue gpushort resources_min.nodes = 1 set queue gpushort resources_default.neednodes = gpunode set queue gpushort resources_default.nodes = 1hex07 state = free np = 14 properties = opteron ntype = cluster status = rectime=1317905378,varattr=,jobs=,state=free,netload=70147843323,g res=,loadave=0.03,ncpus=16,physmem=32876308kb,availmem=84370548kb,totmem=9841230 0kb,idletime=172141,nusers=2,nsessions=15,sessions=2218 9895 10210 10378 10844 1 0964 11065 11150 11253 11338 11423 11508 11815 11902 12123,uname=Linux hex07 2.6 .32-131.12.1.el6.x86_64 #1 SMP Tue Aug 23 10:52:23 EDT 2011 x86_64,opsys=linux mom_service_port = 15002 mom_manager_port = 15003 gpus = 0 hex06 state = free np = 14 properties = opteron ntype = cluster status = rectime=1317905383,varattr=,jobs=,state=free,netload=1177038092,gr es=,loadave=0.05,ncpus=16,physmem=32877076kb,availmem=97417192kb,totmem=98413068 kb,idletime=884847,nusers=0,nsessions=0,uname=Linux hex06 2.6.32-131.6.1.el6.x86 _64 #1 SMP Fri Jul 15 09:29:38 EDT 2011 x86_64,opsys=linux mom_service_port = 15002 mom_manager_port = 15003 gpus = 0 set queue gpushort resources_default.walltime = 24:00:00 set queue gpushort enabled = True set queue gpushort started = True # # Create and define queue optlong # create queue optlong set queue optlong queue_type = Execution set queue optlong resources_default.neednodes = opteron set queue optlong resources_default.nodes = 1 set queue optlong enabled = True set queue optlong started = True # # Create and define queue gpumedium # create queue gpumedium set queue gpumedium queue_type = Execution set queue gpumedium resources_default.neednodes = gpunode set queue gpumedium resources_default.nodes = 1hex07 state = free np = 14 properties = opteron ntype = cluster status = rectime=1317905378,varattr=,jobs=,state=free,netload=70147843323,g res=,loadave=0.03,ncpus=16,physmem=32876308kb,availmem=84370548kb,totmem=9841230 0kb,idletime=172141,nusers=2,nsessions=15,sessions=2218 9895 10210 10378 10844 1 0964 11065 11150 11253 11338 11423 11508 11815 11902 12123,uname=Linux hex07 2.6 .32-131.12.1.el6.x86_64 #1 SMP Tue Aug 23 10:52:23 EDT 2011 x86_64,opsys=linux mom_service_port = 15002 mom_manager_port = 15003 gpus = 0 hex06 state = free np = 14 properties = opteron ntype = cluster status = rectime=1317905383,varattr=,jobs=,state=free,netload=1177038092,gr es=,loadave=0.05,ncpus=16,physmem=32877076kb,availmem=97417192kb,totmem=98413068 kb,idletime=884847,nusers=0,nsessions=0,uname=Linux hex06 2.6.32-131.6.1.el6.x86 _64 #1 SMP Fri Jul 15 09:29:38 EDT 2011 x86_64,opsys=linux mom_service_port = 15002 mom_manager_port = 15003 gpus = 0 set queue gpumedium resources_default.walltime = 72:00:00 set queue gpumedium enabled = True set queue gpumedium started = True # # Create and define queue gpulong # create queue gpulong set queue gpulong queue_type = Execution set queue gpulong resources_default.neednodes = gpunode set queue gpulong resources_default.nodes = 1 set queue gpulong enabled = True set queue gpulong started = True # # Create and define queue batch # create queue batch set queue batch queue_type = Execution set queue batch resources_default.nodes = 1 set queue batch resources_default.walltime = 01:00:00 set queue batch enabled = True set queue batch started = True # # Create and define queue optshort # create queue optshort set queue optshort queue_type = Execution set queue optshort resources_default.neednodes = opteron set queue optshort resources_default.nodes = 1 set queue optshort resources_default.walltime = 24:00:00 set queue optshort enabled = True set queue optshort started = True # # Create and define queue optmedium # create queue optmedium set queue optmedium queue_type = Execution set queue optmedium resources_default.neednodes = opteron set queue optmedium resources_default.nodes = 1 set queue optmedium resources_default.walltime = 72:00:00 set queue optmedium enabled = True set queue optmedium started = True # # Create and define queue short # create queue short set queue short queue_type = Execution set queue short resources_default.walltime = 24:00:00 set queue short enabled = True set queue short started = True # # Set server attributes. # set server scheduling = True set server acl_hosts = gpu set server managers = forsthof at gpu set server managers += peter at gpu set server managers += root at gpu set server managers += vlad at gpu set server operators = forsthof at gpu set server operators += peter at gpu set server operators += root at gpu set server operators += vlad at gpu set server default_queue = batch set server log_events = 511 set server mail_from = adm set server scheduler_iteration = 600 set server node_check_rate = 150 set server tcp_timeout = 6 set server log_level = 7 set server mom_job_sync = True set server keep_completed = 300 set server next_job_number = 293 # set server scheduling = True set server acl_hosts = gpu set server managers = forsthof at gpu set server managers += peter at gpu set server managers += root at gpu set server managers += vlad at gpu set server operators = forsthof at gpu set server operators += peter at gpu set server operators += root at gpu set server operators += vlad at gpu set server default_queue = batch set server log_events = 511 set server mail_from = adm set server scheduler_iteration = 600 set server node_check_rate = 150 set server tcp_timeout = 6 set server log_level = 7 set server mom_job_sync = True set server keep_completed = 300 set server next_job_number = 293 [vlad at gpu ~]$ showconfig # Maui version 3.3.1 (PID: 22120) # global policies REJECTNEGPRIOJOBS[0] FALSE ENABLENEGJOBPRIORITY[0] FALSE ENABLEMULTINODEJOBS[0] TRUE ENABLEMULTIREQJOBS[0] FALSE BFPRIORITYPOLICY[0] [NONE] JOBPRIOACCRUALPOLICY QUEUEPOLICY NODELOADPOLICY ADJUSTSTATE USEMACHINESPEEDFORFS FALSE USEMACHINESPEED FALSE USESYSTEMQUEUETIME TRUE USELOCALMACHINEPRIORITY FALSE NODEUNTRACKEDLOADFACTOR 1.2 JOBNODEMATCHPOLICY[0] JOBMAXSTARTTIME[0] INFINITY METAMAXTASKS[0] 0 NODESETPOLICY[0] [NONE] NODESETATTRIBUTE[0] [NONE] NODESETLIST[0] NODESETDELAY[0] 00:00:00 NODESETPRIORITYTYPE[0] MINLOSS NODESETTOLERANCE[0] 0.00 BACKFILLPOLICY[0] FIRSTFIT BACKFILLDEPTH[0] 0 BACKFILLPROCFACTOR[0] 0 BACKFILLMAXSCHEDULES[0] 10000 BACKFILLMETRIC[0] PROCS BFCHUNKDURATION[0] 00:00:00 BFCHUNKSIZE[0] 0 PREEMPTPOLICY[0] REQUEUE MINADMINSTIME[0] 00:00:00 RESOURCELIMITPOLICY[0] NODEAVAILABILITYPOLICY[0] COMBINED:[DEFAULT] NODEALLOCATIONPOLICY[0] CPULOAD TASKDISTRIBUTIONPOLICY[0] DEFAULT RESERVATIONPOLICY[0] CURRENTHIGHEST RESERVATIONRETRYTIME[0] 00:00:00 RESERVATIONTHRESHOLDTYPE[0] NONE RESERVATIONTHRESHOLDVALUE[0] 0 FSPOLICY [NONE] FSPOLICY [NONE] FSINTERVAL 12:00:00 FSDEPTH 8 FSDECAY 1.00 # Priority Weights SERVICEWEIGHT[0] 1 TARGETWEIGHT[0] 1 CREDWEIGHT[0] 1 ATTRWEIGHT[0] 1 FSWEIGHT[0] 1 RESWEIGHT[0] 1 USAGEWEIGHT[0] 1 QUEUETIMEWEIGHT[0] 1 XFACTORWEIGHT[0] 0 SPVIOLATIONWEIGHT[0] 0 BYPASSWEIGHT[0] 0 TARGETQUEUETIMEWEIGHT[0] 0 TARGETXFACTORWEIGHT[0] 0 USERWEIGHT[0] 0 GROUPWEIGHT[0] 0 ACCOUNTWEIGHT[0] 0 QOSWEIGHT[0] 0 CLASSWEIGHT[0] 0 FSUSERWEIGHT[0] 0 FSGROUPWEIGHT[0] 0 FSACCOUNTWEIGHT[0] 0 FSQOSWEIGHT[0] 0 FSCLASSWEIGHT[0] 0 ATTRATTRWEIGHT[0] 0 ATTRSTATEWEIGHT[0] 0 NODEWEIGHT[0] 0 PROCWEIGHT[0] 0 MEMWEIGHT[0] 0 SWAPWEIGHT[0] 0 DISKWEIGHT[0] 0 PSWEIGHT[0] 0 PEWEIGHT[0] 0 WALLTIMEWEIGHT[0] 0 UPROCWEIGHT[0] 0 UJOBWEIGHT[0] 0 CONSUMEDWEIGHT[0] 0 USAGEEXECUTIONTIMEWEIGHT[0] 0 REMAININGWEIGHT[0] 0 PERCENTWEIGHT[0] 0 XFMINWCLIMIT[0] 00:02:00 # partition DEFAULT policies REJECTNEGPRIOJOBS[1] FALSE ENABLENEGJOBPRIORITY[1] FALSE ENABLEMULTINODEJOBS[1] TRUE ENABLEMULTIREQJOBS[1] FALSE BFPRIORITYPOLICY[1] [NONE] JOBPRIOACCRUALPOLICY QUEUEPOLICY NODELOADPOLICY ADJUSTSTATE JOBNODEMATCHPOLICY[1] JOBMAXSTARTTIME[1] INFINITY METAMAXTASKS[1] 0 NODESETPOLICY[1] [NONE] NODESETATTRIBUTE[1] [NONE] NODESETLIST[1] NODESETDELAY[1] 00:00:00 NODESETPRIORITYTYPE[1] MINLOSS NODESETTOLERANCE[1] 0.00 # Priority Weights XFMINWCLIMIT[1] 00:00:00 RMAUTHTYPE[0] CHECKSUM CLASSCFG[[NONE]] DEFAULT.FEATURES=[NONE] CLASSCFG[[ALL]] DEFAULT.FEATURES=[NONE] CLASSCFG[gpushort] DEFAULT.FEATURES=[gpunode] CLASSCFG[optlong] DEFAULT.FEATURES=[opteron] CLASSCFG[gpumedium] DEFAULT.FEATURES=[gpunode] CLASSCFG[gpulong] DEFAULT.FEATURES=[gpunode] CLASSCFG[batch] DEFAULT.FEATURES=[NONE] CLASSCFG[optshort] DEFAULT.FEATURES=[opteron] CLASSCFG[optmedium] DEFAULT.FEATURES=[opteron] CLASSCFG[short] DEFAULT.FEATURES=[NONE] QOSPRIORITY[0] 0 QOSQTWEIGHT[0] 0 QOSXFWEIGHT[0] 0 QOSTARGETXF[0] 0.00 QOSTARGETQT[0] 00:00:00 QOSFLAGS[0] QOSPRIORITY[1] 0 QOSQTWEIGHT[1] 0 QOSXFWEIGHT[1] 0 QOSTARGETXF[1] 0.00 QOSTARGETQT[1] 00:00:00 QOSFLAGS[1] # SERVER MODULES: MX SERVERMODE NORMAL SERVERNAME SERVERHOST gpu SERVERPORT 42559 LOGFILE maui.log LOGFILEMAXSIZE 10000000 LOGFILEROLLDEPTH 1 LOGLEVEL 9 LOGFACILITY fALL SERVERHOMEDIR /var/spool/maui/ TOOLSDIR /var/spool/maui/tools/ LOGDIR /var/spool/maui/log/ STATDIR /var/spool/maui/stats/ LOCKFILE /var/spool/maui/maui.pid SERVERCONFIGFILE /var/spool/maui/maui.cfg CHECKPOINTFILE /var/spool/maui/maui.ck CHECKPOINTINTERVAL 00:05:00 CHECKPOINTEXPIRATIONTIME 3:11:20:00 TRAPJOB TRAPNODE TRAPFUNCTION RESDEPTH 24 RMPOLLINTERVAL 00:00:30 NODEACCESSPOLICY SHARED ALLOCLOCALITYPOLICY [NONE] SIMTIMEPOLICY [NONE] ADMIN1 root vlad peter forsthof ADMINHOSTS ALL NODEPOLLFREQUENCY 0 DISPLAYFLAGS DEFAULTDOMAIN DEFAULTCLASSLIST [DEFAULT:1] FEATURENODETYPEHEADER FEATUREPROCSPEEDHEADER FEATUREPARTITIONHEADER DEFERTIME 1:00:00 DEFERCOUNT 24 DEFERSTARTCOUNT 1 JOBPURGETIME 0 NODEPURGETIME 2140000000 APIFAILURETHRESHHOLD 6 NODESYNCTIME 600 JOBSYNCTIME 600 JOBMAXOVERRUN 00:10:00 NODEMAXLOAD 0.0 PLOTMINTIME 120 PLOTMAXTIME 245760 PLOTTIMESCALE 11 PLOTMINPROC 1 PLOTMAXPROC 512 PLOTPROCSCALE 9 SCHEDCFG[] MODE=NORMAL SERVER=gpu:42559 # RM MODULES: PBS SSS WIKI NATIVE RMCFG[GPU] AUTHTYPE=CHECKSUM EPORT=15004 TIMEOUT=00:00:09 TYPE=PBS SIMWORKLOADTRACEFILE workload SIMRESOURCETRACEFILE resource SIMAUTOSHUTDOWN OFF SIMSTARTTIME 0 SIMSCALEJOBRUNTIME FALSE SIMFLAGS SIMJOBSUBMISSIONPOLICY CONSTANTJOBDEPTH SIMINITIALQUEUEDEPTH 16 SIMWCACCURACY 0.00 SIMWCACCURACYCHANGE 0.00 SIMNODECOUNT 0 SIMNODECONFIGURATION NORMAL SIMWCSCALINGPERCENT 100 SIMCOMRATE 0.10 SIMCOMTYPE ROUNDROBIN COMINTRAFRAMECOST 0.30 COMINTERFRAMECOST 0.30 SIMSTOPITERATION -1 SIMEXITITERATION -1 From JRS221 at bham.ac.uk Thu Oct 13 05:08:42 2011 From: JRS221 at bham.ac.uk (Jonathan Smale) Date: Thu, 13 Oct 2011 11:08:42 +0000 Subject: [Mauiusers] Assigning Nodes to specific nodes Message-ID: Dear Torque & Maui users, I'm crossposting this to maui & torque users as I'm not sure which of these is posing a problem. I'm trying to make three separate queue for the three different types of nodes on our cluster using the combination of qmgr commands: set queue firstgen resources_default.neednodes = 1stgennodes set nodes compute-0-0 properties = 1stgennodes I've found a fair few previous emails about this and have followed their solution without sucess. I can submit the jobs to the queues but they remain in a queued state, qstat -f gives:- [root at che-hydra /]# qstat -f 48691 Job Id: 48691.che-hydra.bham.ac.uk Job_Name = allenega_p2000_g300_r2.cmd Job_Owner = jsmale at che-hydra.bham.ac.uk resources_used.cput = 529:06:32 resources_used.mem = 16872kb resources_used.vmem = 245908kb resources_used.walltime = 529:11:31 job_state = R queue = default server = che-hydra.bham.ac.uk Checkpoint = u ctime = Wed Sep 21 10:45:00 2011 Error_Path = che-hydra.bham.ac.uk:/home/jsmale/allenega/allenega_p2000_g30 0_r2.cmd.e48691 exec_host = compute-0-16/1 Hold_Types = n Join_Path = n Keep_Files = n Mail_Points = a mtime = Wed Sep 21 10:45:01 2011 Output_Path = che-hydra.bham.ac.uk:/home/jsmale/allenega/allenega_p2000_g3 00_r2.cmd.o48691 Priority = 0 qtime = Wed Sep 21 10:45:00 2011 Rerunable = True Resource_List.neednodes = 1 Resource_List.nodect = 1 Resource_List.nodes = 1 session_id = 20687 substate = 42 Variable_List = PBS_O_HOME=/home/jsmale,PBS_O_LANG=en_US.iso885915, PBS_O_LOGNAME=jsmale, PBS_O_PATH=/home/jsmale/mctdh90.svn/bin/x86_64:/home/jsmale/mctdh90.s vn/bin:/usr/lib64/openmpi/1.3.2-gcc/bin:/usr/kerberos/bin:/usr/java/la test/bin:/usr/local/bin:/bin:/usr/bin:/opt/maui/bin:/opt/torque/bin:/o pt/torque/sbin:/usr/share/pvm3/pvm3//bin/LINUX64:/opt/rocks/bin:/opt/r ocks/sbin:/global64/pgi/linux86-64/10.4/bin:/user/worth/gaussian/bin:/ user/jsmale/bin:/home/gaussian/bin:~/mctdh90.svn/bin:/home/jsmale/bin, PBS_O_MAIL=/var/spool/mail/jsmale,PBS_O_SHELL=/bin/bash, PBS_SERVER=che-hydra.bham.ac.uk,PBS_O_HOST=che-hydra.bham.ac.uk, PBS_O_WORKDIR=/home/jsmale/allenega,PBS_O_QUEUE=default euser = jsmale egroup = worth hashname = 48691.che-hydra.bham.ac.uk queue_rank = 35491 queue_type = E etime = Wed Sep 21 10:45:00 2011 submit_args = allenega_p2000_g300_r2.cmd start_time = Wed Sep 21 10:45:01 2011 start_count = 1 and checkjob gives the following: [root at che-hydra /]# checkjob 48691 checking job 48691 State: Running Creds: user:jsmale group:worth class:default qos:DEFAULT WallTime: 22:01:12:16 of 99:23:59:59 SubmitTime: Wed Sep 21 10:45:00 (Time Queued Total: 00:00:01 Eligible: 00:00:00) StartTime: Wed Sep 21 10:45:01 Total Tasks: 1 Req[0] TaskCount: 1 Partition: DEFAULT Network: [NONE] Memory >= 0 Disk >= 0 Swap >= 0 Opsys: [NONE] Arch: [NONE] Features: [NONE] NodeCount: 1 Allocated Nodes: [compute-0-16:1] IWD: [NONE] Executable: [NONE] Bypass: 0 StartCount: 1 PartitionMask: [ALL] Flags: RESTARTABLE Reservation '48691' ( -INFINITY -> 77:22:47:56 Duration: 99:23:59:59) PE: 1.00 StartPriority: 20 I'm not sure why the job isn't running, there doesn't seem to be any reason given in either the maui or torque (server&mom) logs. Could anyone help me decipher the cause? Configuration of the server follows. Top of maui.cfg file: # maui.cfg 3.2.6p20 SERVERHOST che-hydra.bham.ac.uk # primary admin must be first in list ADMIN1 root # Resource Manager Definition RMCFG[che-hydra.bham.ac.uk] TYPE=PBS # Allocation Manager Definition AMCFG[bank] TYPE=NONE # full parameter docs at http://supercluster.org/mauidocs/a.fparameters.html # use the 'schedctl -l' command to display current configuration RMPOLLINTERVAL 00:00:30 SERVERPORT 42559 SERVERMODE NORMAL # Admin: http://supercluster.org/mauidocs/a.esecurity.html LOGFILE maui.log LOGFILEMAXSIZE 100000000 LOGLEVEL 3 # Setting up node information for throttling policies # NODECFG[compute-0-0] SPEED=1 MAXJOB=4 nodetype=firstgennodes NODECFG[compute-0-1] SPEED=1 MAXJOB=4 nodetype=firstgennodes NODECFG[compute-0-2] SPEED=1 MAXJOB=4 nodetype=firstgennodes NODECFG[compute-0-3] SPEED=1 MAXJOB=4 nodetype=firstgennodes NODECFG[compute-0-4] SPEED=1.2 MAXJOB=8 nodetype=secondgennodes NODECFG[compute-0-5] SPEED=1.2 MAXJOB=8 nodetype=secondgennodes NODECFG[compute-0-6] SPEED=1.2 MAXJOB=8 nodetype=secondgennodes NODECFG[compute-0-7] SPEED=1.2 MAXJOB=8 nodetype=secondgennodes NODECFG[compute-0-8] SPEED=1.4 MAXJOB=8 nodetype=secondgennodes NODECFG[compute-0-9] SPEED=1.4 MAXJOB=8 nodetype=secondgennodes NODECFG[compute-0-10] SPEED=1.4 MAXJOB=8 nodetype=secondgennodes NODECFG[compute-0-11] SPEED=1.5 MAXJOB=8 nodetype=secondgennodes NODECFG[compute-0-12] SPEED=1.5 MAXJOB=8 nodetype=secondgennodes NODECFG[compute-0-13] SPEED=1.5 MAXJOB=8 nodetype=secondgennodes NODECFG[compute-0-14] SPEED=1.5 MAXJOB=8 nodetype=secondgennodes NODECFG[compute-0-15] SPEED=1.5 MAXJOB=8 nodetype=secondgennodes NODECFG[compute-0-16] SPEED=1.7 MAXJOB=16 nodetype=thirdgennodes NODECFG[compute-0-17] SPEED=1.7 MAXJOB=16 nodetype=thirdgennodes NODECFG[compute-0-18] SPEED=1.7 MAXJOB=16 nodetype=thirdgennodes NODECFG[compute-0-19] SPEED=1.7 MAXJOB=16 nodetype=thirdgennodes # Setting up queue information to allow allocation to specfic types of nodes via queues CLASSCFG[firstgen] hostlist = compute-0-0,compute-0-1,compute-0-2,compute-0-3 CLASSCFG[secondgen] hostlist = compute-0-4,compute-0-5,compute-0-6,compute-0-7,compute-0-8,compute-0-9,compute-0-10,compute-0-11,compute-0-12,compute-0-13,compute-0-14,compute-0-15 CLASSCFG[thirdgen] hostlist = compute-0-16,compute-0-17,compute-0-18,compute-0-19 # Backfill: http://supercluster.org/mauidocs/8.2backfill.html BACKFILLPOLICY FIRSTFIT RESERVATIONPOLICY CURRENTHIGHEST # Node Allocation: http://supercluster.org/mauidocs/5.2nodeallocation.html NODEALLOCATIONPOLICY CPULOAD Some torque setting that might be of use: [root at che-hydra]# pbsnodes (truncated, one example of each type of node compute-0-0 state = free np = 4 properties = firstgennodes ntype = cluster status = opsys=linux,uname=Linux compute-0-0.local 2.6.18-164.6.1.el5 #1 SMP Tue Nov 3 16:12:36 EST 2009 x86_64,sessions=? 15201,nsessions=? 15201,nusers=0,idletime=13287377,totmem=9195716kb,availmem=8926420kb,physmem=8175600kb,ncpus=4,loadave=0.00,netload=54589282244,state=free,jobs=,varattr=,rectime=1318502240 compute-0-4 state = free np = 8 properties = secondgennodes ntype = cluster status = opsys=linux,uname=Linux compute-0-4.local 2.6.18-164.6.1.el5 #1 SMP Tue Nov 3 16:12:36 EST 2009 x86_64,sessions=? 15201,nsessions=? 15201,nusers=0,idletime=21840103,totmem=17464156kb,availmem=17170748kb,physmem=16444040kb,ncpus=8,loadave=0.00,netload=494140575539,state=free,jobs=,varattr=,rectime=1318502242 compute-0-16 state = free np = 16 properties = thirdgennodes ntype = cluster jobs = 0/48738.che-hydra.bham.ac.uk, 1/48691.che-hydra.bham.ac.uk, 3/48693.che-hydra.bham.ac.uk status = opsys=linux,uname=Linux compute-0-16.local 2.6.18-164.6.1.el5 #1 SMP Tue Nov 3 16:12:36 EST 2009 x86_64,sessions=6691 20687 20764,nsessions=3,nusers=2,idletime=7342084,totmem=17461096kb,availmem=8761384kb,physmem=16440980kb,ncpus=16,loadave=3.08,netload=647310098799,state=free,jobs=48691.che-hydra.bham.ac.uk 48693.che-hydra.bham.ac.uk 48738.che-hydra.bham.ac.uk,varattr=,rectime=1318502257 [root at che-hydra]# qmgr -c "p s" # # Create queues and set their attributes. # # # Create and define queue default # create queue default set queue default queue_type = Execution set queue default Priority = 100 set queue default resources_default.nodes = 1 set queue default enabled = True set queue default started = True # # Create and define queue secondgen # create queue secondgen set queue secondgen queue_type = Execution set queue secondgen Priority = 100 set queue secondgen acl_host_enable = False set queue secondgen acl_hosts = che-hydra+localhost set queue secondgen resources_default.neednodes = secondgennodes set queue secondgen resources_default.nodes = 1 set queue secondgen enabled = True set queue secondgen started = True # # Create and define queue thirdgen # create queue thirdgen set queue thirdgen queue_type = Execution set queue thirdgen Priority = 100 set queue thirdgen acl_host_enable = False set queue thirdgen acl_hosts = che-hydra+localhost set queue thirdgen resources_default.neednodes = thirdgennodes set queue thirdgen resources_default.nodes = 1 set queue thirdgen enabled = True set queue thirdgen started = True # # Create and define queue firstgen # create queue firstgen set queue firstgen queue_type = Execution set queue firstgen Priority = 100 set queue firstgen acl_host_enable = False set queue firstgen acl_hosts = che-hydra+localhost set queue firstgen resources_default.neednodes = firstgennodes set queue firstgen resources_default.nodes = 1 set queue firstgen enabled = True set queue firstgen started = True # # Set server attributes. # set server scheduling = True set server acl_host_enable = False set server acl_hosts = che-hydra.bham.ac.uk set server default_queue = default set server log_events = 511 set server mail_from = adm set server scheduler_iteration = 600 set server node_check_rate = 150 set server tcp_timeout = 6 set server auto_node_np = True set server next_job_number = 49702 Jonathan Smale From matt.morabito at gmail.com Thu Oct 13 08:44:53 2011 From: matt.morabito at gmail.com (Matthew Morabito) Date: Thu, 13 Oct 2011 10:44:53 -0400 Subject: [Mauiusers] Stacking jobs using Maui/Running on shared nodes Message-ID: I'm using maui v. 3.3.1 with pbs/torque v.2.4.11 and it seems maui is refusing to share different jobs on the same nodes (from maui.log): 10/12 12:13:59 INFO: located resources for 64 tasks (76) in best partition DEFAULT for job 1519 at time 00:00:00 10/12 12:13:59 ERROR: invalid nodelist for job 1519:0 (inadequate nodecount, 29 < 32) 10/12 12:13:59 WARNING: cannot allocate tasks for job 1519 at 00:00:00 There are clearly enough processors (76) free for the job (which needs 64, at 32 nodes/2 processors per node) and then maui decides that it can't start despite having enough processors to do so. Where is the configuration for allowing the stacking of jobs in maui? My maui.cfg has: BACKFILLPOLICY FIRSTFIT RESERVATIONPOLICY CURRENTHIGHEST RESERVATIONDEPTH = 20 NODEAVAILABILITYPOLICY UTILIZED NODEACCESSPOLICY SHARED NODEALLOCATIONPOLICY CPULOAD #also has a line for each node: NODECFG[node01] MAXJOB=2 ... etc Any help would be greatly appreciated. Thanks, Matthew -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.supercluster.org/pipermail/mauiusers/attachments/20111013/8e3147f3/attachment.html From brianm at usc.edu Thu Oct 13 09:34:39 2011 From: brianm at usc.edu (Brian Mendenhall) Date: Thu, 13 Oct 2011 08:34:39 -0700 Subject: [Mauiusers] Assigning Nodes to specific nodes In-Reply-To: References: Message-ID: <1318520079.15612.412.camel@orion.usc.edu> On Thu, 2011-10-13 at 11:08 +0000, Jonathan Smale wrote: > > set queue firstgen resources_default.neednodes = 1stgennodes > set nodes compute-0-0 properties = 1stgennodes > Not sure if you were just trying to be brief or if it was simply a typo, but here you have said you set the node compute-0-0's properties to '1stgennodes', and later you have it as 'firstgennodes'. If there is a difference, then that could be the problem. > I've found a fair few previous emails about this and have followed their solution without sucess. I can submit the jobs to the queues but they remain in a queued state, qstat -f gives:- > > [root at che-hydra /]# qstat -f 48691 > Job Id: 48691.che-hydra.bham.ac.uk > Job_Name = allenega_p2000_g300_r2.cmd > Job_Owner = jsmale at che-hydra.bham.ac.uk > resources_used.cput = 529:06:32 > resources_used.mem = 16872kb > resources_used.vmem = 245908kb > resources_used.walltime = 529:11:31 > job_state = R I do believe that 'job_state = R' means it is running from torque's perspective... Someone correct me if I'm wrong, please. > queue = default Please note the queue it is in, default, *not* firstgen, this is due to the definition of the default queue, below I'll comment more ... > > checking job 48691 > > State: Running > Creds: user:jsmale group:worth class:default qos:DEFAULT According to Maui, the job is running as well, in class 'default' > [root at che-hydra]# qmgr -c "p s" > # > # Create queues and set their attributes. > # > # > # Create and define queue default > # > create queue default > set queue default queue_type = Execution > set queue default Priority = 100 > set queue default resources_default.nodes = 1 > set queue default enabled = True > set queue default started = True So, I think that here is the root of your problem. You have configured the first queue that a job will be set to (in the server config below) is the 'default' queue, but this queue is an execution queue which means that jobs are going to 'run' in this queue. What I believe you want this queue to be is a routing queue, with route_destinations of the queues you want your jobs to run in. For example, my default queue configuration is thus: create queue default set queue default queue_type = Route set queue default max_queuable = 2000 set queue default resources_default.nodes = 1 set queue default resources_default.walltime = 00:30:00 set queue default route_destinations = quick set queue default route_destinations += long set queue default route_destinations += large_long set queue default route_destinations += large_route set queue default route_destinations += main_route set queue default route_waiting_jobs = True set queue default route_lifetime = 600 set queue default enabled = True set queue default started = True Please take note that torque will run the job in the FIRST place that it *can* run in (based on neednodes, resources_min, resources_max), so you should also order your queues appropriately. > # > # Set server attributes. > # > set server scheduling = True > set server acl_host_enable = False > set server acl_hosts = che-hydra.bham.ac.uk > set server default_queue = default ^^^ this is the important line that sets the first queue destination. All jobs that do not specify a queue name will go through this named queue. I hope this helps, -- Brian Mendenhall Linux/HPCC Administrator University of Southern California From jkusznir at gmail.com Thu Oct 13 11:22:29 2011 From: jkusznir at gmail.com (Jim Kusznir) Date: Thu, 13 Oct 2011 10:22:29 -0700 Subject: [Mauiusers] maui (or torque?) policy on node runtime and allocation Message-ID: Hi all: I would like to configure maui on my cluster to ensure that at least 60% of the cluster is available within 24 hours. The goal here is to prevent users from consuming the majority of the cluster on jobs that take more than a day (thus causing turn-around problems for other users, many of whom have short jobs to queue). I originally tried to do this with a default queue (max_walltime=24hrs) and a long queue (no max walltime), with the intent to restrict that long cannot use more than 33% of the nodes. However, I haven't figured out how to do that. At this point, its that that important to me that I do get it exactly like that, but if there's a way to ensure that at least 66% of the online cluster will be available within 24hrs, then that will work. I've been trying to figure out how to do that with maui's docs, but I don't really know what I'm looking for, so I haven't been able to locate it. Thanks! --Jim From scrusan at ur.rochester.edu Thu Oct 13 11:45:49 2011 From: scrusan at ur.rochester.edu (Steve Crusan) Date: Thu, 13 Oct 2011 13:45:49 -0400 Subject: [Mauiusers] maui (or torque?) policy on node runtime and allocation In-Reply-To: References: Message-ID: <0DB1632C-46D6-4907-A55F-3B03123059BE@ur.rochester.edu> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Oct 13, 2011, at 1:22 PM, Jim Kusznir wrote: > Hi all: > > I would like to configure maui on my cluster to ensure that at least > 60% of the cluster is available within 24 hours. The goal here is to > prevent users from consuming the majority of the cluster on jobs that > take more than a day (thus causing turn-around problems for other > users, many of whom have short jobs to queue). > > I originally tried to do this with a default queue > (max_walltime=24hrs) and a long queue (no max walltime), with the > intent to restrict that long cannot use more than 33% of the nodes. > However, I haven't figured out how to do that. At this point, its > that that important to me that I do get it exactly like that, but if > there's a way to ensure that at least 66% of the online cluster will > be available within 24hrs, then that will work. I was in a similar situation, where we had some infiniband nodes that we wanted to be dual purpose. Basically, if no one is using the infiniband functionality, normal ethernet jobs should run on those nodes. What I didn't want to happen was that users whom queued their jobs to use the infiniband queue would have to wait for normal jobs to finish. Especially if you consider our normal jobs span 5 days, someone could be waiting in the infiniband queue a long time. Even worse, what if they wanted to run jobs that span ALL the nodes??? That could take quite awhile to be scheduled. Now, mind you, I did this with Moab, but I don't see how this wouldn't work with Maui. Basically, I created a standing reservation with the proposed infiniband nodes, and set a max job time to 2 days. # SR - ibres # only allow jobs running less than 48 hours to use the infiniband nodes SRCFG[ibres] QOSLIST=ibqos,shared MAXTIME=*48:00:00 SRCFG[ibres] HOSTLIST=bh07[1-9],bh08[0-4] SRCFG[ibres] FLAGS=IGNSTATE,NOCHARGE SRCFG[ibres] PERIOD=INFINITY The QOS pieces are particular to our setup, but this should work. In a but shell, any jobs that are part of the shared OR ibqos QOS's have access to those nodes. The QOS's are mapped via CLASSCFG... A benefit of this is that MANY short running jobs are placed on these nodes, so we've seeing better scheduling efficiency also. This might work for you. > > I've been trying to figure out how to do that with maui's docs, but I > don't really know what I'm looking for, so I haven't been able to > locate it. > > Thanks! > --Jim > _______________________________________________ > mauiusers mailing list > mauiusers at supercluster.org > http://www.supercluster.org/mailman/listinfo/mauiusers ---------------------- Steve Crusan System Administrator Center for Research Computing University of Rochester https://www.crc.rochester.edu/ -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org iQEcBAEBAgAGBQJOlyPUAAoJENS19LGOpgqKAsEH/igWGQn6QHhc75ZxXauX19Vj hbVuX4iFTbojqrkx1C093pVpDuZNGLAnboYU7fR7lP7d5+bsrdshpIB4EHVwNTFy bIJ7TEzpSbNsWyCD/SVimLuK26PMkTvDM0vzLYRE9bLv+EZ84D4Oy6ZT/tYbtaz1 M0Ey+BHWgWq9PeM0WgjnGUXlTMZeNiH+DdA0aSff8gZiQpBnIv3KYrgBnNpbKDEn jpKabFn7LggLt5qvcc2pV2kCXdBl2/QA43px1QVyOxWUEknZcBg0Xte2RkDRoBUs M/HEUhVHZooVYDTlWcic+vEyz2KmY6XQpC60RVkdrBf/HPrGOD5cQoAaBRMbO8k= =98W5 -----END PGP SIGNATURE----- From JRS221 at bham.ac.uk Tue Oct 18 09:24:55 2011 From: JRS221 at bham.ac.uk (Jonathan Smale) Date: Tue, 18 Oct 2011 15:24:55 +0000 Subject: [Mauiusers] Assigning Nodes to specific nodes Message-ID: Dear Maui & Torque users, Following on from my previous email, where Brian Mendenhall was good enough to help, my post did have a couple of typos. I am trying to create separate queues for the three different types of nodes in our cluster that the user can specify to submit to. Submitting to the default queue works fine but submitting to the specific queues results in the jobs being queued indefinitely. The problem appears to be shown on the last line of the 'checkjob -v' command: cannot select job 1015 for partition DEFAULT (Class) the same checkjob command shows the class to be firstgen, which has the following settings according to the "qmgr -c 'p s'" command: # # Create and define queue firstgen # create queue firstgen set queue firstgen queue_type = Execution set queue firstgen Priority = 100 set queue firstgen acl_host_enable = False set queue firstgen acl_hosts = che-hydra+localhost set queue firstgen resources_default.neednodes = firstgennodes set queue firstgen resources_default.nodes = 1 set queue firstgen enabled = True set queue firstgen started = True # # Set server attributes. # set server scheduling = True set server acl_host_enable = False set server acl_hosts = che-hydra.bham.ac.uk set server default_queue = default set server log_events = 511 set server mail_from = adm set server scheduler_iteration = 600 set server node_check_rate = 150 set server tcp_timeout = 6 set server auto_node_np = True set server next_job_number = 1022 The important line being the "set queue firstgen resources_default.neednodes = firstgennodes" which requires a paticular property of the node. Both the pbsnodes command and the TORQUE_HOME/server_priv/nodes file show that there are nodes available with this property: compute-0-1 state = free np = 4 properties = firstgennodes ntype = cluster status = opsys=linux,uname=Linux compute-0-1.local 2.6.18-164.6.1.el5 #1 SMP Tue Nov 3 16:12:36 EST 2009 x86_64,sessions=? 15201,nsessions=? 15201,nusers=0,idletime=22288885,totmem=9195716kb,availmem=8914764kb,physmem=8175600kb,ncpus=4,loadave=0.00,netload=260038400155,state=free,jobs=,varattr=,rectime=1318951067 compute-0-1 np=4 firstgennodes Both qstat -f and tracejob show no errors, just that it is queued. In my maui.cfg file this is also specified: NODECFG[compute-0-0] SPEED=1 MAXJOB=4 nodetype=firstgennodes CLASSCFG[firstgen] hostlist = compute-0-0,compute-0-1,compute-0-2,compute-0-3 The only thing that looks wrong to me is the results of the diagnose command. When running diagnose -j on a job submitted to the firstgen queue I receive the following: [root at che-hydra home]# diagnose -j 1015 Name State Par Proc QOS WCLimit R Min User Group Account QueuedTime Network Opsys Arch Mem Disk Procs Class Features 1015 Idle ALL 1 DEF 99:23:59:59 0 1 jsmale worth - 00:39:35 [NONE] [NONE] [NONE] >=0 >=0 NC0 [firstgen:1] [firstgennodes] Which shows 2 class features, I may be misreading this output but it doesn't look right to me. What is really worrying is that the diagnose -Q command doesn't show the firstgen queue at all: diagnose -Q QOS Status System QOS Settings: QList: DEFAULT:0 (Def: DEFAULT) Flags: 0 Name * Priority QTWeight QTTarget XFWeight XFTarget QFlags JobFlags Limits DEFAULT 0 0 0 0 0.00 [NONE] [NONE] [NONE] [ALL] 0 0 0 0 0.00 [NONE] [NONE] [NONE] Any help would be appreciated. Jonathan Smale Postgraduate Research Student School of Chemistry From fcaba at uns.edu.ar Tue Oct 18 13:35:31 2011 From: fcaba at uns.edu.ar (Fernando Caba) Date: Tue, 18 Oct 2011 16:35:31 -0300 Subject: [Mauiusers] =?iso-8859-1?q?Can=B4t_get_busy_nodes?= In-Reply-To: References: <4E824CF2.2080204@uns.edu.ar> <4E825736.8080103@ldeo.columbia.edu> <4E829149.7060309@uns.edu.ar> Message-ID: <4E9DD503.3030100@uns.edu.ar> Hi everybody, finaly my cluster is working fine. I suggested the users to use this script: #!/bin/bash #PBS -l nodes=1:ppn=4 cd $PBS_O_WORKDIR mpirun -np 4 /usr/local/vasp/vasp or #!/bin/bash #PBS -l nodes=1:ppn=8 cd $PBS_O_WORKDIR mpirun -np 8 /usr/local/vasp/vasp ALWAYS paying atention to the numbers of ppn, indicating the same amount in the line mpirun.... and the line #PBS ...... I will try with CPUSET later, as Antonio Messina (amessina at ictp.it) indicated me. Thanks everybody ---------------------------------------------------- Ing. Fernando Caba Director General de Telecomunicaciones Universidad Nacional del Sur http://www.dgt.uns.edu.ar Tel/Fax: (54)-291-4595166 Tel: (54)-291-4595101 int. 2050 Avda. Alem 1253, (B8000CPB) Bah?a Blanca - Argentina ---------------------------------------------------- El 28/09/2011 12:26 AM, Denis escribi?: > Hello, Fernando! > > *it goes in English because my Portuguese would probably not help. ;) > > 2011/9/28 Fernando Caba: >> Hi Gus, my node file /var/spool/torque /server_priv/nodes looks like: >> >> [root at fe server_priv]# more nodes >> n10 np=12 >> n11 np=12 >> n12 np=12 >> n13 np=12 >> [root at fe server_priv]# >> >> it is exact as your comment. >> >> My script: >> >> #!/bin/bash >> >> cd $PBS_O_WORKDIR >> >> mpirun -np 8 /usr/local/vasp/vasp >> > You are missing to inform pbs that you are using 8 cores. > you have to add before anything runs in your script a line: > #PBS -lnodes=8 > > Torque cannot trace the number of mpi processes. A user could request > 4 cpus and start n mpi processes for example. > > So, requesting it in your script as > #PBS -lnodes=8 > then running mpirun -np 8 will do the job. > > > Regards, > From stevenx.a.duchene at intel.com Tue Oct 25 11:30:53 2011 From: stevenx.a.duchene at intel.com (DuChene, StevenX A) Date: Tue, 25 Oct 2011 10:30:53 -0700 Subject: [Mauiusers] user root not authorized to run commands Message-ID: Hello all: I have a small 256 node atom based cluster setup running maui 3.2.6p21 and torque 2.5.7 I have the following in my maui.cfg file: SERVERHOST myhost.atom ADMIN1 root,maui RMCFG[EMCUTIL1.ENDEAVOUR.ATOM] TYPE=PBS AMCFG[bank] TYPE=NONE RMPOLLINTERVAL 00:00:30 SERVERPORT 42559 SERVERMODE NORMAL LOGFILE maui.log LOGFILEMAXSIZE 10000000 LOGLEVEL 3 QUEUETIMEWEIGHT 1 BACKFILLPOLICY FIRSTFIT RESERVATIONPOLICY CURRENTHIGHEST NODEALLOCATIONPOLICY MINRESOURCE Whenever I try to run various maui diagnostic commands as root I get the following sort of message: # diagnose -j 2 ERROR: 'diagnose' failed ERROR: user 'root' is not authorized to execute command 'diagnose' # showstats ERROR: 'showstats' failed ERROR: user 'root' is not authorized to execute command 'showstats' I can however run these same commands just fine as the maui user. Is there something I am overlooking? I thought the only thing I needed to do was list the root user in the ADMIN1 line of my maui.cfg file. -- Steven DuChene From lance at quantumbioinc.com Wed Oct 26 09:31:28 2011 From: lance at quantumbioinc.com (Lance Westerhoff) Date: Wed, 26 Oct 2011 11:31:28 -0400 Subject: [Mauiusers] torque/maui disregarding pmem with procs Message-ID: <6BC7322E-43F5-4D17-8372-B6FB9D2DA965@quantumbioinc.com> Hello all- (I sent this email to the torque list, but I'm wondering if it might be a maui problem). We are trying to use procs= and pmem= on an 18 node (152core) cluster with nodes of various memory size. pbsnodes shows the correct memory complement for each node, so apparently PBS is getting the right specs (see the output of pbsnodes below for more information). If we use the following settings in the PBS script, invariably torque/maui will try to fill up the all 8 of the 8 cores of each node. That is even though there is nowhere near enough memory on any of these nodes for 8*3700mb=29600mb. Considering the physical memory limit goes from 8GB to 24GB depending upon the node, this is just taking down nodes left and right. Below I have provided a small example along with the associated output. I also provided the output for pbsnodes in case there is something I am missing here. Thanks for your help! -Lance torque version: tried 2.5.4, 2.5.8, and 3.0.2 - all exhibit the same problem. maui version: 3.2.6p21 (also tried maui 3.3.1 but it is a complete fail in terms of the procs option and it only asks for a single CPU) $ cat tmp.pbs #!/bin/bash #PBS -S /bin/bash #PBS -l procs=24 #PBS -l pmem=3700mb #PBS -l walltime=6:00:00 #PBS -j oe cat $PBS_NODEFILE $ qsub tmp.pbs 337003.XXXX $ wc -l tmp.pbs.o337003 24 tmp.pbs.o337003 $ cat tmp.pbs.o337003 compute-0-14 compute-0-14 compute-0-14 compute-0-14 compute-0-14 compute-0-14 compute-0-14 compute-0-14 compute-0-15 compute-0-15 compute-0-15 compute-0-15 compute-0-15 compute-0-15 compute-0-15 compute-0-15 compute-0-16 compute-0-16 compute-0-16 compute-0-16 compute-0-16 compute-0-16 compute-0-16 compute-0-16 $ pbsnodes -a compute-0-16 state = free np = 8 ntype = cluster status = rectime=1319219085,varattr=,jobs=,state=free,netload=1834011936,gres=,loadave=0.00,ncpus=8,physmem=8177300kb,availmem=10095652kb,totmem=10225576kb,idletime=5582,nusers=0,nsessions=? 0,sessions=? 0,uname=Linux compute-0-16.local 2.6.18-274.7.1.el5 #1 SMP Thu Oct 20 16:21:01 EDT 2011 x86_64,opsys=linux gpus = 0 compute-0-15 state = free np = 8 ntype = cluster status = rectime=1319219090,varattr=,jobs=,state=free,netload=700017694,gres=,loadave=0.00,ncpus=8,physmem=8177300kb,availmem=10150996kb,totmem=10225576kb,idletime=5606,nusers=0,nsessions=? 0,sessions=? 0,uname=Linux compute-0-15.local 2.6.18-274.7.1.el5 #1 SMP Thu Oct 20 16:21:01 EDT 2011 x86_64,opsys=linux gpus = 0 compute-0-14 state = free np = 8 ntype = cluster status = rectime=1319219090,varattr=,jobs=,state=free,netload=1003164957,gres=,loadave=0.00,ncpus=8,physmem=8177300kb,availmem=10131180kb,totmem=10225576kb,idletime=5615,nusers=0,nsessions=? 0,sessions=? 0,uname=Linux compute-0-14.local 2.6.18-274.7.1.el5 #1 SMP Thu Oct 20 16:21:01 EDT 2011 x86_64,opsys=linux gpus = 0 compute-0-13 state = free np = 8 ntype = cluster status = rectime=1319219090,varattr=,jobs=,state=free,netload=1173266470,gres=,loadave=0.00,ncpus=8,physmem=8177300kb,availmem=10132104kb,totmem=10225576kb,idletime=5637,nusers=0,nsessions=? 0,sessions=? 0,uname=Linux compute-0-13.local 2.6.18-274.7.1.el5 #1 SMP Thu Oct 20 16:21:01 EDT 2011 x86_64,opsys=linux gpus = 0 compute-0-12 state = free np = 8 ntype = cluster status = rectime=1319219090,varattr=,jobs=,state=free,netload=3991477,gres=,loadave=0.00,ncpus=8,physmem=12301956kb,availmem=14276448kb,totmem=14350232kb,idletime=5604,nusers=0,nsessions=? 0,sessions=? 0,uname=Linux compute-0-12.local 2.6.18-274.7.1.el5 #1 SMP Thu Oct 20 16:21:01 EDT 2011 x86_64,opsys=linux gpus = 0 compute-0-11 state = free np = 8 ntype = cluster status = rectime=1319219090,varattr=,jobs=,state=free,netload=2947879,gres=,loadave=0.00,ncpus=8,physmem=12301956kb,availmem=14274604kb,totmem=14350232kb,idletime=5588,nusers=0,nsessions=? 0,sessions=? 0,uname=Linux compute-0-11.local 2.6.18-274.7.1.el5 #1 SMP Thu Oct 20 16:21:01 EDT 2011 x86_64,opsys=linux gpus = 0 compute-0-9 state = free np = 8 ntype = cluster status = rectime=1319219090,varattr=,jobs=,state=free,netload=3721396,gres=,loadave=0.05,ncpus=8,physmem=12301956kb,availmem=14253816kb,totmem=14350232kb,idletime=5660,nusers=0,nsessions=? 0,sessions=? 0,uname=Linux compute-0-9.local 2.6.18-274.7.1.el5 #1 SMP Thu Oct 20 16:21:01 EDT 2011 x86_64,opsys=linux gpus = 0 compute-0-8 state = free np = 8 ntype = cluster status = rectime=1319219090,varattr=,jobs=,state=free,netload=2934478,gres=,loadave=0.00,ncpus=8,physmem=12301956kb,availmem=14254796kb,totmem=14350232kb,idletime=5675,nusers=0,nsessions=? 0,sessions=? 0,uname=Linux compute-0-8.local 2.6.18-274.7.1.el5 #1 SMP Thu Oct 20 16:21:01 EDT 2011 x86_64,opsys=linux gpus = 0 compute-0-7 state = free np = 8 ntype = cluster status = rectime=1319219090,varattr=,jobs=,state=free,netload=2909406,gres=,loadave=0.00,ncpus=8,physmem=12301956kb,availmem=14254812kb,totmem=14350232kb,idletime=5489,nusers=0,nsessions=? 0,sessions=? 0,uname=Linux compute-0-7.local 2.6.18-274.7.1.el5 #1 SMP Thu Oct 20 16:21:01 EDT 2011 x86_64,opsys=linux gpus = 0 compute-0-6 state = free np = 8 ntype = cluster status = rectime=1319219090,varattr=,jobs=,state=free,netload=2936791,gres=,loadave=0.00,ncpus=8,physmem=12301956kb,availmem=14275644kb,totmem=14350232kb,idletime=5748,nusers=0,nsessions=? 0,sessions=? 0,uname=Linux compute-0-6.local 2.6.18-274.7.1.el5 #1 SMP Thu Oct 20 16:21:01 EDT 2011 x86_64,opsys=linux gpus = 0 compute-0-5 state = free np = 8 ntype = cluster status = rectime=1319219090,varattr=,jobs=,state=free,netload=2966183,gres=,loadave=0.00,ncpus=8,physmem=12301956kb,availmem=14276260kb,totmem=14350232kb,idletime=5695,nusers=0,nsessions=? 0,sessions=? 0,uname=Linux compute-0-5.local 2.6.18-274.7.1.el5 #1 SMP Thu Oct 20 16:21:01 EDT 2011 x86_64,opsys=linux gpus = 0 compute-0-4 state = free np = 8 ntype = cluster status = rectime=1319219090,varattr=,jobs=,state=free,netload=2886627,gres=,loadave=0.00,ncpus=8,physmem=16438900kb,availmem=18412332kb,totmem=18487176kb,idletime=5634,nusers=0,nsessions=? 0,sessions=? 0,uname=Linux compute-0-4.local 2.6.18-274.7.1.el5 #1 SMP Thu Oct 20 16:21:01 EDT 2011 x86_64,opsys=linux gpus = 0 compute-0-3 state = free np = 8 properties = lustre ntype = cluster status = rectime=1319219108,varattr=,jobs=,state=free,netload=436527254,gres=,loadave=0.00,ncpus=8,physmem=24688212kb,availmem=26636656kb,totmem=26736488kb,idletime=2224,nusers=0,nsessions=? 0,sessions=? 0,uname=Linux compute-0-3.local 2.6.18-274.7.1.el5 #1 SMP Thu Oct 20 16:21:01 EDT 2011 x86_64,opsys=linux gpus = 0 compute-0-2 state = free np = 8 properties = lustre ntype = cluster status = rectime=1319219106,varattr=,jobs=,state=free,netload=1184385,gres=,loadave=0.00,ncpus=8,physmem=24688212kb,availmem=26659668kb,totmem=26736488kb,idletime=2223,nusers=0,nsessions=? 0,sessions=? 0,uname=Linux compute-0-2.local 2.6.18-274.7.1.el5 #1 SMP Thu Oct 20 16:21:01 EDT 2011 x86_64,opsys=linux gpus = 0 compute-0-1 state = free np = 8 properties = lustre ntype = cluster status = rectime=1319219102,varattr=,jobs=,state=free,netload=1258074,gres=,loadave=0.00,ncpus=8,physmem=24688212kb,availmem=26657304kb,totmem=26736488kb,idletime=2228,nusers=0,nsessions=? 0,sessions=? 0,uname=Linux compute-0-1.local 2.6.18-274.7.1.el5 #1 SMP Thu Oct 20 16:21:01 EDT 2011 x86_64,opsys=linux gpus = 0 compute-0-0 state = free np = 8 ntype = cluster status = rectime=1319219090,varattr=,jobs=,state=free,netload=3416356,gres=,loadave=0.00,ncpus=8,physmem=24688212kb,availmem=26635624kb,totmem=26736488kb,idletime=5603,nusers=0,nsessions=? 0,sessions=? 0,uname=Linux compute-0-0.local 2.6.18-274.7.1.el5 #1 SMP Thu Oct 20 16:21:01 EDT 2011 x86_64,opsys=linux gpus = 0 compute-0-10 state = free np = 2 ntype = cluster status = rectime=1319219090,varattr=,jobs=,state=free,netload=283846193,gres=,loadave=0.23,ncpus=8,physmem=12301956kb,availmem=13762696kb,totmem=14350232kb,idletime=5622,nusers=1,nsessions=1,sessions=3410,uname=Linux compute-0-10.local 2.6.18-274.7.1.el5 #1 SMP Thu Oct 20 16:21:01 EDT 2011 x86_64,opsys=linux gpus = 0 compute-0-17 state = free np = 8 properties = testbox ntype = cluster status = rectime=1319219090,varattr=,jobs=,state=free,netload=2948331,gres=,loadave=0.00,ncpus=8,physmem=8177300kb,availmem=10144432kb,totmem=10225576kb,idletime=5558,nusers=0,nsessions=? 0,sessions=? 0,uname=Linux compute-0-17.local 2.6.18-274.7.1.el5 #1 SMP Thu Oct 20 16:21:01 EDT 2011 x86_64,opsys=linux gpus = 0 From scrusan at ur.rochester.edu Wed Oct 26 09:59:10 2011 From: scrusan at ur.rochester.edu (Steve Crusan) Date: Wed, 26 Oct 2011 11:59:10 -0400 Subject: [Mauiusers] user root not authorized to run commands In-Reply-To: References: Message-ID: <80E265B3-55E1-4DCE-91F5-34BDFE56A655@ur.rochester.edu> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Oct 25, 2011, at 1:30 PM, DuChene, StevenX A wrote: > Hello all: > I have a small 256 node atom based cluster setup running maui 3.2.6p21 and torque 2.5.7 > I have the following in my maui.cfg file: > > SERVERHOST myhost.atom > ADMIN1 root,maui > RMCFG[EMCUTIL1.ENDEAVOUR.ATOM] TYPE=PBS > AMCFG[bank] TYPE=NONE > RMPOLLINTERVAL 00:00:30 > SERVERPORT 42559 > SERVERMODE NORMAL > LOGFILE maui.log > LOGFILEMAXSIZE 10000000 > LOGLEVEL 3 > QUEUETIMEWEIGHT 1 > BACKFILLPOLICY FIRSTFIT > RESERVATIONPOLICY CURRENTHIGHEST > NODEALLOCATIONPOLICY MINRESOURCE > > Whenever I try to run various maui diagnostic commands as root I get the following sort of message: > # diagnose -j 2 > ERROR: 'diagnose' failed > ERROR: user 'root' is not authorized to execute command 'diagnose' > > # showstats > ERROR: 'showstats' failed > ERROR: user 'root' is not authorized to execute command 'showstats' > > I can however run these same commands just fine as the maui user. > > Is there something I am overlooking? I thought the only thing I needed to do was list the root user in the ADMIN1 line of my maui.cfg file. I believe you need spaces in-between the usernames: http://www.adaptivecomputing.com/resources/docs/maui/a.fparameters.php#admin1 After you apply that change, run the showconfig command to check and see if the ADMIN1 line is recognized, then have the scheduler reload the configuration. Also, the other thing to check would be to make sure maui is actually using the right configuration file. I believe Maui will run in a default configuration if it's environment isn't setup right. > -- > Steven DuChene > _______________________________________________ > mauiusers mailing list > mauiusers at supercluster.org > http://www.supercluster.org/mailman/listinfo/mauiusers ---------------------- Steve Crusan System Administrator Center for Research Computing University of Rochester https://www.crc.rochester.edu/ -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org iQEcBAEBAgAGBQJOqC5VAAoJENS19LGOpgqKqPkIAKqCVA57aLVObkpXQD4kutko fcKHEsKJ7gcDvfCKMIdFI7cx0z1voJ652ItkEz7LlAfjKdRx+RuZ7MPIurdL9ODY h/mGbGVsApcmSHCzphdnVVarSbUvTQOlqi33vt06fG2824NITEfSYHeJbwF73/JA JL8j9dej23+ZAkmtE3+EOghovdTnzrI6O9xdNBnaoUz2kFFWNjSMn25WA7xY0TsZ HvqDSfrAAbFY3qPi7ZIhZFh5u69Tjob7c/2rfo5yjA9ww6T9YIVVrpyKntdIcBai Pz3lYW3VVTrgmIQY/J1VNd50ngoLp7TO/268n7A5uRb8AHrKoDEB9FOAVjc6FGI= =p2Gj -----END PGP SIGNATURE----- From patrick.jaeger at fr.ibm.com Wed Oct 26 14:07:13 2011 From: patrick.jaeger at fr.ibm.com (Patrick Jaeger) Date: Wed, 26 Oct 2011 22:07:13 +0200 Subject: [Mauiusers] =?iso-8859-1?q?AUTO=3A_Patrick_Jaeger_is_out_of_the_o?= =?iso-8859-1?q?ffice_-_en_cong=E9s__=28returning_30/10/2011=29?= Message-ID: I am out of the office until 30/10/2011. Je suis en cong?s si besoin contactez mon manager Thierry Paugam qui saura me retrouver ... I am Out of office ( vacations) , if necessary please call my manager Thierry Paugam Note: This is an automated response to your message "mauiusers Digest, Vol 87, Issue 4" sent on 26/10/2011 17:33:16. This is the only notification you will receive while this person is away. From Gareth.Williams at csiro.au Wed Oct 26 17:07:23 2011 From: Gareth.Williams at csiro.au (Gareth.Williams at csiro.au) Date: Thu, 27 Oct 2011 10:07:23 +1100 Subject: [Mauiusers] torque/maui disregarding pmem with procs In-Reply-To: <6BC7322E-43F5-4D17-8372-B6FB9D2DA965@quantumbioinc.com> References: <6BC7322E-43F5-4D17-8372-B6FB9D2DA965@quantumbioinc.com> Message-ID: <007DECE986B47F4EABF823C1FBB19C620102BE58CAE4@exvic-mbx04.nexus.csiro.au> Hi Lance, Does maui locate appropriate nodes if you specify: -l procs=24,vmem=29600mb ? That's what I'd do. It will not limit the memory per process (loosely speaking) but the main problem is which nodes are allocated. Gareth > -----Original Message----- > From: Lance Westerhoff [mailto:lance at quantumbioinc.com] > Sent: Thursday, 27 October 2011 2:31 AM > To: mauiusers at supercluster.org > Subject: [Mauiusers] torque/maui disregarding pmem with procs > > > Hello all- > > (I sent this email to the torque list, but I'm wondering if it might be > a maui problem). > > We are trying to use procs= and pmem= on an 18 node (152core) cluster > with nodes of various memory size. pbsnodes shows the correct memory > complement for each node, so apparently PBS is getting the right specs > (see the output of pbsnodes below for more information). If we use the > following settings in the PBS script, invariably torque/maui will try > to fill up the all 8 of the 8 cores of each node. That is even though > there is nowhere near enough memory on any of these nodes for > 8*3700mb=29600mb. Considering the physical memory limit goes from 8GB > to 24GB depending upon the node, this is just taking down nodes left > and right. > > Below I have provided a small example along with the associated output. > I also provided the output for pbsnodes in case there is something I am > missing here. > > Thanks for your help! -Lance > > torque version: tried 2.5.4, 2.5.8, and 3.0.2 - all exhibit the same > problem. > maui version: 3.2.6p21 (also tried maui 3.3.1 but it is a complete fail > in terms of the procs option and it only asks for a single CPU) > > $ cat tmp.pbs > #!/bin/bash > #PBS -S /bin/bash > #PBS -l procs=24 > #PBS -l pmem=3700mb > #PBS -l walltime=6:00:00 > #PBS -j oe > > cat $PBS_NODEFILE > > $ qsub tmp.pbs > 337003.XXXX > $ wc -l tmp.pbs.o337003 > 24 tmp.pbs.o337003 > $ cat tmp.pbs.o337003 > compute-0-14 > compute-0-14 > compute-0-14 > compute-0-14 > compute-0-14 > compute-0-14 > compute-0-14 > compute-0-14 > compute-0-15 > compute-0-15 > compute-0-15 > compute-0-15 > compute-0-15 > compute-0-15 > compute-0-15 > compute-0-15 > compute-0-16 > compute-0-16 > compute-0-16 > compute-0-16 > compute-0-16 > compute-0-16 > compute-0-16 > compute-0-16 > > $ pbsnodes -a > compute-0-16 > state = free > np = 8 > ntype = cluster > status = > rectime=1319219085,varattr=,jobs=,state=free,netload=1834011936,gres=,l > oadave=0.00,ncpus=8,physmem=8177300kb,availmem=10095652kb,totmem=102255 > 76kb,idletime=5582,nusers=0,nsessions=? 0,sessions=? 0,uname=Linux > compute-0-16.local 2.6.18-274.7.1.el5 #1 SMP Thu Oct 20 16:21:01 EDT > 2011 x86_64,opsys=linux > gpus = 0 > > compute-0-15 > state = free > np = 8 > ntype = cluster > status = > rectime=1319219090,varattr=,jobs=,state=free,netload=700017694,gres=,lo > adave=0.00,ncpus=8,physmem=8177300kb,availmem=10150996kb,totmem=1022557 > 6kb,idletime=5606,nusers=0,nsessions=? 0,sessions=? 0,uname=Linux > compute-0-15.local 2.6.18-274.7.1.el5 #1 SMP Thu Oct 20 16:21:01 EDT > 2011 x86_64,opsys=linux > gpus = 0 > > compute-0-14 > state = free > np = 8 > ntype = cluster > status = > rectime=1319219090,varattr=,jobs=,state=free,netload=1003164957,gres=,l > oadave=0.00,ncpus=8,physmem=8177300kb,availmem=10131180kb,totmem=102255 > 76kb,idletime=5615,nusers=0,nsessions=? 0,sessions=? 0,uname=Linux > compute-0-14.local 2.6.18-274.7.1.el5 #1 SMP Thu Oct 20 16:21:01 EDT > 2011 x86_64,opsys=linux > gpus = 0 > > compute-0-13 > state = free > np = 8 > ntype = cluster > status = > rectime=1319219090,varattr=,jobs=,state=free,netload=1173266470,gres=,l > oadave=0.00,ncpus=8,physmem=8177300kb,availmem=10132104kb,totmem=102255 > 76kb,idletime=5637,nusers=0,nsessions=? 0,sessions=? 0,uname=Linux > compute-0-13.local 2.6.18-274.7.1.el5 #1 SMP Thu Oct 20 16:21:01 EDT > 2011 x86_64,opsys=linux > gpus = 0 > > compute-0-12 > state = free > np = 8 > ntype = cluster > status = > rectime=1319219090,varattr=,jobs=,state=free,netload=3991477,gres=,load > ave=0.00,ncpus=8,physmem=12301956kb,availmem=14276448kb,totmem=14350232 > kb,idletime=5604,nusers=0,nsessions=? 0,sessions=? 0,uname=Linux > compute-0-12.local 2.6.18-274.7.1.el5 #1 SMP Thu Oct 20 16:21:01 EDT > 2011 x86_64,opsys=linux > gpus = 0 > > compute-0-11 > state = free > np = 8 > ntype = cluster > status = > rectime=1319219090,varattr=,jobs=,state=free,netload=2947879,gres=,load > ave=0.00,ncpus=8,physmem=12301956kb,availmem=14274604kb,totmem=14350232 > kb,idletime=5588,nusers=0,nsessions=? 0,sessions=? 0,uname=Linux > compute-0-11.local 2.6.18-274.7.1.el5 #1 SMP Thu Oct 20 16:21:01 EDT > 2011 x86_64,opsys=linux > gpus = 0 > > compute-0-9 > state = free > np = 8 > ntype = cluster > status = > rectime=1319219090,varattr=,jobs=,state=free,netload=3721396,gres=,load > ave=0.05,ncpus=8,physmem=12301956kb,availmem=14253816kb,totmem=14350232 > kb,idletime=5660,nusers=0,nsessions=? 0,sessions=? 0,uname=Linux > compute-0-9.local 2.6.18-274.7.1.el5 #1 SMP Thu Oct 20 16:21:01 EDT > 2011 x86_64,opsys=linux > gpus = 0 > > compute-0-8 > state = free > np = 8 > ntype = cluster > status = > rectime=1319219090,varattr=,jobs=,state=free,netload=2934478,gres=,load > ave=0.00,ncpus=8,physmem=12301956kb,availmem=14254796kb,totmem=14350232 > kb,idletime=5675,nusers=0,nsessions=? 0,sessions=? 0,uname=Linux > compute-0-8.local 2.6.18-274.7.1.el5 #1 SMP Thu Oct 20 16:21:01 EDT > 2011 x86_64,opsys=linux > gpus = 0 > > compute-0-7 > state = free > np = 8 > ntype = cluster > status = > rectime=1319219090,varattr=,jobs=,state=free,netload=2909406,gres=,load > ave=0.00,ncpus=8,physmem=12301956kb,availmem=14254812kb,totmem=14350232 > kb,idletime=5489,nusers=0,nsessions=? 0,sessions=? 0,uname=Linux > compute-0-7.local 2.6.18-274.7.1.el5 #1 SMP Thu Oct 20 16:21:01 EDT > 2011 x86_64,opsys=linux > gpus = 0 > > compute-0-6 > state = free > np = 8 > ntype = cluster > status = > rectime=1319219090,varattr=,jobs=,state=free,netload=2936791,gres=,load > ave=0.00,ncpus=8,physmem=12301956kb,availmem=14275644kb,totmem=14350232 > kb,idletime=5748,nusers=0,nsessions=? 0,sessions=? 0,uname=Linux > compute-0-6.local 2.6.18-274.7.1.el5 #1 SMP Thu Oct 20 16:21:01 EDT > 2011 x86_64,opsys=linux > gpus = 0 > > compute-0-5 > state = free > np = 8 > ntype = cluster > status = > rectime=1319219090,varattr=,jobs=,state=free,netload=2966183,gres=,load > ave=0.00,ncpus=8,physmem=12301956kb,availmem=14276260kb,totmem=14350232 > kb,idletime=5695,nusers=0,nsessions=? 0,sessions=? 0,uname=Linux > compute-0-5.local 2.6.18-274.7.1.el5 #1 SMP Thu Oct 20 16:21:01 EDT > 2011 x86_64,opsys=linux > gpus = 0 > > compute-0-4 > state = free > np = 8 > ntype = cluster > status = > rectime=1319219090,varattr=,jobs=,state=free,netload=2886627,gres=,load > ave=0.00,ncpus=8,physmem=16438900kb,availmem=18412332kb,totmem=18487176 > kb,idletime=5634,nusers=0,nsessions=? 0,sessions=? 0,uname=Linux > compute-0-4.local 2.6.18-274.7.1.el5 #1 SMP Thu Oct 20 16:21:01 EDT > 2011 x86_64,opsys=linux > gpus = 0 > > compute-0-3 > state = free > np = 8 > properties = lustre > ntype = cluster > status = > rectime=1319219108,varattr=,jobs=,state=free,netload=436527254,gres=,lo > adave=0.00,ncpus=8,physmem=24688212kb,availmem=26636656kb,totmem=267364 > 88kb,idletime=2224,nusers=0,nsessions=? 0,sessions=? 0,uname=Linux > compute-0-3.local 2.6.18-274.7.1.el5 #1 SMP Thu Oct 20 16:21:01 EDT > 2011 x86_64,opsys=linux > gpus = 0 > > compute-0-2 > state = free > np = 8 > properties = lustre > ntype = cluster > status = > rectime=1319219106,varattr=,jobs=,state=free,netload=1184385,gres=,load > ave=0.00,ncpus=8,physmem=24688212kb,availmem=26659668kb,totmem=26736488 > kb,idletime=2223,nusers=0,nsessions=? 0,sessions=? 0,uname=Linux > compute-0-2.local 2.6.18-274.7.1.el5 #1 SMP Thu Oct 20 16:21:01 EDT > 2011 x86_64,opsys=linux > gpus = 0 > > compute-0-1 > state = free > np = 8 > properties = lustre > ntype = cluster > status = > rectime=1319219102,varattr=,jobs=,state=free,netload=1258074,gres=,load > ave=0.00,ncpus=8,physmem=24688212kb,availmem=26657304kb,totmem=26736488 > kb,idletime=2228,nusers=0,nsessions=? 0,sessions=? 0,uname=Linux > compute-0-1.local 2.6.18-274.7.1.el5 #1 SMP Thu Oct 20 16:21:01 EDT > 2011 x86_64,opsys=linux > gpus = 0 > > compute-0-0 > state = free > np = 8 > ntype = cluster > status = > rectime=1319219090,varattr=,jobs=,state=free,netload=3416356,gres=,load > ave=0.00,ncpus=8,physmem=24688212kb,availmem=26635624kb,totmem=26736488 > kb,idletime=5603,nusers=0,nsessions=? 0,sessions=? 0,uname=Linux > compute-0-0.local 2.6.18-274.7.1.el5 #1 SMP Thu Oct 20 16:21:01 EDT > 2011 x86_64,opsys=linux > gpus = 0 > > compute-0-10 > state = free > np = 2 > ntype = cluster > status = > rectime=1319219090,varattr=,jobs=,state=free,netload=283846193,gres=,lo > adave=0.23,ncpus=8,physmem=12301956kb,availmem=13762696kb,totmem=143502 > 32kb,idletime=5622,nusers=1,nsessions=1,sessions=3410,uname=Linux > compute-0-10.local 2.6.18-274.7.1.el5 #1 SMP Thu Oct 20 16:21:01 EDT > 2011 x86_64,opsys=linux > gpus = 0 > > compute-0-17 > state = free > np = 8 > properties = testbox > ntype = cluster > status = > rectime=1319219090,varattr=,jobs=,state=free,netload=2948331,gres=,load > ave=0.00,ncpus=8,physmem=8177300kb,availmem=10144432kb,totmem=10225576k > b,idletime=5558,nusers=0,nsessions=? 0,sessions=? 0,uname=Linux > compute-0-17.local 2.6.18-274.7.1.el5 #1 SMP Thu Oct 20 16:21:01 EDT > 2011 x86_64,opsys=linux > gpus = 0 > > From lance at quantumbioinc.com Wed Oct 26 22:15:47 2011 From: lance at quantumbioinc.com (Lance Westerhoff) Date: Thu, 27 Oct 2011 00:15:47 -0400 Subject: [Mauiusers] torque/maui disregarding pmem with procs In-Reply-To: <007DECE986B47F4EABF823C1FBB19C620102BE58CAE4@exvic-mbx04.nexus.csiro.au> References: <6BC7322E-43F5-4D17-8372-B6FB9D2DA965@quantumbioinc.com> <007DECE986B47F4EABF823C1FBB19C620102BE58CAE4@exvic-mbx04.nexus.csiro.au> Message-ID: Hi Gareth- The vmem and pvmem options doesn't seem to make any difference. Incidentally, since the different nodes have different size memory compliments, and I will be distributing multi-processor jobs across nodes, I absolutely need something that will allocate on a per-processor memory basis. Here's a more detailed rundown of what is available in the cluster: $ diagnose -n diagnosing node table (5120 slots) Name State Procs Memory Disk Swap Speed Opsys Arch Par Load Res Classes Network Features compute-0-16 Busy 0:8 489:7985 1:1 8341:9985 1.00 linux [NONE] DEF 8.02 005 [developer_8:8][lowprio_8:8][b [DEFAULT] [NONE] compute-0-15 Busy 0:8 1185:7985 1:1 9455:9985 1.00 linux [NONE] DEF 8.03 009 [developer_8:8][lowprio_8:8][b [DEFAULT] [NONE] compute-0-14 Busy 0:8 1185:7985 1:1 9453:9985 1.00 linux [NONE] DEF 8.00 001 [developer_8:8][lowprio_8:8][b [DEFAULT] [NONE] compute-0-13 Busy 0:8 1185:7985 1:1 9494:9985 1.00 linux [NONE] DEF 8.00 001 [developer_8:8][lowprio_8:8][b [DEFAULT] [NONE] compute-0-12 Busy 0:8 5213:12013 1:1 13340:14013 1.00 linux [NONE] DEF 8.04 001 [developer_8:8][lowprio_8:8][b [DEFAULT] [NONE] compute-0-11 Busy 0:8 5213:12013 1:1 11338:14013 1.00 linux [NONE] DEF 8.01 008 [developer_8:8][lowprio_8:8][b [DEFAULT] [NONE] compute-0-9 Busy 0:8 5213:12013 1:1 11402:14013 1.00 linux [NONE] DEF 8.00 008 [developer_8:8][lowprio_8:8][b [DEFAULT] [NONE] compute-0-8 Busy 0:8 5213:12013 1:1 11379:14013 1.00 linux [NONE] DEF 8.00 008 [developer_8:8][lowprio_8:8][b [DEFAULT] [NONE] compute-0-7 Busy 0:8 5213:12013 1:1 11391:14013 1.00 linux [NONE] DEF 8.01 008 [developer_8:8][lowprio_8:8][b [DEFAULT] [NONE] compute-0-6 Busy 0:8 5213:12013 1:1 11404:14013 1.00 linux [NONE] DEF 8.00 008 [developer_8:8][lowprio_8:8][b [DEFAULT] [NONE] compute-0-5 Busy 0:8 5213:12013 1:1 11371:14013 1.00 linux [NONE] DEF 8.00 008 [developer_8:8][lowprio_8:8][b [DEFAULT] [NONE] compute-0-4 Busy 0:8 9253:16053 1:1 15451:18053 1.00 linux [NONE] DEF 8.00 008 [developer_8:8][lowprio_8:8][b [DEFAULT] [NONE] compute-0-3 Busy 0:8 7725:24109 1:1 25353:26109 1.00 linux [NONE] DEF 8.00 001 [developer_8:8][lowprio_8:8][b [DEFAULT] [lustre] compute-0-2 Busy 0:8 7725:24109 1:1 25374:26109 1.00 linux [NONE] DEF 7.99 001 [developer_8:8][lowprio_8:8][b [DEFAULT] [lustre] compute-0-1 Busy 0:8 7725:24109 1:1 25146:26109 1.00 linux [NONE] DEF 7.94 001 [developer_8:8][lowprio_8:8][b [DEFAULT] [lustre] compute-0-0 Busy 0:8 7725:24109 1:1 25379:26109 1.00 linux [NONE] DEF 8.00 001 [developer_8:8][lowprio_8:8][b [DEFAULT] [NONE] compute-0-10 Busy 0:2 10313:12013 1:1 12673:14013 1.00 linux [NONE] DEF 2.62 002 [developer_2:2][lowprio_2:2][b [DEFAULT] [NONE] compute-0-17 Busy 0:8 1185:7985 1:1 7420:9985 1.00 linux [NONE] DEF 8.00 008 [developer_8:8][lowprio_8:8][b [DEFAULT] [testbox] ----- --- 0:138 92186:248518 18:18 255164:284518 Total Nodes: 18 (Active: 18 Idle: 0 Down: 0) Thanks! -Lance On Oct 26, 2011, at 7:07 PM, wrote: > Hi Lance, > > Does maui locate appropriate nodes if you specify: > -l procs=24,vmem=29600mb > ? > That's what I'd do. It will not limit the memory per process (loosely speaking) but the main problem is which nodes are allocated. > > Gareth > > >> -----Original Message----- >> From: Lance Westerhoff [mailto:lance at quantumbioinc.com] >> Sent: Thursday, 27 October 2011 2:31 AM >> To: mauiusers at supercluster.org >> Subject: [Mauiusers] torque/maui disregarding pmem with procs >> >> >> Hello all- >> >> (I sent this email to the torque list, but I'm wondering if it might be >> a maui problem). >> >> We are trying to use procs= and pmem= on an 18 node (152core) cluster >> with nodes of various memory size. pbsnodes shows the correct memory >> complement for each node, so apparently PBS is getting the right specs >> (see the output of pbsnodes below for more information). If we use the >> following settings in the PBS script, invariably torque/maui will try >> to fill up the all 8 of the 8 cores of each node. That is even though >> there is nowhere near enough memory on any of these nodes for >> 8*3700mb=29600mb. Considering the physical memory limit goes from 8GB >> to 24GB depending upon the node, this is just taking down nodes left >> and right. >> >> Below I have provided a small example along with the associated output. >> I also provided the output for pbsnodes in case there is something I am >> missing here. >> >> Thanks for your help! -Lance >> >> torque version: tried 2.5.4, 2.5.8, and 3.0.2 - all exhibit the same >> problem. >> maui version: 3.2.6p21 (also tried maui 3.3.1 but it is a complete fail >> in terms of the procs option and it only asks for a single CPU) >> >> $ cat tmp.pbs >> #!/bin/bash >> #PBS -S /bin/bash >> #PBS -l procs=24 >> #PBS -l pmem=3700mb >> #PBS -l walltime=6:00:00 >> #PBS -j oe >> >> cat $PBS_NODEFILE >> >> $ qsub tmp.pbs >> 337003.XXXX >> $ wc -l tmp.pbs.o337003 >> 24 tmp.pbs.o337003 >> $ cat tmp.pbs.o337003 >> compute-0-14 >> compute-0-14 >> compute-0-14 >> compute-0-14 >> compute-0-14 >> compute-0-14 >> compute-0-14 >> compute-0-14 >> compute-0-15 >> compute-0-15 >> compute-0-15 >> compute-0-15 >> compute-0-15 >> compute-0-15 >> compute-0-15 >> compute-0-15 >> compute-0-16 >> compute-0-16 >> compute-0-16 >> compute-0-16 >> compute-0-16 >> compute-0-16 >> compute-0-16 >> compute-0-16 >> >> $ pbsnodes -a >> compute-0-16 >> state = free >> np = 8 >> ntype = cluster >> status = >> rectime=1319219085,varattr=,jobs=,state=free,netload=1834011936,gres=,l >> oadave=0.00,ncpus=8,physmem=8177300kb,availmem=10095652kb,totmem=102255 >> 76kb,idletime=5582,nusers=0,nsessions=? 0,sessions=? 0,uname=Linux >> compute-0-16.local 2.6.18-274.7.1.el5 #1 SMP Thu Oct 20 16:21:01 EDT >> 2011 x86_64,opsys=linux >> gpus = 0 >> >> compute-0-15 >> state = free >> np = 8 >> ntype = cluster >> status = >> rectime=1319219090,varattr=,jobs=,state=free,netload=700017694,gres=,lo >> adave=0.00,ncpus=8,physmem=8177300kb,availmem=10150996kb,totmem=1022557 >> 6kb,idletime=5606,nusers=0,nsessions=? 0,sessions=? 0,uname=Linux >> compute-0-15.local 2.6.18-274.7.1.el5 #1 SMP Thu Oct 20 16:21:01 EDT >> 2011 x86_64,opsys=linux >> gpus = 0 >> >> compute-0-14 >> state = free >> np = 8 >> ntype = cluster >> status = >> rectime=1319219090,varattr=,jobs=,state=free,netload=1003164957,gres=,l >> oadave=0.00,ncpus=8,physmem=8177300kb,availmem=10131180kb,totmem=102255 >> 76kb,idletime=5615,nusers=0,nsessions=? 0,sessions=? 0,uname=Linux >> compute-0-14.local 2.6.18-274.7.1.el5 #1 SMP Thu Oct 20 16:21:01 EDT >> 2011 x86_64,opsys=linux >> gpus = 0 >> >> compute-0-13 >> state = free >> np = 8 >> ntype = cluster >> status = >> rectime=1319219090,varattr=,jobs=,state=free,netload=1173266470,gres=,l >> oadave=0.00,ncpus=8,physmem=8177300kb,availmem=10132104kb,totmem=102255 >> 76kb,idletime=5637,nusers=0,nsessions=? 0,sessions=? 0,uname=Linux >> compute-0-13.local 2.6.18-274.7.1.el5 #1 SMP Thu Oct 20 16:21:01 EDT >> 2011 x86_64,opsys=linux >> gpus = 0 >> >> compute-0-12 >> state = free >> np = 8 >> ntype = cluster >> status = >> rectime=1319219090,varattr=,jobs=,state=free,netload=3991477,gres=,load >> ave=0.00,ncpus=8,physmem=12301956kb,availmem=14276448kb,totmem=14350232 >> kb,idletime=5604,nusers=0,nsessions=? 0,sessions=? 0,uname=Linux >> compute-0-12.local 2.6.18-274.7.1.el5 #1 SMP Thu Oct 20 16:21:01 EDT >> 2011 x86_64,opsys=linux >> gpus = 0 >> >> compute-0-11 >> state = free >> np = 8 >> ntype = cluster >> status = >> rectime=1319219090,varattr=,jobs=,state=free,netload=2947879,gres=,load >> ave=0.00,ncpus=8,physmem=12301956kb,availmem=14274604kb,totmem=14350232 >> kb,idletime=5588,nusers=0,nsessions=? 0,sessions=? 0,uname=Linux >> compute-0-11.local 2.6.18-274.7.1.el5 #1 SMP Thu Oct 20 16:21:01 EDT >> 2011 x86_64,opsys=linux >> gpus = 0 >> >> compute-0-9 >> state = free >> np = 8 >> ntype = cluster >> status = >> rectime=1319219090,varattr=,jobs=,state=free,netload=3721396,gres=,load >> ave=0.05,ncpus=8,physmem=12301956kb,availmem=14253816kb,totmem=14350232 >> kb,idletime=5660,nusers=0,nsessions=? 0,sessions=? 0,uname=Linux >> compute-0-9.local 2.6.18-274.7.1.el5 #1 SMP Thu Oct 20 16:21:01 EDT >> 2011 x86_64,opsys=linux >> gpus = 0 >> >> compute-0-8 >> state = free >> np = 8 >> ntype = cluster >> status = >> rectime=1319219090,varattr=,jobs=,state=free,netload=2934478,gres=,load >> ave=0.00,ncpus=8,physmem=12301956kb,availmem=14254796kb,totmem=14350232 >> kb,idletime=5675,nusers=0,nsessions=? 0,sessions=? 0,uname=Linux >> compute-0-8.local 2.6.18-274.7.1.el5 #1 SMP Thu Oct 20 16:21:01 EDT >> 2011 x86_64,opsys=linux >> gpus = 0 >> >> compute-0-7 >> state = free >> np = 8 >> ntype = cluster >> status = >> rectime=1319219090,varattr=,jobs=,state=free,netload=2909406,gres=,load >> ave=0.00,ncpus=8,physmem=12301956kb,availmem=14254812kb,totmem=14350232 >> kb,idletime=5489,nusers=0,nsessions=? 0,sessions=? 0,uname=Linux >> compute-0-7.local 2.6.18-274.7.1.el5 #1 SMP Thu Oct 20 16:21:01 EDT >> 2011 x86_64,opsys=linux >> gpus = 0 >> >> compute-0-6 >> state = free >> np = 8 >> ntype = cluster >> status = >> rectime=1319219090,varattr=,jobs=,state=free,netload=2936791,gres=,load >> ave=0.00,ncpus=8,physmem=12301956kb,availmem=14275644kb,totmem=14350232 >> kb,idletime=5748,nusers=0,nsessions=? 0,sessions=? 0,uname=Linux >> compute-0-6.local 2.6.18-274.7.1.el5 #1 SMP Thu Oct 20 16:21:01 EDT >> 2011 x86_64,opsys=linux >> gpus = 0 >> >> compute-0-5 >> state = free >> np = 8 >> ntype = cluster >> status = >> rectime=1319219090,varattr=,jobs=,state=free,netload=2966183,gres=,load >> ave=0.00,ncpus=8,physmem=12301956kb,availmem=14276260kb,totmem=14350232 >> kb,idletime=5695,nusers=0,nsessions=? 0,sessions=? 0,uname=Linux >> compute-0-5.local 2.6.18-274.7.1.el5 #1 SMP Thu Oct 20 16:21:01 EDT >> 2011 x86_64,opsys=linux >> gpus = 0 >> >> compute-0-4 >> state = free >> np = 8 >> ntype = cluster >> status = >> rectime=1319219090,varattr=,jobs=,state=free,netload=2886627,gres=,load >> ave=0.00,ncpus=8,physmem=16438900kb,availmem=18412332kb,totmem=18487176 >> kb,idletime=5634,nusers=0,nsessions=? 0,sessions=? 0,uname=Linux >> compute-0-4.local 2.6.18-274.7.1.el5 #1 SMP Thu Oct 20 16:21:01 EDT >> 2011 x86_64,opsys=linux >> gpus = 0 >> >> compute-0-3 >> state = free >> np = 8 >> properties = lustre >> ntype = cluster >> status = >> rectime=1319219108,varattr=,jobs=,state=free,netload=436527254,gres=,lo >> adave=0.00,ncpus=8,physmem=24688212kb,availmem=26636656kb,totmem=267364 >> 88kb,idletime=2224,nusers=0,nsessions=? 0,sessions=? 0,uname=Linux >> compute-0-3.local 2.6.18-274.7.1.el5 #1 SMP Thu Oct 20 16:21:01 EDT >> 2011 x86_64,opsys=linux >> gpus = 0 >> >> compute-0-2 >> state = free >> np = 8 >> properties = lustre >> ntype = cluster >> status = >> rectime=1319219106,varattr=,jobs=,state=free,netload=1184385,gres=,load >> ave=0.00,ncpus=8,physmem=24688212kb,availmem=26659668kb,totmem=26736488 >> kb,idletime=2223,nusers=0,nsessions=? 0,sessions=? 0,uname=Linux >> compute-0-2.local 2.6.18-274.7.1.el5 #1 SMP Thu Oct 20 16:21:01 EDT >> 2011 x86_64,opsys=linux >> gpus = 0 >> >> compute-0-1 >> state = free >> np = 8 >> properties = lustre >> ntype = cluster >> status = >> rectime=1319219102,varattr=,jobs=,state=free,netload=1258074,gres=,load >> ave=0.00,ncpus=8,physmem=24688212kb,availmem=26657304kb,totmem=26736488 >> kb,idletime=2228,nusers=0,nsessions=? 0,sessions=? 0,uname=Linux >> compute-0-1.local 2.6.18-274.7.1.el5 #1 SMP Thu Oct 20 16:21:01 EDT >> 2011 x86_64,opsys=linux >> gpus = 0 >> >> compute-0-0 >> state = free >> np = 8 >> ntype = cluster >> status = >> rectime=1319219090,varattr=,jobs=,state=free,netload=3416356,gres=,load >> ave=0.00,ncpus=8,physmem=24688212kb,availmem=26635624kb,totmem=26736488 >> kb,idletime=5603,nusers=0,nsessions=? 0,sessions=? 0,uname=Linux >> compute-0-0.local 2.6.18-274.7.1.el5 #1 SMP Thu Oct 20 16:21:01 EDT >> 2011 x86_64,opsys=linux >> gpus = 0 >> >> compute-0-10 >> state = free >> np = 2 >> ntype = cluster >> status = >> rectime=1319219090,varattr=,jobs=,state=free,netload=283846193,gres=,lo >> adave=0.23,ncpus=8,physmem=12301956kb,availmem=13762696kb,totmem=143502 >> 32kb,idletime=5622,nusers=1,nsessions=1,sessions=3410,uname=Linux >> compute-0-10.local 2.6.18-274.7.1.el5 #1 SMP Thu Oct 20 16:21:01 EDT >> 2011 x86_64,opsys=linux >> gpus = 0 >> >> compute-0-17 >> state = free >> np = 8 >> properties = testbox >> ntype = cluster >> status = >> rectime=1319219090,varattr=,jobs=,state=free,netload=2948331,gres=,load >> ave=0.00,ncpus=8,physmem=8177300kb,availmem=10144432kb,totmem=10225576k >> b,idletime=5558,nusers=0,nsessions=? 0,sessions=? 0,uname=Linux >> compute-0-17.local 2.6.18-274.7.1.el5 #1 SMP Thu Oct 20 16:21:01 EDT >> 2011 x86_64,opsys=linux >> gpus = 0 >> >> > From avb at ssau.ru Fri Oct 28 06:53:35 2011 From: avb at ssau.ru (Alexandr Baskakov) Date: Fri, 28 Oct 2011 16:53:35 +0400 Subject: [Mauiusers] Allocate job on node only if it requested by job node properties Message-ID: <4EAAA5CF.1000508@ssau.ru> Hi. Can anybody help me, please. I need configure TORQUE 3.0.2/Maui 3.3 to allocate particular job on node, only if node has been requested by job node properties. For example, we have 3 nodes: n1 np=8 n2 np=8 n3 np=8 bigmem If I submit job with "-l nodes=2:ppn=8", it must run on n1 and n2. If nodes n1,n2 already executing a some jobs, then if I submit job with "-l nodes=2:ppn=8", it must been queued, and wait n1,n2. If I submit job with "-l nodes=1:ppn=8:bigmem+1:ppn=8", it must be run on n3,n1 or n3,n2. Other words, regular jobs, if node properties "bigmem" has not requested, should't be run on node n3. In Maui, there is a standing reservation parameters, that can be used SRCFG. SRCFG[bigmem] PERIOD=INFINITY SRCFG[bigmem] CLASSLIST=batch SRCFG[bigmem] HOSTLIST=n3 But there is no ACL properties in SRCFG to allow use node n3 only if "bigmem" node's properties requested by qsub. -- Alexandr Baskakov, Samara State Aerospace University e-mail: avb at ssau.ru