[Mauiusers] Memory resource

[Yaroslav Halchenko]

is there something I overlooked in documentation, so that my question is
not worth detailed answer due to RTFM?

On Tue, 21 Aug 2007, Yaroslav Halchenko wrote:

> Dear Maui People,

> Sorry for a possibly lame question.
> We have a cluster running
> maui                          3.2.6p20-snap.1176920941-1
> torque                        2.1.8-1

> before now, most of the tasks were cpu-bound, so I didn't request users
> to explicitly request memory resources for their jobs. I've setup
> virtual cpus so we have 1 real cpu per queue per node.  Lately we got
> more and more memory bound tasks so I am about to implement memory
> policy and to request users to provide sensible estimates for the memory
> while scheduling the tasks.  But I got surprised by the amounts of RAM
> maui is thinking to have per each node.  For instance node25 real amount
> of memory (RAM+swap) is 12GB whenever maui thinks 34GB (see screendump
> below). The same kind of weird proportion is for the other nodes as
> well.

> Should memory be specified manually per node via NODECFG in maui.cfg?

> Should I just get freshier snapshots?

> If I impose memory policy for the tasks (RESOURCELIMITPOLICY
> MEM:ALWAYS), how can I prevent users to abuse it and request too much
> memory for their tasks, effectively wasting the resources and forbidding other
> users to run their tasks even though in reality amount of memory available is
> appropriate? Is there a way to punish such strategy (ie comparing
> requested and really used memory for the task)?

> Thanks everyone in advance for ideas
> ,---
> | itanix:/etc/maui# checknode node25
> | checking node node25
> | State:   Running  (in current state for 00:00:00)
> | Expected State:     Idle   SyncDeadline: Sat Oct 24 08:26:40
> | Configured Resources: PROCS: 6  MEM: 34G  SWAP: 34G  DISK: 1M
> | Utilized   Resources: [NONE]
> | Dedicated  Resources: PROCS: 2  MEM: 1536M
> | Opsys:         linux  Arch:      [NONE]
> | Speed:      1.00  Load:       2.150
> | Network:    [DEFAULT]
> | Features:   [matlab][matlab5]
> | Attributes: [Batch]
> | Classes:    [long 2:2][verylong 0:2][medium 2:2][small 2:2][rumbalong 2:2][rumba 2:2][directors 2:2][rumbasvm 2:2][default 2:2]

> | Total Time:   INFINITY  Up:   INFINITY (99.24%)  Active:   INFINITY (59.04%)

> | Reservations:
> |   User 'SYSTEM.5'(x1)  -16:59:38 ->   INFINITY (  INFINITY)
> |     Blocked Resources at 00:00:00    Procs: 6/6 (100.00%)
> |   Job '377257'(x1)   -INFINITY -> 58:02:42:41 (83:08:00:00)
> |   Job '377898'(x1)  -9:10:55:42 -> 73:21:04:18 (83:08:00:00)
> | JobList:  377257,377898
> | ALERT:  node is overcommitted at time 00:00:00 (P: -2)
> | ALERT:  node is overcommitted at time 58:02:42:41 (P: -1)

> | itanix:/etc/maui# pbsnodes node25 | grep totmem
> |      status = opsys=linux,uname=Linux node25 2.6.18-4-amd64 #1 SMP Mon Mar 26 11:36:53 CEST 2007 x86_64,sessions=2744 3530 12613 19528 6598 14880 17067,nsessions=7,nusers=5,idletime=957600,totmem=12354868kb,availmem=5005220kb,physmem=6057428kb,ncpus=2,loadave=2.07,netload=121060477190,state=free,jobs=377257.itanix.ravana.rutgers.edu 377898.itanix.ravana.rutgers.edu,rectime=1187716369

> `---
-- 
                                  .-.
=------------------------------   /v\  ----------------------------=
Keep in touch                    // \\     (yoh@|www.)onerussian.com
Yaroslav Halchenko              /(   )\               ICQ#: 60653192
                   Linux User    ^^-^^    [175555]