showstats

showstats [FLAGS]

Purpose

Show resource usage statistics

Access

This command can be run by any Maui level 1, 2, or 3 Administrator.

Parameters

[NONE]

Flags

NOTE: this command supports all generic maui command flags

Flag Description

-a [<ACCOUNTID>] display account statistics

-g [<GROUPID>] display group statistics

-n [<NODEID>] display node statistics

-S display summary information. NOTE: only valid with the '-n' flag

-s display general scheduler statistics

-u [<USERID>] display user statistics

-v display verbose information

Description

This command shows various accounting statistics for the system. Historical statistics cover the timeframe from the most recent execution of the resetstats command.

Example 1

% showstats -a

Account Statistics Initialized Tue Aug 26 14:32:39

              |----- Running ------|--------------------------------- Completed ----------------------------------|
  Account     Jobs Procs ProcHours Jobs    %   PHReq    %    PHDed    %   FSTgt  AvgXF  MaxXF  AvgQH  Effic  WCAcc
   137651       16    92   1394.52  229  39.15 18486  45.26 7003.5  41.54 40.00   0.77   8.15   5.21  90.70  34.69
   462212       11    63    855.27   43   7.35  6028  14.76 3448.4  20.45  6.25   0.71   5.40   3.14  98.64  40.83
   462213        6    72    728.12   90  15.38  5974  14.63 3170.7  18.81  6.25   0.37   4.88   0.52  82.01  24.14
   005810        3    24    220.72   77  13.16  2537   6.21 1526.6   9.06 -----   1.53  14.81   0.42  98.73  28.40
   175436        0     0      0.00   12   2.05  6013  14.72  958.6   5.69  2.50   1.78   8.61   5.60  83.64  17.04
   000102        0     0      0.00    1   0.17    64   0.16    5.1   0.03 -----  10.85  10.85  10.77  27.90   7.40
   000023        0     0      0.00    1   0.17    12   0.03    0.2   0.00 -----   0.04   0.04   0.19  21.21   1.20

This example shows a statistical listing of all active accounts. The top line (Account Statistics Initialized...) of the output indicates the beginning of the timeframe covered by the displayed statistics.

The statistical output is divided into two categories, Running and Completed. Running statistics include information about jobs that are currently running. Completed statistics are compiled using historical information from both running and completed jobs.

The fields are as follows:

Account Account Number.

Jobs Number of running jobs.

Procs Number of processors allocated to running jobs.

ProcHours Number of proc-hours required to complete running jobs.

Jobs* Number of jobs completed.

% Percentage of total jobs that were completed by account.

PHReq* Total proc-hours requested by completed jobs.

% Percentage of total proc-hours requested by completed jobs that were requested by account.

PHDed Total proc-hours dedicated to active and completed jobs. The proc-hours dedicated to a job are calculated by multiplying the number of allocated procs by the length of time the procs were allocated, regardless of the job's CPU usage.

% Percentage of total proc-hours dedicated that were dedicated by account.

FSTgt Fairshare target. An account's fairshare target is specified in the fs.cfg file. This value should be compared to the account's node-hour dedicated percentage to determine if the target is being met.

AvgXF* Average expansion factor for jobs completed. A job's XFactor (expansion factor) is calculated by the following formula: (QueuedTime + RunTime) / WallClockLimit.

MaxXF* Highest expansion factor received by jobs completed.

AvgQH* Average queue time (in hours) of jobs.

Effic Average job efficiency. Job efficiency is calculated by dividing the actual node-hours of CPU time used by the job by the node-hours allocated to the job.

WCAcc* Average wall clock accuracy for jobs completed. Wall clock accuracy is calculated by dividing a job's actual run time by its specified wall clock limit.

* These fields are empty until an account has completed at least one job.

Example 2

% showstats -g

Group Statistics Initialized Tue Aug 26 14:32:39

              |----- Running ------|--------------------------------- Completed ----------------------------------|
GroupName  GID Jobs Procs ProcHours Jobs    %   PHReq    %    PHDed    %   FSTgt  AvgXF  MaxXF  AvgQH  Effic  WCAcc
     univ  214   16    92   1394.52  229  39.15 18486  45.26 7003.5  41.54 40.00   0.77   8.15   5.21  90.70  34.69
      daf  204   11    63    855.27   43   7.35  6028  14.76 3448.4  20.45  6.25   0.71   5.40   3.14  98.64  40.83
    dnavy  207    6    72    728.12   90  15.38  5974  14.63 3170.7  18.81  6.25   0.37   4.88   0.52  82.01  24.14
     govt  232    3    24    220.72   77  13.16  2537   6.21 1526.6   9.06 -----   1.53  14.81   0.42  98.73  28.40
      asp  227    0     0      0.00   12   2.05  6013  14.72  958.6   5.69  2.50   1.78   8.61   5.60  83.64  17.04
    derim  229    0     0      0.00   74  12.65   669   1.64  352.5   2.09 -----   0.50   1.93   0.51  96.03  32.60
   dchall  274    0     0      0.00    3   0.51   447   1.10  169.2   1.00 25.00   0.52   0.88   2.49  95.82  33.67
      nih  239    0     0      0.00   17   2.91   170   0.42  148.1   0.88 -----   0.95   1.83   0.14  97.59  84.31
    darmy  205    0     0      0.00   31   5.30   366   0.90   53.9   0.32  6.25   0.14   0.59   0.07  81.33  12.73
  systems   80    0     0      0.00    6   1.03    67   0.16   22.4   0.13 -----   4.07   8.49   1.23  28.68  37.34
      pdc  252    0     0      0.00    1   0.17    64   0.16    5.1   0.03 -----  10.85  10.85  10.77  27.90   7.40
    staff    1    0     0      0.00    1   0.17    12   0.03    0.2   0.00 -----   0.04   0.04   0.19  21.21   1.20

This example shows a statistical listing of all active groups. The top line (Group Statistics Initialized...) of the output indicates the beginning of the timeframe covered by the displayed statistics.

The fields are as follows:

GroupName Name of group.

GID Group ID of group.

Jobs Number of running jobs.

Procs Number of procs allocated to running jobs.

ProcHours Number of proc-hours required to complete running jobs.

Jobs* Number of jobs completed.

% Percentage of total jobs that were completed by group.

PHReq* Total proc-hours requested by completed jobs.

% Percentage of total proc-hours requested by completed jobs that were requested by group.

PHDed Total proc-hours dedicated to active and completed jobs. The proc-hours dedicated to a job are calculated by multiplying the number of allocated procs by the length of time the procs were allocated, regardless of the job's CPU usage.

% Percentage of total proce-hours dedicated that were dedicated by group.

FSTgt Fairshare target. A group's fairshare target is specified in the fs.cfg file. This value should be compared to the group's node-hour dedicated percentage to determine if the target is being met.

AvgXF* Average expansion factor for jobs completed. A job's XFactor (expansion factor) is calculated by the following formula: (QueuedTime + RunTime) / WallClockLimit.

MaxXF* Highest expansion factor received by jobs completed.

AvgQH* Average queue time (in hours) of jobs.

Effic Average job efficiency. Job efficiency is calculated by dividing the actual node-hours of CPU time used by the job by the node-hours allocated to the job.

WCAcc* Average wall clock accuracy for jobs completed. Wall clock accuracy is calculated by dividing a job's actual run time by its specified wall clock limit.

* These fields are empty until a group has completed at least one job.

Example 3

% showstats -n -S

Memory Requirement Breakdown:

  Memory Nodes Percent InitialNH Percent  NodeHours Percent
      64     8    2.78      9799  794.92       1232  100.00
     128   144   50.00      9162   41.29      22190  100.00
     256    32   11.11     20290  411.47       4931  100.00
     512    96   33.33      5080   34.34      14793  100.00
    1024     8    2.78        48    3.89       1232  100.00
    2048     0    0.00         0    0.00          0    0.00
   TOTAL   288  100.00     44381  100.00      44381  100.00


Node Statistics

Summary:    8   64MB Nodes   99.26% Avail   79.18% Busy  (Current: 100.00% Avail  100.00% Busy)
Summary:  144  128MB Nodes   98.99% Avail   75.92% Busy  (Current: 100.00% Avail  100.00% Busy)
Summary:   32  256MB Nodes   97.69% Avail   85.66% Busy  (Current: 100.00% Avail  100.00% Busy)
Summary:   96  512MB Nodes   96.12% Avail   82.92% Busy  (Current:  98.96% Avail   94.79% Busy)
Summary:    8 1024MB Nodes   99.87% Avail   81.77% Busy  (Current: 100.00% Avail   75.00% Busy)
System Summary:  288 Nodes   97.92% Avail   79.59% Busy  (Current:  99.65% Avail   97.57% Busy)

This example shows a statistical listing of nodes and memory. Memory Requirement Breakdown portion shows information about the current workload profile. In this example, the system monitored is a heterogeneous environment consisting of eight 64 MB (RAM) nodes, 144 128 MB nodes, etc., with a total of 288 nodes. The third column indicates the percentage of total nodes that meet this memory criteria. For example, the eight 64 MB nodes make up 2.78% of the 288 total nodes in the system.

The idle job queue monitored in this example consists of numerous jobs consisting of a total of 44,381 node-hours of work. The node-hour workload of jobs that have specific node memory requirements are assigned to the corresponding memory class. If no specific memory requirement is specified, the job's node-hours are assigned to the lowest memory class, in this case, the 64 MB nodes.

Example 4

% showstats

Maui running for 22:01:00 stats initialized on Mon Mar 26 17:43:34

Eligible/Idle Jobs:                   15/45         (33.333%)
Active Jobs:                          42
Successful/Completed Jobs:           873/875        (99.7%)
Avg/Max QTime (Hours):              2.71/4.50
Avg/Max XFactor:                    1.03/4.79

Dedicated/Total ProcHours: 4353.55/4782.10 (91.038%)

Current Active/Total Procs: 183/192 (95.312%)

Avg WallClock Accuracy:            43.25%
Avg Job Proc Efficiency:           98.17%
Est/Avg Backlog (Hours):            34.5/41.8

This example shows a concise summary of the system scheduling state. Note that showstats and showstats -s are equivalent.

The first line of output indicates the number of scheduling iterations performed by the current scheduling process, followed by the time the scheduler started. The second line indicates the amount of time the Maui Scheduler has been scheduling in HH:MM:SS notation followed by the statistics initialization time.

The fields are as follows:

Active Jobs Number of jobs currently active (Running or Starting).

Eligible Jobs Number of jobs in the system queue (jobs that are considered when scheduling).

Idle Jobs Number of jobs both in and out of the system queue that are in the LoadLeveler Idle state.

Completed Jobs Number of jobs completed since statistics were initialized.

Successful Jobs Jobs that completed successfully without abnormal termination.

XFactor Average expansion factor of all completed jobs.

Max XFactor Maximum expansion factor of completed jobs.

Max Bypass Maximum bypass of completed jobs.

Available ProcHours Total proc-hours available to the scheduler.

Dedicated ProcHours Total proc-hours made available to jobs.

Effic Scheduling efficiency (DedicatedProcHours / Available ProcHours).

Min Efficiency Minimum scheduling efficiency obtained since scheduler was started.

Iteration Iteration on which the minimum scheduling efficiency occurred.

Available Procs Number of procs currently available.

Busy Procs Number of procs currently busy.

Effic Current system efficiency (BusyProcs/AvailableProcs).

WallClock Accuracy Average wall clock accuracy of completed jobs (job-weighted average).

Job Efficiency Average job efficiency (UtilizedTime / DedicatedTime).

Est Backlog Estimated backlog of queued work in hours.

Avg Backlog Average backlog of queued work in hours.

Example 5

% showstats -u

User Statistics Initialized Tue Aug 26 14:32:39

              |----- Running ------|--------------------------------- Completed ----------------------------------|
 UserName  UID Jobs Procs ProcHours Jobs    %   PHReq    %    PHDed    %   FSTgt  AvgXF  MaxXF  AvgQH  Effic  WCAcc
  moorejt 2617    1    16     58.80    2   0.34   221   0.54 1896.6  11.25 -----   1.02   1.04   0.14  99.52 100.00
    zhong 1767    3    24    220.72   20   3.42  2306   5.65 1511.3   8.96 -----   0.71   0.96   0.49  99.37  67.48
      lui 2467    0     0      0.00   16   2.74  1970   4.82 1505.1   8.93 -----   1.02   6.33   0.25  98.96  57.72
    evans 3092    0     0      0.00   62  10.60  4960  12.14 1464.3   8.69   5.0   0.62   1.64   5.04  87.64  30.62
   wengel 2430    2    64    824.90    1   0.17   767   1.88  630.3   3.74 -----   0.18   0.18   4.26  99.63   0.40
    mukho 2961    2    16     71.06    6   1.03   776   1.90  563.5   3.34 -----   0.31   0.82   0.20  93.15  30.28
  jimenez 1449    1    16    302.29    2   0.34   768   1.88  458.3   2.72 -----   0.80   0.98   2.31  97.99  70.30
     neff 3194    0     0      0.00   74  12.65   669   1.64  352.5   2.09  10.0   0.50   1.93   0.51  96.03  32.60
   cholik 1303    0     0      0.00    2   0.34   552   1.35  281.9   1.67 -----   1.72   3.07  25.35  99.69  66.70
 jshoemak 2508    1    24    572.22    1   0.17   576   1.41  229.1   1.36 -----   0.55   0.55   3.74  99.20  39.20
     kudo 2324    1     8    163.35    6   1.03  1152   2.82  211.1   1.25 -----   0.12   0.34   1.54  96.77   5.67
   xztang 1835    1     8     18.99 ---- ------ ----- ------  176.3   1.05  10.0 ------ ------ ------  99.62 ------
   feller 1880    0     0      0.00   17   2.91   170   0.42  148.1   0.88 -----   0.95   1.83   0.14  97.59  84.31
    maxia 2936    0     0      0.00    1   0.17   191   0.47  129.1   0.77   7.5   0.88   0.88   4.49  99.84  69.10
 ktgnov71 2838    0     0      0.00    1   0.17   192   0.47   95.5   0.57 -----   0.53   0.53   0.34  90.07  51.20

This example shows a statistical listing of all active users. The top line (User Statistics Initialized...) of the output indicates the timeframe covered by the displayed statistics.

The statistical output is divided into two statistics categories, Running and Completed. Running statistics include information about jobs that are currently running. Completed statistics are compiled using historical information from both running and completed jobs.

The fields are as follows:

UserName Name of user.

UID User ID of user.

Jobs Number of running jobs.

Procs Number of procs allocated to running jobs.

ProcHours Number of proc-hours required to complete running jobs.

Jobs* Number of jobs completed.

% Percentage of total jobs that were completed by user.

PHReq* Total proc-hours requested by completed jobs.

% Percentage of total proc-hours requested by completed jobs that were requested by user.

PHDed Total proc-hours dedicated to active and completed jobs. The proc-hours dedicated to a job are calculated by multiplying the number of allocated procs by the length of time the procs were allocated, regardless of the job's CPU usage.

% Percentage of total prochours dedicated that were dedicated by user.

FSTgt Fairshare target. A user's fairshare target is specified in the fs.cfg file. This value should be compared to the user's node-hour dedicated percentage to determine if the target is being met.

AvgXF* Average expansion factor for jobs completed. A job's XFactor (expansion factor) is calculated by the following formula: (QueuedTime + RunTime) / WallClockLimit.

MaxXF* Highest expansion factor received by jobs completed.

AvgQH* Average queue time (in hours) of jobs.

Effic Average job efficiency. Job efficiency is calculated by dividing the actual node-hours of CPU time used by the job by the node-hours allocated to the job.

WCAcc* Average wall clock accuracy for jobs completed. Wall clock accuracy is calculated by dividing a job's actual run time by its specified wall clock limit.

* These fields are empty until a user has completed at least one job.

Related Commands

Use the resetstats command to re-initialize statistics.

Notes

See the Statistics document for more details about scheduler statistics.