[torqueusers] Help: Unauthorized Request

Aaron T Perry atp42 at cornell.edu
Tue Sep 27 13:15:55 MDT 2011


That might be part of the issue. I also just checked my system resources and
there is a lot more being used than i anticipated, I'm using 94% of
available memory, and most of the cores are operating at >60%. I was working
in a virtual machine to troubleshoot (I have multiple VM with different
configurations, not running simultaneously).

I'm using Open MPI version 1.4.3, and I did test it with a very basic
program to make sure it compiles and runs properly (I had no errors,
warnings, or other odd behavior).

My ultimate goal is to move some of the fixes I found on the VM back to our
cluster, but our sysadmin isn't very familiar with torque either, so trying
to get it work here was another one of my goals.

Are you using a commercial compiler to run the CESM? I was trying to get it
to work with gcc version 4.4.2, but was running into a multitude of
compilation errors.

I'll let you know if I get everything working on the virtual machine.

Thanks,
Aaron

On Tue, Sep 27, 2011 at 3:01 PM, Gus Correa <gus at ldeo.columbia.edu> wrote:

> Now I wonder if part of the problem is due to it being
> a virtual machine.
>
> - Does torque work in a virtual environment?
> - How does MPI {whatever MPI you're using] behave
> [works?, performs well?] in a virtual environment?
> - Does something as big as ccsm [your ultimate goal apparently]
> work in a virtual environment?
>
> Honestly, I don't really know.
>
> For what it is worth, we run ccsm/cesm in a Linux cluster with
> Torque, OpenMPI, etc.
> No virtualization, though.
>
> Gus Correa
>
> Aaron T Perry wrote:
> > This is a single machine, it's a virtual machine running on my Windows 7
> > desktop. Thanks, I'm trying your suggestion now.
> >
> > Thanks,
> > Aaron
> >
> > On Tue, Sep 27, 2011 at 2:10 PM, Gus Correa <gus at ldeo.columbia.edu
> > <mailto:gus at ldeo.columbia.edu>> wrote:
> >
> >     Aron
> >
> >     You can set the stack size unlimited in /etc/security/limits.conf
> >     (here along with locked memory and number of open files):
> >
> >     *   -   memlock     -1
> >     *   -   stack       -1
> >     *   -   nofile      4096
> >
> >     Granted that the above is RHEL/CentOS style,
> >     Debian/Ubuntu may be different/different file.
> >
> >     Also, you may want to check your /var/log/messages [or whatever
> Ubuntu
> >     uses for system logs] and see if it sheds more light into
> >     the pbs_server errors.
> >
> >     My guess is that you need consistent server names in server_name,
> >     server_priv/nodes [assuming your server is also a work
> >     node running pbs_mom],  mom_priv/config (for $pbsserver).
> >     My recollection is that these default to 'localhost' [and 127.0.0.1],
> >     if your installation is in a *single standalone machine*,
> >     but I am not sure.
> >     And you need right name resolution in /etc/hosts, as Mike Reppert
> >     and Jerry Smith pointed out.
> >
> >     Also, not related, but you need to enable scheduling [after the
> >     current problem is sorted out]:
> >
> >     qmgr -c 'set server scheduling = True'
> >
> >     Out of curiosity, is it a single machine or a small cluster?
> >
> >     I hope this helps,
> >     Gus Correa
> >
> >     Aaron T Perry wrote:
> >      > Yes, that I do i have, that was the first thing I came across when
> >      > looking through help online.
> >      >
> >      > And I added the nodes file with appropriate settings for my
> >     machine, but
> >      > I still get the same errors.
> >      >
> >      > I have a completely unrelated question. I'm doing all this to run
> a
> >      > model that I've been trying to port. I'm trying to figure out
> >     whether a
> >      > segmentation fault I'm getting at runtime (using mpirun
> >     ./ccsm.exe) is
> >      > due to a compiler error, or a stack/memory error (the code works
> >     on many
> >      > other machines, not necessarily the compiler I'm using though).
> >     If I can
> >      > install torque I can use an automated script that also
> >      > sets appropriate stack size, among other things. I am on 1
> computer,
> >      > with 1 node, and I have no desire to scale this instance of the
> >     model.
> >      > Basically I'm wondering if you think there might be an
> easier/better
> >      > alternative?
> >      >
> >      > Thank you,
> >      > Aaron
> >      >
> >      >
> >      > On Tue, Sep 27, 2011 at 1:04 PM, Smith, Jerry Don II
> >     <jdsmit at sandia.gov <mailto:jdsmit at sandia.gov>
> >      > <mailto:jdsmit at sandia.gov <mailto:jdsmit at sandia.gov>>> wrote:
> >      >
> >      >     $PBS_HOME/server_priv/nodes needs to encompass your compute
> nodes
> >      >
> >      >     node1 np=4 # or however many cores you have
> >      >     node2 np=4
> >      >
> >      >     Make sure that those nodes can be resolved via those names
> >     from the
> >      >     admin node.
> >      >
> >      >     Do you have $PBS_HOME/server_name file with the resolvable
> >     name of
> >      >     your admin server?
> >      >
> >      >     -Jerry
> >      >
> >      >     From: Aaron T Perry <atp42 at cornell.edu
> >     <mailto:atp42 at cornell.edu> <mailto:atp42 at cornell.edu
> >     <mailto:atp42 at cornell.edu>>>
> >      >     Reply-To: Torque Users Mailing List
> >     <torqueusers at supercluster.org <mailto:torqueusers at supercluster.org>
> >      >     <mailto:torqueusers at supercluster.org
> >     <mailto:torqueusers at supercluster.org>>>
> >      >     Date: Tue, 27 Sep 2011 12:58:50 -0400
> >      >
> >      >     To: Torque Users Mailing List <torqueusers at supercluster.org
> >     <mailto:torqueusers at supercluster.org>
> >      >     <mailto:torqueusers at supercluster.org
> >     <mailto:torqueusers at supercluster.org>>>
> >      >     Subject: Re: [torqueusers] Help: Unauthorized Request
> >      >
> >      >     I think I have, I needed to create the file, and I was unsure
> >     about
> >      >     the formatting required.
> >      >     This is what I have there.
> >      >
> >      >     # + + ubuntu atp42
> >      >
> >      >     Do I also need to create the nodes file in the
> torque>server_priv
> >      >     directory?
> >      >
> >      >     Thanks,
> >      >     Aaron
> >      >
> >      >     On Tue, Sep 27, 2011 at 12:40 PM, Smith, Jerry Don II
> >      >     <jdsmit at sandia.gov <mailto:jdsmit at sandia.gov>
> >     <mailto:jdsmit at sandia.gov <mailto:jdsmit at sandia.gov>>> wrote:
> >      >
> >      >         Have you set up hosts.equiv?
> >      >
> >      >         see:
> >     http://www.clusterresources.com/torquedocs/1.3advconfig.shtml
> >      >
> >      >
> >      >               1.3.2.1 Server Configuration Overview
> >      >
> >      >         There are several steps to ensure that the server and the
> >     nodes
> >      >         are completely aware of each other and able to communicate
> >      >         directly. Some of this configuration takes place within
> >     TORQUE
> >      >         directly using the *qmgr* command. Other configuration
> >     settings
> >      >         are managed using the *pbs_server*nodes file, DNS files
> >     such as
> >      >         /etc/hosts and the /etc/hosts.equiv file.
> >      >
> >      >
> >      >               1.3.2.2 Name Service Configuration
> >      >
> >      >         Each node, as well as the server, must be able to resolve
> the
> >      >         name of every node with which it will interact. This can
> be
> >      >         accomplished using /etc/hosts, *DNS*, *NIS*, or other
> >      >         mechanisms. In the case of /etc/hosts, the file can be
> shared
> >      >         across systems in most cases.
> >      >
> >      >
> >      >         -Jerry
> >      >
> >      >
> >      >         From: Aaron T Perry <atp42 at cornell.edu
> >     <mailto:atp42 at cornell.edu> <mailto:atp42 at cornell.edu
> >     <mailto:atp42 at cornell.edu>>>
> >      >         Reply-To: Torque Users Mailing List
> >      >         <torqueusers at supercluster.org
> >     <mailto:torqueusers at supercluster.org>
> >     <mailto:torqueusers at supercluster.org
> >     <mailto:torqueusers at supercluster.org>>>
> >      >         Date: Tue, 27 Sep 2011 12:33:31 -0400
> >      >
> >      >         To: Torque Users Mailing List
> >     <torqueusers at supercluster.org <mailto:torqueusers at supercluster.org>
> >      >         <mailto:torqueusers at supercluster.org
> >     <mailto:torqueusers at supercluster.org>>>
> >      >         Subject: Re: [torqueusers] Help: Unauthorized Request
> >      >
> >      >         With the execption of the unauthorized request entries it
> >     looks
> >      >         like almost everything is okay, execpt for the node file
> and
> >      >         root <at> localhost (this should be root <at> ubuntu.
> >      >
> >      >         Thank you for your help!
> >      >         Aaron
> >      >
> >      >         Here is an except from the server log...
> >      >
> >      >         09/27/2011 09:51:31;0002;PBS_Server;Svr;Log;Log opened
> >      >         09/27/2011 09:51:31;0006;PBS_Server;Svr;PBS_Server;Server
> >     ubuntu
> >      >         started, initialization type = 4
> >      >         09/27/2011 09:51:42;0002;PBS_Server;Svr;Log;Log opened
> >      >         09/27/2011 09:51:42;0006;PBS_Server;Svr;PBS_Server;Server
> >     ubuntu
> >      >         started, initialization type = 4
> >      >         09/27/2011 09:51:44;0002;PBS_Server;Svr;Act;Account file
> >      >         /var/spool/torque/server_priv/accounting/20110927 opened
> >      >         09/27/2011
> >     09:51:44;0040;PBS_Server;Req;setup_nodes;setup_nodes()
> >      >         09/27/2011 09:51:44;0004;PBS_Server;Svr;ubuntu;cannot
> >     open node
> >      >         description file '/var/spool/torque/server_priv/nodes' in
> >      >         setup_nodes()
> >      >         09/27/2011
> >     09:51:44;0002;PBS_Server;Svr;PBS_Server;Expected 0,
> >      >         recovered 0 queues
> >      >         09/27/2011
> >     09:51:44;0002;PBS_Server;Svr;PBS_Server;Expected 0,
> >      >         recovered 0 jobs
> >      >         09/27/2011 09:51:44;0006;PBS_Server;Svr;PBS_Server;Using
> >     ports
> >      >         Server:15001  Scheduler:15004  MOM:15002 (server:
> 'ubuntu')
> >      >         09/27/2011
> >     09:51:44;0002;PBS_Server;Svr;daemonize_server;INFO:
> >      >            parent is exiting
> >      >         09/27/2011
> >     09:51:44;0002;PBS_Server;Svr;daemonize_server;INFO:
> >      >            parent is exiting
> >      >         09/27/2011
> >     09:51:44;0002;PBS_Server;Svr;daemonize_server;INFO:
> >      >            child process in background
> >      >         09/27/2011 09:51:44;0002;PBS_Server;Svr;PBS_Server;Server
> >     Ready,
> >      >         pid = 11995, loglevel=0
> >      >         09/27/2011 09:51:44;0080;PBS_Server;Req;req_reject;Reject
> >     reply
> >      >         code=15007(Unauthorized Request ), aux=0, type=Manager,
> from
> >      >         root at localhost
> >      >         09/27/2011 09:51:44;0080;PBS_Server;Req;req_reject;Reject
> >     reply
> >      >         code=15007(Unauthorized Request ), aux=0, type=Manager,
> from
> >      >         root at localhost
> >      >         09/27/2011 09:51:44;0080;PBS_Server;Req;req_reject;Reject
> >     reply
> >      >         code=15007(Unauthorized Request ), aux=0, type=Manager,
> from
> >      >         root at localhost
> >      >         09/27/2011 09:51:44;0080;PBS_Server;Req;req_reject;Reject
> >     reply
> >      >         code=15007(Unauthorized Request ), aux=0, type=Manager,
> from
> >      >         root at localhost
> >      >         09/27/2011 09:51:44;0080;PBS_Server;Req;req_reject;Reject
> >     reply
> >      >         code=15007(Unauthorized Request ), aux=0, type=Manager,
> from
> >      >         root at localhost
> >      >         09/27/2011 09:51:44;0080;PBS_Server;Req;req_reject;Reject
> >     reply
> >      >         code=15007(Unauthorized Request ), aux=0, type=Manager,
> from
> >      >         root at localhost
> >      >         09/27/2011 09:51:44;0080;PBS_Server;Req;req_reject;Reject
> >     reply
> >      >         code=15007(Unauthorized Request ), aux=0, type=Manager,
> from
> >      >         root at localhost
> >      >         09/27/2011 09:51:44;0080;PBS_Server;Req;req_reject;Reject
> >     reply
> >      >         code=15007(Unauthorized Request ), aux=0, type=Manager,
> from
> >      >         root at localhost
> >      >         09/27/2011 09:51:44;0080;PBS_Server;Req;req_reject;Reject
> >     reply
> >      >         code=15007(Unauthorized Request ), aux=0, type=Manager,
> from
> >      >         root at localhost
> >      >         09/27/2011 09:51:44;0080;PBS_Server;Req;req_reject;Reject
> >     reply
> >      >         code=15007(Unauthorized Request ), aux=0, type=Manager,
> from
> >      >         root at localhost
> >      >         09/27/2011 09:51:44;0080;PBS_Server;Req;req_reject;Reject
> >     reply
> >      >         code=15007(Unauthorized Request ), aux=0, type=Manager,
> from
> >      >         root at localhost
> >      >         09/27/2011 09:51:44;0080;PBS_Server;Req;req_reject;Reject
> >     reply
> >      >         code=15007(Unauthorized Request ), aux=0, type=Manager,
> from
> >      >         root at localhost
> >      >         09/27/2011 09:51:49;0002;PBS_Server;Svr;PBS_Server;Torque
> >     Server
> >      >         Version = 3.0.2, loglevel = 0
> >      >         09/27/2011 09:56:49;0002;PBS_Server;Svr;PBS_Server;Torque
> >     Server
> >      >         Version = 3.0.2, loglevel = 0
> >      >         ...
> >      >
> >      >         On Tue, Sep 27, 2011 at 12:13 PM, Smith, Jerry Don II
> >      >         <jdsmit at sandia.gov <mailto:jdsmit at sandia.gov>
> >     <mailto:jdsmit at sandia.gov <mailto:jdsmit at sandia.gov>>> wrote:
> >      >
> >      >             Are you seeing anything in the pbs_server logs?
> >      >
> >      >             -Jerry
> >      >
> >      >             From: Aaron T Perry <atp42 at cornell.edu
> >     <mailto:atp42 at cornell.edu>
> >      >             <mailto:atp42 at cornell.edu <mailto:atp42 at cornell.edu
> >>>
> >      >             Reply-To: Torque Users Mailing List
> >      >             <torqueusers at supercluster.org
> >     <mailto:torqueusers at supercluster.org>
> >      >             <mailto:torqueusers at supercluster.org
> >     <mailto:torqueusers at supercluster.org>>>
> >      >             Date: Tue, 27 Sep 2011 09:53:28 -0400
> >      >             To: <torqueusers at supercluster.org
> >     <mailto:torqueusers at supercluster.org>
> >      >             <mailto:torqueusers at supercluster.org
> >     <mailto:torqueusers at supercluster.org>>>
> >      >             Subject: Re: [torqueusers] Help: Unauthorized Request
> >      >
> >      >             Please, any help you can give would be greatly
> >     appreciated,
> >      >             I'm completely stuck. All the solutions I found
> >     online have
> >      >             failed.
> >      >
> >      >             On Mon, Sep 26, 2011 at 2:35 PM, Aaron
> >     <atp42 at cornell.edu <mailto:atp42 at cornell.edu>
> >      >             <mailto:atp42 at cornell.edu
> >     <mailto:atp42 at cornell.edu>>> wrote:
> >      >
> >      >                 Hi,
> >      >
> >      >                 I've just tried to install torque, and I ran the
> >      >                 following commands,
> >      >
> >      >                 ./configure
> >      >                 sudo make
> >      >                 sudo make install
> >      >
> >      >                 however when I run ./torque.setup username I get
> the
> >      >                 following...
> >      >
> >      >                 initializing TORQUE (admin: username at ubuntu)
> >      >                 PBS_Server ubuntu: Create mode and server
> >     database exists,
> >      >                 do you wish to continue y/(n)?y
> >      >                 Max open servers: 9
> >      >                 qmgr obj= svr=default: Unauthorized Request
> >      >                 Max open servers: 9
> >      >                 qmgr obj= svr=default: Unauthorized Request
> >      >                 qmgr obj= svr=default: Unauthorized Request
> >      >                 qmgr obj= svr=default: Unauthorized Request
> >      >                 qmgr obj= svr=default: Unauthorized Request
> >      >                 qmgr obj=batch svr=default: Unauthorized Request
> >      >                 qmgr obj=batch svr=default: Unauthorized Request
> >      >                 qmgr obj=batch svr=default: Unauthorized Request
> >      >                 qmgr obj=batch svr=default: Unauthorized Request
> >      >                 qmgr obj=batch svr=default: Unauthorized Request
> >      >                 qmgr obj=batch svr=default: Unauthorized Request
> >      >                 qmgr obj= svr=default: Unauthorized Request
> >      >
> >      >                 The server lanched and I cannot stop it, nor can
> >     issue
> >      >                 any command related to torque (qterm, gmgr, qsub,
> >     etc)
> >      >                 under my current username or under root. Help!
> >      >
> >      >                 Thank you,
> >      >                 -Aaron
> >      >
> >      >
> >      >             _______________________________________________
> >     torqueusers
> >      >             mailing list torqueusers at supercluster.org
> >     <mailto:torqueusers at supercluster.org>
> >      >             <mailto:torqueusers at supercluster.org
> >     <mailto:torqueusers at supercluster.org>>
> >      >
> http://www.supercluster.org/mailman/listinfo/torqueusers
> >      >
> >      >             _______________________________________________
> >      >             torqueusers mailing list
> >      >             torqueusers at supercluster.org
> >     <mailto:torqueusers at supercluster.org>
> >      >             <mailto:torqueusers at supercluster.org
> >     <mailto:torqueusers at supercluster.org>>
> >      >
> http://www.supercluster.org/mailman/listinfo/torqueusers
> >      >
> >      >
> >      >         _______________________________________________
> torqueusers
> >      >         mailing list torqueusers at supercluster.org
> >     <mailto:torqueusers at supercluster.org>
> >      >         <mailto:torqueusers at supercluster.org
> >     <mailto:torqueusers at supercluster.org>>
> >      >         http://www.supercluster.org/mailman/listinfo/torqueusers
> >      >
> >      >         _______________________________________________
> >      >         torqueusers mailing list
> >      >         torqueusers at supercluster.org
> >     <mailto:torqueusers at supercluster.org>
> >     <mailto:torqueusers at supercluster.org
> >     <mailto:torqueusers at supercluster.org>>
> >      >         http://www.supercluster.org/mailman/listinfo/torqueusers
> >      >
> >      >
> >      >     _______________________________________________ torqueusers
> >     mailing
> >      >     list torqueusers at supercluster.org
> >     <mailto:torqueusers at supercluster.org>
> >      >     <mailto:torqueusers at supercluster.org
> >     <mailto:torqueusers at supercluster.org>>
> >      >     http://www.supercluster.org/mailman/listinfo/torqueusers
> >      >
> >      >     _______________________________________________
> >      >     torqueusers mailing list
> >      >     torqueusers at supercluster.org
> >     <mailto:torqueusers at supercluster.org>
> >     <mailto:torqueusers at supercluster.org
> >     <mailto:torqueusers at supercluster.org>>
> >      >     http://www.supercluster.org/mailman/listinfo/torqueusers
> >      >
> >      >
> >      >
> >      >
> >
> ------------------------------------------------------------------------
> >      >
> >      > _______________________________________________
> >      > torqueusers mailing list
> >      > torqueusers at supercluster.org <mailto:torqueusers at supercluster.org
> >
> >      > http://www.supercluster.org/mailman/listinfo/torqueusers
> >
> >     _______________________________________________
> >     torqueusers mailing list
> >     torqueusers at supercluster.org <mailto:torqueusers at supercluster.org>
> >     http://www.supercluster.org/mailman/listinfo/torqueusers
> >
> >
> >
> > ------------------------------------------------------------------------
> >
> > _______________________________________________
> > torqueusers mailing list
> > torqueusers at supercluster.org
> > http://www.supercluster.org/mailman/listinfo/torqueusers
>
> _______________________________________________
> torqueusers mailing list
> torqueusers at supercluster.org
> http://www.supercluster.org/mailman/listinfo/torqueusers
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.supercluster.org/pipermail/torqueusers/attachments/20110927/cadfeef5/attachment-0001.html 


More information about the torqueusers mailing list