[torqueusers] Help: Unauthorized Request

Gus Correa gus at ldeo.columbia.edu
Tue Sep 27 13:41:54 MDT 2011


Aaron T Perry wrote:
> That might be part of the issue. I also just checked my system resources 
> and there is a lot more being used than i anticipated, I'm using 94% of 
> available memory, and most of the cores are operating at >60%. I was 
> working in a virtual machine to troubleshoot (I have multiple VM with 
> different configurations, not running simultaneously).
> 

If this is while running ccsm, you should expect near 100% core use.
As for RAM, it depends on how much you have and which ccsm configuration
you are trying to run.
Ccsm uses a lot of resources, even on small configurations.
If you are paging memory, that is not a good sign,
and may as well explain a segfault.
Anyway, this is more of a NCAR CGD Bulletin Board discussion
than a Torque discussion.

> I'm using Open MPI version 1.4.3, and I did test it with a very basic 
> program to make sure it compiles and runs properly (I had no errors, 
> warnings, or other odd behavior).

You're right.
Testing with the basic programs [connectivity_c.c, ring_c.c, hello_c.c]
is the right thing to do.

> 
> My ultimate goal is to move some of the fixes I found on the VM back to 
> our cluster, but our sysadmin isn't very familiar with torque either, so 
> trying to get it work here was another one of my goals.
> 

My guess is that your development platform [VM] is more complicated
than your production one [cluster]
Have you tried  the cluster directly?
Does it have torque installed?
You could either ask the sysadmin to install torque [if not available]
and OpenMPI --with-tm support.
Or, if torque is not available and there is no resource manager,
just install OpenMPI in your home directory [assuming it is NFS mounted
on the compute nodes] and run ccsm straight from mpiexec.


> Are you using a commercial compiler to run the CESM? I was trying to get 
> it to work with gcc version 4.4.2, but was running into a multitude of 
> compilation errors.
> 

I haven't tried it with gcc/gfortran, but it may work.
I have been using Intel compilers icc/ifort and OpenMPI 1.4.3 also.
There are several road blocks to ccsm/cesm.
The compiler may be the lesser of them.

Gus Correa

> I'll let you know if I get everything working on the virtual machine.
> 
> Thanks,
> Aaron
> 
> On Tue, Sep 27, 2011 at 3:01 PM, Gus Correa <gus at ldeo.columbia.edu 
> <mailto:gus at ldeo.columbia.edu>> wrote:
> 
>     Now I wonder if part of the problem is due to it being
>     a virtual machine.
> 
>     - Does torque work in a virtual environment?
>     - How does MPI {whatever MPI you're using] behave
>     [works?, performs well?] in a virtual environment?
>     - Does something as big as ccsm [your ultimate goal apparently]
>     work in a virtual environment?
> 
>     Honestly, I don't really know.
> 
>     For what it is worth, we run ccsm/cesm in a Linux cluster with
>     Torque, OpenMPI, etc.
>     No virtualization, though.
> 
>     Gus Correa
> 
>     Aaron T Perry wrote:
>      > This is a single machine, it's a virtual machine running on my
>     Windows 7
>      > desktop. Thanks, I'm trying your suggestion now.
>      >
>      > Thanks,
>      > Aaron
>      >
>      > On Tue, Sep 27, 2011 at 2:10 PM, Gus Correa
>     <gus at ldeo.columbia.edu <mailto:gus at ldeo.columbia.edu>
>      > <mailto:gus at ldeo.columbia.edu <mailto:gus at ldeo.columbia.edu>>> wrote:
>      >
>      >     Aron
>      >
>      >     You can set the stack size unlimited in /etc/security/limits.conf
>      >     (here along with locked memory and number of open files):
>      >
>      >     *   -   memlock     -1
>      >     *   -   stack       -1
>      >     *   -   nofile      4096
>      >
>      >     Granted that the above is RHEL/CentOS style,
>      >     Debian/Ubuntu may be different/different file.
>      >
>      >     Also, you may want to check your /var/log/messages [or
>     whatever Ubuntu
>      >     uses for system logs] and see if it sheds more light into
>      >     the pbs_server errors.
>      >
>      >     My guess is that you need consistent server names in server_name,
>      >     server_priv/nodes [assuming your server is also a work
>      >     node running pbs_mom],  mom_priv/config (for $pbsserver).
>      >     My recollection is that these default to 'localhost' [and
>     127.0.0.1],
>      >     if your installation is in a *single standalone machine*,
>      >     but I am not sure.
>      >     And you need right name resolution in /etc/hosts, as Mike Reppert
>      >     and Jerry Smith pointed out.
>      >
>      >     Also, not related, but you need to enable scheduling [after the
>      >     current problem is sorted out]:
>      >
>      >     qmgr -c 'set server scheduling = True'
>      >
>      >     Out of curiosity, is it a single machine or a small cluster?
>      >
>      >     I hope this helps,
>      >     Gus Correa
>      >
>      >     Aaron T Perry wrote:
>      >      > Yes, that I do i have, that was the first thing I came
>     across when
>      >      > looking through help online.
>      >      >
>      >      > And I added the nodes file with appropriate settings for my
>      >     machine, but
>      >      > I still get the same errors.
>      >      >
>      >      > I have a completely unrelated question. I'm doing all this
>     to run a
>      >      > model that I've been trying to port. I'm trying to figure out
>      >     whether a
>      >      > segmentation fault I'm getting at runtime (using mpirun
>      >     ./ccsm.exe) is
>      >      > due to a compiler error, or a stack/memory error (the code
>     works
>      >     on many
>      >      > other machines, not necessarily the compiler I'm using
>     though).
>      >     If I can
>      >      > install torque I can use an automated script that also
>      >      > sets appropriate stack size, among other things. I am on 1
>     computer,
>      >      > with 1 node, and I have no desire to scale this instance
>     of the
>      >     model.
>      >      > Basically I'm wondering if you think there might be an
>     easier/better
>      >      > alternative?
>      >      >
>      >      > Thank you,
>      >      > Aaron
>      >      >
>      >      >
>      >      > On Tue, Sep 27, 2011 at 1:04 PM, Smith, Jerry Don II
>      >     <jdsmit at sandia.gov <mailto:jdsmit at sandia.gov>
>     <mailto:jdsmit at sandia.gov <mailto:jdsmit at sandia.gov>>
>      >      > <mailto:jdsmit at sandia.gov <mailto:jdsmit at sandia.gov>
>     <mailto:jdsmit at sandia.gov <mailto:jdsmit at sandia.gov>>>> wrote:
>      >      >
>      >      >     $PBS_HOME/server_priv/nodes needs to encompass your
>     compute nodes
>      >      >
>      >      >     node1 np=4 # or however many cores you have
>      >      >     node2 np=4
>      >      >
>      >      >     Make sure that those nodes can be resolved via those names
>      >     from the
>      >      >     admin node.
>      >      >
>      >      >     Do you have $PBS_HOME/server_name file with the resolvable
>      >     name of
>      >      >     your admin server?
>      >      >
>      >      >     -Jerry
>      >      >
>      >      >     From: Aaron T Perry <atp42 at cornell.edu
>     <mailto:atp42 at cornell.edu>
>      >     <mailto:atp42 at cornell.edu <mailto:atp42 at cornell.edu>>
>     <mailto:atp42 at cornell.edu <mailto:atp42 at cornell.edu>
>      >     <mailto:atp42 at cornell.edu <mailto:atp42 at cornell.edu>>>>
>      >      >     Reply-To: Torque Users Mailing List
>      >     <torqueusers at supercluster.org
>     <mailto:torqueusers at supercluster.org>
>     <mailto:torqueusers at supercluster.org
>     <mailto:torqueusers at supercluster.org>>
>      >      >     <mailto:torqueusers at supercluster.org
>     <mailto:torqueusers at supercluster.org>
>      >     <mailto:torqueusers at supercluster.org
>     <mailto:torqueusers at supercluster.org>>>>
>      >      >     Date: Tue, 27 Sep 2011 12:58:50 -0400
>      >      >
>      >      >     To: Torque Users Mailing List
>     <torqueusers at supercluster.org <mailto:torqueusers at supercluster.org>
>      >     <mailto:torqueusers at supercluster.org
>     <mailto:torqueusers at supercluster.org>>
>      >      >     <mailto:torqueusers at supercluster.org
>     <mailto:torqueusers at supercluster.org>
>      >     <mailto:torqueusers at supercluster.org
>     <mailto:torqueusers at supercluster.org>>>>
>      >      >     Subject: Re: [torqueusers] Help: Unauthorized Request
>      >      >
>      >      >     I think I have, I needed to create the file, and I was
>     unsure
>      >     about
>      >      >     the formatting required.
>      >      >     This is what I have there.
>      >      >
>      >      >     # + + ubuntu atp42
>      >      >
>      >      >     Do I also need to create the nodes file in the
>     torque>server_priv
>      >      >     directory?
>      >      >
>      >      >     Thanks,
>      >      >     Aaron
>      >      >
>      >      >     On Tue, Sep 27, 2011 at 12:40 PM, Smith, Jerry Don II
>      >      >     <jdsmit at sandia.gov <mailto:jdsmit at sandia.gov>
>     <mailto:jdsmit at sandia.gov <mailto:jdsmit at sandia.gov>>
>      >     <mailto:jdsmit at sandia.gov <mailto:jdsmit at sandia.gov>
>     <mailto:jdsmit at sandia.gov <mailto:jdsmit at sandia.gov>>>> wrote:
>      >      >
>      >      >         Have you set up hosts.equiv?
>      >      >
>      >      >         see:
>      >     http://www.clusterresources.com/torquedocs/1.3advconfig.shtml
>      >      >
>      >      >
>      >      >               1.3.2.1 Server Configuration Overview
>      >      >
>      >      >         There are several steps to ensure that the server
>     and the
>      >     nodes
>      >      >         are completely aware of each other and able to
>     communicate
>      >      >         directly. Some of this configuration takes place
>     within
>      >     TORQUE
>      >      >         directly using the *qmgr* command. Other configuration
>      >     settings
>      >      >         are managed using the *pbs_server*nodes file, DNS
>     files
>      >     such as
>      >      >         /etc/hosts and the /etc/hosts.equiv file.
>      >      >
>      >      >
>      >      >               1.3.2.2 Name Service Configuration
>      >      >
>      >      >         Each node, as well as the server, must be able to
>     resolve the
>      >      >         name of every node with which it will interact.
>     This can be
>      >      >         accomplished using /etc/hosts, *DNS*, *NIS*, or other
>      >      >         mechanisms. In the case of /etc/hosts, the file
>     can be shared
>      >      >         across systems in most cases.
>      >      >
>      >      >
>      >      >         -Jerry
>      >      >
>      >      >
>      >      >         From: Aaron T Perry <atp42 at cornell.edu
>     <mailto:atp42 at cornell.edu>
>      >     <mailto:atp42 at cornell.edu <mailto:atp42 at cornell.edu>>
>     <mailto:atp42 at cornell.edu <mailto:atp42 at cornell.edu>
>      >     <mailto:atp42 at cornell.edu <mailto:atp42 at cornell.edu>>>>
>      >      >         Reply-To: Torque Users Mailing List
>      >      >         <torqueusers at supercluster.org
>     <mailto:torqueusers at supercluster.org>
>      >     <mailto:torqueusers at supercluster.org
>     <mailto:torqueusers at supercluster.org>>
>      >     <mailto:torqueusers at supercluster.org
>     <mailto:torqueusers at supercluster.org>
>      >     <mailto:torqueusers at supercluster.org
>     <mailto:torqueusers at supercluster.org>>>>
>      >      >         Date: Tue, 27 Sep 2011 12:33:31 -0400
>      >      >
>      >      >         To: Torque Users Mailing List
>      >     <torqueusers at supercluster.org
>     <mailto:torqueusers at supercluster.org>
>     <mailto:torqueusers at supercluster.org
>     <mailto:torqueusers at supercluster.org>>
>      >      >         <mailto:torqueusers at supercluster.org
>     <mailto:torqueusers at supercluster.org>
>      >     <mailto:torqueusers at supercluster.org
>     <mailto:torqueusers at supercluster.org>>>>
>      >      >         Subject: Re: [torqueusers] Help: Unauthorized Request
>      >      >
>      >      >         With the execption of the unauthorized request
>     entries it
>      >     looks
>      >      >         like almost everything is okay, execpt for the
>     node file and
>      >      >         root <at> localhost (this should be root <at> ubuntu.
>      >      >
>      >      >         Thank you for your help!
>      >      >         Aaron
>      >      >
>      >      >         Here is an except from the server log...
>      >      >
>      >      >         09/27/2011 09:51:31;0002;PBS_Server;Svr;Log;Log opened
>      >      >         09/27/2011
>     09:51:31;0006;PBS_Server;Svr;PBS_Server;Server
>      >     ubuntu
>      >      >         started, initialization type = 4
>      >      >         09/27/2011 09:51:42;0002;PBS_Server;Svr;Log;Log opened
>      >      >         09/27/2011
>     09:51:42;0006;PBS_Server;Svr;PBS_Server;Server
>      >     ubuntu
>      >      >         started, initialization type = 4
>      >      >         09/27/2011
>     09:51:44;0002;PBS_Server;Svr;Act;Account file
>      >      >         /var/spool/torque/server_priv/accounting/20110927
>     opened
>      >      >         09/27/2011
>      >     09:51:44;0040;PBS_Server;Req;setup_nodes;setup_nodes()
>      >      >         09/27/2011 09:51:44;0004;PBS_Server;Svr;ubuntu;cannot
>      >     open node
>      >      >         description file
>     '/var/spool/torque/server_priv/nodes' in
>      >      >         setup_nodes()
>      >      >         09/27/2011
>      >     09:51:44;0002;PBS_Server;Svr;PBS_Server;Expected 0,
>      >      >         recovered 0 queues
>      >      >         09/27/2011
>      >     09:51:44;0002;PBS_Server;Svr;PBS_Server;Expected 0,
>      >      >         recovered 0 jobs
>      >      >         09/27/2011
>     09:51:44;0006;PBS_Server;Svr;PBS_Server;Using
>      >     ports
>      >      >         Server:15001  Scheduler:15004  MOM:15002 (server:
>     'ubuntu')
>      >      >         09/27/2011
>      >     09:51:44;0002;PBS_Server;Svr;daemonize_server;INFO:
>      >      >            parent is exiting
>      >      >         09/27/2011
>      >     09:51:44;0002;PBS_Server;Svr;daemonize_server;INFO:
>      >      >            parent is exiting
>      >      >         09/27/2011
>      >     09:51:44;0002;PBS_Server;Svr;daemonize_server;INFO:
>      >      >            child process in background
>      >      >         09/27/2011
>     09:51:44;0002;PBS_Server;Svr;PBS_Server;Server
>      >     Ready,
>      >      >         pid = 11995, loglevel=0
>      >      >         09/27/2011
>     09:51:44;0080;PBS_Server;Req;req_reject;Reject
>      >     reply
>      >      >         code=15007(Unauthorized Request ), aux=0,
>     type=Manager, from
>      >      >         root at localhost
>      >      >         09/27/2011
>     09:51:44;0080;PBS_Server;Req;req_reject;Reject
>      >     reply
>      >      >         code=15007(Unauthorized Request ), aux=0,
>     type=Manager, from
>      >      >         root at localhost
>      >      >         09/27/2011
>     09:51:44;0080;PBS_Server;Req;req_reject;Reject
>      >     reply
>      >      >         code=15007(Unauthorized Request ), aux=0,
>     type=Manager, from
>      >      >         root at localhost
>      >      >         09/27/2011
>     09:51:44;0080;PBS_Server;Req;req_reject;Reject
>      >     reply
>      >      >         code=15007(Unauthorized Request ), aux=0,
>     type=Manager, from
>      >      >         root at localhost
>      >      >         09/27/2011
>     09:51:44;0080;PBS_Server;Req;req_reject;Reject
>      >     reply
>      >      >         code=15007(Unauthorized Request ), aux=0,
>     type=Manager, from
>      >      >         root at localhost
>      >      >         09/27/2011
>     09:51:44;0080;PBS_Server;Req;req_reject;Reject
>      >     reply
>      >      >         code=15007(Unauthorized Request ), aux=0,
>     type=Manager, from
>      >      >         root at localhost
>      >      >         09/27/2011
>     09:51:44;0080;PBS_Server;Req;req_reject;Reject
>      >     reply
>      >      >         code=15007(Unauthorized Request ), aux=0,
>     type=Manager, from
>      >      >         root at localhost
>      >      >         09/27/2011
>     09:51:44;0080;PBS_Server;Req;req_reject;Reject
>      >     reply
>      >      >         code=15007(Unauthorized Request ), aux=0,
>     type=Manager, from
>      >      >         root at localhost
>      >      >         09/27/2011
>     09:51:44;0080;PBS_Server;Req;req_reject;Reject
>      >     reply
>      >      >         code=15007(Unauthorized Request ), aux=0,
>     type=Manager, from
>      >      >         root at localhost
>      >      >         09/27/2011
>     09:51:44;0080;PBS_Server;Req;req_reject;Reject
>      >     reply
>      >      >         code=15007(Unauthorized Request ), aux=0,
>     type=Manager, from
>      >      >         root at localhost
>      >      >         09/27/2011
>     09:51:44;0080;PBS_Server;Req;req_reject;Reject
>      >     reply
>      >      >         code=15007(Unauthorized Request ), aux=0,
>     type=Manager, from
>      >      >         root at localhost
>      >      >         09/27/2011
>     09:51:44;0080;PBS_Server;Req;req_reject;Reject
>      >     reply
>      >      >         code=15007(Unauthorized Request ), aux=0,
>     type=Manager, from
>      >      >         root at localhost
>      >      >         09/27/2011
>     09:51:49;0002;PBS_Server;Svr;PBS_Server;Torque
>      >     Server
>      >      >         Version = 3.0.2, loglevel = 0
>      >      >         09/27/2011
>     09:56:49;0002;PBS_Server;Svr;PBS_Server;Torque
>      >     Server
>      >      >         Version = 3.0.2, loglevel = 0
>      >      >         ...
>      >      >
>      >      >         On Tue, Sep 27, 2011 at 12:13 PM, Smith, Jerry Don II
>      >      >         <jdsmit at sandia.gov <mailto:jdsmit at sandia.gov>
>     <mailto:jdsmit at sandia.gov <mailto:jdsmit at sandia.gov>>
>      >     <mailto:jdsmit at sandia.gov <mailto:jdsmit at sandia.gov>
>     <mailto:jdsmit at sandia.gov <mailto:jdsmit at sandia.gov>>>> wrote:
>      >      >
>      >      >             Are you seeing anything in the pbs_server logs?
>      >      >
>      >      >             -Jerry
>      >      >
>      >      >             From: Aaron T Perry <atp42 at cornell.edu
>     <mailto:atp42 at cornell.edu>
>      >     <mailto:atp42 at cornell.edu <mailto:atp42 at cornell.edu>>
>      >      >             <mailto:atp42 at cornell.edu
>     <mailto:atp42 at cornell.edu> <mailto:atp42 at cornell.edu
>     <mailto:atp42 at cornell.edu>>>>
>      >      >             Reply-To: Torque Users Mailing List
>      >      >             <torqueusers at supercluster.org
>     <mailto:torqueusers at supercluster.org>
>      >     <mailto:torqueusers at supercluster.org
>     <mailto:torqueusers at supercluster.org>>
>      >      >             <mailto:torqueusers at supercluster.org
>     <mailto:torqueusers at supercluster.org>
>      >     <mailto:torqueusers at supercluster.org
>     <mailto:torqueusers at supercluster.org>>>>
>      >      >             Date: Tue, 27 Sep 2011 09:53:28 -0400
>      >      >             To: <torqueusers at supercluster.org
>     <mailto:torqueusers at supercluster.org>
>      >     <mailto:torqueusers at supercluster.org
>     <mailto:torqueusers at supercluster.org>>
>      >      >             <mailto:torqueusers at supercluster.org
>     <mailto:torqueusers at supercluster.org>
>      >     <mailto:torqueusers at supercluster.org
>     <mailto:torqueusers at supercluster.org>>>>
>      >      >             Subject: Re: [torqueusers] Help: Unauthorized
>     Request
>      >      >
>      >      >             Please, any help you can give would be greatly
>      >     appreciated,
>      >      >             I'm completely stuck. All the solutions I found
>      >     online have
>      >      >             failed.
>      >      >
>      >      >             On Mon, Sep 26, 2011 at 2:35 PM, Aaron
>      >     <atp42 at cornell.edu <mailto:atp42 at cornell.edu>
>     <mailto:atp42 at cornell.edu <mailto:atp42 at cornell.edu>>
>      >      >             <mailto:atp42 at cornell.edu
>     <mailto:atp42 at cornell.edu>
>      >     <mailto:atp42 at cornell.edu <mailto:atp42 at cornell.edu>>>> wrote:
>      >      >
>      >      >                 Hi,
>      >      >
>      >      >                 I've just tried to install torque, and I
>     ran the
>      >      >                 following commands,
>      >      >
>      >      >                 ./configure
>      >      >                 sudo make
>      >      >                 sudo make install
>      >      >
>      >      >                 however when I run ./torque.setup username
>     I get the
>      >      >                 following...
>      >      >
>      >      >                 initializing TORQUE (admin: username at ubuntu)
>      >      >                 PBS_Server ubuntu: Create mode and server
>      >     database exists,
>      >      >                 do you wish to continue y/(n)?y
>      >      >                 Max open servers: 9
>      >      >                 qmgr obj= svr=default: Unauthorized Request
>      >      >                 Max open servers: 9
>      >      >                 qmgr obj= svr=default: Unauthorized Request
>      >      >                 qmgr obj= svr=default: Unauthorized Request
>      >      >                 qmgr obj= svr=default: Unauthorized Request
>      >      >                 qmgr obj= svr=default: Unauthorized Request
>      >      >                 qmgr obj=batch svr=default: Unauthorized
>     Request
>      >      >                 qmgr obj=batch svr=default: Unauthorized
>     Request
>      >      >                 qmgr obj=batch svr=default: Unauthorized
>     Request
>      >      >                 qmgr obj=batch svr=default: Unauthorized
>     Request
>      >      >                 qmgr obj=batch svr=default: Unauthorized
>     Request
>      >      >                 qmgr obj=batch svr=default: Unauthorized
>     Request
>      >      >                 qmgr obj= svr=default: Unauthorized Request
>      >      >
>      >      >                 The server lanched and I cannot stop it,
>     nor can
>      >     issue
>      >      >                 any command related to torque (qterm,
>     gmgr, qsub,
>      >     etc)
>      >      >                 under my current username or under root. Help!
>      >      >
>      >      >                 Thank you,
>      >      >                 -Aaron
>      >      >
>      >      >
>      >      >             _______________________________________________
>      >     torqueusers
>      >      >             mailing list torqueusers at supercluster.org
>     <mailto:torqueusers at supercluster.org>
>      >     <mailto:torqueusers at supercluster.org
>     <mailto:torqueusers at supercluster.org>>
>      >      >             <mailto:torqueusers at supercluster.org
>     <mailto:torqueusers at supercluster.org>
>      >     <mailto:torqueusers at supercluster.org
>     <mailto:torqueusers at supercluster.org>>>
>      >      >            
>     http://www.supercluster.org/mailman/listinfo/torqueusers
>      >      >
>      >      >             _______________________________________________
>      >      >             torqueusers mailing list
>      >      >             torqueusers at supercluster.org
>     <mailto:torqueusers at supercluster.org>
>      >     <mailto:torqueusers at supercluster.org
>     <mailto:torqueusers at supercluster.org>>
>      >      >             <mailto:torqueusers at supercluster.org
>     <mailto:torqueusers at supercluster.org>
>      >     <mailto:torqueusers at supercluster.org
>     <mailto:torqueusers at supercluster.org>>>
>      >      >            
>     http://www.supercluster.org/mailman/listinfo/torqueusers
>      >      >
>      >      >
>      >      >         _______________________________________________
>     torqueusers
>      >      >         mailing list torqueusers at supercluster.org
>     <mailto:torqueusers at supercluster.org>
>      >     <mailto:torqueusers at supercluster.org
>     <mailto:torqueusers at supercluster.org>>
>      >      >         <mailto:torqueusers at supercluster.org
>     <mailto:torqueusers at supercluster.org>
>      >     <mailto:torqueusers at supercluster.org
>     <mailto:torqueusers at supercluster.org>>>
>      >      >        
>     http://www.supercluster.org/mailman/listinfo/torqueusers
>      >      >
>      >      >         _______________________________________________
>      >      >         torqueusers mailing list
>      >      >         torqueusers at supercluster.org
>     <mailto:torqueusers at supercluster.org>
>      >     <mailto:torqueusers at supercluster.org
>     <mailto:torqueusers at supercluster.org>>
>      >     <mailto:torqueusers at supercluster.org
>     <mailto:torqueusers at supercluster.org>
>      >     <mailto:torqueusers at supercluster.org
>     <mailto:torqueusers at supercluster.org>>>
>      >      >        
>     http://www.supercluster.org/mailman/listinfo/torqueusers
>      >      >
>      >      >
>      >      >     _______________________________________________
>     torqueusers
>      >     mailing
>      >      >     list torqueusers at supercluster.org
>     <mailto:torqueusers at supercluster.org>
>      >     <mailto:torqueusers at supercluster.org
>     <mailto:torqueusers at supercluster.org>>
>      >      >     <mailto:torqueusers at supercluster.org
>     <mailto:torqueusers at supercluster.org>
>      >     <mailto:torqueusers at supercluster.org
>     <mailto:torqueusers at supercluster.org>>>
>      >      >     http://www.supercluster.org/mailman/listinfo/torqueusers
>      >      >
>      >      >     _______________________________________________
>      >      >     torqueusers mailing list
>      >      >     torqueusers at supercluster.org
>     <mailto:torqueusers at supercluster.org>
>      >     <mailto:torqueusers at supercluster.org
>     <mailto:torqueusers at supercluster.org>>
>      >     <mailto:torqueusers at supercluster.org
>     <mailto:torqueusers at supercluster.org>
>      >     <mailto:torqueusers at supercluster.org
>     <mailto:torqueusers at supercluster.org>>>
>      >      >     http://www.supercluster.org/mailman/listinfo/torqueusers
>      >      >
>      >      >
>      >      >
>      >      >
>      >    
>     ------------------------------------------------------------------------
>      >      >
>      >      > _______________________________________________
>      >      > torqueusers mailing list
>      >      > torqueusers at supercluster.org
>     <mailto:torqueusers at supercluster.org>
>     <mailto:torqueusers at supercluster.org
>     <mailto:torqueusers at supercluster.org>>
>      >      > http://www.supercluster.org/mailman/listinfo/torqueusers
>      >
>      >     _______________________________________________
>      >     torqueusers mailing list
>      >     torqueusers at supercluster.org
>     <mailto:torqueusers at supercluster.org>
>     <mailto:torqueusers at supercluster.org
>     <mailto:torqueusers at supercluster.org>>
>      >     http://www.supercluster.org/mailman/listinfo/torqueusers
>      >
>      >
>      >
>      >
>     ------------------------------------------------------------------------
>      >
>      > _______________________________________________
>      > torqueusers mailing list
>      > torqueusers at supercluster.org <mailto:torqueusers at supercluster.org>
>      > http://www.supercluster.org/mailman/listinfo/torqueusers
> 
>     _______________________________________________
>     torqueusers mailing list
>     torqueusers at supercluster.org <mailto:torqueusers at supercluster.org>
>     http://www.supercluster.org/mailman/listinfo/torqueusers
> 
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> torqueusers mailing list
> torqueusers at supercluster.org
> http://www.supercluster.org/mailman/listinfo/torqueusers



More information about the torqueusers mailing list