Hi,
we have an EMI-2 infrastructure running with CREAM CE's and a central Torque server. The torque comes from EPEL repo and is 2.5.7-7. Everything is working nicely between the torque server, the CREAM CE's and the workernodes (mom's) with thousands of jobs being submitted and processed.
However we also wanted to allow local submissions so we installed torque-client on a separate node where users do their local computations. We copied the munge key over and started munge, which seems to be working according to munge.log
2012-10-02 20:04:13 Notice: Running on "phys.hep.kbfi.ee" (127.0.0.1)
2012-10-02 20:04:13 Info: PRNG seeded with 1024 bytes from "/var/lib/munge/munge.seed"
2012-10-02 20:04:13 Info: Updating supplementary group mapping every 3600 seconds
2012-10-02 20:04:13 Info: Enabled supplementary group mtime check of "/etc/group"
2012-10-02 20:04:13 Notice: Starting munge-0.5.8 daemon (pid 11312)
2012-10-02 20:04:13 Info: Created 2 work threads
2012-10-02 20:04:13 Info: Found 15 users with supplementary groups in 0.148 seconds
We configured the server name into /var/torque/server_name and added the host to authorized users list in pbs_server:
[root@torque-v-1 tmp]# qmgr -c "p s"|grep phys
set server authorized_users += *@phys.hep.kbfi.ee
however we seem to be unable to get any torque commands to work even as root user:
[root@phys munge]# qstat -q
qstat: Invalid credential (null)
[root@phys munge]# pbsnodes -l
pbsnodes: Invalid credential
[root@phys munge]#
I've tried adding a user that works on all other CE's and torque itself and that didn't change anything. The nodes are in sync time wise. What torque server sees:
10/02/2012 20:12:08;0080;PBS_Server;Req;req_reject;Reject reply code=15021(Invalid credential), aux=0, type=StatusQueue, from [log in to unmask]
10/02/2012 20:12:12;0080;PBS_Server;Req;req_reject;Reject reply code=15021(Invalid credential), aux=0, type=StatusNode, from [log in to unmask]
We've run out of ideas on what else to debug. Suggestions are welcome.
Mario Kadastik, PhD
Researcher
---
"Physics is like sex, sure it may have practical reasons, but that's not why we do it"
-- Richard P. Feynman
|