Dear all,
our WMS suddenly stopped working and after looking for couple of
hours, we have no further idea, what is going wrong.
Symptom: all jobs end in a "BrokerHelper: Problems during rank
evaluation (e.g. GRISes down, wrong JDL rank expression, etc.)" state.
Even if no requirements are given in the JDL.
The most interesting hints we've found are:
[root@glite-wms log]# grep c3TGiiCg messages
May 28 16:46:45 glite-wms glite-proxy-renewd[2704]: Received command
code 1 for proxy /opt/glite/var/proxycache/
50aab4940d2868a9-061f2bd5fed8fbe6 and jobid https://glite-wms.physik.uni-wuppertal.de:9000/tlxLgnj3v8qCScc3TGiiCg
May 28 16:46:47 glite-wms glite-proxy-renewd[2704]: Received command
code 3 for proxy (unspecified) and jobid https://glite-wms.physik.uni-wuppertal.de:9000/tlxLgnj3v8qCScc3TGiiCg
-bash-3.00$ glite-wms-job-logging-info -v 2 -i jobid
**********************************************************************
LOGGING INFORMATION:
Printing info for the Job : https://glite-wms.physik.uni-wuppertal.de:9000/tlxLgnj3v8qCScc3TGiiCg
---
Event: RegJob
- Arrived = Wed May 28 16:46:45 2008 CEST
- Host = glite-wms.physik.uni-wuppertal.de
- Ns = https://132.195.104.218:7443/glite_wms_wmproxy_server
- Nsubjobs = 0
- Seed = edg_wll_RegisterJobProxy()
- Source = NetworkServer
- Src instance = https://132.195.104.218:7443/glite_wms_wmproxy_server
- Timestamp = Wed May 28 16:46:44 2008 CEST
- User = /O=GermanGrid/OU=UniWuppertal/
CN=Torsten Harenberg
---
Event: RegJob
- Arrived = Wed May 28 16:46:45 2008 CEST
- Host = glite-wms.physik.uni-wuppertal.de
- Ns = https://132.195.104.218:7443/glite_wms_wmproxy_server
- Nsubjobs = 0
- Seed = edg_wll_RegisterJobProxy()
- Source = NetworkServer
- Src instance = https://132.195.104.218:7443/glite_wms_wmproxy_server
- Timestamp = Wed May 28 16:46:45 2008 CEST
- User = /O=GermanGrid/OU=UniWuppertal/
CN=Torsten Harenberg
---
Event: UserTag
- Arrived = Wed May 28 16:46:45 2008 CEST
- Host = glite-wms.physik.uni-wuppertal.de
- Name = delegation_id
- Source = NetworkServer
- Src instance = https://132.195.104.218:7443/glite_wms_wmproxy_server
- Timestamp = Wed May 28 16:46:45 2008 CEST
- User = /O=GermanGrid/OU=UniWuppertal/
CN=Torsten Harenberg
- Value = 2K9oOtVhlhFuVXCuSGKZjA
---
Event: UserTag
- Arrived = Wed May 28 16:46:45 2008 CEST
- Host = glite-wms.physik.uni-wuppertal.de
- Name = jdl_original
- Source = NetworkServer
- Src instance = https://132.195.104.218:7443/glite_wms_wmproxy_server
- Timestamp = Wed May 28 16:46:45 2008 CEST
- User = /O=GermanGrid/OU=UniWuppertal/
CN=Torsten Harenberg
- Value = [ requirements =
other.GlueCEStateStatus == "Production"; RetryCount = 3; MyProxyServer
= "grid-rb.physik.uni-wuppertal.de"; AllowZippedISB = true; JobType =
"normal"; SignificantAttributes =
{ "Requirements","Rank","FuzzyRank" }; Executable = "muon2.sh";
StdOutput = "std.out"; OutputSandbox = { "std.out","std.err" };
VirtualOrganisation = "atlas"; rank = -
other.GlueCEStateEstimatedResponseTime; Type = "job";
ShallowRetryCount = 10; StdError = "std.err"; DefaultRank = -
other.GlueCEStateEstimatedResponseTime; ZippedISB =
{ "ISBfiles_kdxsnadgdxxWMk3uevBY0g_0.tar.gz" }; InputSandbox = { "file:///common/home/harenber/muon2.sh
","file:///common/home/harenber/muonana.tar.gz" } ]
---
Event: UserTag
- Arrived = Wed May 28 16:46:45 2008 CEST
- Host = glite-wms.physik.uni-wuppertal.de
- Name = lb_sequence_code
- Source = NetworkServer
- Src instance = https://132.195.104.218:7443/glite_wms_wmproxy_server
- Timestamp = Wed May 28 16:46:45 2008 CEST
- User = /O=GermanGrid/OU=UniWuppertal/
CN=Torsten Harenberg
- Value =
UI
=
000000
:NS
=
0000000004
:WM=000000:BH=0000000000:JSS=000000:LM=000000:LRMS=000000:APP=000000
---
Event: Accepted
- Arrived = Wed May 28 16:46:47 2008 CEST
- From = NetworkServer
- From host = https://132.195.104.218:7443/glite_wms_wmproxy_server
- Host = glite-wms.physik.uni-wuppertal.de
- Source = NetworkServer
- Src instance = https://132.195.104.218:7443/glite_wms_wmproxy_server
- Timestamp = Wed May 28 16:46:47 2008 CEST
- User = /O=GermanGrid/OU=UniWuppertal/
CN=Torsten Harenberg
---
Event: EnQueued
- Arrived = Wed May 28 16:46:47 2008 CEST
- Host = glite-wms.physik.uni-wuppertal.de
- Queue = /opt/glite/var/workload_manager/
input.fl
- Result = START
- Source = NetworkServer
- Src instance = https://132.195.104.218:7443/glite_wms_wmproxy_server
- Timestamp = Wed May 28 16:46:47 2008 CEST
- User = /O=GermanGrid/OU=UniWuppertal/
CN=Torsten Harenberg
---
Event: EnQueued
- Arrived = Wed May 28 16:46:47 2008 CEST
- Host = glite-wms.physik.uni-wuppertal.de
- Queue = /opt/glite/var/workload_manager/
input.fl
- Result = OK
- Source = NetworkServer
- Src instance = https://132.195.104.218:7443/glite_wms_wmproxy_server
- Timestamp = Wed May 28 16:46:47 2008 CEST
- User = /O=GermanGrid/OU=UniWuppertal/
CN=Torsten Harenberg
---
Event: DeQueued
- Arrived = Wed May 28 16:46:47 2008 CEST
- Host = glite-wms.physik.uni-wuppertal.de
- Queue = /opt/glite/var/workload_manager/
input.fl
- Source = WorkloadManager
- Src instance = 2477
- Timestamp = Wed May 28 16:46:47 2008 CEST
- User = /O=GermanGrid/OU=UniWuppertal/
CN=Torsten Harenberg/CN=proxy/CN=proxy
---
Event: Pending
- Arrived = Wed May 28 16:46:48 2008 CEST
- Host = glite-wms.physik.uni-wuppertal.de
- Reason = BrokerHelper: Problems during rank
evaluation (e.g. GRISes down, wrong JDL rank expression, etc.)
- Source = WorkloadManager
- Src instance = 2477
- Timestamp = Wed May 28 16:46:48 2008 CEST
- User = /O=GermanGrid/OU=UniWuppertal/
CN=Torsten Harenberg/CN=proxy/CN=proxy
**********************************************************************
Anybody an idea? It would be very appreciated.
Cheers,
Torsten
--
<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>
<> <>
<> Dr. Torsten Harenberg [log in to unmask] <>
<> Bergische Universitaet <>
<> FB C - Physik Tel.: +49 (0)202 439-3521 <>
<> Gaussstr. 20 Fax : +49 (0)202 439-2811 <>
<> 42097 Wuppertal <>
<> <>
<><><><><><><>< Of course it runs NetBSD http://www.netbsd.org ><>
|