 |
 |
Hi thanks for the suggestions we did increased the MaxStartups value in the sshd_config on our CE to 400 ( this is way too much but is just for staying in the safe zone ) and, at the same time, reduced the MaxAuthTries to 3 as the problem is only sporadic, we will monitor the system with this new config and see how it behaves for a long period of time we will keep you updated on this thanks again Cheers Sergio :) On 07, May 2008 01:39 PM, Steve Traylen <[log in to unmask]> wrote: > 2008/5/7 IEETA_Grid_initiative <[log in to unmask]>: > > Hi Sergio, > > > > So... any news about the 15001 error? > > > > think I'm having a similar problem here. > > > > from the WN logs (/var/spool/pbs/mom_logs/) I can see: > > > > 05/07/2008 08:25:00;0080; pbs_mom;Req;req_reject;Reject reply > > code=15001(Unknown Job Id REJHOST=axon-g06.ieeta.pt MSG=modify job > > failed, unknown job 6733.axon-g01.ieeta.pt), aux=0, type=ModifyJob, > > from > > [log in to unmask] > > > > the odd thing is that in 4 equal jobs submitted by me 3 have > > concluded > > successfully and 1 falls in "BatchHold" for eternity... > > > > Sergio, > > How many job slots, typical # jobs do you have in the first instance? > > You may need to increase the number of permitted scp connections in > sshd_config > on your CE. > > On one of the apparently affected WNs can you run. > > # momctl -d 1 > > You may be able to clear the stale job with a > > # momctl -c <jobid> -h <WN.example.org> > > I'm not sure if this will just delete the job though, I think it > depends on the retry policy. > > http://scotgrid.blogspot.com/2007/04/intervention-at-edinburgh.html > has a possible solution but it has never been confirmed if this is > even related. > It needs to be updated for new versions of torque as well. > > > Like the following ones: > > > > [luis@axon-g01 ~]$ showq > > ACTIVE JOBS-------------------- > > JOBNAME USERNAME STATE PROC REMAINING > > STARTTIME > > > > 6736 bio012 Running 1 2:22:44:42 Wed May 7 > > 10:20:39 > > 6737 bio012 Running 1 2:22:47:43 Wed May 7 > > 10:23:40 > > > > 2 Active Jobs 2 of 6 Processors Active (33.33%) > > 1 of 3 Nodes Active (33.33%) > > > > IDLE JOBS---------------------- > > JOBNAME USERNAME STATE PROC WCLIMIT > > QUEUETIME > > > > > > 0 Idle Jobs > > > > BLOCKED JOBS---------------- > > JOBNAME USERNAME STATE PROC WCLIMIT > > QUEUETIME > > > > 6684 dteam004 BatchHold 1 3:00:00:00 Tue May 6 > > 17:39:42 > > 6687 dteam004 BatchHold 1 3:00:00:00 Tue May 6 > > 17:40:46 > > 6697 dteam004 BatchHold 1 3:00:00:00 Tue May 6 > > 17:47:44 > > 6710 opssgm BatchHold 1 3:00:00:00 Tue May 6 > > 22:32:06 > > 6712 dteam004 BatchHold 1 3:00:00:00 Tue May 6 > > 22:58:34 > > 6724 opssgm BatchHold 1 3:00:00:00 Wed May 7 > > 03:17:38 > > 6739 opssgm Deferred 1 3:00:00:00 Wed May 7 > > 11:30:00 > > > > Total Jobs: 9 Active Jobs: 2 Idle Jobs: 0 Blocked Jobs: 7 > > > > For instance: > > > > [root@axon-g01 ~]# checkjob 6710 > > > > > > checking job 6710 > > > > State: Idle > > Creds: user:opssgm group:ops class:ops qos:DEFAULT > > WallTime: 00:00:16 of 3:00:00:00 > > SubmitTime: Tue May 6 22:32:06 > > (Time Queued Total: 13:05:59 Eligible: 00:00:00) > > > > StartDate: -12:39:13 Tue May 6 22:58:52 > > Total Tasks: 1 > > > > Req[0] TaskCount: 1 Partition: ALL > > Network: [NONE] Memory >= 0 Disk >= 0 Swap >= 0 > > Opsys: [NONE] Arch: [NONE] Features: [NONE] > > NodeCount: 1 > > > > > > IWD: [NONE] Executable: [NONE] > > Bypass: 0 StartCount: 26 > > PartitionMask: [ALL] > > Holds: Batch (hold reason: RMFailure) > > Messages: cannot start job - RM failure, rc: 15057, msg: 'Cannot > > execute at specified host because of checkpoint or stagein files' > > PE: 1.00 StartPriority: 1000758 > > cannot select job 6710 for partition DEFAULT (job hold active) > > > > ...and... > > > > [root@axon-g01 ~]# diagnose -j 6710 > > Name State Par Proc QOS WCLimit R Min User > > Group Account QueuedTime Network Opsys Arch Mem Disk Procs > > Class Features > > > > 6710 Idle ALL 1 DEF 3:00:00:00 0 1 opssgm > > ops - 12:41:36 [NONE] [NONE] [NONE] >=0 >=0 NC0 > > [ops:1] [NONE] > > WARNING: job '6710' has failed to start 26 times > > > > > > [root@axon-g01 ~]# releasehold -a ALL; tail > > -f /var/spool/pbs/server_logs/20080507 > > > > job holds adjusted > > > > (...) > > > > 05/07/2008 12:00:09;0100;PBS_Server;Req;;Type StatusQueue request > > received from [log in to unmask], sock=9 > > 05/07/2008 12:00:09;0100;PBS_Server;Req;;Type StatusJob request > > received > > from [log in to unmask], sock=9 > > 05/07/2008 12:00:09;0100;PBS_Server;Req;;Type ModifyJob request > > received > > from [log in to unmask], sock=9 > > 05/07/2008 12:00:09;0008;PBS_Server;Job;6710.axon-g01.ieeta.pt;Job > > Modified at request of [log in to unmask] > > 05/07/2008 12:00:09;0100;PBS_Server;Req;;Type RunJob request > > received > > from [log in to unmask], sock=9 > > 05/07/2008 12:00:09;0080;PBS_Server;Req;req_reject;Reject reply > > code=15057(Cannot execute at specified host because of checkpoint or > > stagein files), aux=0, type=RunJob, from [log in to unmask] > > 05/07/2008 12:00:09;0100;PBS_Server;Req;;Type ModifyJob request > > received > > from [log in to unmask], sock=9 > > 05/07/2008 12:00:09;0008;PBS_Server;Job;6710.axon-g01.ieeta.pt;Job > > Modified at request of [log in to unmask] > > 05/07/2008 12:00:09;0100;PBS_Server;Req;;Type ModifyJob request > > received > > from [log in to unmask], sock=9 > > 05/07/2008 12:00:09;0008;PBS_Server;Job;6724.axon-g01.ieeta.pt;Job > > Modified at request of [log in to unmask] > > 05/07/2008 12:00:09;0100;PBS_Server;Req;;Type RunJob request > > received > > from [log in to unmask], sock=9 > > 05/07/2008 12:00:09;0080;PBS_Server;Req;req_reject;Reject reply > > code=15057(Cannot execute at specified host because of checkpoint or > > stagein files), aux=0, type=RunJob, from [log in to unmask] > > 05/07/2008 12:00:09;0100;PBS_Server;Req;;Type ModifyJob request > > received > > from [log in to unmask], sock=9 > > 05/07/2008 12:00:09;0008;PBS_Server;Job;6724.axon-g01.ieeta.pt;Job > > Modified at request of [log in to unmask] > > 05/07/2008 12:00:16;0100;PBS_Server;Req;;Type AuthenticateUser > > request > > received from [log in to unmask], sock=12 > > 05/07/2008 12:00:16;0100;PBS_Server;Req;;Type StatusJob request > > received > > from [log in to unmask], sock=10 > > 05/07/2008 12:00:22;0040;PBS_Server;Svr;axon-g01.ieeta.pt;Scheduler > > sent > > command time > > 05/07/2008 12:00:22;0100;PBS_Server;Req;;Type StatusNode request > > received from [log in to unmask], sock=9 > > > > > > (...) > > > > > > This problem is driving me crazy for weeks... > > > > - don't matter if I turn the firewall off > > - CE/WN connection exists > > - I have reinstalled the WNs metapackages > > > > Please, any hint will be precious! > > > > P.S. - Sorry for the long mail. > > > > Thanks, > > > > Luís > > > > > > > > > > On Mon, 2008-05-05 at 23:43 +0200, Sergio Maffioletti wrote: > > > Hi Mario > > > > > > could you detail what problem these two nodes are having ? > > > we are experiencing a similar problem, except that it is not > > > systematic > > > > > > basically we are observing spooradics > > > "MOM rejected modify request, error: 15001" > > > messages; sometimes the job get started anyway > > > some other times it fails the stagin operation > > > then the job is sent back to the server and is placed in Q state, > > > but > > > then maui does not select it anymore. > > > > > > we had two period of time during last weekend when we did observer > > > the > > > Globus error 94, and we wander whether the two things are > > > correlated > > > with each other or not > > > > > > Cheers > > > Sergio :) > > > > > > On 05, May 2008 02:57 PM, Mario Kadastik <[log in to unmask]> > > > wrote: > > > > > > >Well actually we may have figured out the problem. It seems two > > > >workernodes had problems with stageout, but not something one > > > >would > > > >notice immediately out of hand. We have isolated them and now SAM > > > >tests seem to be running fine (but we'll have to wait a bit > > > >longer to > > > >make sure this was the problem indeed). We also ran a separate > > > >test of > > > >a job on one of that workernodes and the logging information came > > > >back > > > >with exactly the known error so we do hope we have isolated it > > > >now. We > > > >will know in about 24h if all the SAM tests run through nicely. > > > > > > > >Mario > > > > > > > >On May 5, 2008, at 3:36 PM, <[log in to unmask]> > > > ><[log in to unmask] > > > >>wrote: > > > > > > > >>Hi Mario, Ilja, > > > >> > > > >>>>Anyway, the exact details are available from this ggus ticket: > > > >>>>https://gus.fzk.de/pages/ticket_details.php?ticket=35655 > > > >>>> > > > >>>>I have increased the maxproc settings of both "marshal"s as it > > > >>>>seemed to be somehow related to the error ( Globus error 94: > > > >>>>the > > > >>>>jobmanager does not accept any new requests (shutting down)), > > > >>>>will > > > >>>>see if it helps. > > > >>>> > > > >>>>Any other ideas are still very welcome! > > > >> > > > >>It appears that the failing jobs were in fact successfully > > > >>submitted > > > >>to Torque. For example, in /opt/edg/var/gatekeeper/grid- > > > >>jobmap_20080505 > > > >>(spaces replaced with newlines for clarity): > > > >> > > > >>"localUser=11860" > > > >>"userDN=/DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=samoper/ > > > >>CN=582979/CN=Judit Novak" > > > >>"userFQAN=/ops/Role=lcgadmin/Capability=NULL" > > > >>"userFQAN=/ops/Role=NULL/Capability=NULL" > > > >>"jobID=https://rb113.cern.ch:9000/X3pf3fHmWWZTWr9nY5VvKQ" > > > >>"ceID=oberon.hep.kbfi.ee:2119/jobmanager-lcgpbs-short" > > > >>"lrmsID=42444.oberon.hep.kbfi.ee" > > > >>"timestamp=2008-05-05 10:07:18" > > > >> > > > >>The job then may have been reported in such a way that the > > > >>lcgpbs job > > > >>manager considered the job as having failed. For example, the > > > >>'W' > > > >>state > > > >>is treated like that. In that case you would see a cancellation > > > >>(qdel) > > > >>request in the Torque logs. Can you check what happened to job > > > >>42444? > > > >> > > > > > > > > > > > > Cheers > > > Sergio :) > > > > > > --------------------------------------------- > > > Dr. Sergio Maffioletti > > > > > > Grid Group > > > CSCS, Swiss National Supercomputing Centre > > > Via Cantonale > > > CH-6928 Manno > > > Tel: +41916108218 > > > Fax: +41916108282 > > > email: [log in to unmask] > > > --------------------------------------------- > > > > > > -- > Steve Traylen Cheers Sergio :) --------------------------------------------- Dr. Sergio Maffioletti Grid Group CSCS, Swiss National Supercomputing Centre Via Cantonale CH-6928 Manno Tel: +41916108218 Fax: +41916108282 email: [log in to unmask] ---------------------------------------------