Hi Nagaraj
The answer sort of depends on how many cores you have in your farm.
We have a few techniques that we use here to ensure that dteam/ops
jobs run in a timely fashion. The most important are:
* don't allow any single VO to take all your cores. We typically have a
cap of about 80% (or lower) for each VO. This means for example with 100
cores, you tell Maui "MAXPROC=80" (or even some lower number) for each
group running jobs.
* use priorities. we have a very high priority assigned to ops jobs
here, this means that it is always the case that ops jobs, if present,
will be scheduled before any other job. So it doesn't matter if there
are 200 cms jobs waiting, the ops job will run first.
* cap the high-priority groups. otherwise someone could flood your
system with dteam jobs. we typically don't allow more than 32
simultaneous ops / dteams jobs to run. BUT don't forget we have a
thousand cores. We used to have this number set at 2 when the cluster
was much smaller.
If you don't have many WN cores, this may not be enough and you may need
to make a standing reservation for the ops jobs. I am not exactly sure
how many you need to have "enough". 400 cores is certainly enough, and
I am pretty sure that 10 cores is too few.
I would (if I were you) shy away from things like special queues and/or
a WN specially reserved to ops. These would make sure that ops jobs
always run promptly, BUT then these ops jobs do not take the same path
thru your system as do other jobs. So it might be that for normal jobs,
your system is broken, but this will not be detected because the path
for ops jobs is not broken.
here is some example stuff from our Maui cfg. We have about 1200 cores
right now. The stuff about fair share you could leave off for now
unless you need it, i include it for completeness.
J "prioritizing the fair scare" T
> QUEUETIMEWEIGHT 0
> XFACTORWEIGHT 1
> XFACTORCAP 100000
> RESWEIGHT 10
>
> CREDWEIGHT 10
> USERWEIGHT 10
> GROUPWEIGHT 10
>
> FSWEIGHT 1
> FSUSERWEIGHT 1
> FSGROUPWEIGHT 43
> FSQOSWEIGHT 2000
>
> # FairShare
> # use dedicated CPU ("wallclocktime used") metering
> # decays over 24 "days"
> FSPOLICY DEDICATEDPES%
> FSDEPTH 24
> FSINTERVAL 24:00:00
> FSDECAY 0.99
> FSCAP 100000
> ##############################################################################
> #
> # use PRIORITY to define various levels.
> # test groups have highest priority, e.g. dteam PRIORITY=5000
> # Tier-1 HEP VOs have next PRIORITY, all = 100
> # other VOs have less, e.g biomed PRIORITY 10, esr PRIORITY 50,
> # geant PRIORITY 80
>
> # USERs in Maui map to real users
> # GROUPs in Maui map to unix GIDs which map to VOs or VO subgroups
> # QoS in Maui map to VOs (bundle together VO subgroups)
>
> # for fair scare percentages: see spreadsheet on wiki.
> #
> # installed capacities
> #
> # note that fair-share competition between users is absolutely disabled
> # unless users have a fair share. A first guess: use 1%. This didn't
> # work since it meant that any user actually using cycles had a
> # rather large discrepancy from the target. Retry: right now set at 50%.
>
> USERCFG[DEFAULT] FSTARGET=50 MAXJOBQUEUED=1000
> GROUPCFG[DEFAULT] FSTARGET=1 PRIORITY=1 MAXPROC=1000
>
> GROUPCFG[tutor] FSTARGET=1 PRIORITY=200 MAXPROC=42
>
> # the limits applied appear to be a MIN() of all applicable limits
>
> ############## more or less local people, and monitoring VOs
>
> GROUPCFG[users] FSTARGET=1 PRIORITY=10 MAXPROC=50
> GROUPCFG[dteam] FSTARGET=1 PRIORITY=500 MAXPROC=32
> GROUPCFG[ops] FSTARGET=1 PRIORITY=500 MAXPROC=32
> GROUPCFG[pvier] FSTARGET=1 PRIORITY=500 MAXPROC=5
> GROUPCFG[deploy] FSTARGET=1 PRIORITY=500 MAXPROC=2
>
> ############# standard supported VOs with dedicated fair shares
>
> GROUPCFG[atlas] FSTARGET=10 PRIORITY=100 QDEF=lhcatlas
> GROUPCFG[atlb] FSTARGET=39 PRIORITY=100 QDEF=lhcatlas
> GROUPCFG[atlc] FSTARGET=7 PRIORITY=120 QDEF=lhcatlas
> GROUPCFG[atlsgm] PRIORITY=200 MAXPROC=2 QDEF=lhcatlas
> QOSCFG[lhcatlas] FSTARGET=49 MAXPROC=1000
>
Nagaraj Panyam wrote:
> Hi,
>
> I have some novice questions.
>
> At our site, we see a long queue of 200 jobs by user cms jobs, and SAM
> jobs are kept waiting. How do I manage the situation? How do I make sure
> that dteam and ops jobs get to run without waiting long?
>
> How do I know what is the id of the edg-job that is running , who
> submitted etc?
>
> Thanks
> Nagaraj
> +----------------------------------+--------------------------------------+
> Nagaraj Panyam | Office tel: +91-22-22782610
> Dept of High Energy Physics | Office fax: +91-22-22804610
> Tata Instt. of Fundamental Research| Home tel : +91-22-22804936
> Mumbai - 400 005, INDIA | **Email** : [log in to unmask]
> +----------------------------------+--------------------------------------+
