Hi Nagaraj

   The answer sort of depends on how many cores you have in your farm.
   We have a few techniques that we use here to ensure that dteam/ops 
jobs run in a timely fashion.  The most important are:

* don't allow any single VO to take all your cores.  We typically have a 
cap of about 80% (or lower) for each VO. This means for example with 100 
cores, you tell Maui "MAXPROC=80" (or even some lower number) for each 
group running jobs.

* use priorities.  we have a very high priority assigned to ops jobs 
here, this means that it is always the case that ops jobs, if present, 
will be scheduled before any other job.  So it doesn't matter if there 
are 200 cms jobs waiting, the ops job will run first.

* cap the high-priority groups.  otherwise someone could flood your 
system with dteam jobs.  we typically don't allow more than 32 
simultaneous ops / dteams jobs to run.  BUT don't forget we have a 
thousand cores.  We used to have this number set at 2 when the cluster 
was much smaller.

If you don't have many WN cores, this may not be enough and you may need 
to make a standing reservation for the ops jobs.  I am not exactly sure 
how many you need to have "enough".  400 cores is certainly enough, and 
I am pretty sure that 10 cores is too few.

I would (if I were you) shy away from things like special queues and/or 
a WN specially reserved to ops.  These would make sure that ops jobs 
always run promptly, BUT then these ops jobs do not take the same path 
thru your system as do other jobs.  So it might be that for normal jobs, 
your system is broken, but this will not be detected because the path 
for ops jobs is not broken.

here is some example stuff from our Maui cfg.  We have about 1200 cores 
right now.  The stuff about fair share you could leave off for now 
unless you need it, i include it for completeness.

	J "prioritizing the fair scare" T

> QUEUETIMEWEIGHT         0
> XFACTORWEIGHT           1
> XFACTORCAP         100000
> RESWEIGHT              10
> 
> CREDWEIGHT             10
> USERWEIGHT             10
> GROUPWEIGHT            10
> 
> FSWEIGHT                1
> FSUSERWEIGHT            1
> FSGROUPWEIGHT          43
> FSQOSWEIGHT          2000
>                                                                                 
> # FairShare
> # use dedicated CPU ("wallclocktime used") metering
> # decays over 24 "days"
> FSPOLICY              DEDICATEDPES%
> FSDEPTH               24
> FSINTERVAL            24:00:00
> FSDECAY               0.99
> FSCAP                 100000          
> ##############################################################################
> #
> # use PRIORITY to define various levels.
> # test groups have highest priority, e.g. dteam PRIORITY=5000
> # Tier-1 HEP VOs have next PRIORITY, all = 100
> # other VOs have less, e.g biomed PRIORITY 10, esr PRIORITY 50,
> # geant PRIORITY 80
> 
> # USERs in Maui map to real users
> # GROUPs in Maui map to unix GIDs which map to VOs or VO subgroups
> # QoS in Maui map to VOs (bundle together VO subgroups)
> 
> # for fair scare percentages: see spreadsheet on wiki.
> #
> # installed capacities
> #
> # note that fair-share competition between users is absolutely disabled
> # unless users have a fair share.  A first guess: use 1%.  This didn't
> # work since it meant that any user actually using cycles had a
> # rather large discrepancy from the target.  Retry: right now set at 50%.
>
> USERCFG[DEFAULT]    FSTARGET=50          MAXJOBQUEUED=1000
> GROUPCFG[DEFAULT]   FSTARGET=1    PRIORITY=1      MAXPROC=1000
> 
> GROUPCFG[tutor]     FSTARGET=1    PRIORITY=200    MAXPROC=42
> 
> # the limits applied appear to be a MIN() of all applicable limits
> 
> ##############  more or less local people, and monitoring VOs
> 
> GROUPCFG[users]     FSTARGET=1    PRIORITY=10     MAXPROC=50
> GROUPCFG[dteam]     FSTARGET=1    PRIORITY=500    MAXPROC=32
> GROUPCFG[ops]       FSTARGET=1    PRIORITY=500    MAXPROC=32
> GROUPCFG[pvier]     FSTARGET=1    PRIORITY=500    MAXPROC=5
> GROUPCFG[deploy]    FSTARGET=1    PRIORITY=500    MAXPROC=2
> 
> ############# standard supported VOs with dedicated fair shares
> 
> GROUPCFG[atlas]     FSTARGET=10   PRIORITY=100                  QDEF=lhcatlas
> GROUPCFG[atlb]      FSTARGET=39   PRIORITY=100                  QDEF=lhcatlas
> GROUPCFG[atlc]      FSTARGET=7    PRIORITY=120                  QDEF=lhcatlas
> GROUPCFG[atlsgm]                  PRIORITY=200    MAXPROC=2     QDEF=lhcatlas
> QOSCFG[lhcatlas]    FSTARGET=49                   MAXPROC=1000
>                                                                                        




Nagaraj Panyam wrote:
> Hi,
> 
> I have some novice questions.
> 
> At our site, we see a long queue of 200 jobs  by user cms jobs, and SAM 
> jobs are kept waiting. How do I manage the situation? How do I make sure 
> that dteam and ops jobs  get to run without waiting long?
> 
> How do I know what is the id of the edg-job that is running , who 
> submitted etc?
> 
> Thanks
> Nagaraj
> +----------------------------------+--------------------------------------+
> Nagaraj Panyam                     | Office tel: +91-22-22782610
> Dept of High Energy Physics        | Office fax: +91-22-22804610
> Tata Instt. of Fundamental Research| Home  tel : +91-22-22804936
> Mumbai - 400 005, INDIA            | **Email** : [log in to unmask]
> +----------------------------------+--------------------------------------+