In fact, after thinking about this, considering that some sites do
have less than 100 slots, I think it would be useful for *all* sites
to start at 10 jobs, and scale to 25 before doing their own thing up
to their own limit.
That way, we have at least two points that overlap for all the sites,
regardless of size (and we have a feel for how the site behaves with
essentially zero contention, hopefully).
Sam
2009/7/15 brian davies <[log in to unmask]>:
> As an aside , Netwerok limits may well be the most important factor. ( I
> know some sites only have 1or 2 cores on each WN so this factor is likely to
> be low.
> Also in terms of how high a max current jobs you want to run during the
> test, you might want to try to run up to 50% of you total number of slots
> in this tests. ( this is fo rsites who have less than the 100 slots that Sam
> suggested as a starting point!!) IIRC 50% is what is currently thought to be
> an estimation of th enumber of analysis jobs that ATLAS would like to run at
> aT2 site.
> Brian
>
>
> 2009/7/15 James Cullen <[log in to unmask]>
>>
>> Thanks Sam, that's a very thorough guide. What is the expected total
>> duration of the hammercloud test? Will be useful for planning how many maui
>> increments and their size we will make.
>>
>> Cheers,
>>
>> James
>>
>> ________________________________________
>> From: Testbed Support for GridPP member institutes
>> [[log in to unmask]] On Behalf Of Sam Skipsey
>> [[log in to unmask]]
>> Sent: 15 July 2009 14:04
>> To: [log in to unmask]
>> Subject: Next week's HammerCloud test: how to measure throughput scaling
>> for your site.
>>
>> *Preface
>>
>> In the interests of investigating how ATLAS analysis jobs scale at
>> sites in the UK, we are planning a UK-wide HammerCloud test at the
>> start of next week. In yesterday's Dteam meeting, it was generally
>> agreed that it would be good for sites to have some kind of
>> description of the process we're looking for, to produce results that
>> are comparable.
>> So, for any sites + site admins interested, here is such a description:
>>
>> *Testing your throughput/concurrency relation.
>>
>> We suspect that the number of jobs per node is the most important
>> factor in throughput/concurrency for filestaging jobs like PanDA
>> analysis and the Filestager WMS jobs. For these, you will want to
>> scale until you get at least some fraction of nodes with 3 or 4 jobs
>> concurrent on them.
>> (For 8 core nodes, this is around 25% to 50% of the cluster, with
>> statistical variation taken into account.)
>> It is advisable to start at around 100 concurrent jobs (or less, for
>> WMS analysis), unless your storage is insufficient to cope with even
>> that many from experience, and try to increase in equally sized steps,
>> such that you get a reasonable number of data points without taking
>> too long overall.
>>
>> You will also want to keep the number of jobs constant at each step
>> for a reasonable amount of time. The expected duration of HammerCloud
>> analysis jobs is around 40 minutes, so at least 1 hour for each step
>> is a good idea (2 hours is better).
>>
>> *Maui configuration
>>
>> So, looking in /var/spool/maui/maui.cfg (or in the copy of maui.cfg in
>> your configuration management system if you're using cfengine or
>> Quattor or something), Glasgow, for example has:
>>
>> GROUPCFG[atlas] FSTARGET=4 MAXPROC=2000,2000 QDEF=atlas
>> GROUPCFG[atlasprd] FSTARGET=21 MAXPROC=2000,2000 QDEF=atlas
>> GROUPCFG[atlaspil] FSTARGET=17 MAXPROC=600,600 QDEF=atlas
>> GROUPCFG[atlassgm] FSTARGET=100 MAXPROC=10,10
>> and, later:
>> USERCFG[atlas476] MAXJOB=20
>>
>>
>> ignore the QDEFs, as those are for QoS level management of fairshares.
>> The important point is that we can use the MAXPROC and MAXJOB
>> injunctions (the difference only exists for multi-process jobs) to set
>> a maximum number of a particular group or user's jobs to be allowed on
>> the cluster.
>>
>> In this case, we know that user analysis pilots (which do PanDA
>> analysis) are going to be mapped to atlaspil group, and that the user
>> atlas476 corresponds to Johannes's DN used for WMS analysis on our
>> cluster. Thus,
>> PanDA analysis jobs are limited to a max of 600 concurrent, and WMS
>> analysis to 20.
>>
>> In order to accurately control the actual number of concurrent jobs to
>> a reasonable degree of stability, one needs to also give the relevant
>> group a very large fairshare, like
>> GROUPCFG[atlaspil] FSTARGET=1000000000 MAXPROC=400,400
>>
>> this ensures that maui always prefers to give free slots to that
>> group, and that it will never run more than 400 jobs from that group -
>> the result being that (assuming the headroom in your cluster, and
>> enough job rate from the user), there will always be 400 atlaspil jobs
>> running.
>>
>> You may also want to turn *down* the MAXPROC/MAXJOB for other users
>> with i/o intensive workloads if you want to avoid "contamination" of
>> the results (we did this during STEP09 to separate out the WMS and the
>> PanDA scaling from each other). On the other hand, of course, there is
>> value to testing against normal competition with other jobs.
>>
>> Note that each time you change maui.cfg, you need to restart the maui
>> service (and check that it is still running - try tailing the output
>> of /var/log/maui.log ) to make it pick up the configuration change.
>>
>> Once you have scaled your job concurrency through enough steps over a
>> wide enough range, you can then use the pbs record parser (which will
>> be distributed soon) to analyse the efficiency and throughput of your
>> jobs.
>>
>> Example (Glasgow):
>>
>> Start of test:
>> GROUPCFG[atlaspil] FSTARGET=1000000000 MAXPROC=100,100
>> Two hours later:
>> GROUPCFG[atlaspil] FSTARGET=1000000000 MAXPROC=200,200
>> (optionally, wait for equilibration of jobs to new level), then two hours
>> later:
>> GROUPCFG[atlaspil] FSTARGET=1000000000 MAXPROC=300,300
>> ... all the way to 800,800, which is well after the turn over in our
>> throughput graph
>>
>> Since we know that our turn over is around 500 to 600, I might be
>> tempted to stick an step in at 550 as well for better resolution near
>> the turning point.
>>
>> Sam
>
>
|