Brian, what will ATLAS be doing with the other 50% of slots? At some stage we should test with that being done. Ideally one would like the HC work to be spread evenly across cpus - ie 50% of the cores per cpu. Which batch systems can achieve that?

john

-----Original Message-----
From: "Davies, BGE (Brian)" <[log in to unmask]>
To: "[log in to unmask]" <[log in to unmask]>
Sent: 16/07/09 09:39
Subject: Re: Next week's HammerCloud test: how to measure throughput scaling for your site.

By "their own limit", I would suggest that this site "limit" be (approximately) a value which is half the number of slots that a site expects the total atlas fairshare for the site to be.
Below shows the figures that ATLAS have been using as the maximum  slot usage ( taken from production usage over the last couple of months, rounded to nearest 50). So the suggestion is to go up to at least half this value at each site. (IMHO, if the number below is less than 400, then profiling up to 100% of this number is plausible with a short timescale.)
RALPPD	400
OXFORD	350
CAMBRIDGE	200
BIRMINGHAM	50
GLASGOW	1600
SHEFFIELD	150
MANCHESTER	1300
LANCASTER	550
LIVERPOOL	650
RHUL		400
QMUL		1150
UCL		50
 
Brian

-----Original Message-----
From: Testbed Support for GridPP member institutes [mailto:[log in to unmask]] On Behalf Of Sam Skipsey
Sent: 16 July 2009 06:45
To: [log in to unmask]
Subject: Re: Next week's HammerCloud test: how to measure throughput scaling for your site.

In fact, after thinking about this, considering that some sites do have less than 100 slots, I think it would be useful for *all* sites to start at 10 jobs, and scale to 25 before doing their own thing up to their own limit.
That way, we have at least two points that overlap for all the sites, regardless of size (and we have a feel for how the site behaves with essentially zero contention, hopefully).

Sam

2009/7/15 brian davies <[log in to unmask]>:
> As an aside , Netwerok limits may well be the most important factor. ( 
> I know some sites only have 1or 2 cores on each WN so this factor is 
> likely to be low.
> Also in terms of how high  a max current jobs you want to run during 
> the test, you might want to try to run up to 50%  of you total number 
> of slots in this tests. ( this is fo rsites who have less than the 100 
> slots that Sam suggested as a starting point!!) IIRC 50% is what is 
> currently thought to be an estimation of th enumber of analysis jobs 
> that ATLAS would like to run at
> aT2 site.
> Brian
>
>
> 2009/7/15 James Cullen <[log in to unmask]>
>>
>> Thanks Sam, that's a very thorough guide. What is the expected total 
>> duration of the hammercloud test? Will be useful for planning how 
>> many maui increments and their size we will make.
>>
>> Cheers,
>>
>> James
>>
>> ________________________________________
>> From: Testbed Support for GridPP member institutes 
>> [[log in to unmask]] On Behalf Of Sam Skipsey 
>> [[log in to unmask]]
>> Sent: 15 July 2009 14:04
>> To: [log in to unmask]
>> Subject: Next week's HammerCloud test: how to measure throughput 
>> scaling for your site.
>>
>> *Preface
>>
>> In the interests of investigating how ATLAS analysis jobs scale at 
>> sites in the UK, we are planning a UK-wide HammerCloud test at the 
>> start of next week. In yesterday's Dteam meeting, it was generally 
>> agreed that it would be good for sites to have some kind of 
>> description of the process we're looking for, to produce results that 
>> are comparable.
>> So, for any sites + site admins interested, here is such a description:
>>
>> *Testing your throughput/concurrency relation.
>>
>> We suspect that the number of jobs per node is the most important 
>> factor in throughput/concurrency for filestaging jobs like PanDA 
>> analysis and the Filestager WMS jobs. For these, you will want to 
>> scale until you get at least some fraction of nodes with 3 or 4 jobs 
>> concurrent on them.
>> (For 8 core nodes, this is around 25% to 50% of the cluster, with 
>> statistical variation taken into account.) It is advisable to start 
>> at around 100 concurrent jobs (or less, for WMS analysis), unless 
>> your storage is insufficient to cope with even that many from 
>> experience, and try to increase in equally sized steps, such that you 
>> get a reasonable number of data points without taking too long 
>> overall.
>>
>> You will also want to keep the number of jobs constant at each step 
>> for a reasonable amount of time. The expected duration of HammerCloud 
>> analysis jobs is around 40 minutes, so at least 1 hour for each step 
>> is a good idea (2 hours is better).
>>
>> *Maui configuration
>>
>> So, looking in /var/spool/maui/maui.cfg (or in the copy of maui.cfg 
>> in your configuration management system if you're using cfengine or 
>> Quattor or something), Glasgow, for example has:
>>
>> GROUPCFG[atlas]         FSTARGET=4      MAXPROC=2000,2000       
>> QDEF=atlas GROUPCFG[atlasprd]      FSTARGET=21     MAXPROC=2000,2000       
>> QDEF=atlas GROUPCFG[atlaspil]      FSTARGET=17     MAXPROC=600,600         
>> QDEF=atlas GROUPCFG[atlassgm]      FSTARGET=100    MAXPROC=10,10 and, 
>> later:
>> USERCFG[atlas476]       MAXJOB=20
>>
>>
>> ignore the QDEFs, as those are for QoS level management of fairshares.
>> The important point is that we can use the MAXPROC and MAXJOB 
>> injunctions (the difference only exists for multi-process jobs) to 
>> set a maximum number of a particular group or user's jobs to be 
>> allowed on the cluster.
>>
>> In this case, we know that user analysis pilots (which do PanDA
>> analysis) are going to be mapped to atlaspil group, and that the user
>> atlas476 corresponds to Johannes's DN used for WMS analysis on our 
>> cluster. Thus, PanDA analysis jobs are limited to a max of 600 
>> concurrent, and WMS analysis to 20.
>>
>> In order to accurately control the actual number of concurrent jobs 
>> to a reasonable degree of stability, one needs to also give the 
>> relevant group a very large fairshare, like GROUPCFG[atlaspil]      
>> FSTARGET=1000000000     MAXPROC=400,400
>>
>> this ensures that maui always prefers to give free slots to that 
>> group, and that it will never run more than 400 jobs from that group 
>> - the result being that (assuming the headroom in your cluster, and 
>> enough job rate from the user), there will always be 400 atlaspil 
>> jobs running.
>>
>> You may also want to turn *down* the MAXPROC/MAXJOB for other users 
>> with i/o intensive workloads if you want to avoid "contamination" of 
>> the results (we did this during STEP09 to separate out the WMS and 
>> the PanDA scaling from each other). On the other hand, of course, 
>> there is value to testing against normal competition with other jobs.
>>
>> Note that each time you change maui.cfg, you need to restart the maui 
>> service (and check that it is still running - try tailing the output 
>> of /var/log/maui.log ) to make it pick up the configuration change.
>>
>> Once you have scaled your job concurrency through enough steps over a 
>> wide enough range, you can then use the pbs record parser (which will 
>> be distributed soon) to analyse the efficiency and throughput of your 
>> jobs.
>>
>> Example (Glasgow):
>>
>> Start of test:
>> GROUPCFG[atlaspil]      FSTARGET=1000000000     MAXPROC=100,100 Two 
>> hours later:
>> GROUPCFG[atlaspil]      FSTARGET=1000000000     MAXPROC=200,200 
>> (optionally, wait for equilibration of jobs to new level), then two 
>> hours
>> later:
>> GROUPCFG[atlaspil]      FSTARGET=1000000000     MAXPROC=300,300 ... 
>> all the way to 800,800, which is well after the turn over in our 
>> throughput graph
>>
>> Since we know that our turn over is around 500 to 600, I might be 
>> tempted to stick an step in at 550 as well for better resolution near 
>> the turning point.
>>
>> Sam
>
>
-- 
Scanned by iCritical.
-- 
Scanned by iCritical.