I sent this to the PMB for discussion but some of you might be interested in it.
I had a look at my benchmark results. The spread (= sdev/mean) in raw CPU times across all UK sites and machines is 29%. If I apply the average HS06 for each site obtained from Apel/ATLAS ('HS06 ATLAS') this increases to 38%. I haven't looked in any greater detail if the core gets smaller and the tails wider or anything like that (it wouldn't help us anyway). I also looked at the ATLAS production numbers. For April the spread in production CPU/job increases from 14% to 28% and production CPU/event from 22% to 33% if you scale by 'HS06 ATLAS'. (These are averaged over sites not weighted by size). These bear out other observations made by ATLAS reported at a recent CMB/SPMB meeting that HS06 makes it worse than using raw CPU numbers. Basically correcting for HS06 introduces ~25-30% noise into the measurements.
However, if I use the ATLAS production CPU/event to derive a HS06 number ('HS06 Prod') i.e the production HS06/event should be constant and apply these to my benchmark results then the spread goes down from 29% to 23%. So basically if you derive your benchmark from actually running ATLAS code then it improves things (as you might hope) otherwise it makes it worse. So we should be using 'HS06 Prod' not 'HS06 ATLAS' (or nothing).
+ Steve Lloyd Queen Mary, University of London +
+ E-mail: [log in to unmask] School of Physics +
+ Phone: +44-(0)20-7882-5057 Mile End Road +
+ Fax: +44-(0)20-8981-9465 London E1 4NS, UK +