Hi John, just a follow-on from what Mark said for B'ham:

We had an issue with one VO's jobs (ALICE, so that's not going to be an
issue for you) where its job-built-in-monitoring overhead didn't scale
well with the number of cores and was increasing the overall load. Not
recently though.

The 32-bit HEP-SPEC06 benchmark on SL5 for 48 cores (AMD6234) we've got as
7.68 per core (no turbo).

I did some power measurements early on: see near the bottom of
http://www.ep.ph.bham.ac.uk/twiki/bin/view/Computing/DellC6145Init?cover=print

Lawrie


Tel: 0121 414 4621 Fax: 0121 414 6709 Email: [log in to unmask]

On Mon, 3 Dec 2012, Peter Gronbech wrote:

> We would echo Mark's comments.
> Ours are working well now and are fully loaded with 64 jobs each.
> They do run very warm and we have upgraded the firmware on at least two of our 4 boxes after one of them had switched itself off due to overheating.
> Also due to the fact that we have a lot of RAM in these nodes and 9 local disks each the PSU are loaded quite highly. The net effect is that they are less resistant minor brown outs. Other nodes survive but they usually crash. We will put them on UPS's to cover that issue.
>
> We have the 6145 with AMD Opteron 6276 128GB. We get ~ 7.38 HS06 per core when fully loaded (with Turbo Mode set to Auto)
> We also have 10Gbit NICs.
>
> We have recently bought some Intel Sandy Bridge nodes for our local cluster as we wanted faster throughput so the higher HS06 per core count won there.
> Cheers Pete
>
>
>
>
> --
> ----------------------------------------------------------------------
> Peter Gronbech GridPP Project Manager Tel No. : 01865 273389
> Fax No. : 01865 273418
> Department of Particle Physics,
> University of Oxford,
> Keble Road, Oxford OX1 3RH, UK E-mail : [log in to unmask]
> ----------------------------------------------------------------------
>
>
>> -----Original Message-----
>> From: Testbed Support for GridPP member institutes [mailto:TB-
>> [log in to unmask]] On Behalf Of Mark Slater
>> Sent: 03 December 2012 12:09
>> To: [log in to unmask]
>> Subject: Re: DELL PowerEdge C6145
>>
>> Hi John,
>>
>> We haven't had many issues with our C6145's though be sure to upgrade
>> the BIOS and Fan firmware as there were some heat/fan issues before we
>> did that. They're running smoothly now and have been for quite some
>> time. As regards best bang for the buck, I'm not certain as I haven't
>> done the calcs (Ewan would have a better idea) but I believe that,
>> though they do give a lowish HS06 rating when fully loaded (we clocked
>> ours at 7.68 * 48 cores), you're getting more HS06 in total for the
>> money. This may have been superseded now though with e.g. the
>> Sandybridge Xeons Gareth mentions. I guess compare prices and see what
>> looks better!
>>
>> hope this helps!
>>
>> Mark
>>
>> On 03/12/12 11:57, Gareth Roy wrote:
>>> Hi John,
>>>
>>> And in following up I should say thats 128 Hyper-threaded core in the
>> Sandybridge nodes rather than physical cores (so 64 physical).
>>>
>>> Gareth
>>>
>>>
>>>
>>> On 3 Dec 2012, at 11:45, Gareth Roy wrote:
>>>
>>>> Hi John,
>>>>
>>>> We have 5x C6145's at Glasgow (so 10 nodes with 640 cores total) and if
>> we had it to do again we probably wouldn't go for them. Some of the issues
>> we've had with the boxes have been:
>>>>
>>>> 1) Lower HEPSPEC than stated (probably overcome by a move to SL6).
>>>> 2) Issues with BIOS upgrades not working.
>>>> 3) Unexplained shutdowns (possibly heat related).
>>>> 4) Problems with PBS due to cpu density.
>>>>
>>>> In general we've found they use a lot more power than other nodes we
>> have for the HEPSPEC we get.
>>>>
>>>> In a follow up purchase we actually went for Intel Sandybridge Xeons in a
>> twin-squared chassis from Supermicro which gave us the same 128 cores in a
>> 2U density but spread across 4 nodes rather than 2 (this is without a 10Gig
>> interface though).
>>>>
>>>> I know other sites haven't had the same number of problems we've had
>> but that's the experience with them at Glasgow.
>>>>
>>>> Gareth
>>>>
>>>>
>>>>
>>>>
>>>> On 3 Dec 2012, at 10:47, John Hill wrote:
>>>>
>>>>> Hi all,
>>>>> I'm looking to add some CPUs to our Tier 2. I was considering the
>>>>> possibility of the C6145 (or similar) and I believe that other sites
>>>>> already have these. Hence I was interested in people's experiences. Are
>>>>> there unexpected gotchas from putting 48+ cores in a single box,
>>>>> expecially for a smaller Tier 2 site like ours, or is it all plain sailing?
>>>>> Thanks for any thoughts.
>>>>>
>>>>> John
>