Thanks for the replies guys, you kind of mirror my fears. Our problem
is in order to buy the 256 CPUs and 200TB of Storage we need the 2TB
disk solution seems to be the only way to come under budget. A quick
google suggests that we could buy 10GigE cards for the 5 pools and a
10 GigE switch (to go with our upgrade to a 10GigE infrastructure) and
still make a saving going for the 2TB solution. John's points about
the 2TB technology are worrying, especially about 2TB disk teething
problems. But I suppose someone has to go point on 2TB disks and be
the guinea pig for the storage group...

So do you think a  44 TB pool with two 10GigE uplinks (one LAN, one
WAN) cope? There would be five of them altogether, providing 2/3 of
our total storage. The other third being in six 1-TB disk pools.

Our problem eventually comes down to whats better; (possibly) slow
disk, or no disk? It's not a trick question, as I suspect many of us
have purchases that we wish we had done differently or just not bought
at all.

cheers,
Matt
Missing the days when the Pound was strong.

2009/5/29 John Bland <[log in to unmask]>:
> Matt Doidge wrote:
>>
>> Heya guys,
>>
>> We're looking to buy more storage and we got quoted the standard
>> 24-bay 4-U supermicro chassis BUT filled with 2TB Enterprise disks for
>> a very competitive price (VERY competitive. 2/3 of the price of the
>> nearest quote for 200TB competive). However common wisdom suggests
>> that more then 20TB/box isn't a good idea. We're currently LAN limited
>> but this hopefully will be fixed in the near future, so my plan to
>> reduce potential bottlenecks if we went for these huge pools would be
>> to bond ethernet links out of the boxes to 2Gb internal (farm traffic)
>> and 2Gb external(WAN traffic). Does the storage group have any
>> opinions on pools >> 20TB is size?
>
> Having been through a similar choice recently my opinion is that there is a
> compromise between cost, rack space, performance, and reliability. Assuming
> you're aiming for a fixed total amount of storage I would say the Pros and
> Cons of the 2TB solution are
>
> Pro
> Cheaper (get more storage or kit elsewhere)
> Fewer systems equals less rack space/cabling
> Fewer drives equals fewer drive failures
>
> Con
> Less striping of data files across pools (< aggregate performance)
> Reliability of individual arrays is worse (2TB drives take a very long time
> to rebuild and more chance of hitting an error)
> 2TB drives are a lot newer, more chance of teething problems
>
> For us we stuck with the tested 1TB solutions for our next round of storage
> as we have (just) enough space to easily accommodate more systems and even
> with 3Gbit bonded links to the research LAN it's already looking like
> network bandwidth to the pools is going to be a major bottle neck so the
> more machines the better (10G NICs would certainly help but raise the extra
> infrastructure prices way too high for us atm).
>
> As storage requirements increase we will go with a 2TB solution eventually,
> maybe next year.
>
> John
>
> --
> Dr John Bland, Systems Administrator
> Room 210, Oliver Lodge
> Particle Physics Group, University of Liverpool
> Mail: [log in to unmask]
> Tel : 0151 794 3396
>