Hi Alberto,
We use Scientific Linux 6 (RedHat-based) and it works perfectly fine as
well.
HTH,
Sjors
On 06/14/2016 11:25 AM, Alberto Riera wrote:
> Hi Erik,
>
> Thank you for the advice regarding the cards. It is actually true that
> you usually do not care if one pixel in Fallout 4 does not render
> properly. We will go for reference cards in this build since, sadly, I
> will not use it for gaming. In principle money wise we are OK for a
> single high-end system and the idea is to install 4 1080 on it. I am
> still thinking about the chassis, though. Maybe it is better to go for
> a 14 or 10 core Xeon with 128GB ECC, it should be more reliable than a
> squeezed I7 (and run cooler) and it could be expanded in the future
> with more RAM and another CPU. A system like this would be, with 14
> cores, around 10K.
>
> Regarding the distribution Ubuntu seems actually a very nice option
> and I am much more familiar with this branch than with others. I am
> afraid that maybe we will need to use RedHat in here, but let´s see...
>
> Best!
>
> Alberto
>
>
> On 14/06/2016 11:09, Erik Lindahl wrote:
>> Hi Alberto,
>>
>> We typically use Ubuntu (16.04LTS right now), but we've never had any
>> compatibility issues - the NVIDIA drivers work fine for all modern
>> Linux distros.
>>
>> The problem with overclocking is that vendors optimise the cards for
>> graphics where you probably don't care of one pixel out of a billion
>> has an incorrect color, and even with a high-end game the card will
>> only run at 100% utilisation for very short periods. For computing,
>> the situation is pretty much the opposite: Cards might be running at
>> 100% 24/7/365, and we don't want to accept a single error. There have
>> been many instances where vendors have been to aggressive, so to us
>> it's simply not worth the risks of random errors to get 5-10% better
>> performance.
>>
>> Many of the gaming cards actually have better cooling than the
>> reference/Founder's edition if you only use a single card - they
>> often pull in the air from the long side of the card and push it out
>> on the other to improve the airflow. This works great... provided you
>> only have a single card in the machine. However, with 2-4 cards it
>> is important to get the hot air to flow out of the box rather than
>> circulate inside it - for a server it would even be disastrous since
>> front-to-back is the only possible way for the cooling air to flow.
>>
>> Finally, in particular for autorefine it will be beneficial
>> (memory-wise) to have two cards, since we can put one model on each
>> card in that case. Thus, if you are budget-limited, it will usually
>> be a much better idea to get two 1070 cards than a single 1080.
>>
>> Cheers,
>>
>> Erik
>>
>>
>> On Tue, Jun 14, 2016 at 10:55 AM, Alberto Riera
>> <[log in to unmask] <mailto:[log in to unmask]>>
>> wrote:
>>
>> Thank you all for the great advice!
>>
>> I am very glad to hear that a gaming rig should actually be enough
>> :) 128GB of RAM for an I7 is actually overkill but with the new
>> memories luckily it turns out to be possible. About the edition of
>> the cards, I have always used overclocked EVGAs instead of the
>> reference ones (only EVGA) but since actually the cooling system
>> is quite different we will stick with the standard versions for
>> this build. On a separate note: what operating system are you
>> using right now? It would make sense to have the same distro in
>> here :)
>>
>> Best!
>>
>> Alberto
>>
>>
>> On 13/06/2016 21:45, Erik Lindahl wrote:
>>> PS:
>>>
>>> I'll second Sjors and emphasize that *currently* the initial GPU
>>> version becomes a bit too memory hungry beyond 450^2 pixels.
>>>
>>> 1) There are a few tricks that save a factor 2 already know, but
>>> it can affect quality and is not something we recommend.
>>>
>>> 2) While this is NOT a promise, we have pretty good hopes we can
>>> fix the memory consumption completely and make all images work on
>>> consumer hardware in the future.
>>>
>>> Cheers,
>>>
>>> Erik
>>>
>>> Erik Lindahl <[log in to unmask]
>>> <mailto:[log in to unmask]>>
>>> Professor of Biophysics
>>> Science for Life Laboratory
>>> Stockholm University & KTH
>>> Office (SciLifeLab): +46 8 524 81567 <tel:%2B46%208%20524%2081567>
>>> Cell (Sweden): +46 73 4618050 <tel:%2B46%2073%204618050>
>>> Cell (US): +1 267 3078746 <tel:%2B1%20267%203078746>
>>>
>>>
>>> On 13 juni 2016, at 22:31, Erik Lindahl <[log in to unmask]
>>> <mailto:[log in to unmask]>> wrote:
>>>
>>>> Hi Oliver et al.,
>>>>
>>>> This far we've managed well even with 64GB on a workstation even
>>>> when working with ribosomes. The memory requirements won't be
>>>> proportional to the number of cards.
>>>>
>>>> When it comes to value for money, the king will be the 1070
>>>> cards released this weekend for $450. I haven't forgotten my
>>>> promise to create a Web page with more info, but we got a bit
>>>> delayed finishing a paper - we will have this online with the
>>>> beta.
>>>>
>>>> However, one important thing if anybody wants to get hardware
>>>> right away: Stay away from all overclocked "gaming" cards, and
>>>> if you want to use multiple cards in a machine you should stick
>>>> to the reference "Founder's edition" cards. Long story, but it
>>>> has to do with the cooling airflow.
>>>>
>>>> We'll create recommendations for cheap workstations with 2 cards
>>>> (~$1500), high-end workstations with 4 cards ($5-7k), and
>>>> special cluster nodes with 4 cards ($6-8k). Note that most
>>>> standard nodes will *not* accept consumer cards, so please don't
>>>> rush out and buy any model prematurely, but wait 2-3 days and
>>>> we'll have numbers & specs!
>>>>
>>>>
>>>> Having said that, the whole point of this is that we want to
>>>> make cryo-EM computing ridiculously cheap, and I think you'll be
>>>> happy :-)
>>>>
>>>> Cheers,
>>>>
>>>> Erik
>>>>
>>>> Erik Lindahl <[log in to unmask]
>>>> <mailto:[log in to unmask]>>
>>>> Professor of Biophysics
>>>> Science for Life Laboratory
>>>> Stockholm University & KTH
>>>> Office (SciLifeLab): +46 8 524 81567 <tel:%2B46%208%20524%2081567>
>>>> Cell (Sweden): +46 73 4618050 <tel:%2B46%2073%204618050>
>>>> Cell (US): +1 267 3078746 <tel:%2B1%20267%203078746>
>>>>
>>>>
>>>> On 13 juni 2016, at 20:36, Oliver Clarke <[log in to unmask]
>>>> <mailto:[log in to unmask]>> wrote:
>>>>
>>>>> Hi Sjors,
>>>>>
>>>>> We are currently in the same situation - trying to figure out
>>>>> the optimal GPU-workstation setup for a given budget.
>>>>>
>>>>> How much RAM is too much? That is, if using dual 1080s, how
>>>>> much RAM does one have to have to ensure that the max box size
>>>>> will be limited by the GPU memory, rather than the amount of
>>>>> RAM - because I am presuming that getting any more RAM than
>>>>> that will be a waste (except if you want to preread all your
>>>>> particles into RAM)?
>>>>>
>>>>> Cheers,
>>>>> Oliver.
>>>>>
>>>>> On Mon, Jun 13, 2016 at 2:04 PM, Sjors Scheres
>>>>> <[log in to unmask] <mailto:[log in to unmask]>>
>>>>> wrote:
>>>>>
>>>>> Hi Alberto,
>>>>>
>>>>> Please be advised that hopefully this week we (read Erik
>>>>> Lindahl) will
>>>>> come out with much more detailed information. I'll also
>>>>> speak about this
>>>>> at the GRC in Hong Kong. Meanwhile, have a look at my talk
>>>>> at the CCPEM
>>>>> Spring Symposium here:
>>>>> https://sas.stfc.ac.uk/vportal/index.jsp
>>>>>
>>>>> For small particles as the ones you mention, four 1080s (or
>>>>> even 1070s)
>>>>> will surely do. 128Gb of RAM will also be enough, so your
>>>>> cheap machine
>>>>> should be able to process all those data in a matter of
>>>>> days. You can also
>>>>> buy several of those gaming machines and tie them together
>>>>> using MPI. I
>>>>> don't have experience at all with EC2, so I'll let someone
>>>>> else comment on
>>>>> that.
>>>>>
>>>>> When you do larger systems (e.g. ribosomes at very high
>>>>> mag, viruses, etc)
>>>>> and the box size becomes larger than say 450x450 pixels,
>>>>> then the
>>>>> GPU-memory on the 1080s will become too small, and you'd
>>>>> have to work with
>>>>> the more expensive cards. For the larger box sizes you
>>>>> would also need
>>>>> more RAM (again becoming more expensive as you mention
>>>>> yourself). But as
>>>>> said, for the majority of cases (boxsize <400x400) your
>>>>> 'cheap option'
>>>>> should do extremely well. I'd also buy a fast (ssd) hard
>>>>> disk with it, as
>>>>> access to disk will quickly become a next bottleneck.
>>>>>
>>>>> HTH,
>>>>> Sjors
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> > Hello!
>>>>> >
>>>>> > I am Alberto, a postdoc from the DNA replication group in
>>>>> Imperial College
>>>>> > (London). I know that this has been discussed in before
>>>>> but with the
>>>>> > upcoming release of the new GPU ready Relion it seems
>>>>> that the time for
>>>>> > buying a new system has come and we still have a few
>>>>> questions. Maybe you
>>>>> > will be able to help :)
>>>>> >
>>>>> > In the lab we mainly work with protein complexes that are
>>>>> around 1 million
>>>>> > Dalton or a bit bigger, and in our latest reconstructions
>>>>> we have used
>>>>> > with around 300K particles (sorted from a 1.2 million
>>>>> particle pool) to
>>>>> > reach around 3.9A resolution. For this we have been using
>>>>> a cluster but we
>>>>> > wonder if, with the new Relion version, it would be
>>>>> possible to move the
>>>>> > task to a single workstation. There are different options:
>>>>> >
>>>>> > a) gaming computer with an I7 (10 cores) and up to 128GB
>>>>> of RAM; this
>>>>> > machine would have 4 GTX 1080 cards. This option would be
>>>>> the cheapest,
>>>>> > something like 5 o 6K should do it, but we cannot
>>>>> increase the RAM any
>>>>> > further, in fact, 128GB is already a bit over the top.
>>>>> >
>>>>> > b) workstation with a single xeon or a couple of CPUs.
>>>>> With this we could
>>>>> > go up to 36 cores, 1TB of RAM and the same 4 cards or
>>>>> higher. However, it
>>>>> > would be expensive. Actually, the starting configuration
>>>>> with a single 16
>>>>> > core xeon and 128GB of RAM plus the cards would go for
>>>>> around 10K
>>>>> >
>>>>> > c) we could continue using a cluster and try to something
>>>>> like Amazon
>>>>> > cloud, or a similar solution that offers GPU
>>>>> acceleration. Has anybody
>>>>> > tried this?
>>>>> >
>>>>> > d) We could go for any other approach that you recommend,
>>>>> like pilling a
>>>>> > few gaming computers and build our own cluster from them.
>>>>> >
>>>>> > Any advice will be much appreciated!
>>>>> >
>>>>> > Thanks,
>>>>> >
>>>>> > Alberto
>>>>> >
>>>>>
>>>>>
>>>>> --
>>>>> Sjors Scheres
>>>>> MRC Laboratory of Molecular Biology
>>>>> Francis Crick Avenue, Cambridge Biomedical Campus
>>>>> Cambridge CB2 0QH, U.K.
>>>>> tel: +44 (0)1223 267061 <tel:%2B44%20%280%291223%20267061>
>>>>> http://www2.mrc-lmb.cam.ac.uk/groups/scheres
>>>>>
>>>>>
>>
>> -- Alberto Riera PhD
>> Imperial College London
>> Faculty of Medicine
>> Institute of Clinical Sciences (CRB room 3006)
>> Hammersmith Hospital Campus
>> Du Cane Road
>> London W12 0NN
>>
>> [log in to unmask] <mailto:[log in to unmask]>
>>
>>
>>
>>
>> --
>> --
>> Erik Lindahl <[log in to unmask] <mailto:[log in to unmask]>>
>> Professor of Biophysics, Dept. Biochemistry & Biophysics, Stockholm
>> University
>> Science for Life Laboratory, Box 1031, 17121 Solna, Sweden
>
--
Sjors Scheres
MRC Laboratory of Molecular Biology
Francis Crick Avenue, Cambridge Biomedical Campus
Cambridge CB2 0QH, U.K.
tel: +44 (0)1223 267061
http://www2.mrc-lmb.cam.ac.uk/groups/scheres
|
