JISCMail - CCPEM Archives

Hi Alberto,

We typically use Ubuntu (16.04LTS right now), but we've never had any compatibility issues - the NVIDIA drivers work fine for all modern Linux distros.

The problem with overclocking is that vendors optimise the cards for graphics where you probably don't care of one pixel out of a billion has an incorrect color, and even with a high-end game the card will only run at 100% utilisation for very short periods. For computing, the situation is pretty much the opposite: Cards might be running at 100% 24/7/365, and we don't want to accept a single error. There have been many instances where vendors have been to aggressive, so to us it's simply not worth the risks of random errors to get 5-10% better performance.

Many of the gaming cards actually have better cooling than the reference/Founder's edition if you only use a single card - they often pull in the air from the long side of the card and push it out on the other to improve the airflow. This works great... provided you only have a single card in the machine. However, with 2-4 cards it is important to get the hot air to flow out of the box rather than circulate inside it - for a server it would even be disastrous since front-to-back is the only possible way for the cooling air to flow.

Finally, in particular for autorefine it will be beneficial (memory-wise) to have two cards, since we can put one model on each card in that case. Thus, if you are budget-limited, it will usually be a much better idea to get two 1070 cards than a single 1080.

Cheers,

Erik

On Tue, Jun 14, 2016 at 10:55 AM, Alberto Riera <[log in to unmask]> wrote:

Thank you all for the great advice!

I am very glad to hear that a gaming rig should actually be enough :) 128GB of RAM for an I7 is actually overkill but with the new memories luckily it turns out to be possible. About the edition of the cards, I have always used overclocked EVGAs instead of the reference ones (only EVGA) but since actually the cooling system is quite different we will stick with the standard versions for this build. On a separate note: what operating system are you using right now? It would make sense to have the same distro in here :)

Best!

Alberto

On 13/06/2016 21:45, Erik Lindahl wrote:

PS:

I'll second Sjors and emphasize that *currently* the initial GPU version becomes a bit too memory hungry beyond 450^2 pixels.

1) There are a few tricks that save a factor 2 already know, but it can affect quality and is not something we recommend.

2) While this is NOT a promise, we have pretty good hopes we can fix the memory consumption completely and make all images work on consumer hardware in the future.

Cheers,

Erik

Erik Lindahl <[log in to unmask]>
Professor of Biophysics

Science for Life Laboratory

Stockholm University & KTH

Office (SciLifeLab): +46 8 524 81567

Cell (Sweden): +46 73 4618050

Cell (US): +1 267 3078746

On 13 juni 2016, at 22:31, Erik Lindahl <[log in to unmask]> wrote:

Hi Oliver et al.,

This far we've managed well even with 64GB on a workstation even when working with ribosomes. The memory requirements won't be proportional to the number of cards.

When it comes to value for money, the king will be the 1070 cards released this weekend for $450. I haven't forgotten my promise to create a Web page with more info, but we got a bit delayed finishing a paper - we will have this online with the beta.

However, one important thing if anybody wants to get hardware right away: Stay away from all overclocked "gaming" cards, and if you want to use multiple cards in a machine you should stick to the reference "Founder's edition" cards. Long story, but it has to do with the cooling airflow.

We'll create recommendations for cheap workstations with 2 cards (~$1500), high-end workstations with 4 cards ($5-7k), and special cluster nodes with 4 cards ($6-8k). Note that most standard nodes will *not* accept consumer cards, so please don't rush out and buy any model prematurely, but wait 2-3 days and we'll have numbers & specs!

Having said that, the whole point of this is that we want to make cryo-EM computing ridiculously cheap, and I think you'll be happy :-)

Cheers,

Erik

Erik Lindahl <[log in to unmask]>
Professor of Biophysics

Science for Life Laboratory

Stockholm University & KTH

Office (SciLifeLab): +46 8 524 81567

Cell (Sweden): +46 73 4618050

Cell (US): +1 267 3078746

On 13 juni 2016, at 20:36, Oliver Clarke <[log in to unmask]> wrote:

Hi Sjors,

We are currently in the same situation - trying to figure out the optimal GPU-workstation setup for a given budget.

How much RAM is too much? That is, if using dual 1080s, how much RAM does one have to have to ensure that the max box size will be limited by the GPU memory, rather than the amount of RAM - because I am presuming that getting any more RAM than that will be a waste (except if you want to preread all your particles into RAM)?

Cheers,

Oliver.

On Mon, Jun 13, 2016 at 2:04 PM, Sjors Scheres <[log in to unmask]> wrote:

Hi Alberto,

Please be advised that hopefully this week we (read Erik Lindahl) will
come out with much more detailed information. I'll also speak about this
at the GRC in Hong Kong. Meanwhile, have a look at my talk at the CCPEM
Spring Symposium here: https://sas.stfc.ac.uk/vportal/index.jsp

For small particles as the ones you mention, four 1080s (or even 1070s)
will surely do. 128Gb of RAM will also be enough, so your cheap machine
should be able to process all those data in a matter of days. You can also
buy several of those gaming machines and tie them together using MPI. I
don't have experience at all with EC2, so I'll let someone else comment on
that.

When you do larger systems (e.g. ribosomes at very high mag, viruses, etc)
and the box size becomes larger than say 450x450 pixels, then the
GPU-memory on the 1080s will become too small, and you'd have to work with
the more expensive cards. For the larger box sizes you would also need
more RAM (again becoming more expensive as you mention yourself). But as
said, for the majority of cases (boxsize <400x400) your 'cheap option'
should do extremely well. I'd also buy a fast (ssd) hard disk with it, as
access to disk will quickly become a next bottleneck.

HTH,
Sjors

> Hello!
>
> I am Alberto, a postdoc from the DNA replication group in Imperial College
> (London). I know that this has been discussed in before but with the
> upcoming release of the new GPU ready Relion it seems that the time for
> buying a new system has come and we still have a few questions. Maybe you
> will be able to help :)
>
> In the lab we mainly work with protein complexes that are around 1 million
> Dalton or a bit bigger, and in our latest reconstructions we have used
> with around 300K particles (sorted from a 1.2 million particle pool) to
> reach around 3.9A resolution. For this we have been using a cluster but we
> wonder if, with the new Relion version, it would be possible to move the
> task to a single workstation. There are different options:
>
> a) gaming computer with an I7 (10 cores) and up to 128GB of RAM; this
> machine would have 4 GTX 1080 cards. This option would be the cheapest,
> something like 5 o 6K should do it, but we cannot increase the RAM any
> further, in fact, 128GB is already a bit over the top.
>
> b) workstation with a single xeon or a couple of CPUs. With this we could
> go up to 36 cores, 1TB of RAM and the same 4 cards or higher. However, it
> would be expensive. Actually, the starting configuration with a single 16
> core xeon and 128GB of RAM plus the cards would go for around 10K
>
> c) we could continue using a cluster and try to something like Amazon
> cloud, or a similar solution that offers GPU acceleration. Has anybody
> tried this?
>
> d) We could go for any other approach that you recommend, like pilling a
> few gaming computers and build our own cluster from them.
>
> Any advice will be much appreciated!
>
> Thanks,
>
> Alberto
>

--
Sjors Scheres
MRC Laboratory of Molecular Biology
Francis Crick Avenue, Cambridge Biomedical Campus
Cambridge CB2 0QH, U.K.
tel: +44 (0)1223 267061
http://www2.mrc-lmb.cam.ac.uk/groups/scheres

-- Alberto Riera PhD Imperial College London Faculty of Medicine Institute of Clinical Sciences (CRB room 3006) Hammersmith Hospital Campus Du Cane Road London W12 0NN [log in to unmask]

Erik Lindahl <[log in to unmask]>

Professor of Biophysics, Dept. Biochemistry & Biophysics, Stockholm University

Science for Life Laboratory, Box 1031, 17121 Solna, Sweden