Just a small addition to what Sjors already said. 

We are using the pre-read to RAM option most of the time and I highly recommend it. Our workstations have 512 GB or RAM, 32 CPU cores and four GPUs. A simple rule of thumb for how much memory you'll need is (extracted particles data size) x (number of MPI processes), e.g. if your particle data is 20 GB and you run 3 MPIs (the minimum required for refinements) you'll need 60 GB + a few more GB for other stuff. To utilize all CPU cores of the workstation, just add more threads, e.g. a job with 3 MPIs and 10 threads will utilize 30 CPU cores. We have not tired the SSD scratch option because we don't have SSDs.

Cheers,

Rado


-----Original Message-----
From: Collaborative Computational Project in Electron cryo-Microscopy [mailto:[log in to unmask]] On Behalf Of Sjors Scheres
Sent: 06 December 2016 17:10
To: [log in to unmask]
Subject: Re: [ccpem] Continue 3D refinement run for the final iteration

Hi Lu,
Yes! I would not recommend buying a machine with less than 64GB for exactly that reason. With 64GB you can do 3A-resolution reconstructions of ribosomes (450x450 images), but probably not of large viruses
(600x600 pixels) for example. If you want to reconstruct very large particles, then putting in 256GB or 0.5-1 TB of RAM may be a good idea. 
I think Erik recommends 64GB because after that the prices increase quite steeply, and not many people would need the large amounts of RAM. 
Reading images from a fast SSD disk is alsmot as fast as keeping them all in RAM, and probably a lot cheaper.
HTH,
Sjors


On 12/06/2016 03:49 PM, Lu Gan wrote:
> Hi Dari,
>
> Can I follow up Huy's question with one about about CPU memory 
> requirements (in double precision mode)? In RELION 2, there's an option to "pre-read"
> all the particles into RAM, meaning that if you can install 1TB RAM 
> into your workstation, then you can load 1TB worth of images into 
> memory. In contrast, Erik's blog recommends 64GB for the "high-end" 
> workstation -- presumably for those who will not pre-read particles. 
> Because many of us will not have access to workstations with 1TB (or 
> more!) RAM, what would be the minimum amount of RAM to "future-proof" 
> a workstation? In other words, for the M step, is there a relationship 
> between CPU RAM, box size, particle numbers, and class numbers like 
> you and Bjorn showed in Fig 6 of your Elife paper?
>
> Thanks.
>
> Cheers.
> Lu
>
> --
> Lu Gan
> Assistant Professor
> Department of Biological Sciences
> Centre for BioImaging Sciences
> National University of Singapore
> 14 Science Drive 4
> S1A, Lvl 2
> Singapore 117543
>
> www.anaphase.org
>
> Tel: (65) 6516 8868
> Fax: (65) 6776 7882
>
> On Tue, Dec 6, 2016 at 11:25 PM, Dari Kimanius <[log in to unmask]> wrote:
>
>> The maximization subroutine is the part of refinement that we 
>> consider sensitive to precision loss. This is one of the reason why 
>> it's not currently GPU-accelerated and is hence only effected by the 
>> configured precision of the CPU code. To my knowledge this should 
>> also apply to Relion 1.4.
>>
>> Regards,
>> Dari
>>
>> On Tue, Dec 6, 2016 at 4:16 PM, Huy Bui <[log in to unmask]> wrote:
>>
>>> Thanks, Dari.
>>>
>>> Just want to clarify a bit.
>>> - GPU runs in single precision, but the detrimental effects on the 
>>> results from single precision are only from the CPU part?
>>> - Does the effect of single precision affect Relion 2 only or the 
>>> same effect can be said about single precision compiled Relion 1.4?
>>>
>>> Best regards,
>>> Huy
>>>
>>> On Tue, Dec 6, 2016 at 10:12 AM, Dari Kimanius <[log in to unmask]>
>>> wrote:
>>>
>>>> Similar to the cmake flag DoublePrec_GPU that is set to OFF by 
>>>> default there is also a DoublePrec_CPU that conversely is set to ON by default.
>>>> Here's an example on how to set both CPU and GPU to single 
>>>> precision at configuration time:
>>>>
>>>> cmake -DDoublePrec_GPU=OFF -DDoublePrec_CPU=OFF ..
>>>>
>>>> However, we currently don't recommend running cpu code in single 
>>>> precision since the accuracy loss in some critical parts of the 
>>>> code could have detrimental effects on the results. Please make 
>>>> sure you evaluate the results carefully if you use a singel-cpu build.
>>>>
>>>> Regards,
>>>> Dari
>>>>
>>>> On Tue, Dec 6, 2016 at 4:02 PM, Huy Bui <[log in to unmask]> wrote:
>>>>
>>>>> Thanks a lot, Sjors & Dari. It works.
>>>>>
>>>>> Is there any option to compile Relion2 with single precision for CPU?
>>>>> That would make processing large box size so much easier.
>>>>>
>>>>> Best regards,
>>>>> Huy
>>>>>
>>>>> On Tue, Dec 6, 2016 at 3:18 AM, Sjors Scheres < 
>>>>> [log in to unmask]> wrote:
>>>>>
>>>>>> Hi Huy,
>>>>>> --force_converge as additional argument when continuing an old 
>>>>>> job will do exactly that.
>>>>>> HTH,
>>>>>> Sjors
>>>>>>
>>>>>>> Dear all,
>>>>>>>
>>>>>>> I am doing my 3D refinement with Relion2 and GPU. Often, my
>>>>>> refinement
>>>>>>> will
>>>>>>> report convergence and enter the last iteration and crash due to
>>>>>> lack of
>>>>>>> memory because of the large box size of my sample.
>>>>>>>
>>>>>>> If I tried to continue run, instead of going into the last 
>>>>>>> iteration
>>>>>> to
>>>>>>> construct the unfil.mrc files right away, refinement normally 
>>>>>>> goes
>>>>>> through
>>>>>>> several iterations then converges. With the 1080 GTX card, it 
>>>>>>> will
>>>>>> never
>>>>>>> be
>>>>>>> able to finish the last one due to the limit of 8GB memory.
>>>>>>>
>>>>>>> Is there any way to trick Relion to go directly to the final
>>>>>> iteration
>>>>>>> after convergence and crash? It will be incredibly helpful since 
>>>>>>> the
>>>>>> GPU
>>>>>>> refinement is quick but then will crash due to lack of memory 
>>>>>>> and we
>>>>>> can
>>>>>>> continue the last iteration with CPU.
>>>>>>>
>>>>>>> Best regards,
>>>>>>> Huy
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> Sjors Scheres
>>>>>> MRC Laboratory of Molecular Biology Francis Crick Avenue, 
>>>>>> Cambridge Biomedical Campus Cambridge CB2 0QH, U.K.
>>>>>> tel: +44 (0)1223 267061
>>>>>> http://www2.mrc-lmb.cam.ac.uk/groups/scheres
>>>>>>
>>>>>>

--
Sjors Scheres
MRC Laboratory of Molecular Biology
Francis Crick Avenue, Cambridge Biomedical Campus Cambridge CB2 0QH, U.K.
tel: +44 (0)1223 267061
http://www2.mrc-lmb.cam.ac.uk/groups/scheres