JiscMail Logo
Email discussion lists for the UK Education and Research communities

Help for CCPEM Archives


CCPEM Archives

CCPEM Archives


CCPEM@JISCMAIL.AC.UK


View:

Message:

[

First

|

Previous

|

Next

|

Last

]

By Topic:

[

First

|

Previous

|

Next

|

Last

]

By Author:

[

First

|

Previous

|

Next

|

Last

]

Font:

Proportional Font

LISTSERV Archives

LISTSERV Archives

CCPEM Home

CCPEM Home

CCPEM  December 2016

CCPEM December 2016

Options

Subscribe or Unsubscribe

Subscribe or Unsubscribe

Log In

Log In

Get Password

Get Password

Subject:

Re: Continue 3D refinement run for the final iteration

From:

Sjors Scheres <[log in to unmask]>

Reply-To:

Sjors Scheres <[log in to unmask]>

Date:

Sat, 10 Dec 2016 10:03:42 -0000

Content-Type:

text/plain

Parts/Attachments:

Parts/Attachments

text/plain (548 lines)

Dear Cristina,
This estimate is a very approximate one for the CPU. If you have 128 GB of
RAM, then the GPU memory is probably the problem here. You can restart the
job to do the last iteration on the CPU cluster using the --force_converge
option.
HTH,
Sjors


> Dear all,
>
> just to clarify something for me. In the last iteration, when it
> converged, it estimates the amount of required memory, like here:
> 	 Estimated memory for expectation  step > 9.84028 Gb.
>  	 Estimated memory for maximization step > 4.58846 Gb.
> Does this refer to the “normal” RAM memory or in case of using GPUs
> the memory of the GPU?
>
> I juts had such a case myself that on my 4 GTX1080 (which have roughly
> ~8GB of internal memory) the last step of refinement failed due to lack of
> memory.
> I tried to run the same last iteration by indicating less MPIs/threads in
> the hope it would get enough memory then (my system has 128GB RAM). But it
> failed as well and I had to switch to my cluster.
>
> Best,
> Cristina
>
>
>> On Dec 9, 2016, at 1:09 PM, Lu Gan <[log in to unmask]> wrote:
>>
>> Hi Dari,
>>
>> Thanks. I just confirmed that on our 2-GPU workstation, I can run 3
>> ranks @ 6 threads. Using "top", I see > 300% CPU per rank. It's also
>> amazing that my job didn't crash even though my CPU has 12 hyperthreads
>> max.
>>
>>
>> Cheers.
>> Lu
>>
>> --
>> Lu Gan
>> Assistant Professor
>> Department of Biological Sciences
>> Centre for BioImaging Sciences
>> National University of Singapore
>> 14 Science Drive 4
>> S1A, Lvl 2
>> Singapore 117543
>>
>> www.anaphase.org <http://www.anaphase.org/>
>>
>> Tel: (65) 6516 8868
>> Fax: (65) 6776 7882
>>
>> On Fri, Dec 9, 2016 at 7:48 PM, Dari Kimanius <[log in to unmask]
>> <mailto:[log in to unmask]>> wrote:
>> Hi Lu,
>>
>> It's the threads that are assigned to the GPUs. So you will have to have
>> at least 4 threads in total to use all GPUs on a 4-GPU-system. For two
>> slave ranks, that's a minimum of two threads per rank.
>>
>> There's no way to control the number of threads on individual ranks.
>> However, the threads on the master rank don't occupy any significant
>> resources, so you shouldn't be too worried about them.
>>
>> Cheers,
>> Dari
>>
>>
>> On Fri, Dec 9, 2016 at 12:11 PM, Lu Gan <[log in to unmask]
>> <mailto:[log in to unmask]>> wrote:
>> Hi Dari,
>>
>> I'm a little confused about which factor (threads vs. mpi ranks) feeds
>> the data to the GPUs. Your last comment was: "You can go as low as 2
>> slave ranks (3 with the master rank), even on a 4-GPU-system and still
>> use all GPUs." Are you saying that one could use 2 slave ranks x 2
>> threads, and that the total 4 threads will keep the 4 GPUs busy? Or will
>> 2 slave ranks x 1 thread apiece also keep the 4 GPUs busy? In any case,
>> could we get the most out of CPU RAM by using, say, 2 slave ranks x 6
>> threads apiece, which Sjors might have alluded to?
>>
>> Another related question: it sounds like more threads going to the
>> master rank doesn't help much, so could one somehow give the master only
>> 1 thread to save thread counts for the slave ranks?
>>
>> Thanks.
>>
>>
>> Cheers.
>> Lu
>>
>> --
>> Lu Gan
>> Assistant Professor
>> Department of Biological Sciences
>> Centre for BioImaging Sciences
>> National University of Singapore
>> 14 Science Drive 4
>> S1A, Lvl 2
>> Singapore 117543
>>
>> www.anaphase.org <http://www.anaphase.org/>
>>
>> Tel: (65) 6516 8868
>> Fax: (65) 6776 7882
>>
>> On Fri, Dec 9, 2016 at 7:00 PM, Dari Kimanius <[log in to unmask]
>> <mailto:[log in to unmask]>> wrote:
>> Hi,
>>
>> To sum it up: For optimal performance use 1-2 MPI slave ranks per GPU
>> and >2 threads per rank. If you keep the total number of threads (slave
>> ranks times threads/rank) roughly equal to the number of hyper-threaded
>> cores, you are likely getting very close to optimal performance given
>> the hardware. If you run out of memory, reduce the number of ranks and
>> replace them with threads. You can go as low as 2 slave ranks (3 with
>> the master rank), even on a 4-GPU-system and still use all GPUs.
>>
>> Cheers,
>> Dari
>>
>>
>> On Fri, Dec 9, 2016 at 10:14 AM, Sjors Scheres
>> <[log in to unmask] <mailto:[log in to unmask]>> wrote:
>> Agreed. And depending on how many CPU cores your system has, you can get
>> a small further speed improvement by even using 5-6 threads per mpi.
>> HTH,
>> Sjors
>>
>>
>> On 12/08/2016 09:57 PM, Bharat Reddy wrote:
>> Hi Huy,
>> Generally I use one mpi process (with two treads each) per gpu plus one
>> mpi process for the master process. This requires a minimum of 8-9 cpu
>> cores for a 4 gpu system. Sometimes you will notice your gpus aren't
>> being used fully (use the nvidia-smi command to view this) and if you
>> have enough cpu cores and are not using all your GPU memory you can use
>> 2 mpi processes per gpu plus one master. Often this might give you a
>> bit more performance. If you don't have enough gpu memory during the
>> run this can cause your job to fail or if you don't have enough cpu
>> cores you might not really notice much of a speedup.
>> Cheers,BR
>> On Thu, 2016-12-08 at 14:47 -0500, Huy Bui wrote:
>> Dear Dari,
>> I have another question about numbers of MPI process for refinement.
>> Since I got 4 GPU, does it mean that most optimally I should use 5
>> MPI? As I understand, one MPI will be master one and will not use
>> GPU?
>>
>> Best regards,
>> Huy
>> On Wed, Dec 7, 2016 at 11:30 AM, Dari Kimanius <[log in to unmask]
>> <mailto:[log in to unmask]>>
>> wrote:
>> Yes, and this will leave space for more MPI ranks instead of
>> threads, which currently makes relion go slightly faster.
>> PS. If you're on a workstation you can always put your extract-dir
>> on the scratch space and symlink it into the project-dir. This way
>> you avoid copying to scratch all together.
>>
>> Cheers,
>> Dari
>> On Wed, Dec 7, 2016 at 5:17 PM, Huy Bui <[log in to unmask]
>> <mailto:[log in to unmask]>> wrote:
>> Thanks Dari for the info.
>>  From your email, I guess for my system with SSD Raid scratch
>> disk, 4 x GTX 1080, 256GB ram and 20 cores, I should always use
>> the --scratch_dir option and don't use --pre_read to reduce the
>> RAM usage?
>>
>> Best,
>> Huy
>> On Wed, Dec 7, 2016 at 5:04 AM, Dari Kimanius <[log in to unmask]
>> <mailto:[log in to unmask]>
>> m> wrote:
>> In our benchmarks we observed similar performance reading from
>> two SSDs in RAID as that with preread. We did however not
>> include these benchmarks since the needed RAM size to preread
>> common datasets requires a much more expensive system.
>> Maximum memory reqirenments depend only on the boxsize and
>> number of MPI ranks sharing the node, if you store your image
>> data on RAIDed SSDs. In addition you get the same performance
>> as preread.
>> When using the GPUs with large box sizes the GPU memory is
>> usually far more limiting than RAM. Hence with 8GB in GPU
>> memory you will mostly do fine with 64GB of RAM. However, as
>> mentioned, if you run out of GPU memory in the last refinement
>> steps (approaching Nyqvist) you will need to do a continue with
>> force-converge without the GPUs. Here you may require more RAM
>> than 64GB. For this we have very few CPU only nodes with large
>> RAM size.
>>
>> Also noteworthy: When doing preread without
>> --no_parallel_disc_io, all the slave ranks will create an extra
>> copy of the image data in RAM. With --no_parallel_disc_io only
>> one copy will be stored in RAM by the master rank. This creates
>> an extra MPI overhead when data is distributed by the master
>> rank. We haven't done benchmarks with parallel disc-IO and
>> preread, since this at least double your memory requirements.
>>
>> Cheers,
>> Dari
>> On Wed, Dec 7, 2016 at 10:33 AM, Radostin Danev <[log in to unmask]
>> mpg.de <http://mpg.de/>> wrote:
>> My previous comment about the memory requirement may not be
>> true. I just RTFM for the pre-read option and there it says
>> that if parallel disk I/O is set to No then the particle data
>> will be read only once for the MPI master. Is that so? I
>> don't remember seeing this info in earlier Relion 2.0 builds.
>>
>>
>>
>> Cheers,
>>
>>
>>
>> Rado
>>
>>
>>
>>
>>
>> -----Original Message-----
>>
>> From: Collaborative Computational Project in Electron cryo-
>> Microscopy [mailto:[log in to unmask] <mailto:[log in to unmask]>]
>> On Behalf Of
>> Radostin Danev
>>
>> Sent: 07 December 2016 09:58
>>
>> To: [log in to unmask] <mailto:[log in to unmask]>
>>
>> Subject: Re: [ccpem] Continue 3D refinement run for the final
>> iteration
>>
>>
>>
>> Just a small addition to what Sjors already said.
>>
>>
>>
>> We are using the pre-read to RAM option most of the time and
>> I highly recommend it. Our workstations have 512 GB or RAM,
>> 32 CPU cores and four GPUs. A simple rule of thumb for how
>> much memory you'll need is (extracted particles data size) x
>> (number of MPI processes), e.g. if your particle data is 20
>> GB and you run 3 MPIs (the minimum required for refinements)
>> you'll need 60 GB + a few more GB for other stuff. To utilize
>> all CPU cores of the workstation, just add more threads, e.g.
>> a job with 3 MPIs and 10 threads will utilize 30 CPU cores.
>> We have not tired the SSD scratch option because we don't
>> have SSDs.
>>
>>
>>
>> Cheers,
>>
>>
>>
>> Rado
>>
>>
>>
>>
>>
>> -----Original Message-----
>>
>> From: Collaborative Computational Project in Electron cryo-
>> Microscopy [mailto:[log in to unmask] <mailto:[log in to unmask]>]
>> On Behalf Of Sjors
>> Scheres
>>
>> Sent: 06 December 2016 17:10
>>
>> To: [log in to unmask] <mailto:[log in to unmask]>
>>
>> Subject: Re: [ccpem] Continue 3D refinement run for the final
>> iteration
>>
>>
>>
>> Hi Lu,
>>
>> Yes! I would not recommend buying a machine with less than
>> 64GB for exactly that reason. With 64GB you can do 3A-
>> resolution reconstructions of ribosomes (450x450 images), but
>> probably not of large viruses
>>
>> (600x600 pixels) for example. If you want to reconstruct very
>> large particles, then putting in 256GB or 0.5-1 TB of RAM may
>> be a good idea.
>>
>> I think Erik recommends 64GB because after that the prices
>> increase quite steeply, and not many people would need the
>> large amounts of RAM.
>>
>> Reading images from a fast SSD disk is alsmot as fast as
>> keeping them all in RAM, and probably a lot cheaper.
>>
>> HTH,
>>
>> Sjors
>>
>>
>>
>>
>>
>> On 12/06/2016 03:49 PM, Lu Gan wrote:
>>
>> Hi Dari,
>> Can I follow up Huy's question with one about about CPU
>> memory
>>
>> requirements (in double precision mode)? In RELION 2,
>> there's an option to "pre-read"
>>
>> all the particles into RAM, meaning that if you can install
>> 1TB RAM
>>
>> into your workstation, then you can load 1TB worth of
>> images into
>>
>> memory. In contrast, Erik's blog recommends 64GB for the
>> "high-end"
>>
>> workstation -- presumably for those who will not pre-read
>> particles.
>>
>> Because many of us will not have access to workstations
>> with 1TB (or
>>
>> more!) RAM, what would be the minimum amount of RAM to
>> "future-proof"
>>
>> a workstation? In other words, for the M step, is there a
>> relationship
>>
>> between CPU RAM, box size, particle numbers, and class
>> numbers like
>>
>> you and Bjorn showed in Fig 6 of your Elife paper?
>> Thanks.
>> Cheers.
>> Lu
>> --
>> Lu Gan
>> Assistant Professor
>> Department of Biological Sciences
>> Centre for BioImaging Sciences
>> National University of Singapore
>> 14 Science Drive 4
>> S1A, Lvl 2
>> Singapore 117543
>> www.anaphase.org <http://www.anaphase.org/>
>> Tel: (65) 6516 8868
>> Fax: (65) 6776 7882
>> On Tue, Dec 6, 2016 at 11:25 PM, Dari Kimanius <dkimanius@g
>> mail.com <http://mail.com/>> wrote:
>>
>> The maximization subroutine is the part of refinement that
>> we
>>
>> consider sensitive to precision loss. This is one of the
>> reason why
>>
>> it's not currently GPU-accelerated and is hence only
>> effected by the
>>
>> configured precision of the CPU code. To my knowledge this
>> should
>>
>> also apply to Relion 1.4.
>> Regards,
>> Dari
>> On Tue, Dec 6, 2016 at 4:16 PM, Huy Bui <[log in to unmask]
>> <mailto:[log in to unmask]>
>> wrote:
>> Thanks, Dari.
>> Just want to clarify a bit.
>> - GPU runs in single precision, but the detrimental
>> effects on the
>>
>> results from single precision are only from the CPU part?
>> - Does the effect of single precision affect Relion 2
>> only or the
>>
>> same effect can be said about single precision compiled
>> Relion 1.4?
>>
>> Best regards,
>> Huy
>> On Tue, Dec 6, 2016 at 10:12 AM, Dari Kimanius <dkimanius
>> @gmail.com <http://gmail.com/>>
>>
>> wrote:
>> Similar to the cmake flag DoublePrec_GPU that is set to
>> OFF by
>>
>> default there is also a DoublePrec_CPU that conversely
>> is set to ON by default.
>>
>> Here's an example on how to set both CPU and GPU to
>> single
>>
>> precision at configuration time:
>> cmake -DDoublePrec_GPU=OFF -DDoublePrec_CPU=OFF ..
>> However, we currently don't recommend running cpu code
>> in single
>>
>> precision since the accuracy loss in some critical parts
>> of the
>>
>> code could have detrimental effects on the results.
>> Please make
>>
>> sure you evaluate the results carefully if you use a
>> singel-cpu build.
>>
>> Regards,
>> Dari
>> On Tue, Dec 6, 2016 at 4:02 PM, Huy Bui <[log in to unmask]
>> om> wrote:
>>
>> Thanks a lot, Sjors & Dari. It works.
>> Is there any option to compile Relion2 with single
>> precision for CPU?
>>
>> That would make processing large box size so much
>> easier.
>>
>> Best regards,
>> Huy
>> On Tue, Dec 6, 2016 at 3:18 AM, Sjors Scheres <
>> [log in to unmask] <mailto:[log in to unmask]>> wrote:
>> Hi Huy,
>> --force_converge as additional argument when
>> continuing an old
>>
>> job will do exactly that.
>> HTH,
>> Sjors
>> Dear all,
>> I am doing my 3D refinement with Relion2 and GPU.
>> Often, my
>>
>> refinement
>> will
>> report convergence and enter the last iteration and
>> crash due to
>>
>> lack of
>> memory because of the large box size of my sample.
>> If I tried to continue run, instead of going into the
>> last
>>
>> iteration
>> to
>> construct the unfil.mrc files right away, refinement
>> normally
>>
>> goes
>> through
>> several iterations then converges. With the 1080 GTX
>> card, it
>>
>> will
>> never
>> be
>> able to finish the last one due to the limit of 8GB
>> memory.
>>
>> Is there any way to trick Relion to go directly to
>> the final
>>
>> iteration
>> after convergence and crash? It will be incredibly
>> helpful since
>>
>> the
>> GPU
>> refinement is quick but then will crash due to lack
>> of memory
>>
>> and we
>> can
>> continue the last iteration with CPU.
>> Best regards,
>> Huy
>> --
>> Sjors Scheres
>> MRC Laboratory of Molecular Biology Francis Crick
>> Avenue,
>>
>> Cambridge Biomedical Campus Cambridge CB2 0QH, U.K.
>> tel: +44 (0)1223 267061 <tel:%2B44%20%280%291223%20267061>
>> http://www2.mrc-lmb.cam.ac.uk/groups/scheres
>> <http://www2.mrc-lmb.cam.ac.uk/groups/scheres>
>>
>>
>> --
>>
>> Sjors Scheres
>>
>> MRC Laboratory of Molecular Biology
>>
>> Francis Crick Avenue, Cambridge Biomedical Campus Cambridge
>> CB2 0QH, U.K.
>>
>> tel: +44 (0)1223 267061 <tel:%2B44%20%280%291223%20267061>
>>
>> http://www2.mrc-lmb.cam.ac.uk/groups/scheres
>> <http://www2.mrc-lmb.cam.ac.uk/groups/scheres>
>>
>>
>>
>>
>>
>>
>> --
>> Sjors Scheres
>> MRC Laboratory of Molecular Biology
>> Francis Crick Avenue, Cambridge Biomedical Campus
>> Cambridge CB2 0QH, U.K.
>> tel: +44 (0)1223 267061 <tel:%2B44%20%280%291223%20267061>
>> http://www2.mrc-lmb.cam.ac.uk/groups/scheres
>> <http://www2.mrc-lmb.cam.ac.uk/groups/scheres>
>>
>>
>>
>>
>
>


-- 
Sjors Scheres
MRC Laboratory of Molecular Biology
Francis Crick Avenue, Cambridge Biomedical Campus
Cambridge CB2 0QH, U.K.
tel: +44 (0)1223 267061
http://www2.mrc-lmb.cam.ac.uk/groups/scheres

Top of Message | Previous Page | Permalink

JiscMail Tools


RSS Feeds and Sharing


Advanced Options


Archives

May 2024
April 2024
March 2024
February 2024
January 2024
December 2023
November 2023
October 2023
September 2023
August 2023
July 2023
June 2023
May 2023
April 2023
March 2023
February 2023
January 2023
December 2022
November 2022
October 2022
September 2022
August 2022
July 2022
June 2022
May 2022
April 2022
March 2022
February 2022
January 2022
December 2021
November 2021
October 2021
September 2021
August 2021
July 2021
June 2021
May 2021
April 2021
March 2021
February 2021
January 2021
December 2020
November 2020
October 2020
September 2020
August 2020
July 2020
June 2020
May 2020
April 2020
March 2020
February 2020
January 2020
December 2019
November 2019
October 2019
September 2019
August 2019
July 2019
June 2019
May 2019
April 2019
March 2019
February 2019
January 2019
December 2018
November 2018
October 2018
September 2018
August 2018
July 2018
June 2018
May 2018
April 2018
March 2018
February 2018
January 2018
December 2017
November 2017
October 2017
September 2017
August 2017
July 2017
June 2017
May 2017
April 2017
March 2017
February 2017
January 2017
December 2016
November 2016
October 2016
September 2016
August 2016
July 2016
June 2016
May 2016
April 2016
March 2016
February 2016
January 2016
December 2015
November 2015
October 2015
September 2015
August 2015
July 2015
June 2015
May 2015
April 2015
March 2015
February 2015
January 2015
December 2014
November 2014
October 2014
September 2014
August 2014
July 2014
June 2014
May 2014
April 2014
March 2014
February 2014
January 2014
December 2013
November 2013
October 2013
September 2013
August 2013
July 2013
June 2013
May 2013
April 2013
March 2013
February 2013


JiscMail is a Jisc service.

View our service policies at https://www.jiscmail.ac.uk/policyandsecurity/ and Jisc's privacy policy at https://www.jisc.ac.uk/website/privacy-notice

For help and support help@jisc.ac.uk

Secured by F-Secure Anti-Virus CataList Email List Search Powered by the LISTSERV Email List Manager