JiscMail Logo
Email discussion lists for the UK Education and Research communities

Help for CCPEM Archives


CCPEM Archives

CCPEM Archives


CCPEM@JISCMAIL.AC.UK


View:

Message:

[

First

|

Previous

|

Next

|

Last

]

By Topic:

[

First

|

Previous

|

Next

|

Last

]

By Author:

[

First

|

Previous

|

Next

|

Last

]

Font:

Proportional Font

LISTSERV Archives

LISTSERV Archives

CCPEM Home

CCPEM Home

CCPEM  January 2019

CCPEM January 2019

Options

Subscribe or Unsubscribe

Subscribe or Unsubscribe

Log In

Log In

Get Password

Get Password

Subject:

Re: motionCor2 enhancement request

From:

Heumann John <[log in to unmask]>

Reply-To:

Heumann John <[log in to unmask]>

Date:

Wed, 9 Jan 2019 21:45:19 -0700

Content-Type:

text/plain

Parts/Attachments:

Parts/Attachments

text/plain (217 lines)

Thanks Shawn! That's good to know.

FWIW, I just finished some further testing and  this  was definitely a  
system rather than gpu memory issue and was caused by competition with 
(in this case non-gpu) process(es) I'd overlooked from other users. 
After making sure I had the system to myself, I tried the run again, and 
it completes with no issues at all. Each of the 4 processes reports 52 
GB of VM usage and up to 10 GB RSS, but most of the VM must consist of 
shared code / data because only about 40 out of 128 GB of physical 
memory gets used (as you'd expect from the reported resident set sizes) 
and only 2.5 out of 4 Gb of swap.

Once again, apologies for the false alarm.

Regards,
-jh-

On 1/9/19 8:47 PM, Shawn Zheng wrote:
> Hi John,
>
> I do not know if this will help. The recent UCSF implementation of 
> MotionCor2 adds a function "-GpuUsage" that specifies number of free 
> GPUs to be used in the process to be started. This function is 
> specifically for running multiple MC2 processes in a multi-GPU system. 
> Whenever a process exits, it frees its GPUs for the next process. We 
> have a workstation with 8 1080Ti cards and 256 GB CPU memory. We 
> routinely run 4 processes with "-Gpu 0 1 2 3 4 5 6 7 -GpuUsage 2", 
> which means each process will select 2 free GPUs from total 8 GPUs to 
> do the job. This configuration allows us to catch up with the data 
> collection running Wim's fast, image-shift scheme.
>
> Another note, MotionCor2's performance scales very well from 1 to 2 
> GPUs and decently from 2 to 3 GPUs. Internally, MC2 uses only 50% GPU 
> memory to buffer the movie. If not not enough, the remaining frames 
> stay in CPU memory and cause significant CPU and GPU traffic. When 
> running multiple MC2 processes, please take into account both CPU 
> memory capacity and data traffic.
>
> Best
> Shawn
>
> On Wed, Jan 9, 2019 at 4:07 PM Heumann John <[log in to unmask] 
> <mailto:[log in to unmask]>> wrote:
>
>     As a quick follow up, if all 4 mpi processes simultaneously really
>     wanted 52 GB or VM, that's more than we could support, an we'd be
>     limited to a max of 2 ranks. Fortunately, the intervals where VM
>     usage
>     shoots up to 52 GB are sporadic and brief. Still, depending on how
>     things behave, I may find it prudent to cut back to 2 or 3 processes
>     instead of 4.
>
>     Regards,
>     -jh-
>
>     On 1/9/19 4:19 PM, Heumann John wrote:
>     > Hi Takanori,
>     >
>     > I'd already checked this... the motionCor2 processes are
>     distributed 1
>     > per gpu as you'd hope, and memory usage on each gpu looks well
>     within
>     > limits.  Also, it wouldn't seem to make sense that going to 1
>     mpi rank
>     > would change anything if gpu memory was really the limit.
>     >
>     > Based on this and the stack trace I'm inclined to think is must
>     be a
>     > system rather than gpu memory issue. This is a 128 GB system,
>     and I've
>     > run 4 mpi / 4 gpu motion correction with super-res movies on it
>     quite
>     > a few times  previously with no problems. The motionCor2 processes
>     > seem to use ~ 25 - 52 GB of VM each, and I see resident set
>     sizes of
>     > <= 10 GB. My best guess at the moment is that somebody else must
>     have
>     > run some process(es) which consumed enough memory that we ran
>     out of
>     > swap space or physical RAM. This seems consistent with all the
>     > symptoms, including the sporadic behavior, and doesn't require
>     > assuming a problem in either the wrapper or motionCor2.
>     >
>     > This  system is currently heavily loaded, but I'll check again when
>     > it's idle and let you know if the issue goes away.
>     > If that turns out to be the case, I apologize for the false alarm!
>     >
>     > Regards,
>     > -jh-
>     >
>     > On 1/9/19 12:24 PM, Takanori Nakane wrote:
>     >> Hi,
>     >>
>     >> Please check which MotionCor2 process uses which GPU by
>     >> nvidia-smi.
>     >>
>     >> > if I use 1 mpi rank and manually assign 4 gpus with all the
>     other
>     >> options the same the run completes successfully.
>     >>
>     >> Do you have enough memory to run four MotionCor2 processes
>     >> simultaneously? Super-res K2 movies are very big.
>     >>
>     >> Best regards,
>     >>
>     >> Takanori Nakane
>     >>
>     >> On 2019/01/09 19:06, John Heumann wrote:
>     >>> Hi Takanori (et al),
>     >>>
>     >>> I'm afraid it looks to me like an issue with the motionCor2 mpi
>     >>> wrapper may have reemerged or been reintroduced. Using yesterdays
>     >>> (0ad5bd01eb2d1d3b01f65fad1f5c8cbbefe94704) commit on a particular
>     >>> compressed tiff dataset with super-res K2 movies, if I assign
>     use 4
>     >>> mpi ranks and 4 gpus, I get a signal 6 (allocation) failure,
>     >>> regardless of whether  I assign the gpus manually with the colon
>     >>> operator or leave that field blank in the gui and let Relion
>     do the
>     >>> assignment.
>     >>>
>     >>> Conversely, if I use 1 mpi rank and manually assign 4 gpus
>     with all
>     >>> the other options the same the run completes successfully. (I
>     used
>     >>> blanks as delimiters out of habit,  so I haven't verified
>     Takanori's
>     >>> recent fix to the wrapper, I assume that would have worked as
>     well).
>     >>>
>     >>> The 2 failing runstrings were
>     >>>
>     >>> `which relion_run_motioncorr_mpi` --i
>     Import/job001/movies.star --o
>     >>> MotionCorr/job004/ --first_frame_sum 1 --last_frame_sum -1
>     >>> --use_motioncor2 --motioncor2_exe /usr/bin/MotionCor2 --gpu ""
>     >>> --bin_factor 2 --bfactor 150 --angpix 0.692 --voltage 300
>     >>> --dose_per_frame 0.732 --preexposure 0 --patch_x 5 --patch_y 5
>     >>> --gainref Movies/gainRef.mrc --gain_rot 0 --gain_flip 0
>     >>> --dose_weighting  --only_do_unfinished
>     >>>
>     >>> and
>     >>>
>     >>> `which relion_run_motioncorr_mpi` --i
>     Import/job001/movies.star --o
>     >>> MotionCorr/job006/ --first_frame_sum 1 --last_frame_sum -1
>     >>> --use_motioncor2 --motioncor2_exe /usr/bin/MotionCor2 --gpu
>     >>> "0:1:2:3" --bin_factor 2 --bfactor 150 --angpix 0.692
>     --voltage 300
>     >>> --dose_per_frame 0.732 --preexposure 0 --patch_x 5 --patch_y 5
>     >>> --gainref Movies/gainRef.mrc --gain_rot 0 --gain_flip 0
>     >>> --dose_weighting  --only_do_unfinished
>     >>>
>     >>> The successful one was
>     >>>
>     >>> `which relion_run_motioncorr` --i Import/job001/movies.star --o
>     >>> MotionCorr/job003/ --first_frame_sum 1 --last_frame_sum -1
>     >>> --use_motioncor2  --motioncor2_exe /usr/bin/MotionCor2 --gpu
>     "0 1 2
>     >>> 3" --bin_factor 2 --bfactor 150 --angpix 0.692 --voltage 300
>     >>> --dose_per_frame 0.732 --preexposure 0 --patch_x 5 --patch_y 5
>     >>> --gainref Movies/gainRef.mrc --gain_rot 0 --gain_flip 0
>     >>> --dose_weighting  --only_do_unfinished
>     >>>
>     >>> I've attached one of the run.err files showing the stack trace.
>     >>> These are 8 Gb GTX1080s. During at least the early part of the
>     >>> processing, only 4-5 Gb per gpu seems to be used. Also, a
>     variable
>     >>> number  (~20 - 80) of movies get corrected successfully before
>     the
>     >>> failure occurs, so perhaps this reflect a memory leak of some
>     kind?
>     >>>
>     >>> Of course, running with 1 mpi rank provides an adequate
>     workaround.
>     >>>
>     >>> Thanks in advance!
>     >>>
>     >>> Regards,
>     >>> -jh-
>     >>>
>     >>>
>     >>>
>     >>>
>     >>>
>     ########################################################################
>
>     >>>
>     >>>
>     >>> To unsubscribe from the CCPEM list, click the following link:
>     >>> https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCPEM&A=1
>     >>>
>     >>
>     >
>
>     -- 
>     John M. Heumann
>     Department of Molecular, Cellular, and Developmental Biology
>     347 UCB, University of Colorado
>     Boulder, CO 80309-0347
>
>     ########################################################################
>
>     To unsubscribe from the CCPEM list, click the following link:
>     https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCPEM&A=1
>

-- 
John M. Heumann
Department of Molecular, Cellular, and Developmental Biology
347 UCB, University of Colorado
Boulder, CO 80309-0347

########################################################################

To unsubscribe from the CCPEM list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCPEM&A=1

Top of Message | Previous Page | Permalink

JiscMail Tools


RSS Feeds and Sharing


Advanced Options


Archives

January 2019
December 2018
November 2018
October 2018
September 2018
August 2018
July 2018
June 2018
May 2018
April 2018
March 2018
February 2018
January 2018
December 2017
November 2017
October 2017
September 2017
August 2017
July 2017
June 2017
May 2017
April 2017
March 2017
February 2017
January 2017
December 2016
November 2016
October 2016
September 2016
August 2016
July 2016
June 2016
May 2016
April 2016
March 2016
February 2016
January 2016
December 2015
November 2015
October 2015
September 2015
August 2015
July 2015
June 2015
May 2015
April 2015
March 2015
February 2015
January 2015
December 2014
November 2014
October 2014
September 2014
August 2014
July 2014
June 2014
May 2014
April 2014
March 2014
February 2014
January 2014
December 2013
November 2013
October 2013
September 2013
August 2013
July 2013
June 2013
May 2013
April 2013
March 2013
February 2013


JiscMail is a Jisc service.

View our service policies at https://www.jiscmail.ac.uk/policyandsecurity/ and Jisc's privacy policy at https://www.jisc.ac.uk/website/privacy-notice

Secured by F-Secure Anti-Virus CataList Email List Search Powered by the LISTSERV Email List Manager