JiscMail Logo
Email discussion lists for the UK Education and Research communities

Help for CCPEM Archives


CCPEM Archives

CCPEM Archives


CCPEM@JISCMAIL.AC.UK


View:

Message:

[

First

|

Previous

|

Next

|

Last

]

By Topic:

[

First

|

Previous

|

Next

|

Last

]

By Author:

[

First

|

Previous

|

Next

|

Last

]

Font:

Proportional Font

LISTSERV Archives

LISTSERV Archives

CCPEM Home

CCPEM Home

CCPEM  January 2019

CCPEM January 2019

Options

Subscribe or Unsubscribe

Subscribe or Unsubscribe

Log In

Log In

Get Password

Get Password

Subject:

Re: motionCor2 enhancement request

From:

Heumann John <[log in to unmask]>

Reply-To:

Heumann John <[log in to unmask]>

Date:

Wed, 9 Jan 2019 16:19:15 -0700

Content-Type:

text/plain

Parts/Attachments:

Parts/Attachments

text/plain (121 lines)

Hi Takanori,

I'd already checked this... the motionCor2 processes are distributed 1 
per gpu as you'd hope, and memory usage on each gpu looks well within 
limits.  Also, it wouldn't seem to make sense that going to 1 mpi rank 
would change anything if gpu memory was really the limit.

Based on this and the stack trace I'm inclined to think is must be a 
system rather than gpu memory issue. This is a 128 GB system, and I've 
run 4 mpi / 4 gpu motion correction with super-res movies on it quite a 
few times  previously with no problems. The motionCor2 processes seem to 
use ~ 25 - 52 GB of VM each, and I see resident set sizes of <= 10 GB. 
My best guess at the moment is that somebody else must have run some 
process(es) which consumed enough memory that we ran out of swap space 
or physical RAM. This seems consistent with all the symptoms, including 
the sporadic behavior, and doesn't require assuming a problem in either 
the wrapper or motionCor2.

This  system is currently heavily loaded, but I'll check again when it's 
idle and let you know if the issue goes away.
If that turns out to be the case, I apologize for the false alarm!

Regards,
-jh-

On 1/9/19 12:24 PM, Takanori Nakane wrote:
> Hi,
>
> Please check which MotionCor2 process uses which GPU by
> nvidia-smi.
>
> > if I use 1 mpi rank and manually assign 4 gpus with all the other 
> options the same the run completes successfully.
>
> Do you have enough memory to run four MotionCor2 processes
> simultaneously? Super-res K2 movies are very big.
>
> Best regards,
>
> Takanori Nakane
>
> On 2019/01/09 19:06, John Heumann wrote:
>> Hi Takanori (et al),
>>
>> I'm afraid it looks to me like an issue with the motionCor2 mpi 
>> wrapper may have reemerged or been reintroduced. Using yesterdays  
>> (0ad5bd01eb2d1d3b01f65fad1f5c8cbbefe94704) commit on a particular 
>> compressed tiff dataset with super-res K2 movies, if I assign use 4 
>> mpi ranks and 4 gpus, I get a signal 6 (allocation) failure, 
>> regardless of whether  I assign the gpus manually with the colon 
>> operator or leave that field blank in the gui and let Relion do the 
>> assignment.
>>
>> Conversely, if I use 1 mpi rank and manually assign 4 gpus with all 
>> the other options the same the run completes successfully. (I used 
>> blanks as delimiters out of habit,  so I haven't verified Takanori's 
>> recent fix to the wrapper, I assume that would have worked as well).
>>
>> The 2 failing runstrings were
>>
>> `which relion_run_motioncorr_mpi` --i Import/job001/movies.star --o 
>> MotionCorr/job004/ --first_frame_sum 1 --last_frame_sum -1 
>> --use_motioncor2  --motioncor2_exe /usr/bin/MotionCor2 --gpu "" 
>> --bin_factor 2 --bfactor 150 --angpix 0.692 --voltage 300 
>> --dose_per_frame 0.732 --preexposure 0 --patch_x 5 --patch_y 5 
>> --gainref Movies/gainRef.mrc --gain_rot 0 --gain_flip 0 
>> --dose_weighting  --only_do_unfinished
>>
>> and
>>
>> `which relion_run_motioncorr_mpi` --i Import/job001/movies.star --o 
>> MotionCorr/job006/ --first_frame_sum 1 --last_frame_sum -1 
>> --use_motioncor2  --motioncor2_exe /usr/bin/MotionCor2 --gpu 
>> "0:1:2:3" --bin_factor 2 --bfactor 150 --angpix 0.692 --voltage 300 
>> --dose_per_frame 0.732 --preexposure 0 --patch_x 5 --patch_y 5 
>> --gainref Movies/gainRef.mrc --gain_rot 0 --gain_flip 0 
>> --dose_weighting  --only_do_unfinished
>>
>> The successful one was
>>
>> `which relion_run_motioncorr` --i Import/job001/movies.star --o 
>> MotionCorr/job003/ --first_frame_sum 1 --last_frame_sum -1 
>> --use_motioncor2  --motioncor2_exe /usr/bin/MotionCor2 --gpu "0 1 2 
>> 3" --bin_factor 2 --bfactor 150 --angpix 0.692 --voltage 300 
>> --dose_per_frame 0.732 --preexposure 0 --patch_x 5 --patch_y 5 
>> --gainref Movies/gainRef.mrc --gain_rot 0 --gain_flip 0 
>> --dose_weighting  --only_do_unfinished
>>
>> I've attached one of the run.err files showing the stack trace. These 
>> are 8 Gb GTX1080s. During at least the early part of the processing, 
>> only 4-5 Gb per gpu seems to be used. Also, a variable number  (~20 - 
>> 80) of movies get corrected successfully before the failure occurs, 
>> so perhaps this reflect a memory leak of some kind?
>>
>> Of course, running with 1 mpi rank provides an adequate workaround.
>>
>> Thanks in advance!
>>
>> Regards,
>> -jh-
>>
>>
>>
>>
>> ########################################################################
>>
>> To unsubscribe from the CCPEM list, click the following link:
>> https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCPEM&A=1
>>
>

-- 
John M. Heumann
Department of Molecular, Cellular, and Developmental Biology
347 UCB, University of Colorado
Boulder, CO 80309-0347

########################################################################

To unsubscribe from the CCPEM list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCPEM&A=1

Top of Message | Previous Page | Permalink

JiscMail Tools


RSS Feeds and Sharing


Advanced Options


Archives

January 2019
December 2018
November 2018
October 2018
September 2018
August 2018
July 2018
June 2018
May 2018
April 2018
March 2018
February 2018
January 2018
December 2017
November 2017
October 2017
September 2017
August 2017
July 2017
June 2017
May 2017
April 2017
March 2017
February 2017
January 2017
December 2016
November 2016
October 2016
September 2016
August 2016
July 2016
June 2016
May 2016
April 2016
March 2016
February 2016
January 2016
December 2015
November 2015
October 2015
September 2015
August 2015
July 2015
June 2015
May 2015
April 2015
March 2015
February 2015
January 2015
December 2014
November 2014
October 2014
September 2014
August 2014
July 2014
June 2014
May 2014
April 2014
March 2014
February 2014
January 2014
December 2013
November 2013
October 2013
September 2013
August 2013
July 2013
June 2013
May 2013
April 2013
March 2013
February 2013


JiscMail is a Jisc service.

View our service policies at https://www.jiscmail.ac.uk/policyandsecurity/ and Jisc's privacy policy at https://www.jisc.ac.uk/website/privacy-notice

Secured by F-Secure Anti-Virus CataList Email List Search Powered by the LISTSERV Email List Manager