Hi Takanori,
I'd already checked this... the motionCor2 processes are distributed 1
per gpu as you'd hope, and memory usage on each gpu looks well within
limits. Also, it wouldn't seem to make sense that going to 1 mpi rank
would change anything if gpu memory was really the limit.
Based on this and the stack trace I'm inclined to think is must be a
system rather than gpu memory issue. This is a 128 GB system, and I've
run 4 mpi / 4 gpu motion correction with super-res movies on it quite a
few times previously with no problems. The motionCor2 processes seem to
use ~ 25 - 52 GB of VM each, and I see resident set sizes of <= 10 GB.
My best guess at the moment is that somebody else must have run some
process(es) which consumed enough memory that we ran out of swap space
or physical RAM. This seems consistent with all the symptoms, including
the sporadic behavior, and doesn't require assuming a problem in either
the wrapper or motionCor2.
This system is currently heavily loaded, but I'll check again when it's
idle and let you know if the issue goes away.
If that turns out to be the case, I apologize for the false alarm!
Regards,
-jh-
On 1/9/19 12:24 PM, Takanori Nakane wrote:
> Hi,
>
> Please check which MotionCor2 process uses which GPU by
> nvidia-smi.
>
> > if I use 1 mpi rank and manually assign 4 gpus with all the other
> options the same the run completes successfully.
>
> Do you have enough memory to run four MotionCor2 processes
> simultaneously? Super-res K2 movies are very big.
>
> Best regards,
>
> Takanori Nakane
>
> On 2019/01/09 19:06, John Heumann wrote:
>> Hi Takanori (et al),
>>
>> I'm afraid it looks to me like an issue with the motionCor2 mpi
>> wrapper may have reemerged or been reintroduced. Using yesterdays
>> (0ad5bd01eb2d1d3b01f65fad1f5c8cbbefe94704) commit on a particular
>> compressed tiff dataset with super-res K2 movies, if I assign use 4
>> mpi ranks and 4 gpus, I get a signal 6 (allocation) failure,
>> regardless of whether I assign the gpus manually with the colon
>> operator or leave that field blank in the gui and let Relion do the
>> assignment.
>>
>> Conversely, if I use 1 mpi rank and manually assign 4 gpus with all
>> the other options the same the run completes successfully. (I used
>> blanks as delimiters out of habit, so I haven't verified Takanori's
>> recent fix to the wrapper, I assume that would have worked as well).
>>
>> The 2 failing runstrings were
>>
>> `which relion_run_motioncorr_mpi` --i Import/job001/movies.star --o
>> MotionCorr/job004/ --first_frame_sum 1 --last_frame_sum -1
>> --use_motioncor2 --motioncor2_exe /usr/bin/MotionCor2 --gpu ""
>> --bin_factor 2 --bfactor 150 --angpix 0.692 --voltage 300
>> --dose_per_frame 0.732 --preexposure 0 --patch_x 5 --patch_y 5
>> --gainref Movies/gainRef.mrc --gain_rot 0 --gain_flip 0
>> --dose_weighting --only_do_unfinished
>>
>> and
>>
>> `which relion_run_motioncorr_mpi` --i Import/job001/movies.star --o
>> MotionCorr/job006/ --first_frame_sum 1 --last_frame_sum -1
>> --use_motioncor2 --motioncor2_exe /usr/bin/MotionCor2 --gpu
>> "0:1:2:3" --bin_factor 2 --bfactor 150 --angpix 0.692 --voltage 300
>> --dose_per_frame 0.732 --preexposure 0 --patch_x 5 --patch_y 5
>> --gainref Movies/gainRef.mrc --gain_rot 0 --gain_flip 0
>> --dose_weighting --only_do_unfinished
>>
>> The successful one was
>>
>> `which relion_run_motioncorr` --i Import/job001/movies.star --o
>> MotionCorr/job003/ --first_frame_sum 1 --last_frame_sum -1
>> --use_motioncor2 --motioncor2_exe /usr/bin/MotionCor2 --gpu "0 1 2
>> 3" --bin_factor 2 --bfactor 150 --angpix 0.692 --voltage 300
>> --dose_per_frame 0.732 --preexposure 0 --patch_x 5 --patch_y 5
>> --gainref Movies/gainRef.mrc --gain_rot 0 --gain_flip 0
>> --dose_weighting --only_do_unfinished
>>
>> I've attached one of the run.err files showing the stack trace. These
>> are 8 Gb GTX1080s. During at least the early part of the processing,
>> only 4-5 Gb per gpu seems to be used. Also, a variable number (~20 -
>> 80) of movies get corrected successfully before the failure occurs,
>> so perhaps this reflect a memory leak of some kind?
>>
>> Of course, running with 1 mpi rank provides an adequate workaround.
>>
>> Thanks in advance!
>>
>> Regards,
>> -jh-
>>
>>
>>
>>
>> ########################################################################
>>
>> To unsubscribe from the CCPEM list, click the following link:
>> https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCPEM&A=1
>>
>
--
John M. Heumann
Department of Molecular, Cellular, and Developmental Biology
347 UCB, University of Colorado
Boulder, CO 80309-0347
########################################################################
To unsubscribe from the CCPEM list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCPEM&A=1
|