Hi,
RELION does not always follow the max_memory parameter in auto-refine. I actually noticed this quite a while ago with smaller box sizes. However, now with a larger box size of 432x432 pixels it leads to the process swapping (and potentially crashing the whole cluster job).
max_memory was set to 4, relion runs in auto-refine mode:
Example "top" from one cluster node:
45001 brunner 20 0 4459m 3.8g 2948 S 101 2.7 219:24.94 relion_refine_m
45002 brunner 20 0 6126m 5.6g 2952 S 101 4.0 219:53.65 relion_refine_m
45003 brunner 20 0 6126m 5.5g 2948 S 101 3.9 220:01.13 relion_refine_m
45004 brunner 20 0 6126m 5.6g 2948 S 101 4.0 219:58.11 relion_refine_m
45005 brunner 20 0 6130m 5.6g 2952 S 101 4.0 219:52.58 relion_refine_m
45006 brunner 20 0 6129m 5.6g 2932 S 101 4.0 219:30.97 relion_refine_m
45007 brunner 20 0 6130m 5.6g 2936 S 101 4.0 219:28.75 relion_refine_m
45008 brunner 20 0 6127m 5.5g 2944 S 101 3.9 219:16.86 relion_refine_m
45009 brunner 20 0 6127m 5.5g 2952 S 101 3.8 219:53.24 relion_refine_m
45010 brunner 20 0 4460m 3.8g 2940 S 101 2.7 219:46.30 relion_refine_m
45011 brunner 20 0 6127m 5.6g 2944 S 101 4.0 219:53.37 relion_refine_m
45000 brunner 20 0 1259m 684m 2680 R 25 0.5 187:01.66 relion_refine_m
sampling points:
=============================
Oversampling= 1 NrHiddenVariableSamplingPoints= 14094336
OrientationalSampling= 3.75 NrOrientations= 95232
TranslationalSampling= 1.5 NrTranslations= 148
=============================
The peak memory usage seems to be at the beginning of each expectation iteration, with the usage for instance peaking at 14GB (instead of 4GB) at a sampling of 3.75 degrees.
It seems to get worse the longer RELION is running, probably due to the increase in OrientationalSampling/TranslationalSampling.
Tested with RELION 1.2 beta 9
Is there any remedy for this?
BTW this is not about the last auto-refine iteration!
Cheers
--
Matthias Brunner <[log in to unmask]>
PhD student, Marlovits Lab
-Research Institute of Molecular Pathology (IMP)
-Inst of Molecular Biotechnology of the Austrian Academy of Sciences (IMBA)
Dr. Bohr-Gasse 3, A-1030 Vienna, Austria
Phone: +43 1 79044 4791
|