Dear all,
I am working on data-set: 2.5M particles, 200x200 box (stored on beegfs). After updating to relion 3.0.7, I am still facing the problem with 3D classifications/Refinements. Calculations stall/pause after "Estimating initial noise spectra" for circa 5-6h before the iteration starts. The same happens at the end of each iteration after maximization. I have tested multiple running settings ... I/O, combine iterations, pre-reading all particles in RAM(in this case particles are loaded and then it`s stalled), GPUs (P100/V100) x CPUs (AMD/INTEL), multiple nodes vs single node, different node architectures, relion compiled for intel-cpu only, per socket job allocation, different mpi/thread ratios, splitting the data-set in 3x850k particles, amount of classes 1,2,3,4 ...10, fast subsets, joining all particles into one stack and using it as input, skip_align. In fact, nothing helped.
Interestingly, during this "hanging" time nodes are heavily occupied (cpus only, on gpu nodes no job is running running on the gpu until the iteration starts), or only 5/48 cpus are used (according to different ways of submission). But there is still no difference at the end.
Now, I am a bit clueless, if the problem is on our side in configuration or still in the relion 3.0.7 version?
thank you all in advance for any help!
Best Regards,
Jiri W.
/ Thomas C. Marlovits Laboratory \
/ CSSB/UKE Hamburg-Eppendorf \
########################################################################
To unsubscribe from the CCPEM list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCPEM&A=1
|