Hi CCPEM,

I'm running relion-2.1 with openmpi-2.0.1 with 4x1080 cards. I was (and still am) able to perform 2D-classification on ~10k manually-picked particles. However, when I use ~167k particles picked by Gautomatch, the 2D-classification hangs on the first iteration step. Continuing the job gives the attached error.

The hang-up occurs with or without MPI, pre-loading into ram or copying to an SSD, and re-extracting the 167k set with binning.

Are there potential qualities of the auto-picked particles that would result in this hang-up?
Any other ideas on what might be the problem?

Best,

-Tom
 



 === RELION MPI setup ===
 + Number of MPI processes             = 5
 + Number of threads per MPI process  = 4
 + Total number of threads therefore  = 20
 + Master  (0) runs on host            = squirrel
 + Slave     1 runs on host            = squirrel
 + Slave     2 runs on host            = squirrel
 + Slave     3 runs on host            = squirrel
 + Slave     4 runs on host            = squirrel
 =================
 uniqueHost squirrel has 4 ranks.
GPU-ids not specified for this rank, threads will automatically be mapped to available devices.
 Thread 0 on slave 1 mapped to device 0
 Thread 1 on slave 1 mapped to device 0
 Thread 2 on slave 1 mapped to device 0
 Thread 3 on slave 1 mapped to device 0
GPU-ids not specified for this rank, threads will automatically be mapped to available devices.
 Thread 0 on slave 2 mapped to device 1
 Thread 1 on slave 2 mapped to device 1
 Thread 2 on slave 2 mapped to device 1
 Thread 3 on slave 2 mapped to device 1
GPU-ids not specified for this rank, threads will automatically be mapped to available devices.
 Thread 0 on slave 3 mapped to device 2
 Thread 1 on slave 3 mapped to device 2
 Thread 2 on slave 3 mapped to device 2
 Thread 3 on slave 3 mapped to device 2
GPU-ids not specified for this rank, threads will automatically be mapped to available devices.
 Thread 0 on slave 4 mapped to device 3
 Thread 1 on slave 4 mapped to device 3
 Thread 2 on slave 4 mapped to device 3
 Thread 3 on slave 4 mapped to device 3
 Running CPU instructions in double precision. 
 + WARNING: Changing psi sampling rate (before oversampling) to 11.25 degrees, for more efficient GPU calculations
 + On host squirrel: free scratch space = 1785 Gb.
 Copying particles to scratch directory: /SCRATCH/relion_volatile/
8.32/8.32 min ............................................................~~(,_,">
 Estimating initial noise spectra 
8.58/8.58 min ............................................................~~(,_,">
 Estimating accuracies in the orientational assignment ... 
   0/   0 sec ............................................................~~(,_,">
 Auto-refine: Estimated accuracy angles= 999 degrees; offsets= 999 pixels
 CurrentResolution= 15.4305 Angstroms, which requires orientationSampling of at least 6.54545 degrees for a particle of diameter 270 Angstroms
 Oversampling= 0 NrHiddenVariableSamplingPoints= 134400
 OrientationalSampling= 11.25 NrOrientations= 32
 TranslationalSampling= 2 NrTranslations= 21
=============================
 Oversampling= 1 NrHiddenVariableSamplingPoints= 4300800
 OrientationalSampling= 5.625 NrOrientations= 256
 TranslationalSampling= 1 NrTranslations= 84
=============================
 Expectation iteration 1 of 25
0.82/22.90 hrs 

..~~(,_,"> === RELION MPI setup ===
 + Number of MPI processes             = 5
 + Number of threads per MPI process  = 4
 + Total number of threads therefore  = 20
 + Master  (0) runs on host            = squirrel
 + Slave     1 runs on host            = squirrel
 + Slave     2 runs on host            = squirrel
 + Slave     3 runs on host            = squirrel
 + Slave     4 runs on host            = squirrel
 =================
 uniqueHost squirrel has 4 ranks.
GPU-ids not specified for this rank, threads will automatically be mapped to available devices.
 Thread 0 on slave 1 mapped to device 0
 Thread 1 on slave 1 mapped to device 0
 Thread 2 on slave 1 mapped to device 0
 Thread 3 on slave 1 mapped to device 0
GPU-ids not specified for this rank, threads will automatically be mapped to available devices.
 Thread 0 on slave 2 mapped to device 1
 Thread 1 on slave 2 mapped to device 1
 Thread 2 on slave 2 mapped to device 1
 Thread 3 on slave 2 mapped to device 1
GPU-ids not specified for this rank, threads will automatically be mapped to available devices.
 Thread 0 on slave 3 mapped to device 2
 Thread 1 on slave 3 mapped to device 2
 Thread 2 on slave 3 mapped to device 2
 Thread 3 on slave 3 mapped to device 2
GPU-ids not specified for this rank, threads will automatically be mapped to available devices.
 Thread 0 on slave 4 mapped to device 3
 Thread 1 on slave 4 mapped to device 3
 Thread 2 on slave 4 mapped to device 3
 Thread 3 on slave 4 mapped to device 3
 Running CPU instructions in double precision. 
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
with errorcode 1.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------