Hi Songye (and all others),

I was able to stabilize execution of relion compiled specifically for
the Pascal/Maxwell compute architecture, by updating the version of CUB
we use inside relion. I made a new patched version 2.0.6 with these
changes, so pulling the new code should help with this, and other
similar, reported issues.

Under the tests I've run, there is for instance no longer any 2nd
iteration stalls when compiling using e.g.

-DCUDA_ARCH=61

Cheers,

/Björn

On 03/31/2017 11:10 PM, Chen, Songye wrote:
> Hi Björn,
>
> I just want to report to you something happened to us recently, that may or may not be related to this topic. I was trying to install Relion2 on Ubuntu 16.04 with 2 Nvidia Quadro m5000 and CUDA 8.0. I got the latest Relion v2.0.5, and compiled it for cuda compute 52. When I ran Refind3D, it always stalled or ended on the second iteration start, and gives error report "the launch timed out and was terminated in .../relion/src/gpu-utils/cuda_mem_utils.h at line 311". So I compiled Relion again with the default cuda compute 35, and the problem went away. I hope it helps. Please let me know if you have any questions or advice.
>
> Thanks,
> Songye
>
> -----Original Message-----
> From: Collaborative Computational Project in Electron cryo-Microscopy [mailto:[log in to unmask]] On Behalf Of Bjoern Forsberg
> Sent: Thursday, March 16, 2017 7:19 PM
> To: [log in to unmask]
> Subject: Re: [ccpem] Relion2: Refine3D gets stuck at iteration 2
>
> Hi Tru,
>
> This is in fact not the same "stall" issue that was fixed in v2.0.4, but rather the same as discussed recently with respect to compilation specifically against pascal-generation GPUs, i.e. when specifying cuda compute 61. I know this because your logfile specifies a stall at the start of an iteration rather than at the end of an expectation-step.
>
> I'm guessing that you have compiled relion for compute 61 since your binary is called "CC-61". For some reason (looking like a compiler bug at the moment), compiling for 61 seems to cause stalls on the second iteration start. We're on it, but for now, I recommend compiling for cuda compute 52, even if you're going to run on pascal-generation cards, like the TITAN X(pascal), GTX1080, and so on.
>
> /Björn
>
>
> On 2017-03-16 10:36, Tru Huynh wrote:
>> On Wed, Mar 15, 2017 at 10:09:13PM +0100, Bjoern Forsberg wrote:
>>> Any version after 2.0.4 should not display this stall. If you are
>>> sure you are running at least 2.0.4 and are observing that
>>>
>>> - relion stops at the end of an expectation step without
>>> reporting an error
>>> - one GPU is running at 100%, but all others are not doing anything
>>> - one CPU thread per MPI is running at 100%
>>>
>>> then please check the issue linked below and post your output there.
>>> If you do not see all the above symptoms, it something else, and I'd
>>> be grateful if you create new issue on the 3dem/relion github page,
>>> so that we can sort it out.
>>>
>> git built version as of yesterday/git d401f24
>> 1+8 cores/single gpu run:
>>
>> slurm script:
>> [tru@visu1 slurm.d]$ more
>> 9cpus-1gpu-run-nih-c057-201703015-d401f24-openmpi-1.10.6-libltdl-CC-61
>> .sh
>> #!/bin/sh
>> #SBATCH -N 1
>> #SBATCH -n 9
>> #SBATCH --exclusive
>> #SBATCH --gres=gpu:4 --constraint=CC61 export OMP_NUM_THREADS=1 export
>> TMPDIR=/local-storage/tru/relion2/tmpdir
>> module purge
>> module load slurm/16.05.9-hdf5-1.8-hwloc-1.8-munge-0.5.11
>> module load relion/git/201703015-d401f24-openmpi-1.10.6-libltdl-CC-61
>> export OMPI_MCA_btl="self,sm,tcp"
>> CPUS=9
>> cd /local-storage/tru/relion_benchmark || exit 1 #for GPUS in "0" "1"
>> "2" "3"
>> for GPUS in "0" "1" "2" "3"
>> do
>> dropcache.sh
>> NAME=relion-git-201703015-d401f24-openmpi-1.10.6-libltdl-CC-61-cpu${CP
>> US}gpus$(echo ${GPUS}|tr ':' '.')-OMP_NUM_THREADS1 mkdir -p
>> relion/release/${NAME} (time mpirun -n ${CPUS} `which relion_refine_mpi` --o relion/release/${NAME} \
>> --i Particles/shiny_2sets.star --ref emd_2660.map:mrc --firstiter_cc --ini_high 60 \
>> --dont_combine_weights_via_disc --scratch_dir ${TMPDIR} --pool 100 --ctf \
>> --ctf_corrected_ref --iter 25 --tau2_fudge 4 --particle_diameter 360 --K 6 --flatten_solvent \
>> --zero_mask --oversampling 1 --healpix_order 2 --offset_range 5 --offset_step 2 --sym C1 \
>> --norm --scale --j 2 --random_seed 0 \
>> --gpu ${GPUS} ) > ${NAME}.out-`date +'%Y%m%d-%H%M'` 2>&1 done
>>
>> mpirun -n 9
>> /c6/shared/relion/git-d401f24-openmpi-1.10.6-libltdl-CC-61/bin/relion_
>> refine_mpi --o
>> relion/release/relion-git-201703015-d401f24-openmpi-1.10.6-libltdl-CC-
>> 61-cpu9gpus0-OMP_NUM_THREADS1 --i Particles/shiny_2sets.star --ref
>> emd_2660.map:mrc --firstiter_cc --ini_high 60
>> --dont_combine_weights_via_disc --scratch_dir
>> /local-storage/tru/relion2/tmpdir --pool 100 --ctf --ctf_corrected_ref
>> --iter 25 --tau2_fudge 4 --particle_diameter 360 --K 6
>> --flatten_solvent --zero_mask --oversampling 1 --healpix_order 2
>> --offset_range 5 --offset_step 2 --sym C1 --norm --scale --j 2
>> --random_seed 0 --gpu 0
>>
>> output file attached:
>> relion-git-201703015-d401f24-openmpi-1.10.6-libltdl-CC-61-cpu9gpus0-OM
>> P_NUM_THREADS1.out-20170316-0907
>>
>> [tru@c057 relion_benchmark]$ nvidia-smi Thu Mar 16 10:26:04 2017
>> +-----------------------------------------------------------------------------+
>> | NVIDIA-SMI 367.57 Driver Version: 367.57 |
>> |-------------------------------+----------------------+----------------------+
>> | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
>> | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
>> |===============================+======================+======================|
>> | 0 TITAN X (Pascal) Off | 0000:02:00.0 Off | N/A |
>> | 23% 34C P2 79W / 250W | 9542MiB / 12189MiB | 100% Default |
>> +-------------------------------+----------------------+----------------------+
>> | 1 TITAN X (Pascal) Off | 0000:03:00.0 Off | N/A |
>> | 23% 17C P8 9W / 250W | 2MiB / 12189MiB | 0% Default |
>> +-------------------------------+----------------------+----------------------+
>> | 2 TITAN X (Pascal) Off | 0000:81:00.0 Off | N/A |
>> | 23% 19C P8 8W / 250W | 2MiB / 12189MiB | 0% Default |
>> +-------------------------------+----------------------+----------------------+
>> | 3 TITAN X (Pascal) Off | 0000:82:00.0 Off | N/A |
>> | 23% 15C P8 8W / 250W | 2MiB / 12189MiB | 0% Default |
>> +-------------------------------+----------------------+----------------------+
>>
>> +-----------------------------------------------------------------------------+
>> | Processes: GPU Memory |
>> | GPU PID Type Process name Usage |
>> |=============================================================================|
>> | 0 997 C ....10.6-libltdl-CC-61/bin/relion_refine_mpi 1187MiB |
>> | 0 998 C ....10.6-libltdl-CC-61/bin/relion_refine_mpi 1223MiB |
>> | 0 999 C ....10.6-libltdl-CC-61/bin/relion_refine_mpi 1187MiB |
>> | 0 1000 C ....10.6-libltdl-CC-61/bin/relion_refine_mpi 1191MiB |
>> | 0 1001 C ....10.6-libltdl-CC-61/bin/relion_refine_mpi 1189MiB |
>> | 0 1002 C ....10.6-libltdl-CC-61/bin/relion_refine_mpi 1187MiB |
>> | 0 1003 C ....10.6-libltdl-CC-61/bin/relion_refine_mpi 1187MiB |
>> | 0 1004 C ....10.6-libltdl-CC-61/bin/relion_refine_mpi 1187MiB |
>> +-----------------------------------------------------------------------------+
>>
>> [tru@c057 relion_benchmark]$ nvidia-smi topo -m
>> GPU0 GPU1 GPU2 GPU3 CPU Affinity
>> GPU0 X PHB SOC SOC 0-7
>> GPU1 PHB X SOC SOC 0-7
>> GPU2 SOC SOC X PHB 8-15
>> GPU3 SOC SOC PHB X 8-15
>>
>> Legend:
>>
>> X = Self
>> SOC = Connection traversing PCIe as well as the SMP link between CPU sockets(e.g. QPI)
>> PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
>> PXB = Connection traversing multiple PCIe switches (without traversing the PCIe Host Bridge)
>> PIX = Connection traversing a single PCIe switch
>> NV# = Connection traversing a bonded set of # NVLinks
>>
>>
>> top reports:
>>
>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ P TIME COMMAND
>> 999 tru 20 0 40.8g 2.9g 110m S 100.0 1.2 83:34.96 13 83:34 /c6/shared/relion/git-d401f24-openmpi-1.10.6-libltdl-CC-61/bin/relion_refine_mpi --o relion/release/reli
>> 1003 tru 20 0 40.8g 2.9g 110m S 99.0 1.2 83:34.96 15 83:34 /c6/shared/relion/git-d401f24-openmpi-1.10.6-libltdl-CC-61/bin/relion_refine_mpi --o relion/release/relio
>> 1001 tru 20 0 40.8g 2.9g 110m S 99.0 1.2 83:32.09 10 83:32 /c6/shared/relion/git-d401f24-openmpi-1.10.6-libltdl-CC-61/bin/relion_refine_mpi --o relion/release/relio
>> 997 tru 20 0 40.9g 2.9g 111m S 100.0 1.1 82:45.63 9 82:45 /c6/shared/relion/git-d401f24-openmpi-1.10.6-libltdl-CC-61/bin/relion_refine_mpi --o relion/release/reli
>> 998 tru 20 0 40.9g 2.9g 110m S 100.0 1.1 82:21.05 1 82:21 /c6/shared/relion/git-d401f24-openmpi-1.10.6-libltdl-CC-61/bin/relion_refine_mpi --o relion/release/reli
>> 1004 tru 20 0 40.8g 2.9g 110m S 100.0 1.2 82:18.99 0 82:18 /c6/shared/relion/git-d401f24-openmpi-1.10.6-libltdl-CC-61/bin/relion_refine_mpi --o relion/release/reli
>> 1000 tru 20 0 41.0g 2.9g 110m S 100.0 1.2 82:18.11 3 82:18 /c6/shared/relion/git-d401f24-openmpi-1.10.6-libltdl-CC-61/bin/relion_refine_mpi --o relion/release/reli
>> 1002 tru 20 0 40.9g 3.0g 110m S 100.0 1.2 82:17.71 2 82:17 /c6/shared/relion/git-d401f24-openmpi-1.10.6-libltdl-CC-61/bin/relion_refine_mpi --o relion/release/reli
>> 996 tru 20 0 2979m 2.4g 10m R 100.0 1.0 76:50.42 6 76:50 /c6/shared/relion/git-d401f24-openmpi-1.10.6-libltdl-CC-61/bin/relion_refine_mpi --o relion/release/reli
>> 1784 tru 20 0 27860 1816 1100 R 1.0 0.0 0:00.38 5 0:00 top
>> 994 tru 20 0 149m 3556 2368 S 0.0 0.0 0:00.17 1 0:00 mpirun -n 9 /c6/shared/relion/git-d401f24-openmpi-1.10.6-libltdl-CC-61/bin/relion_refine_mpi --o relion/r
>> 931 tru 20 0 111m 2052 1060 S 0.0 0.0 0:00.13 0 0:00 sshd: tru@pts/0
>> 932 tru 20 0 11576 1964 1436 S 0.0 0.0 0:00.05 9 0:00 -bash
>> 977 tru 20 0 9208 1352 1116 S 0.0 0.0 0:00.00 1 0:00 /bin/sh /var/run/slurm/slurmd.state/job15485628/slurm_script
>> 991 tru 20 0 9208 728 488 S 0.0 0.0 0:00.00 0 0:00 /bin/sh /var/run/slurm/slurmd.state/job15485628/slurm_script
>>
>> stracing the lowest cpu used mpi process:
>>
>> [tru@c057 relion_benchmark]$ strace -f -p 996 Process 996 attached
>> with 2 threads [pid 1012] restart_syscall(<... resuming interrupted
>> call ...> <unfinished ...>
>> [pid 996] poll([{fd=5, events=POLLIN}, {fd=11, events=POLLIN}], 2, 0) = 0 (Timeout)
>> [pid 996] poll([{fd=5, events=POLLIN}, {fd=11, events=POLLIN}], 2, 0) = 0 (Timeout)
>> [pid 996] poll([{fd=5, events=POLLIN}, {fd=11, events=POLLIN}], 2, 0) = 0 (Timeout)
>> [pid 996] poll([{fd=5, events=POLLIN}, {fd=11, events=POLLIN}], 2, 0) = 0 (Timeout)
>> [pid 996] poll([{fd=5, events=POLLIN}, {fd=11, events=POLLIN}], 2, 0) = 0 (Timeout)
>> ...forever...
>>
>> Cheers
>>
>> Tru