Hi all,
We have been running and testing Relion 2.0 on our 8 GPU nodes to try to figure out the optimal parameters. We thought these results might be interesting to share, and we are looking for any suggestions / comments / similar tests that you could provide.
Our configuration is:
8 GPUs node, 48 slots, TITAN X (Pascal, 750 GB RAM, 11GB on-card), sRelion compiled with gcc 4.8.5, kermel 3.10, Centos 7.3, sd hard drive of 4TB
At first, we only varied the number of GPUs, MPIs and threads, leaving the other disk access options constant (No parallel disc I/O, particles pre-read into the RAM, no "combine iterations"). The results for each type of job are as follow:
*** 2D Classification (265k particles of 280x280 pixels, 5 rounds, 50 classes):
2 GPUs, 3 MPIs, 24 threads: 14h04
4 GPUs, 3 MPIs, 24 threads: 5h23
4 GPUs, 5 MPIs, 12 threads: 13h28
8 GPUs, 3 MPIs, 24 threads: 3h14
8 GPUs, 5 MPIs, 12 threads: 5h10
8 GPUs, 8 MPIs, 6 threads: 13h28
*** 3D Classification (226k particles of 280x280 pixels, 5 rounds, 5 classes):
2 GPUs, 3 MPIs, 24 threads: 15h17
4 GPUs, 3 MPIs, 24 threads: 5h53
4 GPUs, 5 MPIs, 12 threads: 8h11
8 GPUs, 3 MPIs, 24 threads: 2h48
8 GPUs, 5 MPIs, 12 threads: 3h16
8 GPUs, 8 MPIs, 6 threads: 4h37
*** 3D Refinement (116k particles of 280x280 pixels):
2 GPUs, 3 MPIs, 24 threads: 12h07
4 GPUs, 3 MPIs, 24 threads: 4h54
4 GPUs, 5 MPIs, 12 threads: 9h12
8 GPUs, 3 MPIs, 24 threads: 4h57
8 GPUs, 5 MPIs, 12 threads: 4h15
8 GPUs, 8 MPIs, 6 threads: 8h47
*** Auto-picking (on 2600 micrographs, generating around 750 k particles):
0 GPU , 48 threads: 79 min
2 GPUs, 2 threads: 52 min
2 GPUs, 48 threads: error (code 11)
4 GPUs, 4 threads: 27 min
4 GPUs, 48 threads: 8 min
8 GPUs, 8 threads: 19 min
8 GPUs, 48 threads: 12 min
Does anyone have particular suggestions / feedback?
We are using these tests to guide the expansion of our centralized GPU capability and any comment is greatly welcome.
Thanks,
Best,
Nicolas Coudray
New York University
------------------------------------------------------------
This email message, including any attachments, is for the sole use of the intended recipient(s) and may contain information that is proprietary, confidential, and exempt from disclosure under applicable law. Any unauthorized review, use, disclosure, or distribution is prohibited. If you have received this email in error please notify the sender by return email and delete the original message. Please note, the recipient should check this email and any attachments for the presence of viruses. The organization accepts no liability for any damage caused by any virus transmitted by this email.
=================================
|