Hi Dominik,
Did you look at the clock speed of your GPU's during the run? The fact
you said you had 80C peaks leads me to believe your GPUs might be
throttling themselves. The consumer based founders editions cards I've
used have horrible default fan settings. They prioritize noise over
performance. This in turn causes your cards to start throttling and can
hurt your performance by 5-20%. In order to bypass the default gpu fan
settings you need to enable coolbits. The problem is that cool bits
only works on screens that have X sessions running on them. To by pass
this problem the following github project by Boris Dimitrov based on
the work of Axel Kohlmeyer solves it:
https://github.com/boris-dimitrov/set_gpu_fans_public . It sets up dummy screens and X sessions on each
gpu and has a nifty script to automatically ramp up and down the gpu
fan as needed based on gpu temperature. I used this on all our
workstations and this helps keep the gpus running at nearly top speeds
at all times and overall temperature below 78C. The only drawback is
noise.
Cheers,
BR
_________________________________
Bharat Reddy, Ph.D.
Perozo Lab, University of Chicago
Email:
[log in to unmask]Tel: (773) 834 - 4734
On Mon, 2017-03-06 at 17:12 +0100, Dominik A. Herbst wrote:
Hi Benoit,
Approx. half a year go we bought together with our HPC team six GPU
nodes from DALCO (dual Xeon E5-2680, 512 GB RAM, 1.6 TB Intel NVMe
PCIe SSD, 4xTitan X, Infiniband, 2 HE chassis, CentOS 7).
Before we bought them we were running Relion2 jobs on 4x GTX 1080 GPU
workstations (DALCO, Samsung M2 NVMe 950 Pro, i7-6900K, 64-128 GB
RAM, 1 GBit/s Ethernet, CentOS 6) as described on Erik Lindahl's
homepage.
I did plenty of benchmarking. All tests were done using the same data
set and random seed. In all cases the Titan-X GPU nodes showed
approx. 25 % higher performance, which is in agreement to literature.
(The workstations with 4x GTX 1080 had a performance of approx. 10
cluster nodes with dual Xeon E5-2650, 64 GB RAM, infiniband (~ 320
cores / no GPUs))
The 2 HE GPU node chassis/boards provide 8 PCIe slots, whereof 4 are
used for TitanXs and one for the scratch SSD. In order to check how
performance scales with more TitanX we equipped one node with 7
TitanX. I was running benchmarks using the scratch with the same
random seed and the same data set (120,000 ptcls, 210px box).
Results for the usage of all possible resources for 7, 6 and 4 GPUs
on one GPU node:
(rpn = MPI ranks per node; ppr = processes per rank / threads; gpu =
gpus)
rpn14_ppr2_gpu7: #28 slots
real 23m46.751s
user 309m5.491s
sys 50m48.316s
rpn8_ppr3_gpu7: #24 slots
real 24m42.213s
user 234m41.726s
sys 41m33.935s
rpn13_ppr2_gpu6: #26 slots
real 25m6.495s
user 312m3.790s
sys 52m24.046s
rpn7_ppr4_gpu6:#28 slots
real 27m9.320s
user 255m1.701s
sys 48m11.140s
rpn9_ppr3_gpu4: #27 slots
real 32m54.921s
user 370m40.358s
sys 61m44.317s
rpn5_ppr5_gpu4: #25 slots
real 38m54.711s
user 283m0.446s
sys 55m42.250s
With more than 4 GPUs the GPUs were never running with full
utilization (>90%), but in a range of 50-70%.
Based on a direct comparison (real time improvement):
GPUs used (relative)
performance increase for 1 rank/GPU performance increase for 2
ranks/GPU
4-->6 = 2 more
30.2% (15%/GPU) 23.8% (12%/GPU)
4-->7 = 3 more
36.5% (12%/GPU) 27.8% (9%/GPU)
This tells me that 6 GPUs scales better than 7 on this 28 core
machine, which is why we plan to upgrade all nodes to 6 TitanX.
Surprisingly, the temperature was very moderate (~ 60-70°C, 80°C at
the peak) despite of the high packing density, but it might be that
our HPC team did some chassis fan tuning.
Currently we are using the Univa Grid Engine, which comes with some
problems for running hybrid smp-mpi jobs, but it works.
Unfortunately, UGE (/SGE) cannot run hybrid smp-mpi-gpu jobs on
several nodes, which limits your job request to one node.
If you want to use GPUs on several nodes, Slurm is a better choice
and we will switch to it soon.
However, in our case we had severe issues with core binding of mpi
processes. Often all of them were bound to the first cores, even when
a second job was started they ended up on the same cores (!!!),
unless mpirun was started with the "--bind-to none" parameter.
Furthermore, I recommend to provide a $GPU_ASSIGN variable with your
Relion2 (module-)installation, that generates the --gpu string from
the SGE variables ($SGE_HGR_gpu_dev, $NSLOTS and $OMP_NUM_THREADS).
If you like, I can provide you the bash script.
In my opinion this is particularly important, because if the --gpu
X,x,x,x:Y,y,y,y:... parameter is not set, Relion2 will use ALL
resources and distribute the job itself. This is particularly bad if
a second job will be started on the same node, because the two jobs
will compete for the same resources and once one job took all the GPU
memory, the other job will die. Note that the --j and the --gpu
parameter work differently: --j takes only what you assign (perfekt
for queueing systems), --gpu takes everything it can get, unless you
restrict it (not ideal for queueing systems).
Concerning the OS, please note that the Nvidia drivers for the Pascal
cards are not well supported by CentOS6/RHEL6 and you might want to
switch to CentOS7/RHEL7.
I hope this helps!
Best,
Dominik
On 03/06/2017 12:00 PM, Beno ît Zuber wrote:
Hi Daniel, Erki, and Masahide,
Thank you for your feedback!
Best
Benoit
De : "[log in to unmask]" <[log in to unmask]>
Date : lundi, 6 mars 2017 à 08:29
À : Benoît Zuber <[log in to unmask]>
Objet : AW: geforce vs tesla
Hallo Benoit,
ETH ist currently setting up a cluster with NVidia GTX1080 GPUs for
big data (https://scicomp.ethz.ch/wiki/Leonhard). We could not test
it yet but Relion2 should run on the GPU nodes.
Best,
Daniel
Von: Collaborative Computational Project in Electron cryo-
Microscopy [[log in to unmask]]" im Auftrag von "Benoît Zuber [be
[log in to unmask]]
Gesendet: Montag, 6. März 2017 06:58
An: [log in to unmask]
Betreff: [ccpem] geforce vs tesla
Hello,
Our HPC cluster team is collecting wishes before building up a new
GPU cluster. They consider implementing Tesla or Geforce cards.
With the new 1080 Ti card and its 11 Gb RAM, is there any reason to
go for Tesla cards when considering performance for Relion 2?
Thanks for your input
Benoit