If you already can fit 5, you might want to run your relion jobs with 5
as well.
:-)
S
On 07/18/2016 12:50 PM, Alberto Riera wrote:
> Hi Sjors,
>
> Thanks for the advice! Luckily, we can use the 4 cards for Relion.
> However, since this system actually allows to fit an extra GPU I was
> considering if it would be worth it :)
>
> Alberto
>
>
> On 18/07/16 12:41, Sjors Scheres wrote:
>> Hi Alberto,
>> Odd number of cards will be used equally efficiency as even numbers
>> for 2D and 3D classification. It's only the 3D auto-refine that makes
>> better use of even number of cards (as each half-set is processed by
>> a different MPI process). If you need 1 card for something else, I
>> would probably choose to run relion with 3 cards rather than
>> complicating your setup with 5 cards....
>> HTH,
>> Sjors
>>
>> On 07/18/2016 10:42 AM, Alberto Riera wrote:
>>> Thank you for all the advice!
>>>
>>> I was thinking a little bit more about this computer and I wonder if
>>> it would be advantageous for us to install an additional GPU. I know
>>> that Relion uses the GPUs better in pairs, so the idea would be to
>>> let 4 cards just for it and use the spare one for some other tasks
>>> if needed. Alternatively, we could use the fifth card with Relion as
>>> well, although it would be less efficient than having six of them.
>>>
>>>
>>> On 14/07/16 16:01, Craig Yoshioka wrote:
>>>> Hi Alberto,
>>>>
>>>> I’d like to add, it might be worthwhile to keep a few of those CPUs
>>>> and RAM on the side for ZFS. I’d highly recommend turning on ZFS
>>>> LZ4 compression, especially on raw frame stacks it’ll save a lot of
>>>> space and be faster I/O.
>>>>
>>>>
>>>>
>>>>> On Jul 14, 2016, at 5:09 AM, Erik Lindahl <[log in to unmask]
>>>>> <mailto:[log in to unmask]>> wrote:
>>>>>
>>>>> Hi Alberto,
>>>>>
>>>>> On vacation, so sorry if I'm a bit brief.
>>>>>
>>>>> I think your machine will work great if it's intended as a staging
>>>>> server where you do all/most processing. We have a highly similar
>>>>> one, where we also got an external 44-disk enclosure for storage.
>>>>>
>>>>> However, if you want to get more than one machine, we prefer to
>>>>> separate CPU-heavy and GPU-heavy nodes. The reason is that even
>>>>> the GPU code will use CPUs, so you will hurt your GPU performance
>>>>> a lot if you start to run a CPU job on the same machine - and if
>>>>> you don't, you waste those expensive CPUs. For this reason we
>>>>> bought one set of CPU-only nodes with fast processors and then
>>>>> cheap CPUs for the GPU nodes - on the whole I think that provides
>>>>> better value for cluster nodes.
>>>>>
>>>>> We no longer use RAID cards, but have moved to ZFS instead. If you
>>>>> use ubuntu 16.04 it even comes built-in.
>>>>>
>>>>> The released version of oculus rift unfortunately no longer
>>>>> supports plain display mode, so unless a program has been adapted
>>>>> to their devkit, you won't be able to use it.
>>>>>
>>>>> Cheers,
>>>>>
>>>>> Erik
>>>>>
>>>>>
>>>>>
>>>>> Erik Lindahl <[log in to unmask]
>>>>> <mailto:[log in to unmask]>>
>>>>> Professor of Biophysics
>>>>> Science for Life Laboratory
>>>>> Stockholm University & KTH
>>>>> Office (SciLifeLab): +46 8 524 81567
>>>>> Cell (Sweden): +46 73 4618050
>>>>> Cell (US): +1 267 3078746
>>>>>
>>>>>
>>>>> On 14 juli 2016, at 07:24, Alberto Riera
>>>>> <[log in to unmask] <mailto:[log in to unmask]>>
>>>>> wrote:
>>>>>
>>>>>> Hi guys!
>>>>>>
>>>>>> After a lot of days working on this we finally have defined a
>>>>>> system to work with the GPU implementation of Relion and it would
>>>>>> be great it you could tell us your opinion. Currently it is a bit
>>>>>> on the overkill side from the CPU point of view, but we managed
>>>>>> to obtain a very good price and, on the other hand, it is future
>>>>>> proof in this regard and in case we need to install more RAM (it
>>>>>> goes up to 2TB).
>>>>>>
>>>>>> So, it is going to be a Supermicro 4U GPU chassis (workstation
>>>>>> format) with the following:
>>>>>>
>>>>>> 2x Intel E5-2697v4 2.3Ghz Processors (36 cores)
>>>>>> 8x Supermicro 16GB DDR4 ECC REG Memory
>>>>>> 1x 960GB SATA Read Intensive Enterprise Level SSD
>>>>>> 4x 2TB SATA Mixed Workload Enterprise Level SSD (RAID 5)
>>>>>> 1x LSI 9361-8i RAID Controller inc CacheVault
>>>>>> 4x NVidia GTX 1080
>>>>>> 1x Mellanox Single Port 10Gb SFP+ NIC
>>>>>>
>>>>>> Any opinions on it?
>>>>>>
>>>>>> On the other hand, we are looking to do some 3D visualization
>>>>>> with the system and I wonder if any of you has experience with
>>>>>> Linux and the new VR headsets. They seem very promising but I
>>>>>> still did not have the chance to test them. If you have any other
>>>>>> preferred option, suggestions are very welcome :).
>>>>>>
>>>>>> Thanks!
>>>>>>
>>>>>> Alberto
>>>>>>
>>>>>> --
>>>>>> Alberto Riera PhD
>>>>>> Imperial College London
>>>>>> Faculty of Medicine
>>>>>> Institute of Clinical Sciences (CRB room 3006)
>>>>>> Hammersmith Hospital Campus
>>>>>> Du Cane Road
>>>>>> London W12 0NN
>>>>>>
>>>>>> [log in to unmask]
>>>>
>>>
>>
>
--
Sjors Scheres
MRC Laboratory of Molecular Biology
Francis Crick Avenue, Cambridge Biomedical Campus
Cambridge CB2 0QH, U.K.
tel: +44 (0)1223 267061
http://www2.mrc-lmb.cam.ac.uk/groups/scheres
|