Hi Sjors,
Thanks for the advice! Luckily, we can use the 4 cards for Relion.
However, since this system actually allows to fit an extra GPU I was
considering if it would be worth it :)
Alberto
On 18/07/16 12:41, Sjors Scheres wrote:
> Hi Alberto,
> Odd number of cards will be used equally efficiency as even numbers
> for 2D and 3D classification. It's only the 3D auto-refine that makes
> better use of even number of cards (as each half-set is processed by a
> different MPI process). If you need 1 card for something else, I would
> probably choose to run relion with 3 cards rather than complicating
> your setup with 5 cards....
> HTH,
> Sjors
>
> On 07/18/2016 10:42 AM, Alberto Riera wrote:
>> Thank you for all the advice!
>>
>> I was thinking a little bit more about this computer and I wonder if
>> it would be advantageous for us to install an additional GPU. I know
>> that Relion uses the GPUs better in pairs, so the idea would be to
>> let 4 cards just for it and use the spare one for some other tasks if
>> needed. Alternatively, we could use the fifth card with Relion as
>> well, although it would be less efficient than having six of them.
>>
>>
>> On 14/07/16 16:01, Craig Yoshioka wrote:
>>> Hi Alberto,
>>>
>>> I’d like to add, it might be worthwhile to keep a few of those CPUs
>>> and RAM on the side for ZFS. I’d highly recommend turning on ZFS
>>> LZ4 compression, especially on raw frame stacks it’ll save a lot of
>>> space and be faster I/O.
>>>
>>>
>>>
>>>> On Jul 14, 2016, at 5:09 AM, Erik Lindahl <[log in to unmask]
>>>> <mailto:[log in to unmask]>> wrote:
>>>>
>>>> Hi Alberto,
>>>>
>>>> On vacation, so sorry if I'm a bit brief.
>>>>
>>>> I think your machine will work great if it's intended as a staging
>>>> server where you do all/most processing. We have a highly similar
>>>> one, where we also got an external 44-disk enclosure for storage.
>>>>
>>>> However, if you want to get more than one machine, we prefer to
>>>> separate CPU-heavy and GPU-heavy nodes. The reason is that even the
>>>> GPU code will use CPUs, so you will hurt your GPU performance a lot
>>>> if you start to run a CPU job on the same machine - and if you
>>>> don't, you waste those expensive CPUs. For this reason we bought
>>>> one set of CPU-only nodes with fast processors and then cheap CPUs
>>>> for the GPU nodes - on the whole I think that provides better value
>>>> for cluster nodes.
>>>>
>>>> We no longer use RAID cards, but have moved to ZFS instead. If you
>>>> use ubuntu 16.04 it even comes built-in.
>>>>
>>>> The released version of oculus rift unfortunately no longer
>>>> supports plain display mode, so unless a program has been adapted
>>>> to their devkit, you won't be able to use it.
>>>>
>>>> Cheers,
>>>>
>>>> Erik
>>>>
>>>>
>>>>
>>>> Erik Lindahl <[log in to unmask]
>>>> <mailto:[log in to unmask]>>
>>>> Professor of Biophysics
>>>> Science for Life Laboratory
>>>> Stockholm University & KTH
>>>> Office (SciLifeLab): +46 8 524 81567
>>>> Cell (Sweden): +46 73 4618050
>>>> Cell (US): +1 267 3078746
>>>>
>>>>
>>>> On 14 juli 2016, at 07:24, Alberto Riera
>>>> <[log in to unmask] <mailto:[log in to unmask]>>
>>>> wrote:
>>>>
>>>>> Hi guys!
>>>>>
>>>>> After a lot of days working on this we finally have defined a
>>>>> system to work with the GPU implementation of Relion and it would
>>>>> be great it you could tell us your opinion. Currently it is a bit
>>>>> on the overkill side from the CPU point of view, but we managed to
>>>>> obtain a very good price and, on the other hand, it is future
>>>>> proof in this regard and in case we need to install more RAM (it
>>>>> goes up to 2TB).
>>>>>
>>>>> So, it is going to be a Supermicro 4U GPU chassis (workstation
>>>>> format) with the following:
>>>>>
>>>>> 2x Intel E5-2697v4 2.3Ghz Processors (36 cores)
>>>>> 8x Supermicro 16GB DDR4 ECC REG Memory
>>>>> 1x 960GB SATA Read Intensive Enterprise Level SSD
>>>>> 4x 2TB SATA Mixed Workload Enterprise Level SSD (RAID 5)
>>>>> 1x LSI 9361-8i RAID Controller inc CacheVault
>>>>> 4x NVidia GTX 1080
>>>>> 1x Mellanox Single Port 10Gb SFP+ NIC
>>>>>
>>>>> Any opinions on it?
>>>>>
>>>>> On the other hand, we are looking to do some 3D visualization with
>>>>> the system and I wonder if any of you has experience with Linux
>>>>> and the new VR headsets. They seem very promising but I still did
>>>>> not have the chance to test them. If you have any other preferred
>>>>> option, suggestions are very welcome :).
>>>>>
>>>>> Thanks!
>>>>>
>>>>> Alberto
>>>>>
>>>>> --
>>>>> Alberto Riera PhD
>>>>> Imperial College London
>>>>> Faculty of Medicine
>>>>> Institute of Clinical Sciences (CRB room 3006)
>>>>> Hammersmith Hospital Campus
>>>>> Du Cane Road
>>>>> London W12 0NN
>>>>>
>>>>> [log in to unmask]
>>>
>>
>
--
Alberto Riera PhD
Imperial College London
Faculty of Medicine
Institute of Clinical Sciences (CRB room 3006)
Hammersmith Hospital Campus
Du Cane Road
London W12 0NN
[log in to unmask]
|