Hi Duncan,
to confirm:
should it be a single processor in the server ?
Thanks
Elena
On 5 Dec 2019, at 13:58, Duncan Rand <[log in to unmask]> wrote:
> Hi Matt
>
> It's the next generation of technology, the AMD EPYC has the faster PCIe Gen4 slot(s) whereas the Intel CPUs I looked at have PCIe Gen3. The AMD CPUs can also use faster 3200 MT/s memory, whereas the Intels I looked at are limited to DIMM speeds up to 2666 MT/s.
>
> Note, that only some of the Mellanox ConnectX-5 cards use PCIe Gen4, see page 4 here
>
> https://www.mellanox.com/related-docs/prod_adapter_cards/PB_ConnectX-5_EN_Card.pdf
>
> Duncan
>
> On 05/12/2019 11:00, Matt Doidge wrote:
>> Thanks Duncan,
>> Our vendor came back to us asking why we were specifying AMD over Intel - and I embarrassingly couldn't remember the technical reasoning (something about AMD CPUs being able to better drive those big network cards?). My response was "a wise man told us so" - which whilst correct isn't all that useful!
>> If you (or anyone with better recall then I) could refresh my memory on this that would be great. I'm not a fan of doing things by rote.
>> Cheers,
>> Matt
>> On 05/12/2019 10:18, Duncan Rand wrote:
>>> This email originated from outside of the University. Do not click links or open attachments unless you recognise the sender and know the content is safe.
>>>
>>> Hi
>>>
>>> So the vendor-neutral specification that includes 4 memory DIMMs would be
>>>
>>> AMD EPYC 7262 3.20GHz, 8C/16T, 128M Cache (155W) DDR4-3200
>>> (4) 8GB RDIMM, 3200MT/s, Single Rank
>>> (2) 240GB SSD SATA
>>> Dual, Hot Plug, Redundant Power Supply (1+1)
>>> iDRAC9, Enterprise
>>> Mellanox ConnectX-5 Dual Port 10/25GbE SFP28 Adapter, PCIe Low Profile
>>>
>>> and then those sites needing 40G/100G would buy this in addition:
>>>
>>> Mellanox ConnectX-5 EX Dual Port 40/100GbE QSFP28 Adapter, PCIe Low Profile
>>>
>>> As I understand it all sites should buy their own apart from QMUL,
>>> Glasgow and Bristol.
>>>
>>> Duncan
>>>
>>> On 25/11/2019 14:18, Duncan Rand wrote:
>>>> Thanks Pete. Is that the latest version of that document? It doesn't
>>>> mention the 7262.
>>>>
>>>> Duncan
>>>>
>>>> On 22/11/2019 15:10, Pete Gronbech wrote:
>>>>> Please see :
>>>>> https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdeveloper.amd.com%2Fwp-content%2Fresources%2F56301_1.0.pdf&data=02%7C01%7Cm.doidge%40lancaster.ac.uk%7C10140b14efac4f254bd308d7796c9453%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C1%7C637111379578124485&sdata=KLdLVCoVfL4KUclAByiUVjPNXC%2BQ7iAkbFX2YJTSqSs%3D&reserved=0 for
>>>>> details of the memory channels.
>>>>> Performance is reduced if you specify less than the ideal of 4 DIMMS
>>>>> per CPU.
>>>>>
>>>>> I'm assuming this config is based on 1 AMD CPU.
>>>>>
>>>>> Pete
>>>>>
>>>>> --
>>>>> ----------------------------------------------------------------------
>>>>> Peter Gronbech
>>>>>
>>>>> Head of IT
>>>>> Department of Physics,
>>>>> University of Oxford,
>>>>> Denys Wilkinson Building, Tel No. : 01865 272233
>>>>> Keble Road, Oxford OX1 3RH, UK E-mail :
>>>>> [log in to unmask]<mailto:[log in to unmask]>
>>>>> ----------------------------------------------------------------------
>>>>>
>>>>>
>>>>> From: Testbed Support for GridPP member institutes
>>>>> <[log in to unmask]> On Behalf Of Alexander Dibbo - UKRI STFC
>>>>> Sent: 22 November 2019 14:48
>>>>> To: [log in to unmask]
>>>>> Subject: Re: perfSONAR specification
>>>>>
>>>>> Hi All,
>>>>>
>>>>> I don't believe the PCI-E 4 versions of these nics are generally
>>>>> available yet. The last time I spoke to Mellanox they we looking at Q1
>>>>> 2020.
>>>>>
>>>>> You would need 2 of the PCI-E 3 nics to do 2 100Gb links at full
>>>>> bandwidth.
>>>>>
>>>>> Regards
>>>>>
>>>>> Alexander Dibbo - Cloud Architect
>>>>> For STFC Cloud Documentation visit
>>>>> https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fstfc-cloud-docs.readthedocs.io&data=02%7C01%7Cm.doidge%40lancaster.ac.uk%7C10140b14efac4f254bd308d7796c9453%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C1%7C637111379578124485&sdata=LbEEIPdRPk32%2FIYsC9MFBP%2F7Z4SsBSxElwxYd0IMUBo%3D&reserved=0<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fstfc-cloud-docs.readthedocs.io%2F&data=02%7C01%7Cm.doidge%40lancaster.ac.uk%7C10140b14efac4f254bd308d7796c9453%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C1%7C637111379578124485&sdata=AEpkCpdx7%2BjJh387DInOeXo8ZH1ME7JJTzV%2FDSLMDJ8%3D&reserved=0>
>>>>>
>>>>> To raise a support ticket with the cloud team please email
>>>>> [log in to unmask]<mailto:[log in to unmask]>
>>>>> To receive notifications about the service please subscribe to our
>>>>> mailing list at:
>>>>> https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.jiscmail.ac.uk%2Fcgi-bin%2Fwebadmin%3FA0%3DSTFC-CLOUD&data=02%7C01%7Cm.doidge%40lancaster.ac.uk%7C10140b14efac4f254bd308d7796c9453%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C1%7C637111379578124485&sdata=vveW%2FcOXKw0TouO%2BauiZQgcvsaqxlwT4zpgkJt%2BCFco%3D&reserved=0
>>>>> To receive fast notifications or to discuss usage of the cloud please
>>>>> join our Slack: https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fstfc-cloud.slack.com%2F&data=02%7C01%7Cm.doidge%40lancaster.ac.uk%7C10140b14efac4f254bd308d7796c9453%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C1%7C637111379578124485&sdata=B%2F8IpTPzHRLCVJ%2FYNhgu7AKD7E146AnvO%2FrL5T2zgXw%3D&reserved=0
>>>>>
>>>>> From: Testbed Support for GridPP member institutes
>>>>> <[log in to unmask]<mailto:[log in to unmask]>> On
>>>>> Behalf Of George, Simon
>>>>> Sent: 22 November 2019 14:42
>>>>> To: [log in to unmask]<mailto:[log in to unmask]>
>>>>> Subject: Re: perfSONAR specification
>>>>>
>>>>> Thanks for this Duncan.
>>>>> I can see the advantage of focusing on one vendor as an example but we
>>>>> will presumably need to get quotes from other vendors so satisfy our
>>>>> purchasing rules.
>>>>> So once you have settled on the spec, please could you list in a
>>>>> vendor-neutral way the required features of the server model you have
>>>>> chosen (e.g. PCIe slots & bandwidth, CPU features)?
>>>>> Thanks,
>>>>> Simon
>>>>>
>>>>> ________________________________
>>>>> From: Testbed Support for GridPP member institutes
>>>>> <[log in to unmask]<mailto:[log in to unmask]>> on
>>>>> behalf of Daniel Traynor
>>>>> <[log in to unmask]<mailto:[log in to unmask]>>
>>>>> Sent: 22 November 2019 13:24
>>>>> To: [log in to unmask]<mailto:[log in to unmask]>
>>>>> <[log in to unmask]<mailto:[log in to unmask]>>
>>>>> Subject: Re: perfSONAR specification
>>>>>
>>>>> AMD EPYC have quad memory channels so 4 memory DIMMS would have higher
>>>>> memory bandwidth than 2 DIMMS (is it important for perfsonar?)
>>>>>
>>>>> Is the 100G network card PCI-3 x16 ? this only has a bandwidth of
>>>>> 128Gb/s, is this enough to drive 2 100G links? I guess the latency
>>>>> link won't need the full bandwidth to run tests.
>>>>>
>>>>> Don't forget to add the price of the cables! Might be able to get away
>>>>> with 3/5 M direct attached cables. 100G cables not that cheep.
>>>>>
>>>>> dan
>>>>>
>>>>> * Dr Daniel Traynor, Grid cluster system manager
>>>>> * Tel +44(0)20 7882 6560, Particle Physics,QMUL
>>>>>
>>>>>
>>>>> ________________________________________
>>>>> From: Testbed Support for GridPP member institutes
>>>>> <[log in to unmask]<mailto:[log in to unmask]>> on
>>>>> behalf of Duncan Rand
>>>>> <[log in to unmask]<mailto:[log in to unmask]>>
>>>>> Sent: 22 November 2019 12:31
>>>>> To: [log in to unmask]<mailto:[log in to unmask]>
>>>>> Subject: perfSONAR specification
>>>>>
>>>>> Hi
>>>>>
>>>>> Just to remind everyone we are looking for GridPP sites to refresh their
>>>>> perfSONAR hosts. There is a requirement for the hosts to be the same
>>>>> specification at each site and capable of 100G at sites that either are
>>>>> already connected at 100G (such as RAL and Imperial) or likely soon to
>>>>> be upgraded. Rather than two hosts we propose only one host now with
>>>>> three interfaces, one for management, one for latency and loss and one
>>>>> for throughput (either 10G or 100G).
>>>>>
>>>>> Following on from Martin's suggestion to try AMD, it seems that Dell
>>>>> R6515 which is 1U
>>>>>
>>>>> https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.dell.com%2Fen-us%2Fwork%2Fshop%2Fpovw%2Fpoweredge-r6515&data=02%7C01%7Cm.doidge%40lancaster.ac.uk%7C10140b14efac4f254bd308d7796c9453%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C1%7C637111379578124485&sdata=Ghf%2FR4vMlNQQx0cCkMZCBA6q2KOeIg5Ly58ABDtkRlg%3D&reserved=0
>>>>>
>>>>>
>>>>> or R7515 which is 2U would seem good value. These come with 8 core AMD
>>>>> EPYC 3.2GHz and 3200 MT/s memory. Furthermore they both have one or more
>>>>> PCIe Gen4 slots. So I would like to propose the following initial
>>>>> possible draft specification:
>>>>>
>>>>> PowerEdge R6515
>>>>> Chassis with up to 4x3.5" Drives
>>>>> PowerEdge R6515/R7515 Motherboard, with 2 x 1Gb Onboard LOM (BCM5720)
>>>>> AMD EPYC 7262 3.20GHz, 8C/16T, 128M Cache (155W) DDR4-3200
>>>>> (2) 16GB RDIMM, 3200MT/s, Dual Rank
>>>>> (2) 240GB SSD SATA Mixed Use 6Gbps 512e 2.5in Hot plug, 3.5in HYB CARR
>>>>> S4610
>>>>> Dual, Hot Plug, Redundant Power Supply (1+1), 550W
>>>>> iDRAC9, Enterprise X5
>>>>> Mellanox ConnectX-5 Dual Port 10/25GbE SFP28 Adapter, PCIe Low Profile
>>>>>
>>>>> and then those sites needing 40G/100G could buy this in addition:
>>>>>
>>>>> Mellanox ConnectX-5 EX Dual Port 40/100GbE QSFP28 Adapter, PCIe Low
>>>>> Profile;
>>>>>
>>>>> The Dell website indicates the cost would be about £4000 for the basic
>>>>> version and about £5400 with the extra 100G card.
>>>>>
>>>>> Thoughts?
>>>>>
>>>>> Duncan
>>>>>
>>>>> ########################################################################
>>>>>
>>>>> To unsubscribe from the TB-SUPPORT list, click the following link:
>>>>> https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.jiscmail.ac.uk%2Fcgi-bin%2Fwebadmin%3FSUBED1%3DTB-SUPPORT%26A%3D1&data=02%7C01%7Cm.doidge%40lancaster.ac.uk%7C10140b14efac4f254bd308d7796c9453%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C1%7C637111379578134479&sdata=pYfySRha8QYaZv%2F3efAxw49wNHd1gqx882FiKZfxOX0%3D&reserved=0
>>>>>
>>>>>
>>>>> ########################################################################
>>>>>
>>>>> To unsubscribe from the TB-SUPPORT list, click the following link:
>>>>> https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.jiscmail.ac.uk%2Fcgi-bin%2Fwebadmin%3FSUBED1%3DTB-SUPPORT%26A%3D1&data=02%7C01%7Cm.doidge%40lancaster.ac.uk%7C10140b14efac4f254bd308d7796c9453%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C1%7C637111379578134479&sdata=pYfySRha8QYaZv%2F3efAxw49wNHd1gqx882FiKZfxOX0%3D&reserved=0
>>>>>
>>>>> ________________________________
>>>>>
>>>>> To unsubscribe from the TB-SUPPORT list, click the following link:
>>>>> https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.jiscmail.ac.uk%2Fcgi-bin%2Fwebadmin%3FSUBED1%3DTB-SUPPORT%26A%3D1&data=02%7C01%7Cm.doidge%40lancaster.ac.uk%7C10140b14efac4f254bd308d7796c9453%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C1%7C637111379578134479&sdata=pYfySRha8QYaZv%2F3efAxw49wNHd1gqx882FiKZfxOX0%3D&reserved=0
>>>>>
>>>>> ________________________________
>>>>>
>>>>> To unsubscribe from the TB-SUPPORT list, click the following link:
>>>>> https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.jiscmail.ac.uk%2Fcgi-bin%2Fwebadmin%3FSUBED1%3DTB-SUPPORT%26A%3D1&data=02%7C01%7Cm.doidge%40lancaster.ac.uk%7C10140b14efac4f254bd308d7796c9453%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C1%7C637111379578134479&sdata=pYfySRha8QYaZv%2F3efAxw49wNHd1gqx882FiKZfxOX0%3D&reserved=0
>>>>>
>>>>> ########################################################################
>>>>>
>>>>> To unsubscribe from the TB-SUPPORT list, click the following link:
>>>>> https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.jiscmail.ac.uk%2Fcgi-bin%2Fwebadmin%3FSUBED1%3DTB-SUPPORT%26A%3D1&data=02%7C01%7Cm.doidge%40lancaster.ac.uk%7C10140b14efac4f254bd308d7796c9453%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C1%7C637111379578134479&sdata=pYfySRha8QYaZv%2F3efAxw49wNHd1gqx882FiKZfxOX0%3D&reserved=0
>>>>>
>>>>
>>>> ########################################################################
>>>>
>>>> To unsubscribe from the TB-SUPPORT list, click the following link:
>>>> https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.jiscmail.ac.uk%2Fcgi-bin%2Fwebadmin%3FSUBED1%3DTB-SUPPORT%26A%3D1&data=02%7C01%7Cm.doidge%40lancaster.ac.uk%7C10140b14efac4f254bd308d7796c9453%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C1%7C637111379578134479&sdata=pYfySRha8QYaZv%2F3efAxw49wNHd1gqx882FiKZfxOX0%3D&reserved=0
>>>
>>> ########################################################################
>>>
>>> To unsubscribe from the TB-SUPPORT list, click the following link:
>>> https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.jiscmail.ac.uk%2Fcgi-bin%2Fwebadmin%3FSUBED1%3DTB-SUPPORT%26A%3D1&data=02%7C01%7Cm.doidge%40lancaster.ac.uk%7C10140b14efac4f254bd308d7796c9453%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C1%7C637111379578134479&sdata=pYfySRha8QYaZv%2F3efAxw49wNHd1gqx882FiKZfxOX0%3D&reserved=0
>> ########################################################################
>> To unsubscribe from the TB-SUPPORT list, click the following link:
>> https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=TB-SUPPORT&A=1
>
> ########################################################################
>
> To unsubscribe from the TB-SUPPORT list, click the following link:
> https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=TB-SUPPORT&A=1
########################################################################
To unsubscribe from the TB-SUPPORT list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=TB-SUPPORT&A=1
|