Thanks Duncan,
Our vendor came back to us asking why we were specifying AMD over Intel - and I embarrassingly couldn't remember the technical reasoning (something about AMD CPUs being able to better drive those big network cards?). My response was "a wise man told us so" - which whilst correct isn't all that useful!
If you (or anyone with better recall then I) could refresh my memory on this that would be great. I'm not a fan of doing things by rote.
Cheers,
Matt
On 05/12/2019 10:18, Duncan Rand wrote:
> This email originated from outside of the University. Do not click links or open attachments unless you recognise the sender and know the content is safe.
>
> Hi
>
> So the vendor-neutral specification that includes 4 memory DIMMs would be
>
> AMD EPYC 7262 3.20GHz, 8C/16T, 128M Cache (155W) DDR4-3200
> (4) 8GB RDIMM, 3200MT/s, Single Rank
> (2) 240GB SSD SATA
> Dual, Hot Plug, Redundant Power Supply (1+1)
> iDRAC9, Enterprise
> Mellanox ConnectX-5 Dual Port 10/25GbE SFP28 Adapter, PCIe Low Profile
>
> and then those sites needing 40G/100G would buy this in addition:
>
> Mellanox ConnectX-5 EX Dual Port 40/100GbE QSFP28 Adapter, PCIe Low Profile
>
> As I understand it all sites should buy their own apart from QMUL,
> Glasgow and Bristol.
>
> Duncan
>
> On 25/11/2019 14:18, Duncan Rand wrote:
>> Thanks Pete. Is that the latest version of that document? It doesn't
>> mention the 7262.
>>
>> Duncan
>>
>> On 22/11/2019 15:10, Pete Gronbech wrote:
>>> Please see :
>>> https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdeveloper.amd.com%2Fwp-content%2Fresources%2F56301_1.0.pdf&data=02%7C01%7Cm.doidge%40lancaster.ac.uk%7C10140b14efac4f254bd308d7796c9453%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C1%7C637111379578124485&sdata=KLdLVCoVfL4KUclAByiUVjPNXC%2BQ7iAkbFX2YJTSqSs%3D&reserved=0 for
>>> details of the memory channels.
>>> Performance is reduced if you specify less than the ideal of 4 DIMMS
>>> per CPU.
>>>
>>> I'm assuming this config is based on 1 AMD CPU.
>>>
>>> Pete
>>>
>>> --
>>> ----------------------------------------------------------------------
>>> Peter Gronbech
>>>
>>> Head of IT
>>> Department of Physics,
>>> University of Oxford,
>>> Denys Wilkinson Building, Tel No. : 01865 272233
>>> Keble Road, Oxford OX1 3RH, UK E-mail :
>>> [log in to unmask]<mailto:[log in to unmask]>
>>> ----------------------------------------------------------------------
>>>
>>>
>>> From: Testbed Support for GridPP member institutes
>>> <[log in to unmask]> On Behalf Of Alexander Dibbo - UKRI STFC
>>> Sent: 22 November 2019 14:48
>>> To: [log in to unmask]
>>> Subject: Re: perfSONAR specification
>>>
>>> Hi All,
>>>
>>> I don't believe the PCI-E 4 versions of these nics are generally
>>> available yet. The last time I spoke to Mellanox they we looking at Q1
>>> 2020.
>>>
>>> You would need 2 of the PCI-E 3 nics to do 2 100Gb links at full
>>> bandwidth.
>>>
>>> Regards
>>>
>>> Alexander Dibbo - Cloud Architect
>>> For STFC Cloud Documentation visit
>>> https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fstfc-cloud-docs.readthedocs.io&data=02%7C01%7Cm.doidge%40lancaster.ac.uk%7C10140b14efac4f254bd308d7796c9453%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C1%7C637111379578124485&sdata=LbEEIPdRPk32%2FIYsC9MFBP%2F7Z4SsBSxElwxYd0IMUBo%3D&reserved=0<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fstfc-cloud-docs.readthedocs.io%2F&data=02%7C01%7Cm.doidge%40lancaster.ac.uk%7C10140b14efac4f254bd308d7796c9453%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C1%7C637111379578124485&sdata=AEpkCpdx7%2BjJh387DInOeXo8ZH1ME7JJTzV%2FDSLMDJ8%3D&reserved=0>
>>>
>>> To raise a support ticket with the cloud team please email
>>> [log in to unmask]<mailto:[log in to unmask]>
>>> To receive notifications about the service please subscribe to our
>>> mailing list at:
>>> https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.jiscmail.ac.uk%2Fcgi-bin%2Fwebadmin%3FA0%3DSTFC-CLOUD&data=02%7C01%7Cm.doidge%40lancaster.ac.uk%7C10140b14efac4f254bd308d7796c9453%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C1%7C637111379578124485&sdata=vveW%2FcOXKw0TouO%2BauiZQgcvsaqxlwT4zpgkJt%2BCFco%3D&reserved=0
>>> To receive fast notifications or to discuss usage of the cloud please
>>> join our Slack: https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fstfc-cloud.slack.com%2F&data=02%7C01%7Cm.doidge%40lancaster.ac.uk%7C10140b14efac4f254bd308d7796c9453%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C1%7C637111379578124485&sdata=B%2F8IpTPzHRLCVJ%2FYNhgu7AKD7E146AnvO%2FrL5T2zgXw%3D&reserved=0
>>>
>>> From: Testbed Support for GridPP member institutes
>>> <[log in to unmask]<mailto:[log in to unmask]>> On
>>> Behalf Of George, Simon
>>> Sent: 22 November 2019 14:42
>>> To: [log in to unmask]<mailto:[log in to unmask]>
>>> Subject: Re: perfSONAR specification
>>>
>>> Thanks for this Duncan.
>>> I can see the advantage of focusing on one vendor as an example but we
>>> will presumably need to get quotes from other vendors so satisfy our
>>> purchasing rules.
>>> So once you have settled on the spec, please could you list in a
>>> vendor-neutral way the required features of the server model you have
>>> chosen (e.g. PCIe slots & bandwidth, CPU features)?
>>> Thanks,
>>> Simon
>>>
>>> ________________________________
>>> From: Testbed Support for GridPP member institutes
>>> <[log in to unmask]<mailto:[log in to unmask]>> on
>>> behalf of Daniel Traynor
>>> <[log in to unmask]<mailto:[log in to unmask]>>
>>> Sent: 22 November 2019 13:24
>>> To: [log in to unmask]<mailto:[log in to unmask]>
>>> <[log in to unmask]<mailto:[log in to unmask]>>
>>> Subject: Re: perfSONAR specification
>>>
>>> AMD EPYC have quad memory channels so 4 memory DIMMS would have higher
>>> memory bandwidth than 2 DIMMS (is it important for perfsonar?)
>>>
>>> Is the 100G network card PCI-3 x16 ? this only has a bandwidth of
>>> 128Gb/s, is this enough to drive 2 100G links? I guess the latency
>>> link won't need the full bandwidth to run tests.
>>>
>>> Don't forget to add the price of the cables! Might be able to get away
>>> with 3/5 M direct attached cables. 100G cables not that cheep.
>>>
>>> dan
>>>
>>> * Dr Daniel Traynor, Grid cluster system manager
>>> * Tel +44(0)20 7882 6560, Particle Physics,QMUL
>>>
>>>
>>> ________________________________________
>>> From: Testbed Support for GridPP member institutes
>>> <[log in to unmask]<mailto:[log in to unmask]>> on
>>> behalf of Duncan Rand
>>> <[log in to unmask]<mailto:[log in to unmask]>>
>>> Sent: 22 November 2019 12:31
>>> To: [log in to unmask]<mailto:[log in to unmask]>
>>> Subject: perfSONAR specification
>>>
>>> Hi
>>>
>>> Just to remind everyone we are looking for GridPP sites to refresh their
>>> perfSONAR hosts. There is a requirement for the hosts to be the same
>>> specification at each site and capable of 100G at sites that either are
>>> already connected at 100G (such as RAL and Imperial) or likely soon to
>>> be upgraded. Rather than two hosts we propose only one host now with
>>> three interfaces, one for management, one for latency and loss and one
>>> for throughput (either 10G or 100G).
>>>
>>> Following on from Martin's suggestion to try AMD, it seems that Dell
>>> R6515 which is 1U
>>>
>>> https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.dell.com%2Fen-us%2Fwork%2Fshop%2Fpovw%2Fpoweredge-r6515&data=02%7C01%7Cm.doidge%40lancaster.ac.uk%7C10140b14efac4f254bd308d7796c9453%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C1%7C637111379578124485&sdata=Ghf%2FR4vMlNQQx0cCkMZCBA6q2KOeIg5Ly58ABDtkRlg%3D&reserved=0
>>>
>>>
>>> or R7515 which is 2U would seem good value. These come with 8 core AMD
>>> EPYC 3.2GHz and 3200 MT/s memory. Furthermore they both have one or more
>>> PCIe Gen4 slots. So I would like to propose the following initial
>>> possible draft specification:
>>>
>>> PowerEdge R6515
>>> Chassis with up to 4x3.5" Drives
>>> PowerEdge R6515/R7515 Motherboard, with 2 x 1Gb Onboard LOM (BCM5720)
>>> AMD EPYC 7262 3.20GHz, 8C/16T, 128M Cache (155W) DDR4-3200
>>> (2) 16GB RDIMM, 3200MT/s, Dual Rank
>>> (2) 240GB SSD SATA Mixed Use 6Gbps 512e 2.5in Hot plug, 3.5in HYB CARR
>>> S4610
>>> Dual, Hot Plug, Redundant Power Supply (1+1), 550W
>>> iDRAC9, Enterprise X5
>>> Mellanox ConnectX-5 Dual Port 10/25GbE SFP28 Adapter, PCIe Low Profile
>>>
>>> and then those sites needing 40G/100G could buy this in addition:
>>>
>>> Mellanox ConnectX-5 EX Dual Port 40/100GbE QSFP28 Adapter, PCIe Low
>>> Profile;
>>>
>>> The Dell website indicates the cost would be about £4000 for the basic
>>> version and about £5400 with the extra 100G card.
>>>
>>> Thoughts?
>>>
>>> Duncan
>>>
>>> ########################################################################
>>>
>>> To unsubscribe from the TB-SUPPORT list, click the following link:
>>> https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.jiscmail.ac.uk%2Fcgi-bin%2Fwebadmin%3FSUBED1%3DTB-SUPPORT%26A%3D1&data=02%7C01%7Cm.doidge%40lancaster.ac.uk%7C10140b14efac4f254bd308d7796c9453%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C1%7C637111379578134479&sdata=pYfySRha8QYaZv%2F3efAxw49wNHd1gqx882FiKZfxOX0%3D&reserved=0
>>>
>>>
>>> ########################################################################
>>>
>>> To unsubscribe from the TB-SUPPORT list, click the following link:
>>> https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.jiscmail.ac.uk%2Fcgi-bin%2Fwebadmin%3FSUBED1%3DTB-SUPPORT%26A%3D1&data=02%7C01%7Cm.doidge%40lancaster.ac.uk%7C10140b14efac4f254bd308d7796c9453%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C1%7C637111379578134479&sdata=pYfySRha8QYaZv%2F3efAxw49wNHd1gqx882FiKZfxOX0%3D&reserved=0
>>>
>>> ________________________________
>>>
>>> To unsubscribe from the TB-SUPPORT list, click the following link:
>>> https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.jiscmail.ac.uk%2Fcgi-bin%2Fwebadmin%3FSUBED1%3DTB-SUPPORT%26A%3D1&data=02%7C01%7Cm.doidge%40lancaster.ac.uk%7C10140b14efac4f254bd308d7796c9453%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C1%7C637111379578134479&sdata=pYfySRha8QYaZv%2F3efAxw49wNHd1gqx882FiKZfxOX0%3D&reserved=0
>>>
>>> ________________________________
>>>
>>> To unsubscribe from the TB-SUPPORT list, click the following link:
>>> https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.jiscmail.ac.uk%2Fcgi-bin%2Fwebadmin%3FSUBED1%3DTB-SUPPORT%26A%3D1&data=02%7C01%7Cm.doidge%40lancaster.ac.uk%7C10140b14efac4f254bd308d7796c9453%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C1%7C637111379578134479&sdata=pYfySRha8QYaZv%2F3efAxw49wNHd1gqx882FiKZfxOX0%3D&reserved=0
>>>
>>> ########################################################################
>>>
>>> To unsubscribe from the TB-SUPPORT list, click the following link:
>>> https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.jiscmail.ac.uk%2Fcgi-bin%2Fwebadmin%3FSUBED1%3DTB-SUPPORT%26A%3D1&data=02%7C01%7Cm.doidge%40lancaster.ac.uk%7C10140b14efac4f254bd308d7796c9453%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C1%7C637111379578134479&sdata=pYfySRha8QYaZv%2F3efAxw49wNHd1gqx882FiKZfxOX0%3D&reserved=0
>>>
>>
>> ########################################################################
>>
>> To unsubscribe from the TB-SUPPORT list, click the following link:
>> https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.jiscmail.ac.uk%2Fcgi-bin%2Fwebadmin%3FSUBED1%3DTB-SUPPORT%26A%3D1&data=02%7C01%7Cm.doidge%40lancaster.ac.uk%7C10140b14efac4f254bd308d7796c9453%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C1%7C637111379578134479&sdata=pYfySRha8QYaZv%2F3efAxw49wNHd1gqx882FiKZfxOX0%3D&reserved=0
>
> ########################################################################
>
> To unsubscribe from the TB-SUPPORT list, click the following link:
> https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.jiscmail.ac.uk%2Fcgi-bin%2Fwebadmin%3FSUBED1%3DTB-SUPPORT%26A%3D1&data=02%7C01%7Cm.doidge%40lancaster.ac.uk%7C10140b14efac4f254bd308d7796c9453%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C1%7C637111379578134479&sdata=pYfySRha8QYaZv%2F3efAxw49wNHd1gqx882FiKZfxOX0%3D&reserved=0
########################################################################
To unsubscribe from the TB-SUPPORT list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=TB-SUPPORT&A=1
|