Hi Elena
Yes, just the one.
Duncan
On 05/12/2019 17:00, Elena Korolkova wrote:
> Hi Duncan,
>
> to confirm:
> should it be a single processor in the server ?
>
> Thanks
> Elena
>
> On 5 Dec 2019, at 13:58, Duncan Rand <[log in to unmask]> wrote:
>
>> Hi Matt
>>
>> It's the next generation of technology, the AMD EPYC has the faster PCIe Gen4 slot(s) whereas the Intel CPUs I looked at have PCIe Gen3. The AMD CPUs can also use faster 3200 MT/s memory, whereas the Intels I looked at are limited to DIMM speeds up to 2666 MT/s.
>>
>> Note, that only some of the Mellanox ConnectX-5 cards use PCIe Gen4, see page 4 here
>>
>> https://www.mellanox.com/related-docs/prod_adapter_cards/PB_ConnectX-5_EN_Card.pdf
>>
>> Duncan
>>
>> On 05/12/2019 11:00, Matt Doidge wrote:
>>> Thanks Duncan,
>>> Our vendor came back to us asking why we were specifying AMD over Intel - and I embarrassingly couldn't remember the technical reasoning (something about AMD CPUs being able to better drive those big network cards?). My response was "a wise man told us so" - which whilst correct isn't all that useful!
>>> If you (or anyone with better recall then I) could refresh my memory on this that would be great. I'm not a fan of doing things by rote.
>>> Cheers,
>>> Matt
>>> On 05/12/2019 10:18, Duncan Rand wrote:
>>>> This email originated from outside of the University. Do not click links or open attachments unless you recognise the sender and know the content is safe.
>>>>
>>>> Hi
>>>>
>>>> So the vendor-neutral specification that includes 4 memory DIMMs would be
>>>>
>>>> AMD EPYC 7262 3.20GHz, 8C/16T, 128M Cache (155W) DDR4-3200
>>>> (4) 8GB RDIMM, 3200MT/s, Single Rank
>>>> (2) 240GB SSD SATA
>>>> Dual, Hot Plug, Redundant Power Supply (1+1)
>>>> iDRAC9, Enterprise
>>>> Mellanox ConnectX-5 Dual Port 10/25GbE SFP28 Adapter, PCIe Low Profile
>>>>
>>>> and then those sites needing 40G/100G would buy this in addition:
>>>>
>>>> Mellanox ConnectX-5 EX Dual Port 40/100GbE QSFP28 Adapter, PCIe Low Profile
>>>>
>>>> As I understand it all sites should buy their own apart from QMUL,
>>>> Glasgow and Bristol.
>>>>
>>>> Duncan
>>>>
>>>> On 25/11/2019 14:18, Duncan Rand wrote:
>>>>> Thanks Pete. Is that the latest version of that document? It doesn't
>>>>> mention the 7262.
>>>>>
>>>>> Duncan
>>>>>
>>>>> On 22/11/2019 15:10, Pete Gronbech wrote:
>>>>>> Please see :
>>>>>> https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdeveloper.amd.com%2Fwp-content%2Fresources%2F56301_1.0.pdf&data=02%7C01%7Cm.doidge%40lancaster.ac.uk%7C10140b14efac4f254bd308d7796c9453%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C1%7C637111379578124485&sdata=KLdLVCoVfL4KUclAByiUVjPNXC%2BQ7iAkbFX2YJTSqSs%3D&reserved=0 for
>>>>>> details of the memory channels.
>>>>>> Performance is reduced if you specify less than the ideal of 4 DIMMS
>>>>>> per CPU.
>>>>>>
>>>>>> I'm assuming this config is based on 1 AMD CPU.
>>>>>>
>>>>>> Pete
>>>>>>
>>>>>> --
>>>>>> ----------------------------------------------------------------------
>>>>>> Peter Gronbech
>>>>>>
>>>>>> Head of IT
>>>>>> Department of Physics,
>>>>>> University of Oxford,
>>>>>> Denys Wilkinson Building, Tel No. : 01865 272233
>>>>>> Keble Road, Oxford OX1 3RH, UK E-mail :
>>>>>> [log in to unmask]<mailto:[log in to unmask]>
>>>>>> ----------------------------------------------------------------------
>>>>>>
>>>>>>
>>>>>> From: Testbed Support for GridPP member institutes
>>>>>> <[log in to unmask]> On Behalf Of Alexander Dibbo - UKRI STFC
>>>>>> Sent: 22 November 2019 14:48
>>>>>> To: [log in to unmask]
>>>>>> Subject: Re: perfSONAR specification
>>>>>>
>>>>>> Hi All,
>>>>>>
>>>>>> I don't believe the PCI-E 4 versions of these nics are generally
>>>>>> available yet. The last time I spoke to Mellanox they we looking at Q1
>>>>>> 2020.
>>>>>>
>>>>>> You would need 2 of the PCI-E 3 nics to do 2 100Gb links at full
>>>>>> bandwidth.
>>>>>>
>>>>>> Regards
>>>>>>
>>>>>> Alexander Dibbo - Cloud Architect
>>>>>> For STFC Cloud Documentation visit
>>>>>> https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fstfc-cloud-docs.readthedocs.io&data=02%7C01%7Cm.doidge%40lancaster.ac.uk%7C10140b14efac4f254bd308d7796c9453%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C1%7C637111379578124485&sdata=LbEEIPdRPk32%2FIYsC9MFBP%2F7Z4SsBSxElwxYd0IMUBo%3D&reserved=0<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fstfc-cloud-docs.readthedocs.io%2F&data=02%7C01%7Cm.doidge%40lancaster.ac.uk%7C10140b14efac4f254bd308d7796c9453%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C1%7C637111379578124485&sdata=AEpkCpdx7%2BjJh387DInOeXo8ZH1ME7JJTzV%2FDSLMDJ8%3D&reserved=0>
>>>>>>
>>>>>> To raise a support ticket with the cloud team please email
>>>>>> [log in to unmask]<mailto:[log in to unmask]>
>>>>>> To receive notifications about the service please subscribe to our
>>>>>> mailing list at:
>>>>>> https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.jiscmail.ac.uk%2Fcgi-bin%2Fwebadmin%3FA0%3DSTFC-CLOUD&data=02%7C01%7Cm.doidge%40lancaster.ac.uk%7C10140b14efac4f254bd308d7796c9453%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C1%7C637111379578124485&sdata=vveW%2FcOXKw0TouO%2BauiZQgcvsaqxlwT4zpgkJt%2BCFco%3D&reserved=0
>>>>>> To receive fast notifications or to discuss usage of the cloud please
>>>>>> join our Slack: https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fstfc-cloud.slack.com%2F&data=02%7C01%7Cm.doidge%40lancaster.ac.uk%7C10140b14efac4f254bd308d7796c9453%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C1%7C637111379578124485&sdata=B%2F8IpTPzHRLCVJ%2FYNhgu7AKD7E146AnvO%2FrL5T2zgXw%3D&reserved=0
>>>>>>
>>>>>> From: Testbed Support for GridPP member institutes
>>>>>> <[log in to unmask]<mailto:[log in to unmask]>> On
>>>>>> Behalf Of George, Simon
>>>>>> Sent: 22 November 2019 14:42
>>>>>> To: [log in to unmask]<mailto:[log in to unmask]>
>>>>>> Subject: Re: perfSONAR specification
>>>>>>
>>>>>> Thanks for this Duncan.
>>>>>> I can see the advantage of focusing on one vendor as an example but we
>>>>>> will presumably need to get quotes from other vendors so satisfy our
>>>>>> purchasing rules.
>>>>>> So once you have settled on the spec, please could you list in a
>>>>>> vendor-neutral way the required features of the server model you have
>>>>>> chosen (e.g. PCIe slots & bandwidth, CPU features)?
>>>>>> Thanks,
>>>>>> Simon
>>>>>>
>>>>>> ________________________________
>>>>>> From: Testbed Support for GridPP member institutes
>>>>>> <[log in to unmask]<mailto:[log in to unmask]>> on
>>>>>> behalf of Daniel Traynor
>>>>>> <[log in to unmask]<mailto:[log in to unmask]>>
>>>>>> Sent: 22 November 2019 13:24
>>>>>> To: [log in to unmask]<mailto:[log in to unmask]>
>>>>>> <[log in to unmask]<mailto:[log in to unmask]>>
>>>>>> Subject: Re: perfSONAR specification
>>>>>>
>>>>>> AMD EPYC have quad memory channels so 4 memory DIMMS would have higher
>>>>>> memory bandwidth than 2 DIMMS (is it important for perfsonar?)
>>>>>>
>>>>>> Is the 100G network card PCI-3 x16 ? this only has a bandwidth of
>>>>>> 128Gb/s, is this enough to drive 2 100G links? I guess the latency
>>>>>> link won't need the full bandwidth to run tests.
>>>>>>
>>>>>> Don't forget to add the price of the cables! Might be able to get away
>>>>>> with 3/5 M direct attached cables. 100G cables not that cheep.
>>>>>>
>>>>>> dan
>>>>>>
>>>>>> * Dr Daniel Traynor, Grid cluster system manager
>>>>>> * Tel +44(0)20 7882 6560, Particle Physics,QMUL
>>>>>>
>>>>>>
>>>>>> ________________________________________
>>>>>> From: Testbed Support for GridPP member institutes
>>>>>> <[log in to unmask]<mailto:[log in to unmask]>> on
>>>>>> behalf of Duncan Rand
>>>>>> <[log in to unmask]<mailto:[log in to unmask]>>
>>>>>> Sent: 22 November 2019 12:31
>>>>>> To: [log in to unmask]<mailto:[log in to unmask]>
>>>>>> Subject: perfSONAR specification
>>>>>>
>>>>>> Hi
>>>>>>
>>>>>> Just to remind everyone we are looking for GridPP sites to refresh their
>>>>>> perfSONAR hosts. There is a requirement for the hosts to be the same
>>>>>> specification at each site and capable of 100G at sites that either are
>>>>>> already connected at 100G (such as RAL and Imperial) or likely soon to
>>>>>> be upgraded. Rather than two hosts we propose only one host now with
>>>>>> three interfaces, one for management, one for latency and loss and one
>>>>>> for throughput (either 10G or 100G).
>>>>>>
>>>>>> Following on from Martin's suggestion to try AMD, it seems that Dell
>>>>>> R6515 which is 1U
>>>>>>
>>>>>> https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.dell.com%2Fen-us%2Fwork%2Fshop%2Fpovw%2Fpoweredge-r6515&data=02%7C01%7Cm.doidge%40lancaster.ac.uk%7C10140b14efac4f254bd308d7796c9453%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C1%7C637111379578124485&sdata=Ghf%2FR4vMlNQQx0cCkMZCBA6q2KOeIg5Ly58ABDtkRlg%3D&reserved=0
>>>>>>
>>>>>>
>>>>>> or R7515 which is 2U would seem good value. These come with 8 core AMD
>>>>>> EPYC 3.2GHz and 3200 MT/s memory. Furthermore they both have one or more
>>>>>> PCIe Gen4 slots. So I would like to propose the following initial
>>>>>> possible draft specification:
>>>>>>
>>>>>> PowerEdge R6515
>>>>>> Chassis with up to 4x3.5" Drives
>>>>>> PowerEdge R6515/R7515 Motherboard, with 2 x 1Gb Onboard LOM (BCM5720)
>>>>>> AMD EPYC 7262 3.20GHz, 8C/16T, 128M Cache (155W) DDR4-3200
>>>>>> (2) 16GB RDIMM, 3200MT/s, Dual Rank
>>>>>> (2) 240GB SSD SATA Mixed Use 6Gbps 512e 2.5in Hot plug, 3.5in HYB CARR
>>>>>> S4610
>>>>>> Dual, Hot Plug, Redundant Power Supply (1+1), 550W
>>>>>> iDRAC9, Enterprise X5
>>>>>> Mellanox ConnectX-5 Dual Port 10/25GbE SFP28 Adapter, PCIe Low Profile
>>>>>>
>>>>>> and then those sites needing 40G/100G could buy this in addition:
>>>>>>
>>>>>> Mellanox ConnectX-5 EX Dual Port 40/100GbE QSFP28 Adapter, PCIe Low
>>>>>> Profile;
>>>>>>
>>>>>> The Dell website indicates the cost would be about £4000 for the basic
>>>>>> version and about £5400 with the extra 100G card.
>>>>>>
>>>>>> Thoughts?
>>>>>>
>>>>>> Duncan
>>>>>>
>>>>>> ########################################################################
>>>>>>
>>>>>> To unsubscribe from the TB-SUPPORT list, click the following link:
>>>>>> https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.jiscmail.ac.uk%2Fcgi-bin%2Fwebadmin%3FSUBED1%3DTB-SUPPORT%26A%3D1&data=02%7C01%7Cm.doidge%40lancaster.ac.uk%7C10140b14efac4f254bd308d7796c9453%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C1%7C637111379578134479&sdata=pYfySRha8QYaZv%2F3efAxw49wNHd1gqx882FiKZfxOX0%3D&reserved=0
>>>>>>
>>>>>>
>>>>>> ########################################################################
>>>>>>
>>>>>> To unsubscribe from the TB-SUPPORT list, click the following link:
>>>>>> https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.jiscmail.ac.uk%2Fcgi-bin%2Fwebadmin%3FSUBED1%3DTB-SUPPORT%26A%3D1&data=02%7C01%7Cm.doidge%40lancaster.ac.uk%7C10140b14efac4f254bd308d7796c9453%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C1%7C637111379578134479&sdata=pYfySRha8QYaZv%2F3efAxw49wNHd1gqx882FiKZfxOX0%3D&reserved=0
>>>>>>
>>>>>> ________________________________
>>>>>>
>>>>>> To unsubscribe from the TB-SUPPORT list, click the following link:
>>>>>> https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.jiscmail.ac.uk%2Fcgi-bin%2Fwebadmin%3FSUBED1%3DTB-SUPPORT%26A%3D1&data=02%7C01%7Cm.doidge%40lancaster.ac.uk%7C10140b14efac4f254bd308d7796c9453%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C1%7C637111379578134479&sdata=pYfySRha8QYaZv%2F3efAxw49wNHd1gqx882FiKZfxOX0%3D&reserved=0
>>>>>>
>>>>>> ________________________________
>>>>>>
>>>>>> To unsubscribe from the TB-SUPPORT list, click the following link:
>>>>>> https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.jiscmail.ac.uk%2Fcgi-bin%2Fwebadmin%3FSUBED1%3DTB-SUPPORT%26A%3D1&data=02%7C01%7Cm.doidge%40lancaster.ac.uk%7C10140b14efac4f254bd308d7796c9453%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C1%7C637111379578134479&sdata=pYfySRha8QYaZv%2F3efAxw49wNHd1gqx882FiKZfxOX0%3D&reserved=0
>>>>>>
>>>>>> ########################################################################
>>>>>>
>>>>>> To unsubscribe from the TB-SUPPORT list, click the following link:
>>>>>> https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.jiscmail.ac.uk%2Fcgi-bin%2Fwebadmin%3FSUBED1%3DTB-SUPPORT%26A%3D1&data=02%7C01%7Cm.doidge%40lancaster.ac.uk%7C10140b14efac4f254bd308d7796c9453%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C1%7C637111379578134479&sdata=pYfySRha8QYaZv%2F3efAxw49wNHd1gqx882FiKZfxOX0%3D&reserved=0
>>>>>>
>>>>>
>>>>> ########################################################################
>>>>>
>>>>> To unsubscribe from the TB-SUPPORT list, click the following link:
>>>>> https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.jiscmail.ac.uk%2Fcgi-bin%2Fwebadmin%3FSUBED1%3DTB-SUPPORT%26A%3D1&data=02%7C01%7Cm.doidge%40lancaster.ac.uk%7C10140b14efac4f254bd308d7796c9453%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C1%7C637111379578134479&sdata=pYfySRha8QYaZv%2F3efAxw49wNHd1gqx882FiKZfxOX0%3D&reserved=0
>>>>
>>>> ########################################################################
>>>>
>>>> To unsubscribe from the TB-SUPPORT list, click the following link:
>>>> https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.jiscmail.ac.uk%2Fcgi-bin%2Fwebadmin%3FSUBED1%3DTB-SUPPORT%26A%3D1&data=02%7C01%7Cm.doidge%40lancaster.ac.uk%7C10140b14efac4f254bd308d7796c9453%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C1%7C637111379578134479&sdata=pYfySRha8QYaZv%2F3efAxw49wNHd1gqx882FiKZfxOX0%3D&reserved=0
>>> ########################################################################
>>> To unsubscribe from the TB-SUPPORT list, click the following link:
>>> https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=TB-SUPPORT&A=1
>>
>> ########################################################################
>>
>> To unsubscribe from the TB-SUPPORT list, click the following link:
>> https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=TB-SUPPORT&A=1
>
> ########################################################################
>
> To unsubscribe from the TB-SUPPORT list, click the following link:
> https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=TB-SUPPORT&A=1
>
########################################################################
To unsubscribe from the TB-SUPPORT list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=TB-SUPPORT&A=1
|