 |
 |
Now I have quotes back, my vendors are saying that the Mellanox 10/25GbE SFP28 Adapter is only available in gen 3 PCIe. (It seems currently only the 100/200Gb NICs are gen 4.) Is this correct and I should settle for a Gen 3 card? If so doesn't this somewhat undermine the CPU requirement? Thanks, Simon ________________________________ From: Testbed Support for GridPP member institutes <[log in to unmask]> on behalf of Duncan Rand <[log in to unmask]> Sent: 05 December 2019 13:58 To: [log in to unmask] <[log in to unmask]> Subject: [EXT] Re: perfSONAR specification Hi Matt It's the next generation of technology, the AMD EPYC has the faster PCIe Gen4 slot(s) whereas the Intel CPUs I looked at have PCIe Gen3. The AMD CPUs can also use faster 3200 MT/s memory, whereas the Intels I looked at are limited to DIMM speeds up to 2666 MT/s. Note, that only some of the Mellanox ConnectX-5 cards use PCIe Gen4, see page 4 here https://www.mellanox.com/related-docs/prod_adapter_cards/PB_ConnectX-5_EN_Card.pdf Duncan On 05/12/2019 11:00, Matt Doidge wrote: > Thanks Duncan, > Our vendor came back to us asking why we were specifying AMD over Intel > - and I embarrassingly couldn't remember the technical reasoning > (something about AMD CPUs being able to better drive those big network > cards?). My response was "a wise man told us so" - which whilst correct > isn't all that useful! > > If you (or anyone with better recall then I) could refresh my memory on > this that would be great. I'm not a fan of doing things by rote. > > Cheers, > Matt > > On 05/12/2019 10:18, Duncan Rand wrote: >> This email originated from outside of the University. Do not click >> links or open attachments unless you recognise the sender and know the >> content is safe. >> >> Hi >> >> So the vendor-neutral specification that includes 4 memory DIMMs would be >> >> AMD EPYC 7262 3.20GHz, 8C/16T, 128M Cache (155W) DDR4-3200 >> (4) 8GB RDIMM, 3200MT/s, Single Rank >> (2) 240GB SSD SATA >> Dual, Hot Plug, Redundant Power Supply (1+1) >> iDRAC9, Enterprise >> Mellanox ConnectX-5 Dual Port 10/25GbE SFP28 Adapter, PCIe Low Profile >> >> and then those sites needing 40G/100G would buy this in addition: >> >> Mellanox ConnectX-5 EX Dual Port 40/100GbE QSFP28 Adapter, PCIe Low >> Profile >> >> As I understand it all sites should buy their own apart from QMUL, >> Glasgow and Bristol. >> >> Duncan >> >> On 25/11/2019 14:18, Duncan Rand wrote: >>> Thanks Pete. Is that the latest version of that document? It doesn't >>> mention the 7262. >>> >>> Duncan >>> >>> On 22/11/2019 15:10, Pete Gronbech wrote: >>>> Please see : >>>> https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdeveloper.amd.com%2Fwp-content%2Fresources%2F56301_1.0.pdf&data=02%7C01%7Cm.doidge%40lancaster.ac.uk%7C10140b14efac4f254bd308d7796c9453%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C1%7C637111379578124485&sdata=KLdLVCoVfL4KUclAByiUVjPNXC%2BQ7iAkbFX2YJTSqSs%3D&reserved=0 >>>> for >>>> details of the memory channels. >>>> Performance is reduced if you specify less than the ideal of 4 DIMMS >>>> per CPU. >>>> >>>> I'm assuming this config is based on 1 AMD CPU. >>>> >>>> Pete >>>> >>>> -- >>>> ---------------------------------------------------------------------- >>>> Peter Gronbech >>>> >>>> Head of IT >>>> Department of Physics, >>>> University of Oxford, >>>> Denys Wilkinson Building, Tel No. : 01865 272233 >>>> Keble Road, Oxford OX1 3RH, UK E-mail : >>>> [log in to unmask]<mailto:[log in to unmask]> >>>> ---------------------------------------------------------------------- >>>> >>>> >>>> From: Testbed Support for GridPP member institutes >>>> <[log in to unmask]> On Behalf Of Alexander Dibbo - UKRI STFC >>>> Sent: 22 November 2019 14:48 >>>> To: [log in to unmask] >>>> Subject: Re: perfSONAR specification >>>> >>>> Hi All, >>>> >>>> I don't believe the PCI-E 4 versions of these nics are generally >>>> available yet. The last time I spoke to Mellanox they we looking at Q1 >>>> 2020. >>>> >>>> You would need 2 of the PCI-E 3 nics to do 2 100Gb links at full >>>> bandwidth. >>>> >>>> Regards >>>> >>>> Alexander Dibbo - Cloud Architect >>>> For STFC Cloud Documentation visit >>>> https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fstfc-cloud-docs.readthedocs.io&data=02%7C01%7Cm.doidge%40lancaster.ac.uk%7C10140b14efac4f254bd308d7796c9453%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C1%7C637111379578124485&sdata=LbEEIPdRPk32%2FIYsC9MFBP%2F7Z4SsBSxElwxYd0IMUBo%3D&reserved=0<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fstfc-cloud-docs.readthedocs.io%2F&data=02%7C01%7Cm.doidge%40lancaster.ac.uk%7C10140b14efac4f254bd308d7796c9453%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C1%7C637111379578124485&sdata=AEpkCpdx7%2BjJh387DInOeXo8ZH1ME7JJTzV%2FDSLMDJ8%3D&reserved=0> >>>> >>>> >>>> To raise a support ticket with the cloud team please email >>>> [log in to unmask]<mailto:[log in to unmask]> >>>> To receive notifications about the service please subscribe to our >>>> mailing list at: >>>> https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.jiscmail.ac.uk%2Fcgi-bin%2Fwebadmin%3FA0%3DSTFC-CLOUD&data=02%7C01%7Cm.doidge%40lancaster.ac.uk%7C10140b14efac4f254bd308d7796c9453%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C1%7C637111379578124485&sdata=vveW%2FcOXKw0TouO%2BauiZQgcvsaqxlwT4zpgkJt%2BCFco%3D&reserved=0 >>>> >>>> To receive fast notifications or to discuss usage of the cloud please >>>> join our Slack: >>>> https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fstfc-cloud.slack.com%2F&data=02%7C01%7Cm.doidge%40lancaster.ac.uk%7C10140b14efac4f254bd308d7796c9453%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C1%7C637111379578124485&sdata=B%2F8IpTPzHRLCVJ%2FYNhgu7AKD7E146AnvO%2FrL5T2zgXw%3D&reserved=0 >>>> >>>> >>>> From: Testbed Support for GridPP member institutes >>>> <[log in to unmask]<mailto:[log in to unmask]>> On >>>> Behalf Of George, Simon >>>> Sent: 22 November 2019 14:42 >>>> To: [log in to unmask]<mailto:[log in to unmask]> >>>> Subject: Re: perfSONAR specification >>>> >>>> Thanks for this Duncan. >>>> I can see the advantage of focusing on one vendor as an example but we >>>> will presumably need to get quotes from other vendors so satisfy our >>>> purchasing rules. >>>> So once you have settled on the spec, please could you list in a >>>> vendor-neutral way the required features of the server model you have >>>> chosen (e.g. PCIe slots & bandwidth, CPU features)? >>>> Thanks, >>>> Simon >>>> >>>> ________________________________ >>>> From: Testbed Support for GridPP member institutes >>>> <[log in to unmask]<mailto:[log in to unmask]>> on >>>> behalf of Daniel Traynor >>>> <[log in to unmask]<mailto:[log in to unmask]>> >>>> Sent: 22 November 2019 13:24 >>>> To: [log in to unmask]<mailto:[log in to unmask]> >>>> <[log in to unmask]<mailto:[log in to unmask]>> >>>> Subject: Re: perfSONAR specification >>>> >>>> AMD EPYC have quad memory channels so 4 memory DIMMS would have higher >>>> memory bandwidth than 2 DIMMS (is it important for perfsonar?) >>>> >>>> Is the 100G network card PCI-3 x16 ? this only has a bandwidth of >>>> 128Gb/s, is this enough to drive 2 100G links? I guess the latency >>>> link won't need the full bandwidth to run tests. >>>> >>>> Don't forget to add the price of the cables! Might be able to get away >>>> with 3/5 M direct attached cables. 100G cables not that cheep. >>>> >>>> dan >>>> >>>> * Dr Daniel Traynor, Grid cluster system manager >>>> * Tel +44(0)20 7882 6560, Particle Physics,QMUL >>>> >>>> >>>> ________________________________________ >>>> From: Testbed Support for GridPP member institutes >>>> <[log in to unmask]<mailto:[log in to unmask]>> on >>>> behalf of Duncan Rand >>>> <[log in to unmask]<mailto:[log in to unmask]>> >>>> Sent: 22 November 2019 12:31 >>>> To: [log in to unmask]<mailto:[log in to unmask]> >>>> Subject: perfSONAR specification >>>> >>>> Hi >>>> >>>> Just to remind everyone we are looking for GridPP sites to refresh >>>> their >>>> perfSONAR hosts. There is a requirement for the hosts to be the same >>>> specification at each site and capable of 100G at sites that either are >>>> already connected at 100G (such as RAL and Imperial) or likely soon to >>>> be upgraded. Rather than two hosts we propose only one host now with >>>> three interfaces, one for management, one for latency and loss and one >>>> for throughput (either 10G or 100G). >>>> >>>> Following on from Martin's suggestion to try AMD, it seems that Dell >>>> R6515 which is 1U >>>> >>>> https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.dell.com%2Fen-us%2Fwork%2Fshop%2Fpovw%2Fpoweredge-r6515&data=02%7C01%7Cm.doidge%40lancaster.ac.uk%7C10140b14efac4f254bd308d7796c9453%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C1%7C637111379578124485&sdata=Ghf%2FR4vMlNQQx0cCkMZCBA6q2KOeIg5Ly58ABDtkRlg%3D&reserved=0 >>>> >>>> >>>> >>>> or R7515 which is 2U would seem good value. These come with 8 core AMD >>>> EPYC 3.2GHz and 3200 MT/s memory. Furthermore they both have one or >>>> more >>>> PCIe Gen4 slots. So I would like to propose the following initial >>>> possible draft specification: >>>> >>>> PowerEdge R6515 >>>> Chassis with up to 4x3.5" Drives >>>> PowerEdge R6515/R7515 Motherboard, with 2 x 1Gb Onboard LOM (BCM5720) >>>> AMD EPYC 7262 3.20GHz, 8C/16T, 128M Cache (155W) DDR4-3200 >>>> (2) 16GB RDIMM, 3200MT/s, Dual Rank >>>> (2) 240GB SSD SATA Mixed Use 6Gbps 512e 2.5in Hot plug, 3.5in HYB CARR >>>> S4610 >>>> Dual, Hot Plug, Redundant Power Supply (1+1), 550W >>>> iDRAC9, Enterprise X5 >>>> Mellanox ConnectX-5 Dual Port 10/25GbE SFP28 Adapter, PCIe Low Profile >>>> >>>> and then those sites needing 40G/100G could buy this in addition: >>>> >>>> Mellanox ConnectX-5 EX Dual Port 40/100GbE QSFP28 Adapter, PCIe Low >>>> Profile; >>>> >>>> The Dell website indicates the cost would be about £4000 for the basic >>>> version and about £5400 with the extra 100G card. >>>> >>>> Thoughts? >>>> >>>> Duncan >>>> >>>> ######################################################################## >>>> >>>> >>>> To unsubscribe from the TB-SUPPORT list, click the following link: >>>> https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.jiscmail.ac.uk%2Fcgi-bin%2Fwebadmin%3FSUBED1%3DTB-SUPPORT%26A%3D1&data=02%7C01%7Cm.doidge%40lancaster.ac.uk%7C10140b14efac4f254bd308d7796c9453%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C1%7C637111379578134479&sdata=pYfySRha8QYaZv%2F3efAxw49wNHd1gqx882FiKZfxOX0%3D&reserved=0 >>>> >>>> >>>> >>>> ######################################################################## >>>> >>>> >>>> To unsubscribe from the TB-SUPPORT list, click the following link: >>>> https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.jiscmail.ac.uk%2Fcgi-bin%2Fwebadmin%3FSUBED1%3DTB-SUPPORT%26A%3D1&data=02%7C01%7Cm.doidge%40lancaster.ac.uk%7C10140b14efac4f254bd308d7796c9453%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C1%7C637111379578134479&sdata=pYfySRha8QYaZv%2F3efAxw49wNHd1gqx882FiKZfxOX0%3D&reserved=0 >>>> >>>> >>>> ________________________________ >>>> >>>> To unsubscribe from the TB-SUPPORT list, click the following link: >>>> https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.jiscmail.ac.uk%2Fcgi-bin%2Fwebadmin%3FSUBED1%3DTB-SUPPORT%26A%3D1&data=02%7C01%7Cm.doidge%40lancaster.ac.uk%7C10140b14efac4f254bd308d7796c9453%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C1%7C637111379578134479&sdata=pYfySRha8QYaZv%2F3efAxw49wNHd1gqx882FiKZfxOX0%3D&reserved=0 >>>> >>>> >>>> ________________________________ >>>> >>>> To unsubscribe from the TB-SUPPORT list, click the following link: >>>> https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.jiscmail.ac.uk%2Fcgi-bin%2Fwebadmin%3FSUBED1%3DTB-SUPPORT%26A%3D1&data=02%7C01%7Cm.doidge%40lancaster.ac.uk%7C10140b14efac4f254bd308d7796c9453%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C1%7C637111379578134479&sdata=pYfySRha8QYaZv%2F3efAxw49wNHd1gqx882FiKZfxOX0%3D&reserved=0 >>>> >>>> >>>> ######################################################################## >>>> >>>> >>>> To unsubscribe from the TB-SUPPORT list, click the following link: >>>> https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.jiscmail.ac.uk%2Fcgi-bin%2Fwebadmin%3FSUBED1%3DTB-SUPPORT%26A%3D1&data=02%7C01%7Cm.doidge%40lancaster.ac.uk%7C10140b14efac4f254bd308d7796c9453%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C1%7C637111379578134479&sdata=pYfySRha8QYaZv%2F3efAxw49wNHd1gqx882FiKZfxOX0%3D&reserved=0 >>>> >>>> >>> >>> ######################################################################## >>> >>> To unsubscribe from the TB-SUPPORT list, click the following link: >>> https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.jiscmail.ac.uk%2Fcgi-bin%2Fwebadmin%3FSUBED1%3DTB-SUPPORT%26A%3D1&data=02%7C01%7Cm.doidge%40lancaster.ac.uk%7C10140b14efac4f254bd308d7796c9453%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C1%7C637111379578134479&sdata=pYfySRha8QYaZv%2F3efAxw49wNHd1gqx882FiKZfxOX0%3D&reserved=0 >>> >> >> ######################################################################## >> >> To unsubscribe from the TB-SUPPORT list, click the following link: >> https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.jiscmail.ac.uk%2Fcgi-bin%2Fwebadmin%3FSUBED1%3DTB-SUPPORT%26A%3D1&data=02%7C01%7Cm.doidge%40lancaster.ac.uk%7C10140b14efac4f254bd308d7796c9453%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C1%7C637111379578134479&sdata=pYfySRha8QYaZv%2F3efAxw49wNHd1gqx882FiKZfxOX0%3D&reserved=0 >> > > ######################################################################## > > To unsubscribe from the TB-SUPPORT list, click the following link: > https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=TB-SUPPORT&A=1 ######################################################################## To unsubscribe from the TB-SUPPORT list, click the following link: https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=TB-SUPPORT&A=1 ######################################################################## To unsubscribe from the TB-SUPPORT list, click the following link: https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=TB-SUPPORT&A=1