 |
 |
Hiya, Spoke to the CERN network guys this morning. It appears that Geant had a BGP route update issue. It was fixed around 1 AM this morning. Traceroute Below traceroute to svr001.gla.scotgrid.ac.uk (130.209.239.101), 64 hops max, 52 byte packets 1 r40-s2-rhpzm-1-da1 (128.141.224.1) 2.429 ms 2.450 ms 2.494 ms 2 r40-b-rhpzl-1-gb50 (194.12.130.49) 2.472 ms 2.531 ms 2.541 ms 3 r513-b-rbrml-2-zb53 (194.12.141.21) 2.992 ms 0.900 ms 0.816 ms 4 b513-b-rbrml-2-sc2 (194.12.149.5) 0.852 ms 0.910 ms 0.812 ms 5 g513-a-rci65-2-sg2 (194.12.144.17) 0.996 ms 0.997 ms 0.901 ms 6 g513-a-rci65-2-fe2 (192.65.184.177) 1.065 ms 0.995 ms 0.929 ms 7 e513-a-rbrxl-2-se1 (192.65.184.169) 3.340 ms 1.183 ms 1.073 ms 8 swice3-te20 (192.65.184.69) 12.573 ms 1.248 ms 1.175 ms 9 switch-bckp.rt1.par.fr.geant.net (62.40.124.81) 11.470 ms 9.989 ms 9.901 ms 10 ae5.mx1.gen.ch.geant.net (62.40.112.184) 9.993 ms 9.920 ms 10.208 ms 11 ae1.mx1.lon.uk.geant.net (62.40.98.76) 17.434 ms 16.027 ms 16.167 ms 12 ae1-vlan110.rt1.lon.uk.geant.net (62.40.98.75) 84.282 ms 15.849 ms 17.895 ms 13 janet-gw.rt1.lon.uk.geant.net (62.40.124.198) 16.042 ms 15.981 ms 15.973 ms 14 ae13.read-sbr1.ja.net<http://ae13.read-sbr1.ja.net> (146.97.33.145) 17.450 ms 17.565 ms 17.316 ms 15 as3.warr-sbr1.ja.net<http://as3.warr-sbr1.ja.net> (146.97.33.90) 21.217 ms 20.973 ms 21.329 ms 16 ae14.glas-sbr1.ja.net<http://ae14.glas-sbr1.ja.net> (146.97.33.122) 25.958 ms 29.529 ms 26.089 ms 17 clydenet.glas-sbr1.ja.net<http://clydenet.glas-sbr1.ja.net> (146.97.40.98) 26.116 ms 26.337 ms 26.338 ms 18 glasgowuni-ge1-1-glasgowpop-ge1-2-v152.clyde.net.uk<http://glasgowuni-ge1-1-glasgowpop-ge1-2-v152.clyde.net.uk> (194.81.62.154) 26.758 ms 26.630 ms 26.482 ms 19 *^C What is of interest is that the alarms are lower level than I thought they would be. As the service "stayed up" is this an issue for us? I am really surprised that the experiments don't look at these types of topology changes but this was requested yesterday at the meeting. Should this be something we want to raise as an issue? I think we should. Also, Chris the CERN network guys said to say thank you for picking this up. regards, Mark On 3 May 2013, at 08:51, Alastair Dewhurst wrote: Hi I did a trace route from RAL last night and saw the same issue. I have checked now and it is normal. This was on a UI, I didn't check a machine on the OPN subnet. Alastair 2/5/13 # traceroute www.cern.ch<http://www.cern.ch> traceroute to www.cern.ch<http://www.cern.ch> (137.138.144.168), 30 hops max, 40 byte packets 1 gw (130.246.180.254) 0.714 ms 1.088 ms 1.477 ms 2 130.246.0.6 (130.246.0.6) 0.516 ms 0.659 ms 0.794 ms 3 192.100.78.61 (192.100.78.61) 2.332 ms 2.325 ms 2.807 ms 4 ae6.read-sbr1.ja.net<http://ae6.read-sbr1.ja.net> (146.97.41.157) 3.249 ms 3.232 ms 3.226 ms 5 ae13.lond-sbr3.ja.net<http://ae13.lond-sbr3.ja.net> (146.97.33.146) 4.898 ms 5.086 ms 4.878 ms 6 janet.rt1.lon.uk.geant.net (62.40.124.197) 4.866 ms 4.764 ms 4.647 ms 7 ae3-vlan110.mx1.lon.uk.geant.net (62.40.98.74) 4.747 ms 4.886 ms 4.973 ms 8 ae0.mx1.ams.nl.geant.net (62.40.98.81) 11.813 ms 11.448 ms 11.407 ms 9 ae0.mx1.fra.de.geant.net (62.40.98.128) 18.381 ms 17.583 ms 17.716 ms 10 ae4.rt1.fra.de.geant.net (62.40.98.135) 18.900 ms 17.412 ms 22.400 ms 11 abilene-wash-gw.rt1.fra.de.geant.net (62.40.125.18) 135.625 ms 135.617 ms 135.653 ms 12 ae-2.10.rtr.clev.net.internet2.edu (64.57.28.162) 140.278 ms 140.317 ms 140.408 ms 13 ae-7.10.rtr.chic.net.internet2.edu (64.57.28.158) 152.877 ms 156.432 ms 150.719 ms 14 e513-a-rbrxl-2-te6.cern.ch<http://e513-a-rbrxl-2-te6.cern.ch> (192.91.246.126) 248.055 ms 240.901 ms 240.828 ms 3/5/13 # traceroute www.cern.ch<http://www.cern.ch> traceroute to www.cern.ch<http://www.cern.ch> (137.138.144.168), 30 hops max, 40 byte packets 1 gw (130.246.180.254) 0.730 ms 1.532 ms 1.970 ms 2 130.246.0.6 (130.246.0.6) 0.310 ms 0.446 ms 0.488 ms 3 192.100.78.61 (192.100.78.61) 2.271 ms 2.265 ms 2.298 ms 4 ae6.read-sbr1.ja.net<http://ae6.read-sbr1.ja.net> (146.97.41.157) 24.752 ms 24.745 ms 24.736 ms 5 ae13.lond-sbr3.ja.net<http://ae13.lond-sbr3.ja.net> (146.97.33.146) 3.751 ms 18.762 ms 18.801 ms 6 janet.rt1.lon.uk.geant.net (62.40.124.197) 4.459 ms 4.265 ms 4.131 ms 7 ae3-vlan110.mx1.lon.uk.geant.net (62.40.98.74) 4.078 ms 3.525 ms 3.765 ms 8 ae0.mx1.par.fr.geant.net (62.40.98.77) 8.852 ms 8.819 ms 8.852 ms 9 ae4.rt1.par.fr.geant.net (62.40.112.185) 59.005 ms 9.371 ms 58.473 ms 10 switch-bckp-gw.rt1.par.fr.geant.net (62.40.124.82) 17.973 ms 17.966 ms 18.052 ms 11 e513-a-rbrxl-2-te20.cern.ch<http://e513-a-rbrxl-2-te20.cern.ch> (192.65.184.70) 21.149 ms 20.655 ms 17.651 ms On 2 May 2013, at 23:19, Christopher J. Walker wrote: On 02/05/13 21:19, Mark Mitchell wrote: Hiya, I think Chris wins the network vigilance award this week :-) There appears to be an issue with the Geant network. On a serious note, Chris how did you pick this up? Someone came into the office and asked was there a problem with the network... Further investigation revealed he'd been trying to videoconference to CERN and had poor performance so ended up using the phone bridge. The QMUL link to Janet was fine, and whilst pinging CERN didn't have huge packet loss, ping times of 230ms seemed surprisingly long, and traceroute revealed packets looked like they were going across the atlantic. Did any of the experiment software pick up on this? Email from Edoardo is attached below. Thanks. Chris PS Although IPv6 traffic is going to the US, IPv6 traffic isn't. [root@nm03 ~]# ping -c 2 www.ipv6.cern.ch<http://www.ipv6.cern.ch> PING cernis22-21.cern.ch<http://cernis22-21.cern.ch> (137.138.218.32) 56(84) bytes of data. 64 bytes from cernis22-21.cern.ch<http://cernis22-21.cern.ch> (137.138.218.32): icmp_seq=1 ttl=110 time=221 ms 64 bytes from cernis22-21.cern.ch<http://cernis22-21.cern.ch> (137.138.218.32): icmp_seq=1 ttl=110 time=221 ms (DUP!) 64 bytes from cernis22-21.cern.ch<http://cernis22-21.cern.ch> (137.138.218.32): icmp_seq=2 ttl=110 time=219 ms --- cernis22-21.cern.ch<http://cernis22-21.cern.ch> ping statistics --- 2 packets transmitted, 2 received, +1 duplicates, 0% packet loss, time 1000ms rtt min/avg/max/mdev = 219.561/220.903/221.577/0.948 ms [root@nm03 ~]# ping6 -c 2 www.ipv6.cern.ch<http://www.ipv6.cern.ch> PING www.ipv6.cern.ch<http://www.ipv6.cern.ch>(cernis22-21.cern.ch<http://cernis22-21.cern.ch>) 56 data bytes 64 bytes from cernis22-21.cern.ch<http://cernis22-21.cern.ch>: icmp_seq=0 ttl=50 time=28.7 ms 64 bytes from cernis22-21.cern.ch<http://cernis22-21.cern.ch>: icmp_seq=0 ttl=50 time=28.7 ms (DUP!) 64 bytes from cernis22-21.cern.ch<http://cernis22-21.cern.ch>: icmp_seq=1 ttl=50 time=16.0 ms 64 bytes from cernis22-21.cern.ch<http://cernis22-21.cern.ch>: icmp_seq=1 ttl=50 time=16.0 ms (DUP!) Traceroute6/tracepath6 don't work to that host - they stop somewhere with no name (no idea if it's inside or outside CERN). Chris regards, Mark -------- Original Message -------- Subject: few prefixes from Geant? Date: Thu, 2 May 2013 21:59:23 +0200 From: Edoardo Martelli <[log in to unmask]<mailto:[log in to unmask]> <mailto:[log in to unmask]>> To: NOC Switch <[log in to unmask]<mailto:[log in to unmask]> <mailto:[log in to unmask]>>, NOC Geant <[log in to unmask]<mailto:[log in to unmask]> <mailto:[log in to unmask]>> CC: Cern Extip <[log in to unmask]<mailto:[log in to unmask]> <mailto:[log in to unmask]>> Hello Geant and SWITCH NOC We don't get many prefixes from the peerings with SWitch and most of our traffic to the Europen NRENs is going via Internet2. It seems to me that the prefixes are not even received by Switch. Is there any issue? cheers Edoardo Router: swiCE2.switch.ch<http://swiCE2.switch.ch> <http://swiCE2.switch.ch/> Command: show ip bgp 62.40.105.162 BGP routing table entry for 62.40.96.0/19, version 23788441 Paths: (1 available, no best path) Multipath: eBGP Not advertised to any peer 3356 20965, (received-only) 213.242.73.73 from 213.242.73.73 (4.69.187.34) Origin IGP, metric 0, localpref 100, valid, external Community: 3356:2 3356:22 3356:100 3356:123 3356:501 3356:2065 20965:155 On 2 May 2013, at 20:22, John Gordon wrote: Mark the workshop is timely, I was about to suggest that this routing might be a 'benefit ' of LHCONE. We have already seen one instance of Germany assuming that everyone is on LHCONE. John Mark Mitchell <[log in to unmask]<mailto:[log in to unmask]> <mailto:[log in to unmask]>> wrote: Hi Chris, Same pattern at Glasgow I am at the LHCONE workshop so I can ask in the morning as there are multiple devices from multiple networks on both sides of the Atlantic doing WAN links and it may be a case of naming conventions? However given that on one visual trace route it appears to have been sourced in the US, I am a tad confused. I will ask the internet 2 guys tomorrow morning. regards, Mark traceroute to www.cern.ch<http://www.cern.ch> <http://www.cern.ch><http://www.cern.ch> (137.138.144.168), 30 hops max, 40 byte packets 1 130.209.239.2 (130.209.239.2) 0.452 ms 0.554 ms 0.677 ms 2 glasgowpop-ge1-2-glasgowuni-ge1-1-v152.clyde.net.uk<http://glasgowpop-ge1-2-glasgowuni-ge1-1-v152.clyde.net.uk> <http://glasgowpop-ge1-2-glasgowuni-ge1-1-v152.clyde.net.uk><http://glasgowpop-ge1-2-glasgowuni-ge1-1-v152.clyde.net.uk> (194.81.62.153) 0.399 ms 0.380 ms 0.413 ms 3 so-2-0-0.glas-sbr1.ja.net<http://so-2-0-0.glas-sbr1.ja.net> <http://so-2-0-0.glas-sbr1.ja.net><http://so-2-0-0.glas-sbr1.ja.net> (146.97.40.97) 0.510 ms 0.562 ms 0.646 ms 4 ae14.warr-sbr1.ja.net<http://ae14.warr-sbr1.ja.net> <http://ae14.warr-sbr1.ja.net><http://ae14.warr-sbr1.ja.net> (146.97.33.121) 5.207 ms 5.211 ms 5.206 ms 5 as3.read-sbr1.ja.net<http://as3.read-sbr1.ja.net> <http://as3.read-sbr1.ja.net><http://as3.read-sbr1.ja.net> (146.97.33.89) 8.975 ms 8.982 ms 8.978 ms 6 ae13.lond-sbr3.ja.net<http://ae13.lond-sbr3.ja.net> <http://ae13.lond-sbr3.ja.net><http://ae13.lond-sbr3.ja.net> (146.97.33.146) 10.336 ms 10.327 ms 10.348 ms 7 janet.rt1.lon.uk.geant.net (62.40.124.197) 66.367 ms 64.430 ms 63.623 ms 8 ae3-vlan110.mx1.lon.uk.geant.net (62.40.98.74) 10.525 ms 10.528 ms 10.525 ms 9 ae0.mx1.ams.nl.geant.net (62.40.98.81) 18.056 ms 18.061 ms 18.055 ms 10 ae0.mx1.fra.de.geant.net (62.40.98.128) 24.746 ms 24.793 ms 24.762 ms 11 ae4.rt1.fra.de.geant.net (62.40.98.135) 24.854 ms 24.790 ms 24.894 ms 12 abilene-wash-gw.rt1.fra.de.geant.net (62.40.125.18) 118.807 ms 118.923 ms 119.183 ms 13 ae-2.10.rtr.clev.net.internet2.edu (64.57.28.162) 126.910 ms 126.890 ms 126.978 ms 14 ae-7.10.rtr.chic.net.internet2.edu (64.57.28.158) 135.798 ms 137.945 ms 135.682 ms 15 e513-a-rbrxl-2-te6.cern.ch<http://e513-a-rbrxl-2-te6.cern.ch> <http://e513-a-rbrxl-2-te6.cern.ch><http://e513-a-rbrxl-2-te6.cern.ch> (192.91.246.126) 232.462 ms 232.407 ms 232.565 ms 16 * * * 17 * * * 18 * * * 19 * * * 20 * * * 21 * * On 2 May 2013, at 19:05, Christopher J. Walker wrote: Packets to (and indeed from) CERN seem to be taking the scenic route via the USA. I'm curious if this is deliberate... walker@heppc046:~$ traceroute www.cern.ch<http://www.cern.ch> <http://www.cern.ch><http://www.cern.ch> traceroute to www.cern.ch<http://www.cern.ch> <http://www.cern.ch><http://www.cern.ch> (137.138.144.168), 30 hops max, 60 byte packets 1 138.37.51.254 (138.37.51.254) 0.239 ms 0.172 ms 0.133 ms 2 ebr3.witless.core-net.qmul.ac.uk (138.37.3.196) 0.359 ms 0.290 ms 0.296 ms 3 Te2-2.lond-bar6.ja.net<http://Te2-2.lond-bar6.ja.net> <http://Te2-2.lond-bar6.ja.net><http://Te2-2.lond-bar6.ja.net> (146.97.40.233) 0.684 ms 0.742 ms 0.740 ms 4 ae2.lond-sbr4.ja.net<http://ae2.lond-sbr4.ja.net> <http://ae2.lond-sbr4.ja.net><http://ae2.lond-sbr4.ja.net> (146.97.35.137) 0.685 ms 0.651 ms 0.639 ms 5 ae12.read-sbr1.ja.net<http://ae12.read-sbr1.ja.net> <http://ae12.read-sbr1.ja.net><http://ae12.read-sbr1.ja.net> (146.97.33.141) 2.098 ms 2.159 ms 2.128 ms 6 ae13.lond-sbr3.ja.net<http://ae13.lond-sbr3.ja.net> <http://ae13.lond-sbr3.ja.net><http://ae13.lond-sbr3.ja.net> (146.97.33.146) 8.198 ms 6.232 ms 6.192 ms 7 janet.rt1.lon.uk.geant.net (62.40.124.197) 3.783 ms 3.773 ms 3.735 ms 8 ae3-vlan110.mx1.lon.uk.geant.net (62.40.98.74) 3.728 ms 3.718 ms 3.722 ms 9 ae0.mx1.ams.nl.geant.net (62.40.98.81) 11.240 ms 11.293 ms 11.297 ms 10 ae0.mx1.fra.de.geant.net (62.40.98.128) 18.069 ms 18.058 ms 18.065 ms 11 ae4.rt1.fra.de.geant.net (62.40.98.135) 18.011 ms 18.035 ms 64.931 ms 12 abilene-wash-gw.rt1.fra.de.geant.net (62.40.125.18) 113.006 ms 113.047 ms 112.805 ms 13 ae-2.10.rtr.clev.net.internet2.edu (64.57.28.162) 120.277 ms 119.935 ms 119.917 ms 14 ae-7.10.rtr.chic.net.internet2.edu (64.57.28.158) 135.090 ms 129.386 ms 129.438 ms 15 e513-a-rbrxl-2-te6.cern.ch<http://e513-a-rbrxl-2-te6.cern.ch> <http://e513-a-rbrxl-2-te6.cern.ch><http://e513-a-rbrxl-2-te6.cern.ch> (192.91.246.126) 227.000 ms 222.710 ms 223.052 ms 16 * * * -------------------------------------------- Mark Mitchell, ScotGrid Technical Co-ordinator, Rm 481, Kelvin Building, School of Physics and Astronomy, University of Glasgow, G12 8QQ, UK Telephone: +44-141-330 6439 E Mail: [log in to unmask]<mailto:[log in to unmask]> <mailto:[log in to unmask]><mailto:[log in to unmask]> -- Scanned by iCritical. -------------------------------------------- Mark Mitchell, ScotGrid Technical Co-ordinator, Rm 481, Kelvin Building, School of Physics and Astronomy, University of Glasgow, G12 8QQ, UK Telephone: +44-141-330 6439 E Mail: [log in to unmask]<mailto:[log in to unmask]> <mailto:[log in to unmask]> -------------------------------------------- Mark Mitchell, ScotGrid Technical Co-ordinator, Rm 481, Kelvin Building, School of Physics and Astronomy, University of Glasgow, G12 8QQ, UK Telephone: +44-141-330 6439 E Mail: [log in to unmask]<mailto:[log in to unmask]>