On 03/05/13 08:16, Mark Mitchell wrote:
> Hiya,
> Spoke to the CERN network guys this morning. It appears that Geant had a
> BGP route update issue.
> It was fixed around 1 AM this morning.
>
Excellent.
[snip]
> What is of interest is that the alarms are lower level than I thought
> they would be.
> As the service "stayed up" is this an issue for us? I am really
> surprised that the experiments don't look at these types of topology
> changes but this was requested yesterday at the meeting. Should this be
> something we want to raise as an issue? I think we should.
> Also, Chris the CERN network guys said to say thank you for picking this
> up.
No problem.
I actually have some monitoring of routes from CERN to QMUL for a few
weeks (from my ripe atlas probe).
http://collect.qmul.ac.uk/down?t=R83ETS82VKPC0S4D/5142PDTSF3S9QKLJLR4AE30
It should be there for 2 weeks. Format is json - which I haven't tried
to interpret. If the network guys at CERN want it, they are welcome to it.
Chris
> regards,
> Mark
>
>
>
> On 3 May 2013, at 08:51, Alastair Dewhurst wrote:
>
>> Hi
>>
>> I did a trace route from RAL last night and saw the same issue. I have
>> checked now and it is normal. This was on a UI, I didn't check a
>> machine on the OPN subnet.
>>
>> Alastair
>>
>> 2/5/13
>> # traceroute www.cern.ch <http://www.cern.ch>
>> traceroute to www.cern.ch <http://www.cern.ch> (137.138.144.168), 30
>> hops max, 40 byte packets
>> 1 gw (130.246.180.254) 0.714 ms 1.088 ms 1.477 ms
>> 2 130.246.0.6 (130.246.0.6) 0.516 ms 0.659 ms 0.794 ms
>> 3 192.100.78.61 (192.100.78.61) 2.332 ms 2.325 ms 2.807 ms
>> 4 ae6.read-sbr1.ja.net <http://ae6.read-sbr1.ja.net> (146.97.41.157)
>> 3.249 ms 3.232 ms 3.226 ms
>> 5 ae13.lond-sbr3.ja.net <http://ae13.lond-sbr3.ja.net> (146.97.33.146)
>> 4.898 ms 5.086 ms 4.878 ms
>> 6 janet.rt1.lon.uk.geant.net (62.40.124.197) 4.866 ms 4.764 ms 4.647 ms
>> 7 ae3-vlan110.mx1.lon.uk.geant.net (62.40.98.74) 4.747 ms 4.886 ms
>> 4.973 ms
>> 8 ae0.mx1.ams.nl.geant.net (62.40.98.81) 11.813 ms 11.448 ms 11.407 ms
>> 9 ae0.mx1.fra.de.geant.net (62.40.98.128) 18.381 ms 17.583 ms 17.716 ms
>> 10 ae4.rt1.fra.de.geant.net (62.40.98.135) 18.900 ms 17.412 ms 22.400 ms
>> 11 abilene-wash-gw.rt1.fra.de.geant.net (62.40.125.18) 135.625 ms
>> 135.617 ms 135.653 ms
>> 12 ae-2.10.rtr.clev.net.internet2.edu (64.57.28.162) 140.278 ms
>> 140.317 ms 140.408 ms
>> 13 ae-7.10.rtr.chic.net.internet2.edu (64.57.28.158) 152.877 ms
>> 156.432 ms 150.719 ms
>> 14 e513-a-rbrxl-2-te6.cern.ch <http://e513-a-rbrxl-2-te6.cern.ch>
>> (192.91.246.126) 248.055 ms 240.901 ms 240.828 ms
>>
>> 3/5/13
>> # traceroute www.cern.ch <http://www.cern.ch>
>> traceroute to www.cern.ch <http://www.cern.ch> (137.138.144.168), 30
>> hops max, 40 byte packets
>> 1 gw (130.246.180.254) 0.730 ms 1.532 ms 1.970 ms
>> 2 130.246.0.6 (130.246.0.6) 0.310 ms 0.446 ms 0.488 ms
>> 3 192.100.78.61 (192.100.78.61) 2.271 ms 2.265 ms 2.298 ms
>> 4 ae6.read-sbr1.ja.net <http://ae6.read-sbr1.ja.net> (146.97.41.157)
>> 24.752 ms 24.745 ms 24.736 ms
>> 5 ae13.lond-sbr3.ja.net <http://ae13.lond-sbr3.ja.net> (146.97.33.146)
>> 3.751 ms 18.762 ms 18.801 ms
>> 6 janet.rt1.lon.uk.geant.net (62.40.124.197) 4.459 ms 4.265 ms 4.131 ms
>> 7 ae3-vlan110.mx1.lon.uk.geant.net (62.40.98.74) 4.078 ms 3.525 ms
>> 3.765 ms
>> 8 ae0.mx1.par.fr.geant.net (62.40.98.77) 8.852 ms 8.819 ms 8.852 ms
>> 9 ae4.rt1.par.fr.geant.net (62.40.112.185) 59.005 ms 9.371 ms 58.473 ms
>> 10 switch-bckp-gw.rt1.par.fr.geant.net (62.40.124.82) 17.973 ms 17.966
>> ms 18.052 ms
>> 11 e513-a-rbrxl-2-te20.cern.ch <http://e513-a-rbrxl-2-te20.cern.ch>
>> (192.65.184.70) 21.149 ms 20.655 ms 17.651 ms
>>
>>
>>
>> On 2 May 2013, at 23:19, Christopher J. Walker wrote:
>>
>>> On 02/05/13 21:19, Mark Mitchell wrote:
>>>> Hiya,
>>>>
>>>> I think Chris wins the network vigilance award this week :-)
>>>> There appears to be an issue with the Geant network.
>>>> On a serious note, Chris how did you pick this up?
>>>
>>> Someone came into the office and asked was there a problem with the
>>> network...
>>>
>>> Further investigation revealed he'd been trying to videoconference to
>>> CERN and had poor performance so ended up using the phone bridge.
>>>
>>> The QMUL link to Janet was fine, and whilst pinging CERN didn't have
>>> huge packet loss, ping times of 230ms seemed surprisingly long, and
>>> traceroute revealed packets looked like they were going across the
>>> atlantic.
>>>
>>>> Did any of the experiment software pick up on this?
>>>> Email from Edoardo is attached below.
>>>
>>> Thanks.
>>>
>>> Chris
>>>
>>> PS Although IPv6 traffic is going to the US, IPv6 traffic isn't.
>>>
>>> [root@nm03 ~]# ping -c 2 www.ipv6.cern.ch <http://www.ipv6.cern.ch>
>>> PING cernis22-21.cern.ch <http://cernis22-21.cern.ch>
>>> (137.138.218.32) 56(84) bytes of data.
>>> 64 bytes from cernis22-21.cern.ch <http://cernis22-21.cern.ch>
>>> (137.138.218.32): icmp_seq=1 ttl=110 time=221 ms
>>> 64 bytes from cernis22-21.cern.ch <http://cernis22-21.cern.ch>
>>> (137.138.218.32): icmp_seq=1 ttl=110 time=221 ms (DUP!)
>>> 64 bytes from cernis22-21.cern.ch <http://cernis22-21.cern.ch>
>>> (137.138.218.32): icmp_seq=2 ttl=110 time=219 ms
>>>
>>> --- cernis22-21.cern.ch <http://cernis22-21.cern.ch> ping statistics ---
>>> 2 packets transmitted, 2 received, +1 duplicates, 0% packet loss,
>>> time 1000ms
>>> rtt min/avg/max/mdev = 219.561/220.903/221.577/0.948 ms
>>> [root@nm03 ~]# ping6 -c 2 www.ipv6.cern.ch <http://www.ipv6.cern.ch>
>>> PING www.ipv6.cern.ch <http://www.ipv6.cern.ch>(cernis22-21.cern.ch
>>> <http://cernis22-21.cern.ch>) 56 data bytes
>>> 64 bytes from cernis22-21.cern.ch <http://cernis22-21.cern.ch>:
>>> icmp_seq=0 ttl=50 time=28.7 ms
>>> 64 bytes from cernis22-21.cern.ch <http://cernis22-21.cern.ch>:
>>> icmp_seq=0 ttl=50 time=28.7 ms (DUP!)
>>> 64 bytes from cernis22-21.cern.ch <http://cernis22-21.cern.ch>:
>>> icmp_seq=1 ttl=50 time=16.0 ms
>>> 64 bytes from cernis22-21.cern.ch <http://cernis22-21.cern.ch>:
>>> icmp_seq=1 ttl=50 time=16.0 ms (DUP!)
>>>
>>> Traceroute6/tracepath6 don't work to that host - they stop somewhere
>>> with no name (no idea if it's inside or outside CERN).
>>>
>>> Chris
>>>
>>>> regards,
>>>> Mark
>>>>
>>>> -------- Original Message --------
>>>> Subject: few prefixes from Geant?
>>>> Date: Thu, 2 May 2013 21:59:23 +0200
>>>> From: Edoardo Martelli <[log in to unmask]
>>>> <mailto:[log in to unmask]>
>>>> <mailto:[log in to unmask]>>
>>>> To: NOC Switch <[log in to unmask] <mailto:[log in to unmask]>
>>>> <mailto:[log in to unmask]>>, NOC Geant
>>>> <[log in to unmask] <mailto:[log in to unmask]> <mailto:[log in to unmask]>>
>>>> CC: Cern Extip <[log in to unmask] <mailto:[log in to unmask]>
>>>> <mailto:[log in to unmask]>>
>>>>
>>>> Hello Geant and SWITCH NOC
>>>>
>>>> We don't get many prefixes from the peerings with SWitch and most of
>>>> our traffic to the Europen NRENs is going via Internet2.
>>>>
>>>> It seems to me that the prefixes are not even received by Switch.
>>>>
>>>> Is there any issue?
>>>>
>>>> cheers
>>>> Edoardo
>>>>
>>>>
>>>> Router: swiCE2.switch.ch <http://swiCE2.switch.ch>
>>>> <http://swiCE2.switch.ch/>
>>>> Command: show ip bgp 62.40.105.162
>>>>
>>>> BGP routing table entry for 62.40.96.0/19, version 23788441
>>>> Paths: (1 available, no best path)
>>>> Multipath: eBGP
>>>> Not advertised to any peer
>>>> 3356 20965, (received-only)
>>>> 213.242.73.73 from 213.242.73.73 (4.69.187.34)
>>>> Origin IGP, metric 0, localpref 100, valid, external
>>>> Community: 3356:2 3356:22 3356:100 3356:123 3356:501 3356:2065
>>>> 20965:155
>>>>
>>>>
>>>>
>>>> On 2 May 2013, at 20:22, John Gordon wrote:
>>>>
>>>>> Mark the workshop is timely, I was about to suggest that this routing
>>>>> might be a 'benefit ' of LHCONE. We have already seen one instance of
>>>>> Germany assuming that everyone is on LHCONE.
>>>>>
>>>>> John
>>>>>
>>>>> Mark Mitchell <[log in to unmask]
>>>>> <mailto:[log in to unmask]>
>>>>> <mailto:[log in to unmask]>> wrote:
>>>>>
>>>>>
>>>>>
>>>>> Hi Chris,
>>>>> Same pattern at Glasgow
>>>>> I am at the LHCONE workshop so I can ask in the morning as there are
>>>>> multiple devices from multiple networks on both sides of the Atlantic
>>>>> doing WAN links and it may be a case of naming conventions? However
>>>>> given that on one visual trace route it appears to have been sourced
>>>>> in the US, I am a tad confused.
>>>>> I will ask the internet 2 guys tomorrow morning.
>>>>> regards,
>>>>> Mark
>>>>>
>>>>> traceroute to www.cern.ch <http://www.cern.ch>
>>>>> <http://www.cern.ch><http://www.cern.ch>
>>>>> (137.138.144.168), 30 hops max, 40 byte packets
>>>>> 1 130.209.239.2 (130.209.239.2) 0.452 ms 0.554 ms 0.677 ms
>>>>> 2 glasgowpop-ge1-2-glasgowuni-ge1-1-v152.clyde.net.uk
>>>>> <http://glasgowpop-ge1-2-glasgowuni-ge1-1-v152.clyde.net.uk>
>>>>> <http://glasgowpop-ge1-2-glasgowuni-ge1-1-v152.clyde.net.uk><http://glasgowpop-ge1-2-glasgowuni-ge1-1-v152.clyde.net.uk>
>>>>> (194.81.62.153) 0.399 ms 0.380 ms 0.413 ms
>>>>> 3 so-2-0-0.glas-sbr1.ja.net <http://so-2-0-0.glas-sbr1.ja.net>
>>>>> <http://so-2-0-0.glas-sbr1.ja.net><http://so-2-0-0.glas-sbr1.ja.net>
>>>>> (146.97.40.97) 0.510 ms 0.562 ms 0.646 ms
>>>>> 4 ae14.warr-sbr1.ja.net <http://ae14.warr-sbr1.ja.net>
>>>>> <http://ae14.warr-sbr1.ja.net><http://ae14.warr-sbr1.ja.net>
>>>>> (146.97.33.121) 5.207 ms 5.211 ms 5.206 ms
>>>>> 5 as3.read-sbr1.ja.net <http://as3.read-sbr1.ja.net>
>>>>> <http://as3.read-sbr1.ja.net><http://as3.read-sbr1.ja.net>
>>>>> (146.97.33.89) 8.975 ms 8.982 ms 8.978 ms
>>>>> 6 ae13.lond-sbr3.ja.net <http://ae13.lond-sbr3.ja.net>
>>>>> <http://ae13.lond-sbr3.ja.net><http://ae13.lond-sbr3.ja.net>
>>>>> (146.97.33.146) 10.336 ms 10.327 ms 10.348 ms
>>>>> 7 janet.rt1.lon.uk.geant.net (62.40.124.197) 66.367 ms 64.430 ms
>>>>> 63.623 ms
>>>>> 8 ae3-vlan110.mx1.lon.uk.geant.net (62.40.98.74) 10.525 ms 10.528 ms
>>>>> 10.525 ms
>>>>> 9 ae0.mx1.ams.nl.geant.net (62.40.98.81) 18.056 ms 18.061 ms 18.055 ms
>>>>> 10 ae0.mx1.fra.de.geant.net (62.40.98.128) 24.746 ms 24.793 ms
>>>>> 24.762 ms
>>>>> 11 ae4.rt1.fra.de.geant.net (62.40.98.135) 24.854 ms 24.790 ms
>>>>> 24.894 ms
>>>>> 12 abilene-wash-gw.rt1.fra.de.geant.net (62.40.125.18) 118.807 ms
>>>>> 118.923 ms 119.183 ms
>>>>> 13 ae-2.10.rtr.clev.net.internet2.edu (64.57.28.162) 126.910 ms
>>>>> 126.890 ms 126.978 ms
>>>>> 14 ae-7.10.rtr.chic.net.internet2.edu (64.57.28.158) 135.798 ms
>>>>> 137.945 ms 135.682 ms
>>>>> 15 e513-a-rbrxl-2-te6.cern.ch <http://e513-a-rbrxl-2-te6.cern.ch>
>>>>> <http://e513-a-rbrxl-2-te6.cern.ch><http://e513-a-rbrxl-2-te6.cern.ch>
>>>>> (192.91.246.126) 232.462 ms 232.407 ms 232.565 ms
>>>>> 16 * * *
>>>>> 17 * * *
>>>>> 18 * * *
>>>>> 19 * * *
>>>>> 20 * * *
>>>>> 21 * *
>>>>> On 2 May 2013, at 19:05, Christopher J. Walker wrote:
>>>>>
>>>>> Packets to (and indeed from) CERN seem to be taking the scenic
>>>>> route via
>>>>> the USA.
>>>>>
>>>>> I'm curious if this is deliberate...
>>>>>
>>>>> walker@heppc046:~$ traceroute www.cern.ch <http://www.cern.ch>
>>>>> <http://www.cern.ch><http://www.cern.ch>
>>>>> traceroute to www.cern.ch <http://www.cern.ch>
>>>>> <http://www.cern.ch><http://www.cern.ch>
>>>>> (137.138.144.168), 30 hops max, 60 byte packets
>>>>> 1 138.37.51.254 (138.37.51.254) 0.239 ms 0.172 ms 0.133 ms
>>>>> 2 ebr3.witless.core-net.qmul.ac.uk (138.37.3.196) 0.359 ms 0.290 ms
>>>>> 0.296 ms
>>>>> 3 Te2-2.lond-bar6.ja.net <http://Te2-2.lond-bar6.ja.net>
>>>>> <http://Te2-2.lond-bar6.ja.net><http://Te2-2.lond-bar6.ja.net>
>>>>> (146.97.40.233) 0.684 ms 0.742 ms 0.740 ms
>>>>> 4 ae2.lond-sbr4.ja.net <http://ae2.lond-sbr4.ja.net>
>>>>> <http://ae2.lond-sbr4.ja.net><http://ae2.lond-sbr4.ja.net>
>>>>> (146.97.35.137) 0.685 ms 0.651 ms 0.639 ms
>>>>> 5 ae12.read-sbr1.ja.net <http://ae12.read-sbr1.ja.net>
>>>>> <http://ae12.read-sbr1.ja.net><http://ae12.read-sbr1.ja.net>
>>>>> (146.97.33.141) 2.098 ms 2.159 ms 2.128 ms
>>>>> 6 ae13.lond-sbr3.ja.net <http://ae13.lond-sbr3.ja.net>
>>>>> <http://ae13.lond-sbr3.ja.net><http://ae13.lond-sbr3.ja.net>
>>>>> (146.97.33.146) 8.198 ms 6.232 ms 6.192 ms
>>>>> 7 janet.rt1.lon.uk.geant.net (62.40.124.197) 3.783 ms 3.773 ms 3.735 ms
>>>>> 8 ae3-vlan110.mx1.lon.uk.geant.net (62.40.98.74) 3.728 ms 3.718 ms
>>>>> 3.722 ms
>>>>> 9 ae0.mx1.ams.nl.geant.net (62.40.98.81) 11.240 ms 11.293 ms 11.297 ms
>>>>> 10 ae0.mx1.fra.de.geant.net (62.40.98.128) 18.069 ms 18.058 ms
>>>>> 18.065 ms
>>>>> 11 ae4.rt1.fra.de.geant.net (62.40.98.135) 18.011 ms 18.035 ms
>>>>> 64.931 ms
>>>>> 12 abilene-wash-gw.rt1.fra.de.geant.net (62.40.125.18) 113.006 ms
>>>>> 113.047 ms 112.805 ms
>>>>> 13 ae-2.10.rtr.clev.net.internet2.edu (64.57.28.162) 120.277 ms
>>>>> 119.935 ms 119.917 ms
>>>>> 14 ae-7.10.rtr.chic.net.internet2.edu (64.57.28.158) 135.090 ms
>>>>> 129.386 ms 129.438 ms
>>>>> 15 e513-a-rbrxl-2-te6.cern.ch <http://e513-a-rbrxl-2-te6.cern.ch>
>>>>> <http://e513-a-rbrxl-2-te6.cern.ch><http://e513-a-rbrxl-2-te6.cern.ch>
>>>>> (192.91.246.126) 227.000 ms 222.710 ms
>>>>> 223.052 ms
>>>>> 16 * * *
>>>>>
>>>>> --------------------------------------------
>>>>> Mark Mitchell,
>>>>> ScotGrid Technical Co-ordinator,
>>>>> Rm 481,
>>>>> Kelvin Building,
>>>>> School of Physics and Astronomy,
>>>>> University of Glasgow,
>>>>> G12 8QQ, UK
>>>>> Telephone: +44-141-330 6439
>>>>> E Mail: [log in to unmask]
>>>>> <mailto:[log in to unmask]>
>>>>> <mailto:[log in to unmask]><mailto:[log in to unmask]>
>>>>>
>>>>> --
>>>>> Scanned by iCritical.
>>>>
>>>> --------------------------------------------
>>>> Mark Mitchell,
>>>> ScotGrid Technical Co-ordinator,
>>>> Rm 481,
>>>> Kelvin Building,
>>>> School of Physics and Astronomy,
>>>> University of Glasgow,
>>>> G12 8QQ, UK
>>>> Telephone: +44-141-330 6439
>>>> E Mail: [log in to unmask]
>>>> <mailto:[log in to unmask]>
>>>> <mailto:[log in to unmask]>
>>>>
>
> --------------------------------------------
> Mark Mitchell,
> ScotGrid Technical Co-ordinator,
> Rm 481,
> Kelvin Building,
> School of Physics and Astronomy,
> University of Glasgow,
> G12 8QQ, UK
> Telephone: +44-141-330 6439
> E Mail: [log in to unmask] <mailto:[log in to unmask]>
>
|