Looking at March
vm015.gla.scotgrid.ac.uk
vm016.gla.scotgrid.ac.uk
seem to have downloaded considerably more than any other host. Indeed
Glasgow seems to account for 28GB out of the total of 69GB.
http://wlcg-squid-monitor.cern.ch/awstats/bin/awstats.pl?month=03&year=2020&output=allhosts&config=cvmfsbproxy.cern.ch&framename=index
Duncan
On 28/02/2020 08:49, Alessandra Forti wrote:
> Hi,
>
> adding /dns_v4_first on/ to the squid caches configuration has
> dramatically reduced access to cvmfsbproxy.cern.ch from Manchester
>
> I don't know how this fits overall with moving to ipv6, but for now can
> all UK sites add it so we can see if that reduces the access from the UK
> overall?
>
> cheers
> alessandra
>
> On 26/02/2020 13:36, Alessandra Forti wrote:
>> On 26/02/2020 13:18, Jose Caballero Bejar - UKRI STFC wrote:
>>> Not Alastair… but I will try…
>>>
>>> I have just checked the access logs for our Squid servers, and I see
>>> they are still accepting connections from outside RAL. Here is from
>>> one of them [1]. So it seems like is not a firewall issue.
>>> WARNING: I checked the logs from today. If needed, I can search for
>>> the past days.
>>>
>>> I am just wondering if the geoIP could be involved here somehow. I
>>> mean, if it is failing finding the closest CVMFS Stratum-1, the
>>> default configuration for the clients may be to go to CERN… Not sure
>>> if this makes sense at all or not :)
>> yes, and we should understand why this is happening.
>>
>>>
>>>
>>> [1]
>>> .gla.ac.uk<http://gla.ac.uk>.
>>> .ic.ac.uk<http://ic.ac.uk>.
>>> .grid.hep.ph.ic.ac.uk.
>>> .gla.scotgrid.ac.uk<http://gla.scotgrid.ac.uk>.
>>> .qmul.ac.uk<http://qmul.ac.uk>.
>>> .nat44-compute.esc.qmul.ac.uk<http://nat44-compute.esc.qmul.ac.uk>.
>>> .dns.cam.ac.uk<http://dns.cam.ac.uk>.
>>> .ox.ac.uk<http://ox.ac.uk>.
>>> .physics.ox.ac.uk<http://physics.ox.ac.uk>.
>>> .aspadmin.com<http://aspadmin.com>.
>>> .googledomains.com<http://googledomains.com>.
>>> .census.shodan.io<http://census.shodan.io>.
>>> .sfj.corp.censys.io<http://sfj.corp.censys.io>.
>>> .ja.net<http://ja.net>.
>>> .ovh.net<http://ovh.net>.
>>>
>>>
>>> Cheers,
>>> Jose
>>>
>>>
>>>
>>>
>>>
>>> On 26 Feb 2020, at 11:23, Brew, Chris (STFC,RAL,PPD)
>>> <[log in to unmask]<mailto:[log in to unmask]>> wrote:
>>>
>>> Hi all,
>>>
>>> We’re becoming aware of some problem in the RAL firewall affecting
>>> web traffic through it, causing connection resets and empty
>>> responses. Alastair are the Tier 1 Squids inside or outside the RAL
>>> firewall?
>>>
>>> Yours,
>>> Chris.
>>>
>>> On 26/02/2020, 10:47, "Testbed Support for GridPP member institutes
>>> on behalf of Alessandra Forti"
>>> <[log in to unmask]<mailto:[log in to unmask]> on
>>> behalf of [log in to unmask]<mailto:[log in to unmask]>>
>>> wrote:
>>>
>>> Looking at the plots in "Full list" under countries for previous
>>> months, it looks like other countries had this problem in the past
>>> but now is mostly the UK and a bit Germany.
>>>
>>> It seems something has changed in other countries infrastructure.
>>> On 26/02/2020 09:45, Alessandra Forti wrote:
>>> I agree I have similar lines but we don't run CMS. It does that for
>>> all repos
>>>
>>> Feb 26 09:08:21 wn1905250 cvmfs2:
>>> (clicdp.cern.ch<http://clicdp.cern.ch>) switching proxy from
>>> http://195.194.104.1:3128<http://195.194.104.1:3128/> to
>>> http://195.194.104.2:3128<http://195.194.104.2:3128/> (reset
>>> load-balanced proxies)
>>> Feb 26 09:08:21 wn1905250 cvmfs2:
>>> (clicdp.cern.ch<http://clicdp.cern.ch>) switching host from
>>> http://cvmfs-stratum-one.cern.ch:8000/cvmfs/clicdp.cern.ch to
>>> http://cernvmfs.gridpp.rl.ac.uk:8000/cvmfs/clicdp.cern.ch (reset host)
>>> Feb 26 09:08:21 wn1905250 cvmfs2:
>>> (clicdp.cern.ch<http://clicdp.cern.ch>) switching host from
>>> http://cernvmfs.gridpp.rl.ac.uk:8000/cvmfs/clicdp.cern.ch to
>>> http://cvmfs-stratum-one.cern.ch:8000/cvmfs/clicdp.cern.ch
>>> Feb 26 09:10:02 wn1905250 cvmfs2:
>>> (sft-nightlies.cern.ch<http://sft-nightlies.cern.ch>) switching host
>>> from
>>> http://cvmfs-stratum-one.cern.ch:8000/cvmfs/sft-nightlies.cern.ch to
>>> http://cernvmfs.gridpp.rl.ac.uk:8000/cvmfs/sft-nightlies.cern.ch
>>> (reset host)
>>> Feb 26 09:10:02 wn1905250 cvmfs2:
>>> (sft-nightlies.cern.ch<http://sft-nightlies.cern.ch>) switching host
>>> from http://cernvmfs.gridpp.rl.ac.uk:8000/cvmfs/sft-nightlies.cern.ch
>>> to http://cvmfs-stratum-one.cern.ch:8000/cvmfs/sft-nightlies.cern.ch
>>> Feb 26 09:11:45 wn1905250 cvmfs2:
>>> (lhcbdev.cern.ch<http://lhcbdev.cern.ch>) switched to catalog
>>> revision 732206
>>> Feb 26 09:15:05 wn1905250 cvmfs2:
>>> (atlas.cern.ch<http://atlas.cern.ch>) switched to catalog revision 60818
>>> Feb 26 09:17:01 wn1905250 cvmfs2:
>>> (lhcbdev.cern.ch<http://lhcbdev.cern.ch>) switched to catalog
>>> revision 732210
>>> Feb 26 09:20:20 wn1905250 cvmfs2: (cms.cern.ch<http://cms.cern.ch>)
>>> switched to catalog revision 88926
>>> Feb 26 09:25:01 wn1905250 cvmfs2:
>>> (lhcbdev.cern.ch<http://lhcbdev.cern.ch>) switching host from
>>> http://cvmfs-stratum-one.cern.ch:8000/cvmfs/lhcbdev.cern.ch to
>>> http://cernvmfs.gridpp.rl.ac.uk:8000/cvmfs/lhcbdev.cern.ch (reset host)
>>> Feb 26 09:25:01 wn1905250 cvmfs2:
>>> (lhcbdev.cern.ch<http://lhcbdev.cern.ch>) switching host from
>>> http://cernvmfs.gridpp.rl.ac.uk:8000/cvmfs/lhcbdev.cern.ch to
>>> http://cvmfs-stratum-one.cern.ch:8000/cvmfs/lhcbdev.cern.ch
>>> Feb 26 09:26:02 wn1905250 cvmfs2:
>>> (lhcbdev.cern.ch<http://lhcbdev.cern.ch>) switched to catalog
>>> revision 732211
>>> Feb 26 09:31:19 wn1905250 cvmfs2:
>>> (lhcbdev.cern.ch<http://lhcbdev.cern.ch>) switched to catalog
>>> revision 732216
>>> Feb 26 09:36:29 wn1905250 cvmfs2:
>>> (lhcbdev.cern.ch<http://lhcbdev.cern.ch>) switched to catalog
>>> revision 732219
>>>
>>> cheers
>>> alessandra
>>>
>>> On 26/02/2020 09:24, Daniel Traynor wrote:
>>>
>>> If I look at our some workernodes (which is where the NAT traffic is
>>> coming from I see in the logs line like)
>>>
>>>
>>>
>>> Feb 26 03:01:03 cn201 cvmfs2: (cms-ib.cern.ch<http://cms-ib.cern.ch>)
>>> switching host from http://cvmfs-stratum-one.cern.ch/opt/cms-ib to
>>> http://cernvmfs.gridpp.rl.ac.uk/opt/cms-ib (reset host)
>>>
>>> Feb 26 03:01:03 cn201 cvmfs2: (cms-ib.cern.ch<http://cms-ib.cern.ch>)
>>> switching host from http://cernvmfs.gridpp.rl.ac.uk/opt/cms-ib to
>>> http://cvmfs-stratum-one.cern.ch/opt/cms-ib
>>>
>>> Feb 26 03:03:03 cn201 cvmfs2: (cms-ib.cern.ch<http://cms-ib.cern.ch>)
>>> switched to catalog revision 180975
>>>
>>> Feb 26 03:08:05 cn201 cvmfs2: (cms-ib.cern.ch<http://cms-ib.cern.ch>)
>>> switched to catalog revision 180977
>>>
>>> Feb 26 03:18:03 cn201 cvmfs2: (cms-ib.cern.ch<http://cms-ib.cern.ch>)
>>> switched to catalog revision 180981
>>>
>>> Feb 26 03:23:04 cn201 cvmfs2: (cms-ib.cern.ch<http://cms-ib.cern.ch>)
>>> switched to catalog revision 180983
>>>
>>> Feb 26 03:31:04 cn201 cvmfs2: (cms-ib.cern.ch<http://cms-ib.cern.ch>)
>>> switching host from http://cvmfs-stratum-one.cern.ch/opt/cms-ib to
>>> http://cernvmfs.gridpp.rl.ac.uk/opt/cms-ib (reset host)
>>>
>>> Feb 26 03:31:13 cn201 cvmfs2: (cms-ib.cern.ch<http://cms-ib.cern.ch>)
>>> switching host from http://cernvmfs.gridpp.rl.ac.uk/opt/cms-ib to
>>> http://cvmfs-stratum-one.cern.ch/opt/cms-ib
>>>
>>>
>>>
>>> Look like it's trying to access RAL then switches back to cern.
>>>
>>>
>>>
>>> As far as i can tell our non CMS cvmfs traffic is using the local
>>> squids (but we don't normally running so much cms work so this might
>>> be an exceptional).
>>>
>>>
>>>
>>> * Dr Daniel Traynor, Grid cluster system manager
>>>
>>> * Tel +44(0)20 7882 6560, Particle Physics,QMUL
>>>
>>>
>>>
>>>
>>>
>>> ________________________________________
>>>
>>> From: Testbed Support for GridPP member institutes
>>> <[log in to unmask]><mailto:[log in to unmask]> on
>>> behalf of Alessandra Forti
>>> <[log in to unmask]><mailto:[log in to unmask]>
>>>
>>> Sent: 26 February 2020 09:10
>>>
>>> To: [log in to unmask]<mailto:[log in to unmask]>
>>>
>>> Subject: Re: Frontier squids in the UK
>>>
>>>
>>>
>>> Hi Dan,
>>>
>>>
>>>
>>> the problem is not the activity, is that the WNs shouldn't access
>>> CERN. The UK is doing so disproportionately.
>>>
>>>
>>>
>>> [cid:part1.19C41B5F.4722DEFB@cern.ch]
>>>
>>>
>>>
>>>
>>>
>>> On 26/02/2020 09:06, Daniel Traynor wrote:
>>>
>>>
>>>
>>> At QM we updated to frontier-squid-4.10-2.1.x86_64 on the 14th Feb.
>>> Internally I don't see any difference in behaviour/ performance of
>>> the squids before or after the update.
>>>
>>>
>>>
>>> I see both our NATs our listed near the top list but that's probable
>>> related to the large amount of CMS we doing at the moment.
>>>
>>>
>>>
>>> dan
>>>
>>>
>>>
>>>
>>>
>>> * Dr Daniel Traynor, Grid cluster system manager
>>>
>>> * Tel +44(0)20 7882 6560, Particle Physics,QMUL
>>>
>>>
>>>
>>>
>>>
>>> ________________________________________
>>>
>>> From: Testbed Support for GridPP member institutes
>>> <[log in to unmask]><mailto:[log in to unmask]><mailto:[log in to unmask]><mailto:[log in to unmask]>
>>> on behalf of Alessandra Forti
>>> <[log in to unmask]><mailto:[log in to unmask]><mailto:[log in to unmask]><mailto:[log in to unmask]>
>>>
>>>
>>> Sent: 26 February 2020 08:15
>>>
>>> To:
>>> [log in to unmask]<mailto:[log in to unmask]><mailto:[log in to unmask]><mailto:[log in to unmask]>
>>>
>>>
>>> Subject: Frontier squids in the UK
>>>
>>>
>>>
>>> Hi,
>>>
>>>
>>>
>>> it seems there is a problem with the UK sites WNs accessing frontier
>>>
>>> squid. In particular Manchester, RAL, RALPP and Oxford the others tooo
>>>
>>> but they are behind a NAT and not as many nodes appear in the list.
>>>
>>>
>>>
>>> https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwlcg-squid-monitor.cern.ch%2Fawstats%2Fbin%2Fawstats.pl%3Fconfig%3Dcvmfsbproxy.cern.ch%26databasebreak%3Dday%26day%3D25&data=02%7C01%7C%7Cebdecf99190248c3600a08d7ba941a8f%7C569df091b01340e386eebd9cb9e25814%7C0%7C0%7C637183017572955219&sdata=tri9QzQT0sN%2BJAO1jlmf0X21X3yQ%2Bo%2BTdTnD%2BEVcZhM%3D&reserved=0<https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwlcg-squid-monitor.cern.ch%2Fawstats%2Fbin%2Fawstats.pl%3Fconfig%3Dcvmfsbproxy.cern.ch%26databasebreak%3Dday%26day%3D25&data=02%7C01%7C%7Cd4aa7154bfeb4660e3db08d7ba9bdc21%7C569df091b01340e386eebd9cb9e25814%7C0%7C0%7C637183050884940544&sdata=uUXYXuha6GUG8cn%2FEs2a0Gyx13tQbAKvkG86U%2Fe9oy0%3D&reserved=0><https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwlcg-squid-monitor.cern.ch%2Fawstats%2Fbin%2Fawstats.pl%3Fconfig%3Dcvmfsbproxy.cern.ch%26databasebreak%3Dday%26day%3D25&data=02%7C01%7C%7Cd4aa7154bfeb4660e3db08d7ba9bdc21%7C569df091b01340e386eebd9cb9e25814%7C0%7C0%7C637183050884940544&sdata=uUXYXuha6GUG8cn%2FEs2a0Gyx13tQbAKvkG86U%2Fe9oy0%3D&reserved=0>
>>>
>>>
>>>
>>>
>>> you need to filter for ac.uk<http://ac.uk>
>>>
>>>
>>>
>>> Can I ask what version of squid you are using and if you are using
>>>
>>> frontier squid or standard OS squid?
>>>
>>>
>>>
>>> Frontier changed the baseline from 3.5.28 to 4.10.1-1 two weeks ago
>>>
>>> which is not even available on centos8.
>>>
>>>
>>>
>>> thanks
>>>
>>>
>>>
>>> cheers
>>>
>>> alessandra
>>>
>>>
>>>
>>> --
>>>
>>> Inference: a conclusion reached on the basis of evidence and reasoning
>>>
>>> Respect is a rational process. \\//
>>>
>>> For Ur-Fascism, disagreement is treason. (U. Eco)
>>>
>>>
>>>
>>> ########################################################################
>>>
>>>
>>>
>>> To unsubscribe from the TB-SUPPORT list, click the following link:
>>>
>>> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.jiscmail.ac.uk%2Fcgi-bin%2Fwebadmin%3FSUBED1%3DTB-SUPPORT%26A%3D1&data=02%7C01%7C%7Cebdecf99190248c3600a08d7ba941a8f%7C569df091b01340e386eebd9cb9e25814%7C0%7C0%7C637183017572955219&sdata=F6MgNpBRfiP3GYbzAm09jAy8NP5OJyhbjGR2Qw5Kv9Q%3D&reserved=0<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.jiscmail.ac.uk%2Fcgi-bin%2Fwebadmin%3FSUBED1%3DTB-SUPPORT%26A%3D1&data=02%7C01%7C%7Cd4aa7154bfeb4660e3db08d7ba9bdc21%7C569df091b01340e386eebd9cb9e25814%7C0%7C0%7C637183050884940544&sdata=QHC2ESn4JQC07OiWx7%2BRb%2F1J3EyoC5dWb6ZHJq3tEQ8%3D&reserved=0><https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.jiscmail.ac.uk%2Fcgi-bin%2Fwebadmin%3FSUBED1%3DTB-SUPPORT%26A%3D1&data=02%7C01%7C%7Cd4aa7154bfeb4660e3db08d7ba9bdc21%7C569df091b01340e386eebd9cb9e25814%7C0%7C0%7C637183050884940544&sdata=QHC2ESn4JQC07OiWx7%2BRb%2F1J3EyoC5dWb6ZHJq3tEQ8%3D&reserved=0>
>>>
>>>
>>>
>>>
>>> ########################################################################
>>>
>>>
>>>
>>> To unsubscribe from the TB-SUPPORT list, click the following link:
>>>
>>> https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=TB-SUPPORT&A=1<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.jiscmail.ac.uk%2Fcgi-bin%2Fwebadmin%3FSUBED1%3DTB-SUPPORT%26A%3D1&data=02%7C01%7C%7Cd4aa7154bfeb4660e3db08d7ba9bdc21%7C569df091b01340e386eebd9cb9e25814%7C0%7C0%7C637183050884950541&sdata=iOac5TL90beyHsfXcJ39y5Lbc6aNFCHULbN7QTjQ%2BHA%3D&reserved=0><https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.jiscmail.ac.uk%2Fcgi-bin%2Fwebadmin%3FSUBED1%3DTB-SUPPORT%26A%3D1&data=02%7C01%7C%7Cd4aa7154bfeb4660e3db08d7ba9bdc21%7C569df091b01340e386eebd9cb9e25814%7C0%7C0%7C637183050884950541&sdata=iOac5TL90beyHsfXcJ39y5Lbc6aNFCHULbN7QTjQ%2BHA%3D&reserved=0>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> --
>>>
>>> Inference: a conclusion reached on the basis of evidence and reasoning
>>>
>>> Respect is a rational process. \\//
>>>
>>> For Ur-Fascism, disagreement is treason. (U. Eco)
>>>
>>>
>>>
>>>
>>>
>>> ________________________________
>>>
>>>
>>>
>>> To unsubscribe from the TB-SUPPORT list, click the following link:
>>>
>>> https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=TB-SUPPORT&A=1<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.jiscmail.ac.uk%2Fcgi-bin%2Fwebadmin%3FSUBED1%3DTB-SUPPORT%26A%3D1&data=02%7C01%7C%7Cd4aa7154bfeb4660e3db08d7ba9bdc21%7C569df091b01340e386eebd9cb9e25814%7C0%7C0%7C637183050884960536&sdata=dhNIv9Y9I%2BEXNTlE3B%2Fi%2FYvbzOi7EnhVHFjN%2BJnwtNA%3D&reserved=0><https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.jiscmail.ac.uk%2Fcgi-bin%2Fwebadmin%3FSUBED1%3DTB-SUPPORT%26A%3D1&data=02%7C01%7C%7Cd4aa7154bfeb4660e3db08d7ba9bdc21%7C569df091b01340e386eebd9cb9e25814%7C0%7C0%7C637183050884960536&sdata=dhNIv9Y9I%2BEXNTlE3B%2Fi%2FYvbzOi7EnhVHFjN%2BJnwtNA%3D&reserved=0>
>>>
>>>
>>>
>>>
>>> ########################################################################
>>>
>>>
>>>
>>> To unsubscribe from the TB-SUPPORT list, click the following link:
>>>
>>> https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=TB-SUPPORT&A=1
>>>
>>>
>>>
>>>
>>>
>>> --
>>>
>>> Inference: a conclusion reached on the basis of evidence and reasoning
>>>
>>> Respect is a rational process. \\//
>>>
>>> For Ur-Fascism, disagreement is treason. (U. Eco)
>>>
>>>
>>> ________________________________
>>>
>>> To unsubscribe from the TB-SUPPORT list, click the following link:
>>> https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=TB-SUPPORT&A=1
>>>
>>>
>>>
>>> --
>>>
>>> Inference: a conclusion reached on the basis of evidence and reasoning
>>>
>>> Respect is a rational process. \\//
>>>
>>> For Ur-Fascism, disagreement is treason. (U. Eco)
>>>
>>>
>>> ________________________________
>>>
>>> To unsubscribe from the TB-SUPPORT list, click the following link:
>>> https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=TB-SUPPORT&A=1
>>>
>>> ________________________________
>>>
>>> To unsubscribe from the TB-SUPPORT list, click the following link:
>>> https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=TB-SUPPORT&A=1
>>>
>>>
>>> ########################################################################
>>>
>>> To unsubscribe from the TB-SUPPORT list, click the following link:
>>> https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=TB-SUPPORT&A=1
>>
>
########################################################################
To unsubscribe from the TB-SUPPORT list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=TB-SUPPORT&A=1
|