Print

Print


Looking at the plots in "Full list" under countries for previous months, it looks like other countries had this problem in the past but now is mostly the UK and a bit Germany.

It seems something has changed in other countries infrastructure.

On 26/02/2020 09:45, Alessandra Forti wrote:
[log in to unmask]"> I agree I have similar lines but we don't run CMS. It does that for all repos

Feb 26 09:08:21 wn1905250 cvmfs2: (clicdp.cern.ch) switching proxy from http://195.194.104.1:3128 to http://195.194.104.2:3128 (reset load-balanced proxies)
Feb 26 09:08:21 wn1905250 cvmfs2: (clicdp.cern.ch) switching host from http://cvmfs-stratum-one.cern.ch:8000/cvmfs/clicdp.cern.ch to http://cernvmfs.gridpp.rl.ac.uk:8000/cvmfs/clicdp.cern.ch (reset host)
Feb 26 09:08:21 wn1905250 cvmfs2: (clicdp.cern.ch) switching host from http://cernvmfs.gridpp.rl.ac.uk:8000/cvmfs/clicdp.cern.ch to http://cvmfs-stratum-one.cern.ch:8000/cvmfs/clicdp.cern.ch
Feb 26 09:10:02 wn1905250 cvmfs2: (sft-nightlies.cern.ch) switching host from http://cvmfs-stratum-one.cern.ch:8000/cvmfs/sft-nightlies.cern.ch to http://cernvmfs.gridpp.rl.ac.uk:8000/cvmfs/sft-nightlies.cern.ch (reset host)
Feb 26 09:10:02 wn1905250 cvmfs2: (sft-nightlies.cern.ch) switching host from http://cernvmfs.gridpp.rl.ac.uk:8000/cvmfs/sft-nightlies.cern.ch to http://cvmfs-stratum-one.cern.ch:8000/cvmfs/sft-nightlies.cern.ch
Feb 26 09:11:45 wn1905250 cvmfs2: (lhcbdev.cern.ch) switched to catalog revision 732206
Feb 26 09:15:05 wn1905250 cvmfs2: (atlas.cern.ch) switched to catalog revision 60818
Feb 26 09:17:01 wn1905250 cvmfs2: (lhcbdev.cern.ch) switched to catalog revision 732210
Feb 26 09:20:20 wn1905250 cvmfs2: (cms.cern.ch) switched to catalog revision 88926
Feb 26 09:25:01 wn1905250 cvmfs2: (lhcbdev.cern.ch) switching host from http://cvmfs-stratum-one.cern.ch:8000/cvmfs/lhcbdev.cern.ch to http://cernvmfs.gridpp.rl.ac.uk:8000/cvmfs/lhcbdev.cern.ch (reset host)
Feb 26 09:25:01 wn1905250 cvmfs2: (lhcbdev.cern.ch) switching host from http://cernvmfs.gridpp.rl.ac.uk:8000/cvmfs/lhcbdev.cern.ch to http://cvmfs-stratum-one.cern.ch:8000/cvmfs/lhcbdev.cern.ch
Feb 26 09:26:02 wn1905250 cvmfs2: (lhcbdev.cern.ch) switched to catalog revision 732211
Feb 26 09:31:19 wn1905250 cvmfs2: (lhcbdev.cern.ch) switched to catalog revision 732216
Feb 26 09:36:29 wn1905250 cvmfs2: (lhcbdev.cern.ch) switched to catalog revision 732219


cheers
alessandra


On 26/02/2020 09:24, Daniel Traynor wrote:
[log in to unmask]">
If I look at our some workernodes (which is where the NAT traffic is coming from I see in the logs line like)

Feb 26 03:01:03 cn201 cvmfs2: (cms-ib.cern.ch) switching host from http://cvmfs-stratum-one.cern.ch/opt/cms-ib to http://cernvmfs.gridpp.rl.ac.uk/opt/cms-ib (reset host)
Feb 26 03:01:03 cn201 cvmfs2: (cms-ib.cern.ch) switching host from http://cernvmfs.gridpp.rl.ac.uk/opt/cms-ib to http://cvmfs-stratum-one.cern.ch/opt/cms-ib
Feb 26 03:03:03 cn201 cvmfs2: (cms-ib.cern.ch) switched to catalog revision 180975
Feb 26 03:08:05 cn201 cvmfs2: (cms-ib.cern.ch) switched to catalog revision 180977
Feb 26 03:18:03 cn201 cvmfs2: (cms-ib.cern.ch) switched to catalog revision 180981
Feb 26 03:23:04 cn201 cvmfs2: (cms-ib.cern.ch) switched to catalog revision 180983
Feb 26 03:31:04 cn201 cvmfs2: (cms-ib.cern.ch) switching host from http://cvmfs-stratum-one.cern.ch/opt/cms-ib to http://cernvmfs.gridpp.rl.ac.uk/opt/cms-ib (reset host)
Feb 26 03:31:13 cn201 cvmfs2: (cms-ib.cern.ch) switching host from http://cernvmfs.gridpp.rl.ac.uk/opt/cms-ib to http://cvmfs-stratum-one.cern.ch/opt/cms-ib

Look like it's trying to access RAL then switches back to cern. 

As far as i can tell our non CMS cvmfs traffic is using the local squids (but we don't normally running so much cms work so this might be an exceptional).

* Dr Daniel Traynor, Grid cluster system manager
* Tel +44(0)20 7882 6560, Particle Physics,QMUL


________________________________________
From: Testbed Support for GridPP member institutes <[log in to unmask]> on behalf of Alessandra Forti <[log in to unmask]>
Sent: 26 February 2020 09:10
To: [log in to unmask]
Subject: Re: Frontier squids in the UK

Hi Dan,

the problem is not the activity, is that the WNs shouldn't access CERN. The UK is doing so disproportionately.

[cid:part1.19C41B5F.4722DEFB@cern.ch]


On 26/02/2020 09:06, Daniel Traynor wrote:

At QM we updated to frontier-squid-4.10-2.1.x86_64 on the 14th Feb. Internally I don't see any difference in behaviour/ performance of the squids before or after the update.

I see both our NATs our listed near the top list but that's probable related to the large amount of CMS we doing at the moment.

dan


* Dr Daniel Traynor, Grid cluster system manager
* Tel +44(0)20 7882 6560, Particle Physics,QMUL


________________________________________
From: Testbed Support for GridPP member institutes <[log in to unmask]><mailto:[log in to unmask]> on behalf of Alessandra Forti <[log in to unmask]><mailto:[log in to unmask]>
Sent: 26 February 2020 08:15
To: [log in to unmask]<mailto:[log in to unmask]>
Subject: Frontier squids in the UK

Hi,

it seems there is a problem with the UK sites WNs accessing frontier
squid. In particular Manchester, RAL, RALPP and Oxford the others tooo
but they are behind a NAT and not as many nodes appear in the list.

https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwlcg-squid-monitor.cern.ch%2Fawstats%2Fbin%2Fawstats.pl%3Fconfig%3Dcvmfsbproxy.cern.ch%26databasebreak%3Dday%26day%3D25&amp;data=02%7C01%7C%7Cebdecf99190248c3600a08d7ba941a8f%7C569df091b01340e386eebd9cb9e25814%7C0%7C0%7C637183017572955219&amp;sdata=tri9QzQT0sN%2BJAO1jlmf0X21X3yQ%2Bo%2BTdTnD%2BEVcZhM%3D&amp;reserved=0<https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwlcg-squid-monitor.cern.ch%2Fawstats%2Fbin%2Fawstats.pl%3Fconfig%3Dcvmfsbproxy.cern.ch%26databasebreak%3Dday%26day%3D25&data=02%7C01%7C%7Cd4aa7154bfeb4660e3db08d7ba9bdc21%7C569df091b01340e386eebd9cb9e25814%7C0%7C0%7C637183050884940544&sdata=uUXYXuha6GUG8cn%2FEs2a0Gyx13tQbAKvkG86U%2Fe9oy0%3D&reserved=0>

you need to filter for ac.uk

Can I ask what version of squid you are using and if you are using
frontier squid or standard OS squid?

Frontier changed the baseline from 3.5.28 to 4.10.1-1 two weeks ago
which is not even available on centos8.

thanks

cheers
alessandra

--
Inference: a conclusion reached on the basis of evidence and reasoning
Respect is a rational process. \\//
For Ur-Fascism, disagreement is treason. (U. Eco)

########################################################################

To unsubscribe from the TB-SUPPORT list, click the following link:
https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.jiscmail.ac.uk%2Fcgi-bin%2Fwebadmin%3FSUBED1%3DTB-SUPPORT%26A%3D1&amp;data=02%7C01%7C%7Cebdecf99190248c3600a08d7ba941a8f%7C569df091b01340e386eebd9cb9e25814%7C0%7C0%7C637183017572955219&amp;sdata=F6MgNpBRfiP3GYbzAm09jAy8NP5OJyhbjGR2Qw5Kv9Q%3D&amp;reserved=0<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.jiscmail.ac.uk%2Fcgi-bin%2Fwebadmin%3FSUBED1%3DTB-SUPPORT%26A%3D1&data=02%7C01%7C%7Cd4aa7154bfeb4660e3db08d7ba9bdc21%7C569df091b01340e386eebd9cb9e25814%7C0%7C0%7C637183050884940544&sdata=QHC2ESn4JQC07OiWx7%2BRb%2F1J3EyoC5dWb6ZHJq3tEQ8%3D&reserved=0>

########################################################################

To unsubscribe from the TB-SUPPORT list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=TB-SUPPORT&A=1<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.jiscmail.ac.uk%2Fcgi-bin%2Fwebadmin%3FSUBED1%3DTB-SUPPORT%26A%3D1&data=02%7C01%7C%7Cd4aa7154bfeb4660e3db08d7ba9bdc21%7C569df091b01340e386eebd9cb9e25814%7C0%7C0%7C637183050884950541&sdata=iOac5TL90beyHsfXcJ39y5Lbc6aNFCHULbN7QTjQ%2BHA%3D&reserved=0>



--
Inference: a conclusion reached on the basis of evidence and reasoning
Respect is a rational process. \\//
For Ur-Fascism, disagreement is treason. (U. Eco)


________________________________

To unsubscribe from the TB-SUPPORT list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=TB-SUPPORT&A=1<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.jiscmail.ac.uk%2Fcgi-bin%2Fwebadmin%3FSUBED1%3DTB-SUPPORT%26A%3D1&data=02%7C01%7C%7Cd4aa7154bfeb4660e3db08d7ba9bdc21%7C569df091b01340e386eebd9cb9e25814%7C0%7C0%7C637183050884960536&sdata=dhNIv9Y9I%2BEXNTlE3B%2Fi%2FYvbzOi7EnhVHFjN%2BJnwtNA%3D&reserved=0>

########################################################################

To unsubscribe from the TB-SUPPORT list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=TB-SUPPORT&A=1


-- 
Inference: a conclusion reached on the basis of evidence and reasoning
Respect is a rational process. \\//
For Ur-Fascism, disagreement is treason. (U. Eco)


To unsubscribe from the TB-SUPPORT list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=TB-SUPPORT&A=1


-- 
Inference: a conclusion reached on the basis of evidence and reasoning
Respect is a rational process. \\//
For Ur-Fascism, disagreement is treason. (U. Eco)


To unsubscribe from the TB-SUPPORT list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=TB-SUPPORT&A=1