I've updated Jens on what I'm guessing the situation is (note - that what I wrote below is my best understanding based on the evidence you guys have provided - so feel free to contradict if I've mis-understood something).
JK
> -----Original Message-----
> From: David Groep [mailto:[log in to unmask]]
> Sent: 28 November 2017 15:44
> To: Stephen Jones <[log in to unmask]>; Testbed Support for GridPP
> member institutes <[log in to unmask]>
> Cc: Grid MW security developers at NIKHEF <[log in to unmask]>;
> Jens Jensen <[log in to unmask]>; Kewley, John (STFC,DL,SC)
> <[log in to unmask]>; [log in to unmask]; [log in to unmask]; Joao
> Pina <[log in to unmask]>
> Subject: Re: We think trust anchors 1.88-1 breaks on SL6
>
> Hi Stephen, all,
>
> On 2017-11-28 16:29, Stephen Jones wrote:
> > I'll do some more research once everything is running.
>
> Also John Kewley did a summary now:
> > So, in simple terms, is the situation as follows:
> > * If client and server are using the same versions, all is well
> > * If client and server have different versions then some m/w [e.g.
> > VOMS and ARGUS] have issues
> > * In theory this should have worked OK, but maybe such a situation
> > (re-signing of the same keys/DN with diff signature alg) wasn't
> > foreseen by the underlying m/w in these cases (BC / Java maybe)
>
> which means that - given that now part of the Infra will have moved to 1.88,
> some are still on 1.87 - moving back now to the old
> SHA-1 version of the ICA will equally cause issues with sites that have already
> moved to 1.88. It's one of those big bang situations that was not foreseen for
> this parficular combination of Java/BC on this version.
>
> I propose that, since moving back now is as painful as moving forward, we
> might as well push forward vigurously and move everyone to 1.88 with the new
> SHA-256 UKeScienceCA-2B ICA.
>
> So for Joao: I propose that - unless Jens or the joint UK really wants to move
> back - we
> - STICK with release 1.88, including the SHA_256 UKeScienceCA-2B ICA
> - the UK SENDS NOTICE to the sites to upgrade vigorously to 1.88
> - if non-UK sites experience trouble in GGUS, we also send out an
> announcement to EGI for this
>
> This means that any 1.89 you may have seen will NOT be distributed, unless
> specifically notified otherwise.
>
> Is that OK?
>
> Cheers,
> DavidG.
>
>
> >
> > I've rolled back to previous version, 1.87-1, and the messages have
> > dried up, some clusters are working OK, about 80% back to capacity
> > here. Our storage seems to be online.
> >
> > (BTW: BouncyCastle always makes me shudder. Whenever it is mentioned,
> > I suffer the horrors of Java versionitis.)
> >
> > Cheers,
> >
> > Ste
> >
> >
> > On 28/11/17 15:04, David Groep wrote:
> >> Hi Stephen,
> >>
> >> On 2017-11-28 15:58, Stephen Jones wrote:
> >>> Did just ARGUS break, or diod other things? Daniela says her UI was
> broken.
> >>> WHat was the software and the version? What else is broken besides
> >>> ARGUS, UIs... I thing our DPM is failing too.
> >> It seems it affects Java/BouncyCastle on EL6, maybe with specific
> >> versions of BouncyCastle that are relatively old, and any software derived
> from it.
> >> This includes ARGUS and VOMS, and of course then affects services
> >> that talk to such services. If you have DPM linked to ARGUS, it will
> >> so fail.
> >
> >> One would have to check if the version of Argus is up to date and
> >> supports SHA-2, and for the VOMS issue if the server does not
> >> inadvertently sends an old version.
> >>
> >> # rpm -qa | grep -i argus
> >> argus-pap-1.6.2-1.el6.noarch
> >> argus-pepcli-2.2.0-1.el6.x86_64
> >> emi-argus-1.6.0-1.el6.noarch
> >> argus-pdp-1.6.0-1.el6.noarch
> >> yaim-argus_server-1.6.0-1.el6.noarch
> >> argus-pep-api-c-2.2.0-1.el6.x86_64
> >> argus-pep-server-1.6.1-1.el6.noarch
> >> argus-pdp-pep-common-1.4.0-2.el6.noarch
> >> argus-pep-common-2.3.0-1.el6.noarch
> >>
> >> I cannot test form non-UK locations for other combinations :(( If we
> >> cannot resolve this, Jens as the CA manager can trigger the next step
> >> (provided EGI ops is ready as well - I'm checking for emergency
> >> 'roll-forward' back-to-the-future right now with the old SHA-1 ICA)
> >>
> >> DavidG.
> >>
> >> PS:
> >> To get better resolution, please keep everyone in CC that can help.
> >> This thread is fragmenting uickly...
> >>
> >>
> >>
> >>> Cheers,
> >>>
> >>> Ste
> >>>
> >>>
> >>>
> >>> On 28/11/17 14:26, Matt Doidge wrote:
> >>>> Just an FYI that I've had great success rolling back thanks to the
> >>>> link Steve shared.
> >>>>
> >>>> Just in case it's useful, the dummy yum repo snippet I used was:
> >>>>
> >>>> [egi-igtf-187]
> >>>> name=egi-igtf-187
> >>>> baseurl=https://egi-igtf.ndpf.info/distribution/egi-1.87-1/ca-polic
> >>>> y-egi-core-1.87-1/
> >>>>
> >>>>
> >>>> enabled=0
> >>>> gpgcheck=1
> >>>> gpgkey=https://egi-igtf.ndpf.info/distribution/egi-1.87-1/GPG-KEY-E
> >>>> UGridPMA-RPM-3
> >>>>
> >>>>
> >>>>
> >>>> Cheers,
> >>>> Matt
> >>>>
> >>>> On 28/11/17 13:50, John Kewley wrote:
> >>>>> Just to let you know that I'm aware of the issue; I wasn't
> >>>>> involved in this release so wasn't involved in any testing, but
> >>>>> I'll see if I can work out the issue.
> >>>>>
> >>>>> My understanding is that Jens is out of the office, but I'm hoping
> >>>>> he'll be online at some point this afternoon.
> >>>>>
> >>>>> FYI, I haven't yet updated the CA repository, so the "old" 2B
> >>>>> certificate should still be downloadable from there:
> >>>>> http://www.ngs.ac.uk/ukca/certificates/cacerts
> >>>>>
> >>>>> Cheers
> >>>>>
> >>>>> JK
> >>>>>
> >>>>>> -----Original Message-----
> >>>>>> From: Testbed Support for GridPP member institutes [mailto:TB-
> >>>>>> [log in to unmask]] On Behalf Of Robert Frank
> >>>>>> Sent: 28 November 2017 13:45
> >>>>>> To: [log in to unmask]
> >>>>>> Subject: Re: We think trust anchors 1.88-1 breaks on SL6
> >>>>>>
> >>>>>> Have a look here:
> >>>>>>
> >>>>>> http://mirror.tier2.hep.manchester.ac.uk/Repositories/EMI/CA/
> >>>>>>
> >>>>>> Robert
> >>>>>>
> >>>>>> On 28/11/17 13:36, Stephen Jones wrote:
> >>>>>>> On 28/11/17 13:32, Daniela Bauer wrote:
> >>>>>>>> How did you roll back to 1.87 ?
> >>>>>>>
> >>>>>>> They've taken it away.
> >>>>>>>
> >>>>>>> (note to self: always download and KEEP the last good CAs)
> >>>>>>>
> >>>>>>> Ste
> >>>>>>>
> >>>>>>>
> >>>>>>>> Cheers,
> >>>>>>>> Daniela
> >>>>>>>>
> >>>>>>>> On 28 November 2017 at 13:30, Robert Frank
> >>>>>> <[log in to unmask]
> >>>>>> <mailto:[log in to unmask]>>
> >>>>>> wrote:
> >>>>>>>> I've seen it as well in Manchester when I tried to update this
> >>>>>>>> morning. I've rolled everything back to 1.87 for now.
> >>>>>>>> I got the impression that it works when both, the server and the
> >>>>>>>> client use the same version, but more testing is needed to
> confirm
> >>>>>>>> this.
> >>>>>>>>
> >>>>>>>> Cheers,
> >>>>>>>> Robert
> >>>>>>>>
> >>>>>>>> On 28/11/17 13:21, Stephen Jones wrote:
> >>>>>>>>
> >>>>>>>> Don't update to 1.88-1
> >>>>>>>>
> >>>>>>>> We have same problems too!
> >>>>>>>>
> >>>>>>>> Working on it; site is down because ARGUS (SL6) is clobbered
> >>>>>>>> by this...
> >>>>>>>>
> >>>>>>>> Cheers,
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Ste
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On 28/11/17 13:17, Daniela Bauer wrote:
> >>>>>>>>
> >>>>>>>> Hi All,
> >>>>>>>>
> >>>>>>>> the latest trust anchor release contains this chage:
> >>>>>>>>
> >>>>>>>> * updated UKeScience 2B ICA based on a SHA-2 family digest
> >>>>>>>> (UK)
> >>>>>>>>
> >>>>>>>> When I try and run the cvmfs UI on SL6 I get the following
> >>>>>>>> error:
> >>>>>>>>
> >>>>>>>> lx01:~ > voms-proxy-init --voms gridpp
> >>>>>>>> Enter GRID pass phrase for this identity:
> >>>>>>>> Contacting voms03.gridpp.ac.uk:15000
> >>>>>>>> <http://voms03.gridpp.ac.uk:15000>
> >>>>>>>> <http://voms03.gridpp.ac.uk:15000
> >>>>>>>> <http://voms03.gridpp.ac.uk:15000>>
> >>>>>>>>
> [/C=UK/O=eScience/OU=Imperial/L=Physics/CN=voms03.gridpp.ac.u
> >>>>>> k
> >>>>>>>> <http://voms03.gridpp.ac.uk> <http://voms03.gridpp.ac.uk>]
> >>>>>>>> "gridpp"...
> >>>>>>>> Certificate validation error: Can not verify the CRL as
> >>>>>>>> its issuer's public key is unknown or can not be validated
> >>>>>>>> Cause: Certification path could not be validated. Cause:
> >>>>>>>> NullPointerException
> >>>>>>>> Error contacting voms03.gridpp.ac.uk:15000
> >>>>>>>> <http://voms03.gridpp.ac.uk:15000>
> >>>>>>>> <http://voms03.gridpp.ac.uk:15000
> >>>>>>>> <http://voms03.gridpp.ac.uk:15000>> for VO gridpp:
> >>>>>>>> java.security.cert.CertificateException: The peer's
> >>>>>>>> certificate with subject's DN CN=voms03.gridpp.ac.uk
> >>>>>>>> <http://voms03.gridpp.ac.uk>
> >>>>>>>> <http://voms03.gridpp.ac.uk>,L=Physics,OU=Imperial,O=eScience,C
> >>>>>>>> =
> >>>>>> UK
> >>>>>>>> was rejected. The peer's certificate status is: FAILED The
> >>>>>>>> following validation errors were found:
> >>>>>>>> error at position 0 in chain, problematic certificate
> >>>>>>>> subject: CN=voms03.gridpp.ac.uk
> >>>>>>>> <http://voms03.gridpp.ac.uk>
> >>>>>>>> <http://voms03.gridpp.ac.uk>,L=Physics,OU=Imperial,O=eScience,C
> >>>>>>>> =
> >>>>>> UK
> >>>>>>>> (category: CRL): Can not verify the CRL as its issuer's
> >>>>>>>> public key is unknown or can not be validated Cause:
> >>>>>>>> Certification path could not be validated. Cause:
> >>>>>>>> NullPointerException
> >>>>>>>> Certificate validation error: Can not verify the CRL as
> >>>>>>>> its issuer's public key is unknown or can not be validated
> >>>>>>>> Cause: Certification path could not be validated. Cause:
> >>>>>>>> NullPointerException
> >>>>>>>> Error contacting voms03.gridpp.ac.uk:15000
> >>>>>>>> <http://voms03.gridpp.ac.uk:15000>
> >>>>>>>> <http://voms03.gridpp.ac.uk:15000
> >>>>>>>> <http://voms03.gridpp.ac.uk:15000>> for VO gridpp: peer
> >>>>>>>> not authenticated
> >>>>>>>> Error contacting voms03.gridpp.ac.uk:15000
> >>>>>>>> <http://voms03.gridpp.ac.uk:15000>
> >>>>>>>> <http://voms03.gridpp.ac.uk:15000
> >>>>>>>> <http://voms03.gridpp.ac.uk:15000>> for VO gridpp: REST
> >>>>>>>> and legacy VOMS endpoints failed.
> >>>>>>>> Contacting voms02.gridpp.ac.uk:15000
> >>>>>>>> <http://voms02.gridpp.ac.uk:15000>
> >>>>>>>> <http://voms02.gridpp.ac.uk:15000
> >>>>>>>> <http://voms02.gridpp.ac.uk:15000>>
> >>>>>>>> [/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=voms02.gridpp.ac.uk
> >>>>>>>> <http://voms02.gridpp.ac.uk>
> <http://voms02.gridpp.ac.uk>]
> >>>>>>>> "gridpp"...
> >>>>>>>> Certificate validation error: Can not verify the CRL as
> >>>>>>>> its issuer's public key is unknown or can not be validated
> >>>>>>>> Cause: Certification path could not be validated. Cause:
> >>>>>>>> NullPointerException
> >>>>>>>> Error contacting voms02.gridpp.ac.uk:15000
> >>>>>>>> <http://voms02.gridpp.ac.uk:15000>
> >>>>>>>> <http://voms02.gridpp.ac.uk:15000
> >>>>>>>> <http://voms02.gridpp.ac.uk:15000>> for VO gridpp:
> >>>>>>>> java.security.cert.CertificateException: The peer's
> >>>>>>>> certificate with subject's DN CN=voms02.gridpp.ac.uk
> >>>>>>>> <http://voms02.gridpp.ac.uk>
> >>>>>>>>
> <http://voms02.gridpp.ac.uk>,L=OeSC,OU=Oxford,O=eScience,C=UK
> >>>>>>>> was rejected. The peer's certificate status is: FAILED The
> >>>>>>>> following validation errors were found:
> >>>>>>>> error at position 0 in chain, problematic certificate
> >>>>>>>> subject: CN=voms02.gridpp.ac.uk
> >>>>>>>> <http://voms02.gridpp.ac.uk>
> >>>>>>>>
> <http://voms02.gridpp.ac.uk>,L=OeSC,OU=Oxford,O=eScience,C=UK
> >>>>>>>> (category: CRL): Can not verify the CRL as its issuer's
> >>>>>>>> public key is unknown or can not be validated Cause:
> >>>>>>>> Certification path could not be validated. Cause:
> >>>>>>>> NullPointerException
> >>>>>>>> Certificate validation error: Can not verify the CRL as
> >>>>>>>> its issuer's public key is unknown or can not be validated
> >>>>>>>> Cause: Certification path could not be validated. Cause:
> >>>>>>>> NullPointerException
> >>>>>>>> Error contacting voms02.gridpp.ac.uk:15000
> >>>>>>>> <http://voms02.gridpp.ac.uk:15000>
> >>>>>>>> <http://voms02.gridpp.ac.uk:15000
> >>>>>>>> <http://voms02.gridpp.ac.uk:15000>> for VO gridpp: peer
> >>>>>>>> not authenticated
> >>>>>>>> Error contacting voms02.gridpp.ac.uk:15000
> >>>>>>>> <http://voms02.gridpp.ac.uk:15000>
> >>>>>>>> <http://voms02.gridpp.ac.uk:15000
> >>>>>>>> <http://voms02.gridpp.ac.uk:15000>> for VO gridpp: REST
> >>>>>>>> and legacy VOMS endpoints failed.
> >>>>>>>> Contacting voms.gridpp.ac.uk:15000
> >>>>>>>> <http://voms.gridpp.ac.uk:15000>
> >>>>>>>> <http://voms.gridpp.ac.uk:15000
> >>>>>>>> <http://voms.gridpp.ac.uk:15000>>
> >>>>>>>>
> [/C=UK/O=eScience/OU=Manchester/L=HEP/CN=voms.gridpp.ac.uk
> >>>>>>>> <http://voms.gridpp.ac.uk> <http://voms.gridpp.ac.uk>]
> >>>>>>>> "gridpp"...
> >>>>>>>> Certificate validation error: Can not verify the CRL as
> >>>>>>>> its issuer's public key is unknown or can not be validated
> >>>>>>>> Cause: Certification path could not be validated. Cause:
> >>>>>>>> NullPointerException
> >>>>>>>> Error contacting voms.gridpp.ac.uk:15000
> >>>>>>>> <http://voms.gridpp.ac.uk:15000>
> >>>>>>>> <http://voms.gridpp.ac.uk:15000
> >>>>>>>> <http://voms.gridpp.ac.uk:15000>> for VO gridpp:
> >>>>>>>> java.security.cert.CertificateException: The peer's
> >>>>>>>> certificate with subject's DN CN=voms.gridpp.ac.uk
> >>>>>>>> <http://voms.gridpp.ac.uk>
> >>>>>>>>
> <http://voms.gridpp.ac.uk>,L=HEP,OU=Manchester,O=eScience,C=U
> >>>>>> K
> >>>>>>>> was rejected. The peer's certificate status is: FAILED The
> >>>>>>>> following validation errors were found:
> >>>>>>>> error at position 0 in chain, problematic certificate
> >>>>>>>> subject: CN=voms.gridpp.ac.uk
> >>>>>>>> <http://voms.gridpp.ac.uk>
> >>>>>>>>
> <http://voms.gridpp.ac.uk>,L=HEP,OU=Manchester,O=eScience,C=U
> >>>>>> K
> >>>>>>>> (category: CRL): Can not verify the CRL as its issuer's
> >>>>>>>> public key is unknown or can not be validated Cause:
> >>>>>>>> Certification path could not be validated. Cause:
> >>>>>>>> NullPointerException
> >>>>>>>> Certificate validation error: Can not verify the CRL as
> >>>>>>>> its issuer's public key is unknown or can not be validated
> >>>>>>>> Cause: Certification path could not be validated. Cause:
> >>>>>>>> NullPointerException
> >>>>>>>> Error contacting voms.gridpp.ac.uk:15000
> >>>>>>>> <http://voms.gridpp.ac.uk:15000>
> >>>>>>>> <http://voms.gridpp.ac.uk:15000
> >>>>>>>> <http://voms.gridpp.ac.uk:15000>> for VO gridpp: peer not
> >>>>>>>> authenticated
> >>>>>>>> Error contacting voms.gridpp.ac.uk:15000
> >>>>>>>> <http://voms.gridpp.ac.uk:15000>
> >>>>>>>> <http://voms.gridpp.ac.uk:15000
> >>>>>>>> <http://voms.gridpp.ac.uk:15000>> for VO gridpp: REST and
> >>>>>>>> legacy VOMS endpoints failed.
> >>>>>>>> None of the contacted servers for gridpp were capable of
> >>>>>>>> returning a valid AC for the user.
> >>>>>>>> User's request for VOMS attributes could not be fulfilled.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> It works on SL7.
> >>>>>>>>
> >>>>>>>> This error is fairly deadly for a lot of stuff we are
> >>>>>>>> doing here.
> >>>>>>>>
> >>>>>>>> Any ideas ?
> >>>>>>>>
> >>>>>>>> Regards,
> >>>>>>>> Daniela
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> -- Sent from the pit of despair
> >>>>>>>>
> >>>>>>>> -----------------------------------------------------------
> >>>>>>>> [log in to unmask]
> >>>>>>>> <mailto:[log in to unmask]>
> >>>>>>>> <mailto:[log in to unmask]
> >>>>>>>> <mailto:[log in to unmask]>>
> >>>>>>>> HEP Group/Physics Dep
> >>>>>>>> Imperial College
> >>>>>>>> London, SW7 2BW
> >>>>>>>> Tel: +44-(0)20-75947810 <tel:%2B44-%280%2920-75947810>
> >>>>>>>> http://www.hep.ph.ic.ac.uk/~dbauer/
> >>>>>>>> <http://www.hep.ph.ic.ac.uk/%7Edbauer/>
> >>>>>>>> <http://www.hep.ph.ic.ac.uk/%7Edbauer/
> >>>>>>>> <http://www.hep.ph.ic.ac.uk/%7Edbauer/>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> --
> >>>>>>>> Sent from the pit of despair
> >>>>>>>>
> >>>>>>>> -----------------------------------------------------------
> >>>>>>>> [log in to unmask]
> >>>>>>>> <mailto:[log in to unmask]>
> >>>>>>>> HEP Group/Physics Dep
> >>>>>>>> Imperial College
> >>>>>>>> London, SW7 2BW
> >>>>>>>> Tel: +44-(0)20-75947810
> >>>>>>>> http://www.hep.ph.ic.ac.uk/~dbauer/
> >>>>>> <http://www.hep.ph.ic.ac.uk/%7Edbauer/>
> >>>>>>>
> >>
> >
>
>
> --
> David Groep
>
> ** Nikhef, Dutch National Institute for Subatomic Physics, PDP(ACR) group **
> ** Room: H1.50 Phone: +31 20 5922179, PObox 41882, NL-1009DB Amsterdam
> NL **
> ** PGP: 0xD80134C2 308E076A FP:
> 2facebea12803ba145685a21d80134c2308e076a **
|