Hi Maarten,
we got our broken ARC back working after brutally downgrading by
scopying all files in the updated packages [1] from our remaining
working ARC over to the broken one.
We will try to debug tomorrow what package actually breaks the ARC
(globus or openssl, condor??)
Cheers,
Thomas
[1]
globus-common-16.7-1.el6.x86_64
openssl-1.0.1e-48.sl6_8.3.x86_64
globus-xio-5.13-1.el6.x86_64
globus-gsi-openssl-error-3.6-1.el6.x86_64
globus-gsi-sysconfig-6.10-1.el6.x86_64
globus-gsi-proxy-ssl-5.9-1.el6.x86_64
globus-openssl-module-4.7-1.el6.x86_64
globus-gsi-cert-utils-9.14-1.el6.x86_64
globus-gsi-callback-5.10-1.el6.x86_64
globus-gsi-credential-7.10-1.el6.x86_64
globus-callout-3.15-1.el6.x86_64
condor-classads-8.4.9-1.el6.x86_64
globus-gsi-proxy-core-8.4-1.el6.x86_64
globus-gssapi-gsi-12.5-2.el6.x86_64
globus-gss-assist-10.19-1.el6.x86_64
globus-gssapi-error-5.5-1.el6.x86_64
globus-io-11.7-1.el6.x86_64
globus-ftp-control-7.4-1.el6.x86_64
globus-xio-popen-driver-3.6-1.el6.x86_64
globus-xio-udt-driver-1.24-1.el6.x86_64
globus-xio-pipe-driver-3.9-1.el6.x86_64
globus-gridftp-server-control-4.2-1.el6.x86_64
globus-gfork-4.9-1.el6.x86_64
globus-usage-4.5-1.el6.x86_64
condor-external-libs-8.4.9-1.el6.x86_64
condor-procd-8.4.9-1.el6.x86_64
condor-8.4.9-1.el6.x86_64
globus-authz-callout-error-3.6-1.el6.x86_64
globus-authz-3.14-1.el6.x86_64
dcap-libs-2.47.10-4.el6.x86_64
dcap-tunnel-gsi-2.47.10-4.el6.x86_64
globus-gridftp-server-11.3-2.el6.x86_64
condor-python-8.4.9-1.el6.x86_64
globus-ftp-client-8.32-1.el6.x86_64
openssl-devel-1.0.1e-48.sl6_8.3.x86_64
On 2016-10-05 18:37, Maarten Litmaath wrote:
> Hallo Thomas,
> it appears the proxy on the WN has some problem that is
> not affecting the C version of Globus GSI, while the Java version
> does not like it. We have occasionally seen similar issues in the past,
> for example legacy proxies delegated by CREAM to the WN lack(ed?)
> a flag that only causes problems for some operations.
>
> Could you capture such a half-broken proxy for further analysis?
> It may be best to open a ticket for MW security, but do not attach
> the proxy there! In the ticket we can discuss how to make it available
> to the experts.
>
> ________________________________________
> From: LHC Computer Grid - Rollout [[log in to unmask]] on behalf of Thomas Hartmann [[log in to unmask]]
> Sent: 05 October 2016 18:21
> To: [log in to unmask]
> Subject: Re: [LCG-ROLLOUT] srm-tools problems for jobs after updating globus-gsi packages?
>
> Hi Maarten,
>
> yes, for the srm-tools we also realized that everything should stay in
> Java/BouncyCastle(?) and we decided to postpone the issue since
> everything seemed to work as usual.
>
> However, later on we triggered a yum updated on one of our ARC CEs and
> run into new trouble with submissions starting to fail.
> Unfortunately, downgrades seem not to be possible for globus packages [1]
>
> Maybe it is a red herring, but at the moment we are suspecting the
> recent globus update to be the culprit - but we have no idea, if/how it
> would have triggered the srm-tool problems in the beginning (ssl/tls
> versions/handshakes??)
>
> Cheers,
> Thomas
>
> [1]
>> grep "Oct 05 15" /var/log/yum.log | grep globus | cut -d " " -f 5
> globus-common-16.7-1.el6.x86_64
> globus-xio-5.13-1.el6.x86_64
> globus-gsi-openssl-error-3.6-1.el6.x86_64
> globus-gsi-sysconfig-6.10-1.el6.x86_64
> globus-gsi-proxy-ssl-5.9-1.el6.x86_64
> globus-openssl-module-4.7-1.el6.x86_64
> globus-gsi-cert-utils-9.14-1.el6.x86_64
> globus-gsi-callback-5.10-1.el6.x86_64
> globus-gsi-credential-7.10-1.el6.x86_64
> globus-callout-3.15-1.el6.x86_64
> globus-gsi-proxy-core-8.4-1.el6.x86_64
> globus-gssapi-gsi-12.5-2.el6.x86_64
> globus-gss-assist-10.19-1.el6.x86_64
> globus-gssapi-error-5.5-1.el6.x86_64
> globus-io-11.7-1.el6.x86_64
> globus-ftp-control-7.4-1.el6.x86_64
> globus-xio-popen-driver-3.6-1.el6.x86_64
> globus-xio-udt-driver-1.24-1.el6.x86_64
> globus-xio-pipe-driver-3.9-1.el6.x86_64
> globus-gridftp-server-control-4.2-1.el6.x86_64
> globus-gfork-4.9-1.el6.x86_64
> globus-usage-4.5-1.el6.x86_64
> globus-authz-callout-error-3.6-1.el6.x86_64
> globus-authz-3.14-1.el6.x86_64
> globus-gridftp-server-11.3-2.el6.x86_64
> globus-ftp-client-8.32-1.el6.x86_64
>
>
> On 2016-10-05 17:36, Maarten Litmaath wrote:
>> Hallo Thomas,
>> AFAIK the "srm" tools use the Java version of Globus,
>> not the packages that you reported below?
>>
>> Easy to check: if you downgrade _only_ those packages,
>> does the problem persist or go away?
>>
>> If the latter, try upgrading them again and check once more.
>>
>> My suspicion would be that something else changed as well
>> and actually caused the failures you reported...
>>
>> ________________________________________
>> From: LHC Computer Grid - Rollout [[log in to unmask]] on behalf of Thomas Hartmann [[log in to unmask]]
>> Sent: 05 October 2016 13:28
>> To: [log in to unmask]
>> Subject: [LCG-ROLLOUT] srm-tools problems for jobs after updating globus-gsi packages?
>>
>> Hi all,
>>
>> since yesterday srm-tools (as srmls/srm -ls) are failing in our local
>> tests [1]. Apparently, the start of the problem coincides with updates
>> of globus packages [2] (other packages as Java etc. have not been updated).
>>
>> As far as we see, other grid tools (gfal2,...) are not affected but only
>> srm-tools.
>>
>> We have not been able to reproduce the behaviour directly on nodes using
>> legacy/rfc proxies. So, the problem only occures when submitting jobs
>> via our ARC CE into the HTCondor LRMS.
>>
>> Has somebody else noticed similar behaviour or maybe has an idea, how to
>> pinpoint the issue?
>>
>> Cheers and thanks for ideas,
>> Andreas and Thomas
>>
>>
>> [1]
>> Couldn't getGssCredential.
>> srm client error:
>> org.globus.gsi.CredentialException: no key
>>
>> [2]
>> Oct 04 05:45:43 Updated: globus-common-16.7-1.el6.x86_64
>> Oct 04 05:45:43 Updated: globus-gsi-openssl-error-3.6-1.el6.x86_64
>> Oct 04 05:45:44 Updated: globus-gsi-sysconfig-6.10-1.el6.x86_64
>> Oct 04 05:45:44 Updated: globus-gsi-proxy-ssl-5.9-1.el6.x86_64
>> Oct 04 05:45:44 Updated: globus-openssl-module-4.7-1.el6.x86_64
>> Oct 04 05:45:45 Updated: globus-gsi-cert-utils-9.14-1.el6.x86_64
>> Oct 04 05:45:45 Updated: globus-gsi-callback-5.10-1.el6.x86_64
>> Oct 04 05:45:46 Updated: globus-gsi-credential-7.10-1.el6.x86_64
>> Oct 04 05:45:46 Updated: globus-gsi-proxy-core-8.4-1.el6.x86_64
>> Oct 04 05:45:47 Updated: globus-gssapi-gsi-12.5-2.el6.x86_64
>> Oct 04 05:45:47 Updated: globus-callout-3.15-1.el6.x86_64
>> Oct 04 05:45:48 Updated: globus-gss-assist-10.19-1.el6.x86_64
>>
>
|