Hello,
On 08/31/2012 09:31 AM, Alvise Dorigo wrote:
> Hi Matt,
> could you please execute this query in the WMS node:
>
> sqlite3 /var/ice/persist_dir/ice.db "select userdn from event_id where userdn like '%kashif%'" ?
Please find it attached.
Many thanks to the WMS admins at Imperial for sending it to me before I
even had a chance to ask.
If (as it is looking) there is no "simple" a fix then I will bring
forward the reinstall of this CE by a few weeks. This has already been a
lot of work for a lot of people to sooth a dying server!
Thanks again to everyone for the help.
Matt
>
> thanks,
>
> Alvise
>
> On Aug 30, 2012, at 5:06 PM, Matt Doidge wrote:
>
>>> CREAM DB ID (if the data given is in UTC) is 1288278171
>>> ICE DB ID (already saved in UTC) is 1288274571 as reported by Matt.
>>>
>>> ICE is late of exactly 3600 seconds (1 hr).
>>>
>>> If, and only if they save the timestamp in CREAM's DB in UTC (Eric ?) then the two DB ID differ. And this is not good. Then continue to read, otherwise something bad happend on the CE to the EventID.
>>>
>>> After first ICE installation (DB created from scratch), ICE worked a while with that CE and received, in total, X events (later we will calculate X); at the very first event ICE set the DB ID to 1288274571 (CREAM tells ICE this number as parameter of the QueryEvent response); and at a certain point (1 hr later) CREAM has been scratched (DB ID changed from timestamp 1288274571 to date 28 Oct 2010, 15:02:51 (I repeat if the reported CE's DB ID is not UTC all this discussion is pointless). ICE for some reason (bug maybe) didn't detect the DBID change.
>>>
>>> now consider the big gap of eventID (the number X mentioned before): last Event of ICE for Kashif (logged in ICE's log files) is about 36M, while last event in CREAM is about 3M (user kashif, as reported by Matt). It is quite impossible that in that short time (1hr) ICE managed about 33M events (~6.5M jobs!).
>>>
>>> Then I would exclude that CREAM has been scratched (exactly 3600 seconds later first installation, that is probably a localization issue that I did not take in consideration when comparing IDs) and then I think (90% sure) that something/someone changed or reset the event id table in the CREAM database ("something" could be even a bug... also consider that the CE is very old, 2010-10).
>>>
>>> I do not have any better idea about that 33M events lost in the CREAM CE.
>>>
>>> does my discussion make any sense ? I could miss some detail...
>>
>> The cream CE is currently using British Summer Time (UTC + 1), but I'm unsure what the database is logging timestamps in. When the CREAM CE crashed last week it was in it's bad state for ~16 hours. It could be that the original cause of the crash was the same thing that caused the reset of the event id table, but as it left no evidence in the logs I can't be sure.
>>
>> Do you have any advice on how to proceed? Or would it be easier to step up our reinstallation of this machine?
>>
>> Thanks,
>> Matt
>>>
>>> Alvise
>>>
>>> On Aug 30, 2012, at 3:36 PM, Matt Doidge wrote:
>>>
>>>> Hello,
>>>>
>>>>> news ?
>>>>>
>>>>> On Aug 30, 2012, at 2:08 PM, Alvise Dorigo wrote:
>>>>>
>>>>>> this seems a big mess...
>>>>>> I'm not yet sure about the DBID for abaddon registered in the ICE DB (I get it from the logs, but multiple threads are interleaved and I could mess different log lines).
>>>>>> Please guys, execute this query on the WMS node:
>>>>>>
>>>>>> sqlite3 /var/ice/persist_dir/ice.db "select db_id from ce_dbid where ceurl like '%abaddon%';"
>>>> The value the WMS admins had returned from that query was:
>>>>
>>>> 1288274571000
>>>>
>>>> Cheers,
>>>> Matt
>>>>
>>>>>>
>>>>>> Alvise
>>>>>>
>>>>>> ICE has a DBID for abaddon CE equal to
>>>>>> On Aug 30, 2012, at 1:54 PM, Matt Doidge wrote:
>>>>>>
>>>>>>> Hello,
>>>>>>>> What's the output of the following queries?
>>>>>>>
>>>>>>> mysql> select max(status.id) as eventId, j.userId from job j, job_status AS status LEFT OUTER JOIN job_status AS latest ON latest.jobId=status.jobId AND status.id< latest.id WHERE latest.id IS null and j.id=status.jobId and j.userId like '%kashif%' group by j.userId;
>>>>>>> +---------+----------------------------------------------------------------------------------------+
>>>>>>> | eventId | userId |
>>>>>>> +---------+----------------------------------------------------------------------------------------+
>>>>>>> | 3138440 | _C_UK_O_eScience_OU_Oxford_L_OeSC_CN_kashif_mohammad_ops_Role_lcgadmin_Capability_NULL |
>>>>>>> +---------+----------------------------------------------------------------------------------------+
>>>>>>> 1 row in set (0.00 sec)
>>>>>>>
>>>>>>> mysql> select creationTime from db_info;
>>>>>>> +---------------------+
>>>>>>> | creationTime |
>>>>>>> +---------------------+
>>>>>>> | 2010-10-28 15:02:51 |
>>>>>>> +---------------------+
>>>>>>> 1 row in set (0.00 sec)
>>>>>>>
>>>>>>> As problems started after the cream service crashed then it is likely that the cream database has got itself into a "bad" state.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Matt
>>>>>>>
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> E r i c.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>
>>>
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad-/ops/Role=lcgadmin/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad-/ops/Role=lcgadmin/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad-/ops/Role=pilot/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad-/vo.southgrid.ac.uk/Role=NULL/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad-/vo.southgrid.ac.uk/Role=NULL/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad/CN=Robot:GridClient-/ops/Role=lcgadmin/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad/CN=Robot:GridClient-/ops/Role=lcgadmin/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad/CN=Robot:GridClient-/ops/Role=lcgadmin/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad/CN=Robot:GridClient-/ops/Role=lcgadmin/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad/CN=Robot:GridClient-/ops/Role=lcgadmin/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad/CN=Robot:GridClient-/ops/Role=lcgadmin/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad/CN=Robot:GridClient-/ops/Role=lcgadmin/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad/CN=Robot:GridClient-/ops/Role=lcgadmin/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad/CN=Robot:GridClient-/ops/Role=pilot/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad/CN=Robot:GridClient-/ops/Role=lcgadmin/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad/CN=Robot:GridClient-/ops/Role=lcgadmin/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad/CN=Robot:GridClient-/ops/Role=lcgadmin/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad/CN=Robot:GridClient-/ops/Role=pilot/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad/CN=Robot:GridClient-/ops/Role=lcgadmin/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad/CN=Robot:GridClient-/ops/Role=lcgadmin/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad/CN=Robot:GridClient-/ops/Role=lcgadmin/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad/CN=Robot:GridClient-/ops/Role=lcgadmin/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad/CN=Robot:GridClient-/ops/Role=lcgadmin/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad/CN=Robot:GridClient-/ops/Role=pilot/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad/CN=Robot:GridClient-/ops/Role=lcgadmin/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad/CN=Robot:GridClient-/ops/Role=lcgadmin/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad/CN=Robot:GridClient-/ops/Role=pilot/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad/CN=Robot:GridClient-/ops/Role=pilot/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad/CN=Robot:GridClient-/ops/Role=lcgadmin/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad/CN=Robot:GridClient-/ops/Role=lcgadmin/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad/CN=Robot:GridClient-/ops/Role=lcgadmin/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad/CN=Robot:GridClient-/ops/Role=lcgadmin/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad/CN=Robot:GridClient-/ops/Role=lcgadmin/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad/CN=Robot:GridClient-/ops/Role=pilot/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad/CN=Robot:GridClient-/ops/Role=lcgadmin/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad/CN=Robot:GridClient-/ops/Role=lcgadmin/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad/CN=Robot:GridClient-/ops/Role=pilot/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad/CN=Robot:GridClient-/ops/Role=lcgadmin/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad/CN=Robot:GridClient-/ops/Role=lcgadmin/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad/CN=Robot:GridClient-/ops/Role=pilot/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad/CN=Robot:GridClient-/ops/Role=lcgadmin/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad/CN=Robot:GridClient-/ops/Role=lcgadmin/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad/CN=Robot:GridClient-/ops/Role=lcgadmin/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad/CN=Robot:GridClient-/ops/Role=pilot/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad/CN=Robot:GridClient-/ops/Role=lcgadmin/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad/CN=Robot:GridClient-/ops/Role=lcgadmin/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad/CN=Robot:GridClient-/ops/Role=lcgadmin/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad/CN=Robot:GridClient-/ops/Role=pilot/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad/CN=Robot:GridClient-/ops/Role=lcgadmin/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad/CN=Robot:GridClient-/ops/Role=pilot/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad/CN=Robot:GridClient-/ops/Role=pilot/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad/CN=Robot:GridClient-/ops/Role=lcgadmin/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad/CN=Robot:GridClient-/ops/Role=pilot/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad/CN=Robot:GridClient-/ops/Role=pilot/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad/CN=Robot:GridClient-/ops/Role=lcgadmin/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad/CN=Robot:GridClient-/ops/Role=lcgadmin/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad/CN=Robot:GridClient-/ops/Role=pilot/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad/CN=Robot:GridClient-/ops/Role=lcgadmin/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad/CN=Robot:GridClient-/ops/Role=lcgadmin/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad/CN=Robot:GridClient-/ops/Role=lcgadmin/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad/CN=Robot:GridClient-/ops/Role=lcgadmin/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad/CN=Robot:GridClient-/ops/Role=lcgadmin/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad/CN=Robot:GridClient-/ops/Role=pilot/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad/CN=Robot:GridClient-/ops/Role=lcgadmin/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad/CN=Robot:GridClient-/ops/Role=pilot/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad/CN=Robot:GridClient-/ops/Role=lcgadmin/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad/CN=Robot:GridClient-/ops/Role=pilot/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad/CN=Robot:GridClient-/ops/Role=lcgadmin/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad/CN=Robot:GridClient-/ops/Role=pilot/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad/CN=Robot:GridClient-/ops/Role=pilot/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad/CN=Robot:GridClient-/ops/Role=pilot/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad/CN=Robot:GridClient-/ops/Role=pilot/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad/CN=Robot:GridClient-/ops/Role=lcgadmin/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad/CN=Robot:GridClient-/ops/Role=pilot/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad/CN=Robot:GridClient-/ops/Role=pilot/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad-/ops/Role=lcgadmin/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad-/ops/Role=lcgadmin/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad-/ops/Role=pilot/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad-/ops/Role=pilot/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad-/ops/Role=lcgadmin/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad-/ops/Role=lcgadmin/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad-/ops/Role=lcgadmin/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad-/ops/Role=lcgadmin/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad-/ops/Role=lcgadmin/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad-/ops/Role=lcgadmin/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad-/ops/Role=lcgadmin/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad-/ops/Role=lcgadmin/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad-/ops/Role=pilot/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad-/ops/Role=pilot/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad-/ops/Role=lcgadmin/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad-/ops/Role=pilot/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad-/ops/Role=lcgadmin/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad-/ops/Role=lcgadmin/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad-/ops/Role=lcgadmin/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad-/ops/Role=lcgadmin/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad-/ops/Role=lcgadmin/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad-/ops/Role=lcgadmin/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad-/ops/Role=lcgadmin/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad-/ops/Role=lcgadmin/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad-/ops/Role=lcgadmin/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad-/ops/Role=lcgadmin/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad-/ops/Role=lcgadmin/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad-/ops/Role=lcgadmin/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad-/ops/Role=lcgadmin/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad-/ops/Role=pilot/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad-/ops/Role=lcgadmin/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad-/ops/Role=pilot/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad-/ops/Role=pilot/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad-/ops/Role=pilot/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad-/ops/Role=pilot/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad-/ops/Role=lcgadmin/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad-/ops/Role=lcgadmin/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad-/ops/Role=lcgadmin/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad-/ops/Role=pilot/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad-/ops/Role=pilot/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad-/ops/Role=lcgadmin/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad-/ops/Role=lcgadmin/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad-/ops/Role=lcgadmin/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad-/ops/Role=lcgadmin/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad-/ops/Role=lcgadmin/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad-/ops/Role=lcgadmin/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad-/ops/Role=lcgadmin/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad-/ops/Role=lcgadmin/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad-/ops/Role=pilot/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad-/ops/Role=pilot/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad-/ops/Role=pilot/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad-/ops/Role=pilot/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad-/ops/Role=lcgadmin/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad-/ops/Role=lcgadmin/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad-/ops/Role=lcgadmin/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad-/ops/Role=lcgadmin/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad-/ops/Role=lcgadmin/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad-/ops/Role=lcgadmin/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad-/ops/Role=pilot/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad-/ops/Role=pilot/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad-/ops/Role=pilot/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad-/ops/Role=pilot/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad-/ops/Role=lcgadmin/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad-/ops/Role=lcgadmin/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad-/ops/Role=lcgadmin/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad-/ops/Role=lcgadmin/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad-/ops/Role=lcgadmin/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad-/ops/Role=pilot/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad-/ops/Role=pilot/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad-/ops/Role=pilot/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad-/ops/Role=pilot/Capability=NULL
/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad-/ops/Role=pilot/Capability=NULL
|