The rucio setup change was on the 11th. This latest job is using the
most recent wrapper so doesn't help with this problem. I guess DPM
experts need to take a closer look or crank-up the debug logging on
xrootd+argus.

Cheers,
Peter

On 20 July 2016 at 17:37, Andrew Washbrook <[log in to unmask]> wrote:
> Hi Peter,
>
> The last one I can find is this one:
> http://bigpanda.cern.ch/job?pandaid=2928394785
> which ran over 24 hours ago now.
>
> Pretty much all our dpd-production jobs also failed with the same error:
> http://bigpanda.cern.ch/errors/?computingsite=UKI-SCOTGRID-ECDF_SL7&days=14&jobstatus=failed&piloterrorcode=1137&hours=336&display_limit=100&produsername=atlas-dpd-production
>
> When did the rucio change go in and do you have any details?
>
> I can pretty much reproduce this permission denied error now grabbing the proxies of these users and can provide full details if relevant. The production role is either being ignored or dropped when writing a file to DPM - so atlpil01 cannot write into a directory that atlpil02 owns (and vice versa).
>
> Thanks,
> Andy.
>
>> On 20 Jul 2016, at 11:56, Love, Peter <[log in to unmask]> wrote:
>>
>> Those are rather old jobs from last week and there has been a wrapper change in the meantime related to ddm/rucio setup. I'm not saying this is related but its worth checking more recent jobs:
>> http://bigpanda.cern.ch/jobs/?computingsite=UKI-SCOTGRID-ECDF_SL7&hours=12&display_limit=100
>>
>> The two types of errors are different, are you still see permission denied in xrootd logs?
>> Cheers,
>> Peter
>>
>>
>>
>>> On 20 Jul 2016, at 11:36, Andrew Washbrook <[log in to unmask]> wrote:
>>>
>>> Hi Alessandra,
>>>
>>>> On 20 Jul 2016, at 09:38, Alessandra Forti <[log in to unmask]> wrote:
>>>>
>>>> Hi,
>>>>
>>>> I'm adding atlas-adc-ddm-support I have vague recollection of wrong permissions at some sites recently they might have better memory than I do.
>>>>
>>>
>>> Ok thanks.
>>>
>>> To answer Peter’s question - here are some sample jobs:
>>>
>>> Not working: http://bigpanda.cern.ch/job?pandaid=2924067646
>>> Working: http://bigpanda.cern.ch/job?pandaid=2924020865
>>>
>>> These are the same type of the HC tests (template 621) to the same queue. It is not just HC jobs I see this - but I can make a clearer comparison with working and non working jobs.
>>>
>>> You'll see from the pilot output that the proxy information is the same:
>>>
>>> subject   : /DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=atlpilo2/CN=531497/CN=Robot: ATLAS Pilot2/CN=2077262095/CN=1616390467/CN=1208191811
>>> issuer    : /DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=atlpilo2/CN=531497/CN=Robot: ATLAS Pilot2/CN=2077262095/CN=1616390467
>>> identity  : /DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=atlpilo2/CN=531497/CN=Robot: ATLAS Pilot2/CN=2077262095/CN=1616390467
>>> type      : RFC compliant proxy
>>> strength  : 1024 bits
>>> path      : /local/7836133.1.eddie/6SaNDmSg4hon5zp09pXeGelnABFKDmABFKDmQ7KKDmABFKDmN3SSsn/user.proxy
>>> timeleft  : 158:34:02
>>> key usage : Digital Signature, Key Encipherment
>>> === VO atlas extension information ===
>>> VO        : atlas
>>> subject   : /DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=atlpilo2/CN=531497/CN=Robot: ATLAS Pilot2
>>> issuer    : /DC=ch/DC=cern/OU=computers/CN=lcg-voms2.cern.ch
>>> attribute : /atlas/Role=production/Capability=NULL
>>> attribute : /atlas/Role=NULL/Capability=NULL
>>> attribute : /atlas/lcg1/Role=NULL/Capability=NULL
>>> attribute : /atlas/usatlas/Role=NULL/Capability=NULL
>>> attribute : nickname = atlpilo2 (atlas)
>>> timeleft  : 86:34:02
>>> uri       : lcg-voms2.cern.ch:15001
>>>
>>> and that these get mapped to the to same local user (gl02pr1). The production role is seen by ARGUS (see earlier mail)
>>>
>>> As far as I can see from the logs and from checking dpns-getacl on file transfers in the logs:
>>> - atlpilot2 cannot write to directories owned by atlpilot1 that have a group: atlas/Role=production
>>> - In the cases that transfers are working the file itself has a group set as "atlas" and NOT "atlas/Role=production"
>>>
>>> Unless I am way off track this leads me to conclude that although there is a production role for the incoming pilot user it is not being interpreted properly somewhere along the chain. I checked myself with voms-proxy-info with the proxy on a sample worker node and it has the correct credentials - so something is wrong when writing to storage.
>>>
>>> Thanks,
>>> Andy.
>>>
>>>
>>>> On 19/07/2016 17:42, Elena Korolkova wrote:
>>>>> Hi Alessandra, Peter
>>>>>
>>>>> if this is  analysis job, should it write to scratch disk and not to the data disk?
>>>>>
>>>>> Thanks
>>>>> Elena
>>>>>
>>>>>
>>>>>
>>>>> On 19 Jul 2016, at 16:32, Andrew Washbrook <[log in to unmask]> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I am seeing an odd permission denied error on our storage which I think has something to do with ACLs and was wondering if anyone had seen this before.
>>>>>>
>>>>>> First we get the following error from Xrootd:
>>>>>>
>>>>>> 160715 10:47:54 9912 dpmfinder_Locate: Error during DoFileAccessRequest: [#00.000013] Need write access on /dpm/ecdf.ed.ac.uk/home/atlas/atlasdatadisk/rucio/hc_test/57/3a
>>>>>> 160715 10:47:54 9912 gl02pr1.114775:[log in to unmask] XrootdResponse: 0100 sending err 3010: Permission denied
>>>>>>
>>>>>> This is a HC test which only occasionally fails despite being mapped to the same local user each time. The file transfer can both and pass and fail on the same worker node and transferring to the same disk server. After much searching and many blind alleys I see the only pattern is in the ACLs of the parent directory:
>>>>>>
>>>>>> Pilot user is /DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=atlpilo2/CN=531497/CN=Robot: ATLAS Pilot2
>>>>>>
>>>>>> File copy to this directory fails:
>>>>>>
>>>>>> dpns-getacl /dpm/ecdf.ed.ac.uk/home/atlas/atlasdatadisk/rucio/hc_test/57/3a
>>>>>> # file: /dpm/ecdf.ed.ac.uk/home/atlas/atlasdatadisk/rucio/hc_test/57/3a
>>>>>> # owner: /DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=atlpilo1/CN=614260/CN=Robot: ATLAS Pilot1
>>>>>> # group: atlas/Role=production
>>>>>> user::rwx
>>>>>> group::rwx                #effective:rwx
>>>>>> other::r-x
>>>>>> default:user::rwx
>>>>>> default:group::rwx
>>>>>> default:other::r-x
>>>>>>
>>>>>> File copy to these directories completes:
>>>>>>
>>>>>> # file: /dpm/ecdf.ed.ac.uk/home/atlas/atlasdatadisk/rucio/hc_test/5d/50
>>>>>> # owner: /DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=atlpilo2/CN=531497/CN=Robot: ATLAS Pilot2
>>>>>> # group: atlas/Role=production
>>>>>> user::rwx
>>>>>> group::rwx                #effective:rwx
>>>>>> other::r-x
>>>>>> default:user::rwx
>>>>>> default:group::rwx
>>>>>> default:other::r-x
>>>>>>
>>>>>> # file: /dpm/ecdf.ed.ac.uk/home/atlas/atlasdatadisk/rucio/hc_test/4f/bc
>>>>>> # owner: /DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=atlpilo2/CN=531497/CN=Robot: ATLAS Pilot2
>>>>>> # group: atlas
>>>>>> user::rwx
>>>>>> group::rwx                #effective:rwx
>>>>>> other::r-x
>>>>>> default:user::rwx
>>>>>> default:group::rwx
>>>>>> default:other::r-x
>>>>>> ```
>>>>>>
>>>>>> I don't have too much experience with ACL setup in DPM. I read this as only the same owner and/or the same group (i.e VO role) can write to this directory. Is that correct?
>>>>>>
>>>>>> If so I would conclude that ATLAS Pilot2 does not have (or is not advertising) the role "atlas/Role=production" when writing to storage. How is the role checked and enforced by DPM/Xrootd? Our ARGUS server recognises the production role:
>>>>>>
>>>>>> 2016-07-15 13:47:09.223Z - INFO [DFPMObligationHandler] - ACCOUNTMAPPER_OH: DN: CN=Robot: ATLAS Pilot2,CN=531497,CN=atlpilo2,OU=Users,OU=Organic Units,DC=cern,DC=ch pFQAN: /atlas/Role=production FQANs: [/atlas/lcg1, /atlas, /atlas/Role=production, /atlas/usatlas] mapped to POSIX account: PosixAccount{user=gl02pr1 group=gl02pr groups=gl02us}
>>>>>>
>>>>>> For background, I seem to only get this error on our new cluster where DNs are mapped to a brand new set of local accounts. Does anything need to be configured on DPM head node or on storage nodes when new local accounts are added? I would have thought that this wouldn't matter.
>>>>>>
>>>>>> This problem is preventing us putting our new queue online so I need to fix this somehow.
>>>>>>
>>>>>> If you can also give me some troubleshooting tips that would be much appreciated!
>>>>>>
>>>>>> Thanks for your help,
>>>>>> Andy.
>>>>>>
>>>>>>
>>>>>> --
>>>>>> The University of Edinburgh is a charitable body, registered in
>>>>>> Scotland, with registration number SC005336.
>>>>
>>>> --
>>>> Respect is a rational process. \\//
>>>> Fatti non foste a viver come bruti (Dante)
>>>>
>>>>
>>>
>>>
>>> --
>>> The University of Edinburgh is a charitable body, registered in
>>> Scotland, with registration number SC005336.
>>>
>>
>>
>
>
> --
> The University of Edinburgh is a charitable body, registered in
> Scotland, with registration number SC005336.
>