Things do indeed look a lot better now, things seem to have fixed
themselves (maybe the gremlins showed me some pity?). My gridmapdir is
still quite empty of mappings, so I'm not sure what's up with that,
but we're working so I'll take us out of downtime.
Thanks y'all,
Matt
On 16 September 2010 10:46, Wahid Bhimji <[log in to unmask]> wrote:
>
> Morning Matt -
> Did this go away?
> I don't see any errors in the nagios now.
> And I can copy a file to your se and list them
> I notice that the steve lloyd SE test can also copy / list etc.
>
> Unfortunately the SL sam test page seems to be showing the tests from yesterday so thats not useful.
>
> Wahid
>
> On 15 Sep 2010, at 23:25, Matt Doidge wrote:
>
>> Well we aren't working yet, but we've changed the error message to errors like:
>> ERROR: [SE][Ls][]
>> httpg://fal-pygrid-30.lancs.ac.uk:8446/srm/managerv2: CGSI-gSOAP
>> running on gridppnagios.physics.ox.ac.uk reports Error reading token
>> data header: Connection closed
>>
>> Cert and CA should be okay, so this is likely caused by my completely
>> unpopulated gridmapdir (no wierd % filled user names are appearing,
>> just a bunch of pool accounts that yaim shoved in there). Or it might
>> be a shift.conf problem if that's not the case.
>>
>> Also our nagios tests are having an error "CRITICAL: problem listing
>> Storage Path(s).", but looking at our BDII output and gstat things
>> look fine, so maybe this is a red herring of some kind.
>>
>> It's bedtime for me, Night Night all,
>> Matt
>>
>>
>> On 15 September 2010 18:40, Matt Doidge <[log in to unmask]> wrote:
>>> I think we might have got it, turns out all the 15-09-2010 directories
>>> on all our poolnodes were created using the bad dpmmgr UID. After a
>>> lot of chowning hopefully I hopefully have a working DPM once more.
>>>
>>> Thanks everyone!
>>> Matt
>>>
>>> On 15 September 2010 14:00, Matt Doidge <[log in to unmask]> wrote:
>>>> Hello Alessandra, Wahid (and everyone else),
>>>>
>>>> I usermod'ed dpmmgr to the old uid/gid and then went on a chowning
>>>> spree. Once I got everything that dpm touches owned by the right UID
>>>> everything worked, but things are still not happy. It looks still
>>>> looks like transfers are getting to the pools then dying. I can see in
>>>> the dpns logs entries being created then almost immediately destroyed.
>>>>
>>>> A snippet from the gridftp log on one of my pools:
>>>> [4102] Wed Sep 15 13:47:43 2010 :: lcgfts01.gridpp.rl.ac.uk:36749:
>>>> [CLIENT]: STOR /fal-pygrid-38.lancs.ac.uk:/thepark/atlas/2010-09-15/user.JamesCatmore.periodD1.t0pro04_v01.Muons.v03-pass1-2010D_RPCloose_muon.staco.EarlyOnia.EXT0._01877.ntuple.root__DQ2-1284553887.15188488.0
>>>> [4102] Wed Sep 15 13:47:43 2010 :: Finished transferring
>>>> "/fal-pygrid-38.lancs.ac.uk:/thepark/atlas/2010-09-15/user.JamesCatmore.periodD1.t0pro04_v01.Muons.v03-pass1-2010D_RPCloose_muon.staco.EarlyOnia.EXT0._01877.ntuple.root__DQ2-1284553887.15188488.0".
>>>> [4102] Wed Sep 15 13:47:43 2010 :: lcgfts01.gridpp.rl.ac.uk:36749:
>>>> [SERVER]: 500 Command failed. : open/create error: Permission denied
>>>> [4102] Wed Sep 15 13:47:43 2010 :: Closed connection from
>>>> lcgfts01.gridpp.rl.ac.uk:36749
>>>>
>>>> Not sure I've seen the phrase "open/create error" before. The
>>>> edg/lcg-dm grid map stuff on the head node and pools have a few
>>>> indescrepencies (which I'll need to sort out), but nothing huge and,
>>>> most importantly, the atlas and ops entries are identical. Another,
>>>> symptom, possibly a smoking gun, is that the gridmapdir on our
>>>> headnode has no mappings in it yet (despite people knocking at our
>>>> door), so this could be now just a boring authentication system
>>>> problem.
>>>>
>>>> Another thought I had was whether the changes might warrant a restart
>>>> of all the pool dpm services? It sounds like a wise thing I probably
>>>> should have done anyway.
>>>>
>>>> cheers,
>>>> Matt
>>>>
>>>>
>>>>
>>>> On 15 September 2010 11:37, Alessandra Forti <[log in to unmask]> wrote:
>>>>> We change the directory permissions on the pool nodes with a cron job. I
>>>>> think we tried usermod and didn't work. I have to look back at the tickets.
>>>>> For us the problem arose when we reinstalled the pool nodes. Everything will
>>>>> hopefully go back in sync when we finally upgrade the head node.
>>>>>
>>>>> cheers
>>>>> alessandra
>>>>>
>>>>>
>>>>> On 09/15/2010 11:07 AM, Matt Doidge wrote:
>>>>>>
>>>>>> That looks like it could be a winner, thanks! Did simply usermod-ing
>>>>>> the dpmmgr user on your headnode to the right UID fix things for you?
>>>>>>
>>>>>> Cheers,
>>>>>> Matt
>>>>>>
>>>>>> On 15 September 2010 10:53, Alessandra Forti<[log in to unmask]>
>>>>>> wrote:
>>>>>>>
>>>>>>> Do you have the same UID for dpmmgr on the head node and on the pools?
>>>>>>>
>>>>>>> In Manchester we had an UID and during the upgrade to SL5 of the pool
>>>>>>> nodes
>>>>>>> YAIM created the user with another by default. The pool nodes always go
>>>>>>> back
>>>>>>> to the head node for the UID so they have to match.
>>>>>>>
>>>>>>> cheers
>>>>>>> alessandra
>>>>>>>
>>>>>>> On 09/15/2010 10:46 AM, Matt Doidge wrote:
>>>>>>>>
>>>>>>>> Hello guys,
>>>>>>>> As mentioned in the meeting 5-minutes ago, we're having some trouble
>>>>>>>> after upgrading our DPM headnode to new hardware and SL5 (DPM
>>>>>>>> 1.7.4-7). On the surface things look good, after fixing a few gotchas
>>>>>>>> dpns seems to work, dpm-qryconf output looks good and things chug
>>>>>>>> along nicely. Except that transfers don't seem to work, at the moment
>>>>>>>> they all seem to be failing with permission denied type errors:
>>>>>>>>
>>>>>>>> From the FTS (atlas users):
>>>>>>>> ERROR MSG: [FTS] FTS State [Failed] FTS Retries [1] Reason [TRANSFER
>>>>>>>> error during TRANSFER phase: [SECURITY_ERROR] globu
>>>>>>>> s_ftp_client: the server responded with an error500 Command failed. :
>>>>>>>> open/create error: Permission denied]
>>>>>>>>
>>>>>>>> From lcg-cr (ops tests):
>>>>>>>> *snip*
>>>>>>>> Destination URL for copy:
>>>>>>>>
>>>>>>>>
>>>>>>>> gsiftp://fal-pygrid-46.lancs.ac.uk/fal-pygrid-46.lancs.ac.uk:/bobbin/ops/2010-09-15/file4f5afac2-e2a2-45ec-82a1-cbe0f0842a8e.151868.75.0
>>>>>>>> # streams: 1
>>>>>>>> 0 bytes 0.00 KB/sec avg 0.00 KB/sec inst 0 bytes 0.00 KB/sec avg 0.00
>>>>>>>> KB/sec instglobus_ftp_client: the server responded with an error
>>>>>>>> 500 Command failed. : open/create error: Permission denied
>>>>>>>>
>>>>>>>> At first I thought this may have been a problem with pool-headnode
>>>>>>>> communications (due to an oversight I had to produce the new
>>>>>>>> shift.conf largely by hand), but it's now looking to me like it could
>>>>>>>> be a user mapping problem, but on the surface everything looks fine.
>>>>>>>> Maybe I'm looking in the wrong place? Any suggestions would be greatly
>>>>>>>> appreciated.
>>>>>>>>
>>>>>>>> Thanks in advance,
>>>>>>>> Matt
>>>>>>>>
>>>>>>>> For the sake of thoroughness here's our new headnode shift.conf:
>>>>>>>> RFIOD TRUST fal-pygrid-30.lancs.ac.uk
>>>>>>>> RFIOD WTRUST fal-pygrid-30.lancs.ac.uk
>>>>>>>> RFIOD RTRUST fal-pygrid-30.lancs.ac.uk
>>>>>>>> RFIOD XTRUST fal-pygrid-30.lancs.ac.uk
>>>>>>>> RFIOD FTRUST fal-pygrid-30.lancs.ac.uk
>>>>>>>> DPM TRUST fal-pygrid-28.lancs.ac.uk fal-pygrid-35.lancs.ac.uk
>>>>>>>> fal-pygrid-37.lancs.ac.uk fal-pygrid-38.lancs.ac.uk
>>>>>>>> fal-pygrid-39.lancs.ac.uk fal-pygrid-40.lancs.ac.uk
>>>>>>>> fal-pygrid-41.lancs.ac.uk fal-pygrid-42.lancs.ac.uk
>>>>>>>> fal-pygrid-46.lancs.ac.uk fal-pygrid-47.lancs.ac.uk
>>>>>>>> fal-pygrid-48.lancs.ac.uk fal-pygrid-49.lancs.ac.uk
>>>>>>>> fal-pygrid-50.lancs.ac.uk fal-pygrid-51.lancs.ac.uk
>>>>>>>> fal-pygrid-52.lancs.ac.uk
>>>>>>>> DPNS TRUST fal-pygrid-28.lancs.ac.uk fal-pygrid-35.lancs.ac.uk
>>>>>>>> fal-pygrid-37.lancs.ac.uk fal-pygrid-38.lancs.ac.uk
>>>>>>>> fal-pygrid-39.lancs.ac.uk fal-pygrid-40.lancs.ac.uk
>>>>>>>> fal-pygrid-41.lancs.ac.uk fal-pygrid-42.lancs.ac.uk
>>>>>>>> fal-pygrid-46.lancs.ac.uk fal-pygrid-47.lancs.ac.uk
>>>>>>>> fal-pygrid-48.lancs.ac.uk fal-pygrid-49.lancs.ac.uk
>>>>>>>> fal-pygrid-50.lancs.ac.uk fal-pygrid-51.lancs.ac.uk
>>>>>>>> fal-pygrid-52.lancs.ac.uk
>>>>>>>> DPM PROTOCOLS rfio gsiftp
>>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
>
> --
> The University of Edinburgh is a charitable body, registered in
> Scotland, with registration number SC005336.
>
>
|
