Mystery solved!
The script I was using to allocate spacetokens totalled 116% of the space
available. It resized the first spacetokens but then failed to resize the
last ones. They kept their (over allocated) original sizes, leading to
these accounting difficulties.
This problem was fixed by resizing one token to something much smaller
causing the other spacetokens to take normal capacities. They were then
resized to allow for the first spacetoken to be resized up to something
more sensible.
Thanks for your help,
Chris
On Fri, 14 May 2010, Wahid Bhimji wrote:
>
> Hi,
>
> So basically if you do dpm-listspaces and add up the values next to Capacity you get a different number to what dpm-qryconf says next to reserved?
> Did it used to be different - is there a particular token which is now less - or are they all reduced.
>
> What happens if you try and resize a token to use up more of the space eg.:
> /opt/lcg/bin/dpm-updatespace --gspace 1500G --lifetime Inf --token_desc CMS_DEFAULT
>
> Perhaps you have a secret token that has the same name (token_desc) but a different id.
> Try doing
> dpm-getspacetokens --token_desc ATLASDATADISK
> etc. for each of the spacetokens you have
> that may reveal something
>
> Cheers
>
> Wahid
>
> On 14 May 2010, at 13:05, Chris Curtis wrote:
>
>>
>> Hi -
>>
>> All of the file systems in the pool appear to be ok (see below).
>>
>> Chris
>>
>> -------------------------------------------------------------------
>> POOL epgsr-f13 DEFSIZE 200.00M GC_START_THRESH 0 GC_STOP_THRESH 0 DEF_LIFETIME 7.0d DEFPINTIME 2.0h MAX_LIFETIME 1.0m MAXPINTIME 12.0h FSS_POLICY maxfreespace GC_POLICY lru RS_POLICY fifo GIDS 1307,17062 S_TYPE - MIG_POLICY none RET_POLICY R
>> CAPACITY 100.03T FREE 0 ( 0.0%)
>> epgsr1.ph.bham.ac.uk /disk/f12a CAPACITY 4.55T FREE 3.81T ( 83.7%)
>> epgsr1.ph.bham.ac.uk /disk/f12b CAPACITY 4.55T FREE 3.80T ( 83.6%)
>> epgsr1.ph.bham.ac.uk /disk/f12c CAPACITY 4.55T FREE 3.80T ( 83.7%)
>> epgsr1.ph.bham.ac.uk /disk/f12d CAPACITY 4.55T FREE 3.85T ( 84.8%)
>> epgsr1.ph.bham.ac.uk /disk/f13a CAPACITY 4.55T FREE 3.45T ( 75.9%)
>> epgsr1.ph.bham.ac.uk /disk/f13b CAPACITY 4.55T FREE 3.40T ( 74.9%)
>> epgsr1.ph.bham.ac.uk /disk/f13c CAPACITY 4.55T FREE 3.44T ( 75.7%)
>> epgsr1.ph.bham.ac.uk /disk/f13d CAPACITY 4.55T FREE 3.42T ( 75.2%)
>> epgsr1.ph.bham.ac.uk /disk/f14a CAPACITY 4.55T FREE 4.37T ( 96.1%)
>> epgsr1.ph.bham.ac.uk /disk/f14b CAPACITY 4.55T FREE 4.37T ( 96.0%)
>> epgsr1.ph.bham.ac.uk /disk/f14c CAPACITY 4.55T FREE 4.36T ( 95.9%)
>> epgsr1.ph.bham.ac.uk /disk/f14d CAPACITY 4.55T FREE 4.37T ( 96.1%)
>> epgsr1.ph.bham.ac.uk /disk/f15a CAPACITY 4.55T FREE 3.79T ( 83.4%)
>> epgsr1.ph.bham.ac.uk /disk/f15b CAPACITY 4.55T FREE 3.80T ( 83.6%)
>> epgsr2.ph.bham.ac.uk /disk/f16a CAPACITY 4.55T FREE 4.21T ( 92.6%)
>> epgsr2.ph.bham.ac.uk /disk/f16b CAPACITY 4.55T FREE 4.22T ( 92.8%)
>> epgsr2.ph.bham.ac.uk /disk/f16c CAPACITY 4.55T FREE 4.22T ( 92.9%)
>> epgsr2.ph.bham.ac.uk /disk/f16d CAPACITY 4.55T FREE 4.25T ( 93.4%)
>> epgsr2.ph.bham.ac.uk /disk/f17a CAPACITY 4.55T FREE 4.22T ( 92.9%)
>> epgsr2.ph.bham.ac.uk /disk/f17b CAPACITY 4.55T FREE 4.22T ( 92.7%)
>> epgsr2.ph.bham.ac.uk /disk/f17c CAPACITY 4.55T FREE 4.22T ( 92.8%)
>> epgsr2.ph.bham.ac.uk /disk/f17d CAPACITY 4.55T FREE 4.20T ( 92.4%)
>> --------------------------------------------------------------------
>>
>> On Fri, 14 May 2010, Sam Skipsey wrote:
>>
>>> So, Chris, what does dpm-qryconf think about your disk servers?
>>> Are any of them DISABLED, RD_ONLY or otherwise not in a "0" state?
>>>
>>> Sam
>>>
>>> On 14 May 2010 11:00, Chris Curtis <[log in to unmask]> wrote:
>>>> Hi -
>>>>
>>>> I've lost about 20 TB from our DPM.
>>>>
>>>> We have defined a pool with a total capacity of 100.03TB. This pool was
>>>> operating as expected until very recently, when a gus ticket pointed out
>>>> that some of the ATLAS spacetokens had negative free space. If I total up
>>>> the amount of Capacity reported by the spacetokens belonging to the pool, I
>>>> get just over 77TB. Previously this matched the value of reserved space,
>>>> which is still reported as being just shy of 100 TB.
>>>>
>>>> I've tried rebooting, re-running yaim and re-adding the filesystems to the
>>>> pool, without success. I've also tried writing to all of the filesystems -
>>>> no problems reported so far.
>>>>
>>>> How can I find out more about the lost space?
>>>>
>>>> Cheers,
>>>>
>>>> Chris
>>>>
>>>> --
>>>> West 326
>>>> Physics and Astronomy
>>>> University of Birmingham
>>>> Edgbaston
>>>> Birmingham
>>>> B15 2TT
>>>>
>>>> (Office) 0121 414 4700
>>>> (Mobile) 0798 666 1959
>>>>
>>>
>>
>> --
>> West 326
>> Physics and Astronomy
>> University of Birmingham
>> Edgbaston
>> Birmingham
>> B15 2TT
>>
>> (Office) 0121 414 4700
>> (Mobile) 0798 666 1959
>
>
>
--
West 326
Physics and Astronomy
University of Birmingham
Edgbaston
Birmingham
B15 2TT
(Office) 0121 414 4700
(Mobile) 0798 666 1959
|