On 12 April 2011 10:19, Alessandra Forti <[log in to unmask]> wrote:
> Hi Sam,
>
>> Do you want me to package this up into a "list files with replicas> 2"
>> util for you?
>
> yes, since you have most of the code already.....
>
Here you go.
Invocation is
python dpm-sql-dpns-by-replication.py #
(or if you set it executable you can execute it as normal)
where # is the minimum number of replicas you want to consider reporting.
The output is a CR-separated list of
DPNS_entry <tab> #
where # is the number of replicas for that DPNS path, and the list is
sorted in descending order.
Sam
> thanks
>
> cheers
> alessandra
>
> On 12/04/2011 10:12, Sam Skipsey wrote:
>>
>> On 12 April 2011 09:59, Alessandra Forti<[log in to unmask]> wrote:
>>>
>>> Hi Wahid,
>>>
>>> it doesn't replicate when --del is used (is it supposed to work?). I
>>> rerun
>>> it without --del and with --verbose in this way it is making the copies.
>>> Still I don't think it'll be enough, yesterday I had 2 problems
>>>
>>> 1) the same file was reuested 5000 times 2 copies were not enough
>>> 2) reco jobs wreaked avoc with proddisk files
>>>
>>> Problem 2) means we need to integrate your tool with something that also
>>> looks at the hotfiles list of the day and makes copies of those too.
>>>
>>> It be nice to know which files have more than one replica independently
>>> from
>>> the space token. Do you know what mysql query or API one is supposed to
>>> use?
>>
>> select fileid, count(*) from cns_db.Cns_file_replica GROUP BY
>> cns_db.Cns_file_replica.fileid
>>
>> will give you a list of fileid s and how many replicas exist for them.
>>
>> You could then do something like:
>>
>> def sfn_lookup(cc, fileid):
>> namelist = ['']
>> try:
>> cc.execute('''
>> select parent_fileid, name from Cns_file_metadata where fileid = %s
>> '' % fileid)
>>
>> name = ''
>> parent_fileid = 0L
>> p = cc.fetchone()
>> #handle zero results here by raising exception!!!
>> (parent_fileid, name) = p #this gets the "head" of the
>> namei
>> namelist.append(str(name))
>> while parent_fileid> 1:
>> cc.execute('''select parent_fileid, name from
>> Cns_file_metadata where Cns_file_metadata.fileid = %s''' %
>> parent_fileid)
>> (parent_fileid, name) = cc.fetchone()
>> #the above fetchone() is a zero length tuple
>> in the case of no results, so that should except?
>> namelist.append(str(name))
>> except MySQLdb.Error, e:
>> sys.exit("Error %d: %s" % (e.args[0], e.args[1]))
>> #sys.exit (1)
>> namelist.reverse() #put entries in "right" order for joining
>> together
>> return '/'.join(namelist)[1:] #and print dpns name (minus srm
>> bits)
>>
>> to generate the dpns name for a particular fileid. (cc is a Cursor
>> object from a MySQLdb Connection object)
>>
>> Do you want me to package this up into a "list files with replicas>
>> 2" util for you?
>>
>> Sam
>>
>>> thanks
>>>
>>> cheers
>>> alessandra
>>>
>>>
>>> On 12/04/2011 09:52, Wahid Bhimji wrote:
>>>
>>> Hello
>>>
>>> I have version gridpp-dpm-tools-2.6.7.DPM174-1
>>>
>>> That's good.
>>>
>>> According to the output it only deleted 3 files because there were 3
>>> replicas instead of 2. When I run it again it did nothing even though
>>> most
>>> files have a single replica. I'm running without --del now.
>>>
>>> I am surprised it isn't working without the --del option.
>>> If you use the --list option does it say that there are only one replica
>>> for
>>> most of the files.
>>> There is a --verbose option that gives slightly more output which may
>>> shed
>>> some light.
>>> I presume you have DPM_HOST, DPNS_HOST set?
>>> Sorry you having troubles with it.
>>> Wahid
>>>
>>> cheers
>>> alessandra
>>>
>>> On 11/04/2011 17:28, Wahid Bhimji wrote:
>>>
>>> WAIT!
>>>
>>> This isn't really an answer to your question though and so likely to
>>> cause
>>> confusion (!) For your answer you can skip down but I would read the
>>> first
>>> bit as a warning.
>>>
>>> But I was just about to send a warning to the list about using this tool
>>> with the --del option (interesting timing of your email).
>>>
>>> Basically (I have mentioned this before but maybe not stressed its
>>> importance) there was a bug with the first version of my script that the
>>> replicas it created were not permanent.
>>>
>>> This means that if:
>>>
>>> 1) You used an old version of the tool to make replicas AND
>>>
>>> 2) Your run either old or new version with --del to reduce the number
>>> (e.g.
>>> from 3 to 2) AND
>>>
>>> 3) You run dpm-drain on a filesystem containing these replicas
>>>
>>> there is a risk of removing all the replicas.
>>>
>>> I want to write a tool to list if people have non-permanent replicas and
>>> to
>>> mark them all permanent - so will send another post when I have done
>>> that.
>>> But I just wanted to flag it here in case people are in this position.
>>>
>>> (If you do suspect you may have done this then let me know (and for
>>> atlashotdisk) I can run an integrity check and if there are missing files
>>> I
>>> can also mark them lost and they will be restored quite easily). To be
>>> clear
>>> though - you need to have done all 3 above otherwise there is no problem.
>>>
>>> So to answer you actual question !
>>>
>>> If you have never used the tool before and use the latest version (which
>>> is
>>> I think your actual question) then then:
>>>
>>> A) You do not need --del that just deletes from (e.g 3 to 2) so in your
>>> case
>>> it will see there is not more than 2 and do nothing.
>>>
>>> B) The first time it will take a long time and so I would run it first
>>> outside cron.
>>>
>>> C) Then once that is done it will only run on new files (those that do
>>> not
>>> have 2 replicas in ATLASHOTDISK) so you can put it in a daily cron.
>>>
>>> Let me know if this is not clear
>>>
>>> (and apologies to anyone caught out by the above bug. It is actually
>>> easily
>>> put right so do let me know if you are concerned and I will send out
>>> another
>>> post with more details).
>>>
>>> Cheers
>>>
>>> Wahid
>>>
>>>
>>> On 11 Apr 2011, at 17:04, Alessandra Forti wrote:
>>>
>>> Hi,
>>>
>>> I'm going to try this command
>>>
>>> dpm-sql-spacetoken-replicate-hotfiles --nreps=2 --st=ATLASHOTDISK --del
>>>
>>> I'm wondering how do I catch the new files though and how often can I run
>>> it. Is it really something I can put in a cron job? Otherwise I'll go
>>> with
>>> the normal list-hotfiles and replicate only those files that have>NNN
>>> requests unless they have been already replicated.
>>>
>>> cheers
>>>
>>> alessandra
>>>
>>>
>>>
>>>
>>>
>>> The University of Edinburgh is a charitable body, registered in
>>> Scotland, with registration number SC005336.
>>>
>>>
>
>
