Hi Sam,
> Do you want me to package this up into a "list files with replicas> 2" util for you?
yes, since you have most of the code already.....
thanks
cheers
alessandra
On 12/04/2011 10:12, Sam Skipsey wrote:
> On 12 April 2011 09:59, Alessandra Forti<[log in to unmask]> wrote:
>> Hi Wahid,
>>
>> it doesn't replicate when --del is used (is it supposed to work?). I rerun
>> it without --del and with --verbose in this way it is making the copies.
>> Still I don't think it'll be enough, yesterday I had 2 problems
>>
>> 1) the same file was reuested 5000 times 2 copies were not enough
>> 2) reco jobs wreaked avoc with proddisk files
>>
>> Problem 2) means we need to integrate your tool with something that also
>> looks at the hotfiles list of the day and makes copies of those too.
>>
>> It be nice to know which files have more than one replica independently from
>> the space token. Do you know what mysql query or API one is supposed to use?
> select fileid, count(*) from cns_db.Cns_file_replica GROUP BY
> cns_db.Cns_file_replica.fileid
>
> will give you a list of fileid s and how many replicas exist for them.
>
> You could then do something like:
>
> def sfn_lookup(cc, fileid):
> namelist = ['']
> try:
> cc.execute('''
> select parent_fileid, name from Cns_file_metadata where fileid = %s
> '' % fileid)
>
> name = ''
> parent_fileid = 0L
> p = cc.fetchone()
> #handle zero results here by raising exception!!!
> (parent_fileid, name) = p #this gets the "head" of the namei
> namelist.append(str(name))
> while parent_fileid> 1:
> cc.execute('''select parent_fileid, name from
> Cns_file_metadata where Cns_file_metadata.fileid = %s''' %
> parent_fileid)
> (parent_fileid, name) = cc.fetchone()
> #the above fetchone() is a zero length tuple
> in the case of no results, so that should except?
> namelist.append(str(name))
> except MySQLdb.Error, e:
> sys.exit("Error %d: %s" % (e.args[0], e.args[1]))
> #sys.exit (1)
> namelist.reverse() #put entries in "right" order for joining together
> return '/'.join(namelist)[1:] #and print dpns name (minus srm bits)
>
> to generate the dpns name for a particular fileid. (cc is a Cursor
> object from a MySQLdb Connection object)
>
> Do you want me to package this up into a "list files with replicas>
> 2" util for you?
>
> Sam
>
>> thanks
>>
>> cheers
>> alessandra
>>
>>
>> On 12/04/2011 09:52, Wahid Bhimji wrote:
>>
>> Hello
>>
>> I have version gridpp-dpm-tools-2.6.7.DPM174-1
>>
>> That's good.
>>
>> According to the output it only deleted 3 files because there were 3
>> replicas instead of 2. When I run it again it did nothing even though most
>> files have a single replica. I'm running without --del now.
>>
>> I am surprised it isn't working without the --del option.
>> If you use the --list option does it say that there are only one replica for
>> most of the files.
>> There is a --verbose option that gives slightly more output which may shed
>> some light.
>> I presume you have DPM_HOST, DPNS_HOST set?
>> Sorry you having troubles with it.
>> Wahid
>>
>> cheers
>> alessandra
>>
>> On 11/04/2011 17:28, Wahid Bhimji wrote:
>>
>> WAIT!
>>
>> This isn't really an answer to your question though and so likely to cause
>> confusion (!) For your answer you can skip down but I would read the first
>> bit as a warning.
>>
>> But I was just about to send a warning to the list about using this tool
>> with the --del option (interesting timing of your email).
>>
>> Basically (I have mentioned this before but maybe not stressed its
>> importance) there was a bug with the first version of my script that the
>> replicas it created were not permanent.
>>
>> This means that if:
>>
>> 1) You used an old version of the tool to make replicas AND
>>
>> 2) Your run either old or new version with --del to reduce the number (e.g.
>> from 3 to 2) AND
>>
>> 3) You run dpm-drain on a filesystem containing these replicas
>>
>> there is a risk of removing all the replicas.
>>
>> I want to write a tool to list if people have non-permanent replicas and to
>> mark them all permanent - so will send another post when I have done that.
>> But I just wanted to flag it here in case people are in this position.
>>
>> (If you do suspect you may have done this then let me know (and for
>> atlashotdisk) I can run an integrity check and if there are missing files I
>> can also mark them lost and they will be restored quite easily). To be clear
>> though - you need to have done all 3 above otherwise there is no problem.
>>
>> So to answer you actual question !
>>
>> If you have never used the tool before and use the latest version (which is
>> I think your actual question) then then:
>>
>> A) You do not need --del that just deletes from (e.g 3 to 2) so in your case
>> it will see there is not more than 2 and do nothing.
>>
>> B) The first time it will take a long time and so I would run it first
>> outside cron.
>>
>> C) Then once that is done it will only run on new files (those that do not
>> have 2 replicas in ATLASHOTDISK) so you can put it in a daily cron.
>>
>> Let me know if this is not clear
>>
>> (and apologies to anyone caught out by the above bug. It is actually easily
>> put right so do let me know if you are concerned and I will send out another
>> post with more details).
>>
>> Cheers
>>
>> Wahid
>>
>>
>> On 11 Apr 2011, at 17:04, Alessandra Forti wrote:
>>
>> Hi,
>>
>> I'm going to try this command
>>
>> dpm-sql-spacetoken-replicate-hotfiles --nreps=2 --st=ATLASHOTDISK --del
>>
>> I'm wondering how do I catch the new files though and how often can I run
>> it. Is it really something I can put in a cron job? Otherwise I'll go with
>> the normal list-hotfiles and replicate only those files that have>NNN
>> requests unless they have been already replicated.
>>
>> cheers
>>
>> alessandra
>>
>>
>>
>>
>>
>> The University of Edinburgh is a charitable body, registered in
>> Scotland, with registration number SC005336.
>>
>>
|
