On 12 April 2011 09:59, Alessandra Forti <[log in to unmask]> wrote:
> Hi Wahid,
>
> it doesn't replicate when --del is used (is it supposed to work?). I rerun
> it without --del and with --verbose in this way it is making the copies.
> Still I don't think it'll be enough, yesterday I had 2 problems
>
> 1) the same file was reuested 5000 times 2 copies were not enough
> 2) reco jobs wreaked avoc with proddisk files
>
> Problem 2) means we need to integrate your tool with something that also
> looks at the hotfiles list of the day and makes copies of those too.
>
> It be nice to know which files have more than one replica independently from
> the space token. Do you know what mysql query or API one is supposed to use?
select fileid, count(*) from cns_db.Cns_file_replica GROUP BY
cns_db.Cns_file_replica.fileid
will give you a list of fileid s and how many replicas exist for them.
You could then do something like:
def sfn_lookup(cc, fileid):
namelist = ['']
try:
cc.execute('''
select parent_fileid, name from Cns_file_metadata where fileid = %s
'' % fileid)
name = ''
parent_fileid = 0L
p = cc.fetchone()
#handle zero results here by raising exception!!!
(parent_fileid, name) = p #this gets the "head" of the namei
namelist.append(str(name))
while parent_fileid > 1:
cc.execute('''select parent_fileid, name from
Cns_file_metadata where Cns_file_metadata.fileid = %s''' %
parent_fileid)
(parent_fileid, name) = cc.fetchone()
#the above fetchone() is a zero length tuple
in the case of no results, so that should except?
namelist.append(str(name))
except MySQLdb.Error, e:
sys.exit("Error %d: %s" % (e.args[0], e.args[1]))
#sys.exit (1)
namelist.reverse() #put entries in "right" order for joining together
return '/'.join(namelist)[1:] #and print dpns name (minus srm bits)
to generate the dpns name for a particular fileid. (cc is a Cursor
object from a MySQLdb Connection object)
Do you want me to package this up into a "list files with replicas >
2" util for you?
Sam
>
> thanks
>
> cheers
> alessandra
>
>
> On 12/04/2011 09:52, Wahid Bhimji wrote:
>
> Hello
>
> I have version gridpp-dpm-tools-2.6.7.DPM174-1
>
> That's good.
>
> According to the output it only deleted 3 files because there were 3
> replicas instead of 2. When I run it again it did nothing even though most
> files have a single replica. I'm running without --del now.
>
> I am surprised it isn't working without the --del option.
> If you use the --list option does it say that there are only one replica for
> most of the files.
> There is a --verbose option that gives slightly more output which may shed
> some light.
> I presume you have DPM_HOST, DPNS_HOST set?
> Sorry you having troubles with it.
> Wahid
>
> cheers
> alessandra
>
> On 11/04/2011 17:28, Wahid Bhimji wrote:
>
> WAIT!
>
> This isn't really an answer to your question though and so likely to cause
> confusion (!) For your answer you can skip down but I would read the first
> bit as a warning.
>
> But I was just about to send a warning to the list about using this tool
> with the --del option (interesting timing of your email).
>
> Basically (I have mentioned this before but maybe not stressed its
> importance) there was a bug with the first version of my script that the
> replicas it created were not permanent.
>
> This means that if:
>
> 1) You used an old version of the tool to make replicas AND
>
> 2) Your run either old or new version with --del to reduce the number (e.g.
> from 3 to 2) AND
>
> 3) You run dpm-drain on a filesystem containing these replicas
>
> there is a risk of removing all the replicas.
>
> I want to write a tool to list if people have non-permanent replicas and to
> mark them all permanent - so will send another post when I have done that.
> But I just wanted to flag it here in case people are in this position.
>
> (If you do suspect you may have done this then let me know (and for
> atlashotdisk) I can run an integrity check and if there are missing files I
> can also mark them lost and they will be restored quite easily). To be clear
> though - you need to have done all 3 above otherwise there is no problem.
>
> So to answer you actual question !
>
> If you have never used the tool before and use the latest version (which is
> I think your actual question) then then:
>
> A) You do not need --del that just deletes from (e.g 3 to 2) so in your case
> it will see there is not more than 2 and do nothing.
>
> B) The first time it will take a long time and so I would run it first
> outside cron.
>
> C) Then once that is done it will only run on new files (those that do not
> have 2 replicas in ATLASHOTDISK) so you can put it in a daily cron.
>
> Let me know if this is not clear
>
> (and apologies to anyone caught out by the above bug. It is actually easily
> put right so do let me know if you are concerned and I will send out another
> post with more details).
>
> Cheers
>
> Wahid
>
>
> On 11 Apr 2011, at 17:04, Alessandra Forti wrote:
>
> Hi,
>
> I'm going to try this command
>
> dpm-sql-spacetoken-replicate-hotfiles --nreps=2 --st=ATLASHOTDISK --del
>
> I'm wondering how do I catch the new files though and how often can I run
> it. Is it really something I can put in a cron job? Otherwise I'll go with
> the normal list-hotfiles and replicate only those files that have>NNN
> requests unless they have been already replicated.
>
> cheers
>
> alessandra
>
>
>
>
>
> The University of Edinburgh is a charitable body, registered in
> Scotland, with registration number SC005336.
>
>
|