> -----Original Message-----
> From: GRIDPP2: Deployment and support of SRM and local storage
> management [mailto:[log in to unmask]]On Behalf Of
> Owen Synge
> Sent: 14 June 2006 11:19
> To: [log in to unmask]
> Subject: Re: Administration scripts/functionality in dCache and DPM
>
>
> On Wed, 14 Jun 2006 08:43:49 +0100
> Greig A Cowan <[log in to unmask]> wrote:
>
> > Hi everyone,
> >
> > Thanks very much for your useful responses yesterday
> regarding "Quotas
> > for dCache/DPM". Keep them coming if you think of anything else.
> >
> > If possible, I would like to start up a similar thread today on the
> > subject of administration scripts/functionality that you
> would like to
> > see in dCache or DPM. For example, the ability to easily
> drain a disk
> > pool into another filesystem/pool would be useful to have. This is
> > already possible in dCache using the CopyManager, but it
> isn't a very
> > flexible tool nor is it very easy to use. Can you think of anything
> > else that you would like to have available?
> >
> > Thanks in advance for any responses.
> >
One thing I'd like to see is splitting read and writes into seperate queues inside a pool; for example dCache isn't particularly great at coping with pools which are very nearly full, and so you'll end up with a couple of write requests which are waiting for space and sitting doing nothing apart from taking up a Active mover queue spot which could have been used by a read request.
> > Cheers,
> > Greig
>
> For D-cache.
>
> I have grabbed some of Derek's scripts knowing that hes got
> lots of work
> to do today. I hope he gets time to read this email and improve my
> comments.
Most of these scripts consist of nothing more than a for loop to generate a series of admin interface command into a file, catting the file in to the admin interface and redirecting the output to another file, and then grepping the output file for what we're interested in. Complex they are not :-).
>
> The ones Derek has given me so far are:
>
> Something that checks the integrity of pnfs, ie does the file with a
> pnfs ID exist. This needs to be done both by path, and by pnfsid.
By path, because I have wanted to check all of a particular VOs files were on disk.
The by-pnfsid one is probably redundant now, it'd be better to write a perl (or python if you must :-)) script to connect to the companion database and query it directly, as you could do clever things with SQL. See http://www.gridpp.ac.uk/wiki/DCache_Problems_and_Workarounds#Checking_pool_migration for an example, attempting to duplicate that query through the admin interface would be painful.
> Finding the pool that erroring files are meant to be on.
Actually it looks like a script to iterate over all pools finding files in the error state and then deleting them. I can't recall ever actually running that one.
> Checking files have gone to and from tape after they have moved.
That sounds like our hsm get/restore script which likely got bundled up in the tar ball
> Checking files in trash have been deleted from tape and deleting them
> after the event from trash and triggering deletion in ADS. (Overcomes
> D-cache ADS incompatible modes of operation in API design
> particularly
> when D-cache cancels a transfer.)
No, this script is for propagating deletes of tape stored dCache files to the tape system as dCache doesn't do it itself.
> Bulk movement of files from one pool to multiple pools.
>
> I believe Derek has some more ideas, Including a file permissions
> manager as PNFS does not support the sticky bit, so permissions
> management can be less of an issue. (CMS phedex makes this important)
This is possibly a RAL-specific problem, CMS/PhEDEx moves files in via the SRM, but delete files by rm'ing via nfs, when writing the DN gets mapped to a cms pool account, when running rm, it has the uid of the person who started the agents, thus we get permission/ownership problems - files get written with rwxr-xr-x permissions and we have a script that searches the entire CMS directory tree (ouch) for things with these permissions and changes them to rwxrwxr-x permissions
Derek
|