Hi Steve
I poked around last night and could reproduce the bug with DPM too.
It shouldn't happen, of course. lcg-utils does do an
srmGetFileMetadata call, as it should, but it ignores the filetype
mask which is returned.
Doing an srm-get-metadata by hand gives:
ui1-gla:~$ srm-get-metadata -debug=false srm://se2-gla.scotgrid.ac.uk:
8443/dpm/scotgrid.ac.uk/home/dteam/lesc-transfer-test
...
permMode :16893
...
which is 100664 in octal (dunno what the top bit is for).
On a directory
ui1-gla:~$ srm-get-metadata -debug=false srm://se2-gla.scotgrid.ac.uk:
8443/dpm/scotgrid.ac.uk/home/dteam
...
permMode :33204
...
which is 40775 in octal, i.e.,
#define S_IFDIR 0040000 /* directory */
To my mind this _is_ a bug. First it's a file catalog - there's no
reason I can think of that one could usefully enter a dir. Second, a
replica for a file makes sense. Replicating a directory is clearly
impossible.
If you try and rep from a DPM you get
ui1-gla:~$ lcg-rf -v --vo dteam srm://se2-gla.scotgrid.ac.uk/dpm/
scotgrid.ac.uk/home/dteam
Using grid catalog type: lfc
Using grid catalog : prod-lfc-shared-central.cern.ch
set timeout to 0 seconds
Site URL to be registered: srm://se2-gla.scotgrid.ac.uk/dpm/
scotgrid.ac.uk/home/dteam
File size: 0
Alias created in Catalog: lfn:/grid/dteam/generated/2006-06-16/
file-4de347c7-f42a-4048-863d-1efd7a4c1d1e
guid:5d9902d9-33e9-4113-b4cd-b1f017acd045
ui1-gla:~$ lcg-rep -v --vo dteam -d dpm.epcc.ed.ac.uk srm://se2-
gla.scotgrid.ac.uk/dpm/scotgrid.ac.uk/home/dteam
Using grid catalog type: lfc
Using grid catalog : prod-lfc-shared-central.cern.ch
lcg_rep: No such file or directory
At least it fails cleanly, even if the error message is just wrong.
I would have thought a savannah bug rather than a GGUS ticket? (CC me
in.)
Cheers
Graeme
On 13 Jun 2006, at 18:02, Steve Traylen wrote:
> I was about to submit this as a GGUS ticket but thought as you are
> thinking
> about SRMs today someone may have comment here first.
>
>
> Hi,
>
> I found this problem after debugging a script with an atlas user for a
> couple of hours.
> While the mistake was in the script the lcg- commands should really
> be more
> resillant to this.
>
> When registering a SURL in an LcgFC the existance of the SURL is
> checked with
> the SRM and quite correctly fails if the SURL does not exist.
>
> However it appears that this check is also succesful if the SRM path
> corresponds to a directory. dCache and Castor handle replications
> of these
> LFNs in different ways.
>
> 1) Register an existing SURL in the LcgFC.
>
> For dCache
>
> $ lcg-cr -v --vo dteam
> srm://dcache.gridpp.rl.ac.uk/pnfs/gridpp.rl.ac.uk/data/dteam/
> existing/file
>
> lfn:/grid/dteam/generated/2006-06-13/file-3030ecfd-43fa-46c7-b287-
> e287c984ef23
>
> and for CASTOR
>
> $ lcg-cr -v --vo dteam
> srm://castorsrm.cern.ch/castor/cern.ch/grid/dteam/generated/
> 2006-06-13/file4589136b-d6dd-4b1a-aab3-eeb796ba1b48
>
> Alias created in Catalog:
> lfn:/grid/dteam/generated/2006-06-13/file-
> e02454c6-149d-444c-805d-0c5f7554b7cd
>
> 2) Now try registering a non-existing SURL.
>
> For CASTOR
> $ lcg-rf -v --vo dteam
> srm://castorsrm.cern.ch/castor/cern.ch/grid/dteam/generated/not/
> existing/file
>
> fails quite correctly with
>
> lcg_rf: No such file or directory
>
> a similar thing can be seen for dcache as well.
>
>
> 3) Now try registering an existing directory
>
> For dCache
>
> $ lcg-rf -v --vo dteam
> srm://dcache.gridpp.rl.ac.uk/pnfs/gridpp.rl.ac.uk/data/dteam/qmul
>
> returns
>
> Alias created in Catalog:
> lfn:/grid/dteam/generated/2006-06-13/
> file-971a3036-1521-4035-9674-081665f94ea3
>
> And for CASTOR
>
> $ lcg-rf -v --vo dteam
> srm://castorsrm.cern.ch/castor/cern.ch/grid/dteam/generated
>
> lfn:/grid/dteam/generated/2006-06-13/file-19b4eee3-4f59-480e-
> ad7f-0f18586ffc04
>
> so we have managed to be able to register and LFN and guid for these
> directories.
>
> 4) Now try and replicate each of these LFNs to the other server.
>
> For castor->dcache
>
> $ lcg-rep -v --vo dteam
> lfn:/grid/dteam/generated/2006-06-13/file-19b4eee3-4f59-480e-
> ad7f-0f18586ffc04
> -d dcache.gridpp.rl.ac.uk
> Using grid catalog type: lfc
> Using grid catalog : prod-lfc-shared-central.cern.ch
>
> this just hangs and has to be killed.
>
> For dcache -> castor
>
> $ lcg-rep -v --vo dteam
> lfn:/grid/dteam/generated/2006-06-13/
> file-971a3036-1521-4035-9674-081665f94ea3
> -d castorsrm.cern.ch
> Using grid catalog type: lfc
> Using grid catalog : prod-lfc-shared-central.cern.ch
> # streams: 1
> # set timeout to 0
> 0 bytes 0.00 KB/sec avg 0.00 KB/sec inst
>
> again this just hangs and needs to be killed.
>
> 5) doing just and lcg-cp
>
> for dcache
>
> lcg-cp -v --vo dteam
> lfn:/grid/dteam/generated/2006-06-13/
> file-971a3036-1521-4035-9674-081665f94ea3
> file:/etc/group
>
> # streams: 1
> # set timeout to 0 (seconds)
> a system call failed (Permission denied)
> lcg_cp: Transport endpoint is not connected
>
> which is sensible in that it at least fails.
>
> and for castor
>
> $ lcg-cp -v --vo dteam
> lfn:/grid/dteam/generated/2006-06-13/file-19b4eee3-4f59-480e-
> ad7f-0f18586ffc04
> file:/etc/junk1
> Using grid catalog type: lfc
> Using grid catalog : prod-lfc-shared-central.cern.ch
>
> and it just hangs like this.
>
>
> So in conclusion there are a number of points here.
>
> 1) Should one be able to register a SURL directory with lcg-rf
> The short answer is probably no you should not be able to do
> this but it is tricky since SRMs don't know anything about
> directories
> and a directory is a file after all. Can one distinguish between
> a file and directory at the SRM level?
>
> 2) Both castor and dCache should handle better the case where some
> one tries to either replicate or copy out a directory. Although
> the commands above were done with LFNs the same happens when SURLs
> are used that are actually directory paths.
>
> Steve
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> --
> Steve Traylen
> [log in to unmask]
> http://www.gridpp.ac.uk/
--
Dr Graeme Stewart - http://wiki.gridpp.ac.uk/wiki/User:Graeme_stewart
GridPP DM Wiki - http://wiki.gridpp.ac.uk/wiki/Data_Management
|