Testbed Support for GridPP member institutes [mailto:TB-
> [log in to unmask]] On Behalf Of Christopher J.Walker said:
> I'm curious why it is expensive. I didn't really think ls was an
> expensive operation (but perhaps it is). Is it that it can return huge
> amounts of data - which you don't want to send down a potentially
small
> network connection.
Partly the data volume, you could potentially have a vast number of
files and transmitting the data through a web-service interface isn't
very efficient. Also it may by implementation-dependent - for storm the
underlying operation is presumably just doing an ls on a file system,
but in most cases it's a database query and not necessarily one which
has been optimised given that the general use-case is a lookup of a
single file by name. The theory is that SURLs should be stored in an
external catalogue so you don't need to do an ls on the SE itself - but
of course that breaks down when you're looking for dark data ... there
is now some work going on to use the messaging system to synchronise SEs
and catalogues, but I don't know when or if that will bear fruit.
Stephen
|