 |
 |
Hi Alessandra, Alessandra Forti // EOJ пишет: > I do believe that a lot of files left behind are waste that hasn't been > cleaned up by failed jobs and since we are now in a shared environment it > should be good practice of the experiments to periodically and > frequently clean up. While this is true, there are 2 points: (a) unfortunately, this is largely due to LCG [mis]features: the jobs fail far too often, and VERY many exactly on the stage of copying and registering files; if LCG data management tools would have been more reliable, we wouldn't have had that much garbage; (b) imagine the "leftover" files are not garbage: they were produced by very valid LCG jobs, and may well could have been perfectly valid data (most actually are). LCG was capable of producing it, how come it is not capable of storing it? There's nothing wrong with a full SE - or there shouldn't be. > I think this is more of a problem of data and storage management. Any site > can put what they have but there will be an big variety of hardware to > deal with. I don't see in place neither the software nor a good policy to > use it properly. It's really easy to ask the sites to push the data > management developers to implement features, perhaps it is the experiments > that should do something about it. Oh, we do, but who ever listens to what the _users_ are mumbling! You guys have more weight :-) > It's your data after all (on our > machines). It looks like the law of the jungle if jobs don't even check > the storage element space and don't respect any kind of policy. Well, with the existing software you can't quite enforce any kind of policy. Of course one could check the available disk space, but it's basically identical to attempting to write to the SE: the attempt will fail if there's no enough space, and the job will try to write to another location. In short, if I can fill the allocated space, I *WILL*. Law of the Jungle, true. Stop me. Make a dedicated partition for me. I can't follow the peculiarities of 80+ sites. Quite often I can't even trust what is published in infosys, as many attributes are modified manually - if ever. Oxana.