Hi,

we seem to have a problem with the storage.

Most of the jobs are failing to stagein the input and the problem is xrootd AFAICT and it might be a problem with our configuration.

Although a user also wrote me he cannot dq2-get but I still have to investigate that because I can use lcg-cp fine. xrootd logs say the following

One job example

http://panda.cern.ch/server/pandamon/query?job=2098194781

Last server error 10000 ('') Error accessing path/file for root://bohr3226.tier2.hep.manchester.ac.uk//dpm/tier2.hep.manchester.ac.uk/home/atlas/atlasdatadisk/rucio/mc12_8TeV/5d/bd/EVNT.01001789._000001.pool.root.1
03 Mar 14:15:05|xrdcpSiteMov| !!WARNING!!2990!! Command failed: source /cvmfs/atlas.cern.ch/repo/sw/local/xrootdsetup.sh; xrdcp root://bohr3226.tier2.hep.manchester.ac.uk//dpm/tier2.hep.manchester.ac.uk/home/atlas/atlasdatadisk/rucio/mc12_8TeV/5d/bd/EVNT.01001789._000001.pool.root.1 /scratch/1223444.ce02.tier2.hep.manchester.ac.uk/condorg_vhDcxF8w/pilot3/Panda_Pilot_17101_1393855151/PandaJob_2098194781_1393855213/EVNT.01001789._000001.pool.root.1 03 Mar 14:15:05|futil.py | WARNING: Abnormal termination: ecode=256, ec=1, sig=-, len(etext)=1140 03 Mar 14:15:05|futil.py | WARNING: Error message: Created /home/prdatl012/home_cream_470305009/.asetup. Please look and (optional) edit it.

if I try with lcg-cp I can copy the file

[aforti@bohr2825 ~]$ lcg-cp --verbose srm://bohr3226.tier2.hep.manchester.ac.uk//dpm/tier2.hep.manchester.ac.uk/home/atlas/atlasdatadisk/rucio/mc12_8TeV/5d/bd/EVNT.01001789._000001.pool.root.1 ./lcgcp-test
Using grid catalog type: UNKNOWN
Using grid catalog : prod-lfc-atlas.cern.ch
VO name: atlas
Checksum type: None
Trying SURL srm://bohr3226.tier2.hep.manchester.ac.uk//dpm/tier2.hep.manchester.ac.uk/home/atlas/atlasdatadisk/rucio/mc12_8TeV/5d/bd/EVNT.01001789._000001.pool.root.1 ...
Source SE type: SRMv2
Source SRM Request Token: 53897457-1d97-4f41-babb-3601da1bc326
Source URL: srm://bohr3226.tier2.hep.manchester.ac.uk//dpm/tier2.hep.manchester.ac.uk/home/atlas/atlasdatadisk/rucio/mc12_8TeV/5d/bd/EVNT.01001789._000001.pool.root.1
File size: 143179334
Source URL for copy: gsiftp://se06.tier2.hep.manchester.ac.uk/se06.tier2.hep.manchester.ac.uk:/raid/atlas/2014-01-06/EVNT.01001789._000001.pool.root.1.123581076.0
Destination URL: file:/home/aforti/./lcgcp-test
# streams: 1
    131072000 bytes  63839.22 KB/sec avg  63839.22 KB/sec inst
Transfer took 3010 ms

if I try with xrdcp it fails


and in the log files it is full of this error

140303 08:16:33 14919 XrdAccept: Unable to perform accept; too many open files

it looks to me I should change something in the configuration. The only reasonable thread I found is this but it's not DPM-xrootd

https://listserv.slac.stanford.edu/cgi-bin/wa?A2=ind0804&L=XROOTD-L&D=0&P=5756

This problem sums up with the FAX problems we have but are kind of more urgent since they are blocking production.

Thanks for any help.

cheers
alessandra