Hi all,
We installed cvmfs (2.0.6, later 2.0.7) across our cluster last last
year, and initially it seemed to work. But all was not well - stacktrace
files were showing up amongst the cache files, e.g.
/data2/var/cache/cvmfs2/*/stacktrace. These contained data like below.
The error also shows in syslog.
-- Signal: 6, errno: 619 version: 2.0.7 cvmfs2[0x42de37]
/lib64/libc.so.6(gsignal+0x35)[0x2b09bd9b1265]
/lib64/libc.so.6(gsignal+0x35)[0x2b09bd9b1265]
/lib64/libc.so.6(abort+0x110)[0x2b09bd9b2d10]
/lib64/libc.so.6(__assert_fail+0xf6)[0x2b09bd9aa6e6] cvmfs2[0x4229a2]
/lib64/libpthread.so.0[0x2b09bd76c73d]
/lib64/libc.so.6(clone+0x6d)[0x2b09bda554bd] Timestamp: Sat Feb 4
21:39:43 2012
We had hundreds of these. We came across this bug:
https://savannah.cern.ch/bugs/?84872. It describes a stacktrace, but
relates to a condition where space was exhausted. We have plenty of
space. So we contacted Jakob Blomer, who fixes cvmfs, and he found some
race condition. He has released a new copy (cvmfs-2.0.10). So far
(between 5 and 10 days), we've had no sign of the problem. It may or may
not be fixed - the jury is out. But now the problem has developed, I'd
like to know this: does anyone else find
/data2/var/cache/cvmfs2/*/stacktrace files in their caches? If so, what
version of cvmfs do you use?
Cheers in advance,
Steve
PS: the number of occurrences seemed slightly correlated with the
frequency of automount/unmount operations. We reduced the frequency of
mounts, and got less stacktraces.
--
Steve Jones [log in to unmask]
System Administrator office: 220
High Energy Physics Division tel (int): 42334
Oliver Lodge Laboratory tel (ext): +44 (0)151 794 2334
University of Liverpool http://www.liv.ac.uk/physics/hep/
|