Wow.
I don't know about the rest of you, but I got told three times.
Gerard is, of course, right about pixel non-independence (think "point
spread function", among other things), and I wouldn't care to argue
statistics with him, but as far as I know (and I could well be wrong) most
of the integration programs out there _do_ use counting statistics (i.e.
Poisson statistics) at least as a first approximation for the random error
in measurement; this may be modified by some "detector inefficiency
factor" (See Borek, Minor & Otwinowski, Acta Cryst (2003) D59 2031 -
2038), but it's still there and being used by "everyone", nonetheless.
Having said that, regarding the storage of images, my personal feeling is
that there's no real point in using a lossy compression when there are
good lossless systems out there. I also think that almost no-one would
ever bother to reprocess deposited images anyway; my guess is that
"unusual" structures would be detected by other means, and that examining
the original images would rarely shed light on the problem.
> I think we need to stop and think right here. The errors in pixel
> values of images are neither Poisson (i.e. forget about taking square roots)
> nor independent. Our ideas about image statistics are already disastrously
> poor enough: the last thing we need is to make matters even worse by using
> compression methods based on those erroneous statistical arguments!
>
>
> With best wishes,
>
> Gerard.
>
> --
> On Fri, Aug 24, 2007 at 01:20:29PM +0100, Harry Powell wrote:
>> Hi
>>
>> Lossy compression should be okay, provided that the errors introduced are
>> smaller than those expected for counting statistics (assuming that the
>> pixels are more-or-less independent) - i.e. less than the square-root of
>> the individual pixel intensities (though I don't see why this can't be
>> extended to the integrated reflection intensities). So it's more important
>> to accurately retain your weak pixel values than your strong ones - an
>> error of ±10 for a pixel in a background count where the background should
>> be 40 is significant, but an error of ±10 for a saturated pixel on most
>> detectors (say, about 64K for a CCD) wouldn't affect anything.
>>
>>> On the question of lossy compression, I think we'd have to ask some data
>>> reduction guru's how much the "noise" would affect the data reduction. I
>>> suspect that the main problem is that the noise added would be
>>> correlated across the image and would therefore affect the background
>>> statistics in a non-trivial way. Although the intensity measurements may
>>> not be badly affected the error estimates on them could be...
>>
>> Harry
>> --
>> Dr Harry Powell, MRC Laboratory of Molecular Biology, MRC Centre, Hills
>> Road, Cambridge, CB2 2QH
>
>
> --
>
> ===============================================================
> * *
> * Gerard Bricogne [log in to unmask] *
> * *
> * Global Phasing Ltd. *
> * Sheraton House, Castle Park Tel: +44-(0)1223-353033 *
> * Cambridge CB3 0AX, UK Fax: +44-(0)1223-366889 *
> * *
> ===============================================================
>
Harry
--
Dr Harry Powell, MRC Laboratory of Molecular Biology, MRC Centre, Hills
Road, Cambridge, CB2 2QH
|