Dear Harry,

     Sorry, this did end up sounding harsher than I intended, and it could
have done with a few judiciously placed smileys :) . My mailer may also have 
been overzealous: there was no intention of bombarding you with disapproval.

     I was simply trying to say that we are still far from fully
understanding the statistics of images, and that if we are to have a chance
of applying future insights into this open question towards improving some
interesting difficult datasets that would have previously been deposited, we
do not want some assumptions to have been made in the compression of the
original images that would, by then, be known to have been erroneous.

     Raw data are sacred. The question is of course: "how raw?"; but the
answer is always going to be to go back to rawer and rawer forms, rather
than towards more and more cooked ones (especially, if the recipes are open
to doubt). 


     With best wishes,
     
          Gerard.


--
On Fri, Aug 24, 2007 at 03:39:31PM +0100, Harry Powell wrote:
> 
> Wow.
> 
> I don't know about the rest of you, but I got told three times.
> 
> Gerard is, of course, right about pixel non-independence (think "point 
> spread function", among other things), and I wouldn't care to argue 
> statistics with him, but as far as I know (and I could well be wrong) most 
> of the integration programs out there _do_ use counting statistics (i.e. 
> Poisson statistics) at least as a first approximation for the random error 
> in measurement; this may be modified by some "detector inefficiency 
> factor" (See Borek, Minor & Otwinowski, Acta Cryst (2003) D59 2031 - 
> 2038), but it's still there and being used by "everyone", nonetheless.
> 
> Having said that, regarding the storage of images, my personal feeling is 
> that there's no real point in using a lossy compression when there are 
> good lossless systems out there. I also think that almost no-one would 
> ever bother to reprocess deposited images anyway; my guess is that 
> "unusual" structures would be detected by other means, and that examining 
> the original images would rarely shed light on the problem.
> 
> >    I think we need to stop and think right here. The errors in pixel
> >values of images are neither Poisson (i.e. forget about taking square 
> >roots)
> >nor independent. Our ideas about image statistics are already disastrously
> >poor enough: the last thing we need is to make matters even worse by using
> >compression methods based on those erroneous statistical arguments!
> >
> >
> >    With best wishes,
> >
> >         Gerard.
> >
> >--
> >On Fri, Aug 24, 2007 at 01:20:29PM +0100, Harry Powell wrote:
> >>Hi
> >>
> >>Lossy compression should be okay, provided that the errors introduced are
> >>smaller than those expected for counting statistics (assuming that the
> >>pixels are more-or-less independent) - i.e. less than the square-root of
> >>the individual pixel intensities (though I don't see why this can't be
> >>extended to the integrated reflection intensities). So it's more important
> >>to accurately retain your weak pixel values than your strong ones - an
> >>error of ±10 for a pixel in a background count where the background 
> >>should
> >>be 40 is significant, but an error of ±10 for a saturated pixel on most
> >>detectors (say, about 64K for a CCD) wouldn't affect anything.
> >>
> >>>On the question of lossy compression, I think we'd have to ask some data
> >>>reduction guru's how much the "noise" would affect the data reduction. I
> >>>suspect that the main problem is that the noise added would be
> >>>correlated across the image and would therefore affect the background
> >>>statistics in a non-trivial way. Although the intensity measurements may
> >>>not be badly affected the error estimates on them could be...
> >>
> >>Harry
> >>--
> >>Dr Harry Powell, MRC Laboratory of Molecular Biology, MRC Centre, Hills
> >>Road, Cambridge, CB2 2QH
> >
> >
> >-- 
> >
> >    ===============================================================
> >    *                                                             *
> >    * Gerard Bricogne                     [log in to unmask]  *
> >    *                                                             *
> >    * Global Phasing Ltd.                                         *
> >    * Sheraton House, Castle Park         Tel: +44-(0)1223-353033 *
> >    * Cambridge CB3 0AX, UK               Fax: +44-(0)1223-366889 *
> >    *                                                             *
> >    ===============================================================
> >
> 
> Harry
> -- 
> Dr Harry Powell, MRC Laboratory of Molecular Biology, MRC Centre, Hills
> Road, Cambridge, CB2 2QH