 |
 |
Frank von Delft wrote: > Just looked at the algorithm, how it stores the average "non-spot" > through all the images. > > What happens with dataset where the "non-spot" (e.g. background) > changes systematically through the dataset, i.e. anisotropic datasets > or thin crystals lying flat in a thin loop? How much worse is > compression for that? > Cheers > phx Well, what will happen in that case (with the current "algorithm") is that once a background pixel deviates from the median level by more than 4 "sigmas", it will start to get stored losslessly. Essentially, they will be treated as "spots" and the overall compression ratio will start to approach that of bzip2. A "workaround" for this is simply to store the data set in "chunks" where the background level is similar, but I suppose a more intelligent thing to do would be to simply "scale" each image to the median background image, and store the scale factors (a list of 100 numbers for a 100-image data set) along with the other ancillary data. I haven't done that yet. Didn't want to spend too much time on this in case I incited some kind of revolt. -James Holton MAD Scientist > > > On 07/05/2010 06:07, James Holton wrote: >> Ian Tickle wrote: >>> I found an old e-mail from James Holton where he suggested lossy >>> compression for diffraction images (as long as it didn't change the >>> F's significantly!) - I'm not sure whether anything came of that! >> >> Well, yes, something did come of this.... But I don't think Gerard >> Bricogne is going to like it. >> >> Details are here: >> http://bl831.als.lbl.gov/~jamesh/lossy_compression/ >> >> Short version is that I found a way to compress a test lysozyme >> dataset by a factor of ~33 with no apparent ill effects on the data. >> In fact, anomalous differences were completely unaffected, and Rfree >> dropped from 0.287 for the original data to 0.275 when refined >> against Fs from the compressed images. This is no doubt a fluke of >> the excess noise added by compression, but I think it highlights how >> the errors in crystallography are dominated by the inadequacies of >> the electron density models we use, and not the quality of our data. >> >> The page above lists two data sets: "A" and "B", and I am interested >> to know if and how anyone can "tell" which one of these data sets was >> compressed. The first image of each data set can be found here: >> http://bl831.als.lbl.gov/~jamesh/lossy_compression/firstimage.tar.bz2 >> >> -James Holton >> MAD Scientist