Well, I know it's not the definitive source of anything, but the 
wikipedia entry on JPEG2000 says:
"The PNG (Portable Network Graphics) format is still more 
space-efficient in the case of images with many pixels of the same 
color, and supports special compression features that JPEG 2000 does not." 

So would PNG be better?  It does support 16 bit greyscale.  Then again, 
so does TIFF, and Mar already uses that.  Why don't they use the LZW 
compression feature of TIFF?  The old Mar325 images were compressed 
after all. I think only Mar can answer this, but I imagine the choice to 
drop compression was because the advantages of compression (a factor or 
2 or so in space) are outweighed by the disadvantages (limited speed and 
limited compatibility with data processing packages).

How good could lossless compression of diffraction images possibly be?  
I just ran an entropy calculation on the 44968 images on "/data" at the 
moment at ALS 8.3.1.  I am using a feature of Andy Hammersley's program 
"FIT2D" to compute the entropy.  I don't pretend to completely 
understand the algorithm, but I do understand that the entropy of the 
image reflects the maximum possible compression ratio.  For these 
images, the "theoretical maximum compression ratio" ranged from 1.2 to 
4.8 with mean 2.7 and standard deviation 0.7.  The values for Huffmann 
encoding ranged from 0.95 to 4.7 with mean 2.4 and standard deviation 
1.0.  The correlation coefficient between the Huffmann and "theoretical" 
compression ratio was 0.97.  I had a look at a few of the outlier 
cases.  As one might expect, the best compression ratios are from blank 
images (where all the high-order bits are zero).  The #1 
hardest-to-compress image had many overloads, clear protein diffraction 
and a bunch of ice rings. 

So, unless I am missing something, I think the best we are going to get 
with lossless compression is about 2.5:1.  At least, for individual 
frames.  Compressing a data set as a "video" sequence might have 
substantial gains since only a few pixels change significantly from 
frame-to-frame.  Are there any lossless video codecs out there?  If so, 
can they handle 6144x6144 video?

  What about lossy compression?  Yes yes, I know it sounds like a 
horrible idea to use lossy compression on scientific data, because it 
would change the values of that most precious of numbers: Fobs.  
However, the question I have never heard a good answer to is HOW MUCH 
would it change Fobs?  More practically: how much compression can you do 
before Fobs changes by more than the value of SIGFobs?  Diffraction 
patterns are inherently noisy.  If you take the same image twice, then 
photon counting statistics make sure that no two images are exactly the 
same.  So which one is "right"?  If the changes in pixel values from a 
lossy compression algorithm are always smaller than that introduced by 
photon-counting noise, then is lossy compression really such a bad 
idea?  The errors introduced could be small when compared to errors in 
say, scale factors or bulk solvent parameters.  A great deal can be 
gained in compression ratio if only "random noise" is removed.  I 
remember the days before MP3 when it was lamented that sampled audio 
files could never be compressed very well.  Even today bzip2 does not 
work very well at all at compressing sampled audio (about 1.3:1), but 
mp3 files can be made at a "compression ratio" of 10:1 over CD-quality 
audio and we all seem to still enjoy the music.

I suppose the best "lossy compression" is the one that preserves the 
features of the image you want and throws out the stuff you don't care 
about.  So, in a way, data-reduction programs are probably the best 
"lossy compression" we are going to get.  Unfortunately, accurate 
"external information" is required (such as the beam center 
convention!), so the interface to this "compression algorithm" still a 
little more complicated than pkzip.  ;)

Nevertheless, there are still only about 40-50 "formats" of diffraction 
images floating around (30 operating beamlines, a few in-house vendors 
and some "history").  I would like to collect them all.  So, if anybody 
out there has a lysozyme data set (or even just a single image) from 
anywhere but ALS 8.3.1, please let me know how I can get a copy of it!

-James Holton
MAD Scientist


Harry Powell wrote:
> Hi
>
> Just to add to this. imgCIF (or CBF, which amounts to pretty well the 
> same thing) has fast and efficient compression built in, and has been 
> developed with protein crystallography (particularly) in mind. There 
> are even (a few) detectors out there which will write these instead of 
> (or as well as) the manufacturer's native format, saving the user the 
> trouble of conversion.
>
> If you're looking for a standard format for storing image data in, I 
> wouldn't look any further, since (in principle) imgCIF/CBF can store 
> all the image information you (or a fussy^H^H^H^H^H conscientious 
> reviewer who could be bothered to re-process your dataset) would want 
> about your data collection and you wouldn't need to come up with 
> inventive tags for data items that might be required for other 
> (general purpose) image formats.
>
> There are even conversion programs available to convert to imgCIF/CBF 
> files from some native formats - if your favourite detector isn't one 
> of these, drop Herb Bernstein a line and ask for support ;-)
>
>> I looked at jpeg2000 as a compression for diffraction images for
>> archiving purposes - it works well but is *SLOW*. It's designed with the
>> idea in mind of compressing a single image, not the several hundred
>> typical for our work. There is also no place to put the header.
>>
>> Bzip2 works pretty much as well and is standard, but again slow. This is
>> what people mostly seem to use for putting diffraction images on the
>> web, particularly the JCSG.
>>
>> The ccp4 "pack" format which has been around for a very long time works
>> very well and is jolly quick, and is supported in a number of data
>> processing packages natively (Mosflm, XDS). Likewise there is a new
>> compression being used for the Pilatus detector which is quicker again.
>> These two have the advantage of being designed for diffraction images
>> and with speed in mind.
>>
>> So there are plenty of good compression schemes out there - and if you
>> use CBF these can be supported natively in the image standard... So you
>> don't even need to know or care...
>>
>> Just my 2c on this one.
>>
>> Cheers,
>>
>> Graeme
>>
>> -----Original Message-----
>> From: CCP4 bulletin board [mailto:[log in to unmask]] On Behalf Of
>> Maneesh Yadav
>> Sent: 18 August 2007 00:02
>> To: [log in to unmask]
>> Subject: [ccp4bb] diffraction images images/jpeg2000
>>
>> FWIW, I don't agree with storing image data, I don't think they justify
>> the cost of storage even remotely (some people debate the value of the
>> structures themselves....)...but if you want to do it anyway, maybe we
>> should use a format like jpeg2000.
>>
>> Last time I checked, none of the major image processing suites used it,
>> but it is a very impressive and mature format that (I think) would be
>> suitable for diffraction images.  If anyone is up for experimenting, you
>> can get a nice suite of tools from kakadu (just google kakdu +
>> jpeg2000).
>>
>
> Harry