> I'll second that... can't remember anybody on the barricades about
> "corrected" CCD images, but they've been just so much more practical.
>
> Different kind of problem, I know, but equivalent situation: the people
> to ask are not the purists, but the ones struggling with the huge
> volumes of data. I'll take the lossy version any day if it speeds up
> real-time evaluation of data quality, helps me browse my datasets, and
> allows me to do remote but intelligent data collection.
>
> phx.
>
>
>
> On 08/11/2011 02:22, Herbert J. Bernstein wrote:
>> Dear James,
>>
>> You are _not_ wasting your time. Even if the lossy compression ends
>> up only being used to stage preliminary images forward on the net while
>> full images slowly work their way forward, having such a compression
>> that preserves the crystallography in the image will be an important
>> contribution to efficient workflows. Personally I suspect that
>> such images will have more important, uses, e.g. facilitating
>> real-time monitoring of experiments using detectors providing
>> full images at data rates that simply cannot be handled without
>> major compression. We are already in that world. The reason that
>> the Dectris images use Andy Hammersley's byte-offset compression,
>> rather than going uncompressed or using CCP4 compression is that
>> in January 2007 we were sitting right on the edge of a nasty
>> CPU-performance/disk bandwidth tradeoff, and the byte-offset
>> compression won the competition. In that round a lossless
>> compression was sufficient, but just barely. In the future,
>> I am certain some amount of lossy compression will be
>> needed to sample the dataflow while the losslessly compressed
>> images work their way through a very back-logged queue to the disk.
>>
>> In the longer term, I can see people working with lossy compressed
>> images for analysis of massive volumes of images to select the
>> 1% to 10% that will be useful in a final analysis, and may need
>> to be used in a lossless mode. If you can reject 90% of the images
>> with a fraction of the effort needed to work with the resulting
>> 10% of good images, you have made a good decision.
>>
>> An then there is the inevitable need to work with images on
>> portable devices with limited storage over cell and WIFI networks. ...
>>
>> I would not worry about upturned noses. I would worry about
>> the engineering needed to manage experiments. Lossy compression
>> can be an important part of that engineering.
>>
>> Regards,
>> Herbert
>>
>>
>> At 4:09 PM -0800 11/7/11, James Holton wrote:
>>> So far, all I really have is a "proof of concept" compression
>>> algorithm here:
>>>
http://bl831.als.lbl.gov/~jamesh/lossy_compression/
>>>
>>> Not exactly "portable" since you need ffmpeg and the x264 libraries
>>> set up properly. The latter seems to be constantly changing things
>>> and breaking the former, so I'm not sure how "future proof" my
>>> "algorithm" is.
>>>
>>> Something that caught my eye recently was fractal compression,
>>> particularly since FIASCO has been part of the NetPBM package for
>>> about 10 years now. Seems to give comparable compression vs quality
>>> as x264 (to my eye), but I'm presently wondering if I'd be wasting my
>>> time developing this further? Will the crystallographic world simply
>>> turn up its collective nose at lossy images? Even if it means waiting
>>> 6 years for "Nielsen's Law" to make up the difference in network
>>> bandwidth?
>>>
>>> -James Holton
>>> MAD Scientist
>>>
>>> On Mon, Nov 7, 2011 at 10:01 AM, Herbert J. Bernstein
>>> <
[log in to unmask]> wrote:
>>>> This is a very good question. I would suggest that both versions
>>>> of the old data are useful. If was is being done is simple
>>>> validation
>>>> and regeneration of what was done before, then the lossy compression
>>>> should be fine in most instances. However, when what is being
>>>> done hinges on the really fine details -- looking for lost faint
>>>> spots just peeking out from the background, looking at detailed
>>>> peak profiles -- then the lossless compression version is the
>>>> better choice. The annotation for both sets should be the same.
>>>> The difference is in storage and network bandwidth.
>>>>
>>>> Hopefully the fraud issue will never again rear its ugly head,
>>>> but if it should, then having saved the losslessly compressed
>>>> images might prove to have been a good idea.
>>>>
>>>> To facilitate experimentation with the idea, if there is agreement
>>>> on the particular lossy compression to be used, I would be happy
>>>> to add it as an option in CBFlib. Right now all the compressions
>>> > we have are lossless.
>>>> Regards,
>>>> Herbert
>>>>
>>>>
>>>> =====================================================
>>>> Herbert J. Bernstein, Professor of Computer Science
>>>> Dowling College, Kramer Science Center, KSC 121
>>>> Idle Hour Blvd, Oakdale, NY, 11769
>>>>
>>>>
+1-631-244-3035
>>>>
[log in to unmask]
>>>> =====================================================
>>>>
>>>> On Mon, 7 Nov 2011, James Holton wrote:
>>>>
>>>>> At the risk of sounding like another "poll", I have a pragmatic
>>>>> question
>>>>> for the methods development community:
>>>>>
>>>>> Hypothetically, assume that there was a website where you could
>>>>> download
>>>>> the original diffraction images corresponding to any given PDB file,
>>>>> including "early" datasets that were from the same project, but
>>>>> because of
>>>>> smeary spots or whatever, couldn't be solved. There might even
>>>>> be datasets
>>>>> with "unknown" PDB IDs because that particular project never did
>>>>> work out,
>>>>> or because the relevant protein sequence has been lost.
>>>>> Remember, few of
>>>>> these datasets will be less than 5 years old if we try to allow
>>>>> enough time
>>>>> for the original data collector to either solve it or graduate
>>>>> (and then
>>>>> cease to care). Even for the "final" dataset, there will be a
>>>>> delay, since
>>>>> the half-life between data collection and coordinate deposition
>>>>> in the PDB
>>>>> is still ~20 months. Plenty of time to forget. So, although the
>>>>> images were
>>>>> archived (probably named "test" and in a directory called "john")
>>>>> it may be
>>>>> that the only way to figure out which PDB ID is the "right
>>>>> answer" is by
>>>>> processing them and comparing to all deposited Fs. Assume this
>>>>> was done.
>>>>> But there will always be some datasets that don't match any PDB.
>>>>> Are those
>>>>> interesting? What about ones that can't be processed? What
>>>>> about ones that
>>>>> can't even be indexed? There may be a lot of those!
>>>>> (hypothetically, of
>>>>> course).
>>>>>
>>>>> Anyway, assume that someone did go through all the trouble to
>>>>> make these
>>>>> datasets "available" for download, just in case they are
>>>>> interesting, and
>>>>> annotated them as much as possible. There will be about 20
>>>>> datasets for any
>>>>> given PDB ID.
>>>>>
>>>>> Now assume that for each of these datasets this hypothetical
>>>>> website has
>>>>> two links, one for the "raw data", which will average ~2 GB per
>>>>> wedge (after
>>>>> gzip compression, taking at least ~45 min to download), and a
>>>>> second link
>>>>> for a "lossy compressed" version, which is only ~100 MB/wedge (2 min
>>>>> download). When decompressed, the images will visually look
>>>>> pretty much like
>>>>> the originals, and generally give you very similar Rmerge,
>>>>> Rcryst, Rfree,
>>>>> I/sigma, anomalous differences, and all other statistics when
>>>>> processed with
>>>>> contemporary software. Perhaps a bit worse. Essentially, lossy
>>>>> compression
>>>>> is equivalent to adding noise to the images.
>>>>>
>>>>> Which one would you try first? Does lossy compression make it
>>>>> easier to
>>>>> hunt for "interesting" datasets? Or is it just too repugnant to
>>>>> have
>>>>> "modified" the data in any way shape or form ... after the detector
>>>>> manufacturer's software has "corrected" it? Would it suffice to
>>>>> simply
>>>>> supply a couple of "example" images for download instead?
>>>>>
>>>>> -James Holton
>>>>> MAD Scientist
>>>>>
>>
>