I keep meaning to get back to that algorithm and make it a bit more
"slick" so that people don't need all the third-party stuff to use it.
In fact, I have a better idea for how to do it now I'm calling "noise
replacement compression", but that is awaiting something I like to call
"time" which appears to be in increasingly short supply these days.
Enthusiasm for lossy compression has also been somewhat dampened by
things like this:
http://blog.backblaze.com/2013/02/20/180tb-of-good-vibrations-storage-pod-3-0/
That's 180 TB for 2000 USD. Admittedly not exactly the fire-hardened
highly available redundant fault-tolerant "archive" that you might want
to preserve your data for all time. But if you build one of these
things and (ulp) install Windoze on it, then backblaze.com will be your
off-site redundant copy and maintain that copy for only $5/month. Also
has a nice web interface for retrieving your files. Which, I suppose,
you might be able to tell someone at TARDIS the password for the
particular backblaze account on this "pod" machine and make all the
image data you want "freely available" to the crystallographic community.
Of course, if EVERYONE does this, then backblaze might decide to change
their pricing schedule, but it looks like some astronomers have already
done something like this to them and they thought it was cool enough to
put it on their blog. I suppose if it really is a problem for them, they
might be interested in "licensing" my lossy-compression algorithm. ;)
-James Holton
MAD Scientist
On 3/11/2013 7:50 AM, Pete Meyer wrote:
> I use bzip2 as well. In addition, I generally store md5sums of the
> images (before and after compression) because it's quicker to check
> the hash for validity than load up the images - but this may be overkill.
>
> Pete
>
> Graeme Winter wrote:
>> Dear Eugene,
>>
>> Personally I have a habit of using bzip2 for archival of data.
>> Negative points: very slow. Positive points: universally supported,
>> lossless. I have lots of data. To be honest most of it I keep in the
>> native format.
>>
>> I expect to see plenty of comments of lossless vs. lossy compression
>> now :o)
>>
>> N.B. well processed unmerged raw-from-integration .x, INTEGRATE.HKL,
>> mosflm mtz represents pretty good lossy compression.
>>
>> Long term storage: depends on your definition of long term, which will
>> also depend on what you want to do with them... I would guess
>> useful-long-term would correspond to ~ 10 years to ~ 20 years tops. In
>> the past I have written data to tape which I have never attempted to
>> recover. Everything I now have on central file servers (raid systems)
>> on local fast drives and local cheap USB drives which sit unplugged on
>> my desk. Local fast drives fail (more frequently than the
>> manufacturers would admit), raids fail (less frequently but more
>> catastrophically), local cheap USB drives fail. I bank on them not all
>> failing at the same time :o) esp. the ones I leave unplugged.
>>
>> The cheap drives store about 2TB in something the size of a paperback
>> book and are relatively universal with USB connections. But they will
>> fail one day. You weigh up how sad you will be at the loss against the
>> cost of being more certain of not losing the data...
>>
>> Cheerio,
>>
>> Graeme
>>
>>
>>
>>
>> On 11 March 2013 08:39, Eugene Osipov <[log in to unmask]> wrote:
>>> Hello everyone,
>>>
>>> I have a couple of questions about images compression and storage:
>>> 1)do someone use it in routine work and does it works well for them?
>>> 2)I found this page by google:
>>> http://bl831.als.lbl.gov/~jamesh/lossy_compression/
>>> And want to ask about reports of usage of this program (of course if
>>> someone
>>> already uses it)
>>> 3)Is there any advice for long-term diffraction image storage?
>>> Thank you for your attention,
>>>
>>> --
>>> Eugene Osipov
>>> Junior Research Scientist
>>> Laboratory of Enzyme Engineering
>>> A.N. Bach Institute of Biochemistry
>>> Russian Academy of Sciences
>>> Leninsky pr. 33, 119071 Moscow, Russia
>>> e-mail: [log in to unmask]
|
