Speaking of non-gain corrected compression, does anybody know if anyone has experimented with an electron impact [ X, Y, Z (time), Confidence ] parametrisation for counting data yet (and if so - could point me towards a reference or discussion of the relative merits)? It seems like it might better preserve information, particularly in Z, and be more sensible than saving frames and frames full of mostly zeros, no matter how efficient LZW compression can be for them?
Just my tuppence worth -
Chris
****
> On 16 Dec 2018, at 13:44, Henning Stahlberg <[log in to unmask]> wrote:
>
> Hi,
>
> You can save a non-gain-corrected image as integers, which can compressed loss-lessly when saving as TIFF with LZV compression; and you can save separately a gain reference field as floating point file. To process these, you would decompress the integer image and multiply it with the gain reference, to obtain a gain-corrected image. This is what SerialEM offers, and what allows highly efficient data collection.
>
> Unfortunately, I believe that for several camera manufacturers, this is not the same as saving a gain-corrected image directly.
>
> As Robb McLeod once explained to me, the assumption that gain correction can be done by simple multiplication with a floating point field, is only true in first approximation. There might be non-linearities in the gain profile of each pixel, and in addition the camera driver might also apply an envelope profile in Fourier space to the floating-point image, before handing it over to the user.
>
> If you look at a camera-driver-gain-corrected image in Fourier space, and compare that with the non-gain-corrected image that you multiply yourself with the linear gain reference, you might notice significant differences.
>
> For example, already 10 years ago, several camera drivers were using non-linear intensity look-up tables for each pixel for gain correcting their images, as for example done by TVIPS or Gatan. I believe these were at the time already interpolated or spline-fit from four gain reference images that were recorded at different electron doses. Similar non-linearities may even apply to counted images.
>
> To our luck, the difference between the two may be negligible, especially if MTF correction and Guinier-plot flattening is applied later during image processing.
>
> There are ideas around for compressing floating-point images. For example, Robb McLeod had proposed the MRCZ format, here: https://www.sciencedirect.com/science/article/pii/S1047847717302083 .
> He found that blosc-compressing and saving in his proposed MRCZ format would take less time than saving the non-compressed file, since the time required for computing the MRCZ compression would be shorter than the time transferring the image to the hard drive, or even to another computing node.
> And he also suggested direct detector normalization strategies, here: https://www.biorxiv.org/content/early/2017/03/13/116533
>
> Or, one could also simply use lossy JPEG compression, if you have larger datasets available: https://www.biorxiv.org/content/early/2018/10/24/451427
>
> Henning.
>
>
> Henning Stahlberg, PhD
> Prof. for Structural Biology, C-CINA, Biozentrum, University of Basel
> Mattenstrasse 26 | D-BSSE | WRO-1058 | CH-4058 Basel | Switzerland
> http://c-cina.org | Tel. +41-61-387 32 62
>
>
>
>
>
>
>
>> On Dec 16, 2018, at 7:43 AM, Takanori Nakane <[log in to unmask]> wrote:
>>
>> Hi,
>>
>> Sorry, I was not clear. Thanks for pointing it out.
>>
>> I meant the quality of motion correction is the same for the
>> following two strategies:
>>
>> (1) saving gain-non-corrected movies and the gain reference
>> separately and applying the gain correction during
>> data processing
>> (2) saving gain corrected movies during data collection
>>
>>> My recommendation would be keep the uncorrected (compressed if you want)
>>> images for storage efficiency,
>>
>> Yes.
>>
>>> and always correct for the gain before doing any operation with those movies
>>> movies (the corrected version can be
>>> deleted after finishing the operation so that it does not take space).
>>
>> In most programs, this is done in memory, so users don't have to
>> write (huge) gain-corrected movies during data processing.
>> Of course, they must specify the gain reference and geometry
>> transformations (rotations and flipping) if applicable.
>>
>> Best regards,
>>
>> Takanori Nakane
>>
>> On 2018/12/16 3:21, Carlos Oscar S. Sorzano wrote:
>>> Dear Bum and Takanori,
>>> El 12/12/2018 a las 9:02, Takanori Nakane escribió:
>>>> Hi,
>>>>
>>>> The advantage of non-gain-corrected data is that it is easier to
>>>> compress.
>>> I totally agree with this.
>>>> The quality of motion correction is the same.
>>> I would disagree with this statement. Depending on the range of the gain image (case 1: from 0.999 to 1.001; or case 2: from 0.5 to 2), the movie alignment may be driven by the particles in the image (case 1) or by the gain image (case 2, in this case I would expect that the alignment algorithm does not want to move the images at all because it would mean disaligning the gain images which have a lot of energy).
>>> My recommendation would be keep the uncorrected (compressed if you want) images for storage efficiency, and always correct for the gain before doing any operation with those movies (the corrected version can be deleted after finishing the operation so that it does not take space).
>>> Kind regards, Carlos Oscar
>>>>
>>>> Unfortunately, you cannot save non-gain corrected data from Falcon.
>>>> It is available only for K2/K3.
>>>>
>>>> Best regards,
>>>>
>>>> Takanori Nakane
>>>>
>>>> On 2018/12/12 2:58, 류범한 wrote:
>>>>> Hello, CCPEM.
>>>>>
>>>>> So far I have always collected gain-corrected data with EPU (& Titan Krios+Falcon3 / FEI).
>>>>>
>>>>> But in somewhere I read that ‘non-gain-corrected data' would be better in motion tracking with separately prepared gain reference image. So I am about to check how much it would be helpful, but I never tried this before.
>>>>>
>>>>> Regarding this, I have several questions.
>>>>>
>>>>>
>>>>> 1. What is the actual advantage of ’non-gain-corrected data’? in what terms exactly?
>>>>> (Simply tracking algorithms of motion correction softwares are slightly better than EPU?)
>>>>>
>>>>> 2. In practice, how much is it advantageous? in what terms?
>>>>> (Especially in the aspect of final resolution of refined 3D map)
>>>>>
>>>>> 3. How can I collect non-gain-corrected data ?
>>>>> By selecting ‘Bias’ option of Bias/Gain correction setting in CCD/TV camera TAB on TEM user interface?
>>>>>
>>>>> 4. How can I prepare gain reference separately and correctly?
>>>>> - Simply take from Falcon Reference manager? But file extension is .RAW other than .mrc. :(
>>>>> - Set the imaging condition exactly same with that of data collection, and just take gain reference image w/o sample or in empty hole area? (then check it via TIA?)
>>>>>
>>>>>
>>>>> Thanks in advance.
>>>>>
>>>>>
>>>>>
>>>>> Bum Han Ryu
>>>>>
>>>>> Postdoctoral Researcher
>>>>> Korea Basic Science Institute (KBSI)
>>>>> 161 Yeongudanji-ro, Ochang-eup, Cheongwon-gu, Cheongju-si
>>>>> Chungcheongbuk-do, Republic of Korea
>>>>>
>>>>> Office) +82-43-240-5329
>>>>> Mobile) +82-10-7119-0505
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> ########################################################################
>>>>>
>>>>> To unsubscribe from the CCPEM list, click the following link:
>>>>> https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCPEM&A=1
>>>>>
>>>>
>>>> ########################################################################
>>>>
>>>> To unsubscribe from the CCPEM list, click the following link:
>>>> https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCPEM&A=1
>>
>> ########################################################################
>>
>> To unsubscribe from the CCPEM list, click the following link:
>> https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCPEM&A=1
>
>
> To unsubscribe from the CCPEM list, click the following link:
> https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCPEM&A=1
>
########################################################################
To unsubscribe from the CCPEM list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCPEM&A=1
|
