|
Hi,
For cold storage gzip and its parallel version pigz works quite well for us. It compresses considerably faster than bzip2 and the compression ratio is not much worse. The crucial part is to compress movies before drift correction, as they compress much better. Applying or not applying the gain reference actually has a smaller influence, as 0 pixels, which account for a large number of pixels in low-dose movies, stay 0 even after gain-correction (at least for DM K2 images).
But I agree with Steve that it is suboptimal for live data, as it adds additional compression/decompression steps to the workflow.
For the 'defect' pixels there is a correction within DM. It looks like the values for the defect pixels are interpolated from neighboring pixels, but I don't know what exact method is used. EPU probably just uses whatever DM writes out.
Best,
Andreas
-------
Andreas Schenk, Ph.D.
Friedrich Miescher Institute
for Biomedical Research
CH-4056 Basel
Switzerland
> -----Original Message-----
> From: Collaborative Computational Project in Electron cryo-Microscopy
> <[log in to unmask]> On Behalf Of Mike Strauss
> Sent: Monday, June 4, 2018 01:35
> To: [log in to unmask]
> Subject: Re: [ccpem] CCPEM Digest - 1 Jun 2018 to 3 Jun 2018 (#2018-136)
>
> Hi Yehuda,
>
> mrc2tif from IMOD has some compression algorithms, the relevant flag from
> the man page is:
>
> -c val Compress data; val can be lzw, zip, jpeg, or numbers defined in libtiff
>
> If you have integer or float data from gain corrected images, I suspect you
> won’t gain much more than 10%. If it’s K2 data and you have the Gain
> reference that was applied, you could try to get back to the actual counted
> values, and store those as compressed tifs. That could give you a factor of 8
> or so, depending on the dose rate and frame time.
>
> Good luck.
>
> mike
>
> -------
> Mike Strauss - former cryoEM Facility Manager MPI Biochemistry
> [log in to unmask]
> tel: +49 89-8578-2474
> mob: +49 151 55 308040
>
> > On 4. Jun 2018, at 01:00, CCPEM automatic digest system
> <[log in to unmask]> wrote:
> >
> > There are 12 messages totaling 5503 lines in this issue.
> >
> > Topics of the day:
> >
> > 1. Converting MRC to tiff (11)
> > 2. [3dem] [ccpem] Converting MRC to tiff
> >
> >
> ##########################################################
> ############
> > ##
> >
> > To unsubscribe from the CCPEM list, click the following link:
> > https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCPEM&A=1
> >
> > ----------------------------------------------------------------------
> >
> > Date: Sun, 3 Jun 2018 09:02:21 +0000
> > From: Yehuda Halfon <[log in to unmask]>
> > Subject: Converting MRC to tiff
> >
> > Hi there,
> >
> > We have a bunch of MRC movie files the are eating at out storage, and
> since more are coming I was wondering if there is a good way to convert
> them into tiff to save space?
> >
> > I know that the best way is to save them directly as tif from EPU/ serialEM
> and that is what we will to in the future. But we need to find a solution to the
> ones we have now.
> >
> > Thanks,
> >
> > Yehuda Halfon
> >
> >
> ##########################################################
> ############
> > ##
> >
> > To unsubscribe from the CCPEM list, click the following link:
> > https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCPEM&A=1
> >
> > ------------------------------
> >
> > Date: Sun, 3 Jun 2018 10:17:29 +0100
> > From: Joshua Lobo <[log in to unmask]>
> > Subject: Re: Converting MRC to tiff
> >
> > Hi Dr Halfon
> >
> > Spider has a cp to tiff , imod has mrc2tiff . I haven't tried them
> > but I think it's what you are looking for
> >
> > Sincerely
> > Joshua Lobo
> >
> > On Sun, Jun 3, 2018, 10:02 AM Yehuda Halfon
> > <[log in to unmask]>
> > wrote:
> >
> >> Hi there,
> >>
> >> We have a bunch of MRC movie files the are eating at out storage, and
> >> since more are coming I was wondering if there is a good way to
> >> convert them into tiff to save space?
> >>
> >> I know that the best way is to save them directly as tif from EPU/
> >> serialEM and that is what we will to in the future. But we need to
> >> find a solution to the ones we have now.
> >>
> >> Thanks,
> >>
> >> Yehuda Halfon
> >>
> >> ------------------------------
> >>
> >> To unsubscribe from the CCPEM list, click the following link:
> >> https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCPEM&A=1
> >>
> >
> >
> ##########################################################
> ############
> > ##
> >
> > To unsubscribe from the CCPEM list, click the following link:
> > https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCPEM&A=1
> >
> > ------------------------------
> >
> > Date: Sun, 3 Jun 2018 11:15:36 +0100
> > From: Takanori Nakane <[log in to unmask]>
> > Subject: Re: Converting MRC to tiff
> >
> > Hi,
> >
> > If it is from K2 and not gain normalised, compression is trivial:
> > mrc2tif -s -c zip input.mrcs ouput.tiff (using IMOD) The output
> > usually becomes less than 5-10 % of the input.
> > (Of course this depends on the dose rate)
> >
> > If it is gain-normalised, the file is in 32-bit float and very hard to
> > compress. You must first divide them by the gain reference to bring it
> > back to integer and then compress. If you don't have the gain
> > reference, you can estimate them by a simple program that finds common
> > divisor over all frames for each pixel. This works fine for data from serial
> EM.
> > I could loss-lessly compress EMPIAR-10061 from 12.6 TB to 306 GB this
> > way. See my old post at
> > https://www.jiscmail.ac.uk/cgi-
> bin/webadmin?A2=ind1703&L=CCPEM&P=R33113&1=CCPEM.
> >
> > Unfortunately, EPU (or DM?) seems to do something with 'defect' pixels.
> > I don't know what is happening here. Their values are not multiples of
> > the gain reference and the above method is no longer lossless.
> > But these pixels are defects anyway and probably don't matter.
> >
> > For images from Falcon III, the data are 16-bit integers. So you can
> > use mrc2tif in the same way as for K2.
> > However, because of the convolution algorithm, the compression ratio
> > is not as high as K2. Usually I get about 60 % of the original size.
> > If we had access to 'centroid-ed' super-resolution images before
> > convolution, we could save huge amount of storage...
> >
> > For images from Falcon II, I don't know any good tricks.
> >
> > Best regards,
> >
> > Takanori Nakane
> >
> > On 2018/06/03 10:17, Joshua Lobo wrote:
> >> Hi Dr Halfon
> >>
> >> Spider has a cp to tiff , imod has mrc2tiff . I haven't tried them
> >> but I think it's what you are looking for
> >>
> >> Sincerely
> >> Joshua Lobo
> >>
> >> On Sun, Jun 3, 2018, 10:02 AM Yehuda Halfon
> >> <[log in to unmask]>
> >> wrote:
> >>
> >>> Hi there,
> >>>
> >>> We have a bunch of MRC movie files the are eating at out storage,
> >>> and since more are coming I was wondering if there is a good way to
> >>> convert them into tiff to save space?
> >>>
> >>> I know that the best way is to save them directly as tif from EPU/
> >>> serialEM and that is what we will to in the future. But we need to
> >>> find a solution to the ones we have now.
> >>>
> >>> Thanks,
> >>>
> >>> Yehuda Halfon
> >>>
> >>> ------------------------------
> >>>
> >>> To unsubscribe from the CCPEM list, click the following link:
> >>> https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCPEM&A=1
> >>>
> >>
> >>
> ##########################################################
> ###########
> >> ###
> >>
> >> To unsubscribe from the CCPEM list, click the following link:
> >> https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCPEM&A=1
> >>
> >
> >
> ##########################################################
> ############
> > ##
> >
> > To unsubscribe from the CCPEM list, click the following link:
> > https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCPEM&A=1
> >
> > ------------------------------
> >
> > Date: Sun, 3 Jun 2018 07:18:02 -0300
> > From: Marin van Heel <[log in to unmask]>
> > Subject: Re: Converting MRC to tiff
> >
> >
> > Dear Yehuda Halfon
> >
> > The em2em converter (Image-Science.de) should do the trick but storing
> > (4-bit/8-bit?) MRC movies in tiff is not a good idea: very few
> > programs can handle tiff stacks. Moreover if the TIFFs are not
> > compressed you will not necessarily win any space. You are probably
> > best off using a standard loss-less compression program ("zip"), and
> > convert them back when you need it again. You must always keep your
> > original raw data "forever" in a loss-less form.
> >
> > Marin van Heel
> >
> >
> > On 03/06/2018 06:02, Yehuda Halfon wrote:
> >> Hi there,
> >>
> >> We have a bunch of MRC movie files the are eating at out storage, and
> >> since more are coming I was wondering if there is a good way to
> >> convert them into tiff to save space?
> >>
> >> I know that the best way is to save them directly as tif from EPU/
> >> serialEM and that is what we will to in the future. But we need to
> >> find a solution to the ones we have now.
> >>
> >> Thanks,
> >>
> >> Yehuda Halfon
> >>
> >> ---------------------------------------------------------------------
> >> ---
> >>
> >> To unsubscribe from the CCPEM list, click the following link:
> >> https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCPEM&A=1
> >>
> >
> > --
> >
> ==========================================================
> ====
> >
> > Prof Dr Ir Marin van Heel
> >
> > Laboratório Nacional de Nanotecnologia - LNNano
> > CNPEM/LNNano, Campinas, Brazil
> >
> > tel: +55-19-3518-2316
> > mobile +55-19-983455450 (current)
> > mobile +55-19-981809332
> > (041-19-981809332 TIM)
> > Skype: Marin.van.Heel
> > email: marin.vanheel(A_T)gmail.com
> > marin.vanheel(A_T)lnnano.cnpem.br
> > and: mvh.office(A_T)gmail.com
> >
> > --------------------------------------------------
> > Emeritus Professor of Cryo-EM Data Processing
> > Leiden University
> > Mobile NL: +31(0)652736618 (ALWAYS ACTIVE SMS)
> > --------------------------------------------------
> > Emeritus Professor of Structural Biology
> > Imperial College London
> > Faculty of Natural Sciences
> > email: m.vanheel(A_T)imperial.ac.uk
> > --------------------------------------------------
> >
> > I receive many emails per day and, although I try, there is no
> > guarantee that I will actually read each incoming email.
> >
> >
> >
> ##########################################################
> ############
> > ##
> >
> > To unsubscribe from the CCPEM list, click the following link:
> > https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCPEM&A=1
> >
> > ------------------------------
> >
> > Date: Sun, 3 Jun 2018 07:18:02 -0300
> > From: Marin van Heel <[log in to unmask]>
> > Subject: Re: [3dem] [ccpem] Converting MRC to tiff
> >
> > _______________________________________________
> > 3dem mailing list
> > [log in to unmask]
> > https://mail.ncmir.ucsd.edu/mailman/listinfo/3dem
> >
> >
> ##########################################################
> ############
> > ##
> >
> > To unsubscribe from the CCPEM list, click the following link:
> > https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCPEM&A=1
> >
> > ------------------------------
> >
> > Date: Sun, 3 Jun 2018 14:00:28 +0100
> > From: Dimitry Tegunov <[log in to unmask]>
> > Subject: Re: Converting MRC to tiff
> >
> > Hey!
> >
> > In addition to what Takanori wrote: If you're stuck with EPU for data
> collection, this tool can run on the K2 computer to convert gain-uncorrected
> MRCs to compressed TIFFs on-the-fly:
> https://github.com/dtegunov/stacker2. It will also happily compress your old
> files, assuming they contain only integers.
> >
> > I very much hope that one day we will get an EPU version with TIFF
> > support for K2/3 data. And with most of SerialEM's functionality. And
> > a pony. In this exact order, please ;-)
> >
> > Cheers,
> > Dimitry
> >
> >
> ##########################################################
> ############
> > ##
> >
> > To unsubscribe from the CCPEM list, click the following link:
> > https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCPEM&A=1
> >
> > ------------------------------
> >
> > Date: Sun, 3 Jun 2018 21:55:59 +0900
> > From: Tomohiro Nishizawa <[log in to unmask]>
> > Subject: Re: Converting MRC to tiff
> >
> > Dear EM community,
> >
> > I would post a somewhat related problem.
> > In our institute, Talos was recently installed and just started running.
> > I have a question about EPU file format.
> >
> > Latest version of EPU should support "unnormalized packed" TIFF
> > format, but when we selected "Unnormalized packed (with gain
> > reference) tif" at the session setup for data collection (please find
> > the attached file), EPU (or Digital Micrograph?) only wrote unnormalized
> packed "MRC" files.
> > Our EPU version is 1.10.1.1938REL.
> > From which version does EPU support packed tif format, and any special
> > settings are required to save TIFF?
> > I would be happy if anybody helps us.
> >
> >
> > Tomohiro Nishizawa
> >
> >
> >
> >
> > On 2018/06/03 19:18, Marin van Heel wrote:
> >>
> >> Dear Yehuda Halfon
> >>
> >> The em2em converter (Image-Science.de) should do the trick but
> >> storing
> >> (4-bit/8-bit?) MRC movies in tiff is not a good idea: very few
> >> programs can handle tiff stacks. Moreover if the TIFFs are not
> >> compressed you will not necessarily win any space. You are probably
> >> best off using a standard loss-less compression program ("zip"), and
> >> convert them back when you need it again. You must always keep your
> >> original raw data "forever" in a loss-less form.
> >>
> >> Marin van Heel
> >>
> >>
> >> On 03/06/2018 06:02, Yehuda Halfon wrote:
> >>> Hi there,
> >>>
> >>> We have a bunch of MRC movie files the are eating at out storage,
> >>> and since more are coming I was wondering if there is a good way to
> >>> convert them into tiff to save space?
> >>>
> >>> I know that the best way is to save them directly as tif from EPU/
> >>> serialEM and that is what we will to in the future. But we need to
> >>> find a solution to the ones we have now.
> >>>
> >>> Thanks,
> >>>
> >>> Yehuda Halfon
> >>>
> >>> --------------------------------------------------------------------
> >>> ----
> >>>
> >>> To unsubscribe from the CCPEM list, click the following link:
> >>> https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCPEM&A=1
> >>>
> >>
> >> --
> >>
> ==========================================================
> ====
> >>
> >> Prof Dr Ir Marin van Heel
> >>
> >> Laboratório Nacional de Nanotecnologia - LNNano
> >> CNPEM/LNNano, Campinas, Brazil
> >>
> >> tel: +55-19-3518-2316
> >> mobile +55-19-983455450 (current)
> >> mobile +55-19-981809332
> >> (041-19-981809332 TIM)
> >> Skype: Marin.van.Heel
> >> email: marin.vanheel(A_T)gmail.com
> >> marin.vanheel(A_T)lnnano.cnpem.br
> >> and: mvh.office(A_T)gmail.com
> >>
> >> --------------------------------------------------
> >> Emeritus Professor of Cryo-EM Data Processing
> >> Leiden University
> >> Mobile NL: +31(0)652736618 (ALWAYS ACTIVE SMS)
> >> --------------------------------------------------
> >> Emeritus Professor of Structural Biology
> >> Imperial College London
> >> Faculty of Natural Sciences
> >> email: m.vanheel(A_T)imperial.ac.uk
> >> --------------------------------------------------
> >>
> >> I receive many emails per day and, although I try, there is no
> >> guarantee that I will actually read each incoming email.
> >>
> >> ---------------------------------------------------------------------
> >> ---
> >>
> >> To unsubscribe from the CCPEM list, click the following link:
> >> https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCPEM&A=1
> >>
> >
> >
> >
> ##########################################################
> ############
> > ##
> >
> > To unsubscribe from the CCPEM list, click the following link:
> > https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCPEM&A=1
> >
> > ------------------------------
> >
> > Date: Sun, 3 Jun 2018 09:15:01 -0400
> > From: Oliver Clarke <[log in to unmask]>
> > Subject: Re: Converting MRC to tiff
> >
> > I would vote for bzip2 over compressed tiff.
> >
> > For a test 50 frame, 1.4G stack, pbzip2 gives a 214M archive, while
> > mrc2tif gives a 264M tiff - and pbzip2 is much faster to
> > compress/decompress at least on my system (GPU workstation with 24
> > cores/48 threads)
> >
> > Cheers
> > Oli
> >
> >> On Jun 3, 2018, at 9:00 AM, Dimitry Tegunov <[log in to unmask]>
> wrote:
> >>
> >> Hey!
> >>
> >> In addition to what Takanori wrote: If you're stuck with EPU for data
> collection, this tool can run on the K2 computer to convert gain-uncorrected
> MRCs to compressed TIFFs on-the-fly:
> https://github.com/dtegunov/stacker2. It will also happily compress your old
> files, assuming they contain only integers.
> >>
> >> I very much hope that one day we will get an EPU version with TIFF
> >> support for K2/3 data. And with most of SerialEM's functionality. And
> >> a pony. In this exact order, please ;-)
> >>
> >> Cheers,
> >> Dimitry
> >>
> >>
> ##########################################################
> ###########
> >> ###
> >>
> >> To unsubscribe from the CCPEM list, click the following link:
> >> https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCPEM&A=1
> >
> >
> ##########################################################
> ############
> > ##
> >
> > To unsubscribe from the CCPEM list, click the following link:
> > https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCPEM&A=1
> >
> > ------------------------------
> >
> > Date: Sun, 3 Jun 2018 13:26:59 +0000
> > From: Henning Stahlberg <[log in to unmask]>
> > Subject: Re: Converting MRC to tiff
> >
> > Hi,
> >
> > Robb McLeod studies this problem in detail, and came up with the MRCZ
> > standard, as described at
> > https://doi.org/10.1016/j.jsb.2017.11.012
> > Or
> > https://www.biorxiv.org/content/early/2017/03/13/116533
> >
> > The programs are available at
> > https://github.com/em-MRCZ
> >
> > Henning.
> >
> > Henning Stahlberg, PhD
> > Prof. for Structural Biology, C-CINA, Biozentrum, University Basel
> > Mattenstrasse 26 | D-BSSE | WRO-1058 | CH-4058 Basel | Switzerland
> > http://c-cina.org | Tel. +41-61-387 32 62
> >
> >
> >
> > On Jun 3, 2018, at 3:15 PM, Oliver Clarke
> <[log in to unmask]<mailto:[log in to unmask]>> wrote:
> >
> > I would vote for bzip2 over compressed tiff.
> >
> > For a test 50 frame, 1.4G stack, pbzip2 gives a 214M archive, while
> > mrc2tif gives a 264M tiff - and pbzip2 is much faster to
> > compress/decompress at least on my system (GPU workstation with 24
> > cores/48 threads)
> >
> > Cheers
> > Oli
> >
> > On Jun 3, 2018, at 9:00 AM, Dimitry Tegunov
> <[log in to unmask]<mailto:[log in to unmask]>> wrote:
> >
> > Hey!
> >
> > In addition to what Takanori wrote: If you're stuck with EPU for data
> collection, this tool can run on the K2 computer to convert gain-uncorrected
> MRCs to compressed TIFFs on-the-fly:
> https://github.com/dtegunov/stacker2. It will also happily compress your old
> files, assuming they contain only integers.
> >
> > I very much hope that one day we will get an EPU version with TIFF
> > support for K2/3 data. And with most of SerialEM's functionality. And
> > a pony. In this exact order, please ;-)
> >
> > Cheers,
> > Dimitry
> >
> >
> ##########################################################
> ############
> > ##
> >
> > To unsubscribe from the CCPEM list, click the following link:
> > https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCPEM&A=1
> >
> >
> ##########################################################
> ############
> > ##
> >
> > To unsubscribe from the CCPEM list, click the following link:
> > https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCPEM&A=1
> >
> >
> >
> ##########################################################
> ############
> > ##
> >
> > To unsubscribe from the CCPEM list, click the following link:
> > https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCPEM&A=1
> >
> > ------------------------------
> >
> > Date: Sun, 3 Jun 2018 21:16:49 +0100
> > From: Takanori Nakane <[log in to unmask]>
> > Subject: Re: Converting MRC to tiff
> >
> > Hi,
> >
> > Yes, for long-term 'cold' storage, (p)bzip2 can be better.
> >
> > The advantage of TIFF for processing is that it is being supported by
> > many programs (MotionCor2, cisTEM, RELION 3, etc) and
> > compression/decompression is handled transparently. Otherwise, users
> > have to decompress huge movies before processing.
> >
> > In RELION 3, our new motion refinement directly reads raw movie frames
> > (in MRCS or TIFF). You no longer have to write aligned movies and
> > movie particles. So if you record images in TIFF from the beginning,
> > you can save lot of storage.
> >
> > Best regards,
> >
> > Takanori Nakane
> >
> > On 2018/06/03 14:15, Oliver Clarke wrote:
> >> I would vote for bzip2 over compressed tiff.
> >>
> >> For a test 50 frame, 1.4G stack, pbzip2 gives a 214M archive, while
> >> mrc2tif gives a 264M tiff - and pbzip2 is much faster to
> >> compress/decompress at least on my system (GPU workstation with 24
> >> cores/48 threads)
> >>
> >> Cheers
> >> Oli
> >>
> >>> On Jun 3, 2018, at 9:00 AM, Dimitry Tegunov <[log in to unmask]>
> wrote:
> >>>
> >>> Hey!
> >>>
> >>> In addition to what Takanori wrote: If you're stuck with EPU for data
> collection, this tool can run on the K2 computer to convert gain-uncorrected
> MRCs to compressed TIFFs on-the-fly:
> https://github.com/dtegunov/stacker2. It will also happily compress your old
> files, assuming they contain only integers.
> >>>
> >>> I very much hope that one day we will get an EPU version with TIFF
> >>> support for K2/3 data. And with most of SerialEM's functionality.
> >>> And a pony. In this exact order, please ;-)
> >>>
> >>> Cheers,
> >>> Dimitry
> >>>
> >>>
> ##########################################################
> ##########
> >>> ####
> >>>
> >>> To unsubscribe from the CCPEM list, click the following link:
> >>> https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCPEM&A=1
> >>
> >>
> ##########################################################
> ###########
> >> ###
> >>
> >> To unsubscribe from the CCPEM list, click the following link:
> >> https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCPEM&A=1
> >>
> >
> >
> ##########################################################
> ############
> > ##
> >
> > To unsubscribe from the CCPEM list, click the following link:
> > https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCPEM&A=1
> >
> > ------------------------------
> >
> > Date: Sun, 3 Jun 2018 22:29:26 +0200
> > From: Jose Miguel de la Rosa Trevin
> <[log in to unmask]>
> > Subject: Re: Converting MRC to tiff
> >
> > Hi,
> >
> > I agree that TIFF seems to be a good compromise between saving disk
> > space (by its internal compression) and "processing availability" as
> > input data. Moreover, TIFF is a well established format beyond the
> > cryoEM community and other image processing tools also support it by
> > using the provided library. This is a major downside of more specific
> > solutions proposed. So, from a developer perspective, dealing with
> > TIFF files (compressed or not) will be transparent and there are many
> > more tools for users as well.
> >
> > Best,
> > Jose Miguel
> >
> >
> > On Sun, Jun 3, 2018 at 10:16 PM, Takanori Nakane
> > <[log in to unmask]>
> > wrote:
> >
> >> Hi,
> >>
> >> Yes, for long-term 'cold' storage, (p)bzip2 can be better.
> >>
> >> The advantage of TIFF for processing is that it is being supported by
> >> many programs (MotionCor2, cisTEM, RELION 3, etc) and
> >> compression/decompression is handled transparently. Otherwise, users
> >> have to decompress huge movies before processing.
> >>
> >> In RELION 3, our new motion refinement directly reads raw movie
> >> frames (in MRCS or TIFF). You no longer have to write aligned movies
> >> and movie particles. So if you record images in TIFF from the
> >> beginning, you can save lot of storage.
> >>
> >> Best regards,
> >>
> >> Takanori Nakane
> >>
> >>
> >> On 2018/06/03 14:15, Oliver Clarke wrote:
> >>
> >>> I would vote for bzip2 over compressed tiff.
> >>>
> >>> For a test 50 frame, 1.4G stack, pbzip2 gives a 214M archive, while
> >>> mrc2tif gives a 264M tiff - and pbzip2 is much faster to
> >>> compress/decompress at least on my system (GPU workstation with 24
> >>> cores/48
> >>> threads)
> >>>
> >>> Cheers
> >>> Oli
> >>>
> >>> On Jun 3, 2018, at 9:00 AM, Dimitry Tegunov <[log in to unmask]>
> wrote:
> >>>>
> >>>> Hey!
> >>>>
> >>>> In addition to what Takanori wrote: If you're stuck with EPU for
> >>>> data collection, this tool can run on the K2 computer to convert
> >>>> gain-uncorrected MRCs to compressed TIFFs on-the-fly:
> >>>> https://github.com/dtegunov/stacker2. It will also happily compress
> >>>> your old files, assuming they contain only integers.
> >>>>
> >>>> I very much hope that one day we will get an EPU version with TIFF
> >>>> support for K2/3 data. And with most of SerialEM's functionality.
> >>>> And a pony. In this exact order, please ;-)
> >>>>
> >>>> Cheers,
> >>>> Dimitry
> >>>>
> >>>>
> ##########################################################
> #########
> >>>> #####
> >>>>
> >>>> To unsubscribe from the CCPEM list, click the following link:
> >>>> https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCPEM&A=1
> >>>>
> >>>
> >>>
> ##########################################################
> ##########
> >>> ####
> >>>
> >>> To unsubscribe from the CCPEM list, click the following link:
> >>> https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCPEM&A=1
> >>>
> >>>
> >>
> ##########################################################
> ###########
> >> ###
> >>
> >> To unsubscribe from the CCPEM list, click the following link:
> >> https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCPEM&A=1
> >>
> >
> >
> ##########################################################
> ############
> > ##
> >
> > To unsubscribe from the CCPEM list, click the following link:
> > https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCPEM&A=1
> >
> > ------------------------------
> >
> > Date: Sun, 3 Jun 2018 21:52:29 +0000
> > From: "Ludtke, Steven J" <[log in to unmask]>
> > Subject: Re: Converting MRC to tiff
> >
> > A STRONG vote for compressed TIFF stacks for raw movies:
> >
> > I think the 8 bit (losslessly) compressed TIFF stack has been emerging as
> the dominant standard for raw frames for some time. While it may not have
> achieved a majority, at least I think it's achieved a plurality. Advantages:
> > a) standard format with native compression AND stacks supported
> > b) widely supported (EMAN2 has also supported it for quite some time,
> > and can trivially convert to any other cryoEM format)
> > c) while compression is slightly worse than you can achieve with
> > bzip2, it is MUCH faster
> > d) support for 16 and 32 bits if people really feel they need it
> > e) for developers, both compression and stacks are supported by the
> > standard libTIFF
> >
> > The alternatives each have substantial downsides:
> > Compressed filesystem (ZFS, BTRFS):
> > - none of these are really considered completely stable
> > - generally quite slow compared to the approach above
> > - compression also not as good as bzip2
> >
> > "normal" MRC files with separate bzip2/gzip/zip compression:
> > - compression is very slow and only marginally better than the
> > faster/simpler compression in libTIFF
> > - turns operations into a 2-step process requiring extra disk space in
> > most cases
> >
> > MRCZ files:
> > - despite the apparent simplicity of the the MRC format, due to the
> number of variants and extensions in the wild, the MRC reader/writer in
> EMAN2 is the most complex code by a fair margin, even when compared
> against the ostensibly trickier to code formats like HDF5. Any time anyone
> proposes another change/extension to MRC, I cringe in fear.
> > - WHY? it is completely un-necessary, and has zero advantages over the
> > already widely supported 8 bit compressed TIFF stacks
> >
> > I think really then only thing you can't do easily with TIFF is stacks of 3-D
> volumes. That is, as long as use is limited to storing raw camera movie data,
> there really isn't a problem. You can still use the other format(s) for other
> purposes.
> >
> > --------------------------------------------------------------------------------------
> > Steven Ludtke, Ph.D. <[log in to unmask]<mailto:[log in to unmask]>>
> Baylor College of Medicine
> > Charles C. Bell Jr., Professor of Structural Biology
> > Dept. of Biochemistry and Molecular Biology
> (www.bcm.edu/biochem<http://www.bcm.edu/biochem>)
> > Academic Director, CryoEM Core
> (cryoem.bcm.edu<http://cryoem.bcm.edu>)
> > Co-Director CIBR Center
> (www.bcm.edu/research/cibr<http://www.bcm.edu/research/cibr>)
> > Co-Director National Center For Macromolecular Imaging
> (ncmi.bcm.edu<http://ncmi.bcm.edu>)
> >
> >
> >
> > On Jun 3, 2018, at 3:29 PM, Jose Miguel de la Rosa Trevin
> <[log in to unmask]<mailto:delarosatrevin+ccpem@GMAI
> L.COM>> wrote:
> >
> > ***CAUTION:*** This email is not from a BCM Source. Only click links or
> open attachments you know are safe.
> > ________________________________
> > Hi,
> >
> > I agree that TIFF seems to be a good compromise between saving disk
> > space (by its internal compression) and "processing availability" as
> > input data. Moreover, TIFF is a well established format beyond the
> > cryoEM community and other image processing tools also support it by
> > using the provided library. This is a major downside of more specific
> > solutions proposed. So, from a developer perspective, dealing with
> > TIFF files (compressed or not) will be transparent and there are many
> > more tools for users as well.
> >
> > Best,
> > Jose Miguel
> >
> >
> > On Sun, Jun 3, 2018 at 10:16 PM, Takanori Nakane <tnakane@mrc-
> lmb.cam.ac.uk<mailto:[log in to unmask]>> wrote:
> > Hi,
> >
> > Yes, for long-term 'cold' storage, (p)bzip2 can be better.
> >
> > The advantage of TIFF for processing is that it is being supported by
> > many programs (MotionCor2, cisTEM, RELION 3, etc) and
> > compression/decompression is handled transparently. Otherwise, users
> > have to decompress huge movies before processing.
> >
> > In RELION 3, our new motion refinement directly reads raw movie frames
> > (in MRCS or TIFF). You no longer have to write aligned movies and
> > movie particles. So if you record images in TIFF from the beginning,
> > you can save lot of storage.
> >
> > Best regards,
> >
> > Takanori Nakane
> >
> >
> > On 2018/06/03 14:15, Oliver Clarke wrote:
> > I would vote for bzip2 over compressed tiff.
> >
> > For a test 50 frame, 1.4G stack, pbzip2 gives a 214M archive, while
> > mrc2tif gives a 264M tiff - and pbzip2 is much faster to
> > compress/decompress at least on my system (GPU workstation with 24
> > cores/48 threads)
> >
> > Cheers
> > Oli
> >
> > On Jun 3, 2018, at 9:00 AM, Dimitry Tegunov
> <[log in to unmask]<mailto:[log in to unmask]>> wrote:
> >
> > Hey!
> >
> > In addition to what Takanori wrote: If you're stuck with EPU for data
> collection, this tool can run on the K2 computer to convert gain-uncorrected
> MRCs to compressed TIFFs on-the-fly:
> https://github.com/dtegunov/stacker2<https://urldefense.proofpoint.com/
> v2/url?u=https-3A__github.com_dtegunov_stacker2&d=DwMFaQ&c=ZQs-
> KZ8oxEw0p81sqgiaRA&r=GWA2IF6nkq8sZMXHpp1Xpg&m=ZrfP_Oel0_nDGlC
> hKOIgHlr_YeMmIYu9QPtGC3za4bs&s=ona5_3oZyUK9rWTa-
> TiqE_4OnQdg5ODev6huiRlCbnQ&e=>. It will also happily compress your old
> files, assuming they contain only integers.
> >
> > I very much hope that one day we will get an EPU version with TIFF
> > support for K2/3 data. And with most of SerialEM's functionality. And
> > a pony. In this exact order, please ;-)
> >
> > Cheers,
> > Dimitry
> >
> >
> ##########################################################
> ############
> > ##
> >
> > To unsubscribe from the CCPEM list, click the following link:
> > https://www.jiscmail.ac.uk/cgi-
> bin/webadmin?SUBED1=CCPEM&A=1<https://u
> > rldefense.proofpoint.com/v2/url?u=https-3A__www.jiscmail.ac.uk_cgi-
> 2Db
> > in_webadmin-3FSUBED1-3DCCPEM-26A-3D1&d=DwMFaQ&c=ZQs-
> KZ8oxEw0p81sqgiaRA
> >
> &r=GWA2IF6nkq8sZMXHpp1Xpg&m=ZrfP_Oel0_nDGlChKOIgHlr_YeMmIYu9
> QPtGC3za4b
> > s&s=MLe6yyAjgih-X8kwVAuHwhkEVJ-EXYSIJ385Zty8TTU&e=>
> >
> >
> ##########################################################
> ############
> > ##
> >
> > To unsubscribe from the CCPEM list, click the following link:
> > https://www.jiscmail.ac.uk/cgi-
> bin/webadmin?SUBED1=CCPEM&A=1<https://u
> > rldefense.proofpoint.com/v2/url?u=https-3A__www.jiscmail.ac.uk_cgi-
> 2Db
> > in_webadmin-3FSUBED1-3DCCPEM-26A-3D1&d=DwMFaQ&c=ZQs-
> KZ8oxEw0p81sqgiaRA
> >
> &r=GWA2IF6nkq8sZMXHpp1Xpg&m=ZrfP_Oel0_nDGlChKOIgHlr_YeMmIYu9
> QPtGC3za4b
> > s&s=MLe6yyAjgih-X8kwVAuHwhkEVJ-EXYSIJ385Zty8TTU&e=>
> >
> >
> >
> ##########################################################
> ############
> > ##
> >
> > To unsubscribe from the CCPEM list, click the following link:
> > https://www.jiscmail.ac.uk/cgi-
> bin/webadmin?SUBED1=CCPEM&A=1<https://u
> > rldefense.proofpoint.com/v2/url?u=https-3A__www.jiscmail.ac.uk_cgi-
> 2Db
> > in_webadmin-3FSUBED1-3DCCPEM-26A-3D1&d=DwMFaQ&c=ZQs-
> KZ8oxEw0p81sqgiaRA
> >
> &r=GWA2IF6nkq8sZMXHpp1Xpg&m=ZrfP_Oel0_nDGlChKOIgHlr_YeMmIYu9
> QPtGC3za4b
> > s&s=MLe6yyAjgih-X8kwVAuHwhkEVJ-EXYSIJ385Zty8TTU&e=>
> >
> >
> > ________________________________
> >
> > To unsubscribe from the CCPEM list, click the following link:
> > https://www.jiscmail.ac.uk/cgi-
> bin/webadmin?SUBED1=CCPEM&A=1<https://u
> > rldefense.proofpoint.com/v2/url?u=https-3A__www.jiscmail.ac.uk_cgi-
> 2Db
> > in_webadmin-3FSUBED1-3DCCPEM-26A-3D1&d=DwMFaQ&c=ZQs-
> KZ8oxEw0p81sqgiaRA
> >
> &r=GWA2IF6nkq8sZMXHpp1Xpg&m=ZrfP_Oel0_nDGlChKOIgHlr_YeMmIYu9
> QPtGC3za4b
> > s&s=MLe6yyAjgih-X8kwVAuHwhkEVJ-EXYSIJ385Zty8TTU&e=>
> >
> >
> >
> ##########################################################
> ############
> > ##
> >
> > To unsubscribe from the CCPEM list, click the following link:
> > https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCPEM&A=1
> >
> > ------------------------------
> >
> > End of CCPEM Digest - 1 Jun 2018 to 3 Jun 2018 (#2018-136)
> >
> **********************************************************
>
> ##########################################################
> ##############
>
> To unsubscribe from the CCPEM list, click the following link:
> https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCPEM&A=1
########################################################################
To unsubscribe from the CCPEM list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCPEM&A=1
|
