Hi,
> I suspect this is because the movies are not where the program expects them to be. If so, can I fix this with symbolic links? The Fractions.star
files in both the motion correction job directories point to their respective movies via the same symbolic link "Movies", so I can't figure out a way
of using a data set specific link short of editing all the individual Fractions.star files - maybe there is a clever way to do that?
This should work:
mkdir Movies
cd Movies
ln -s /absolute/path/to/dataset1/Movies/*.tif .
ln -s /absolute/path/to/dataset2/Movies/*.tif .
If the file names collide, you have to edit STAR files or polish dataset1,
replace the links and then polish dataset2.
Takanori
On 2021/01/15 12:58, David Lawson (JIC) wrote:
> Hi Takanori
>
> I've just realised that I screwed up - apologies!
>
> I was originally comparing the format of the corrected_micrographs.star file with the output for the join step, when in fact I had imported the corrected movies to the project, rather than the corrected_micrographs.star file. So the extra columns were already missing from my input files which came from the import jobs - doh!
>
> Now I have a join_mics.star file that looks correct at last. I have also checked that the paths to:
> _rlnCtfPowerSpectrum #1
> _rlnMicrographName #2
> _rlnMicrographMetadata #3
> ...are correct for both datasets
>
> However, now my polish job is failing with:
> "ERROR
> There is no movie metadata STAR file for any micrographs"
>
> I suspect this is because the movies are not where the program expects them to be. If so, can I fix this with symbolic links? The Fractions.star files in both the motion correction job directories point to their respective movies via the same symbolic link "Movies", so I can't figure out a way of using a data set specific link short of editing all the individual Fractions.star files - maybe there is a clever way to do that?
>
> Thanks again
>
> Dave
>
> -------------------------------
>
> Prof. David M. Lawson
> Department of Biological Chemistry,
> John Innes Centre,
> Norwich,
> NR4 7UH, UK.
> Tel: +44-(0)1603-450725
> Web: https://www.jic.ac.uk/people/david-lawson
> Email: [log in to unmask]
>
> -----Original Message-----
> From: Takanori Nakane <[log in to unmask]>
> Sent: 15 January 2021 10:15
> To: David Lawson (JIC) <[log in to unmask]>; [log in to unmask]
> Subject: Re: [ccpem] missing rlnMicrographMetadata column after joining star files
>
> Hi,
>
> Usually you don't need it but to make sure your new line codes are LF, you can try runing dos2unix on micrograph star files.
>
> Takanori
>
> On 2021/01/15 10:10, David Lawson (JIC) wrote:
>> Hi Takanori,
>>
>> I am using the GUI, do I need to worry about dos2unix? If so what do I run it on?
>>
>>> Run two Polish jobs with each STAR file and then combine two shiny.star files.
>>
>> Nice tip - I will try that.
>>
>> Thanks again
>>
>> Dave
>>
>> -------------------------------
>>
>> Prof. David M. Lawson
>> Department of Biological Chemistry,
>> John Innes Centre,
>> Norwich,
>> NR4 7UH, UK.
>> Tel: +44-(0)1603-450725
>> Web: https://www.jic.ac.uk/people/david-lawson
>> Email: [log in to unmask]
>>
>> -----Original Message-----
>> From: Takanori Nakane <[log in to unmask]>
>> Sent: 15 January 2021 10:05
>> To: David Lawson (JIC) <[log in to unmask]>; [log in to unmask]
>> Subject: Re: [ccpem] missing rlnMicrographMetadata column after
>> joining star files
>>
>> Hi,
>>
>> Probably your input is weird. Did you try "dos2unix" to make sure the new line code?
>>
>> > I'm now thinking that I should combine the movie datasets and re-run the motion correction - does that sound sensible?
>>
>> You don't have to.
>>
>> Polish will process only particles that are present in the given motion correction STAR file.
>> Run two Polish jobs with each STAR file and then combine two shiny.star files.
>>
>> Best regards,
>>
>> Takanori Nakane
>>
>> On 2021/01/15 10:01, David Lawson (JIC) wrote:
>>> Hi Takanori,
>>>
>>> I ran your test files and the output file is identical to the one you provided. Maybe there is an issue with my input files?
>>>
>>> The only editing I have done in windows was pasting the note.txt file into my previous email.
>>>
>>> I'm now thinking that I should combine the movie datasets and re-run the motion correction - does that sound sensible?
>>>
>>> Thanks again,
>>>
>>> Dave
>>>
>>> -------------------------------
>>>
>>> Prof. David M. Lawson
>>> Department of Biological Chemistry,
>>> John Innes Centre,
>>> Norwich,
>>> NR4 7UH, UK.
>>> Tel: +44-(0)1603-450725
>>> Web: https://www.jic.ac.uk/people/david-lawson
>>> Email: [log in to unmask]
>>>
>>> -----Original Message-----
>>> From: Takanori Nakane <[log in to unmask]>
>>> Sent: 15 January 2021 09:41
>>> To: David Lawson (JIC) <[log in to unmask]>; CCPEM
>>> <[log in to unmask]>
>>> Subject: Re: [ccpem] missing rlnMicrographMetadata column after
>>> joining star files
>>>
>>> Hi,
>>>
>>> I uploaded my tests to ftp://ftp.mrc-lmb.cam.ac.uk/pub/tnakane/join-star-bug/.
>>>
>>> relion_star_handler --o join.star --combine --i "mic1.star mic2.star"
>>>
>>> runs fine.
>>>
>>> Just to make sure, didn't you edit the files in Windows?
>>> Make sure the new line code remains LF, not CR+LF.
>>>
>>> Best regards,
>>>
>>> Takanori Nakane
>>>
>>> On 2021/01/15 9:22, David Lawson (JIC) wrote:
>>>> Hi Takanori,
>>>>
>>>> I am now running 3.3.1 and I get the same problem. Here is the note.txt file in case that helps:
>>>>
>>>> ++++ Executing new job on Fri Jan 15 08:42:21 2021
>>>> ++++ with the following command(s):
>>>> `which relion_star_handler` --combine --i " Import/job004/micrographs.star Import/job005/micrographs.star " --check_duplicates rlnMicrographName --o JoinStar/job076/join_mics.star --pipeline_control JoinStar/job076/
>>>> ++++
>>>>
>>>> Thanks
>>>>
>>>> Dave
>>>>
>>>> -------------------------------
>>>>
>>>> Prof. David M. Lawson
>>>> Department of Biological Chemistry,
>>>> John Innes Centre,
>>>> Norwich,
>>>> NR4 7UH, UK.
>>>> Tel: +44-(0)1603-450725
>>>> Web: https://www.jic.ac.uk/people/david-lawson
>>>> Email: [log in to unmask]
>>>>
>>>> -----Original Message-----
>>>> From: Takanori Nakane <[log in to unmask]>
>>>> Sent: 13 January 2021 20:54
>>>> To: David Lawson (JIC) <[log in to unmask]>;
>>>> [log in to unmask]
>>>> Subject: Re: [ccpem] missing rlnMicrographMetadata column after
>>>> joining star files
>>>>
>>>> Hi,
>>>>
>>>> > relion-3.1-beta
>>>>
>>>> Please try with the latest version (3.1.1).
>>>>
>>>> I could not reproduce your problem locally.
>>>>
>>>> Takanori
>>>>
>>>> On 2021/01/13 18:06, David Lawson (JIC) wrote:
>>>>> Hi All,
>>>>>
>>>>> I have combined two datasets with "Join star files" (via the
>>>>> relion-3.1-beta GUI) after they were motion corrected separately.
>>>>> This has enabled me to get as far as postprocessing and CTF
>>>>> refinement, but I cannot do Bayesian Polishing because the
>>>>> rlnMicrographMetadata column (along with several
>>>>> others) was lost in the joining step.
>>>>>
>>>>> The individual corrected_micrographs.star files look like this:
>>>>>
>>>>> # version 30001
>>>>>
>>>>> data_optics
>>>>>
>>>>> loop_
>>>>>
>>>>> _rlnOpticsGroupName #1
>>>>>
>>>>> _rlnOpticsGroup #2
>>>>>
>>>>> _rlnMicrographOriginalPixelSize #3
>>>>>
>>>>> _rlnVoltage #4
>>>>>
>>>>> _rlnSphericalAberration #5
>>>>>
>>>>> _rlnAmplitudeContrast #6
>>>>>
>>>>> _rlnMicrographPixelSize #7
>>>>>
>>>>> opticsGroup2 1 1.065000 300.000000 2.700000
>>>>> 0.100000 1.065000
>>>>>
>>>>> # version 30001
>>>>>
>>>>> data_micrographs
>>>>>
>>>>> loop_
>>>>>
>>>>> _rlnCtfPowerSpectrum #1
>>>>>
>>>>> _rlnMicrographName #2
>>>>>
>>>>> _rlnMicrographMetadata #3
>>>>>
>>>>> _rlnOpticsGroup #4
>>>>>
>>>>> _rlnAccumMotionTotal #5
>>>>>
>>>>> _rlnAccumMotionEarly #6
>>>>>
>>>>> _rlnAccumMotionLate #7
>>>>>
>>>>> MotionCorr/job002/Movies/GridSquare_7596976/Data/FoilHole_7601624_D
>>>>> a t a _7600266_7600267_20190409_1543_Fractions_PS.mrc Mo
>>>>>
>>>>> tionCorr/job002/Movies/GridSquare_7596976/Data/FoilHole_7601624_Dat
>>>>> a
>>>>> _
>>>>> 7 600266_7600267_20190409_1543_Fractions.mrc MotionC
>>>>>
>>>>> orr/job002/Movies/GridSquare_7596976/Data/FoilHole_7601624_Data_760
>>>>> 0
>>>>> 2
>>>>> 6 6_7600267_20190409_1543_Fractions.star
>>>>>
>>>>> .etc.
>>>>>
>>>>> ..and the merged one looks like this:
>>>>>
>>>>> # version 30001
>>>>>
>>>>> data_optics
>>>>>
>>>>> loop_
>>>>>
>>>>> _rlnOpticsGroupName #1
>>>>>
>>>>> _rlnOpticsGroup #2
>>>>>
>>>>> _rlnMicrographPixelSize #3
>>>>>
>>>>> _rlnMicrographOriginalPixelSize #4
>>>>>
>>>>> _rlnVoltage #5
>>>>>
>>>>> _rlnSphericalAberration #6
>>>>>
>>>>> _rlnAmplitudeContrast #7
>>>>>
>>>>> opticsGroup1 1 1.065000 1.065000 300.000000
>>>>> 2.700000 0.100000
>>>>>
>>>>> opticsGroup2 2 1.065000 1.065000 300.000000
>>>>> 2.700000 0.100000
>>>>>
>>>>> # version 30001
>>>>>
>>>>> data_micrographs
>>>>>
>>>>> loop_
>>>>>
>>>>> _rlnMicrographName #1
>>>>>
>>>>> _rlnOpticsGroup #2
>>>>>
>>>>> MotionCorr_08-APR-2019/GridSquare_7596976/Data/FoilHole_7601624_Dat
>>>>> a
>>>>> _
>>>>> 7 600266_7600267_20190409_1543_Fractions.mrc
>>>>>
>>>>> 1
>>>>>
>>>>> .etc.
>>>>>
>>>>> Is there a way to run "Join star files" that retains these columns, or do I need to join the two sets of movies and re-run the motion correction?
>>>>>
>>>>> Thanks in advance,
>>>>>
>>>>> Dave
>>>>>
>>>>> -------------------------------
>>>>>
>>>>> Prof. David M. Lawson
>>>>>
>>>>> Department of Biological Chemistry,
>>>>>
>>>>> John Innes Centre,
>>>>>
>>>>> Norwich,
>>>>>
>>>>> NR4 7UH, UK.
>>>>>
>>>>> Tel: +44-(0)1603-450725
>>>>>
>>>>> Web: https://www.jic.ac.uk/people/david-lawson
>>>>>
>>>>> Email: [log in to unmask]
>>>>>
>>>>>
>>>>> -------------------------------------------------------------------
>>>>> -
>>>>> -
>>>>> -
>>>>> -------------------------------------------------------------------
>>>>> -
>>>>> -
>>>>> -
>>>>> ----------
>>>>>
>>>>> To unsubscribe from the CCPEM list, click the following link:
>>>>> https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCPEM&A=1
>>>>> <https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCPEM&A=1>
>>>>>
>>>>
>>>
>>>
>>> #####################################################################
>>> #
>>> ##
>>>
>>> To unsubscribe from the CCPEM list, click the following link:
>>> https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCPEM&A=1
>>>
>>> This message was issued to members of www.jiscmail.ac.uk/CCPEM, a
>>> mailing list hosted by www.jiscmail.ac.uk, terms & conditions are
>>> available at https://www.jiscmail.ac.uk/policyandsecurity/
>>>
>>
>>
>> ######################################################################
>> ##
>>
>> To unsubscribe from the CCPEM list, click the following link:
>> https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCPEM&A=1
>>
>> This message was issued to members of www.jiscmail.ac.uk/CCPEM, a
>> mailing list hosted by www.jiscmail.ac.uk, terms & conditions are
>> available at https://www.jiscmail.ac.uk/policyandsecurity/
>>
>
>
> ########################################################################
>
> To unsubscribe from the CCPEM list, click the following link:
> https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCPEM&A=1
>
> This message was issued to members of www.jiscmail.ac.uk/CCPEM, a mailing list hosted by www.jiscmail.ac.uk, terms & conditions are available at https://www.jiscmail.ac.uk/policyandsecurity/
>
########################################################################
To unsubscribe from the CCPEM list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCPEM&A=1
This message was issued to members of www.jiscmail.ac.uk/CCPEM, a mailing list hosted by www.jiscmail.ac.uk, terms & conditions are available at https://www.jiscmail.ac.uk/policyandsecurity/
|
