Hi,
Thank you very much for valuable feedback.
> Do you think you can apply this correction or something similar on
the next versions of Relion, please?
Sure. I applied this change.
Best regards,
Takanori Nakane
On 2018/09/28 19:46, Coudray, Nicolas wrote:
> Hi,
>
>
> To follow up with the issues we had with empty or non-existent logfile.pdf in Relion3. Our HPC Team has found where the issue seems to come from: it could be due to 'synchronization' issues that may only appear when very high speed hardware is used.
>
>
> To get around this issue, they have modified CPlot2D.cpp and just remove the "&" at the end of line 56 (command += " > /dev/null &";). This seems to solve the problem and all logfiles seem to be generated properly now.
>
>
> Do you think you can apply this correction or something similar on the next versions of Relion, please?
>
>
> Thanks,
>
> Best,
>
> Nicolas
>
>
>
> ________________________________
> From: Collaborative Computational Project in Electron cryo-Microscopy <[log in to unmask]> on behalf of Coudray, Nicolas <[log in to unmask]>
> Sent: Friday, September 14, 2018 4:05 PM
> To: [log in to unmask]
> Subject: Re: [ccpem] Empty logfile.pdf with Relion 3 on the tutorial dataset
>
>
> Thanks Takanori for the input.
>
>
> Do you think installing an older/newer version of gs would help? Which one are you using?
>
>
> What bugs me is that it only occurs with these two jobs (MotionCor, Post-process) - the output log files are all ok for the other jobs tested (ctf, auto-pick and ctfrefine). Is there anything in the way those pdf are generated that could explain the behavior or figure out what's the file system issue?
>
>
> Thanks,
>
> Best,
>
> Nicolas
>
>
>
> ________________________________
> From: Takanori Nakane <[log in to unmask]>
> Sent: Friday, September 14, 2018 12:42 PM
> To: Coudray, Nicolas
> Cc: [log in to unmask]
> Subject: Re: [ccpem] Empty logfile.pdf with Relion 3 on the tutorial dataset
>
> Hi,
>
> This sounds like a file system issue. That is, ghostscript
> does not 'see' the EPS files RELION just created, for some reason.
>
> In past, we had a bug where multiple threads of CtfRefine wrote to
> a same file, sometimes corrupting the output. This bug was fixed on July 9th,
> one month before we went to public beta testing.
>
>> * when the job fails, if we tried to use the gs command to convert the
>> intermediate eps files to pdf, it works
>
> This means EPS outputs are fine.
> If EPS files are broken, ghostscript complains.
>
>> * The post-processing job remains stuck in the "running job" section of
>> the GUI and no logfile.pdf is produced either.
>
> The GUI thinks the job is still running because logfile.pdf is not there.
> Since post-processing does not use multi-threading, this also excludes the
> possibility of the above mentioned bug.
>
> Best regards,
>
> Takanori Nakane
>
>> Hi -
>>
>> Thanks a lot for the feedback, Sjors - We have made some more tests and
>> these are the observations:
>>
>> * the MotionCor2 jobs sometimes fails to produce the final logfile.pdf:
>> the file exists but is empty or very small (a few kb, and cannot be
>> opened; "The file has been damaged" message obtained)
>>
>> * When the generation of the logfile fails, no error message is seen in
>> either run.out or run.err. The run.out still ends with this message
>>
>> " Generating logfile.pdf ...
>>
>> 0/ 0 sec
>> ............................................................~~(,_,">
>> Done! Written: MotionCorr/job055/logfile.pdf and
>> MotionCorr/job055/corrected_micrographs.star
>> Program finished with exit code 0 at: Thu Sep 13 16:17:45 EDT 2018"
>>
>> * We tried on different nodes (we have some with 4 or 8 gpus). For some
>> reasons, some of the jobs do generate the logfile.pdf properly (it seems
>> random, though it tends to fail more on nodes with 4 gpus). In all cases,
>> if we check the intermediate eps files, they look the same and correct.
>>
>> * when the job fails, if we tried to use the gs command to convert the
>> intermediate eps files to pdf, it works
>>
>> * if we run the jobs directly from the command line as suggested, the
>> logfile is created properly (command line corresponding to the job that
>> failed when submitted to the queue and by logging into the same node
>> before running the command)
>>
>>
>> Other behavior we have observed and which might be related:
>>
>> * The post-processing job remains stuck in the "running job" section of
>> the GUI and no logfile.pdf is produced either. There is no error message
>> seen in run.err and run.out ends with:
>>
>> " + FINAL RESOLUTION: 3.62045
>>
>> Program finished with exit code 0 at: Fri Sep 14 11:25:42 EDT 2018"
>>
>> When I check the list of jobs running on the cluster, the post-processing
>> job ID cannot be seen and if I log into the node, no relion job is running
>> anymore.
>>
>>
>> We're using gs 9.07, if that matters.
>>
>>
>> If anyone has any idea, thanks for letting us know.
>>
>>
>> Thanks,
>>
>> Best,
>>
>> Nicolas
>>
>>
>>
>> ________________________________
>> From: Sjors Scheres <[log in to unmask]>
>> Sent: Saturday, September 8, 2018 1:00 PM
>> To: Coudray, Nicolas
>> Cc: [log in to unmask]
>> Subject: Re: [ccpem] Empty logfile.pdf with Relion 3 on the tutorial
>> dataset
>>
>> Hi Nicolas,
>> RELION will make a system call to "gs" to combine the EPS files into a
>> single logfile.pdf. If it fails, it should write to stderr. Did you see
>> such an error in the run.err of those directories? You could try executing
>> the command from the command line to find out what the problem on your
>> system is.
>> HTH,
>> Sjors
>>
>> PS: I am pasting the relevant code from src/CPlot2D.cpp in case it helps
>> below:
>>
>>
>>
>> FileName fn_list = fn_pdf + ".lst";
>> std::string command = "gs -sDEVICE=pdfwrite -dNOPAUSE -dBATCH -dSAFER
>> -dDEVICEWIDTHPOINTS=800 -dDEVICEHEIGHTPOINTS=800 -sOutputFile=";
>> command += fn_pdf + " @" + fn_list;
>> std::ofstream filelist(fn_pdf + ".lst");
>> bool have_at_least_one = false;
>> for (int i = 0; i < fn_eps.size(); i++)
>> {
>> // fn_eps[i] could be a Linux wildcard...
>> std::vector<FileName> all_eps_files;
>> fn_eps[i].globFiles(all_eps_files);
>> fn_eps[i].globFiles(all_eps_files);
>> for (long int j= 0; j < all_eps_files.size(); j++)
>> {
>> if (exists(all_eps_files[j]))
>> {
>> filelist << all_eps_files[j] << "\n";
>> have_at_least_one = true;
>> }
>> }
>> }
>> filelist.close();
>>
>>
>>
>>> Hi -
>>>
>>>
>>> We are currently testing Relion 3 and have some weird behavior with the
>>> pdf files generated during the motion-correction they are empty.
>>>
>>>
>>> We are using the tutorial dataset and we ran several very similar
>>> Motion
>>> Correction jobs; 11 out of 15 runs gave empty "logfile.pdf" file (it
>>> seems to not depend on the number of GPU or CPUs - it happens for both
>>> motioncor and "relioncor" - main parameters in the I/O and Motion tabs
>>> were not changed). Also, the eps files of the movies all seem ok though
>>> (non-empty, can be converted to pdf with ps2pdf and can be open
>>> properly) and none of the run generate any error message.
>>>
>>>
>>> Does anyone know how to solve this issue or how we could track down
>>> its
>>> origin?
>>>
>>>
>>> Thanks,
>>>
>>> Best,
>>>
>>> Nicolas
>>>
>>> ------------------------------------------------------------
>>> This email message, including any attachments, is for the sole use of
>>> the
>>> intended recipient(s) and may contain information that is proprietary,
>>> confidential, and exempt from disclosure under applicable law. Any
>>> unauthorized review, use, disclosure, or distribution is prohibited. If
>>> you have received this email in error please notify the sender by return
>>> email and delete the original message. Please note, the recipient should
>>> check this email and any attachments for the presence of viruses. The
>>> organization accepts no liability for any damage caused by any virus
>>> transmitted by this email.
>>> =================================
>>>
>>> ########################################################################
>>>
>>> To unsubscribe from the CCPEM list, click the following link:
>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__www.jiscmail.ac.uk_cgi-2Dbin_webadmin-3FSUBED1-3DCCPEM-26A-3D1&d=DwIDAg&c=j5oPpO0eBH1iio48DtsedbOBGmuw5jHLjgvtN2r4ehE&r=RRJopfWEMpjEmb1q4aEqz5Y4zjDaSRnr5kurG-GlA7g&m=YpneSos6s6bBDAyHgYauYwKyJrAKiYvYgRFdrfD6G1s&s=EXXDqcXpySIlYfZ9IR88mVlQSU0D7Q6l3lTgGc0ShZ4&e=
>>>
>>
>>
>> --
>> Sjors Scheres
>> MRC Laboratory of Molecular Biology
>> Francis Crick Avenue, Cambridge Biomedical Campus
>> Cambridge CB2 0QH, U.K.
>> tel: +44 (0)1223 267061
>> https://urldefense.proofpoint.com/v2/url?u=http-3A__www2.mrc-2Dlmb.cam.ac.uk_groups_scheres&d=DwIDAg&c=j5oPpO0eBH1iio48DtsedbOBGmuw5jHLjgvtN2r4ehE&r=RRJopfWEMpjEmb1q4aEqz5Y4zjDaSRnr5kurG-GlA7g&m=YpneSos6s6bBDAyHgYauYwKyJrAKiYvYgRFdrfD6G1s&s=sjEthfBCV6DItq1Iv_xaDpwaNkDcXKZ36-OzXzijo_0&e=
>>
>>
>> ------------------------------------------------------------
>> This email message, including any attachments, is for the sole use of the
>> intended recipient(s) and may contain information that is proprietary,
>> confidential, and exempt from disclosure under applicable law. Any
>> unauthorized review, use, disclosure, or distribution is prohibited. If
>> you have received this email in error please notify the sender by return
>> email and delete the original message. Please note, the recipient should
>> check this email and any attachments for the presence of viruses. The
>> organization accepts no liability for any damage caused by any virus
>> transmitted by this email.
>> =================================
>>
>> ########################################################################
>>
>> To unsubscribe from the CCPEM list, click the following link:
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__www.jiscmail.ac.uk_cgi-2Dbin_webadmin-3FSUBED1-3DCCPEM-26A-3D1&d=DwIDJg&c=j5oPpO0eBH1iio48DtsedbOBGmuw5jHLjgvtN2r4ehE&r=RRJopfWEMpjEmb1q4aEqz5Y4zjDaSRnr5kurG-GlA7g&m=h7mUXCz1RqEYxR3jGkIkAEEHXFm468tt681bqYxrQks&s=OyV0lwjBolYsZpRVGk7vgfMVU_GshmSyDtWOTfiPGPY&e=
>>
>
>
>
> ------------------------------------------------------------
> This email message, including any attachments, is for the sole use of the intended recipient(s) and may contain information that is proprietary, confidential, and exempt from disclosure under applicable law. Any unauthorized review, use, disclosure, or distribution is prohibited. If you have received this email in error please notify the sender by return email and delete the original message. Please note, the recipient should check this email and any attachments for the presence of viruses. The organization accepts no liability for any damage caused by any virus transmitted by this email.
> =================================
>
> ________________________________
>
> To unsubscribe from the CCPEM list, click the following link:
> https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCPEM&A=1<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.jiscmail.ac.uk_cgi-2Dbin_webadmin-3FSUBED1-3DCCPEM-26A-3D1&d=DwMFAw&c=j5oPpO0eBH1iio48DtsedbOBGmuw5jHLjgvtN2r4ehE&r=RRJopfWEMpjEmb1q4aEqz5Y4zjDaSRnr5kurG-GlA7g&m=O3vaEUxHAWFdAiXgqrJPKwBvtLZiijXoB9gXRgkPV54&s=6-OGyPMls25ae_s5hIlqSoOyDpgYItflF4-jpOSkQ8E&e=>
>
> ------------------------------------------------------------
> This email message, including any attachments, is for the sole use of the intended recipient(s) and may contain information that is proprietary, confidential, and exempt from disclosure under applicable law. Any unauthorized review, use, disclosure, or distribution is prohibited. If you have received this email in error please notify the sender by return email and delete the original message. Please note, the recipient should check this email and any attachments for the presence of viruses. The organization accepts no liability for any damage caused by any virus transmitted by this email.
> =================================
>
> ########################################################################
>
> To unsubscribe from the CCPEM list, click the following link:
> https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCPEM&A=1
>
########################################################################
To unsubscribe from the CCPEM list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCPEM&A=1
|
