Hi John,
OK, we will look into this.
Sjors
On 10/7/21 8:57 PM, John Monroe Heumann wrote:
> Hi Sjors, et al.
>
> Let me expand on this a little further. Forget
> practical implementation details for a moment and just think about
> this from a UI design standpoint. Should users be able to change job
> settings and re-run? That's such an obvious yes it's not even worth
> discussing. Does doing so raise consistency issues if there are
> already dependencies on the output? Of course. But rather than
> forbidding overwriting, there are better alternatives. One non-ideal,
> but acceptable solution is to simply warn the user that dependent
> files may become inconsistent. A better solution, given that you
> already have the complete dependency tree is to automatically
> propagate the failed / out-of-date status to all dependent nodes.
>
> IMO, overwriting (and the ability to mark jobs failed) are the best
> Relion GUI enhancement in the last several years. It makes correcting
> errors (and I make quite a few) *so* much easier. Please don't take it
> away. If the CCPEM scheduler can't handle this, perhaps you could
> restrict overwriting only when run from within CCPEM rather than
> standalone. Better yet, maybe CCPEM could look into adding features
> like this themselves.
>
> In a nutshell, the Relion 3.1 gui has already shown us a window into a
> better way to do things. Please don't take it back and neuter Relion
> for the sake of integration with other packages.
>
> Sorry to run on, and thanks for considering this.
>
> Best regards,
> -jh-
>
> On Thu, Oct 7, 2021 at 10:32 AM Robert Bücker
> <[log in to unmask]
> <mailto:[log in to unmask]>> wrote:
>
> Hi Sjors,
>
> thanks for the clarification. The workflow I described (building a
> short pipeline on top of e.g. an auto-refine, and later building
> another branch on its continuation) is certainly dirty and should
> be only used when you exactly know what you are doing (or just
> not). However, in that scenario, files are still overwritten and
> the follow-up jobs on the „first“ branch (before continuation) are
> then „lingering“/inconsistent, similar to just plain overwriting
> the job. I think this should be handled consistently with the
> overwriting case, i.e. by forbidding continuation of jobs with
> children, or forcing their deletion or flagging them as
> invalid/failed, as suggested by John.
>
> Best wishes,
> Robert
>
>> Am 07.10.2021 um 17:54 schrieb John Monroe Heumann
>> <[log in to unmask] <mailto:[log in to unmask]>>:
>>
>> Hi Sjors,
>>
>> Please see the earlier post on a related topic and reconsider the
>> "no overwriting jobs whose output has been used" decision.
>> In particular, at my (and other's) request, Takanori already
>> changed this from a blanket prohibition to "pop up a dialog and
>> only overwrite these if the user insists".
>>
>> Obviously, it's your software and you can do what you want, but
>> IMO a blanket refusal to overwrite any job whose output has been
>> used is extreme, unjustified, and makes the overwrite feature
>> essentially useless. As I explained before, suppose you've run
>> jobs n through n+m and (with the later jobs depending on n) and
>> then decide you want to change a setting in job n. The logical
>> way to do this is to mark all of jobs n..n_m as failed, and then
>> to sequentially overwrite them one at a time. That's easy, safe,
>> and brainless. Conversely, if you disallow overwriting jobs whose
>> output has been used, the only choice is to delete jobs n+1
>> through n+m, overwrite n, and then re-create the subsequent jobs.
>> That's much more difficult and error prone, since you have to
>> re-edit settings (if any of the jobs involve common steps, so you
>> can't just trust what the gui has stored).
>>
>> Reasonable solutions are to prompt (as Takanori has currently
>> implemented) or to change the rule to "you can't overwrite jobs
>> whose output has been used *unless they''re marked as failed*.
>>
>> Thanks for considering this.
>>
>> Regards,
>> -jh-
>>
>> On Thu, Oct 7, 2021 at 9:36 AM Sjors Scheres - MRC LMB
>> <[log in to unmask] <mailto:[log in to unmask]>> wrote:
>>
>> Hi Robert,
>>
>> This is intended. Previous versions of RELION were less
>> strict with
>> overwriting and continuation of jobs. This potentially causes
>> problems
>> (like the one you mention where jobs depend on overwritten
>> ones). As we
>> are working with CCP-EM to enlarge the scope of the pipeliner
>> to many
>> more software packages, we have decided to be more strict on
>> this: no
>> longer overwrite jobs that have other jobs depending on them,
>> and use
>> continuations just for finishing crashed jobs etc.
>>
>> HTH,
>>
>> Sjors
>>
>>
>> On 10/7/21 1:44 PM, Robert Bücker wrote:
>> > Hi,
>> >
>> > I noticed that in Relion 4.0 (commit 1fb5b8), when continuing a
>> > refinement job with altered parameters, instead of adding
>> the _ctX_itY
>> > suffix to the output files (where X is the iteration on
>> which the
>> > continuation is built and Y the iteration number), the
>> naming scheme
>> > just sticks to the _itY suffix, and omits the _ct part for
>> the final
>> > outputs, too. This implies that files from before the
>> continuation get
>> > overwritten, including the final iteration, and final
>> results before
>> > continuation onto which other parts of the pipeline might
>> have been
>> > built are lost, leaving an inconsistent pipeline.
>> >
>> > Is this intended behavior in 4.0, or a bug? If it’s
>> intended, is there
>> > some way to keep the old behavior?
>> >
>> > Best wishes,
>> > Robert
>> >
>> > --
>> > Dr. Robert Bücker
>> > Centre for Structural Systems Biology // University of Hamburg
>> > Notkestraße 85 // 22607 Hamburg // Germany
>> > [log in to unmask]
>> <mailto:[log in to unmask]>
>> <mailto:[log in to unmask]
>> <mailto:[log in to unmask]>>
>> > +49 157 70210628
>> >
>> >
>> >
>> >
>> >
>> ------------------------------------------------------------------------
>> >
>> > To unsubscribe from the CCPEM list, click the following link:
>> > https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCPEM&A=1
>> >
>> --
>> Sjors Scheres
>> MRC Laboratory of Molecular Biology
>> Francis Crick Avenue, Cambridge Biomedical Campus
>> Cambridge CB2 0QH, U.K.
>> tel: +44 (0)1223 267061
>> http://www2.mrc-lmb.cam.ac.uk/groups/scheres
>>
>> ########################################################################
>>
>> To unsubscribe from the CCPEM list, click the following link:
>> https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCPEM&A=1
>>
>> This message was issued to members of
>> www.jiscmail.ac.uk/CCPEM <http://www.jiscmail.ac.uk/CCPEM>, a
>> mailing list hosted by www.jiscmail.ac.uk
>> <http://www.jiscmail.ac.uk/>, terms & conditions are
>> available at https://www.jiscmail.ac.uk/policyandsecurity/
>>
>>
>> ------------------------------------------------------------------------
>>
>> To unsubscribe from the CCPEM list, click the following link:
>> https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCPEM&A=1
>>
>
>
> ------------------------------------------------------------------------
>
> To unsubscribe from the CCPEM list, click the following link:
> https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCPEM&A=1
>
--
Sjors Scheres
MRC Laboratory of Molecular Biology
Francis Crick Avenue, Cambridge Biomedical Campus
Cambridge CB2 0QH, U.K.
tel: +44 (0)1223 267061
http://www2.mrc-lmb.cam.ac.uk/groups/scheres
########################################################################
To unsubscribe from the CCPEM list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCPEM&A=1
This message was issued to members of www.jiscmail.ac.uk/CCPEM, a mailing list hosted by www.jiscmail.ac.uk, terms & conditions are available at https://www.jiscmail.ac.uk/policyandsecurity/
|
