Hi Ian,
Yes, I guess my rule does work as you say.
If, starting the day from (Rwork = 20, Rfree = 30) abbreviate (20,30),
you do something to get (18,29), yes this means that that something was a bare
minimum acceptable thing to do.
If you do something to get (16,29) (decreased R by 4, Rfree by 1), then
I would immediately suspect that that thing that was done introduced excessive over-fitting.
If you do something to get (18,28) (decreased R by 2, Rfree by 2), then
I would say that the thing that was done was a good thing.
Yes, other arbitrary linear combinations could work. Not great
analysis of this method was performed. I considered that it came to a
question of what degree of over-fitting is acceptable. In practice, this
rule stopped endless additions of water molecules and further alternate
conformations, and for that purpose the precise point seemed unimportant. However,
I also used this rule to determine preferred parameters for BFAC and the matrix
weight.
Do you think this is a bad rule, and can you point me to a better rule?
Replying to BR:
>> This rule of thumb has proven successful in providing a
defined end point >>for building and refining a structure.
>Hmmm. I always thought things like no more significant explainable >(difference)
density define endpoints
>
>in model building and not R-values. This strategy has proven
successful in >nailing ligand structures where
>
>R-value rules of thumb were used to define the end points.
Of course, there are other rules. One has to explain all
significant residual density. But this tends to be a finite task.
The above rule was not applicable to building active sites, or other
things that would be discussed directly in a paper.
The problem I attempt to address is endless fiddling with features of
every-diminishing importance.
Apologies if I have missed a recent relevant thread, but are lists of
rules of thumb for model building and refinement?
Anthony
Anthony Duff Telephone: 02 9717 3493 Mob: 043
189 1076
-----Original Message-----
From: CCP4 bulletin board [mailto:[log in to unmask]] On Behalf Of Ian
Tickle
Sent: Wednesday, 27 October 2010 12:53 AM
To: [log in to unmask]
Subject: Re: [ccp4bb] diverging Rcryst and Rfree [SEC=UNCLASSIFIED]
Anthony,
Your rule actually works on the difference (Rfree - Rwork/2), not
(Rfree - Rwork) as you said, so is rather different from what most
people seem to be using.
For example let's say the current values are Rwork = 20, Rfree = 30,
so your current test value is (30 - 20/2) = 20. Then
according to
your rule Rwork = 18, Rfree = 29 is equally acceptable (29 - 18/2 =
20, i.e. same test value), whereas Rwork = 16, Rfree = 29 would not be
acceptable by your rule (29 - 16/2 = 21, so the test value is higher).
Rwork = 18, Rfree = 28 would represent an improvement by your
rule
(28 - 18/2 = 19, i.e. a lower test value).
You say this criterion "provides a defined end-point", i.e. a
minimum
in the test value above. However wouldn't other linear
combinations
of Rwork & Rfree also have a defined minimum value? In
particular
Rfree itself always has a defined minimum with respect to adding
parameters or changing the weights, so would also satisfy your
criterion. There has to be some additional criterion that you are
relying on to select the particular linear combination (Rfree -
Rwork.2) over any of the other possible ones?
Cheers
-- Ian
On Tue, Oct 26, 2010 at 6:33 AM, DUFF, Anthony <[log in to unmask]>
wrote:
>
>
> One “rule of thumb” based on R and R-free divergence
that I impress onto
> crystallography students is this:
>
>
>
> If a change in refinement strategy or parameters (eg loosening
restraints,
> introducing TLS) or a round of addition of unimportant water
molecules
> results in a reduction of R that is more than double the reduction
in
> R-free, then don’t do it.
>
>
>
> This rule of thumb has proven successful in providing a defined
end point
> for building and refining a structure.
>
>
>
> The rule works on the differential of R – R-free divergence.
I’ve noticed
> that some structures begin with a bigger divergence than others.
Different
> Rmerge might explain.
>
>
>
> Has anyone else found a student in a dark room carefully adding
large
> numbers of partially occupied water molecules?
>
>
>
>
>
>
>
> Anthony
>
> Anthony Duff Telephone: 02 9717 3493 Mob:
043 189 1076
>
>
>
> ________________________________
>
> From: CCP4 bulletin board [mailto:[log in to unmask]] On Behalf
Of Artem
> Evdokimov
> Sent: Tuesday, 26 October 2010 1:45 PM
> To: [log in to unmask]
> Subject: Re: [ccp4bb] diverging Rcryst and Rfree
>
>
>
> Not that rules of thumb always have to have a rationale, nor that
they're
> always correct - but it would seem that noise in the data (of
which Rmerge
> is an indicator) should have a significant relationship with the
R:Rfree
> difference, since Rfree is not (should not be, if selected correctly)
> subject to noise fitting. This rule is easily broken if one
refines against
> very noisy data (e.g. "that last shell with Rmerge of 55% and
<I/sigmaI>
> ratio of 1.3 is still good, right?") or if the structure is
overfit. The
> rule is only an indicative one (i.e. one should get really worried
if
> R-Rfree looks very different from Rmerge) and it breaks down at
very high
> and very low resolution (more complete picture by GK and shown in
BR's
> book).
>
> Since selection of data and refinement procedures is subject to
the
> decisions of the practitioner, I suspect that the extreme
divergence shown
> in the figures that you refer to is probably the result of our own
> collective decisions. I have no proof, but I suspect that if a
large enough
> section of the PDB were to be re-refined using the same methods
and the same
> data trimming practices, the spread would be considerably more
narrow.
> That'd be somewhat hard to do - but may be doable now given the
abundance of
> auto-building and auto-correcting algorithms.
>
> Artem
>
> On Mon, Oct 25, 2010 at 9:07 PM, Bernhard Rupp (Hofkristallrat
a.D.)
> <[log in to unmask]> wrote:
>
> And the rationale for that rule being exactly what?
>
>
>
> For stats, see figures 12-23, 12-24
>
> http://www.ruppweb.org/garland/gallery/Ch12/index_2.htm
>
>
>
> br
>
>
>
> From: CCP4 bulletin board [mailto:[log in to unmask]] On Behalf
Of Artem
> Evdokimov
> Sent: Monday, October 25, 2010 6:36 PM
> To: [log in to unmask]
> Subject: Re: [ccp4bb] diverging Rcryst and Rfree
>
>
>
> http:[log in to unmask]
>
> as well as some notes in the older posts :)
>
> As a very basic rule of thumb, Rfree-Rwork tends to be around
Rmerge for the
> dataset for refinements that are not overfitted.
>
> Artem
>
> On Mon, Oct 25, 2010 at 4:10 PM, Rakesh Joshi
<[log in to unmask]> wrote:
>
> Hi all,
>
> Can anyone comment, in general, on diverging Rcryst and Rfree
values(say>7%)
> for
> structures with kind of low resolutions(2.5-2.9 angstroms)?
>
> Thanks
> RJ
>
>
>
>