Hi Frank and Ian,

We struggled with the small changes in free R-factors when we implementing
the paired refinement for resolution cut-offs in PDB_REDO. It's not just the
lack of a proper test of significance for (weighted) R-factor changes, it's
also a more philosophical problem. When should you reject a higher
resolution cut-off? 
a) When it gives significantly higher R-factors (lenient)
b) When it gives numerically higher R-factors (less lenient, but takes away
the need for a significance test)
c) When it does not give significantly lower R-factors (very strict; if I
take X*sigma(R-free) as a cut-off, with X > 1.0, in most cases I should
reject the higher cut-off).

PDB_REDO uses b), similar to Karplus and Diederichs in their Science paper.

Then the next question is which metric are you going to use? R-free,
weighted R-free, free log likelihood and CCfree are all written out by
Refmac. At least the latter two have proper significance tests (likelihood
ratios and transformation Z-scores respectively). Note that we use different
models, constructed with different (but very much overlapping) data, but the
metrics are calculated with the same data. The different metrics do not
necessarily move in the same direction when moving to a higher resolution.

We ended up using all 4 in PDB_REDO. By default a higher resolution cut-off
is rejected if more than 1 metric gets (numerically) worse, but this can be
changed by the user.

Next question is the size of the resolution steps. How big should those be
and how should they be set up? Karplus and Diederichs used equal steps in
Angstrom, PDB_REDO uses equal steps in number of reflections. That way you
add the same amount of data (but not usable information) with each step.
Anyway, a different choice of steps will give a different final resolution
cut-off. And the exact cut-off doesn't matter that much (see Evans and
Murshudov). Different (versions of) refinement programs will probably also
give somewhat different results. 

We tested our implementation on a number of structures in the PDB with data
extending to higher resolution than marked in the PDB file and we observed
that quite a lot had very conservative resolution cut-offs. In some cases we
could use so much extra data that we could move to a more complex B-factor
model and seriously improve R-factors.

The best resolution cut-off is unclear and may change over time with
improving methods. So whatever you choose, please deposit all the data that
you can get even if you don't use it yourself. I think that the Karplus and
Diederichs papers show us that you should at least realize that your
resolution cut-off is a methodological choice that you should describe and
should be able to defend if somebody asks you why you made that particular
choice.

Cheers,
Robbie


> On 1 September 2013 11:31, Frank von Delft <[log in to unmask]>
> wrote:
> 
> 
> 
> 	2.
> 	I'm struck by how small the improvements in R/Rfree are in
> Diederichs & Karplus (ActaD 2013,
> http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3689524/
> <http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3689524/> );  the authors
> don't discuss it, but what's current thinking on how to estimate the
expected
> variation in R/Rfree - does the Tickle formalism (1998) still apply for ML
with
> very weak data?
> 
> 
> 
> Frank, our paper is still relevant, unfortunately just not to the question
> you're trying to answer!  We were trying to answer 2 questions: 1) what
> value of Rfree would you expect to get if the structure were free of
> systematic error and only random errors were present, so that could be
used
> as a baseline (assuming a fixed cross-validation test set) to identify
models
> with gross (e.g. chain-tracing) errors; and 2) how much would you expect
> Rfree to vary assuming a fixed starting model but with a different random
> sampling of the test set (i.e. the "sampling standard deviation").  The
latter is
> relevant if say you want to compare the same structure (at the same
> resolution obviously) done independently in 2 labs, since it tells you how
big
> the difference in Rfree for an arbitrary choice of test set needs to be
before
> you can claim that it's statistically significant.
> 
> 
> In this case the questions are different because you're certainly not
> comparing different models using the same test set, neither I suspect are
> you comparing the same model with different randomly selected test sets.
I
> assume in this case that the test sets for different resolution cut-offs
are
> highly correlated, which I suspect makes it quite difficult to say what is
a
> significant difference in Rfree (I have not attempted to do the algebra!).
> 
> 
> Rfree is one of a number of "model selection criteria" (see
> http://en.wikipedia.org/wiki/Model_selection#Criteria_for_model_selectio
> n) whose purpose is to provide a metric for comparison of different models
> given specific data, i.e. as for the likelihood function they all take the
form
> f(model | data), so in all cases you're varying the model with fixed data.
It's
> use in the form f(data | model), i.e. where you're varying the data with a
> fixed model I would say is somewhat questionable and certainly requires
> careful analysis to determine whether the results are statistically
significant.
> Even assuming we can argue our way around the inappropriate application of
> model selection methodology to a different problem, unfortunately Rfree is
> far from an ideal criterion in this respect; a better one would surely be
the
> free log-likelihood as originally proposed by Gerard Bricogne.
> 
> 
> Cheers
> 
> 
> -- Ian
>