Um...
* Given that the weighted Rfactor is weighted by the measurement errors
(1/sig^2)
* and given that the errors in our measurements apparently have no
bearing whatsoever on the errors in our models (for macromolecular
crystals, certainly - the "R-vfactor gap")
is the weighted Rfactor even vaguely relevant for anything at all?
phx.
On 26/10/2010 20:44, Ian Tickle wrote:
> Indeed, see: http://scripts.iucr.org/cgi-bin/paper?a07175 .
>
> The Rfree/Rwork ratio that I referred to does strictly use the
> weighted ('Hamilton') R-factors, but because only the unweighted
> values are given in the PDB we were forced to approximate (against our
> better judgment!).
>
> The problem of course is that all refinement software AFAIK writes the
> unweighted Rwork& Rfree to the PDB header; there are no slots for the
> weighted values, which does indeed make doing serious statistics on
> the PDB entries difficult if not impossible!
>
> The unweighted crystallographic R-factor was only ever intended as a
> "rule of thumb", i.e. to give a rough idea of the relative quality of
> related structures; I hardly think the crystallographers of yesteryear
> ever imagined that we would be taking it so seriously now!
>
> In particular IMO it should never be used for something as critical as
> validation (either global or local), or for guiding refinement
> strategy: use the likelihood instead.
>
> Cheers
>
> -- Ian
>
> PS I've always known it as an 'R-factor', e.g. see paper referenced
> above, but then during my crystallographic training I used extensively
> software developed by both authors of the paper (i.e. Geoff Ford& the
> late John Rollett) in Oxford (which eventually became the 'Crystals'
> small-molecule package). Maybe it's a transatlantic thing ...
>
> Cheers
>
> -- Ian
>
> On Tue, Oct 26, 2010 at 7:28 PM, Ethan Merritt<[log in to unmask]> wrote:
>> On Tuesday, October 26, 2010 09:46:46 am Bernhard Rupp (Hofkristallrat a.D.) wrote:
>>> Hi Folks,
>>>
>>> Please allow me a few biased reflections/opinions on the numeRology of the
>>> R-value (not R-factor, because it is neither a factor itself nor does it
>>> factor in anything but ill-posed reviewer's critique. Historically the term
>>> originated from small molecule crystallography, but it is only a
>>> 'Residual-value')
>>>
>>> a) The R-value itself - based on the linear residuals and of apparent
>>> intuitive meaning - is statistically peculiar to say the least. I could not
>>> find it in any common statistics text. So doing proper statistics with R
>>> becomes difficult.
>> As WC Hamilton pointed out originally, two [properly weighted] R factors can
>> be compared by taking their ratio. Significance levels can then be evaluated
>> using the standard F distribution. A concise summary is given in chapter 9
>> of Prince's book, which I highly recommend to all crystallographers.
>>
>> W C Hamilton "Significance tests on the crystallographic R factor"
>> Acta Cryst. (1965). 18, 502-510
>>
>> Edward Prince "Mathematical Techniques in Crystallography and Materials
>> Science". Springer-Verlag, 1982.
>>
>> It is true that we normally indulge in the sloppy habit of paying attention
>> only to the unweighted R factor even though refinement programs report
>> both the weighted and unweighted versions. (shelx users excepted :-)
>> But the weighted form is there also if you want to do statistical tests.
>>
>> You are of course correct that this remains a global test, and as such
>> is of limited use in evaluating local properties of the model.
>>
>> cheers,
>>
>> Ethan
>>
>>
>>
>>
>>> b) rules of thumb (as much as they conveniently obviate the need for
>>> detailed explanations, satisfy student's desire for quick answers, and
>>> allow superficial review of manuscripts) become less valuable if they have a
>>> case-dependent large variance, topped with an unknown parent distribution.
>>> Combined with an odd statistic, that has great potential for misguidance and
>>> unnecessarily lost sleep.
>>>
>>> c) Ian has (once again) explained that for example the Rf-R depends on the
>>> exact knowledge of the restraints and their individual weighting, which we
>>> generally do not have. Caution is advised.
>>>
>>> d) The answer which model is better - which is actually what you want to
>>> know - becomes a question of model selection or hypothesis testing, which,
>>> given the obscurity of R cannot be derived with some nice plug-in method. As
>>> Ian said the models to be compared must also be based on the same and
>>> identical data.
>>>
>>> e) One measure available that is statistically at least defensible is the
>>> log-likelihood. So what you can do is form a log-likelihood ratio (or Bayes
>>> factor (there is the darn factor again, it’s a ratio)) and see where this
>>> falls - and the answers are pretty soft and, probably because of that,
>>> correspondingly realistic. This also makes - based on statistics alone -
>>> deciding between different overall parameterizations difficult.
>>>
>>> http://en.wikipedia.org/wiki/Bayes_factor
>>>
>>> f) so having said that, what really remains is that the model that fits the
>>> primary evidence (minimally biased electron density) best and is at the same
>>> time physically meaningful, is the best model, i. e., all plausibly
>>> accountable electron density (and not more) is modeled. You can convince
>>> yourself of this by taking the most interesting part of the model out (say a
>>> ligand or a binding pocket) and look at the R-values or do a model selection
>>> test - the result will be indecisive. Poof goes the global rule of thumb.
>>>
>>> g) in other words: global measures in general are entirely inadequate to
>>> judge local model quality (noted many times over already by Jones, Kleywegt,
>>> others, in the dark ages of crystallography when poorly restrained
>>> crystallographers used to passionately whack each other over the head with
>>> unfree R-values).
>>>
>>> Best, BR
>>> -----------------------------------------------------------------
>>> Bernhard Rupp, Hofkristallrat a.D.
>>> 001 (925) 209-7429
>>> +43 (676) 571-0536
>>> [log in to unmask]
>>> [log in to unmask]
>>> http://www.ruppweb.org/
>>> ------------------------------------------------------------------
>>> Und wieder ein chillout-mix aus der Hofkristall-lounge
>>> ------------------------------------------------------------------
>>>
>> --
>> Ethan A Merritt
>> Biomolecular Structure Center, K-428 Health Sciences Bldg
>> University of Washington, Seattle 98195-7742
>>
|