On Tuesday, October 26, 2010 04:31:24 pm James Holton wrote:
> Yes, but what I think Frank is trying to point out is that the difference
> between Fobs and Fcalc in any given PDB entry is generally about 4-5 times
> larger than sigma(Fobs).  In such situations, pretty much any standard
> statistical test will tell you that the model is "highly unlikely to be
> correct".

But that's not the question we are normally asking.
It is highly unlikely that any model in biology is correct, if by "correct" 
you mean "cannot be improved". Normally we ask the more modest question
"have I improved my model today over what it was yesterday?".

> I am not saying that everything in the PDB is "wrong", just that the
> dominant source of error is a shortcoming of the models we use.  Whatever
> this "source of error" is, it vastly overpowers the measurement error.  That
> is, errors do not add linearly, but rather as squares, and 20%^2+5%^2 ~
> 20%^2 .
> 
> So, since the experimental error is only a minor contribution to the total
> error, it is arguably inappropriate to use it as a weight for each hkl.

I think your logic has run off the track.  The experimental error is an
appropriate weight for the Fobs(hkl) because that is indeed the error
for that observation.  This is true independent of errors in the model.
If you improve the model, that does not magically change the accuracy
of the data.

	Ethan


 
> Yes, refinement does seem to work better when you use experimental sigmas,
> and weighted statistics are probably "better" than no weights at all, but
> the problem is that until we do have a model that can explain Fobs to within
> experimental error, we will be severely limited in the kinds of conclusions
> we can derive from our data.
> 
> -James Holton
> MAD Scientist
> 
> On Tue, Oct 26, 2010 at 1:59 PM, Ethan Merritt <[log in to unmask]>wrote:
> 
> > On Tuesday, October 26, 2010 01:16:58 pm Frank von Delft wrote:
> > >    Um...
> > >
> > > * Given that the weighted Rfactor is weighted by the measurement errors
> > > (1/sig^2)
> > >
> > > * and given that the errors in our measurements apparently have no
> > > bearing whatsoever on the errors in our models (for macromolecular
> > > crystals, certainly - the "R-vfactor gap")
> >
> > You are overlooking causality :-)
> >
> > Yes, the errors in state-of-the-art models are only weakly limited by the
> > errors in our measurements.  But that is exactly _because_ we can now
> > weight
> > properly by the measurement errors (1/sig^2).  In my salad days,
> > weighting by 1/sig^2 was a mug's game.   Refinement only produced
> > a reasonable model if you applied empirical corrections rather than
> > statistical weights.  Things have improved a bit since then,
> > both on the equipment side (detectors, cryo, ...) and on the processing
> > side (Maximum Likelihood, error propagation, ...).
> > Now the sigmas actually mean something!
> >
> > > is the weighted Rfactor even vaguely relevant for anything at all?
> >
> > Yes, it is.  It is the thing you are minimizing during refinement,
> > at least to first approximation.  Also, as just mentioned, it is a
> > well-defined value that you can do use for statistical significance
> > tests.
> >
> >        Ethan
> >
> >
> > >
> > > phx.
> > >
> > >
> > >
> > > On 26/10/2010 20:44, Ian Tickle wrote:
> > > > Indeed, see: http://scripts.iucr.org/cgi-bin/paper?a07175 .
> > > >
> > > > The Rfree/Rwork ratio that I referred to does strictly use the
> > > > weighted ('Hamilton') R-factors, but because only the unweighted
> > > > values are given in the PDB we were forced to approximate (against our
> > > > better judgment!).
> > > >
> > > > The problem of course is that all refinement software AFAIK writes the
> > > > unweighted Rwork&  Rfree to the PDB header; there are no slots for the
> > > > weighted values, which does indeed make doing serious statistics on
> > > > the PDB entries difficult if not impossible!
> > > >
> > > > The unweighted crystallographic R-factor was only ever intended as a
> > > > "rule of thumb", i.e. to give a rough idea of the relative quality of
> > > > related structures; I hardly think the crystallographers of yesteryear
> > > > ever imagined that we would be taking it so seriously now!
> > > >
> > > > In particular IMO it should never be used for something as critical as
> > > > validation (either global or local), or for guiding refinement
> > > > strategy: use the likelihood instead.
> > > >
> > > > Cheers
> > > >
> > > > -- Ian
> > > >
> > > > PS I've always known it as an 'R-factor', e.g. see paper referenced
> > > > above, but then during my crystallographic training I used extensively
> > > > software developed by both authors of the paper (i.e. Geoff Ford&  the
> > > > late John Rollett) in Oxford (which eventually became the 'Crystals'
> > > > small-molecule package).  Maybe it's a transatlantic thing ...
> > > >
> > > > Cheers
> > > >
> > > > -- Ian
> > > >
> > > > On Tue, Oct 26, 2010 at 7:28 PM, Ethan Merritt<
> > [log in to unmask]>  wrote:
> > > >> On Tuesday, October 26, 2010 09:46:46 am Bernhard Rupp (Hofkristallrat
> > a.D.) wrote:
> > > >>> Hi Folks,
> > > >>>
> > > >>> Please allow me a few biased reflections/opinions on the numeRology
> > of the
> > > >>> R-value (not R-factor, because it is neither a factor itself nor does
> > it
> > > >>> factor in anything but ill-posed reviewer's critique. Historically
> > the term
> > > >>> originated from small molecule crystallography, but it is only a
> > > >>> 'Residual-value')
> > > >>>
> > > >>> a) The R-value itself - based on the linear residuals and of apparent
> > > >>> intuitive meaning - is statistically peculiar to say the least. I
> > could not
> > > >>> find it in any common statistics text. So doing proper statistics
> > with R
> > > >>> becomes difficult.
> > > >> As WC Hamilton pointed out originally, two [properly weighted] R
> > factors can
> > > >> be compared by taking their ratio.  Significance levels can then be
> > evaluated
> > > >> using the standard F distribution.  A concise summary is given in
> > chapter 9
> > > >> of Prince's book, which I highly recommend to all crystallographers.
> > > >>
> > > >> W C Hamilton "Significance tests on the crystallographic R factor"
> > > >> Acta Cryst. (1965). 18, 502-510
> > > >>
> > > >> Edward Prince "Mathematical Techniques in Crystallography and
> > Materials
> > > >> Science". Springer-Verlag, 1982.
> > > >>
> > > >> It is true that we normally indulge in the sloppy habit of paying
> > attention
> > > >> only to the unweighted R factor even though refinement programs report
> > > >> both the weighted and unweighted versions.  (shelx users excepted :-)
> > > >> But the weighted form is there also if you want to do statistical
> > tests.
> > > >>
> > > >> You are of course correct that this remains a global test, and as such
> > > >> is of limited use in evaluating local properties of the model.
> > > >>
> > > >>         cheers,
> > > >>
> > > >>                 Ethan
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>> b) rules of thumb (as much as they conveniently obviate the need for
> > > >>> detailed explanations, satisfy student's desire for quick answers,
> >  and
> > > >>> allow superficial review of manuscripts) become less valuable if they
> > have a
> > > >>> case-dependent large variance, topped with an unknown parent
> > distribution.
> > > >>> Combined with an odd statistic, that has great potential for
> > misguidance and
> > > >>> unnecessarily lost sleep.
> > > >>>
> > > >>> c) Ian has (once again) explained that for example the Rf-R depends
> > on the
> > > >>> exact knowledge of the restraints and their individual weighting,
> > which we
> > > >>> generally do not have. Caution is advised.
> > > >>>
> > > >>> d) The answer which model is better - which is actually what you want
> > to
> > > >>> know - becomes a question of model selection or hypothesis testing,
> > which,
> > > >>> given the obscurity of R cannot be derived with some nice plug-in
> > method. As
> > > >>> Ian said the models to be compared must also be based on the same and
> > > >>> identical data.
> > > >>>
> > > >>> e) One measure available that is statistically at least defensible is
> > the
> > > >>> log-likelihood. So what you can do is form a log-likelihood ratio (or
> > Bayes
> > > >>> factor (there is the darn factor again, it’s a ratio)) and see where
> > this
> > > >>> falls - and the answers are pretty soft and, probably because of
> > that,
> > > >>> correspondingly realistic. This also makes - based on statistics
> > alone -
> > > >>> deciding between different overall parameterizations difficult.
> > > >>>
> > > >>> http://en.wikipedia.org/wiki/Bayes_factor
> > > >>>
> > > >>> f) so having said that, what really remains is that the model that
> > fits the
> > > >>> primary evidence (minimally biased electron density) best and is at
> > the same
> > > >>> time physically meaningful, is the best model, i. e., all plausibly
> > > >>> accountable electron density (and not more) is modeled. You can
> > convince
> > > >>> yourself of this by taking the most interesting part of the model out
> > (say a
> > > >>> ligand or a binding pocket) and look at the R-values or do a model
> > selection
> > > >>> test - the result will be indecisive.  Poof goes the global rule of
> > thumb.
> > > >>>
> > > >>> g) in other words: global measures in general are entirely inadequate
> > to
> > > >>> judge local model quality (noted many times over already by Jones,
> > Kleywegt,
> > > >>> others, in the dark ages of crystallography when poorly restrained
> > > >>> crystallographers used to passionately whack each other over the head
> > with
> > > >>> unfree R-values).
> > > >>>
> > > >>> Best, BR
> > > >>> -----------------------------------------------------------------
> > > >>> Bernhard Rupp, Hofkristallrat a.D.
> > > >>> 001 (925) 209-7429
> > > >>> +43 (676) 571-0536
> > > >>> [log in to unmask]
> > > >>> [log in to unmask]
> > > >>> http://www.ruppweb.org/
> > > >>> ------------------------------------------------------------------
> > > >>> Und wieder ein chillout-mix aus der Hofkristall-lounge
> > > >>> ------------------------------------------------------------------
> > > >>>
> > > >> --
> > > >> Ethan A Merritt
> > > >> Biomolecular Structure Center,  K-428 Health Sciences Bldg
> > > >> University of Washington, Seattle 98195-7742
> > > >>
> > >
> >
> > --
> > Ethan A Merritt
> > Biomolecular Structure Center,  K-428 Health Sciences Bldg
> > University of Washington, Seattle 98195-7742
> >
> 

-- 
Ethan A Merritt
Biomolecular Structure Center,  K-428 Health Sciences Bldg
University of Washington, Seattle 98195-7742