Although George does not mention anything about data reduction programs,
I take from his description that common small-molecule data processing
packages (SAINT, others?), have also been modernized to record all data
(no I/sigmaI > 2 or 3 cutoff). I agree with him that this is a good
thing! And it is also a good thing that small-molecule refinement
programs use all data. I just don't think it is a good idea to use all
data in R factor calculations.

Like Ron, I will probably be dating myself when I say that when I first
got into the macromolecular crystallography business, it was still
commonplace to use a 2-3 sigma spot intensity cutoff. In fact, this is
the reason why the PDB wants to know your "completeness" in the
outermost resolution shell (in those days, the outer resolution was
defined by where completeness drops to ~80% after the 3 sigma spot
cutoff). My experience with this, however, was brief, as the
maximum-likelihood revolution was just starting to take hold, and the
denzo manual specifically stated that only bad people use sigma cutoffs
 > -3.0. Nevertheless, like many crystallographers from this era, I
have fond memories of the REALLY low R factors you can get by using this
arcane and now reviled practice. Rsym values of 1-2% were common.

It was only recently that I learned enough about statistics to
understand the wisdom of my ancestors and that a 3-sigma cutoff is
actually the "right thing to do" if you want to measure a fractional
error (like an R factor). That is all I'm saying.

-James Holton
MAD Scientist


On 3/6/2011 2:50 PM, Ronald E Stenkamp wrote:
> My small molecule experience is old enough (maybe 20 years) that I
> doubt if it's even close to representing current practices (best or
> otherwise). Given George's comments, I suspect (and hope) that
> less-than cutoffs are historical artifacts at this point, kept around
> in software for making comparisons with older structure
> determinations. But a bit of scanning of Acta papers and others might
> be necessary to confirm that. Ron
>
> On Sun, 6 Mar 2011, James Holton wrote:
>
>>
>> Yes, I would classify anything with I/sigmaI < 3 as "weak". And yes,
>> of course it is possible to get "weak" spots from small molecule
>> crystals. After all, there is no spot so "strong" that it cannot be
>> defeated by a sufficient amount of background! I just meant that,
>> relatively speaking, the intensities diffracted from a small molecule
>> crystal are orders of magnitude brighter than those from a
>> macromolecular crystal of the same size, and even the same quality
>> (the 1/Vcell^2 term in Darwin's formula).
>>
>> I find it interesting that you point out the use of a 2 sigma(I)
>> intensity cutoff for small molecule data sets! Is this still common
>> practice? I am not a card-carrying "small molecule
>> crystallographer", so I'm not sure. However, if that is the case,
>> then by definition there are no "weak" intensities in the data set.
>> And this is exactly the kind of data you want for least-squares
>> refinement targets and computing "% error" quality metrics like R
>> factors. For likelihood targets, however, the "weak" data are
>> actually a powerful restraint.
>>
>> -James Holton
>> MAD Scientist
>>
>> On 3/6/2011 11:22 AM, Ronald E Stenkamp wrote:
>>> Could you please expand on your statement that "small-molecule data
>>> has essentially no weak spots."? The small molecule data sets I've
>>> worked with have had large numbers of "unobserved" reflections where
>>> I used 2 sigma(I) cutoffs (maybe 15-30% of the reflections). Would
>>> you consider those "weak" spots or not? Ron
>>>
>>> On Sun, 6 Mar 2011, James Holton wrote:
>>>
>>>> I should probably admit that I might be indirectly responsible for
>>>> the resurgence of this I/sigma > 3 idea, but I never intended this
>>>> in the way described by the original poster's reviewer!
>>>>
>>>> What I have been trying to encourage people to do is calculate R
>>>> factors using only hkls for which the signal-to-noise ratio is >
>>>> 3. Not refinement! Refinement should be done against all data. I
>>>> merely propose that weak data be excluded from R-factor
>>>> calculations after the refinement/scaling/mergeing/etc. is done.
>>>>
>>>> This is because R factors are a metric of the FRACTIONAL error in
>>>> something (aka a "% difference"), but a "% error" is only
>>>> meaningful when the thing being measured is not zero. However, in
>>>> macromolecular crystallography, we tend to measure a lot of
>>>> "zeroes". There is nothing wrong with measuring zero! An
>>>> excellent example of this is confirming that a systematic absence
>>>> is in fact "absent". The "sigma" on the intensity assigned to an
>>>> absent spot is still a useful quantity, because it reflects how
>>>> confident you are in the measurement. I.E. a sigma of "10" vs
>>>> "100" means you are more sure that the intensity is zero. However,
>>>> there is no "R factor" for systematic absences. How could there
>>>> be! This is because the definition of "% error" starts to break
>>>> down as the "true" spot intensity gets weaker, and it becomes
>>>> completely meaningless when the "true" intensity reaches zero.
>>>>
>>>> Historically, I believe the widespread use of R factors came about
>>>> because small-molecule data has essentially no weak spots. With
>>>> the exception of absences (which are not used in refinement), spots
>>>> from "salt crystals" are strong all the way out to edge of the
>>>> detector, (even out to the "limiting sphere", which is defined by
>>>> the x-ray wavelength). So, when all the data are strong, a "%
>>>> error" is an easy-to-calculate quantity that actually describes the
>>>> "sigma"s of the data very well. That is, sigma(I) of strong spots
>>>> tends to be dominated by things like beam flicker, spindle
>>>> stability, shutter accuracy, etc. All these usually add up to ~5%
>>>> error, and indeed even the Braggs could typically get +/-5% for the
>>>> intensity of the diffracted rays they were measuring. Things like
>>>> Rsym were therefore created to check that nothing "funny" happened
>>>> in the measurement.
>>>>
>>>> For similar reasons, the quality of a model refined against
>>>> all-strong data is described very well by a "% error", and this is
>>>> why the refinement R factors rapidly became popular. Most people
>>>> intuitively know what you mean if you say that your model fits the
>>>> data to "within 5%". In fact, a widely used criterion for the
>>>> correctness of a "small molecule" structure is that the refinement
>>>> R factor must be LOWER than Rsym. This is equivalent to saying
>>>> that your curve (model) fit your data "to within experimental
>>>> error". Unfortunately, this has never been the case for
>>>> macromolecular structures!
>>>>
>>>> The problem with protein crystals, of course, is that we have lots
>>>> of "weak" data. And by "weak", I don't mean "bad"! Yes, it is
>>>> always nicer to have more intense spots, but there is nothing
>>>> shameful about knowing that certain intensities are actually very
>>>> close to zero. In fact, from the point of view of the refinement
>>>> program, isn't describing some high-angle spot as: "zero, plus or
>>>> minus 10", better than "I have no idea"? Indeed, several works
>>>> mentioned already as well as the "free lunch algorithm" have
>>>> demonstrated that these "zero" data can actually be useful, even if
>>>> it is well beyond the "resolution limit".
>>>>
>>>> So, what do we do? I see no reason to abandon R factors, since
>>>> they have such a long history and give us continuity of criteria
>>>> going back almost a century. However, I also see no reason to
>>>> punish ourselves by including lots of zeroes in the denominator.
>>>> In fact, using weak data in an R factor calculation defeats their
>>>> best feature. R factors are a very good estimate of the fractional
>>>> component of the total error, provided they are calculated with
>>>> strong data only.
>>>>
>>>> Of course, with strong and weak data, the best thing to do is
>>>> compare the model-data disagreement with the magnitude of the
>>>> error. That is, compare |Fobs-Fcalc| to sigma(Fobs), not Fobs
>>>> itself. Modern refinement programs do this! And I say the more
>>>> data the merrier.
>>>>
>>>>
>>>> -James Holton
>>>> MAD Scientist
>>>>
>>>>
>>>> On 3/4/2011 5:15 AM, Marjolein Thunnissen wrote:
>>>>> hi
>>>>>
>>>>> Recently on a paper I submitted, it was the editor of the journal
>>>>> who wanted exactly the same thing. I never argued with the editor
>>>>> about this (should have maybe), but it could be one cause of the
>>>>> epidemic that Bart Hazes saw....
>>>>>
>>>>>
>>>>> best regards
>>>>>
>>>>> Marjolein
>>>>>
>>>>> On Mar 3, 2011, at 12:29 PM, Roberto Battistutta wrote:
>>>>>
>>>>>> Dear all,
>>>>>> I got a reviewer comment that indicate the "need to refine the
>>>>>> structures at an appropriate resolution (I/sigmaI of>3.0), and
>>>>>> re-submit the revised coordinate files to the PDB for
>>>>>> validation.". In the manuscript I present some crystal structures
>>>>>> determined by molecular replacement using the same protein in a
>>>>>> different space group as search model. Does anyone know the
>>>>>> origin or the theoretical basis of this "I/sigmaI>3.0" rule for
>>>>>> an appropriate resolution?
>>>>>> Thanks,
>>>>>> Bye,
>>>>>> Roberto.
>>>>>>
>>>>>>
>>>>>> Roberto Battistutta
>>>>>> Associate Professor
>>>>>> Department of Chemistry
>>>>>> University of Padua
>>>>>> via Marzolo 1, 35131 Padova - ITALY
>>>>>> tel. +39.049.8275265/67
>>>>>> fax. +39.049.8275239
>>>>>> [log in to unmask]
>>>>>> www.chimica.unipd.it/roberto.battistutta/
>>>>>> VIMM (Venetian Institute of Molecular Medicine)
>>>>>> via Orus 2, 35129 Padova - ITALY
>>>>>> tel. +39.049.7923236
>>>>>> fax +39.049.7923250
>>>>>> www.vimm.it
>>>>
>>>
>>
>>
>