Sorry - I had not twigged that this was a SAD discussion. 

In this case DM is as you say gonna save you from bimodal ambiguities - as we all know that is why the FOMs get so much better during DM - but that seems fine to me as it is pretty much new information coming in from the solvent flattening, histogram matching, probably some averaging. 

So that improvement is not 'artifactual' but in fact part of the experimental detail of just how the structure was solved. I guess FOMs are really only useful in the Crick and Blow centroid case.

Still think any and all info on the experiment(s) is useful to see - if only for people to plan their own experiments building on past experience.

Best wishes and regards
    Martyn 

Martyn Symmons
Cambridge

 



----- Original Message ----
From: Ian Tickle <[log in to unmask]>
To: [log in to unmask]
Sent: Wednesday, 14 April, 2010 15:42:51
Subject: Re: [ccp4bb] Phasing statistics

I have to say that I don't share James' enthusiasm for the FOM as a
useful statistic, even for experimental phases, and particularly not
in the SAD (and SIR) cases, which after all is what this thread is
supposed to be about.   This is even without delving further into the
murky issues of wildly inflated FOM estimates from DM that he and
others have raised.

My point is that the definition of FOM is that it's the *expected*
cos(phase error), as opposed to *actual* cos(phase error) and the
'expected' bit makes all the difference!  It's essentially the
difference between precision and accuracy, i.e. the FOM tells you how
precise the phase estimate is and not at all how accurate it is: for
the latter you need to calculate the errors relative to the phases of
the final model, as James suggested.  The FOM is roughly related
(anti-correlated to be precise) to the variance: this is clear if we
take the case of small phase deviations, then <cos(x)> expands to
<(1-x^2/2)> and the variance is <x^2>, where x is the deviation from
the mean (NOT the same thing as the error!).  So just like the
variance it measures the 'peakedness' of the distribution: a FOM=1
corresponds to variance=0, i.e. an infinitely sharp distribution
(delta function).

Now a particular problem arises in SAD (and SIR) because then in
general we always get a bimodal distribution: i.e. 2 separate peaks.
If these peaks happen to separated by 180 deg then the FOM is 0
(cos(90)=0).  A large separation is most likely to occur when the
anomalous contribution is large, when of course you expect the phase
estimates to be optimal, for example see Fig 2(a) in Wang et al., Acta
Cryst. (2007). D63, 751–758.  Conversely if the anomalous contribution
is small, so the phase estimates are poor, the separation between the
2 estimates is small and the FOM is close to 1!  So we apparently have
a situation where the *better* are the phases, the *lower* is the FOM!
What the Figure in the Wang paper ignores of course are the
experimental errors which will tend to broaden the distribution and
lower the FOM in the case where the anomalous contribution is small
relative to the errors.  Even so it means that the FOM is not a good
measure of phase quality.  Much better IMO is the phasing power (mean
heavy-atom amplitude / mean P-weighted lack of closure error).  This
essentially measures the degree to which the phase ambiguity is
capable of being successfully resolved by DM methods.

One other thing puzzles me: why do many programs that purport to
calculate average phase differences (relative to model phases say) use
statistics such as, well, average |phase difference| when we already
have a perfectly good measure in the FOM!, i.e. average cos(phase
difference)?  Then you would be able to directly compare the FOMs from
experimental phases with those from model phases.  Even better still
would be the average log likelihood: if the phase probability is exp(A
cos(delta_phi)) then the log likelihood is simply A cos(delta_phi) and
you just average that, i.e. it's essentially the averaged FOM-weighted
cos(phase difference), so that the average is weighted according to
the reliability of the phase estimates.  A poor phase estimate is
likely to be associated with a small F which is not going to
contribute much to the map anyway, so it makes no sense to give poor
phases the same weight as good phases in the average.

Cheers

-- Ian

On Tue, Apr 13, 2010 at 6:47 PM, James Holton <[log in to unmask]> wrote:
> Probably the only phasing stat that I pay any attention to these days is the
> Figure of Merit (FOM). This is because, the _definition_ of FOM is that it
> is the cosine of the phase error (or at least your best estimate of it).
>  FOM=1 is perfect phases and FOM=0 is random phases, and a reasonable cutoff
> value for FOM is 0.5 (see Lunin & Woolfson, Acta D, 1993).  Yes, there are
> ways to get various programs to report very inaccurate values for FOM (such
> as running DM for thousands of cycles), and yes, there are often legitimate
> reasons to run these programs in this way.   But, there are also very wrong
> things one can do to get low Rmerge, Rcryst, and especially Rfree.  It is
> simply a matter of knowing (and reporting) what you are doing.
>
>
> If you are worried that your favorite estimate of FOM is inaccurate, then
> you can always turn to your most accurate phases:  those of your final,
> refined model (the one that you have convinced yourself is "right" and ready
> to publish).  Taking these as the "true phases", the "true FOM" can always
> be obtained by comparing the final-model phases to those of your initial map
> (using PHISTATS or SFTOOLS).  This is by no means standard practice, but
> perhaps it should be?
>
>
> Anyway, FOM is _supposed_ to be the cosine of the phase error, and is
> therefore the most relevant statistic when it comes to how good your phases
> were when you started building.  This is why it is important as a reviewer
> to know what it is.  If I am faced with a structure that was built into a
> MAD map with initial FOM = 0.8 to 2 A resolution, then I am already
> convinced that the structure is "right" because I know they had a very clear
> map to build into.  It is hard to do something egregiously wrong with such a
> map (such as tracing it backwards), so I would even excuse a high R/Rfree in
> this case, especially if the map has large absent (disordered) regions that
> the authors were honest enough to not build.
>
>
> On the other hand, if the initial solvent-flattened SAD map had FOM=0.3 to 2
> A, you are really pushing it.  It is possible to get a correct structure
> from such a map, but extremely difficult.  One might combine some MR phases
> with the SAD phases to improve them somewhat, but how does one evaluate such
> a result?  I'd say that if FOM < 0.5, then the phases don't make you right.
>  You need to look at other statistics (like R/Rfree).
>
>
> The extreme case, of course, is MR, where the "starting" FOM=0.  The author
> then makes an assumption about the starting phases (based on prior knowledge
> such as homology with PDB ID = xxxx), and that assumption is then borne out
> by an "acceptable" R/Rfree (Kleywegt & Brunger, 1996).  The "true FOM"
> (comparing final refined phases to those of the initial MR hit) in this case
> might still be interesting because it tells you a lot about how much
> rebuilding had to be done.
>
>
> To answer Frank's question about a 4 A structure with anisotropic
> diffraction (which I assume means that 4 A is in the best direction, and the
> other(s) are 5 A or so), I would first ask that the "true" resolution limit
> be denoted by the point where the average I/sigma drops to ~2 (this is
> _without_ an anisotropic resolution cutoff!).  Then we probably have a 4.5A
> structure.  The "metrics by which we then judge the results?" then depends
> on the bigger question: "Does the evidence presented justify the conclusions
> drawn?".  If the conclusion is that bond lengths in the active site are
> "strained", then the answer is obviously "no".  Indeed, if the conclusions
> rely on the helicies in a 4.5 A map being traced in the right direction,
> then I would also answer "no".  This is because at 4.5 A the image of a
> backward-traced helix looks a _lot_ like the correctly-traced one (see
> http://bl831.als.lbl.gov/~jamesh/movies/index.html#reso).  To put it another
> way, the R-factors alone are not convincing evidence of a correct trace at
> 4.5A, and corroborating evidence must be presented to make the helix
> direction convincing.  By "presented", I mean spelled out in the text, and
> by "corroborating evidence" I mean something as simple as a clear
> connectivity with enough big side chains placed to deduce the register of
> the sequence.  Barring that, something like "SeMet scanning" can also
> clarify tracing ambiguities (for a relevant example, see Chen et al. (2007)
> PNAS 104 p 18999).  I am not saying that every 4.5 A structure needs to do
> this, but I am saying that the number of alternative explanations (models)
> for a given observation (map) increases as the map gets blurrier, and if a
> plausible alternative model could change the conclusions of the paper, then
> it must be eliminated with controls.  You know, basic science stuff.
>
>
> It is a common misconception that MAD/SAD/MIR phasing depends on resolution,
> but nowhere on the Harker diagram does one see the "resolution" of the
> vectors.  The accuracy of the phase depends entirely on the magnitude of the
> signal (delta-F) and the magnitude of the noise (sigma(F)).  This is why you
> only get experimental phases for strong spots, and never all the way out to
> your "resolution limit".  True, this is a "resolution dependence", but it is
> actually the signal-to-noise  ratio itself that is important.  The only part
> of experimental phasing that seems to be reproducibly resolution-dependent
> is the density modification used to clean it up.  This seems to be limited
> to pushing your "good phases" out by ~ 1 A in most cases (i.e. from 4A to 3A
> or from 3.5A to 2.5A, etc.), but I'm not sure why that is.  Probably
> something in histogram matching.  Unfortunately, I am not aware of a good
> comprehensive review of the resolution dependence of phase extension,
> possibly because one cannot do such an analysis with the data currently
> available in the PDB (initial phases are not deposited).
>
>
> I would finally like to note that I am highly uncomfortable with the idea of
> excusing the reporting of data processing statistics if the structure is
> deemed "correct".  Formally, no protein structure is intrinsically "correct"
> if it does not explain the data (Fobs) to withing experimental error (~5%).
>  In the "small molecule world" models with Rcryst > Rmerge are rejected
> out-of-hand (and for good reason).  The only reason protein structures are
> "excused" from this rule is because they have a good "track record" of
> agreeing with experimentally-phased maps.
>
> -James Holton
> MAD Scientist
>
> Frank von Delft wrote:
>>
>> I fully agree, for high quality data.
>>
>> What though if the data are not impeccable and the structure necessarily
>> ropey?  E.g. 4A phases and anisotropic diffraction.  By what metrics do we
>> then judge the results?
>>
>> (I don't know the answer, btw, but our membranous colleagues surely spend
>> quite a bit of time with that question...)
>>
>> phx.
>>
>>
>> On 12/04/2010 12:10, Anastassis Perrakis wrote:
>>>
>>> Hi -
>>>
>>> A year or so ago, I have asked as a referee somebody to provide for a
>>> paper the statistics for their heavy atom derivative dataset,
>>> and for the phasing statistics. For some good reasons, they were unable
>>> to do that, and they (politely) asked me
>>> 'what would it change if you knew these, isn't the structure we present
>>> impeccable?'. Well, I think they were right.
>>> Their structure was surely correct, surely high quality. After that
>>> incident and giving it some thought, I fail to see why should one report
>>> e.g. PP or Rcullis, or why will I care what they were if the structure has a
>>> convincing Rfree and is properly validated. If someone wants to cheat at the
>>> end of the day, its easy to provide two numbers, but its hard to provide a
>>> good validated model that agrees with the data.
>>> (and, yes, you can also make up the data, but we have been there, haven't
>>> we?!?)
>>>
>>> So, my question to that referee, likely being a ccp4bb aficionado that is
>>> reading this email, or to anyone else really, is:
>>>
>>> "What would it help to judge the quality of the structure or the paper if
>>> you know PP, Rcullis and FOM?"
>>>
>>> Best -
>>>
>>> A.
>>>
>>> PS Especially since you used SHELXE for phasing these statistics are
>>> utterly irrelevant, and possibly you could advice the referee to read a bit
>>> about how SHELXE works ... or go to one of the nice courses that George
>>> teaches ...
>>>
>>> On Apr 12, 2010, at 10:37, Eleanor Dodson wrote:
>>>
>>>> You can feed the SHELX sites into phaser_er or CRANK both of which will
>>>> give this sort of information.
>>>>
>>>> Or mlphare if you know how to set it up..
>>>>
>>>> Eleanor
>>>>
>>>>
>>>> Harmer, Nicholas wrote:
>>>>>
>>>>> Dear CCP4ers,
>>>>>
>>>>> I've been asked by a referee to provide the phasing statistics for a
>>>>> SAD dataset that I used to solve a recent structure. Whilst I have been able
>>>>> to find a figure-of-merit for the data after phasing, I can't work out how
>>>>> to get any other statistics (e.g. phasing power or an equivalent or
>>>>> Rcullis). Does anyone know a good route to obtaining useful statistics to
>>>>> put in the paper for SAD data?
>>>>>
>>>>> The structure solution was carried out using SHELX C/D/E and then
>>>>> ARP/wARP.
>>>>>
>>>>> Thanks in advance,
>>>>>
>>>>> Nic Harmer
>>>>>
>>>>> =====================
>>>>> Dr. Nic Harmer
>>>>> School of Biosciences
>>>>> University of Exeter
>>>>> tel: +44 1392 725179
>>>>>
>>>
>>> *P** **please don't print this e-mail unless you really need to*
>>> Anastassis (Tassos) Perrakis, Principal Investigator / Staff Member
>>> Department of Biochemistry (B8)
>>> Netherlands Cancer Institute, Dept. B8, 1066 CX Amsterdam, The
>>> Netherlands
>>> Tel: +31 20 512 1951 Fax: +31 20 512 1954 Mobile / SMS: +31 6 28 597791
>>>
>>>
>>>
>>>
>