-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Dear Douglas,

why don't you try this and publish the results? As Kay already pointed
out everybody would be delighted if we can get better data - if
including negative intensities leads to better models, I think most
crystallographers could not be bothered if they are non-physical.

Best,
Tim

On 06/20/2013 09:46 PM, Douglas Theobald wrote:
> Well, I tend to think Ian is probably right, that doing things the
> "proper" way (vs French-Wilson) will not make much of a difference
> in the end.
>
> Nevertheless, I don't think refining against the (possibly
> negative) intensities is a good solution to dealing with negative
> intensities --- that just ignores the problem, and will end up
> overweighting large negative intensities. Wouldn't it be better to
> correct the negative intensities with FW and then refine against
> that?
>
>
> On Jun 20, 2013, at 3:38 PM, Kay Diederichs
> <[log in to unmask]> wrote:
>
>> Douglas,
>>
>> as soon as you come up with an algorithm that gives accurate,
>> unbiased intensity estimates together with their standard
>> deviations, everybody will be happy. But I'm not aware of
>> progress in this question (Poisson signal with background) in the
>> last decades - I'd be glad to be proven wrong!
>>
>> Kay
>>
>> Am 20.06.13 21:27, schrieb Douglas Theobald:
>>> Kay, I understand the French-Wilson way of currently doing
>>> things, as you outline below. My point is that it is not
>>> optimal --- we could do things better --- since even
>>> French-Wilson accepts the idea of negative intensity
>>> measurements. I am trying to disabuse the (very stubborn) view
>>> that when the background is more than the spot, the only
>>> possible estimate of the intensity is a negative value. This
>>> is untrue, and unjustified by the physics involved. In
>>> principle, there is no reason to use French-Wilson, as we
>>> should never have reported a negative integrated intensity to
>>> begin with.
>>>
>>> I also understand that (Iobs-Icalc)^2 is not the actual
>>> refinement target, but the same point applies, and the actual
>>> target is based on a fundamental Gaussian assumption for the
>>> Is.
>>>
>>>
>>> On Jun 20, 2013, at 2:13 PM, Kay Diederichs
>>> <[log in to unmask]> wrote:
>>>
>>>> Douglas,
>>>>
>>>> the intensity is negative if the integrated spot has a lower
>>>> intensity than the estimate of the background under the spot.
>>>> So yes, we are not _measuring_ negative intensities, rather
>>>> we are estimating intensities, and that estimate may turn out
>>>> to be negative. In a later step we try to "correct" for this,
>>>> because it is non-physical, as you say. At that point, the
>>>> "proper statistical model" comes into play. Essentially we
>>>> use this as a "prior". In the order of increasing
>>>> information, we can have more or less informative priors for
>>>> weak reflections: 1) I > 0 2) I has a distribution looking
>>>> like the right half of a Gaussian, and we estimate its width
>>>> from the variance of the intensities in a resolution shell 3)
>>>> I follows a Wilson distribution, and we estimate its
>>>> parameters from the data in a resolution shell 4) I must be
>>>> related to Fcalc^2 (i.e. once the structure is solved, we
>>>> re-integrate using the Fcalc as prior) For a given
>>>> experiment, the problem is chicken-and-egg in the sense that
>>>> only if you know the characteristics of the data can you
>>>> choose the correct prior. I guess that using prior 4) would
>>>> be heavily frowned upon because there is a danger of model
>>>> bias. You could say: A Bayesian analysis done properly should
>>>> not suffer from model bias. This is probably true, but the
>>>> theory to ensure the word "properly" is not available at the
>>>> moment. Crystallographers usually use prior 3) which, as I
>>>> tried to point out, also has its weak spots, namely if the
>>>> data do not behave like those of an ideal crystal - and
>>>> today's projects often result in data that would have been
>>>> discarded ten years ago, so they are far from ideal. Prior 2)
>>>> is available as an option in XDSCONV Prior 1) seems to be
>>>> used, or is available, in ctruncate in certain cases (I don't
>>>> know the details)
>>>>
>>>> Using intensities instead of amplitudes in refinement would
>>>> avoid having to choose a prior, and refinement would
>>>> therefore not be compromised in case of data violating the
>>>> assumptions underlying the prior.
>>>>
>>>> By the way, it is not (Iobs-Icalc)^2 that would be optimized
>>>> in refinement against intensities, but rather the
>>>> corresponding maximum likelihood formula (which I seem to
>>>> remember is more complicated than the amplitude ML formula,
>>>> or is not an analytical formula at all, but maybe somebody
>>>> knows better).
>>>>
>>>> best,
>>>>
>>>> Kay
>>>>
>>>>
>>>> On Thu, 20 Jun 2013 13:14:28 -0400, Douglas Theobald
>>>> <[log in to unmask]> wrote:
>>>>
>>>>> I still don't see how you get a negative intensity from
>>>>> that. It seems you are saying that in many cases of a low
>>>>> intensity reflection, the integrated spot will be lower
>>>>> than the background. That is not equivalent to having a
>>>>> negative measurement (as the measurement is actually
>>>>> positive, and sometimes things are randomly less positive
>>>>> than backgroiund). If you are using a proper statistical
>>>>> model, after background correction you will end up with a
>>>>> positive (or 0) value for the integrated intensity.
>>>>>
>>>>>
>>>>> On Jun 20, 2013, at 1:08 PM, Andrew Leslie
>>>>> <[log in to unmask]> wrote:
>>>>>
>>>>>>
>>>>>> The integration programs report a negative intensity
>>>>>> simply because that is the observation.
>>>>>>
>>>>>> Because of noise in the Xray background, in a large
>>>>>> sample of intensity estimates for reflections whose true
>>>>>> intensity is very very small one will inevitably get some
>>>>>> measurements that are negative. These must not be
>>>>>> rejected because this will lead to bias (because some of
>>>>>> these intensities for symmetry mates will be estimated
>>>>>> too large rather than too small). It is not unusual for
>>>>>> the intensity to remain negative even after averaging
>>>>>> symmetry mates.
>>>>>>
>>>>>> Andrew
>>>>>>
>>>>>>
>>>>>> On 20 Jun 2013, at 11:49, Douglas Theobald
>>>>>> <[log in to unmask]> wrote:
>>>>>>
>>>>>>> Seems to me that the negative Is should be dealt with
>>>>>>> early on, in the integration step. Why exactly do
>>>>>>> integration programs report negative Is to begin with?
>>>>>>>
>>>>>>>
>>>>>>> On Jun 20, 2013, at 12:45 PM, Dom Bellini
>>>>>>> <[log in to unmask]> wrote:
>>>>>>>
>>>>>>>> Wouldnt be possible to take advantage of negative Is
>>>>>>>> to extrapolate/estimate the decay of scattering
>>>>>>>> background (kind of Wilson plot of background
>>>>>>>> scattering) to flat out the background and push all
>>>>>>>> the Is to positive values?
>>>>>>>>
>>>>>>>> More of a question rather than a suggestion ...
>>>>>>>>
>>>>>>>> D
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> From: CCP4 bulletin board
>>>>>>>> [mailto:[log in to unmask]] On Behalf Of Ian
>>>>>>>> Tickle Sent: 20 June 2013 17:34 To: ccp4bb Subject:
>>>>>>>> Re: [ccp4bb] ctruncate bug?
>>>>>>>>
>>>>>>>> Yes higher R factors is the usual reason people don't
>>>>>>>> like I-based refinement!
>>>>>>>>
>>>>>>>> Anyway, refining against Is doesn't solve the
>>>>>>>> problem, it only postpones it: you still need the Fs
>>>>>>>> for maps! (though errors in Fs may be less critical
>>>>>>>> then). -- Ian
>>>>>>>>
>>>>>>>> On 20 June 2013 17:20, Dale Tronrud
>>>>>>>> <[log in to unmask]<mailto:[log in to unmask]>>
>>>>>>>> wrote: If you are refining against F's you have to
>>>>>>>> find some way to avoid calculating the square root of
>>>>>>>> a negative number. That is why people have
>>>>>>>> historically rejected negative I's and why Truncate
>>>>>>>> and cTruncate were invented.
>>>>>>>>
>>>>>>>> When refining against I, the calculation of (Iobs -
>>>>>>>> Icalc)^2 couldn't care less if Iobs happens to be
>>>>>>>> negative.
>>>>>>>>
>>>>>>>> As for why people still refine against F... When I
>>>>>>>> was distributing a refinement package it could refine
>>>>>>>> against I but no one wanted to do that. The "R
>>>>>>>> values" ended up higher, but they were looking at R
>>>>>>>> values calculated from F's. Of course the F based R
>>>>>>>> values are lower when you refine against F's, that
>>>>>>>> means nothing.
>>>>>>>>
>>>>>>>> If we could get the PDB to report both the F and I
>>>>>>>> based R values for all models maybe we could get a
>>>>>>>> start toward moving to intensity refinement.
>>>>>>>>
>>>>>>>> Dale Tronrud
>>>>>>>>
>>>>>>>>
>>>>>>>> On 06/20/2013 09:06 AM, Douglas Theobald wrote: Just
>>>>>>>> trying to understand the basic issues here. How
>>>>>>>> could refining directly against intensities solve the
>>>>>>>> fundamental problem of negative intensity values?
>>>>>>>>
>>>>>>>>
>>>>>>>> On Jun 20, 2013, at 11:34 AM, Bernhard Rupp
>>>>>>>> <[log in to unmask]<mailto:[log in to unmask]>>
>>>>>>>> wrote: As a maybe better alternative, we should (once
>>>>>>>> again) consider to refine against intensities (and I
>>>>>>>> guess George Sheldrick would agree here).
>>>>>>>>
>>>>>>>> I have a simple question - what exactly, short of
>>>>>>>> some sort of historic inertia (or memory lapse), is
>>>>>>>> the reason NOT to refine against intensities?
>>>>>>>>
>>>>>>>> Best, BR
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>>
>>>>>>>> This e-mail and any attachments may contain
>>>>>>>> confidential, copyright and or privileged material,
>>>>>>>> and are for the use of the intended addressee only.
>>>>>>>> If you are not the intended addressee or an
>>>>>>>> authorised recipient of the addressee please notify
>>>>>>>> us of receipt by returning the e-mail and do not use,
>>>>>>>> copy, retain, distribute or disclose the information
>>>>>>>> in or attached to the e-mail.
>>>>>>>>
>>>>>>>> Any opinions expressed within this e-mail are those
>>>>>>>> of the individual and not necessarily of Diamond
>>>>>>>> Light Source Ltd.
>>>>>>>>
>>>>>>>> Diamond Light Source Ltd. cannot guarantee that this
>>>>>>>> e-mail or any attachments are free from viruses and
>>>>>>>> we cannot accept liability for any damage which you
>>>>>>>> may sustain as a result of software viruses which may
>>>>>>>> be transmitted in or with the message.
>>>>>>>>
>>>>>>>> Diamond Light Source Limited (company no. 4375679).
>>>>>>>> Registered in England and Wales with its registered
>>>>>>>> office at Diamond House, Harwell Science and
>>>>>>>> Innovation Campus, Didcot, Oxfordshire, OX11 0DE,
>>>>>>>> United Kingdom
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>
>>>
>

- --
Dr Tim Gruene
Institut fuer anorganische Chemie
Tammannstr. 4
D-37077 Goettingen

GPG Key ID = A46BEE1A
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iD8DBQFRw10sUxlJ7aRr7hoRAlCtAKCrM+8Mv1BAw/XATV7iRPs4teaXJwCgy/5T
w8bauyNC6VNa2qw8dRs6hqI=
=2Qn9
-----END PGP SIGNATURE-----