Hi Tim,
I don't follow your point...frames are just data, and with more information than after integration. The data after integration is also to some extent dependent on the beamline.
It should indeed give a more accurate description of the crystal contents - whether that in turn will translate into greater chemical or biological insight (now or some time in the future) will depend on the specific case (and on the interpreter).
Mark
Mark J van Raaij
Lab 20B
Dpto de Estructura de Macromoleculas
Centro Nacional de Biotecnologia - CSIC
c/Darwin 3
E-28049 Madrid, Spain
tel. (+34) 91 585 4616
http://www.cnb.csic.es/~mjvanraaij
On 24 Jun 2013, at 13:59, Tim Gruene wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Dear John,
>
> actually I am not a friend of this idea. Processing software make an
> excellent job of removing the instrumental part from our data. If we
> start to integrate against frames, the next structural title might be
> something like "Crystal structure of ABC a xA resolution measured at
> beamline xyz with a frame width of f degrees and a total rotation
> range of phi degreees..." the point I am trying to make: once
> integrating against frames one may have to take a lot of issues into
> account for interpreting the structure.
> And do you think that refining against frames will actually give
> greater chemical or biological insight into the sample, or will it
> only give a more accurate description of the crystal contents? These
> are two different things and the latter is - in my opinion - not what
> structures are about.
>
> Best, Tim
>
> P.S.: I changed the subject line, because the thread based sorting of
> my emails is soon going to exceed the width of my screem for the
> original one.
>
> On 06/24/2013 08:13 AM, Jrh wrote:
>> Dear Tom, I find this suggestion of using the full images an
>> excellent and visionary one. So, how to implement it? We are part
>> way along the path with James Holton's reverse Mosflm. The computer
>> memory challenge could be ameliorated by simple pixel averaging at
>> least initially. The diffuse scattering would be the ultimate gold
>> at the end of the rainbow. Peter Moore's new book, inter alia,
>> carries many splendid insights into the diffuse scattering in our
>> diffraction patterns. Fullprof analyses have become a firm trend in
>> other fields, admittedly with simpler computing overheads.
>> Greetings, John
>>
>> Prof John R Helliwell DSc FInstP
>>
>>
>>
>> On 21 Jun 2013, at 23:16, "Terwilliger, Thomas C"
>> <[log in to unmask]> wrote:
>>
>>> I hope I am not duplicating too much of this fascinating
>>> discussion with these comments: perhaps the main reason there is
>>> confusion about what to do is that neither F nor I is really the
>>> most suitable thing to use in refinement. As pointed out several
>>> times in different ways, we don't measure F or I, we only measure
>>> counts on a detector. As a convenience, we "process" our
>>> diffraction images to estimate I or F and their uncertainties and
>>> model these uncertainties as simple functions (e.g., a Gaussian).
>>> There is no need in principle to do that, and if we were to
>>> refine instead against the raw image data these issues about
>>> positivity would disappear and our structures might even be a
>>> little better.
>>>
>>> Our standard procedure is to estimate F or I from counts on the
>>> detector, then to use these estimates of F or I in refinement.
>>> This is not so easy to do right because F or I contain many terms
>>> coming from many pixels and it is hard to model their statistics
>>> in detail. Further, attempts we make to estimate either F or I
>>> as physically plausible values (e.g., using the fact that they
>>> are not negative) will generally be biased (the values after
>>> correction will generally be systematically low or systematically
>>> high, as is true for the French and Wilson correction and as
>>> would be true for the truncation of I at zero or above).
>>>
>>> Randy's method for intensity refinement is an improvement because
>>> the statistics are treated more fully than just using an estimate
>>> of F or I and assuming its uncertainty has a simple distribution.
>>> So why not avoid all the problems with modeling the statistics of
>>> processed data and instead refine against the raw data. From the
>>> structural model you calculate F, from F and a detailed model of
>>> the experiment (the same model that is currently used in data
>>> processing) you calculate the counts expected on each pixel. Then
>>> you calculate the likelihood of the data given your models of the
>>> structure and of the experiment. This would have lots of
>>> benefits because it would allow improved descriptions of the
>>> experiment (decay, absorption, detector sensitivity, diffuse
>>> scattering and other "background" on the images,....on and on)
>>> that could lead to more accurate structures in the end. Of
>>> course there are some minor issues about putting all this in
>>> computer memory for refinement....
>>>
>>> -Tom T ________________________________________ From: CCP4
>>> bulletin board [[log in to unmask]] on behalf of Phil
>>> [[log in to unmask]] Sent: Friday, June 21, 2013 2:50 PM To:
>>> [log in to unmask] Subject: Re: [ccp4bb] ctruncate bug?
>>>
>>> However you decide to argue the point, you must consider _all_
>>> the observations of a reflection (replicates and symmetry
>>> related) together when you infer Itrue or F etc, otherwise you
>>> will bias the result even more. Thus you cannot (easily) do it
>>> during integration
>>>
>>> Phil
>>>
>>> Sent from my iPad
>>>
>>> On 21 Jun 2013, at 20:30, Douglas Theobald
>>> <[log in to unmask]> wrote:
>>>
>>>> On Jun 21, 2013, at 2:48 PM, Ed Pozharski
>>>> <[log in to unmask]> wrote:
>>>>
>>>>> Douglas,
>>>>>>> Observed intensities are the best estimates that we can
>>>>>>> come up with in an experiment.
>>>>>> I also agree with this, and this is the clincher. You are
>>>>>> arguing that Ispot-Iback=Iobs is the best estimate we can
>>>>>> come up with. I claim that is absurd. How are you
>>>>>> quantifying "best"? Usually we have some sort of
>>>>>> discrepancy measure between true and estimate, like RMSD,
>>>>>> mean absolute distance, log distance, or somesuch. Here is
>>>>>> the important point --- by any measure of discrepancy you
>>>>>> care to use, the person who estimates Iobs as 0 when
>>>>>> Iback>Ispot will *always*, in *every case*, beat the person
>>>>>> who estimates Iobs with a negative value. This is an
>>>>>> indisputable fact.
>>>>>
>>>>> First off, you may find it useful to avoid such words as
>>>>> absurd and indisputable fact. I know political correctness
>>>>> may be sometimes overrated, but if you actually plan to have
>>>>> meaningful discussion, let's assume that everyone responding
>>>>> to your posts is just trying to help figure this out.
>>>>
>>>> I apologize for offending and using the strong words --- my
>>>> intention was not to offend. This is just how I talk when
>>>> brainstorming with my colleagues around a blackboard, but of
>>>> course then you can see that I smile when I say it.
>>>>
>>>>> To address your point, you are right that J=0 is closer to
>>>>> "true intensity" then a negative value. The problem is that
>>>>> we are not after a single intensity, but rather all of them,
>>>>> as they all contribute to electron density reconstruction.
>>>>> If you replace negative Iobs with E(J), you would
>>>>> systematically inflate the averages, which may turn
>>>>> problematic in some cases.
>>>>
>>>> So, I get the point. But even then, using any reasonable
>>>> criterion, the whole estimated dataset will be closer to the
>>>> true data if you set all "negative" intensity estimates to 0.
>>>>
>>>>> It is probably better to stick with "raw intensities" and
>>>>> construct theoretical predictions properly to account for
>>>>> their properties.
>>>>>
>>>>> What I was trying to tell you is that observed intensities is
>>>>> what we get from experiment.
>>>>
>>>> But they are not what you get from the detector. The detector
>>>> spits out a positive value for what's inside the spot. It is
>>>> we, as human agents, who later manipulate and massage that data
>>>> value by subtracting the background estimate. A value that has
>>>> been subjected to a crude background subtraction is not the raw
>>>> experimental value. It has been modified, and there must be
>>>> some logic to why we massage the data in that particular
>>>> manner. I agree, of course, that the background should be
>>>> accounted for somehow. But why just subtract it away? There
>>>> are other ways to massage the data --- see my other post to
>>>> Ian. My argument is that however we massage the experimentally
>>>> observed value should be physically informed, and allowing
>>>> negative intensity estimates violates the basic physics.
>>>>
>>>> [snip]
>>>>
>>>>>>> These observed intensities can be negative because while
>>>>>>> their true underlying value is positive, random errorsmay
>>>>>>> result in Iback>Ispot. There is absolutely nothing
>>>>>>> unphysical here.
>>>>>> Yes there is. The only way you can get a negative estimate
>>>>>> is to make unphysical assumptions. Namely, the estimate
>>>>>> Ispot-Iback=Iobs assumes that both the true value of I and
>>>>>> the background noise come from a Gaussian distribution that
>>>>>> is allowed to have negative values. Both of those
>>>>>> assumptions are unphysical.
>>>>>
>>>>> See, I have a problem with this. Both common sense and laws
>>>>> of physics dictate that number of photons hitting spot on a
>>>>> detector is a positive number. There is no law of physics
>>>>> that dictates that under no circumstances there could be
>>>>> Ispot<Iback.
>>>>
>>>> That's not what I'm saying. Sure, Ispot can be less than Iback
>>>> randomly. That does not mean we have to estimate the detected
>>>> intensity as negative, after accounting for background.
>>>>
>>>>> Yes, E(Ispot)>=E(Iback). Yes, E(Ispot-Iback)>=0. But
>>>>> P(Ispot-Iback=0)>0, and therefore experimental sampling of
>>>>> Ispot-Iback is bound to occasionally produce negative values.
>>>>> What law of physics is broken when for a given reflection
>>>>> total number of photons in spot pixels is less that total
>>>>> number of photons in equal number of pixels in the
>>>>> surrounding background mask?
>>>>>
>>>>> Cheers,
>>>>>
>>>>> Ed.
>>>>>
>>>>> -- Oh, suddenly throwing a giraffe into a volcano to make
>>>>> water is crazy? Julian, King of Lemurs
>>
>
> - --
> Dr Tim Gruene
> Institut fuer anorganische Chemie
> Tammannstr. 4
> D-37077 Goettingen
>
> GPG Key ID = A46BEE1A
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.12 (GNU/Linux)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
> iD8DBQFRyDSWUxlJ7aRr7hoRAlmKAKD0MRGp21frbv8LcNG78Y30PPmi9ACdGgR6
> eTfN0+B0XrOgpjIS+wu+KHY=
> =sFxD
> -----END PGP SIGNATURE-----
|