Well, I was pointing to the Sivia & David (1994) paper because I thought
it might be helpful in the discussion about how to convert intensities
to amplitudes. The paper is probably not so well known in the PX
community, so I decided that I would advertise it on this BB. However,
since I am not one of the authors, I feel that it is inappropriate for
me to go into a detailed defense of every sentence and equation which is
written in it.

The paper is clear and speaks for itself. I can only recommend a careful 
reading of it.


I will nevertheless make some general comments in response to the
criticism that was raised:

Quoting Ian Tickle <[log in to unmask]>:

> But there's a fundamental difference in approach, the authors here
> assume the apparently simpler prior distribution P(I) = 0 for I < 0 &
> P(I) = const for I >= 0.  As users of Bayesian priors well know this is
> an improper prior since it integrates to infinity instead of unity.


Despite of their disparaging name, improper priors can be used in
Bayesian analysis without major difficulties (at least for estimation
problems), provided that the posterior integrates to a finite value.
If you object to the use of an improper prior in the Sivia & David
paper, I suggest to use a prior where P(I) = 0 for I < 0 as well as
for I > 10^30 and P(I) = constant in between these two boundaries.
Technically speaking this would then be a proper prior, but for all
intents and purposes it would not make any difference at all.


> This means that, unlike the case I described for the French & Wilson
> formula based on the Wilson distribution which gives unbiased estimates
> of the true I's and their average, the effect on the corrected
> intensities of using this prior really will be to increase all
> intensities (since the mean I for this prior PDF is also infinite!),
> hence the intensities and their average must be biased (& I'm sure the
> same goes for the corresponding F's).


Two different "bias" concepts in this statement : "... unbiased
estimates of the true I's and their average..."

(1) Regarding "unbiased estimates of the true I's":

The use of a Wilson prior does by no means guarantee that the
posterior expectation values will be unbiased estimates of the true
I's. Whether one uses the Wilson prior or the naive prior of Sivia &
David, the posterior probability distribution on I will be a truncated
normal distribution (see French & Wilson, appendix A). There is nothing
which allows us to claim that the expectation value (which is what we
use as estimate of the true intensity) over such a posterior will be
unbiased (whichever prior was used !).

Simple example: take a reflection which has true F=0. The posterior
probability distribution p_J(J|I) (here I am using the French & Wilson
notation) will be a truncated normal (see French & Wilson, appendix A)
and its expectation value E_J(J|I) will thus always be greater than 0,
even if the Wilson prior is used ! Both the the French & Wilson and the
Sivia & David procedures will yield a biased estimate of the true
intensity: the estimate will always be greater than 0 (the true value),
whatever the measured I is.

(2) Regarding intensity averages:

Here, your argument about "bias" seems to be about averages of
intensities computed in resolution shells, i.e. you are concerned that
the "corrected" I's, averaged over all reflections in a given resolution
bin, should equal the average of the uncorrected intensities in the same
resolution bin. I would like to see a proof that the French & Wilson
procedure actually achieves this goal (none is given in the French & 
Wilson paper - they are actually not addressing this issue). But apart 
from this, I wonder whether this is of any relevance at all. Why would 
this be so important ? Why are you so concerned that the intensity 
averages over many different reflections in a resolution bin is a 
quantity which should at all price be conserved ?


In any event, I think that the discussions about "bias" on corrected
intensities is a somewhat academic side-issue. The real reason why we
use the "truncate" procedure is not so much do get corrected I's, but
rather to get estimates of the amplitudes. In that sense, I think that
the important message conveyed in the Sivia & David paper is the
following: the awkward truncated Gaussian pdf's in intensity space
(whichever prior was used...) are transformed to well-behaved
Gaussian-like pdf's in amplitude-space. This is an argument in favouring 
F's rather than I's (even corrected I's) for subsequent crystallographic 
computations. In that regime (i.e. in the regime where we accept that 
the posterior probability distribution on F's is close to a Gaussian), 
the estimator given by equation (11) in Sivia & David is actually unbiased !

Side argument: to use the French & Wilson procedure, it is necessary to
know the crystal spacegroup (in order to apply the correct statistical
weights for the various classes of reflections). To use the Sivia &
David procedure, you don't need to know the spacegroup. Now, I think
that it should be possible to convert integrated intensities from a
diffraction image to amplitudes without knowing the spacegroup of the
crystal that produced these diffraction images. Simply consider that 
converting I's to F's is just one other step in the data reduction 
process. If you got the spacegroup wrong, the French & Wilson procedure 
will distort the intensities (and amplitudes) towards the statistics 
corresponding to the wrongly assigned spacegroup (bias !). The Sivia & 
David procedure is immune against such problems. It is also immune 
against any other problems that may affect the intensity statistics of 
the data, such as anisotropic intensity falloff, pseudo-symmetries, etc. 
In all these cases, application of the French & Wilson procedure is 
problematic and the Sivia & David method would be a sensible alternative.

And finally, the nice extension of the Sivia & David method to the
case of overlapping reflections in a powder pattern (described in
section 4 of their paper) could easily be adapted to handle overlapping
reflections in single-crystal data (e.g. in the case of twinning).

Thus, the Sivia & David (1994) paper deserves our careful attention.

Marc