> -----Original Message-----
> From: Dale Tronrud [mailto:[log in to unmask]]
> Sent: 24 September 2009 17:21
> To: Ian Tickle
> Cc: [log in to unmask]
> Subject: Re: [ccp4bb] Rfree in similar data set

>    While I agree with Ian on the theoretical level, in practice
> people use free R's to make decisions before the ultimate model
> is finished, and our refinement programs are still limited in
> their abilities to find even a local minimum.

I wasn't saying that Rfree is only useful for the ultimate finished
model.  My argument also applies to all intermediate models; the
criterion is that the refinement has converged against the current
working set, even if it is only an incomplete model, or if it is only to
a local optimum.  So it's perfectly possible to use Rfree for
overfitting & other tests on intermediate models.  The point is that it
doesn't matter how you arrived at that optimum (whether local or
global), Rfree is a function only of the parameters at that point, not
of any previous history.  If you arrived at that same local or global
optimum via a path which didn't involve switching datasets midway, you
must get the same answer for Rfree, so I just don't see how it can be
biased one way and not biased the other.  Note that this is meant as a
'thought experiment', I'm not saying necessarily that it's possible to
perform this experiment in practice!

>    On the automated level the test set is used, sometimes, to
> determine bulk solvent parameter, and more importantly to calibrate
> the likelihood calculations in refinement.  If the test set is
> not "free" the likelihood calculation will overestimate the
reliability
> of the model and I'm not confident that error will not become
> a self-fulfilling prophecy.  It is not useful to divine meaning
> from the free R until convergence is achieved, but the test
> set is used from the first cycle.

That is indeed a fair point, but I would maintain that the test set
becomes 'free', i.e. free of the memory of all previous models, the
first time you reach convergence, so the question of the effect on
sigmaA calculations, which use the test set, is only relevant to the
first refinement after switching test sets, thereafter it should be
irrelevant.  Converging to a local or global optimum wipes out all
memory of previous models because the parameter values at that optimum
are independent of any previous history, and so Rfree must be the same
for that optimum no matter what path you took to get there.

>    Perhaps I'm in one of my more persnickety moods, but every
> paper I've read about optimization algorithms say that the method
> requires a number of iteration many times the number of parameters
> in the model.  The methods used in refinement programs are pretty
> amazing in their ability to drop the residuals with a small number
> of cycles, but we are violating the mathematical warranty on
> each and every one of them.   A refinement program will produce
> a model that is close to optimal, but cannot be expected to be
> optimal.  Since we haven't seen an optimal model yet it's hard
> to say how far we are off.

I thought that for a quadratic approximation CG requires a number of
iterations that is not more than the number of parameters (not that we
ever use even that many iterations!)?  Anyway that's a problem in
theory, but it's possible to refine until nothing more 'interesting'
happens, i.e. further changes appear to be purely random and at the
level of rounding errors.  Plotting the maximum shift of the atom
positions or B factors from one iteration to the next is a very
sensitive way of detecting whether convergence has been achieved;
looking at changes in R factors or in RMSDs of bonds etc is a bad way,
since R factors are not sensitive to small changes and atoms can move in
concert without affecting bond lengths etc. (or it may just be the
waters that are moving!).

As a final point I would note that cell parameters frequently vary by
several % between crystals even from the same batch due to unavoidable
variations in rates of freezing etc, so what you think are independent
test set reflections may in reality overlap significantly in reciprocal
space with working set reflections from another dataset anyway!

-- Ian


Disclaimer
This communication is confidential and may contain privileged information intended solely for the named addressee(s). It may not be used or disclosed except for the purpose for which it has been sent. If you are not the intended recipient you must not review, use, disclose, copy, distribute or take any action in reliance upon it. If you have received this communication in error, please notify Astex Therapeutics Ltd by emailing [log in to unmask] and destroy all copies of the message and any attached documents. 
Astex Therapeutics Ltd monitors, controls and protects all its messaging traffic in compliance with its corporate email policy. The Company accepts no liability or responsibility for any onward transmission or use of emails and attachments having left the Astex Therapeutics domain.  Unless expressly stated, opinions in this message are those of the individual sender and not of Astex Therapeutics Ltd. The recipient should check this email and any attachments for the presence of computer viruses. Astex Therapeutics Ltd accepts no liability for damage caused by any virus transmitted by this email. E-mail is susceptible to data corruption, interception, unauthorized amendment, and tampering, Astex Therapeutics Ltd only send and receive e-mails on the basis that the Company is not liable for any such alteration or any consequences thereof.
Astex Therapeutics Ltd., Registered in England at 436 Cambridge Science Park, Cambridge CB4 0QA under number 3751674