Just to add that the R-free and R-sleep are nothing else than what is
'officially' known as 'test' set and 'validation' set.
In statistical pattern recognition for example, one would use a
'training' set to establish the 'model'; the power of the model
(in this case the 'model' can be for example a neural network scheme)
is judged against a 'test' set that is kept aside from training.
However, the model is chosen and improved against this 'test' set!
Thus, at the end the 'performance' of the model has to be documented
against the 'validation' test, that has not been used for anything
else !
Since R-free set is often - fortunately - referred to as 'test set',
R-sleep should maybe referred to as 'validation set'.
Although in principle very appropriate and quite standard in other
fields, I agree with Ian, that its practical use
can be questioned and might need some investigation. I suspect that
if one would engage in some kind of automation
project such a whole-PDB re-refinement (brrrrr ....) an R-sleep might
be necessary, since refinement protocols will have
to be chosen against the R-free to achieve automation.
Tassos
On Oct 1, 2007, at 16:21, Ian Tickle wrote:
> The question is how significant is this bias, and is the cure (i.e.
> leaving out more reflections from the working set) worse than the
> disease? For refinements at 'medium' typical resolution (around
> 2.5 to 2 Ang) we are working with an observation/parameter count
> ratio of say < 3 (naturally I'm counting the geometric restraints
> with the X-ray observations). The amount of bias in Rwork and
> other statistics derived from the working set depends critically on
> how close the obs/param ratio is to 1. The Rfree optimisation is
> used only to determine weighting parameters (including sigma-A) and
> it's unlikely there will be more than say 20 of these. Typically
> there are at least 1000 refls in the test set, so for the Rfree
> optimisation the obs/param ratio will be around 50. This is much
> larger than the obs/param ratio for Rwork and may well mean that
> the biasing effect on Rfree is negligible. It should be easy to do
> some tests comparing Rfree with Rsleep to check the bias (taking
> into account errors to limited sample sizes of course), and also to
> see what are the effects of leaving out the sleeping set on the
> refinement and the maps. I don't think it would be wise to rush
> into this until we have done proper evaluations.
>
> -- Ian
>
>> -----Original Message-----
>> From: [log in to unmask]
>> [mailto:[log in to unmask]] On Behalf Of Mark J. van Raaij
>> Sent: 01 October 2007 14:58
>> To: [log in to unmask]
>> Subject: R-sleep
>>
>> Dear All,
>>
>> the short paper by Gerard Kleywegt (ActaD 63, 939-940) treats
>> an interesting subject (at least I think so...). I agree that
>> what we are now doing in many cases is effectively refining
>> against Rfree. For example, the standard CNS torsion angle
>> refinement does n refinement trials with randomised starting
>> points. If you then take the one with lowest Rfree (or let a
>> script do this for you), you are biasing Rfree!
>> Therefore, his proposal to put an extra set of reflections in
>> a dormant "vault" (R-sleep) sounds like a good idea to me.
>> However, how would the "vault" be implemented to be
>> effective? If left to the experimenter, it would be very
>> tempting to check R-sleep once in a while (or often) during
>> refinement, rendering it useless as an unbiased validator.
>>
>> or am I being paranoid and too pessimistic?
>>
>> Mark J. van Raaij
>> Unidad de Bioquímica Estructural
>> Dpto de Bioquímica, Facultad de Farmacia
>> and
>> Unidad de Rayos X, Edificio CACTUS
>> Universidad de Santiago
>> 15782 Santiago de Compostela
>> Spain
>> http://web.usc.es/~vanraaij/
>>
>>
>>
>
>
> Disclaimer
> This communication is confidential and may contain privileged
> information intended solely for the named addressee(s). It may not
> be used or disclosed except for the purpose for which it has been
> sent. If you are not the intended recipient you must not review,
> use, disclose, copy, distribute or take any action in reliance upon
> it. If you have received this communication in error, please notify
> Astex Therapeutics Ltd by emailing [log in to unmask]
> and destroy all copies of the message and any attached documents.
> Astex Therapeutics Ltd monitors, controls and protects all its
> messaging traffic in compliance with its corporate email policy.
> The Company accepts no liability or responsibility for any onward
> transmission or use of emails and attachments having left the Astex
> Therapeutics domain. Unless expressly stated, opinions in this
> message are those of the individual sender and not of Astex
> Therapeutics Ltd. The recipient should check this email and any
> attachments for the presence of computer viruses. Astex
> Therapeutics Ltd accepts no liability for damage caused by any
> virus transmitted by this email. E-mail is susceptible to data
> corruption, interception, unauthorized amendment, and tampering,
> Astex Therapeutics Ltd only send and receive e-mails on the basis
> that the Company is not liable for any such alteration or any
> consequences thereof.
> Astex Therapeutics Ltd., Registered in England at 436 Cambridge
> Science Park, Cambridge CB4 0QA under number 3751674
|