JiscMail Logo
Email discussion lists for the UK Education and Research communities

Help for CCP4BB Archives


CCP4BB Archives

CCP4BB Archives


CCP4BB@JISCMAIL.AC.UK


View:

Message:

[

First

|

Previous

|

Next

|

Last

]

By Topic:

[

First

|

Previous

|

Next

|

Last

]

By Author:

[

First

|

Previous

|

Next

|

Last

]

Font:

Proportional Font

LISTSERV Archives

LISTSERV Archives

CCP4BB Home

CCP4BB Home

CCP4BB  September 2009

CCP4BB September 2009

Options

Subscribe or Unsubscribe

Subscribe or Unsubscribe

Log In

Log In

Get Password

Get Password

Subject:

Re: Rfree in similar data set

From:

Ian Tickle <[log in to unmask]>

Reply-To:

Ian Tickle <[log in to unmask]>

Date:

Fri, 25 Sep 2009 11:00:05 +0100

Content-Type:

text/plain

Parts/Attachments:

Parts/Attachments

text/plain (220 lines)

Hi Ed,

I already did essentially that many years ago for the Rfree papers in
Acta (sorry I don't have the exactly the data I used any more since
those exact datasets weren't deposited, but it would not be too hard for
anyone to reconstruct the experiment along the lines you suggest).  The
conclusion was that it made no difference, the Rfree is the same (or at
least insignificantly different) *provided* that in doing your shaking
you haven't shaken it into a different local optimum (usually with a
worse likelihood) - then obviously you don't expect to get the same
answer.  To me this conclusion is hardly earth-shattering - if you
refine two different models using the same data to the same optimum, you
must get the same Rfree, since Rfree (and all other refinement
statistics) depend only on the current model parameters and not on
anything you did previously.

Cheers

-- Ian

> -----Original Message-----
> From: [log in to unmask] [mailto:[log in to unmask]]
On
> Behalf Of Edward A. Berry
> Sent: 24 September 2009 20:53
> To: [log in to unmask]
> Cc: [log in to unmask]
> Subject: Re: [ccp4bb] Rfree in similar data set
> 
> This issue has come up from time to time, and I don't think
> anyone has been convinced to change their mind by the
> theoretical discussions.
> But isn't this amenable to test experimentally?
> Given the ready availability of CPU time . . .
> 
> Take a particular structure, preferably a deposited PDB structure
> so that it is a fixed starting point available to everyone.
> Select a refinement strategy (which will determine radius of
> convergence?).
> 
> 1. Refine the structure to convergence with the original free set.
> R-free may be different from original depositors because of different
> strategy, refinement target, and bulk solvent model, so this will
> be the reference to compare subsequent results with.
> 
> 2. Take that structure (1), select a new R-free set, again refine
> to convergence. R-free will be different, but should not be
significantly
> different, from that in 1. Might need to refine against 5 or 10
different
> R-free choices to see what is significant.
> 
> 3. Take the structure from (2), refine using the original free set.
> According to one school, R-free is hopelessly corrupted and will never
> rise to the original level (1) unless drastic refinement steps are
> taken to "shake out" the bias.
> The other school would predict R-free to converge on the same value
(1),
> provided drastic steps are *not* taken, as they might allow the
refinement
> to jump into a different local maximum.
> 
> Then measure RMS- and maximum- atomic deviations between the models
> and see if there are any differences that a PDB user would care about.
> 
> This does not directly address the original poster's question, in
which
> a new set of data is being used. However I think we would agree that
> if there is no bias when exactly the same data is being used, there
> would be none in the case of different data.
> 
> Even if the R-free is biased by this test, it may not be in the
> case of a new dataset- the "noise" which is being "overfit" in the
> new dataset could be completely independent from that in the old.
> However I think it is generally agreed that there is a component
> of "noise" (in the most general sense, meaning the difference
> between what is calculated from our best model and what is observed)
> which is common between different crystals.
> 
> Ed
> 
> Ian Tickle wrote:
> >> -----Original Message-----
> >> From: Dale Tronrud [mailto:[log in to unmask]]
> >> Sent: 24 September 2009 17:21
> >> To: Ian Tickle
> >> Cc: [log in to unmask]
> >> Subject: Re: [ccp4bb] Rfree in similar data set
> >
> >>    While I agree with Ian on the theoretical level, in practice
> >> people use free R's to make decisions before the ultimate model
> >> is finished, and our refinement programs are still limited in
> >> their abilities to find even a local minimum.
> >
> > I wasn't saying that Rfree is only useful for the ultimate finished
> > model.  My argument also applies to all intermediate models; the
> > criterion is that the refinement has converged against the current
> > working set, even if it is only an incomplete model, or if it is
only to
> > a local optimum.  So it's perfectly possible to use Rfree for
> > overfitting & other tests on intermediate models.  The point is that
it
> > doesn't matter how you arrived at that optimum (whether local or
> > global), Rfree is a function only of the parameters at that point,
not
> > of any previous history.  If you arrived at that same local or
global
> > optimum via a path which didn't involve switching datasets midway,
you
> > must get the same answer for Rfree, so I just don't see how it can
be
> > biased one way and not biased the other.  Note that this is meant as
a
> > 'thought experiment', I'm not saying necessarily that it's possible
to
> > perform this experiment in practice!
> >
> >>    On the automated level the test set is used, sometimes, to
> >> determine bulk solvent parameter, and more importantly to calibrate
> >> the likelihood calculations in refinement.  If the test set is
> >> not "free" the likelihood calculation will overestimate the
> > reliability
> >> of the model and I'm not confident that error will not become
> >> a self-fulfilling prophecy.  It is not useful to divine meaning
> >> from the free R until convergence is achieved, but the test
> >> set is used from the first cycle.
> >
> > That is indeed a fair point, but I would maintain that the test set
> > becomes 'free', i.e. free of the memory of all previous models, the
> > first time you reach convergence, so the question of the effect on
> > sigmaA calculations, which use the test set, is only relevant to the
> > first refinement after switching test sets, thereafter it should be
> > irrelevant.  Converging to a local or global optimum wipes out all
> > memory of previous models because the parameter values at that
optimum
> > are independent of any previous history, and so Rfree must be the
same
> > for that optimum no matter what path you took to get there.
> >
> >>    Perhaps I'm in one of my more persnickety moods, but every
> >> paper I've read about optimization algorithms say that the method
> >> requires a number of iteration many times the number of parameters
> >> in the model.  The methods used in refinement programs are pretty
> >> amazing in their ability to drop the residuals with a small number
> >> of cycles, but we are violating the mathematical warranty on
> >> each and every one of them.   A refinement program will produce
> >> a model that is close to optimal, but cannot be expected to be
> >> optimal.  Since we haven't seen an optimal model yet it's hard
> >> to say how far we are off.
> >
> > I thought that for a quadratic approximation CG requires a number of
> > iterations that is not more than the number of parameters (not that
we
> > ever use even that many iterations!)?  Anyway that's a problem in
> > theory, but it's possible to refine until nothing more 'interesting'
> > happens, i.e. further changes appear to be purely random and at the
> > level of rounding errors.  Plotting the maximum shift of the atom
> > positions or B factors from one iteration to the next is a very
> > sensitive way of detecting whether convergence has been achieved;
> > looking at changes in R factors or in RMSDs of bonds etc is a bad
way,
> > since R factors are not sensitive to small changes and atoms can
move in
> > concert without affecting bond lengths etc. (or it may just be the
> > waters that are moving!).
> >
> > As a final point I would note that cell parameters frequently vary
by
> > several % between crystals even from the same batch due to
unavoidable
> > variations in rates of freezing etc, so what you think are
independent
> > test set reflections may in reality overlap significantly in
reciprocal
> > space with working set reflections from another dataset anyway!
> >
> > -- Ian
> >
> >
> > Disclaimer
> > This communication is confidential and may contain privileged
> information intended solely for the named addressee(s). It may not be
used
> or disclosed except for the purpose for which it has been sent. If you
are
> not the intended recipient you must not review, use, disclose, copy,
> distribute or take any action in reliance upon it. If you have
received
> this communication in error, please notify Astex Therapeutics Ltd by
> emailing [log in to unmask] and destroy all copies of the
> message and any attached documents.
> > Astex Therapeutics Ltd monitors, controls and protects all its
messaging
> traffic in compliance with its corporate email policy. The Company
accepts
> no liability or responsibility for any onward transmission or use of
> emails and attachments having left the Astex Therapeutics domain.
Unless
> expressly stated, opinions in this message are those of the individual
> sender and not of Astex Therapeutics Ltd. The recipient should check
this
> email and any attachments for the presence of computer viruses. Astex
> Therapeutics Ltd accepts no liability for damage caused by any virus
> transmitted by this email. E-mail is susceptible to data corruption,
> interception, unauthorized amendment, and tampering, Astex
Therapeutics
> Ltd only send and receive e-mails on the basis that the Company is not
> liable for any such alteration or any consequences thereof.
> > Astex Therapeutics Ltd., Registered in England at 436 Cambridge
Science
> Park, Cambridge CB4 0QA under number 3751674
> >



Disclaimer
This communication is confidential and may contain privileged information intended solely for the named addressee(s). It may not be used or disclosed except for the purpose for which it has been sent. If you are not the intended recipient you must not review, use, disclose, copy, distribute or take any action in reliance upon it. If you have received this communication in error, please notify Astex Therapeutics Ltd by emailing [log in to unmask] and destroy all copies of the message and any attached documents. 
Astex Therapeutics Ltd monitors, controls and protects all its messaging traffic in compliance with its corporate email policy. The Company accepts no liability or responsibility for any onward transmission or use of emails and attachments having left the Astex Therapeutics domain.  Unless expressly stated, opinions in this message are those of the individual sender and not of Astex Therapeutics Ltd. The recipient should check this email and any attachments for the presence of computer viruses. Astex Therapeutics Ltd accepts no liability for damage caused by any virus transmitted by this email. E-mail is susceptible to data corruption, interception, unauthorized amendment, and tampering, Astex Therapeutics Ltd only send and receive e-mails on the basis that the Company is not liable for any such alteration or any consequences thereof.
Astex Therapeutics Ltd., Registered in England at 436 Cambridge Science Park, Cambridge CB4 0QA under number 3751674

Top of Message | Previous Page | Permalink

JiscMail Tools


RSS Feeds and Sharing


Advanced Options


Archives

March 2024
February 2024
January 2024
December 2023
November 2023
October 2023
September 2023
August 2023
July 2023
June 2023
May 2023
April 2023
March 2023
February 2023
January 2023
December 2022
November 2022
October 2022
September 2022
August 2022
July 2022
June 2022
May 2022
April 2022
March 2022
February 2022
January 2022
December 2021
November 2021
October 2021
September 2021
August 2021
July 2021
June 2021
May 2021
April 2021
March 2021
February 2021
January 2021
December 2020
November 2020
October 2020
September 2020
August 2020
July 2020
June 2020
May 2020
April 2020
March 2020
February 2020
January 2020
December 2019
November 2019
October 2019
September 2019
August 2019
July 2019
June 2019
May 2019
April 2019
March 2019
February 2019
January 2019
December 2018
November 2018
October 2018
September 2018
August 2018
July 2018
June 2018
May 2018
April 2018
March 2018
February 2018
January 2018
December 2017
November 2017
October 2017
September 2017
August 2017
July 2017
June 2017
May 2017
April 2017
March 2017
February 2017
January 2017
December 2016
November 2016
October 2016
September 2016
August 2016
July 2016
June 2016
May 2016
April 2016
March 2016
February 2016
January 2016
December 2015
November 2015
October 2015
September 2015
August 2015
July 2015
June 2015
May 2015
April 2015
March 2015
February 2015
January 2015
December 2014
November 2014
October 2014
September 2014
August 2014
July 2014
June 2014
May 2014
April 2014
March 2014
February 2014
January 2014
December 2013
November 2013
October 2013
September 2013
August 2013
July 2013
June 2013
May 2013
April 2013
March 2013
February 2013
January 2013
December 2012
November 2012
October 2012
September 2012
August 2012
July 2012
June 2012
May 2012
April 2012
March 2012
February 2012
January 2012
December 2011
November 2011
October 2011
September 2011
August 2011
July 2011
June 2011
May 2011
April 2011
March 2011
February 2011
January 2011
December 2010
November 2010
October 2010
September 2010
August 2010
July 2010
June 2010
May 2010
April 2010
March 2010
February 2010
January 2010
December 2009
November 2009
October 2009
September 2009
August 2009
July 2009
June 2009
May 2009
April 2009
March 2009
February 2009
January 2009
December 2008
November 2008
October 2008
September 2008
August 2008
July 2008
June 2008
May 2008
April 2008
March 2008
February 2008
January 2008
December 2007
November 2007
October 2007
September 2007
August 2007
July 2007
June 2007
May 2007
April 2007
March 2007
February 2007
January 2007


JiscMail is a Jisc service.

View our service policies at https://www.jiscmail.ac.uk/policyandsecurity/ and Jisc's privacy policy at https://www.jisc.ac.uk/website/privacy-notice

For help and support help@jisc.ac.uk

Secured by F-Secure Anti-Virus CataList Email List Search Powered by the LISTSERV Email List Manager