JiscMail Logo
Email discussion lists for the UK Education and Research communities

Help for CCP4BB Archives


CCP4BB Archives

CCP4BB Archives


CCP4BB@JISCMAIL.AC.UK


View:

Message:

[

First

|

Previous

|

Next

|

Last

]

By Topic:

[

First

|

Previous

|

Next

|

Last

]

By Author:

[

First

|

Previous

|

Next

|

Last

]

Font:

Proportional Font

LISTSERV Archives

LISTSERV Archives

CCP4BB Home

CCP4BB Home

CCP4BB  September 2009

CCP4BB September 2009

Options

Subscribe or Unsubscribe

Subscribe or Unsubscribe

Log In

Log In

Get Password

Get Password

Subject:

Re: Rfree in similar data set

From:

Dale Tronrud <[log in to unmask]>

Reply-To:

Dale Tronrud <[log in to unmask]>

Date:

Thu, 24 Sep 2009 09:20:41 -0700

Content-Type:

text/plain

Parts/Attachments:

Parts/Attachments

text/plain (117 lines)

   While I agree with Ian on the theoretical level, in practice
people use free R's to make decisions before the ultimate model
is finished, and our refinement programs are still limited in
their abilities to find even a local minimum.

   On the automated level the test set is used, sometimes, to
determine bulk solvent parameter, and more importantly to calibrate
the likelihood calculations in refinement.  If the test set is
not "free" the likelihood calculation will overestimate the reliability
of the model and I'm not confident that error will not become
a self-fulfilling prophecy.  It is not useful to divine meaning
from the free R until convergence is achieved, but the test
set is used from the first cycle.

   Perhaps I'm in one of my more persnickety moods, but every
paper I've read about optimization algorithms say that the method
requires a number of iteration many times the number of parameters
in the model.  The methods used in refinement programs are pretty
amazing in their ability to drop the residuals with a small number
of cycles, but we are violating the mathematical warranty on
each and every one of them.   A refinement program will produce
a model that is close to optimal, but cannot be expected to be
optimal.  Since we haven't seen an optimal model yet it's hard
to say how far we are off.

   If you have the ability to choose a test set that is "unbiased"
you might as well do so.

Dale Tronrud


Ian Tickle wrote:
>> -----Original Message-----
>> From: [log in to unmask] [mailto:[log in to unmask]]
> On
>> Behalf Of Eric Bennett
>> Sent: 24 September 2009 13:31
>> To: [log in to unmask]
>> Subject: Re: [ccp4bb] Rfree in similar data set
>>
>> Ian Tickle wrote:
>>
>>> For that to
>>> be true it would have to be possible to arrive at a different
> unbiased
>>> Rfree from another starting point.  But provided your starting point
>>> wasn't a local maximum LL and you haven't gotten into a local maximum
>>> along the way, convergence will be to a unique global maximum of the
> LL,
>>> so the Rfree must be the same whatever starting point is used (within
>>> the radius of convergence of course).
>> But if you're using a different set of data the minima and maxima of
>> the function aren't necessarily going to be in the same place.  Rfree
>> is supposed to inform about overfitting.  In an overfitting situation
>> there are multiple possible models which describe the data well and
>> which overfit solution you end up with could be sensitive to the data
>> set used.  The provisions that you haven't gotten stuck in a local
>> maximum and are within radius of convergence don't seem safe
>> considering historical situations that led to the introduction of
>> Rfree.  What algorithm is going to converge main chain tracing errors
>> to the correct maximum?  Thinking about that situation, isn't part of
>> the goal of Rfree to give you a hint in situations where you have, in
>> fact, gotten stuck in a local maximum due to a significant error in
>> the model that places it outside the radius of convergence of the
>> refinement algorithm?
> 
> Hi Eric,
> 
> Yes clearly the function optima won't necessarily be in the same place
> for different datasets; the question is whether the distance between the
> optima is less than the convergence radius.  This will depend largely on
> whether the datasets have similar dmin; if they do then the differences
> will be largely random measurement errors (I'm assuming that there's
> nothing fundamentally wrong with the data).  Then there should be no
> problem re-refining against the 2nd dataset, and the Rfree will be
> unbiased at the global optimum.  The more common situation perhaps is
> that the 2nd dataset is at much higher resolution; in that case it's
> quite likely that there are undetected local optima in the model from
> the 1st dataset that only become apparent in the maps when the 2nd
> dataset is used.  In that case refinement is almost certainly not the
> answer (or at least not the whole answer), you're going to have to go
> back to the maps and model building.
> 
> On the question of overfitting, again any problems of local optima
> (possibly indicated by a higher than expected Rfree as you say) have to
> be resolved first for each of your candidate parameterizations of the
> model, as best as the data will allow.  Then if you find that Rfree at
> convergence is higher (or LLfree lower) for one parameterization than
> another, you choose the parameterization with the lower Rfree (higher
> LLfree) to go forward.  You cannot safely reject a model as being
> overfitted if the refinement generating the Rfree didn't converge, so
> that the Rfree is unbiased.  I don't see the problem there (except of
> course in choosing which parameterizations to try).
> 
> I think you misunderstood my provisos, I was only doing that to simplify
> the argument; if there are local optima then they have to be resolved,
> most likely by means other than refinement, but their presence does not
> affect the argument about Rfree bias.  My contention is that once all
> issues of local optima are resolved, by whatever means it takes, you
> will end up at the same unique global optimum no matter where you
> started from (unless of course you're very unlucky and there are
> multiple global optima with identical likelihoods but I think we can
> discount that as unlikely!), and therefore Rfree must be unbiased at
> that point.  At intermediate points in this process (i.e. on the paths
> connecting optima), Rfree has no meaning or indeed usefulness and
> therefore the question whether it's biased or not is also meaningless.
> 
> Cheers
> 
> -- Ian
> 
> 
> Disclaimer
> This communication is confidential and may contain privileged information intended solely for the named addressee(s). It may not be used or disclosed except for the purpose for which it has been sent. If you are not the intended recipient you must not review, use, disclose, copy, distribute or take any action in reliance upon it. If you have received this communication in error, please notify Astex Therapeutics Ltd by emailing [log in to unmask] and destroy all copies of the message and any attached documents. 
> Astex Therapeutics Ltd monitors, controls and protects all its messaging traffic in compliance with its corporate email policy. The Company accepts no liability or responsibility for any onward transmission or use of emails and attachments having left the Astex Therapeutics domain.  Unless expressly stated, opinions in this message are those of the individual sender and not of Astex Therapeutics Ltd. The recipient should check this email and any attachments for the presence of computer viruses. Astex Therapeutics Ltd accepts no liability for damage caused by any virus transmitted by this email. E-mail is susceptible to data corruption, interception, unauthorized amendment, and tampering, Astex Therapeutics Ltd only send and receive e-mails on the basis that the Company is not liable for any such alteration or any consequences thereof.
> Astex Therapeutics Ltd., Registered in England at 436 Cambridge Science Park, Cambridge CB4 0QA under number 3751674

Top of Message | Previous Page | Permalink

JiscMail Tools


RSS Feeds and Sharing


Advanced Options


Archives

March 2024
February 2024
January 2024
December 2023
November 2023
October 2023
September 2023
August 2023
July 2023
June 2023
May 2023
April 2023
March 2023
February 2023
January 2023
December 2022
November 2022
October 2022
September 2022
August 2022
July 2022
June 2022
May 2022
April 2022
March 2022
February 2022
January 2022
December 2021
November 2021
October 2021
September 2021
August 2021
July 2021
June 2021
May 2021
April 2021
March 2021
February 2021
January 2021
December 2020
November 2020
October 2020
September 2020
August 2020
July 2020
June 2020
May 2020
April 2020
March 2020
February 2020
January 2020
December 2019
November 2019
October 2019
September 2019
August 2019
July 2019
June 2019
May 2019
April 2019
March 2019
February 2019
January 2019
December 2018
November 2018
October 2018
September 2018
August 2018
July 2018
June 2018
May 2018
April 2018
March 2018
February 2018
January 2018
December 2017
November 2017
October 2017
September 2017
August 2017
July 2017
June 2017
May 2017
April 2017
March 2017
February 2017
January 2017
December 2016
November 2016
October 2016
September 2016
August 2016
July 2016
June 2016
May 2016
April 2016
March 2016
February 2016
January 2016
December 2015
November 2015
October 2015
September 2015
August 2015
July 2015
June 2015
May 2015
April 2015
March 2015
February 2015
January 2015
December 2014
November 2014
October 2014
September 2014
August 2014
July 2014
June 2014
May 2014
April 2014
March 2014
February 2014
January 2014
December 2013
November 2013
October 2013
September 2013
August 2013
July 2013
June 2013
May 2013
April 2013
March 2013
February 2013
January 2013
December 2012
November 2012
October 2012
September 2012
August 2012
July 2012
June 2012
May 2012
April 2012
March 2012
February 2012
January 2012
December 2011
November 2011
October 2011
September 2011
August 2011
July 2011
June 2011
May 2011
April 2011
March 2011
February 2011
January 2011
December 2010
November 2010
October 2010
September 2010
August 2010
July 2010
June 2010
May 2010
April 2010
March 2010
February 2010
January 2010
December 2009
November 2009
October 2009
September 2009
August 2009
July 2009
June 2009
May 2009
April 2009
March 2009
February 2009
January 2009
December 2008
November 2008
October 2008
September 2008
August 2008
July 2008
June 2008
May 2008
April 2008
March 2008
February 2008
January 2008
December 2007
November 2007
October 2007
September 2007
August 2007
July 2007
June 2007
May 2007
April 2007
March 2007
February 2007
January 2007


JiscMail is a Jisc service.

View our service policies at https://www.jiscmail.ac.uk/policyandsecurity/ and Jisc's privacy policy at https://www.jisc.ac.uk/website/privacy-notice

For help and support help@jisc.ac.uk

Secured by F-Secure Anti-Virus CataList Email List Search Powered by the LISTSERV Email List Manager