JiscMail Logo
Email discussion lists for the UK Education and Research communities

Help for CCP4BB Archives


CCP4BB Archives

CCP4BB Archives


CCP4BB@JISCMAIL.AC.UK


View:

Message:

[

First

|

Previous

|

Next

|

Last

]

By Topic:

[

First

|

Previous

|

Next

|

Last

]

By Author:

[

First

|

Previous

|

Next

|

Last

]

Font:

Proportional Font

LISTSERV Archives

LISTSERV Archives

CCP4BB Home

CCP4BB Home

CCP4BB  September 2009

CCP4BB September 2009

Options

Subscribe or Unsubscribe

Subscribe or Unsubscribe

Log In

Log In

Get Password

Get Password

Subject:

Re: Rfree in similar data set

From:

Ian Tickle <[log in to unmask]>

Reply-To:

Ian Tickle <[log in to unmask]>

Date:

Fri, 25 Sep 2009 11:53:33 +0100

Content-Type:

text/plain

Parts/Attachments:

Parts/Attachments

text/plain (359 lines)

Hi Lijun

One important point your summary didn't cover - the test set may not
actually be the same even though you think it is!  What I mean is that
the test sets for different crystals may have the same indices but may
not sample exactly the same points in reciprocal space either due to
cell parameter changes or to rigid-body movements, or in fact anything
which causes the crystals to be non-isomorphous.  Mike's original
question I think referred to switching datasets when the crystals are
chemically identical, but I suspect this problem arises more often when
a ligand is soaked in and you want to re-refine the complex using the
refined apo model as a starting point with the new data - the question
is does it matter whether or not you use the 'same' test set (i.e. same
indices)?

We frequently observe quite large cell parameter changes (up to 10% in
extreme cases) on soaking and freezing (which are hardly reproducible
treatments of the crystals!).  Let's say a cell parameter has changed by
only 2%, which is fairly typical for many ligand soaks.  The question is
at what resolution does this cause a test set reflection in the
protein-ligand data to become 'contaminated' by a reflection in the
working set of the apo data which differs by 1 in the index in the
direction of that cell parameter?  I think 'contamination' starts to
occur when the positions of the reflections in reciprocal space differ
by less than half a reciprocal cell length, i.e. the points in one
reciprocal lattice appear at points closer than the half-index positions
of the other lattice.  If the change is 2% in the 'a' parameter that
means the shift in the lattice is 1 in 50 rlu's, or at h=25 for 1/2 rlu.
Let's say the cell parameter a=100 Ang, so what resolution is h=25?  The
answer is 100/25 = 4 Ang, so that means all test set reflections with
resolution higher than 4 Ang are 'contaminated' by the working set of
the other crystal to a greater or lesser degree.  If the data go to 2
Ang, that's ~ 90% of the data - and that's from only a 2% change.

So it's just as well that refinement to convergence removes the
resulting Rfree bias, since I suspect very few people resort to
'shaking' (or whatever extreme measures are recommended) their
protein-ligand models before refinement!

Cheers

-- Ian

> -----Original Message-----
> From: Lijun Liu [mailto:[log in to unmask]]
> Sent: 24 September 2009 19:00
> To: Ian Tickle
> Cc: [log in to unmask]
> Subject: Re: [ccp4bb] Rfree in similar data set
> 
> Sticking to the same test set is a great and practical idea!  It
lowers
> the chance to get biased from "self-validation".  However, logically,
> 
> 0) based on Bragg's law and HKL<->XYZ theory, no reflections are truly
> free from the others from the same crystal.  But when the reflection
> number is larger than the number of parameters minus other restriction
> conditions, you have more degree of freedom, statistically.
Uncertainties
> contributed from many aspects increase the freedom.   Even though,
free
> test set is not purely free.
> 
> 1) if the refined model is optimal/quasi-optimal, the model is then
> supposed to be consistent to the data used for test set too;  or the
model
> is far from optimal.  In this regard, switching test sets/using
different
> test sets will not be a problem for this kind of "ideal" case---enough
> refinement cycles should be able to bring consistent models.
> 
> 2) keeping the same reflections for test set will leave these
reflections
> lost any chance to contribute the minimization process in a direct
> fashion, which itself causes a kind of bias.  From the point of data,
if
> any data are excluded, no matter randomly or not, from calculation,
> artificial bias would be resulted!   If the initial model is biased,
it
> will be biased forever if it caused due to the exclusion of test set
(this
> sounds more true when with low resolution data).
> 
> So, a judgement may need be based on your data!  At the "end" (I mean
you
> are going to stop) of a smooth refinement, switching test sets should
not
> be a "huge" problem, or the model is too wrong!
> 
> Back to Mike's question:  I suggest you keep the same test set, since
your
> data were from the exactly same crystal.  At least it saves your
> convergence time.
> 
> Lijun Liu
> 
> On Sep 24, 2009, at 10:24 AM, Ian Tickle wrote:
> 
> 
> 		-----Original Message-----
> 
> 
> 		From: Dale Tronrud [mailto:[log in to unmask]]
> 
> 
> 		Sent: 24 September 2009 17:21
> 
> 
> 		To: Ian Tickle
> 
> 
> 		Cc: [log in to unmask]
> 
> 
> 		Subject: Re: [ccp4bb] Rfree in similar data set
> 
> 
> 
> 		  While I agree with Ian on the theoretical level, in
practice
> 
> 
> 		people use free R's to make decisions before the
ultimate
> model
> 
> 
> 		is finished, and our refinement programs are still
limited in
> 
> 
> 		their abilities to find even a local minimum.
> 
> 
> 
> 	I wasn't saying that Rfree is only useful for the ultimate
finished
> 	model.  My argument also applies to all intermediate models; the
> 	criterion is that the refinement has converged against the
current
> 	working set, even if it is only an incomplete model, or if it is
> only to
> 	a local optimum.  So it's perfectly possible to use Rfree for
> 	overfitting & other tests on intermediate models.  The point is
that
> it
> 	doesn't matter how you arrived at that optimum (whether local or
> 	global), Rfree is a function only of the parameters at that
point,
> not
> 	of any previous history.  If you arrived at that same local or
> global
> 	optimum via a path which didn't involve switching datasets
midway,
> you
> 	must get the same answer for Rfree, so I just don't see how it
can
> be
> 	biased one way and not biased the other.  Note that this is
meant as
> a
> 	'thought experiment', I'm not saying necessarily that it's
possible
> to
> 	perform this experiment in practice!
> 
> 
> 
> 		  On the automated level the test set is used,
sometimes, to
> 
> 
> 		determine bulk solvent parameter, and more importantly
to
> calibrate
> 
> 
> 		the likelihood calculations in refinement.  If the test
set is
> 
> 
> 		not "free" the likelihood calculation will overestimate
the
> 
> 
> 	reliability
> 
> 
> 		of the model and I'm not confident that error will not
become
> 
> 
> 		a self-fulfilling prophecy.  It is not useful to divine
> meaning
> 
> 
> 		from the free R until convergence is achieved, but the
test
> 
> 
> 		set is used from the first cycle.
> 
> 
> 
> 	That is indeed a fair point, but I would maintain that the test
set
> 	becomes 'free', i.e. free of the memory of all previous models,
the
> 	first time you reach convergence, so the question of the effect
on
> 	sigmaA calculations, which use the test set, is only relevant to
the
> 	first refinement after switching test sets, thereafter it should
be
> 	irrelevant.  Converging to a local or global optimum wipes out
all
> 	memory of previous models because the parameter values at that
> optimum
> 	are independent of any previous history, and so Rfree must be
the
> same
> 	for that optimum no matter what path you took to get there.
> 
> 
> 
> 		  Perhaps I'm in one of my more persnickety moods, but
every
> 
> 
> 		paper I've read about optimization algorithms say that
the
> method
> 
> 
> 		requires a number of iteration many times the number of
> parameters
> 
> 
> 		in the model.  The methods used in refinement programs
are
> pretty
> 
> 
> 		amazing in their ability to drop the residuals with a
small
> number
> 
> 
> 		of cycles, but we are violating the mathematical
warranty on
> 
> 
> 		each and every one of them.   A refinement program will
> produce
> 
> 
> 		a model that is close to optimal, but cannot be expected
to be
> 
> 
> 		optimal.  Since we haven't seen an optimal model yet
it's hard
> 
> 
> 		to say how far we are off.
> 
> 
> 
> 	I thought that for a quadratic approximation CG requires a
number of
> 	iterations that is not more than the number of parameters (not
that
> we
> 	ever use even that many iterations!)?  Anyway that's a problem
in
> 	theory, but it's possible to refine until nothing more
'interesting'
> 	happens, i.e. further changes appear to be purely random and at
the
> 	level of rounding errors.  Plotting the maximum shift of the
atom
> 	positions or B factors from one iteration to the next is a very
> 	sensitive way of detecting whether convergence has been
achieved;
> 	looking at changes in R factors or in RMSDs of bonds etc is a
bad
> way,
> 	since R factors are not sensitive to small changes and atoms can
> move in
> 	concert without affecting bond lengths etc. (or it may just be
the
> 	waters that are moving!).
> 
> 	As a final point I would note that cell parameters frequently
vary
> by
> 	several % between crystals even from the same batch due to
> unavoidable
> 	variations in rates of freezing etc, so what you think are
> independent
> 	test set reflections may in reality overlap significantly in
> reciprocal
> 	space with working set reflections from another dataset anyway!
> 
> 	-- Ian
> 
> 
> 	Disclaimer
> 	This communication is confidential and may contain privileged
> information intended solely for the named addressee(s). It may not be
used
> or disclosed except for the purpose for which it has been sent. If you
are
> not the intended recipient you must not review, use, disclose, copy,
> distribute or take any action in reliance upon it. If you have
received
> this communication in error, please notify Astex Therapeutics Ltd by
> emailing [log in to unmask] and destroy all copies of the
> message and any attached documents.
> 	Astex Therapeutics Ltd monitors, controls and protects all its
> messaging traffic in compliance with its corporate email policy. The
> Company accepts no liability or responsibility for any onward
transmission
> or use of emails and attachments having left the Astex Therapeutics
> domain.  Unless expressly stated, opinions in this message are those
of
> the individual sender and not of Astex Therapeutics Ltd. The recipient
> should check this email and any attachments for the presence of
computer
> viruses. Astex Therapeutics Ltd accepts no liability for damage caused
by
> any virus transmitted by this email. E-mail is susceptible to data
> corruption, interception, unauthorized amendment, and tampering, Astex
> Therapeutics Ltd only send and receive e-mails on the basis that the
> Company is not liable for any such alteration or any consequences
thereof.
> 	Astex Therapeutics Ltd., Registered in England at 436 Cambridge
> Science Park, Cambridge CB4 0QA under number 3751674
> 
> 
> 
> Cardiovascular Research Institute
> University of California, San Francisco
> 1700 4th Street, Box 2532
> San Francisco, CA 94158
> Phone & Fax: (415)514-2836
> 
> 
> 
> 
> 



Disclaimer
This communication is confidential and may contain privileged information intended solely for the named addressee(s). It may not be used or disclosed except for the purpose for which it has been sent. If you are not the intended recipient you must not review, use, disclose, copy, distribute or take any action in reliance upon it. If you have received this communication in error, please notify Astex Therapeutics Ltd by emailing [log in to unmask] and destroy all copies of the message and any attached documents. 
Astex Therapeutics Ltd monitors, controls and protects all its messaging traffic in compliance with its corporate email policy. The Company accepts no liability or responsibility for any onward transmission or use of emails and attachments having left the Astex Therapeutics domain.  Unless expressly stated, opinions in this message are those of the individual sender and not of Astex Therapeutics Ltd. The recipient should check this email and any attachments for the presence of computer viruses. Astex Therapeutics Ltd accepts no liability for damage caused by any virus transmitted by this email. E-mail is susceptible to data corruption, interception, unauthorized amendment, and tampering, Astex Therapeutics Ltd only send and receive e-mails on the basis that the Company is not liable for any such alteration or any consequences thereof.
Astex Therapeutics Ltd., Registered in England at 436 Cambridge Science Park, Cambridge CB4 0QA under number 3751674

Top of Message | Previous Page | Permalink

JiscMail Tools


RSS Feeds and Sharing


Advanced Options


Archives

March 2024
February 2024
January 2024
December 2023
November 2023
October 2023
September 2023
August 2023
July 2023
June 2023
May 2023
April 2023
March 2023
February 2023
January 2023
December 2022
November 2022
October 2022
September 2022
August 2022
July 2022
June 2022
May 2022
April 2022
March 2022
February 2022
January 2022
December 2021
November 2021
October 2021
September 2021
August 2021
July 2021
June 2021
May 2021
April 2021
March 2021
February 2021
January 2021
December 2020
November 2020
October 2020
September 2020
August 2020
July 2020
June 2020
May 2020
April 2020
March 2020
February 2020
January 2020
December 2019
November 2019
October 2019
September 2019
August 2019
July 2019
June 2019
May 2019
April 2019
March 2019
February 2019
January 2019
December 2018
November 2018
October 2018
September 2018
August 2018
July 2018
June 2018
May 2018
April 2018
March 2018
February 2018
January 2018
December 2017
November 2017
October 2017
September 2017
August 2017
July 2017
June 2017
May 2017
April 2017
March 2017
February 2017
January 2017
December 2016
November 2016
October 2016
September 2016
August 2016
July 2016
June 2016
May 2016
April 2016
March 2016
February 2016
January 2016
December 2015
November 2015
October 2015
September 2015
August 2015
July 2015
June 2015
May 2015
April 2015
March 2015
February 2015
January 2015
December 2014
November 2014
October 2014
September 2014
August 2014
July 2014
June 2014
May 2014
April 2014
March 2014
February 2014
January 2014
December 2013
November 2013
October 2013
September 2013
August 2013
July 2013
June 2013
May 2013
April 2013
March 2013
February 2013
January 2013
December 2012
November 2012
October 2012
September 2012
August 2012
July 2012
June 2012
May 2012
April 2012
March 2012
February 2012
January 2012
December 2011
November 2011
October 2011
September 2011
August 2011
July 2011
June 2011
May 2011
April 2011
March 2011
February 2011
January 2011
December 2010
November 2010
October 2010
September 2010
August 2010
July 2010
June 2010
May 2010
April 2010
March 2010
February 2010
January 2010
December 2009
November 2009
October 2009
September 2009
August 2009
July 2009
June 2009
May 2009
April 2009
March 2009
February 2009
January 2009
December 2008
November 2008
October 2008
September 2008
August 2008
July 2008
June 2008
May 2008
April 2008
March 2008
February 2008
January 2008
December 2007
November 2007
October 2007
September 2007
August 2007
July 2007
June 2007
May 2007
April 2007
March 2007
February 2007
January 2007


JiscMail is a Jisc service.

View our service policies at https://www.jiscmail.ac.uk/policyandsecurity/ and Jisc's privacy policy at https://www.jisc.ac.uk/website/privacy-notice

For help and support help@jisc.ac.uk

Secured by F-Secure Anti-Virus CataList Email List Search Powered by the LISTSERV Email List Manager