 |
 |
Dear Ed, my head starts smoking ;-) While it's clear that the Rfree goes down when a structure becomes better during refinement, I think, its not the correlation of the _changes_ of |Fobs-Fcalc|, but correlation of the _final_ |Fobs- Fcalc| that is important here. I don't know the answer right now, but maybe the following short summaries help to clarify the topic a bit further (also for myself). If we define bias as an unwanted correlation between Rwork and Rfree thereby artificially lowering the Rfree too close to Rwork, we're left with analyzing any correlation of the final |Fobs-Fcalc| between working set and test set. Let's have a closer to look to the two different main cases (letting aside correlation between different data sets). I first look at correlation between Fobs, then between Fcalcs and finally between | Fobs-Fcalc|. (1) In the general case of NCS, excluding pseudo-higher symmetry cases, we will find only by chance Fobs in the test set that correlate with Fobs' in the working set if they come very close to each other after application of the NCS-operation. The overall correlation in Fobs will be rather low (except in cases of very high NCS, maybe). (1a) If we apply NCS-constraints/restraints during refinement, we exploit any NCS-correlation in the observed data and thereby introduce both correlation between NCS-related Fcalcs and correlation to the corresponding NCS-related working set Fobs. As a result, we introduce some correlation between the final |Fobs-Fcalc| in the working set and the test set, i.e. we introduce bias. The amount of bias will depend on the choice of the test set, the degree of NCS symmetry and how similar the NCS-copies are forced to become during refinement. (1b) If we do not apply any NCS-constraints/restraints during refinement, the amount of correlation between NCS-related Fcalcs depends on how similar the NCS-copies independently refine and how close the corresponding NCS-related Fobs come in the working set and the test set. Without application of NCS-constraints/restraints, I would expect that the correlation between final |Fobs-Fcalc| in the working set and the test set will be low. (2) In the special case of pseudo-higher symmetry NCS or intentionally treating higher symmetry data in a lower symmetry, we will find Fobs in the test set that are highly correlated to higher symmetry-related Fobs' in the working set. (2a) If we apply the NCS-symmetry during refinement or even refine in the higher symmetry, we introduce both correlation between NCS-/ higher symmetry-related Fcalcs and correlation to the corresponding NCS-/higher symmetry-related working set Fobs. Thus, the final |Fobs- Fcalc| in the working set and the test set will be highly correlated and there will be strong bias between Rfree and Rwork. (2b) If we do not apply NCS-symmetry or higher symmetry during refinement, the amount of correlation between NCS-/higher symmetry- related Fcalcs depends on the degree of similarity of the independently refined copies. If they are still similar, there will be both correlation between the NCS-/higher symmetry-related Fcalcs and correlation to the working set Fobs. Consequently, there will be still correlation of the final |Fobs-Fcalc| between working set and test set, i.e. bias of Rfree. The amount of bias depends only on how similar the independently refined copies become. In case of true higher symmetry, I expect them to be very similar and thus, the resulting bias will be rather high. How far can we agree on this? I would also like to hear the opinion of other crystallographers that are mathematically more trained than me, or did we loose attention on this topic, already? Best regards, Dirk. Am 11.02.2008 um 22:50 schrieb Edward Berry: > Dirk Kostrewa wrote: >> Dear Ed, >> although, I don't think that a comparison of refinement in a >> higher and a lower symmetry space group is valid for general NCS >> cases, I will try to answer your question. Here are my thoughts >> for two different cases: >> (1) You have data to atomic resolution with high I/sigma and low >> Rsym (I assume high redundancy). The n copies of the asymmetric >> unit in the unit cell are really identical and obey the higher >> symmetry (so, not a protein crystal). When you process the data in >> lower symmetry (say, P1), the non-averaged "higher-symmetry"- >> equivalent Fobs will differ due to measurement errors, and thus >> reflections in the working-set will differ to "higher-symmetry"- >> related reflections in the test-set due to these measurement >> errors. If you then refine the n copies against the working-set in >> the lower P1 symmetry, you minimize |Fobs(work)-Fcalc|, resulting >> in Fcalcs that become closer to the working-set Fobs. As a >> consequence, the Fcalcs will thus diverge somewhat from the test- >> set Fobs. However, since this atomic model is assumed to be very >> well defined obeying the higher symmetry, and, furthermore, the >> working-set contains well measured "higher-symmetry"-equivalent >> Fobs, the resulting atomic positions, and thus the Fcalcs, will be >> very close to their equivalent values in the higher-symmetry >> refinement. Therefore, the Fcalcs will also be still very similar >> to the "higher-symmetry"-equivalent Fobs in the test-set, and I >> would expect a difference between Rwork and Rfree ranging from "0" >> to the value of Rsym. In other words, the Fobs in the test-set are >> not really independent of the reflections in the working-set, and >> thus Rfree is heavily biased towards Rwork. >> In this case, I would not expect large differences in the outcome >> due to the additional application of "NCS"-constraints/restraints. > > As I see it, this is clearly a case of |Fo-Fc| for the test reflectins > decreasing because the model is getting better, and there is no bias. > Lets say the higher symmetry really does apply, so the correct > structure > is perfectly symmetrical and the "NCS-related" reflections agree to > within > the error level. > Lets also say the initial model is perfectly symmetrical (you > solved the > molecular replacement with two copies of the same monomer, and rigid- > body refinement positioned them exactly). But let's say it is > completely > unrefined- the search model is from a different organism in a > different > space group, and modified by homology modeling to your sequence. > So the Fo obey the NCS within error, The Fc obey the NCS, but the > Fobs don't fit the Fcalc very well. Initially there is no Free-R bias, > because the model has not been refined agaist the data. The free set > can only be biased by refinement, since it is only during refinement > that the the free set is treated differently. Thus it doesn't matter > that the ncs-related Fo are correlated and the ncs-related Fc > are correlated: it is only the CHANGES in Fc that could introduce > model bias, and they are uncorrelated if you do not enforce ncs. > > Now as we refine, the model will converge toward the correct > symmetrical > model as a result of minimizing the |Fo-Fc| for the work reflections. > At the same time the |Fo-Fc| for the test reflections will also > decrease > on the average, but to a lesser extent. I argue that the only > mechanism > for refinement to reduce |Fo-Fc| at a test reflection is by improving > the structure, and I think that constitutes an unbiased Free-R value. > > If you can think of any mechanism to reduce |Fo-Fc| for a test > reflection > because you are refining against a symm-related work reflection, then > the R-free would be biased. This is not the case if you do not > enforce > symmetry. On the average no decrease in |Fo-Fc|(test) will result from > changes that reduce |Fo-Fc| for the work reflection: given an > arbitrary > change in the structure, the change in |Fc| at arbitrary reflections > is a pseudo-random variable with expected value zero, and there is no > correlation between the change at ncs-related reflections. > > The value of |Fo-Fc| at a test reflection goes down, not due to > changes which improve the fit at a sym-related working reflection, > but because of changes that improve the fit at all test reflections, > and then only because the structure is improving. The atoms moved into > symmetrical positions not because they were constrained to do so, > but because that fits the data better, in turn because the true > structure > is symmetrical. If the symmetry doesn't hold for some atoms, they will > tend to move into asymmetric positions to minimize |Fo-Fc| at work > reflections, now *decreasing* the correlation with sym-related work > reflections. But again this will tend to reduce |Fo-Fc| at free > reflections, simply because the model is better approximating the > true structure. > > To make a more obvious parallel, suppose you are refining a low- > resolution > dataset from a microcrystal (with no NCS). In another directory on the > same disk you have a high resolution structure refined against a > larger > but isomorphous crystal from the same well, same cryo treatment, > using a different or no free set. The Fo's will be highly correlated > between the two dataets, because they are isomorphous crystals > of the same protein. > > Now if you constrain your low resolution model to be close to > the high resolution one, your free set will be biased because > those reflections were used in refining the other structure, > and you are constraining the new structure to be the same. > > If you DON'T impose any restraints between the two models, the > new model will STILL tend toward the high-resolution structure, > because it is a good approximation of the true structure. > Hence the Fc's will become highly correlated to the Fc's of > that structure. And |Fo-Fc| of the test reflections will decrease, > not because the structural changes you are making improved the fit > of the high-resolution structure to the reflection in that dataset > which is a test reflection in the new dataset, but only because > the model is improving. > Using your logic, because the model (and hence Fc's) are approaching > those of the structure which was refined against the test reflections, > so the test reflections must be biased. > > Thanks for taking the time to help me work this out, > Ed > > >> (2) You have data to non-atomic lower resolution, weak I/sigma and >> poor Rsym. It is impossible to say whether the n copies of the >> asymmetric unit in the unit cell are really identical, but they >> are treated so assuming the higher symmetry (so, a real protein >> crystal). For data processing, the same holds true as for case >> (1). In contrast, here I think that it makes a difference, whether >> you apply "NCS"-constraints/restraints between the n copies in the >> lower symmetry P1, or not. If you apply "NCS"-constraints or >> strong "NCS"-restraints, the n copies are made equal and you get n >> times the average structure. This is similar to the refinement in >> the higher symmetry, except that again you minimize the >> discrepancy between Fcalcs and working-set Fobs, which will >> increase the discrepancy to the "higher-symmetry"-related Fobs in >> the test-set. But since the Fobs in the test-set are still not >> really independent to the Fobs in the working-set, I would again >> expect maximum differences between Rwork and Rfree in the same >> order of magnitude as Rsym. So, Rfree is still biased towards >> Rwork, but it might be more difficult to notice this. But if you >> do not apply "NCS"-constraints/restraints, you give the less well- >> defined atomic model more freedom to converge against the working- >> set Fobs, resulting in a higher discrepancy between Rwork and >> Rfree. But since the Fobs in the working set still contain "higher- >> symmetry"-equivalent Fobs, you will end up with a model that still >> shows some similarity to the refined structure in the higher >> symmetry. As a result, the Rfree is even then not really >> independent of Rwork, but it might be even more difficult to >> notice this, depending on data resolution and quality. Here, I >> can't give a range of differences between Rwork and Rfree. >> So, this is still not quantitative, and I hope that I'm not >> completely wrong with my argumentation. >> These lower vs. higher symmetry examples given above are only >> transferable to reality in special NCS-cases with pseudo-higher >> symmetry (what Dale Tronrud discussed). Taking these special cases >> aside, what do the NCS experts say to my original statement that >> precautions against NCS bias in Rfree must only be taken if NCS- >> constraints/restraints are really applied during refinement? >> Best regards, >> Dirk. ******************************************************* Dirk Kostrewa Gene Center, A 5.07 Ludwig-Maximilians-University Feodor-Lynen-Str. 25 81377 Munich Germany Phone: +49-89-2180-76845 Fax: +49-89-2180-76999 E-mail: [log in to unmask] *******************************************************