Another likely possibility that has not been mentioned yet is sample vibration.  In my experience this is just about the only way to get Rmerge in P1 up into the 20%s for the lowest-angle bin.  There is even a paper about this: https://doi.org/10.1107/S0021889808032536

Unless you have a big beamstop shadow and you didn't mask it out (in which case you are measuring zeroes), these low-angle spots should be among the brightest in the whole data set.  It is also a good idea (as Phil suggested) to look at the highest intensity bin to see Rmerge for your brightest spots.  The reason to look at the brightest-spot bins is because they are essentially immune to many common culprits like read-out noise, high background or "weak diffraction" arising from short exposures or disorder in the crystals.  What these spots are sensitive to is sources of relative error, like beam flicker, shutter jitter, absorption factors like detector calibration and sample self-absorption, and definitely sample vibration.  Normally, the resultant of all these effects put together is less than 2-5%, depending on how good your equipment is.  This is exactly what Rmerge is good for: telling you about your equipment.  The overall Rmerge is also useful if you are worried that your bright-spot bin has only a few reflections in it.  R1 is even better since it only looks at strong data.  And I won't beat the dead horse of why Rmerge is not a criterion for resolution cutoff because the OP didn't ask about that.

Sample vibration is a tough problem to diagnose because it is very specific to your particular crystal and loop shape, the angle of the cryo-stream, and to the phi rotation itself if your stream blows off-axis.  If you swap in a firmly-mounted lysozyme crystal everything will look fine, but if your next sample has a long-neck loop with a big "sail" of a crystal on the end your vibration problem will be back.  Sometimes adjusting the flow rate of the cold stream can help (a dangerous game), but the real solution is to shore-up your mounts.  Sometimes coating the twisty loop neck with epoxy (before mounting your crystal) can make a huge difference.  MiTeGen's curved sheet design is also very effective against sample vibration.  And no, they don't pay me to say that.

The other reason sample vibration is hard to diagnose is because the frame rate of the video microscope on a remote internet connection is way too slow to see it.  However, if you do see your sample "fluttering" then you should worry.  In the past we used to think this was a refraction effect of the cryo stream gas between the sample and the camera, but it turns out that was wishful thinking.  What is worse, I have also seen plenty of samples with vibration problems that were in no way apparent in the microscope, just in Rmerge.

A follow-up test to diagnose sample vibration is to take a series of "stills" or at least the same narrow rotation about 3-10 times in a row from the same crystal with the same starting phi value each time.  You then integrate these images as usual and look at the variation (aka Rmerge) for each hkl individually.  Plot this variation vs location on the detector.  If you see swaths of reciprocal space with high variation and others low, and the orientation of these swaths is in no way related to the phi axis, then you know you've got vibration issues.

It is also possible you are seeing other instrument problems, like beam flicker, shutter jitter, an unstable phi motor or a serious detector problem. These should not be present on properly-maintained equipment, but it is always possible that something failed or went awry just before you started collecting. The test for this is to look at other data sets from the same instrument.  If the low-angle Rmerge in P1 of your dataset stands out, then it was probably sample vibration.


The only other time I've seen Rmerge in P1 this high without sample vibration was when we intentionally offset the rotation axis from the beam position to do "doughnut" data collection from a very large crystal, similar to the strategy explored in https://doi.org/10.1002/pro.3302 .  In our case the crystal was twinned and highly non-isomorphous within itself, and because of the offset there was a completely different crystal volume in the beam at phi=0 vs phi=180. For this reason the Friedel opposite reflections were effectively coming from different crystals, and that made the P1 Rmerge value abnormally high.  Of course, if we had bathed this whole crystal in a large beam we would have gotten a very low Rmerge, and ironically the final merged data would have been the same: averaged over the whole exposed volume.  The difference is that by shooting different parts of the crystal at different times you can see problems that are masked by pre-averaging.  You can do "pre-averaging" with the beam by illuminating everything at once, or by using statistics derived from semi-merged data like Rsplit, CC1/2, Rpim or I/sigma (after the sigmas have been "corrected").  In either case you are masking an underlying problem.

That said, once the data are collected and you know you had an "underlying problem", what do you do?  Well, the nice thing about random relative errors like sample vibration is that they average out if you have high enough multiplicity.  So Rpim, CC1/2 and I/sigma are very good ways to compare the quality of your data set to other data sets.  But it is also possible your "underlying issue" is not random but systematic, such as non-isomorphous parts of the same crystal, twinning, wrong space group, etc.  In these cases averaging over your problem is a bad idea.

Finally, if Rmerge is high like you are seeing here you should definitely inform whomever is maintaining the instrument you used that there is a problem.  Could have just been your loop, but it never hurts to check.  The next user may thank you.

-James Holton
MAD Scientist


On 9/29/2018 12:58 AM, Zhang Foggy wrote:
[log in to unmask]">
Dear All, 

Thanks for your kindly comments. Here are the summary and my responses:

1.Summary: 

The high value of the R-merge might be due to the weak diffraction, as well as the collection method (low dose per image and high redundancy).
The suggetions is ignore the R-merge value, and condider the Rpim, CC1/2 and I/sigI value instead. 

2. Responses to the comments:

(1) Dr. Herman Schreuder suggested that I can use ADXV to look through my data and see if there is some bad regions, overload (scale only the first 30-40 frames) or ice rings, and I also can use other software to scale the data (xds, mosfilm etc.) 
Response: Actually I have tried these alternative ways. Indeed, the diffraction of my crystals is pretty good (you can see the image from attached jpg file), there is no ice rings, no significant radiation damage, no bad regions through the entire frames. I have also tried to use xds to scale it, unfortunately, the R-merge is still high (~50%).  Additionally, I also tried to only scale part of the frames, however, the R--merge is to ~48%. 

(2) Dr. Shepard  William suggested to try mosfilm or xds, and asked the multiplicity of the data.
Response:   I have tried to scale with XDS, but there is no improvement. The space group is P43. I can refine the structure to R-value 0.19, R-free 0.23, indicating that  the space group should be correct. I have also tried to scale to P21 or P1, and there is no improve in R-merge. 

(3) Dr. Phil Evans mentioned that Rmerge is a terrible criterion (Science, 2012, 336,1030), and CC(1/2) should be generally considered as the best criterion. In my case,  both of the Rmerge (1.59) and CC(1/2) (0.645) in the outer shell are acceptable. However, the Rmerge (0.284) and CC(1/2) (0.975) in the inner shell looks not perfect. I should consider the radiation damage.
Response: Thanks a lot for the comments. As you can see from the attached figure, the diffraction is sharp, and I do not see any significant radiation damage

(4) Dr. Ditlev Egeskov Brodersen suggested to double check the space group and process part of the data.
Response:  as I mentioned in (1) and (2), I have tried to only scale part of the frames, however, the R--merge is to ~48%;  I can refine the structure to R-value 0.19, R-free 0.23 under the current space group P43. Moreover, scale to P21 or P1,can not improve the R-merge significantly.

(5) Dr.  Remy Loris mentioned that a high value of R-merge indicates a wrong symmetry or very weak data. from my data, the reason could be the weak data as well as high redundancy.  
Response: I agree. from the attached image, I can see the the diffraction is sharp but weak. However, increse the  exposure  time will introduce more radiation damage....

(6) Dr. Edward A. Berry mentioned that my data has rather high redundancy as Rpim is much lower than Rmeas value. It could be caused by collecting low dose per image and making up for it with high redundancy, Dr. Edward A. Berry suggeted to Look instead at CC1/2 and I/sigI which seem fine.
Response:  Thanks for the comments, and I agree. 

(7). Dr. Rajesh Kumar raj suggested me to consider Rpim, CC1/2 and I/sigI for cutting the data as Rmerge is old approach and it is data redundancy dependent.

Thank you for your kindly help again!

Best,

Liang
Diffraction image.jpg



Rajesh Kumar <[log in to unmask]> 于2018年9月28日周五 下午11:41写道：
I totally agree with Berry. Please consider Rpim, CC1/2 and I/sigI for cutting the data. Rmerge is old approach as it is data redundancy dependent.

Thank you
Rajesh

---xxxxx----
With regards
Rajesh K. Harijan, Ph.D.
Schramm Laboratory
Albert Einstein College of Medicine
1300 Morris Park Ave., Bronx, NY 10461
Tel: 718.430.2777  |  Fax: 718.430.8565
 

On Fri, Sep 28, 2018 at 11:32 AM Edward A. Berry <[log in to unmask]> wrote:
The fact that chi^2 is approximately 1.0 in all shells says that the deviations are about what is expected from the error model. The fact that Rpim is much lower than Rmeas means that you have rather high redundancy. This would seem to be a case of collecting low dose per image and making up for it with high redundancy, a strategy that has been recommended to ensure a full dataset even in the case of high radiation sensitivity.  In my opinion the high Rmerge is nothing to worry about. Look instead at CC1/2 and I/sigI which seem fine.

On 09/28/2018 04:09 AM, Zhang Foggy wrote:
> Dear All,
>
> Sorry for the off-topic.
>
> I recently collected a set of data. The diffraction spots are extremely sharp. However, When I used HKL3000 to scale it, I get a final resolution at 3.1A with overall R-merge ~0.54 (R-merge in the highest 3.2A-3.1A shell: 1.59). Then I solve the structure with final R value 0.19 and R free value 0.24 although I know this Rmerge value is totally unacceptable, and the density looks perfect.
>
> I also tried to collect other four set of data with different crystals. unfortunately, all of them have same problem.
>
> I ask one of my friend who is an expert in HKL3000, but he had no idea about it. Does anyone has suggestions?
>
> Here is the scale information for your review:
> Space group: P43 (I also tried P1, the Rmerge value is still similar)
>
> Shell Lower Upper Average      Average     Norm. Linear Square
>   limit    Angstrom       I   error   stat. Chi**2  R-fac  R-fac  Rmeas   Rpim  CC1/2    CC*
>        50.00   6.67    11.6     0.9     0.3  1.165  0.191  0.284  0.198  0.052  0.975  0.994
>         6.67   5.30     4.5     0.5     0.3  0.952  0.317  0.313  0.329  0.086  0.971  0.993
>         5.30   4.63     7.3     0.7     0.5  0.961  0.293  0.297  0.304  0.081  0.975  0.994
>         4.63   4.21     7.0     0.8     0.6  0.986  0.369  0.358  0.382  0.101  0.969  0.992
>         4.21   3.91     5.6     0.8     0.6  1.040  0.522  0.491  0.541  0.142  0.955  0.988
>         3.91   3.68     4.6     0.9     0.7  1.064  0.718  0.669  0.746  0.203  0.929  0.981
>         3.68   3.49     3.5     0.9     0.8  1.092  1.059  0.986  1.101  0.299  0.882  0.968
>         3.49   3.34     2.6     0.9     0.8  1.092  1.382  1.298  1.438  0.395  0.829  0.952
>         3.34   3.21     2.1     0.9     0.8  1.084  1.543  1.489  1.614  0.468  0.772  0.933
>         3.21   3.10     1.6     0.9     0.8  1.070  1.591  1.669  1.680  0.529  0.645  0.885
>    All reflections      5.0     0.8     0.6  1.048  0.538  0.487  0.559  0.153
>
> Thank you.
>
> Liang
>
>
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> To unsubscribe from the CCP4BB list, click the following link:
> https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB&A=1
>

########################################################################

To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB&A=1

To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB&A=1


To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB&A=1



To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB&A=1