In my experience when going from XDS via some intermediate file to mtz format, XDS uses in some cases a different reciprocal asymmetric unit as mtz uses, which may result in only half of the reflections being used and/or ccp4 programs getting confused. By using UNIQUE, one makes sure that the reflections are mapped to the correct asymmetric unit. It has nothing to do with missing reflections but is in many cases essential.

Best regards,
Herman


-----Original Message-----
From: CCP4 bulletin board [mailto:[log in to unmask]] On Behalf Of Kay Diederichs
Sent: Thursday, February 21, 2008 3:46 PM
To: [log in to unmask]
Subject: Re: [ccp4bb] XDS and overlaps

Simon Kolstoe schrieb:
> Whilst we are on the subject of XDS...
>
> I had difficulty processing a data-set in mosflm the other day so on
> the recommendation of a colleague switched to xds which, with a bit of
> tweaking, seemed to work really well. I converted the resulting
> XDS_ASCII.HKL using xdsconv and then f2mtz ready for phaser and refmac.

We do it in the same way here.

>
> However, my colleague then told me that xds handled missing
> reflections differently from the usual mosflm/CCP4 route

I have honestly not the slightest idea what your colleague was referring to.

> and thus I had to run the CCP4 program UNIQUE before I tried
> refinement as apparently refmac does not like reflection files
> originally processed with xds. As I couldn't

In the case of a new project, one should run "uniqueify" or some other means of assigning reflections to the free set (thin shells come to mind; see earlier discussions on CCP4BB).

In the case of an old project, one should transfer the free set of reflections from some master data set to the new dataset.

None of this is specific to XDS.

HTH,

Kay

> find anything in the literature about this I was wondering whether
> this advice is still up to date?
>
> Thanks,
>
> Simon
>
>
> On 21 Feb 2008, at 09:44, Kay Diederichs wrote:
>
>> Engin Ozkan schrieb:
>>> Hi everyone,
>>> I have been recently relying on XDS quite a bit, but at the same
>>> time worrying about how XDS treats overlaps. We had one dataset
>>> that both HKL2000 and Mosflm would show to have severe overlaps, as
>>> expected due to unit cell parameters and the unfortunate crystal
>>> orientation in the loop. We always ended up with completeness
>>> percentages in the 70's.
>>> XDS can find the same lattice, index and scale the data, but yields
>>> a 100% complete mtz (and a nice structure). Without the
>>> HKL/Mosflm-like GUI, it is difficult to assess the fate of the
>>> overlapped observations in XDS. What I could see with VIEW was that
>>> some observations were being divided into several ovals, probably
>>> different reflections, but I'm not very certain.
>>> So, the basic question is, how does XDS treat overlaps? I could not
>>> find in the documentation an answer to this question; the single
>>> mention of overlaps I could find tells me that XDS can recognize
>>> overlaps, but does not tell me if it rejects them, or divvies them
>>> up into separate reflections, and if that is the case, how does it
>>> divide them, and how reliable is that? Depending on how it divides
>>> the overlaps, could that affect commonly-used intensity stats and
>>> distributions?
>>> Thanks,
>>> Engin
>>
>> Engin,
>>
>> the basic answer is:
>> a) each pixel of the detector is assigned to its nearest reflection
>> in reciprocal space
>> b) some of these pixels will mostly allow the background estimation,
>> others will mostly contribute to the integration area (but as they
>> are transformed into a local coordinate system there is not a 1:1
>> relationship). At this step, pixels which should be background but
>> are higher than expected (due to overlap) are rejected.
>> c) for each reflection, the background is estimated, and the 3D
>> profile is assembled from the pixels contributing to it
>> d) a comparison is made: for a reflection, is the percentage of its
>> observed profile assembled in c) larger than some constant (called
>> "MINPK" in XDS.INP)? If the answer is no, this reflection will be
>> discarded (you could call this situation "overlap").
>>
>> Among other things, this means that:
>> a) the program does _not_ look around each reflection to detect an
>> overlap situation, it just tries to gather the pixels for each
>> reflection
>> b) as a user, when your crystal-detector distance was chosen too low
>> or the reflections are very broad (resulting in generally strong
>> overlap), you may reduce MINPK down to 50. This will result in more
>> completeness, but you should monitor the quality of the resulting
>> data. Conversely, if you raise MINPK over its default of 75 you will
>> discard more reflections, but the resulting dataset will be a bit
>> cleaner.
>>
>> The reference is
>> W. Kabsch (1988) Evaluation of single-crystal X-ray diffraction data
>> from a position-sensitive detector. J. Appl. Cryst. 21, 916-924.
>> (http://dx.doi.org/10.1107/S0021889888007903)
>>
>> HTH,
>>
>> Kay
>> --Kay Diederichs http://strucbio.biologie.uni-konstanz.de
>> email: [log in to unmask] Tel +49 7531 88 4049 Fax 3183
>> Fachbereich Biologie, Universität Konstanz, Box M647, D-78457
>> Konstanz
>


--
Kay Diederichs http://strucbio.biologie.uni-konstanz.de
email: [log in to unmask] Tel +49 7531 88 4049 Fax 3183
Fachbereich Biologie, Universität Konstanz, Box M647, D-78457 Konstanz