Many good points have been made on this thread so far, but mostly 
addressing the question "how many free reflections is enough", whereas 
the original question was "how many is too many".

I suppose a reasonable definition of "too many" is when the error 
introduced into the map by leaving out all those reflections start to 
become a problem.  It is easy to calculate this error: it is simply the 
difference between the map made using all  reflections (regardless of 
Free-R flag) and the map made with 5% of the reflections left out.  Of 
course, this "difference map" is identical to a map calculated using 
only the 5% "free" reflections, setting all others to zero.  The RMS 
variation of this "error map" is actually independent of the phases used 
(Parseval's theorem), and it ends up being:

RMSerror = RMSall * sqrt( free_frac )
where:
RMSerror is the RMS variation of the "error map"
RMSall is the RMS variation of the map calculated with all reflections
free_frac is the fraction of hkls left out of the calculation.

So, with 5% free reflections, the errors induced in the electron density 
will have an RMS variation that is 22.3% of the full map's RMS 
variation, or 0.223 "sigma units".  1% free reflections will result in 
RMS 10% error, or 0.1 "sigmas".  This means, for example, that with 5% 
free reflections a 1.0 "sigma" peak might come up as a 1.2 or 0.8 
"sigma" feature.  Note that these are not the "sigmas" of the Fo-Fc map, 
(which changes as you build) but rather the "sigma" of the Fo map.  Most 
of us don't look at Fo maps, but rather 2Fo-Fc or 2mFo-DFc maps, with or 
without the missing reflections "filled in".  These are a bit different 
from a straight "Fo" map.  The absolute electron number density (e-/A^3) 
of the 1 "sigma" contour for all these maps is about the same, but no 
doubt the "fill in", extra Fo-Fc term, and the likelihood weights 
reduces the overall RMS error.  By how much?  That is a good question.

Still, we can take this RMS 0.223 "sigma" variation from 5% free 
reflections as a worst-case scenario, and then ask the question: is this 
a "problem"?  Well, any source of error can be a problem, but when you 
are trying to find the best compromise between two 
difficult-to-reconcile considerations (such as the stability of Rfree 
and the interpretability of the map), it is usually helpful to bring in 
a third consideration: such as how much noise is in the map already due 
to other sources?  My colleagues and I measured this recently (doi: 
10.1073/pnas.1302823110), and found that the 1-sigma contour ranges from 
0.8 to 1.2 e-/A^3 (relative to vacuum), experimental measurement errors 
are RMS ~0.04 e-/A^3 and map errors from the model-data difference is 
about RMS 0.13 e-/A^3.  So, 22.3% of "sigma" is around RMS 0.22 e-/A^3.  
This is a bit larger than our biggest empirically-measured error: the 
"modelling error", indicating that 5% free flags may indeed be "too much".

However, 22.3% is the worst-case error, in the absence of all the 
corrections used to make 2mFo-DFc maps, so in reality the modelling 
error and the omitted-reflection errors are probably comparable, 
indicating that 5% is about the right amount.  Any more and the error 
from omitted reflections starts to dominate the total error.   On the 
other hand, the modelling error is (by definition) the Fo-Fc difference, 
so as Rwork/Rfree get smaller the RMS map variation due to modelling 
errors gets smaller as well, eventually exposing the omitted-reflection 
error.  So, once your Rwork/Rfree get to be less than ~22%, the errors 
in the map are starting to be dominated by the missing Fs of the 5% free 
set.

However, early in the refinement, when your R factors are in the 30%s, 
40%s, or even 50%s, I don't think the errors due to missing 5% of the 
reflections are going to be important.  Then again, late in refinement, 
it might be a good idea to start including some or all of the "free" 
reflections back into the working set in order to reduce the overall map 
error (cue lamentations from validation experts such as Jane Richardson).

This is perhaps the most important topic on this thread.  There are so 
many ways to "contaminate", "bias" or otherwise compromise the free set, 
and once done we don't have generally accepted procedures for 
re-sanctifying the free reflections, other that starting over again from 
scratch.  This is especially problematic if your starting structure for 
molecular replacement was refined against all reflections, and your 
ligand soak is nice and isomorphous to those original crystals.  How do 
you remove the evil "bias" from this model?  You can try shaking it, but 
that only really removes bias at high spatial frequencies and is not so 
effective at low resolution.
So, if bias is so easy to generate why not use it to our advantage?  
Instead of leaving the free-flagged reflections out of the refinement, 
put them in, but give them random F values.  Then do everything you can 
to "bias" your model toward these random values.  Loosen the geometry 
weights, turn on B-factors, build dummy atoms into all the "noise peaks" 
in the maps, and refine for a lot of cycles.  Now you've got a model 
that is highly biased, but to a completely unrelated "free" set.  It 
stands to reason that a model cannot be biased toward two completely 
unrelated things.  Now you can take this model and put it back into 
normal refinement against your original data (with the true free-set Fs 
restored and flagged as "free").  Your Rfree will start out very high, 
but then rapidly drop to what can only be an un-biased value.  You can 
then try different random number seeds for the 
over-refine-against-randomized-free-set run, and see how consistent the 
final Rfree becomes.  Theoretically, you could do this "bias 
misdirection" with as little as one reflection at a time.  This would be 
more true to classical cross-validation, where you are supposed to do 
~20 parallel optimizations with 20 different free sets, and look at the 
combined Rfree across them. Takes a lot longer, but these days what do 
we have if not lots of CPUs lying around?

Anyway, to answer the OP question: 5% is close to being "too many", so 
less than that is preferable as long as you have "enough" as others on 
this thread have so aptly described.  In cases where 
observations/parameters is becoming too precious, then perhaps more 
robust (aka time-consuming) cross-validation protocols are called for?

-James Holton
MAD Scientist

On 6/2/2015 3:26 AM, Graeme Winter wrote:
> Hi Folks
>
> Had a vague comment handed my way that "xia2 assigns too many free 
> reflections" - I have a feeling that by default it makes a free set of 
> 5% which was OK back in the day (like I/sig(I) = 2 was OK) but maybe 
> seems excessive now.
>
> This was particularly in the case of high resolution data where you 
> have a lot of reflections, so 5% could be several thousand which would 
> be more than you need to just check Rfree seems OK.
>
> Since I really don't know what is the right # reflections to assign to 
> a free set thought I would ask here - what do you think? Essentially I 
> need to assign a minimum %age or minimum # - the lower of the two 
> presumably?
>
> Any comments welcome!
>
> Thanks & best wishes Graeme