Delete (set occupancies to zero) the side chain back to CA. Do a few rounds of refinement and calculate Fo-Fc and examine.
It is possible that the side chain is disordered, or ordered in multiple conformations. Compare the density for alternate confonformers against the density for CA.
Alternatively, your side chain B-factor restaints might be too tight.
Anthony