Sounds like there might be something wrong with that contrast for that subject. E.g. the weights are wrong. I'd inspect that contrast for that subject---bring it up in the glass brain ("results" button), display the con_####.img image, check the contrast weights, etc.
It's also possible that that's just the way it is. That's more likely the fewer subjects you have.