Print
This article that Mark spotted provides a useful and accessible discussion on the limitations of 'brute force' web crawlers, and the potential value of metadata. T.