 |
 |
1. I would welcome List members' advice on the likely external validity and general soundness (cet.par.) of sample survey findings that are based on `unadjusted' samples whose sampling units (firms) are drawn from a range of industrial sectors with key characteristics (including trading environments) that differ substantially one from another. By `unadjusted' is meant that no steps have been taken, by means of weighting or otherwise, to bring the disparate sampling units within each sample onto a common, or near common basis. (My question is not intended to include purely exploratory sampling, from which rigorous conclusions are unlikely to be required). 2. Some context may clarify further. This query arises from an assessment I am making of the methodological adequacy of a group of survey sample-based studies, most of which exhibit one or more flaws, including very vague or no definitions of their target/survey populations. . Many of them have drawn their samples (simple random) from across two or more industrial sectors that differ very markedly from each other (eg glove making, aircraft manufacture), for instance, in terms of their capital intensitivity and the scale of R & D undertaken. No compensating adjustments were made to these multi-sector samples. And while in analysis the overall sample is sometimes split into different firm-size groups, all sampling units are treated (usually implicitly) as equivalents. There is no separate analysis by sector subgroups, nor is any such distinction made in the reporting of the studies' findings, which most of these authors tend to claim can be generalised. No mention is made either of study limitations in these respects 3. It is appreciated of course that, other things being equal, the ability to generalise may be enhanced by drawing the sample from more than one sector. But, intuitively at least, it seems clear that generalisations based on unadjusted `apples & pears' samples ought not to be regarded as externally valid.- and therefore as being unsuitable for eg the testing of hypotheses. Sampling units differ in unquantified ways. Prima facie, weights should be applied and/ or separate analysis of sectoral subsamples conducted, (with perhaps standardisation and addition to follow)- in order to end up with an overall sample whose constituent sampling units are largely uniform. 4. With this background - and my inability so far to locate technical literature on these specific issues - I shall be much obliged for the advice of List scholars on the following questions: (i) Can externally valid findings be extracted from the analysis of unadjusted random samples drawn from across a range of hetereogeneous sectors? (ii) If not, what, if any, procedures are best employed to maximise the uniformity of sampling units in such overall samples? (iii) Is there not, in sampling theory and best practice, a presumption in favour of samples composed of uniform sampling units? (iv) Which reference texts are best on these specific aspects? With advance thanks = Owen