Thanks to the several people who replied to my question (repeated here
for completeness). It is a most interesting topic and one that I will
continue to ponder.
Those intriqued by the Deming name may wish to consult
http://www.deming.org.uk/
-----
Date: Tue, 29 Jun 1999 15:54:56 +0100 (BST)
From: "R. Allan Reese" <[log in to unmask]>
Subject: Query - Deming and sampling
I have quoted for some years now what I believed to be a teaching of
Deming, that in some circumstances non-random sampling was preferable.
In particular, when a small "representative" sample from a
non-homogeneous population is needed, a judgment sample gives more usable
results. The "cost" is that you cannot make probability estimates, but
with small samples these are likely to be of little practical use.
I have quoted this as based on "Out of the crisis" 1987, but on looking
again for the source found only the contrary: p353, "The only safe plan is
use of random numbers for selection of items from a lot." (Attributed to
Mr Dave West, 1982)
What I then re-found was Deming's 1960 book "Sample Design in Business
Research", p31, "it is well to mention the existence and occasional
utility of judgment samples, deliberately biased samples, and chunks.
... For an evaluation of the reliability of such a survey, we must rely
on the expert's judgment: we can not use the theory of probability.
... There is no way to compare the cost of a probability sample with the
cost of a judgment sample, because ... [c]ost has no meaning without a
measure of quality."
One question is therefore whether any Demingists in the audience know
whether he changed his attitude over the years and became more puritan
with respect to statistical methods, or whether he continued to talk
about non-probability sampling but omitted it from "Out of the crisis" as
not relevant to quality-control sampling.
The second point is to ask if other list members teach this heterodox
view. I like to pose to students the question: if you are studying
education in, say, Hull and Hong Kong, and can study ONE school in each
city, how would you choose the schools? Simple random sampling from a
frame made up of all the school names is clearly ineffective - if you
solemnly then compared the small Moslem school in Hull with the largest
comprehensive in the New Territories, who would read the report, except
for the laughs?
---------- Forwarded message ----------
From: Ian Wilson <[log in to unmask]>
Subject: Re: Query - Deming and sampling
I don't know about Deming at all, but I confess I do teach what you
describe as heterodox. I had never dignified the thought with a title,
but if you've designed samples with practitioners you often see the
small-sample problem with first-stage units in a hierarchical sample. As
illustration of what we've said, I attach a section from a short guidance
booklet drafted for DFID researchers working in overseas development.
It's with various people for review at present so not in the public arena,
but I'd be glad to know if attached fits with your views. Criticism more
than welcome!
I have spent time trying to get DFID-funded researchers NOT to study one
unit (village, usually) in each setting. I try to persuade them to change
the objective and approach so they can have at least two units in each
setting to have some notion of "within" variability. Some are receptive,
but I can't recall any of them coming back with analysis datasets as yet!
From: "L. G. Tassinary" <[log in to unmask]>
Subject: Re: Query - Deming and sampling
I teach a graduate research methods course and the core text remains the
year old book by Cook & Campbell titled "Quasi-Experimentation: Design and
Analysis Issues for Field Settings". In their discussion of improving
external validity they explicity discuss three strategies. The first is the
traditional "Random Sampling for Representativeness Model" but they offer
the "Model of Deliberate Sampling for Hetergeneity" and "Impressionistic
Modal Instance Model" as both necessary and viable in many applied research
settings.
From: "Hutchison, Dougal" <[log in to unmask]>
I don't know about Deming, but Kish (1965), p19 says
'If a research project must be confined to a single city in the United
States, then I would rather use my judgment to chose a "typical" city than
select one at random. Even for a sample of ten cities I would rather trust
my knowledge of US cities than a random selection. (...) For a sample of
(30 to 100 cities) a probability slection should be designed and controlled
with stratification."
From: Tim Davis <[log in to unmask]>
The most important thing from Deming's teaching, related to your
discussion, is the difference between analytical and enumerative studies.=
See Chapter 7 of Deming's 1950 book "Some Theory of Sampling".
You will also find a letter from Deming himself in an edition of RSS News &
Notes (Sept 1992), with comments on samples and chunks of data.
From: Jay Warner <[log in to unmask]>
stated another way, which reached engineering UG's: You don't know how
much it costs until you know what it does. Referring to device design.
A statistically valid sample, from such a sampling plan, _assumes_
homogeneous population. If the pop is known not to be homogeneous, then
you cannot draw statistically valid conclusions, anyway.
If the population has a variance for a particular parameter, then this
will cloud the conclusions for small samples, and even define 'small.'
When we take notes on a sample of one (or few), _after_ it ha s been
selected, then we expect variance. What counted was how it was
selected, _before_ we knew how it deviated.
In your example below (I love that Muslim school in Hull!), we can
sample randomly from a large variance population, and let our
conclusions include the large variance estimate.
Or we can select the schools in the two cities as pairs, matching for
many other variables than the key one I am interested in. I think this
is the direction you are heading toward, and it is what is used for
surveys across diverse populations.
To: "CHASALOW, SCOTT [AG/2165]" <[log in to unmask]>
Subject: RE: Query - Deming and sampling
I've been sent a quote by Kish that reinforces the one I quoted from
Deming. No, a judgment sample would not be based on undefined or vague
criteria; on the contrary, that would be a non-random or arbitrary
choice. The judgment comes specifically from knowing the variation
between units, which may be multidimensional and unquantifiable, but then
choosing a sample which has to be justified as relevant.
To: Ian Wilson <[log in to unmask]>
Subject: Re: Query - Deming and sampling
Thanks for your notes, which have expanded my thinking. One of the
problems is getting the "user" to define the purpose of the study. Is it
to estimate some parameter from a defined population? Or is it to
demonstrate, purely, the existence of some effect? Most books on
experimental design tacitly assume that the outcome will be some
"measurement" with defined distribution (usually assumed Normal) and
hence a measured or controlled variance. The third possibility is that
you are using statistics to study a "system" and look for linkages (ie
associations, but everyone calls them correlations) which may be used to
impute mechanisms. This is strictly a subset of my second type, since
all you can reasonably show with a small sample is evidence for a
non-zero parameter in a model.
Your argument about moving from one sample to two seems to me potentially
misleading, as it puts the emphasis on the mathematical estimation of
variation while not providing a good estimate in just the circumstance I,
and you, indicate - a population known to be heterogeneous from which you
have perforce to choose a small sample that is accepted as
"representative" in some sense. The problem of preaching two is better
than one is that, presumably, the study must be at best half as intensive
at each site, minus the overheads of moving between sites and the problem
of getting started twice (an "edge effect") rather than once. The
contrast of one (intensive) versus many (extensive) is, however, well
worth considering as a practical/philosophical issue prior to the
consideration of getting best value (efficiency) from the selected study.
To: Jay Warner <[log in to unmask]>
Thanks for your comments. The other aspect of the "Hull vs HK" example
is that it presupposes that something is comparable. That's a serious
issue in education, as in the simplistic argument "why don't UK teachers
adopt Japanese teaching style and improve pupils' maths performance?"
Hence, I suggest, you have to fall back on judgment as to what is a
"typical" English school and a "typical" Chinese school and observe
similarities and differences without jumping to causal links.
Yes, there is a Muslim school of which I am aware because some of our
overseas students do teaching there. I think the children probably go to
an English school as well - the Muslim one is complementary.
R. Allan Reese Email: [log in to unmask]
Associate Manager Direct voice: +44 1482 466845
Graduate Research Institute Voice messages: +44 1482 466844
Hull University, Hull HU6 7RX, UK. Fax: +44 1482 466846
====================================================================
If Stephenson, Marconi and Edison had lived in the opposite order, so
the mobile phone got invented before the railway, would people still
find the need to travel, and tell people where they are?
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
