A big thank you to everyone who responded to my surveys on the qsr
(qualitative) and allstat (quantitative) lists. I have written a draft
report and hope to carry out a little more work on the topic. Meanwhile,
here is a text extract from the report.

Background

Participants at a workshop in communication skills organized by the Staff
Development department at the University of Hull were given an exercise in
attaching quantitative values to words or phrases ("Fuzzy Meanings" an
exercise submitted by Jennifer Ramirez in "Still more games trainers play"
by Scannell, E.E. and Newstrom, J.W., 1991, McGraw-Hill). The exercise
seems to have originated in artificial intelligence studies based on fuzzy
logic (see, eg, "Social Fuzziology" by Vladimir Dimitrov,
www.uws.edu.au/vip/dimitrov/fuzziology.htm). As a participant, the
exercise seemed to me worth repeating with larger groups of respondents,
and in particular to compare groups who might be more, or less, likely to
be comfortable with ideas of quantification.


Results

Responses were analyzed from 13 respondents at the workshop, 33 from the
qualitative list, and 118 from the statisticians' list.

The first striking feature was the consistency of interpretation of many
of the terms despite the vagueness of the guidance. The words could be
ordered sensibly based on the central tendency of responses, using either
the mean or the median. There was good agreement between the three
groups, with rankings differing between some of the middle terms but based
on very small differences in the mean percentages. Responses showed less
variation for terms that are implicitly more absolute. Nevertheless, some
respondents were markedly discordant on either particular terms or
overall, so the responses need to be considered both for general patterns
and for exceptions. The central tendencies, ranges of value excluding
outliers (as defined below) and ranks for each group are presented as
Table 1.

Table 1: Terms ordered by the median value for the full data. Tied medians
broken using the mean.

Word or phrase Median Ranks
(variable) percent (Range) Workshop QSR Allstat

Always 100 (90-100) 1 1 1
Almost always 95 (50-99) 2 2 2
Most of the time 85 (50-99) 3 3 3
Usually 80 (40-99) 6 4 4
A lot of the time 75 (25-95) 7 6 5
Frequently 70 (20-90) 5 5 6
Often 70 (20-95) 4 7 7
Quite often 60 (10-92) 8 8 8
Sometimes 30 (1-60) 9 9 9
Occasionally 17.5 (1-70) 10 10 10
Seldom 5 (0-40) 11 11 11
Rarely 5 (0-59) 12 12 12
Never 0 (0-9) 13 13 13

Max/Min
No of respondents 164/162 13/12 33/32 118/116

The distributions of values were tested using exploratory data analysis
(EDA) techniques and non-parametric tests against standard distributions.
A normal distribution was rejected for each of the thirteen variables.
The arcsin transformation is routinely applied to proportional data as a
normalizing and variance-stabilizing transformation [specifically,
sin-1(sqrt(p/100)) ], but all the transformed values were still rejected
by a Kolmogorov-Smirnov test of normality. Examination of histograms and
quantile (q-q) plots showed that this was due for most terms to an excess
at the modal value (high kurtosis) and the presence of unexpected extremes
(outliers). The exceptions were the terms "quite often" and "sometimes"
which showed flatter-than-normal distributions. One reason for the high
kurtosis was that respondents did not discriminate at less than 5%
intervals, but generally used round figures.

The non-normality of the responses suggested that a non-parametric test
should be used to compare the groups. The Kruskal-Wallis test that all
three groups showed similar distributions of responses (against the
alternative that at least one group shows generally higher-ranked values),
using a Bonferroni adjustment on the test statistics, indicated that the
groups differ only in their interpretation of "occasionally". The reason
for this appeared to be that within the statisticians group, higher
proportions opted for the rounded values 10% (chosen by 38% of
statisticians, 17% of rest) and 20% (chosen by 25% of statisticians, 17%
of rest). The Median test indicated possible difference in only
"occasionally" and "rarely", in both cases with the qualitative
researchers suggesting higher values.

Kruskal-Wallis tests also found no differences in distributions of
responses between males and females, English first language or not,
country of origin (grouped as GB, US, Australia/New Zealand, and Other),
or area of work (academic, commercial or other).

The main conclusion is therefore that the general understanding of the
terms was consistent and stable across intellectual groups, work areas,
and social and cultural backgrounds. It seems unlikely that other groups,
if surveyed, would differ markedly in their ordering of the terms, even if
they differed in their choice of numerical equivalents.

[box and whisker plot omitted here]

Cases 134, 139 and 163 were extreme on several variables. All came from
the statisticians' list. Two were from Asia and English was their second
language, but the other was originally from the UK. The striking feature
was that all three failed to use the whole range 0-100 to cover the terms.

Cases 20 and 98 were highly anomalous in their interpretation of "never".
Both were contacted by email and confirmed that they had intended this
response. Case 20 was an academic qualitative researcher who commented
that "people's memories are often short. They may have done whatever once
upon a time. So this just means 'currently' never." Case 98 was a
statistician whose first language was not English and who did not appear
to notice any anomaly in this response. Case 8 (from the workshop) was
anomalous on "almost always". The form was checked for this being a
data-entry error, but it was not. It would not be feasible (or ethical?)
to trace the respondent.

The term "most of the time" has a compact box and whisker but a higher
number of outliers than other variables. Logically, "most of" implies a
majority, and there may be a division between those who interpreted the
phrase as a bare majority and the modal group who placed it above 80%.
Four responses of ">50%" and one of "51%" may indicate rationalizations
rather than subjective practical use.

The ranges of the boxes and whiskers, and the number of outliers can be
seen to be greater on the terms toward the middle of the scale. One
feature that was looked for specifically was the relative values of
"often" and "quite often", since several commentators (eg, Alistair Cooke
on the BBC) have suggested that "quite" is a weakening adverb in England
but an intensifier in the United States. No such effect was observed in
these data. The highest values associated with the phrase came from UK
respondents.


Discussion

Several members of both lists questioned the point of the survey by
suggesting that language has to be contextual. Different results would
have been obtained had the questions been made concrete by asking
respondents to react to specific instances, eg, "I never tell lies, I will
always love you." Contextual variation in meaning, however, depends upon
imbuing the basic meaning with a filter of suggestion, hyperbole, irony,
or ulterior motive. The questions asked simply for a face-value
understanding, and the results confirm that a high degree of agreement can
be achieved. One interpretation of the survey question would be to relate
the percentages to a denominator of times when something was available or
possible. "I almost always get to work on time" relates perhaps to
numbers of days late in a month, while "I always play golf on Sundays" may
implicitly exclude times when the weather or illness prevent play.

Language does allow for ambiguity, either intended or not, and emanating
from either the transmitter or receiver of the message. Some of the terms
in the list are patently more precise or more vague than others. The
numbers of outliers suggest that it would not be reasonable to infer a
firm number when an interviewee uses a word, but the agreement on modes
does suggest that at least some of these terms may be interpreted
quantitatively in written reports. The common understanding of terms may
be less when dealing with someone whose first language is not English,
hardly a surprising conclusion except that the outlying respondents here
were quantitative researchers working in an English medium.

Three respondents reported carrying out similar exercises, and one
mentioned that, as found here, the words spanning the middle range had
been more variable in their ordering.

Some of the widest variations were seen in the Hull workshop group, which
was the smallest but may have been the most diverse, in that the
participants covered academic, clerical and technical staff. No
demographic data were collected, but the group included at least one
person whose first language was not English.

"The reason that language has fuzzy terms is not that human thought is
fuzzy, but that the world is fuzzy." (Bruce Schuman "Introduction to the
theory of concepts" 1994. http://originresearch.com/sd/sd4.cfm)


Acknowledgement

The Hull workshop was organised by Linda Marshall from the University of
Hull Staff Development office, who also provided the reference to Scannell
and Newstrom.



R. Allan Reese Email: [log in to unmask]
Associate Manager GRI Direct voice: +44 1482 466845
Graduate School Voice messages: +44 1482 466844
Hull University, Hull HU6 7RX, UK. Fax: +44 1482 466436
====================================================================