Dear Allan,

Thanks for the interesting extracts from your report on quantifiers.

There is an extensive literature on quantifiers, ranging through
linguistics, psychology, cognitive science, logic, etc.

Some of the work has concerned the meaning and use of frequency and
probability. I am too busy right now to pass on the most
relevant references, but a good overview is by two psychologists in
Glasgow University, viz., Moxey, L.M. & Sanford, A.J. (1993).
Communicating Quantities: A Psychological Perspective. Hove: Lawrence
Erlbaum Associates.

A few other people might be interested and so I have posted a general
reply.

Yours sincerely,

David Routh

On Thu, 28 Nov 2002 12:01:03 +0000 "R. Allan Reese"
<[log in to unmask]> wrote:

> A big thank you to everyone who responded to my surveys on the qsr
> (qualitative) and allstat (quantitative) lists. I have written a draft
> report and hope to carry out a little more work on the topic. Meanwhile,
> here is a text extract from the report.
>
> Background
>
> Participants at a workshop in communication skills organized by the Staff
> Development department at the University of Hull were given an exercise in
> attaching quantitative values to words or phrases ("Fuzzy Meanings" an
> exercise submitted by Jennifer Ramirez in "Still more games trainers play"
> by Scannell, E.E. and Newstrom, J.W., 1991, McGraw-Hill). The exercise
> seems to have originated in artificial intelligence studies based on fuzzy
> logic (see, eg, "Social Fuzziology" by Vladimir Dimitrov,
> www.uws.edu.au/vip/dimitrov/fuzziology.htm). As a participant, the
> exercise seemed to me worth repeating with larger groups of respondents,
> and in particular to compare groups who might be more, or less, likely to
> be comfortable with ideas of quantification.
>
>
> Results
>
> Responses were analyzed from 13 respondents at the workshop, 33 from the
> qualitative list, and 118 from the statisticians' list.
>
> The first striking feature was the consistency of interpretation of many
> of the terms despite the vagueness of the guidance. The words could be
> ordered sensibly based on the central tendency of responses, using either
> the mean or the median. There was good agreement between the three
> groups, with rankings differing between some of the middle terms but based
> on very small differences in the mean percentages. Responses showed less
> variation for terms that are implicitly more absolute. Nevertheless, some
> respondents were markedly discordant on either particular terms or
> overall, so the responses need to be considered both for general patterns
> and for exceptions. The central tendencies, ranges of value excluding
> outliers (as defined below) and ranks for each group are presented as
> Table 1.
>
> Table 1: Terms ordered by the median value for the full data. Tied medians
> broken using the mean.
>
> Word or phrase Median Ranks
> (variable) percent (Range) Workshop QSR Allstat
>
> Always 100 (90-100) 1 1 1
> Almost always 95 (50-99) 2 2 2
> Most of the time 85 (50-99) 3 3 3
> Usually 80 (40-99) 6 4 4
> A lot of the time 75 (25-95) 7 6 5
> Frequently 70 (20-90) 5 5 6
> Often 70 (20-95) 4 7 7
> Quite often 60 (10-92) 8 8 8
> Sometimes 30 (1-60) 9 9 9
> Occasionally 17.5 (1-70) 10 10 10
> Seldom 5 (0-40) 11 11 11
> Rarely 5 (0-59) 12 12 12
> Never 0 (0-9) 13 13 13
>
> Max/Min
> No of respondents 164/162 13/12 33/32 118/116
>
> The distributions of values were tested using exploratory data analysis
> (EDA) techniques and non-parametric tests against standard distributions.
> A normal distribution was rejected for each of the thirteen variables.
> The arcsin transformation is routinely applied to proportional data as a
> normalizing and variance-stabilizing transformation [specifically,
> sin-1(sqrt(p/100)) ], but all the transformed values were still rejected
> by a Kolmogorov-Smirnov test of normality. Examination of histograms and
> quantile (q-q) plots showed that this was due for most terms to an excess
> at the modal value (high kurtosis) and the presence of unexpected extremes
> (outliers). The exceptions were the terms "quite often" and "sometimes"
> which showed flatter-than-normal distributions. One reason for the high
> kurtosis was that respondents did not discriminate at less than 5%
> intervals, but generally used round figures.
>
> The non-normality of the responses suggested that a non-parametric test
> should be used to compare the groups. The Kruskal-Wallis test that all
> three groups showed similar distributions of responses (against the
> alternative that at least one group shows generally higher-ranked values),
> using a Bonferroni adjustment on the test statistics, indicated that the
> groups differ only in their interpretation of "occasionally". The reason
> for this appeared to be that within the statisticians group, higher
> proportions opted for the rounded values 10% (chosen by 38% of
> statisticians, 17% of rest) and 20% (chosen by 25% of statisticians, 17%
> of rest). The Median test indicated possible difference in only
> "occasionally" and "rarely", in both cases with the qualitative
> researchers suggesting higher values.
>
> Kruskal-Wallis tests also found no differences in distributions of
> responses between males and females, English first language or not,
> country of origin (grouped as GB, US, Australia/New Zealand, and Other),

> or area of work (academic, commercial or other).
>
> The main conclusion is therefore that the general understanding of the
> terms was consistent and stable across intellectual groups, work areas,
> and social and cultural backgrounds. It seems unlikely that other groups,
> if surveyed, would differ markedly in their ordering of the terms, even if
> they differed in their choice of numerical equivalents.
>
> [box and whisker plot omitted here]
>
> Cases 134, 139 and 163 were extreme on several variables. All came from
> the statisticians' list. Two were from Asia and English was their second
> language, but the other was originally from the UK. The striking feature
> was that all three failed to use the whole range 0-100 to cover the terms.
>
> Cases 20 and 98 were highly anomalous in their interpretation of "never".
> Both were contacted by email and confirmed that they had intended this
> response. Case 20 was an academic qualitative researcher who commented
> that "people's memories are often short. They may have done whatever once
> upon a time. So this just means 'currently' never." Case 98 was a
> statistician whose first language was not English and who did not appear
> to notice any anomaly in this response. Case 8 (from the workshop) was
> anomalous on "almost always". The form was checked for this being a
> data-entry error, but it was not. It would not be feasible (or ethical?)
> to trace the respondent.
>
> The term "most of the time" has a compact box and whisker but a higher
> number of outliers than other variables. Logically, "most of" implies a
> majority, and there may be a division between those who interpreted the
> phrase as a bare majority and the modal group who placed it above 80%.
> Four responses of ">50%" and one of "51%" may indicate rationalizations
> rather than subjective practical use.
>
> The ranges of the boxes and whiskers, and the number of outliers can be
> seen to be greater on the terms toward the middle of the scale. One
> feature that was looked for specifically was the relative values of
> "often" and "quite often", since several commentators (eg, Alistair Cooke
> on the BBC) have suggested that "quite" is a weakening adverb in England
> but an intensifier in the United States. No such effect was observed in
> these data. The highest values associated with the phrase came from UK
> respondents.
>
>
> Discussion
>
> Several members of both lists questioned the point of the survey by
> suggesting that language has to be contextual. Different results would
> have been obtained had the questions been made concrete by asking
> respondents to react to specific instances, eg, "I never tell lies, I will
> always love you." Contextual variation in meaning, however, depends upon
> imbuing the basic meaning with a filter of suggestion, hyperbole, irony,
> or ulterior motive. The questions asked simply for a face-value
> understanding, and the results confirm that a high degree of agreement can
> be achieved. One interpretation of the survey question would be to relate
> the percentages to a denominator of times when something was available or
> possible. "I almost always get to work on time" relates perhaps to
> numbers of days late in a month, while "I always play golf on Sundays" may
> implicitly exclude times when the weather or illness prevent play.
>
> Language does allow for ambiguity, either intended or not, and emanating
> from either the transmitter or receiver of the message. Some of the terms
> in the list are patently more precise or more vague than others. The
> numbers of outliers suggest that it would not be reasonable to infer a
> firm number when an interviewee uses a word, but the agreement on modes
> does suggest that at least some of these terms may be interpreted
> quantitatively in written reports. The common understanding of terms may
> be less when dealing with someone whose first language is not English,
> hardly a surprising conclusion except that the outlying respondents here
> were quantitative researchers working in an English medium.
>
> Three respondents reported carrying out similar exercises, and one
> mentioned that, as found here, the words spanning the middle range had
> been more variable in their ordering.
>
> Some of the widest variations were seen in the Hull workshop group, which
> was the smallest but may have been the most diverse, in that the
> participants covered academic, clerical and technical staff. No
> demographic data were collected, but the group included at least one
> person whose first language was not English.
>
> "The reason that language has fuzzy terms is not that human thought is
> fuzzy, but that the world is fuzzy." (Bruce Schuman "Introduction to the
> theory of concepts" 1994. http://originresearch.com/sd/sd4.cfm)
>
>
> Acknowledgement
>
> The Hull workshop was organised by Linda Marshall from the University of
> Hull Staff Development office, who also provided the reference to Scannell
> and Newstrom.
>
>
>
> R. Allan Reese Email: [log in to unmask]
> Associate Manager GRI Direct voice: +44 1482 466845
> Graduate School Voice messages: +44 1482 466844
> Hull University, Hull HU6 7RX, UK. Fax: +44 1482 466436
> ====================================================================
>

----------------------
David A. Routh,
Department of Experimental Psychology,
University of Bristol,
8 Woodland Road,
Bristol BS8 1TN, UK

Tel: +44 (0)117 928 8561
Fax: +44 (0)117 928 8588

E-mail: [log in to unmask]