Hi All

Thanks very much to everyone who have me feedback to my question.  The
list of contributors is given here (in alphabetical order of first
name), and the responses in 'random' order given below the list of
names. (The question is repeated right at the end of the mail).


Names of contributors:
~~~~~~~~~~~~~~~~~~~~~
Allan White
Arturo González-Izquierdo
Chris
Eloisa Ruiz
Gjalt-Jorn Peters
Jay Warner
Peter Das
Robert Macrae
Richard Lawson
Siva


Responses:
~~~~~~~~~

My opinion on this is that if a question lends itself to a vast
proportion of "others" than the validity and robustness of the question
must be a concern. Any analysis/inference based on the "other" category
would be jumped upon (IMO) as in this situation the validity of the
other results(1-10) must be questioned.

However, I'm sure this is common sense which you know anyway, and is not
answering your original question.

So I would say if you're thinking long and hard about breaking down the
"others", then do it, but have a degree of reservation about what this
will actually tell you.

----------------------------------

You could use the Pareto principle (also known as the 80-20 rule, the
law of the vital few and the principle of factor sparsity) states that,
for many events, roughly 80% of the effects come from 20% of the causes.
So here if we use a baseline of 20 % or more for the other to be looked
at more carefully.

I have looked at tabulating the comments in the other category in some
of my past work, and it can be quite cumbersome. I found that looking
for key phrases which highlights a comment to be positive, neutral or
negative to help reduce the reporting. Also ranking the top 5  themes in
the comments by searching for key phrases in the other comment box.

----------------------------------

I'm no expert on surveys, but I find your question very interesting.
Recently I worked on a project where we had a similar scenario ~ 24%
stating "other" out of 8 options. The percentage itself is not that
important, what is more important is whether or not it is in the e.g.
top 5 (out of 10) of your options - A pareto chart is very useful to
highlight this and helps identify the options that are more frequent.

If the percentage is relatively high (to the other options) then it is
worth reading through the text and seeing if there are common responses.
This of course will be easier with less data, and issues such as simple
things being raised but people expressing themselves differently will
undoubtedly cause a few headaches, but is a worthwhile exercise
nonetheless. It may highlight an option which is reoccurring but has not
been included in the original design.

----------------------------------

In my opinion, you should aim to case 2 (5% or less select 'other) from
the
very design of the questionnaire (for which pilot studies are very
useful).
The more your real data departs from case 2 the stronger the indication
of a
badly design question.

The idea of providing answers in categories (options) is strongly based
on
the need of collecting data on a specific scale of measurement to allow
data
analysis "as objectively as possible". Therefore your options should
include
as many likely answers as possible in an mutually exclusive, and leave
to
'other' what cannot be predicted as likely answer in order to make your
scale of measurement exhaustive.
This means that if the possible answers were well considered, 'other'
should
be the less frequently selected option, or in other words the smallest
percentage, after the data has been collected.

If data has already been collected and a large proportion of people
selected
'others' then you should give detail (or break down) of this category if
it's greater than any of the other 10 categories or options. The number
of
groups in which this category should be broken down will depend on how
big
and diverse it is (which is most of the times dependant on the
objectives of
the study). Reporting the recoding is also important.

Having said that, it's worth to mention that sometimes the researcher is
ONLY interested in the 10 initial options for particular reasons, in
which
case it does not matter how big the category 'other' is. In this case,
reporting a few cases within 'other' would be enough.

----------------------------------

You do not mean questions like "town of birth" or "preferred composer"
I suppose, questions where there is just naturally a large number of
possibilities.

My subjective limit is 10%, or somewhat lower in a large survey of
n>=1000.
If the number of 'other' is larger, the likely answers were not
anticipated, or they just were too many. I would then consider recoding
the more frequent replies into extra options. If so, the final report
must explain what was done when reporting that survey question.

Anyway, someone with an interest in the study subject, for instance the
person who commissioned the survey, should read all remaining open
answers because there may be some that are valuable even if rare.
Possibly (if privacy allows) in the form of a list like
male 30-39 [comment]
male 15-19 [comment]
female 40-49 [comment]
etc...

----------------------------------

Seems to me, that when "other" becomes large enough to sway your
conclusion away from one of the specified alternatives, then you
should re-evaluate what "other" is. In the first case you cite, it
sounds as if you failed to find the real options that the respondents
cared about. Time to check back with them, see what was _really_ on
their minds.

In the second case, if you drop "other" your conclusions would
probably remain the same. You got the right options, in other words.

Now, for the kicker you really care about: What happens if "Other"
is 30%, or 20%? Suppose option A is 50%, Other is 30%, and B is
20%. Other could be a serious fly in the conclusion ointment. If
Other is only 10%, then it can't distort your conclusions about A and
B (now 40%). Unless A = B is a significant issue.

So, when Other can distort your (major?) conclusions, then look again.

----------------------------------

Interesting question :-)

I would say that breaking down 'other' is essentially a more
exploratory, qualitative approach to the question. I would therefore be
inclined to use a more qualitative reasoning and say 'break it down as
long as it's useful'.

I can imagine you'd prefer a more systematic/structured algorithm
though, but then the problem becomes that you need the assumption that
the frequency, with which something gets mentioned, is indicative of its
relevance . . . And even when you're willing to make that assumption,
you get back at your original question - when is something relevant
enough to break down?

----------------------------------

I am afraid that I do not know the answer to your question, but you
might want to try posting your question on the listserv of the Survey
Research Methods Section of the American Statistical Association

http://www.amstat.org/sections/srms/srms_net.html

(assuming you have not already done so).

----------------------------------

My own rule of thumb is that if it is NOT the category with the smallest
frequency of response, then it should be broekn down further.

----------------------------------

IMHO "Other" should be broken down when it contains more weight than the
largest of the other alternatives.

It might be broken down when there seems to be meaningful structure in
"Other" which would give rise to weights comparable to the other
alternatives.

I see no particular justification for these opinions 8-)

----------------------------------

Regards

Ken Masters


> -------- Original Message --------
> Subject: At what point is "other" broken down?
> From: Ken Masters <[log in to unmask]>
> Date: Tue, January 27, 2009 12:09 pm
> To: [log in to unmask]
>
>
> Hi All
>
> I'm not sure if there is a definitive answer to this question, but here
> goes.
>
> Let's say I have a survey question with, say, 10 options plus an "Other"
> and a text field for explanation of the "Other".
>
> Case 1: Only 10% of my sample select an option from my 10, and 90%
> select "Other", and explain what that "Other" is.  There is a sense
> then, that, in reporting the results, I really need to get some stats on
> the details of the "Other" (and not just report that 90% said "Other").
>
> Case 2: 95% of my sample select options from my 10 options, and only 5%
> select "Other."  There is a sense that, although the percentage of
> "Other" will be reported, and possibly a comment or two, there is no
> need to break it down into groups.
>
> But somewhere between that 90% and 5% is a line, above which details of
> the "Other" need to looked at carefully.  Where is that line?
>
>
> Regards
>
>
> Ken Masters