JISCMail - PSYCH-POSTGRADS Archives

Echoing what Jeremy said.

I'd just add that while people traditionally treat the ratings in these models as continuous you can and probably should treat them as ordinal. Until recently that was very tricky to do as a multilevel model. Software has now advanced so you can run ordinal multilevel models much more easily (but it still a bit of a step up).

The basis model here would be ordinal logistic regression. I use the ordinal package in R for this - or more recently the Bayesian modelling package brms (also in R). The latter is more flexible and has better graphics (but requires more investment). Both ordinal and brms use the same formula syntax (borrowed from the widely used lme4 package in R). I think you could also do something similar in certain SEM packages such as MPLUS.

This tutorial by the author of brms package introduces ordinal models:

https://psyarxiv.com/x8swp/

(I don't think it goes as far as multilevel extensions but it does introduce signal detection models which are a natural extension of these methods).

A half-way house it to use an empirical logistic transformation after scaling the ratings to be 0 to 1 (rather than 1 to 5). The typical regression approach may also work well in some instances but runs into difficulties with bunching of ratings close to the extremes. Not surprisingly it works well if most ratings are in the middle and the distributions are vaguely symmetrical.

Sorry - reply was longer than I intended!

Thom

From: Research of postgraduate psychologists. <[log in to unmask]> on behalf of Jeremy Miles <[log in to unmask]>
Sent: 16 May 2018 17:17
To: [log in to unmask]
Subject: Re: Stuck on experiment analysis

I would analyze this as a multilevel (random effects, mixed) model.

When you do this, you make the data long, rather than wide. Here's an example:

Here are your wide data:

Participant X Private Taxi Public

1 3 4 1 4

2 2 2 4 3

...

X represents a predictor of some sort (or a set of predictors, e.g. time).

Make the data long:

Participant Mode X Score

1 Private 3 4

1 Taxi 3 1

1 Public 3 4

2 Private 2 2

2 Taxi 2 4

2 Public 2 2

Now you only have one DV (outcome variable). So you can do a regular regression (ANOVA), but you have a new problem people are repeated on rows, which is normally a no-no. Multilevel models fix that.

How you do it depends on what program you use. If you tell us, someone might be able to help.

On Mon, 14 May 2018 at 03:32 I Schubert <[log in to unmask]> wrote:

Dear All,

I am a bit stuck on the analysis of an experiment and wanted to ask your advice. We ran a choice experiment with within and between subject measures where participants had to choose between three modes of transport (autonomous private vehicle, autonomous taxi, autonomous public transport). But unlike in a typical choice experiment participants had to indicate the likelihood of choosing all three transport modes each time on the 1-5 likert scale. The experiment varied the attributes of the modes of transport (e.g. traveling time and cost) and participants had to go through 4 blocks of treatments (time, cost, comfort and combined measures of all three, in a random order) and in each block they were randomly presented 1 out of 2-4 scenarios. All scenarios were presented randomly but to equal distributions in each block). Before the experiment all participants answered a baseline scenario question without any scenario variations.

Because there are 3 DVs and not just one I am really unsure how to go about analysing the data. Any thoughts would be highly appreciated.

Iljana Schubert
Basel University

My employer has nothing to do with this email.