Hello,

I'm intrigued by the discussion before and after Common Ground about refereeing
 and reviewing papers. My experience is there are several widespread problems.
Most are classic business issues relating to vision, mission, strategy,
value production, constituent orientation, marketing, and HR training.

One of the core aspects of managing a conference/journal is the need to
have a vision, mission, and strategic position. As Rachel Cooper said
in an earlier post, 'Anyone who organises a conference or who edits a
journal, should be very clear as to the overall philosophy,  aims and
objectives'. These are the foundations of the detailed decision making
about all other issues including the choice of content. 

It is in this context that many conferences and journals choose to use
some form of peer review process (of which the double blind review by
2 or 3 referees is most common and highest status).

The double blind peer review process offers many benefits and is subject
to several problems. Its benefits are: 1. Critique of submissions free
from personal influence (the reviewer and contributor cannot be certain
of each others identity). 2. Multiple review of each contribution (2 or
3 reviewers independently advise the conference chair or editor as to
the quality of each submission - the final judgement resides with the
chair or editor). 3. Feedback to contributors as to content, logic of
argument, quality of writing etc and advice for changes that would make
the contribution acceptable.

Problems include: 1. Reviewing conference and journal papers requires
  skill sets from reviewers different from researching, lecturing or assessing
student work. 2. Many reviewers are untrained in these review skills.
3. There may be confusion as to whether the reviewers advise the conference
chair/editor or whether it is the reviewers that make the decision as
to acceptance or rejection. 4. Reviewers are usually well established
researchers in a field and their personal biases can act to inhibit new
ideas in the field.  5. Sometimes reviewers have an idea about the identity
of contributors and vice-versa that in principle might compromise their
judgement. This can happen by several means  - especially where the name
of the contributor or reviewer can be found in the meta-properties of
an electronic document.

Conferences and journals are different as means of disseminating new knowledge.
 A key role of any conferences is for contributors to put forward new,
well developed concepts for discussion and critique to a range of interested
peers. The purpose of the exercise is to identify problems and to improve
proposals and hypotheses. The author benefits from the critique, and the
audience benefits from obtaining advance knowledge about emerging trends
and theories in the field. In contrast, journals provide readers with
reliable critically assessed documentation of theory in a field at the
time  of publication. In other words, the aim is that what is read in
a journal can be depended upon as being well tested and well justified.
Thus, the contents of journals can be used directly to form the basis
of future research and theory making by other researchers and theorists.
Problems occur when reviewers view conference submissions as if they were
journal submissions and vice versa, or if they address them similarly.

Reviewing via abstracts is different. Reviewing on abstracts offers fewer
benefits to the participants and to the wider audience, and presents more
problems. The main advantages of this method of review are for conference
organisers who benefit in lots of ways, e.g.: reviewers are easier to
find (abstracts are shorter than papers); it is easier to get contributors
interested (writing an abstract is easier than writing a paper); organisers
cannot be criticised for final quality of content (they only judged the
abstract not the final paper); physical problems are reduced (abstracts
take up less space than full papers); proceedings are easier (abstracts
are shorter and easier to typeset than full papers); devising conference
streams is easier (abstracts can be interpreted in lots of ways because
they are less specific). There are, however, problems: many universities
will not fund faculty to attend conferences unless they are double blind
reviewed; quality is reduced in many dimensions; full paper proceedings
are rare because the organisers do not have copies of the participants
papers before the conference.

I feel the worst scenario is when double blind review of papers is combined
with a preliminary review by abstracts.  This approach offers apparent
benefits to the conference organisers in that themes are easier to manage.
Where rejection levels are high, however, it tends to cause high levels
of dissent in the field. Importantly, it can decimate the potential pool
of reviewers for papers. A traditional reason for reviewing by abstracts
is to do with limits to the number of presentations due, for example,
in Art-based design disciplines to limited display space. In conferences
in which research papers are presented, however, it is rarely necessary
to limit the number of presentations, and where the role of the conference
is to 'get the ideas out there and public' or encourage  new researchers,
it may be important that a conference expands in size as necessary.  It
is almost always possible to increase the number of presentation rooms
or change to a larger venue. The funding for such changes is naturally
provided as a result of the larger number of attendees.

The issue of conference themes can be difficult if organisers handle it
in a rigid manner. Clearly, some guidance is needed to contributors as
to what they should contribute. In reality, however, conference organisers
do not have a full picture of the state of a field at any time and do
not know who will contribute. Both factors suggest that theme building
should be flexible, and to a large extent developed after contributions
have been received and reviewed. Reviewers can play a strong role in shaping
the thematic structure of the conference. An exception is where a group
of researchers approach a conference chair with proposals for a theme
or 'panel'.

It is widely accepted the number and quality of academic papers is low
in fields of design research associated with traditional 'design' practices.
One way forward is for conference organisers, journal editors to put in
place an additional mechanism to provide support in parallel with existing
review processes. This might consist of support for new researchers from
a panel of researchers who have expertise and experience in creating published
papers.  

Suggestions:

1.      Greater attention to identifying vision, mission, strategies and objectives
2.      Address value production, constituent orientation, marketing issues
3.      Provide training for staff and reviewers.
4.      Provide an additional loop for assisting new researchers in writing
conference and journal papers
5.      Use only double blind review of full papers.
6.      In conferences, accept as many papers as are of sufficient quality
and fit within the conference's scope. Expand the size of the conference
as appropriate.
7.      More flexible approach to themes in conferences. Identify exact themes
after acceptance of papers.


Best wishes,

Terry

______________________

Dr. Terence Love
Love Design & Research
GPO Box 226
Quinns Rocks
Western Australia 6030
Tel/fax: +61 (0)8 9305 7629
Email: [log in to unmask]
______________________