Dear Colleagues,
In the past few days, I’ve been getting requests for copies of the
preliminary results on our Design Research Journal Study. In this post,
I am sending our preliminary results.
Deirdre Barron, Gavin Melles, and Silvana Ferlazzo from Swinburne
University of Technology have been working on this with me, along with
Jeremy Yuille and Tania Ivanka from RMIT. We’re planning to develop this
further in the next few months.
The very fact that we have done this is creating interest, however, and
some of our staff are getting requests for copies.
Please let me remind you of the background to this study.
We are not advocating a specific metric system for ranking journals.
The new government of Australia has established a ranking system for
journals.
We are responding to the position of design and design research journals
– and gaps for design and design research – in the proposed system.
The proposal is based solely on suggestions from within Australia and
New Zealand.
Our study is based on a world-wide survey of the global design research
community.
The preliminary results you will find here permit us to share what we
have learned so far. As far as I know, this is the first and largest
study of its kind in design and design research. Because it represents a
wide variety of respondents, the results are broad and reasonable.
Nevertheless, the methods need to be refined. There are obvious
methodological problems in a survey done as swiftly as we need to do
this. We were operating on a tight deadline for the response, so this is
a beginning.
We will welcome suggestions for improvement.
The most severe methodological objection, of course, is that we only
have responses from a self-selected group of respondents from among the
lists we queried. This flaw, of course, was also a key problem in the
original proposal, at least as far as design and design research were
concerned.
In contrast, we sent our survey request to a selection of excellent
lists representing the entire world.
We have now discovered one reason for some odd choices in the original
proposal. After completing this report, we learned that some proposed
journal rankings are the result of lobbying and vote-gathering by
special interest groups. This practice may involve more than one group
or subgroup in the design disciplines, and it seems to involve groups in
several other fields seeking to position specific journals to special
advantage.
To avoid embarrassment, I will not name the group, but one design
specialty association in Australia and New Zealand publishes a
relatively new journal. The editor once told me that submissions are not
of the highest quality, and that many submissions are rejected or
subject to drastic revision because they do not meet the standards of
serious research scholarship. Nevertheless, the association waged a
campaign among members to seek an A* ranking (top 5%) for the journal.
While they did not get an A* rating, they did earn an A rating (top
20%).
Based on our worldwide survey, this journal merits only a C. It is not
within the top 40% mentioned, not even among design and design research
scholars in Australia or New Zealand. The lobby for the subdiscipline
seems to have been quite successful in the effort to influence the ERA
proposal. They secured an A rating for a minor journal read mostly by
sub-discipline specialists in Australia and New Zealand -- and they
earned an A* for the senior international journal in their field. While
this international journal is a serious journal from a major publisher,
our survey shows the journal as a B rank journal, coming in at the
bottom of the second 20%.
This suggests that some special interest groups have actively and
effectively campaigned for high rankings, using targeted campaigns to
inflate the stature of the journals where they publish. We have learned
about one or two specific efforts. Others must also have been active to
get the kind of skewed rankings we see in the Eappropriately by journals that hold up to reasonable comparison with
journals of other fields. This requires an intelligent perspective based
on information from the fields of design and design research as a whole,
as well as fairly reflecting the design sub-disciplines.
Feel free to share what we have learned with anyone you wish. Please
make our motives clear: we wish to represent the field and we wish to
ensure that those who work in design and design research have
appropriate credit for the serious work they do. This requires sober and
responsible lists that reflect the relative standing of journals in
design research and design.
We’ll be going further with this in October. If you have suggestions,
please send them to me and to Dr. Deidre Barron, our Associate Dean
Research, at:
[log in to unmask]
Warm wishes,
Ken Friedman
Professor, Ph.D., Dr.Sci. (hc), FDRS
Dean, Swinburne Design
Swinburne University of Technology
Melbourne, Australia
--
Design Research
Journal Ranking
Study
Preliminary Results
2008 July 26
Contents
Background
A Study of Journals in Design and Design Research
Problems for Design research in the ERA Proposal
Problems for Journals in Design and Design Research
Comparative Problems for the Field
Appendix A - Journals by Rank Order
Appendix B - Journals Clustered by Numbers of Responses
Appendix C - Usable Responses and Respondent Profile
Design Research Journal Ranking Study: Preliminary Results
by
Prof. Ken Friedman (1), Dr. Deirdre Barron (1), Ms. Silvana Ferlazzo
(1), Ms. Tania Ivanka (2), Dr. Gavin Melles (1), and Dr. Jeremy Yuille
(2).
(1) Swinburne University
(2) RMIT University
--
This reports the first results of an international study on comparative
rankings of journals in design and design research.
Please send comments and suggestions to
[log in to unmask]
with a copy to
[log in to unmask]
--
Background
Australian universities recently received a list of journals from the
Australian Research Council (ARC). This list was the proposed set of
journal rankings for the new Excellence in Research for Australia (ERA)
initiative. This list includes journals for design and design research.
Few design research journals appear on the list, and moist of those few
have low rankings.
The ERA journal-ranking proposal is intended to cover all appropriate
journals across all fields. Journal rankings for any specific field
should represent an appropriate distribution of journals for that field.
Journals are described as A* journals representing the top 5% of
journals in a field, A journals representing the next 15%, B journals
representing the next 20%, and C journals for all the rest.
The political and financial importance of the ranking system is one
aspect of the new government’s move from the former Research Quality
Framework (RQF) to the new metric assessment system developed for the
ERA.
This system will help to measure the research activity of any specific
university. It should also demonstrate the research activity and quality
of individual units, schools, or faculties. The research points
allocated to design faculties or design schools in Australian
universities depend on the number of points allocated to each article.
That, in turn, depends on the ranking of the journal in which an article
appears. Research income will follow these points, as will the
credibility that researchers have when they apply for grant funding.
We identified several problems in reviewing the proposed list. Some
problems affect the field of design and design research. Other problems
affect all sectors.
To examine some of these problems, we will refer to the results of our
study. We therefore outline our study before going further.
A Study of Journals in Design and Design Research
To identify appropriate journals for the design sector, we conducted a
worldwide study by inviting leading scholars in design and design
research to lisan open request, giving neither titles nor prompts for order ranking. We
derived order ranking for journals based on the number of times a
journal was listed in unprompted response.
We sent an invitation to several high quality lists of scholars and
specialists in design and design research. These lists included: 1) the
membership and fellows list of the international Design Research Society
(ca. 500), 2) the email bulletin of Design Studies Forum, the design and
design research division of the North American College Art Association
(ca. 400), 3) the world’s largest specialized design research discussion
group, the Anthropology in Design research discussion forum (ca. 1,500),
4) the world’s largest general design research discussion forum, the
JISCMAIL-based PhD-Design list with subscribers from many of the world’s
leading universities (ca. 1,400), 5) a broad selection of journal
editors and editorial board members from a wide range of journals (ca.
200), 6) members of the Australian Built Environment and Design Deans
(ca. 17). While there is significant overlap between and among these
lists, these enabled us to reach much of the world’s active design
research community on short notice.
Scholars in several nations and regions sent the survey on to
colleagues. Prof. Lin-Lin Chen, Dean of Design at the National Taiwan
University of Science and Technology, was especially helpful in
circulating the request in Asia. Dr. Anna-Louise Sommer, Head of Design
Research at the Danish Design School, helped us to expand coverage in
Scandinavia.
The survey drew 240 usable responses. These responses represent 297
respondents, a discrepancy that we account for by the fact that some
respondents hold two titles, for example, professor and dean or lecturer
and research fellow.
By title or position, the respondents included 63 professors, 7 deans, 5
associate deans, 43 associate professors, 37 assistant professors, 7
readers, 25 research fellows and senior research fellows, 30 senior
lecturers, 38 lecturers, and 42 others.
In all, the survey yielded 173 journal titles.
The greatest number of responses for any listed journal was 152 for the
journal Design Studies from Elsevier. The greatest number of journals –
over 130 – were mentioned five times or fewer.
Given the number and spread of respondents in terms of field, rank, and
geographical distribution, this study can reasonably be said to
represent the worldwide fields of design and design research. The
absolute number of times any journal is mentioned is proportionate to
the size of the sample. The comparative proportion of times that
different journals are mentioned reflects the relative standing of
journals within the field. A larger sample size would give greater
number of times mentioned, but the relative proportions of times
mentioned would likely be relatively similar.
The extremely long tail with few mentions mirrors the wide distribution
of journals within the sector. In the last United Kingdom Research
Assessment Exercise, the art and design panel received articles from
over 500 different journals. If these were roughly divided between art
journals and design journals, that would give a roughly equivalent
total.
Because the tail is long with few mentions, we have decided that the top
5, the next 15, and the next 20 journals form a reasonable proxy for the
top 5%, 15%, and the next 20%. These percentages are restrictive, given
that 173 journal titles would mean a top 40% (5%, plus 15%, plus 25%) of
68 journals rather than 44 journals with a line drawn at 6 mentions or
more. If we were to end the list of B journals at 7 mentions or more,
there would be 37 journals in the top three levels.
Our next study will use several comparative methods to measure the
validity of this first study and to draw on a larger sample with more
definitive results. We hope to develop a more fine-grained analysis to
answer questions we do not address here. We are completing this study in
the time required to This the first study of its kind in the field, based on a large and
reliable sample. The titles are well known. The reputations, impact
factors, and index standing of the top journals supports our results.
Based on responses to the survey, the journals that we will recommend
for the levels designated A*, A, and B follow. The numbers after each
title indicate the number of unprompted responses from among the 240
completed survey forms:
A*
Design Studies - 152
Design Issues - 146
International Journal of Design - 85
Design Journal - 84
Journal of Design History - 48
5
A
Journal of Design Research - 42
Engineering Design - 35
CoDesign - 32
Artifact - 25
Journal of Engineering Design - 25
10
Research in Engineering Design - 24
Design Philosophy Papers - 23
Applied Ergonomics - 21
Visible Language - 20
CAD - 16
15
Design Management Journal - 15
Design Research Quarterly - 14
Fashion Theory - 13
Human-Computer Interaction - 13
Interacting with Computers - 13
B
Print Annual - 13
Eye - 12
Information Design Journal - 12
Journal of Technology and Design Education - 12
Digital Creativity - 11
25
Journal of Product Innovation Management - 11
Design Management Review - 10
Visual Communication - 9
Design and Technology Education - 8
Environment and Planning - 8
30
Journal of Architectural Education - 8
Winterthur Portfolio - 8
AI for Engineering Design, Analysis and Manufacturing - 7
Art, Design and Communication in Higher Education - 7
Journal of Interior Design - 7
35
Studies in the Decorative Arts - 7
Textile – 7
37
Architectural Review - 6
Communication Arts - 6
Innovation - 6
40
International Journal of Art and Design Education - 6
Journal of Interior Design - 6
Journal of Mechanical Design - 6
The Radical Designist - 6
44
When we return to this project, we will give deeper consideration to
methodological issues. Right now, we are dealing with the need to
respond to the ERA proposal.
The study yielded a list of 173 journal titles in design and design
research. We will present the full results of the study and our
preliminary results after considering some of the problems in the
original ERA proposal.
Problems for Design and Design Research in the ERA Proposal
The major problem in the ERA journal rankings for design and design
research is that the list seems to have been established by scholars in
other fields. While the ERA list seems to function for scholars in art
or architecture, there is no comparable relationship to scholarship in
design or design research.
One explanation is that no one in design research seems to have been
involved in the development process. This may be a result of historical
factors.
Most design programs In Australia and New Zealand seem to be located in
faculties of art or architecture. If a senior professor or the associate
dean for research made suggestions for the ERA proposal on behalf of an
art or architecture faculty, the faculty proposal would generally
represent greater knowledge of art or architecture that design. This is
compounded by a second problem.
Design research is a relatively young field. It is four decades since
Nobel Laureate Herbert Simon wrote the first landmark book in design
science, The Sciences of the Artificial. At roughly the same time, Prof.
Bruce Archer launched the Department of Design Research at the Royal
College of Art in London. The field of design research has developed
slowly since that time.
This parallels the transformation of other professional fields from
guild practice to research-based practice. Medicine went through the
transformation in the late 1800s. It made the decisive shift to a
research base in the wake of the Carnegie Commission’s Flexner Report.
After the report appeared in 1910, many North American medical schools
closed or were reformed. This decisively changed the face of medicine.
Engineering went through a less dramatic transformation in subsequent
decades of. This has now haMuch of the fundamental work in this field has taken place in the United
Kingdom and North America. Australia and New Zealand have relatively few
senior scholars in the field. Since there are few design faculties in
Australia or New Zealand with design research scholars holding position
at the level of dean or associate dean, it is understandable that the
ARC did not get appropriate design research advice for the ERA proposal.
Other fields may face this problem in different ways.
Nevertheless, the lack of involvement from any specific field may not
represent a major problem. The ERA journal proposal is open to
adjustment and the ARC has invited expert commentary. With expert
advice, it should be possible to create a responsible ranking system
that allows for differences among fields.
The use of a single ranked list creates a different kind of problem that
cannot readily be solved in the short term. This is the fact that many
journals serve several adjacent fields. Such journals often have
different impact and different relative value for authors and faculties
in the different adjacent fields.
A small group of scholars from Swinburne University of Technology and
RMIT University completed this study swiftly to permit a reasonable
response to the ERA proposal from the field of design and design
research.
In September, we will begin a deeper, more fine-grained study that
permits us to examine specific problems and challenges in greater depth.
In that study, we plan to examine some of the problems specific to the
design field in any metric ranking system, along with an examination of
broader problems that may affect many fields.
Several international colleagues and one or two Australian deans have
suggested that metric systems or journal rankings may not be useful or
valid for the fields of design and design research. We do not address
this question.
The decision to establish a journal ranking system is a government
policy decision. Australia is moving from the former RQF system to an
ERA system based in great part on metrics. One metric involves journal
rankings. Design faculties don’t set government policy. We can respond
to specific issues in the current proposed journal-ranking scheme. That
is our purpose here.
[continued]
-----
Swinburne University of Technology
CRICOS Provider Code: 00111D
NOTICE
This e-mail and any attachments are confidential and intended only for
the use of the addressee. They may contain information that is
privileged or protected by copyright. If you are not the intended
recipient, any dissemination, distribution, printing, copying or use is
strictly prohibited. The University does not warrant that this e-mail
and any attachments are secure and there is also a risk that it may be
corrupted in transmission. It is your responsibility to check any
attachments for viruses or defects before opening them. If you have
received this transmission in error, please contact us on +61 3 9214
8000 and delete it immediately from your system. We do not accept
liability in connection with computer virus, data corruption, delay,
interruption, unauthorised access or unauthorised amendment.
Please consider the environment before printing this email.
|
