** Apologies for cross postings. Please distribute as appropriate **
===============================================================================
Workshop on Gathering Efficient Assessments of Relevance (GEAR): Call for papers
At SIGIR 2014, July 11th 2014, Gold Coast, Australia
===============================================================================
*** Deadline for papers extended to the 16th May 2014 ***
Workshop website: https://sites.google.com/site/sigirgear
CALL FOR PAPERS
---------------
Evaluation is a fundamental part of Information Retrieval, and in the
conventional Cranfield evaluation paradigm, sets of relevance
assessments are a fundamental part of test collections. In this
workshop, we wish to revisit how relevance assessments can be
efficiently created. A discussion and exploration of this issue will
be facilitated through the presentation of results based and position
papers on the topic. Participants will also be invited to participate
in a design task focusing on developing a benchmarking exercise. After
all of these activities the workshop will conclude with an open
discussion session, part of which will focus on future directions. As
well as providing a forum for discussion and exploration, it is hoped
that this workshop will lead to future activities and collaborations
in this area.
Workshop areas of interest include:
- How the method of generating assessments, via conventional means or
crowdsourcing, affects the judgments gathered, such as issues of
assessor expertise, payment, etc.
- The process by which individuals, or groups of individuals, assess
documents (text, image, video etc.) for relevance
- Issues relating to the effort required to generate relevance
assessments for different types of topic, and different types of
material (text, web, image, video, etc. and multiple languages)
- To revisit the concept of "relevance", from a practical,
operational standpoint, for the purposes of IR evaluation.
An aim of the workshop will be to provide a forum where short research
papers can be presented, reporting work which may not conventionally
be published in papers at formal venues, including "practice and
experience" papers concerning relevance assessment gathering, and
position papers concerning the concepts and issues. Papers reporting
negative results are also welcome.
Important dates
---------------
Submission deadline (*extended*): May 16th, 2014
Workshop date: July 11th, 2014
Note: all deadlines are on the date specified at midnight in the
UTC-12 time zone.
Paper Submissions
-----------------
The workshop welcomes short papers (4 pages) and position papers (2 pages)
Two types of paper are solicited:
- Short papers: 4 pages max, which present research work related to
the workshop's areas of interest
- Position papers: 2 pages max, which present provocative ideas and
challenges related to workshop's areas of interest.
Submissions should be in ACM SIG proceedings format
(http://www.acm.org/sigs/publications/proceedings-templates), and
should be submitted through the EasyChair system at
https://www.easychair.org/conferences/?conf=gear20141.
Further information is available on the workshop website:
https://sites.google.com/site/sigirgear/schedule
Workshop Organisers
-------------------
- Martin Halvey, School of Engineering and Built Environment, Glasgow
Caledonian University, UK
- Robert Villa, Information School, University of Sheffield, UK
- Paul Clough, Information School, University of Sheffield, UK
--
Dr. Robert Villa
Information School
University of Sheffield
Regent Court
Sheffield S1 4DP
Telephone: +44 (0) 114 222 2683
Email: [log in to unmask]
|