RV-CuBES

An International Workshop on Competitions, Usability, Benchmarks, Evaluation, and Standardisation for Runtime Verification Tools

Held in conjunction with the 17th International Conference on Runtime Verification (RV 2017)

The goal of this workshop is to provide a venue to discuss ongoing efforts to improve how we evaluate and compare tools for runtime verification. This workshop invites submissions that contribute to this discussion (see below). The workshop replaces the Runtime Verification Competition this year, giving an opportunity for reflection and planning for the future.

HIGHLIGHTS

• We invite two kinds of submissions: tool overview papers of existing tools and position papers
• Attendance at the workshop is not compulsory for submission, but encouraged
• The workshop will be integrated into RV 2017 to engage with the wider RV community

BACKGROUND

Over the last three years, beginning in 2014, the Competition on Runtime Verification (CRV) has compared 14 different runtime verification tools using over 100 different benchmarks. It has motivated the development of new tools and extensions of existing ones. Thanks to the time and effort of the various organisers and participants it has successfully provided a platform to discuss how we evaluate and compare our tools. However, we are aware that the competition has not served all members of the community and required significant effort from its participants and therefore we have decided not to run the competition in the same form in 2017.

This workshop has two aims.

• Firstly, to kickstart an online repository of tool descriptions that focus on usability, rather than implementation. Much effort within Runtime Verification is spent developing tools. The planned repository will help advertise these tools within and beyond the community and to document their existence. To this end, this workshop invites tool overview papers (details below) that present an opportunity to demonstrate the diverse range of available RV tools and to advertise their advanced features and capabilities. 
• Secondly, to provide a forum in which the future of the competition, and wider efforts to improve how we evaluate RV tools, can be discussed in a structured and informed way. There have been various informal discussions at the previous three RV conferences and in other settings. By inviting position papers (details below) this workshop aims to establish the issues and possible directions before holding a structured panel session during RV 2017.

RELATION TO RV 2017

There is no overlap in scope between this workshop and RV 2017. Any papers containing original technical developments should be submitted to RV 2017 rather than this workshop as the workshop focuses on tool reviews (containing existing work) and position statements. If there is any uncertainty please contact the workshop chairs.

The workshop will be integrated into RV 2017. The workshop activities will include a poster presentation session and a panel decision, both of which will be scheduled within the main conference program. A consequence of this is that there will be no separate registration for the workshop. To attend the workshop activities it will be necessary to attend RV 2017. Although, attendance is not a requirement for submission.

SUBMISSIONS

We invite two forms of submission: Tool Overview papers and Position papers. All submissions will be subject to a lightweight review by the PC to ensure a reasonable standard and to provide constructive feedback to improve the quality of the submission.

Tool Overview Papers

These should describe an existing tool using a minimum of 5 pages. The paper should at least describe how to obtain the tool, the RV problem the tool is aiming to solve, the key defining features of the tool and relevant references. Additionally, we might expect it to include some of the following:

• The history of the tool
• High-level overviews of key aspect such as the input language or architecture from a user’s perspective
• Usability details undocumented elsewhere
• Examples demonstrating key features
• Details of case studies or applications to real world problems
• Discussion of features particular to the tool
• Summaries (rather than detailed tables) of experimental results
• Analysis of the kinds of problem the tool is suited for and those it is less-well suited for

Ideally, the tool and related material (e.g. benchmarks) will be available online and linked to in the submission. The paper should not cover any significant new contributions (these can be submitted as tool papers to RV 2017) and should rely on previous research and tool papers to provide further details. Note that there is no upper page limit however the number of pages used should reflect the level of detail given. Submissions of 2 pages giving minimal details would be suitable in a situation where the tool is well documented elsewhere but an entry in the repository is still desired. All accepted submissions will be invited for presentation at a special Poster session during RV 2017.

Position papers

Initial submissions should use a minimum of 2 pages to explore a particular position related to the evaluation, comparison or standardisation of Runtime Verification tools (and benchmarks). Topics may include, but are not limited to:

• What should a RV benchmark look like?
• Can we have a common specification language for RV? If so, what should it look like?
• Is execution time the most important performance criteria? What might be more important?
• How can we evaluate hardware monitoring tools?
• What are we doing wrong in evaluation? Can we fix this?
• What can be borrowed form other communities?

A selection of position papers will be used to structure a discussion panel to be held at RV 2017. Again, there is no upper page limit however the number of pages used should reflect the level of detail given. There will be an opportunity to update the paper based on discussions before inclusion in post-proceedings (see below).

Publishing and Submission

Contributions will be published as a post-proceedings volume in the Open-Access Scopus-Indexed EasyChair Kalpa Series. To be eligible for inclusion in the post-proceedings, papers should be at least 5 pages. Please see the relevant information for authors when preparing your paper.

Please submit contributions to the following EasyChair page.

https://easychair.org/conferences/?conf=rvcubes2017

DATES

•  Submission Deadline 1 July 2017
  • Abstracts: 8 July 2017
  • Final Submission: 15 July 2017
• Notification 1 August 2017
• Workshop at RV 2017 13-16 September 2017
  
• Post-proceedings deadline 14 October 2017

ORGANISATION

For local and general organisation please see RV 2017.

Program Committee Chairs

Giles Reger, University of Manchester, UK
Klaus Havelund, NASA Jet Propulsion Laboratory, USA

Program Committee

Ezio Bartocci, TU Wien, Austria
Domenico Bianculli, SnT Centre – University of Luxembourg, Luxembourg
Borzoo Bonakdarpour, McMaster University, Cananda
Christian Colombo, University of Malta, Malta
Ylies Falcone, Univ. Grenoble Alpes, Inria, Laboratoire d’Informatique de Grenoble, France
Adrian Francalanza, University of Malta, Malta
Sylvain Hallé, Université du Québec à Chicoutimi, Canada
Felix Klaedtke, NEC Europe Ltd.
Daniel Thoma, University of Lübeck, Germany
Dmitriy Traytel, ETH Zürich, Switzerland
Gordon Pace, University of Malta, Malta
Cesar Sanchez, IMDEA Software Institute, Spain
Leonardo Mariani, University of Milano Bicocca, Italy
Julien Signoles, CEA LIST, France
Tarmo Uustalu, Tallin University of Technology, Estonia