------------------------------------------------------------------------
Special Issue on
Reproducibility in Evolutionary Computation
Evolutionary Computation Journal, MIT Press
https://direct.mit.edu/evco
DEADLINE: November 30, 2021
------------------------------------------------------------------------
Guest Editors:
* Mike Preuss, Universiteit Leiden, The Netherlands, [log in to unmask]
* Luís Paquete, University of Coimbra, Portugal, [log in to unmask]
Associate Editor:
* Manuel López-Ibáñez, University of Málaga, Spain, [log in to unmask]
Description
-----------
Experimental research is crucial in Evolutionary Computation. The scientific
method requires that empirical results are reproducible by the authors
themselves and replicable by others. Computer Science in general, and
Evolutionary Computation in particular, show signs of a reproducibility crisis
despite their digital underpinnings. Interest in improving reproducibility in
Computer Science and other empirical sciences has grown in recent years and
there is a growing number of works analysing current and best practices,
obstacles and guidelines, effectiveness of journal policies, etc.
Reproducibility issues in the context of Evolutionary Computation have been a
topic of discussion for a long time in the context of best practices for
empirical research, but there are few studies analysing reproducibility in EC
research and reproducibility studies themselves are extremely rare. There is
room for improvement to attain the minimum standards for reproducibility
encouraged in other scientific fields. Challenges for reproducibility in EC
research arise from the stochastic nature of the algorithms and, sometimes, the
problems, which requires multiple runs to analyse expected behavior and
variance; sensitivity of the results to the computational environment,
parameter settings or implementation details; and the generalizability of
conclusions to different instances of the same or related problems.
The aim of this special issue is to encourage research that analyses the topic
of reproducibility in EC and showcase excellent examples of both reproducible
research and reproducibility studies. Within the context of EC, we include
non-evolutionary metaheuristics, swarm intelligence methods, stochastic local
search, and hybrids with exact methods, i.e., matheuristics. We welcome papers
that deal with the topic of reproducibility with a specific focus on the EC
context, either by analysing the current state of the field or providing
evidence that supports best practices for authors, journals or funding bodies.
Furthermore, we also welcome papers that, in addition to an original research
contribution to the field of EC, go well above the current standards of
reproducibility. Finally, we also welcome high-quality reproducibility studies
that attempt to reproduce (by using the materials provided by the original
authors) and/or replicate (by reimplementing the materials from scratch)
previously published results of interest.
Submission Guidelines
---------------------
We will accept four different types of contributions:
1. Original and significant empirical research submissions following the
usual requirements and scientific quality of works published by ECJ with the
added contribution of raising the bar in terms of reproducibility of the
results by, satisfying or going beyond the reproducibility requirements
discussed below.
2. Reproducibility studies that confirm, contradict (fail to reproduce) or
widen the scope of (generalize) previously published experimental results. The
paper that is chosen to be reproduced must present results that are of
particular interest to the EC community (for example indicated by citations,
studies use of common benchmarks, use of well-known algorithms) and must have
been previously published in a reputable venue. Acceptable confirmation
studies must extend the previous analysis in a significant manner. Studies that
contradict previous results must make a serious attempt at analysing the
reasons for the different results and provide convincing enough evidence that
the new results are correct. In all cases, submissions must follow the minimum
reproducibility requirements discussed below. If you intend to submit a paper
in this category, please contact the Guest Editors before you embark on the
study who can advise if the study is appropriate.
3. Empirical research on the topic of reproducibility in the context of EC,
including but not limited to the empirical analysis of:
(a) reproducibility aspects in EC in contrast to other fields;
(b) cultural and technical challenges for reproducibility in EC and how to
overcome them; and
(c) the effectiveness of journal and funding bodies policies.
4. Methodological contributions on the topic of reproducibility in the
context of EC, including but not limited to improving reproducibility, reducing
reproducibility effort, and the design and evaluation of reproducibility
studies. We would accept two types of contributions:
(a) Submissions that describe software aimed at improving reproducibility
in EC in original ways. These submission must follow the guidelines for
Software Articles
(https://direct.mit.edu/evco/pages/submission-guidelines#software).
(b) Position papers that do not provide empirical results but make original
methodological contributions. Submissions must go beyond what has been already
proposed in the literature.
Reproducibility Guidelines
--------------------------
In addition to the usual requirements for submissions to Evolutionary
Computation Journal
(https://direct.mit.edu/evco/pages/submission-guidelines#full), all submissions
containing empirical research must adhere to the following minimum requirements
to be considered for this special issue:
1. A hyperlink to a permanent repository containing the "artifacts" required
to reproduce the experiments in the paper. Zenodo, Figshare, OSF and
institutional repositories with a declared plan for permanent accessibility of
a precise version of the artifacts are acceptable. Dropbox, Google Drive,
personal and institutional webpages without a declared plan for permanent
accessibility are not appropriate. We will use OPENDOAR
(https://v2.sherpa.ac.uk/opendoar/) to evaluate the accessibility of data
repositories.
2. Artifacts must contain:
(a) Pre-processing code, e.g. code that generates instance data and
scripts that set up the experimental conditions.
(b) Algorithm code, the implementation of the algorithm(s) to be tested.
(c) Analysis code, scripts that post-process the data produced by the
algorithm and perform statistical analysis.
(d) Presentation code, e.g. scripts that generate tables and figures
reported in the article.
A README.txt file should be included that explains how to reproduce every
result reported in the article. Experimental conditions should be provided with
as much detail as possible. The process used for configuring algorithmic
parameters should also be reproducible. For randomized algorithms, random
seeds should also be provided for allowing the exact repetition of an
experiment. Should exact repetition not be possible due to the complexity of
the code, it should be explained in the README.txt to what extent and why this
is not possible.
Submission
---------
Authors should submit their manuscripts to the Evolutionary Computation
Editorial Manager (https://www.editorialmanager.com/ecj/default.aspx). Authors
must select "Special Issue: Reproducibility" as the article type when submitting.
When submitting a paper, please send also an email to Manuel López-Ibáñez
([log in to unmask]) mentioning the special issue, the paper id, title,
and author list to inform us about the submission.
--
Dr Manuel López-Ibáñez | Senior Lecturer (Assoc Prof) at the Decision and
Cognitive Sciences Research Centre | Alliance Manchester Business School | The
University of Manchester, UK | http://lopez-ibanez.eu
------------------------------------------------------------------------------
MSc Business Analytics: Operations Research and Risk Analysis
(QS Ranking: 2nd in UK and 9th in the world, https://bit.ly/3wfYjSL)
------------------------------------------------------------------------------
11th Workshop on Evolutionary Computation for the Automated Design of
Algorithms (ECADA): https://bonsai.auburn.edu/ecada/GECCO2021
------------------------------------------------------------------------------
Student appointments via https://calendly.com/manuel-lopez-ibanez
------------------------------------------------------------------------------
########################################################################
To unsubscribe from the EVOLUTIONARY-COMPUTING list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=EVOLUTIONARY-COMPUTING&A=1
This message was issued to members of www.jiscmail.ac.uk/EVOLUTIONARY-COMPUTING, a mailing list hosted by www.jiscmail.ac.uk, terms & conditions are available at https://www.jiscmail.ac.uk/policyandsecurity/
|
