Artifact Evaluation

We give the opportunity to authors of accepted papers to submit an artifact establishing the reproducibility of the computational results described in their paper.

The objective of Artifact Evaluation (AE) is to reward efforts made by researchers to allow others to replicate their experiments. The AE enables to reproduce results, which helps researchers to build on top of each other's work and to compare results in a fair manner. The AE reviewers do not evaluate the correctness of the experimental results obtained using the artifact, only that these results can be reproduced.

The reviewing process is single-blind. Reviewers should be able to evaluate submitted artifacts using regular computing resources. Please feel free to contact the AE chair if you would like to submit an artifact for which this is not possible.

Submitted artifacts will be awarded badges as described in the ACM policy. See the program or accepted papers for papers with awarded badges.

  • Artifact submission deadline: TBA
  • Notification of acceptance: TBA

Specify the following fields:

  • Authors: same as for the (already accepted) paper,
  • Title: same as for the (already accepted) paper,
  • Keywords: use this field to specify the url at which we can find the information about the artifact and the artifact code. You should specify at least three keywords. Among these there must be a file containing instructions for the evaluators. If not all three lines are needed, fill three lines of keywords to enable the submission. A typical example:
    • instructions: url of instruction file (for instance a .pdf or .txt file);
    • code: link to the code (for instance a zip file or a repository);
    • vm: link to a virtual machine image (if any).
  • Paper: include the PDF version of your paper. The instruction file should specify which results in the paper are reproducible.

A nice HOWTO for preparing an artifact evaluation package is available online at http://bit.ly/HOWTO-AEC.

To submit an artifact, use the easychair submission website.

Here is a list of useful links related to artifact evaluation in other computer science research communities:

  • How to package an artifact,
  • About Reviewing,
  • Examples of accepted artifacts at other conferences.
  • TBA, affiliation, France

For any questions, concerns or comments, please contact the chair: ae-rtns2018@ensma.fr.