Reproducible papers with artifacts

News

• Upcoming Artifact Evaluation: ASPLOS 2022.
• ACM TechTalk (video) about artifact evaluation (challenges and solutions).
• Artifact Evaluation at MICRO 2021.
• The report from the "Workflows Community Summit: Bringing the Scientific Workflows Community Together" is available in ArXiv.
• Artifact Evaluation: reproducing papers at ASPLOS 2021 (the list of accepted artifacts).
• The paper about automating artifact evaluation has appeared in the Philosophical Transactions A, the world's longest-running journal where Newton published: DOI, ArXiv.
• cTuning foundation is honored to join MLCommons as a founding member to accelerate machine learning innovation and help with best practices, reproducible benchmarking and workflow automation along with 50+ leading companies and universities: press release.
• Reddit discussion about reproducing ML and systems papers.
• Artifact Evaluation: reproducing papers at MLSys 2020 (the list of accepted artifacts).
• Artifact Evaluation: reproducing papers at ACM ASPLOS 2020 (the list of accepted artifacts).
• Building an open repository with reproduced papers, portable workflows and reusable artifacts: cKnowledge.io.
• Working on a common methodology to share research artifacts (code, data, models) at systems and ML conferences: ACM/cTuning.
• All reproduced papers.
• All our prior reproducibility initiatives with shared artifacts.

Motivation

Related initiatives:

• NISO RP-31-2021, Reproducibility Badging and Definitions
• An Open knowledge portal about complex compuational systems (AI, ML, quantum, IoT) with public SOTA scoreboards and reproduced papers;
• SIGPLAN's checklist for empirical evaluation;
• The Machine Learning Reproducibility Checklist (NeurIPS);
• The NASEM report on Reproducibility and Replicability in Science: Online PDF and a BoF at SC19 organized and chaired by Lorena A. Barba;
• HOWTO for AEC Submitters (Dan Borowy, Charlie Cursinger, Emma Tosch, John Vilk, and Emery Berger)
• A Portable workflow to automate the SeiSol application from the Supercomputing'18 Student Cluster Competition reproducibility challenge and the SCC automation workflow;
• PPoPP'19 HotCRP configuration for artifact evaluation;
• Reproducible optimization tournaments on AI/ML/SW/HW co-design and on quantum computing challenges;
• Artifact Evaluation google group.

Sponsors and supporters