Important Dates

Paper submission: 5 September 2018 Acceptance notification: 21 September 2018 Workshop: 11 November 2018

(program)

News

Introduction

Workshop program is now available- looking forward to see you in Dallas!

Reproducibility is critical for the scientific process. Sharing the artifacts, data, and workflows associated with papers forces authors to turn a more careful eye to their own work, and it enables other scientists to easily validate and build on prior work. Over the past five years, many top-tier parallel computing conferences have established Artifact Evaluation (AE) initiatives. The community has bought in, and nearly half of accepted papers now include artifacts.

Unfortunately, recent attempts to reproduce experimental results show that many challenges still remain (AE slides, ACM ReQuEST report). A lack of common tools, along with increasingly deep stacks of dependencies, lead to ad-hoc workflows, and evaluators struggle to install, run, and analyze experiments. These challenges are not unique to artifact evaluation. Users of production simulation codes struggle to reproduce complex workflows, even on the same machine. Benchmark suites are notoriously complex to configure and work with, and reproducing their performance can be a daunting task. Indeed, nearly all shared artifacts still require manual steps and human intuition, which ultimately makes them difficult to customize, port, reuse, and build upon.

ResCuE-HPC will bring together HPC researchers and practitioners to propose and discuss ways to enable reproducible, portable and customizable experimental workflows for HPC. We are interested in contributions that describe state-of-the-art and pitfalls for reproducibility, as well as improvements to existing frameworks, benchmarks and datasets that can be used to run HPC workloads across multiple software versions and hardware architectures. Ultimately, we aim to automate artifact evaluation, benchmarking, and workflows with a common co-design framework, and collaboratively solve reproducibility issues in HPC.

Topics of Interest

We invite position papers of up to 4 pages presenting novel or existing practical solutions to:

automate and unify artifact evaluation at HPC conferences;

share artifacts (workloads, benchmarks, data sets, models, tools), workflows and experiments in a portable, customizable, and reusable format;

automatically and natively install and rebuild all software dependencies required for shared experimental workflows on different machines and environments;

automatically report and visualize experimental results including interactive articles to assist reproducible initiative at SC and other conferences and journals;

continuously validate experiments from past research and report/record unexpected behavior (bugs, numerical instability, variation of empirical results such as execution time or energy measurements, etc) on new and evolving software and hardware stack;

establish open repositories of common benchmarks, data sets and tools to accelerate knowledge exchange between HPC centers;

enable universal, customizable and multi-objective auto-tuning and co-design of HPC software and hardware in a reproducible and reusable way;

unify statistical analysis and predictive modeling techniques to improve reproducibility of empirical experimental results.

Format

We also encourage submissions demonstrating practical use-cases of portable, customizable and reusable HPC workflows by connecting together existing tools including but not limited to Spack (see shared packages ), Collective Knowledge (CK) (see shared repositories with CK workflows packages ), EasyBuild (see shared packages ), Common Workflow Language and many others.

The day will be organized into sessions of 3-4 related papers. To spark discussion, Each author will briefly introduce their techniques (10-15 minutes), and this will be followed by an open panel including the audience.

There will be no formal proceedings for the first edition of this workshop! Instead, all ResCuE-HPC authors will be able to participate in preparation of a single ResCuE-HPC report. The report will focus on gradual convergence on a common experimental methodology, as well as possible formats for workflows and artifact sharing (meta information and API). We plan to make this report available to reproducibility and artifact evaluation chairs at the leading HPC, ML and systems conferences as well as the ACM task force on reproducibility where we are founding members.

We hope that ResCuE-HPC workshop will help the community to gradually converge on a common experimental methodology and possible formats for workflows and artifact sharing (meta information and API). This, in turn, should help our community unify and automate Artifact Evaluation, benchmarking, and workflows. The resulting ability to quickly prototype research ideas will dramatically accelerate development of the next generation of HPC software and hardware by reusing prior work.

Important dates

Paper submission: 24 August 2018

Acceptance notification: 21 September 2018

Workshop: 11 November 2018

Check SC18 home page for registration deadlines and other related information.

Submissions

Authors must submit their position papers (max 4 pages) using double-column, single-spaced letter format \documentclass[10pt,letterpaper,twocolumn]{article} as PDF files.

All papers should be submitted via SC18 submission website. Submissions are single-blind and will be peer-reviewed.

Questions

If you have any questions or comments, don't hesitate to contact workshop organizers!