Common Workflow Language §

The Common Workflow Language (CWL) is an open standard for describing analysis workflows and tools in a way that makes them portable and scalable across a variety of software and hardware environments, from workstations to cluster, cloud, and high performance computing (HPC) environments. CWL is designed to meet the needs of data-intensive science, such as Bioinformatics, Medical Imaging, Astronomy, High Energy Physics, and Machine Learning.

CWL is developed by a multi-vendor working group consisting of organizations and individuals aiming to enable scientists to share data analysis workflows. The CWL project is maintained on Github and we follow the Open-Stand.org principles for collaborative open standards development. Legally CWL is a member project of Software Freedom Conservancy and is formally managed by the elected CWL leadership team, however every-day project decisions are made by the CWL community which is open for participation by anyone.

Getting Started §

The CWL user guide provides a gentle introduction to learning how to write CWL command line tool and workflow descriptions.

Browse CWL Implementations to find a software package that's right for you.

CWL Recommended Practices

CWLの日本語での解説ドキュメント is a 15 minute introduction to the CWL project in Japanese.

A series of video lessons about CWL is available in Russian as part of the Управление вычислениями(Computation Management) free online course.

Support, Community and Contributing §

The recommended place to ask a question about all things CWL is on the CWL Discourse Group. Previously we used biostars.org where you can still read older responses.

If you are interested in learning more or contributing ideas or code, come chat with us on Gitter, check out #CommonWL on Twitter, join the mailing list common-workflow-language on Google Groups or fork the repository and send a pull request!

Besides the web interface for the mailing list, one can also join by sending a blank email to common-workflow-language+subscribe@googlegroups.com and replying to the automated message.

Code of Conduct §

The CWL Project is dedicated to providing a harassment-free experience for everyone, regardless of gender, gender identity and expression, sexual orientation, disability, physical appearance, body size, age, race, or religion. We do not tolerate harassment of participants in any form. This code of conduct applies to all CWL Project spaces, including the Google Group, the Gitter chat room, the Google Hangouts chats, both online and off. Anyone who violates this code of conduct may be sanctioned or expelled from these spaces at the discretion of the leadership team.

For more details, see our Code of Conduct.

For developers and advanced users, the current CWL Standards v1.2.0 provides authoritative documentation of the execution of CWL documents. The previous versions, CWL Standards v1.0.2 and CWL Standards v1.1.0, are also available.

To reference the CWL standards in a scholary work, please use the following citation inclusive of the DOI:

Peter Amstutz, Michael R. Crusoe, Nebojša Tijanić (editors), Brad Chapman, John Chilton, Michael Heuer, Andrey Kartashov, Dan Leehr, Hervé Ménager, Maya Nedeljkovich, Matt Scales, Stian Soiland-Reyes, Luka Stojanovic (2016): Common Workflow Language, v1.0. Specification, Common Workflow Language working group. https://w3id.org/cwl/v1.0/ doi:10.6084/m9.figshare.3115156.v2

A collection of existing references to CWL can be found at https://zotero.org/groups/cwl

In Production:

Software Description Platform support cwltool Reference implementation of CWL Linux, OS X, Windows, local execution only Arvados Distributed computing platform for data analysis on massive data sets. Using CWL on Arvados AWS, GCP, Azure, Slurm Toil Toil is a workflow engine entirely written in Python. AWS, Azure, GCP, Grid Engine, HTCondor, LSF, Mesos, OpenStack, Slurm, PBS/Torque CWL-Airflow Package to run CWL workflows in Apache-Airflow (supported by BioWardrobe Team, CCHMC) Linux, OS X REANA RE usable ANAlyses Kubernetes, CERN OpenStack (OpenStack Magnum)

Partial implementations:

Repository Description Common Workflow Library Git organization for community maintenance of tools and workflows. Dockstore tool registry An open platform for sharing Docker-based tools described with the Common Workflow Language used by the GA4GH. CWLviewer A web application to view and share Common Workflow Language workflows cwl-source Git repository for collections of tools, workflows, metadata, and input parameter files. Administered by xD Bio Inc. Integrates with truwl.com GitHub Search for CWL documents using extension:cwl cwlVersion + <your search terms> , for example extension:cwl cwlVersion picard . Google Search for CWL documents using filetype:cwl cwlVersion + <your search terms> , for example filetype:cwl cwlVersion picard . Workflow Hub A registry for scientific workflows, sponsored by EOSC Life

Software for working with CWL §

Editors and viewers §

Software Description cwldep CWL dependency manager, for importing tool wrappers and workflows into your own project. cwltest CWL testing framework, automated testing of tools and workflows written with CWL cwl2zshcomp generates ZSH auto completions from CWL command line tool descriptions Cerise A REST service for running CWL workflows on remote clusters workflow-service An implementation of the GA4GH Workflow Execution Service (WES) API to run CWL workflows on remote clusters cwl-inspector Tool to inspect properties of tools or workflows written in CWL cwlprov-py Command line tool and Python API to explore CWLProv Research Objects containing provenance of Common Workflow Language executions cwl-utils Example scripts using the new Python 3.6+ CWL parsing library

Converters and code generators §

Software Description cwl-upgrader Upgrade CWL documents from draft-3 to v1.0 argparse2tool Generate CWL CommandLineTool wrappers (and/or Galaxy tool descriptions) from Python programs that use argparse. Also supports the click argument parser. cwl2argparse Generate Python argparse code from CWL CommandLineTool description. pypi2cwl Automatically run argparse2cwl on any package in PyPi acd2cwl ACD (EMBOSS) to CWL generator CTD converter Common Tool Definition (CTD) to CWL converter scriptcwl Create CWL workflows by writing a simple Python script cwl-to-parsl Convert CWL to Parsl Beatrice Pipeline Assembler For CWL zatsu-cwl-generator A simple CWL document generator from given execution commands Janis A Python API that generates portable CWL and WDL workflows cwl-utils New Python 3.6+ CWL parsing library

Code libraries §

Projects the CWL community is participating in §

Contributers and Governance §

Participating Organizations §

Individual Contributors §

(Alphabetical)

Peter Amstutz, Curii / Arvados Project; https://orcid.org/0000-0003-3566-7705

Robin Andeer; https://orcid.org/0000-0003-1132-5305

Brad Chapman; https://orcid.org/0000-0002-3026-1856

John Chilton, Pennsylvania State University / Galaxy Project; https://orcid.org/0000-0002-6794-0756

Michael R. Crusoe, CWL Project Lead; https://orcid.org/0000-0002-2961-9670

Roman Valls Guimerà; https://orcid.org/0000-0002-0034-9697

Guillermo Carrasco Hernandez guille.ch.88@gmail.com

Kenzo-Hugo Hillion; https://orcid.org/0000-0002-6517-6934

Manabu Ishii, RIKEN; https://orcid.org/0000-0002-5843-4712

Sinisa Ivkovic sinisa.ivkovic@sbgenomics.com

Sehrish Kanwal; https://orcid.org/0000-0002-5044-4692

Andrey Kartashov; https://orcid.org/0000-0001-9102-5681

John Kern; https://orcid.org/0000-0001-6977-458X

Farah Zaib Khan; https://orcid.org/0000-0002-6337-3037

Dan Leehr; https://orcid.org/0000-0003-3221-9579

Hervé Ménager, Institut Pasteur; https://orcid.org/0000-0002-7552-1009

Maxim Mikheev mikhmv@biodatomics.com

Michael Miller mdmiller53@comcast.net

Tazro Ohta, DBCLS; http://orcid.org/0000-0003-3777-5945

Tim Pierce twp@unchi.org

Josh Randall; https://orcid.org/0000-0003-1540-203X

Mark Robinson; https://orcid.org/0000-0002-8184-7507

Janko Simonović janko.simonovic@sbgenomics.com

Stian Soiland-Reyes, University of Manchester; https://orcid.org/0000-0001-9842-9718

Luka Stojanovic luka.stojanovic@sbgenomics.com

Tomoya Tanjo, NII; https://orcid.org/0000-0002-4421-9659

Nebojša Tijanić nebojsa.tijanic@sbgenomics.com

Hiromu Ochiai; @otiai10 https://orcid.org/0000-0001-6636-856X

CWL Advisors §

(Alphabetical)

Peter Amstutz, Curii / Arvados Project; https://orcid.org/0000-0003-3566-7705

Artem Barski, Cincinnati Children's Hospital Medical Center / University of Cincinnati College of Medicine; https://orcid.org/0000-0002-1861-5316

John Chilton, Pennsylvania State University / Galaxy Project; https://orcid.org/0000-0002-6794-0756

Kyle Cranmer, New York University; https://orcid.org/0000-0002-5769-7094

Michael R. Crusoe, CWL Project Lead; https://orcid.org/0000-0002-2961-9670

Brandi Davis Dusenbery, Seven Bridges Genomics, Inc.; https://orcid.org/0000-0001-7811-8613

Niels Drost, Netherland eScience Center; https://orcid.org/0000-0001-9795-7981

Scott Edmunds, GigaScience; https://orcid.org/0000-0001-6444-1436

Geet Duggal, DNAnexus; https://orcid.org/0000-0003-3485-359X

Rob Finn, EMBL-EBI; https://orcid.org/0000-0001-8626-2148

Marc Fiume, DNAstack; https://orcid.org/0000-0002-9769-375X

Jeff Gentry, Foundation Medicine; https://orcid.org/0000-0001-5351-8442

Kaushik Ghose, Seven Bridges Genomics, Inc; https://orcid.org/0000-0003-2933-1260

Carole Goble, The University of Manchester; https://orcid.org/0000-0003-1219-2137

Oliver Hofmann, University of Melbourne / bcbio-nextgen; https://orcid.org/0000-0002-7738-1513

Hervé Ménager, Institut Pasteur; https://orcid.org/0000-0002-7552-1009

Folker Meyer, Argonne / University of Chicago; https://orcid.org/0000-0003-1112-2284

Tom Morris, Curoverse; https://orcid.org/0000-0003-0435-7851

Anton Nekrutenko, The Pennsylvania State University / Galaxy Project; https://orcid.org/0000-0002-5987-8032

Brian O'Connor, University of California Santa Cruz; https://orcid.org/0000-0002-7681-6415

Tibor Simko, CERN, https://orcid.org/0000-0001-7202-5803

Nihar Sheth, DNAnexus; https://orcid.org/0000-0003-4128-4364

Stian Soiland-Reyes, University of Manchester; https://orcid.org/0000-0001-9842-9718

James Taylor, Johns Hopkins University / Galaxy Project; https://orcid.org/0000-0001-5079-840X

Nebojša Tijanić, Seven Bridges

Ward Vandewege, Curoverse Inc. / Arvados; https://orcid.org/0000-0002-2527-6949

Alexander Wait Zaranek, Curoverse Inc. / Arvados; https://orcid.org/0000-0002-0415-9655

CWL Leadership Team §

(Alphabetical)