Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks

In ICCV 2017

Abstract

Image-to-image translation is a class of vision and graphics problems where the goal is to learn the mapping between an input image and an output image using a training set of aligned image pairs. However, for many tasks, paired training data will not be available. We present an approach for learning to translate an image from a source domain X to a target domain Y in the absence of paired examples. Our goal is to learn a mapping G: X → Y, such that the distribution of images from G(X) is indistinguishable from the distribution Y using an adversarial loss. Because this mapping is highly under-constrained, we couple it with an inverse mapping F: Y → X and introduce a cycle consistency loss to push F(G(X)) ≈ X (and vice versa). Qualitative results are presented on several tasks where paired training data does not exist, including collection style transfer, object transfiguration, season transfer, photo enhancement, etc. Quantitative comparisons against several prior methods demonstrate the superiority of our approach.

Paper

arxiv 1703.10593, 2017.

Citation

Jun-Yan Zhu*, Taesung Park*, Phillip Isola, and Alexei A. Efros. "Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks", in IEEE International Conference on Computer Vision (ICCV), 2017.

(* indicates equal contributions) Bibtex

If you have questions about our PyTorch code, please check out model training/test tips and frequently asked questions.

Course

CycleGAN course assignment code and handout designed by Prof. Roger Grosse for "Intro to Neural Networks and Machine Learning" at University of Toronto. Please contact the instructor if you would like to adopt this assignment in your course.

Other Implementations

Tensorflow (Harry Yang) | Tensorflow (Archit Rathore) | Tensorflow (Van Huy) | Tensorflow (Xiaowei Hu) | TensorLayer (luoxier)

Tensorflow-simple (Zhenliang He) | Chainer (Yanghua Jin) | Minimal PyTorch (yunjey) | Mxnet (Ldpe2G) | lasagne/keras (tjwei)

ICCV Spotlight Talk

Expository Articles and Videos

Two minute papers Karoly Zsolnai-Feher made the above as part of his very cool "Two minute papers" series. Understanding and Implementing CycleGAN Nice explanation by Hardik Bansal and Archit Rathore, with Tensorflow code documentation.

Creative Applications of CycleGAN

Researchers, developers and artists have tried our code on various image manipulation and artistic creatiion tasks. Here we highlight a few of the many compelling examples. Search CycleGAN in Twitter for more applications.

How to interpret CycleGAN results: CycleGAN, as well as any GAN-based method, is fundamentally hallucinating part of the content it creates. Its outputs are predictions of "what might it look like if ..." and the predictions, thought plausible, may largely differ from the ground truth. CycleGAN should only be used with great care and calibration in domains where critical decisions are to be taken based on its output. This is especially true in medical applications, such as translating MRI to CT data. Just as CycleGAN may add fanciful clouds to a sky to make it look like it was painted by Van Gogh, it may add tumors in medical images where none exist, or remove those that do. More information on dangers like this can be found in Cohen et al.

Popular Press

Applications in our Paper

Driving Applications (CG → Real and Day → Night ) Translation between driving scenes in different style. Each frame was rendered independently. Between Cityscapes and GTA dataset Between Day and Night driving using the Berkeley Deep Drive dataset (not public yet) The GTA → Cityscapes results of CycleGAN can be used for domain adaptation for segmentation.

A segmentation model trained on the Cityscapes-style GTA images yields mIoU of 37.0 on the segmentation task on Cityscapes.

More information can be found at Cycada.

You can download the original GTA images (18GB) and the translated Cityscapes-style GTA images (16GB).

Experiments and comparisons

Failure Cases