A Google Brain research group has published a paper proposing a new method for evaluating generative models by using insights from competitive games between human players. Skill Rating for Generative Models uses Tournament Win Rate and Skill Rating to summarize outcomes between generators and discriminators.

To address the difficulty of evaluating generative models, the research team proposes a new framework that frames evaluation as a latent skill estimation problem through “multiplayer tournaments.” Each player in a tournament takes either the role of a discriminator trying to distinguish between real and fake data, or a generator trying to deceive discriminators into classifying fake data as real.

By observing tournament results, the model provides a novel comparison between various trained GANs. Such a tournament-based rating method also shows conceptual differences from previously developed evaluation approaches, which may prove complementary.

Renowned machine learning researcher and paper co-author Ian Goodfellow tweeted that latent skill level estimation seems “a promising new research direction” for evaluating generative models.