Image Credit: Yiying Lu, Copyright: Gradle Inc.

Gradle is a build automation tool that supports many programming languages. It coordinates the process of building software between multiple different code repositories and automates a number of important related tasks, like checking dependencies and warning programmers if something they have changed might break code written by someone else. It’s already a huge success by any standard. It’s the official build tool for Android Studio, and several leading tech companies rely on it every day as a key part of their build pipeline.

At LinkedIn, Gradle is a key piece of our approach to multiproduct development. But like many open source projects that we adopt at LinkedIn, we felt that we could contribute back to the community to help make it more useful for our developers.

One of the biggest areas of opportunity at LinkedIn was in the area of language support. Historically, Gradle has provided strong support for JVM languages. At LinkedIn, we have an extreme polyglot programming environment, supporting more than 42 programming languages. One of the most important languages we wanted to integrate with Gradle was Python, the language that we use to build the vast majority of our internal developer tools.

That’s why I’m excited to announce that LinkedIn is open sourcing the Gradle plugins that we have created for building Python libraries and applications.

Introducing {py}gradle

One of the most important aspects of ensuring the success of a programming language at any scale is being able to reliably build, package, and distribute easily. The Python programming language is backed by a mature community, so it is no surprise that there has been a strong emphasis on package management for many years. LinkedIn has leveraged the work done by this community and bridged a gap between two similar but different technologies, Setuptools and Gradle, to build a powerful, flexible, and reusable Python packaging system.

At the core of Python’s existing package management system resides Setuptools—a core library focused on giving project maintainers ways to manage dependencies, encode build steps, specify output distributions, and much more. For self-contained Python applications with a small set of external dependencies, it is a fantastic tool. However, Setuptools did not satisfy all our requirements as the organization's Python footprint grew.

At LinkedIn, these are some areas where Setuptools was challenging to leverage:

Dependency management: how do we specify our dependencies so that they’re explorable by external systems and teams, or so that we can do things like build automation around resolving conflicts and shepherding changes to a large number of products? Interfacing with existing metadata systems: how do we reuse the logic already created to power things at LinkedIn like code review, integration testing, deployment, and more? Polyglot builds: how do we build projects that are composed of multiple languages, such as when we have native C/C++ extension along side our Python code, or a frontend written in Ember.js, or a build system that has dependencies on non-Python tooling?

When we began our journey to address these concerns, we didn’t expect Setuptools alone to solve the problems that we were having. We also knew, however, that Setuptools should not be abandoned—it provides an immense amount of value. This is why we were careful to enhance rather than replace the existing and idiomatic Python package management ecosystem with Gradle. From the beginning, it was important for us to not tread on Python. By enhancing the Python package management system with a set of Gradle plugins, we were able to address each of Setuptools’s shortcomings while still reusing the build logic that was written for other languages. With a small amount of effort, we were able to wire in the Python package management ecosystem with LinkedIn’s rich Gradle build system, which we were already using with other languages to achieve continuous integration.

We’ve tried to make our {py}gradle plugins as idiomatic to use as possible. If you are familiar with Setuptools, using {py}gradle should be straightforward. For most of the important keyword arguments passed to the Setuptools class, we have a corresponding Gradle attribute in the build.gradle file that you can use. And, as a general principle moving forward, we’ll continue adding support for more of Setuptools’s features in this way.

Python + Gradle

A {py}gradle project looks nearly identical to a Python project that uses Setuptools. The structure looks something like the following, but may be different depending on your use of Gradle. Curious readers can familiarize themselves with more of Gradle’s core concepts by reading through their documentation.