Testing is a critical activity in all software projects, but one that is often neglected in data pipelines. The complexities introduced by the inherent statefulness of the problem domain and the interdependencies between systems contribute to make pipeline testing difficult to manage. To make this endeavor more manageable Abe Gong and James Campbell have created Great Expectations. In this episode they discuss how you can use the project to create tests in the exploratory phase of building a pipeline and leverage those to monitor your systems in production. They also discussed how Great Expectations works, the difficulties associated with pipeline testing and managing associated technical debt, and their future plans for the project.

Do you want to try out some of the tools and applications that you heard about on Podcast.__init__? Do you have a side project that you want to share with the world? With Linode’s managed Kubernetes platform it’s now even easier to get started with the latest in cloud technologies. With the combined power of the leading container orchestrator and the speed and reliability of Linode’s object storage, node balancers, block storage, and dedicated CPU or GPU instances, you’ve got everything you need to scale up. Go to pythonpodcast.com/linode today and get a $60 credit to launch a new cluster, run a server, upload some data, or… And don’t forget to thank them for being a long time supporter of Podcast.__init__!

GoCD is the on-premise open source continuous delivery server created by ThoughtWorks and modeled after the ideas in the Continuous Delivery book by Jez Humble and David Farley.

With GoCD’s comprehensive pipeline modeling, you can model complex workflows for multiple teams with ease. And GoCD’s Value Stream Map lets you track a change from commit to deploy at a glance.

GoCD’s real power is in the visibility it provides over your end-to-end workflow. So you get complete control of and visibility into your deployments, across multiple teams.

Say goodbye to deployment panic and hello to consistent, predictable deliveries.

To learn more about GoCD, visit gocd.org for a free download. Professional Support and enterprise add-ons, including disaster recovery, are available.

When your website experiences an error, Airbrake alerts you in real-time, and gives you all the details you need to fix the bug fast. Some of the most useful tools that Airbrake provides for faster resolution are:

Exception aggregation to understand the how many users are being affected, so that you can prioritize the work you are doing to have the biggest impact

Contextual information to understand how certain states of the application contribute to the exception being raised, and which environments are affected

Deployment tracking so that you can easily see whether a new feature is the source of an error

Integration with all of the other tools that you use, such as automatically creating issues in GitHub and linking to the line of code where the error came from

Right now, Podcast.__init__ listeners can try Airbrake free for 30 days, plus get 50% off the first 3 months on the Startup plan. To get started, visit airbrake.com/podcastinit today.