Note: This post was originally published on Mar. 22, 2018 and was updated on Oct. 17, 2018 to reflect changes in TestFlight and Google Play.

When it’s time to choose the tools to help you carry out your beta test, a beta app distribution tool is one of the first things you need to consider. Beta distribution platforms allow you to distribute the beta version of your app to your testers over-the-air (OTA) away from the public app store.

There are a lot of different beta distribution tools on the market offering different features and pricing models. In this post, we will take a look at some of the top beta distribution platforms and their key features.

TestFlight

Apple acquired TestFlight in 2014, integrating it into iTunes Connect and adopting it as their solution for beta app distribution. Naturally, TestFlight exclusively supports the iOS, watchOS, and tvOS platforms.

TestFlight allows you to add up to 25 internal testers from your iTunes Connect users as long as they have either the Admin, App Manager, Developer, Marketer, or Technical role. As for external testers, you can add up to 10,000 testers using their emails. Testers receive an email with a link that prompts them to download the TestFlight app. Testers can then use the TestFlight app to download your beta app, and they receive in-app notifications when it’s time to update. Additionally, TestFlight supports open beta testing by generating a public opt-in link to distribute your app to testers. You can also set a maximum number of allowed testers for your open beta test to avoid exceeding your team’s capacity.

TestFlight supports testing of up to 100 apps simultaneously. Builds are available for 90 days after upload, and internal testers have access to all the builds of your apps. You can choose which apps and builds to make available to external testers and organize these testers into different groups. This allows you to test different builds of the same app with different groups simultaneously.

The main issue most people have with TestFlight is that before builds can be sent to external testers, they must first go through Apple’s approval process. The first build’s approval takes longer than subsequent builds and the turn around times depend on the development platform.

API and Integrations

TestFlight doesn’t offer a documented and supported API. Fastlane seems to be the only solution to automate the upload of your app to TestFlight.

Pricing

Free.

Pros

Up to 10,000 external testers without enterprise developer account.

Allows you to set a maximum number of testers for open beta tests.

Simultaneously test different builds of the same app.

The TestFlight app notifies testers about beta updates.

Testers can submit feedback through the TestFlight app.

Cons

Builds have to go through an approval process before being available to beta testers.

Only 25 internal testers allowed.

Basic session and crash reporting.

Bottom Line

TestFlight is a great option for indie developers and small to medium teams developing for the Apple environment. Tester groups and the ability to test builds simultaneously makes it a good choice for beta tests that aren’t overly complicated. However, if you require more detailed analytics about your testers or your app performance, you will need to rely on third-party tools.

Learn More: TestFlight Beta Testing: Setting Up Effective Beta Tests

Google Play

The Google Play Developer Console has abandoned the alpha/beta convention in favor of internal, closed and open test tracks. Apps that opt for an open test are discoverable on the Google Play store and can be downloaded by anyone.

You can invite testers to your app through email, provided it is a Gmail or G Suite account. You can also invite an unlimited number of testers through Google+ communities to your closed or open beta tests. Internal tests are limited to 100 testers per app while closed betas don’t restrict the number of testers you can recruit. When adding testers by email, you are allowed to upload up to 50 lists per test and a total of 200 email lists, each with a maximum of 2000 testers. Once your testers receive the link, they can download the app from the Google Play Store.

Google does not set a limit on the number of apps you can test at the same time, and you can group your users through Google+ communities or email lists. However, Google Play does not allow you to test different builds of the same app at the same time in the open test track.

The main issue with Google Play is its lack of features when it comes to tester management and team management. Builds submitted to Google Play also must undergo a review process, but that is usually done in a couple of hours.

API and integrations

Google Play Developer API.

Pricing

Free.

Pros

Testers don’t need to download a standalone app.

Testers don’t need to enable download from unknown sources in Android settings.

Open beta tests are discoverable in the Play Store.

Can set a maximum number of users for open tests.

Cons

Can’t test more than one build at the same time in open tests.

For closed and open tests, paid apps have to be purchased to gain access to the test.

Apps must undergo a beta review process.

Bottom Line

Google Play is a handy tool for indie Android developers and small to medium teams that don’t have many requirements for their beta test. The ability to list open betas on the Play Store can also be useful to get a head start on user acquisition. However, its lack of tester management outside of Google+ communities and the inability to test several builds simultaneously limit its scope without the use of third-party tools.

Learn More: Google Play Beta Testing: Setting Up Effective Beta Tests

Beta by Crashlytics

Crashlytics is part of the open-source Fabric suite of tools that was acquired by Google at the beginning of 2017. Google has since been working on integrating Fabric into their own platform, Firebase.

Crashlytics is well known for its crash reporting capabilities, but it also offers beta distribution and app analytics. More importantly, it offers these features for Android and iOS (the crash reporter and analytics features are also available for macOS and tvOS.). After integrating their SDK, you can send email invites to your testers with a link to download the app. When it’s time for an update, they will receive an in-app notification with a link to the new build.

Unlike the previous beta app distribution tools, Crashlytics doesn’t place any limits on the number of testers you can include. However, without an Apple Developer Enterprise Program, you will be limited to 100 UDIDs per device type. You can also test as many builds as you like simultaneously. To facilitate that, Crashlytics offers decent tester management capabilities like creating groups and granting or denying access to specific builds by a group or individual tester.

Besides the crash reporter and the beta distribution tool, Crashlytics offers in-app analytics to monitor important metrics for acquisition, engagement, and retention. This makes it a solution for many of your beta test’s needs.

API and integrations

Crashlytics has extensive integrations with most of the major bug trackers like Jira and communication tools like Slack. They have also open-sourced their integration implementation so you can tailor it to your needs.

Pricing

Free.

Pros

Manage Android and iOS betas from the same tool.

Unlimited users and testers on Android.

Tester groups and group permissions.

Powerful crash reporting and app analytics.

Cons

No support for cross-platform.

No support for direct, in-app feedback from testers.

Bottom Line

Crashlytics Beta is a powerful tool that meets many of the needs of a beta test while still scaling well. Crashlytics can be used by any team size, but as the beta test gets bigger or more sophisticated, it will not hold your hand through the complexity. You will need to have a well thought out workflow ready.

HockeyApp / Visual Studio App Center

While HockeyApp and App Center are not the same thing, since their acquisition, Microsoft has been working on integrating HockeyApp into their continuous integration solution. Microsoft currently offers most of HockeyApp’s features through App Center, where it plans to transition HockeyApp’s users.

HockeyApp can be integrated with your Android, iOS, macOS, tvOS, or Windows app. You can choose between sending your testers a link to register an HockeyApp account or a link to the app’s public download page. In addition to distribution, it offers crash reports, segmented in-app notifications, user feedback, and analytics.

While Crashlytics’ crash reporter and analytics features might be well known, HockeyApp’s features for user feedback and segmented notifications give HockeyApp an edge.

HockeyApp also provides great tester and team management features, allowing you to control exactly who has access to each build. Testers can be organized into groups and multiple builds can be sent to multiple groups simultaneously. You can grant or revoke access to individual builds for a group and for individual testers.

API and integrations

HockeyApp offers extensive integrations with most of the popular tools used by developers. Additionally, it offers a public API so you can integrate the tools of your choice.

Pricing

HockeyApp is free to use through App Center. As for App Center, it offers a free tier that you can scale as you go through add-ons. However, there are restrictions on build minutes, testing, and audience segments for push notifications.

Pros

Manage Android, iOS, and Windows betas from the same tool.

Unlimited users and testers on non-Apple platforms.

Tester groups and group permissions.

Powerful crash reporter, analytics, and targeted push notifications.

Cons

Sooner or later, HockeyApp is going to be discontinued and integrated into App Center.

Bottom Line

HockeyApp is a great platform for beta app distribution whose features cover a lot of the needs for beta testing. The tool also scales well and can support indie developers through to big companies.

TestFairy

TestFairy is a beta distribution platform with a focus on enterprise users. It is available for Android and iOS and supports cross-platform like Xamarin, Cordova, and React Native.

TestFairy allows you to invite beta testers through email, SMS, or through a landing page that it creates for each of your apps. This landing page can be open for anyone to join, or closed where testers can request access. Once the testers start testing your app, TestFairy records video sessions so you can see exactly what happened in your app before an issue occurs. TestFairy’s SDK also provides an in-app bug reporting and crash reporting tool to facilitate the process for you and your testers.

TestFairy provides you with tester management features that let you organize testers into groups and manage their permissions. However, you can’t test two different builds of the same app simultaneously.

With its focus on enterprise customers, TestFairy also offers Single Sign-in with the most popular platforms. Additionally, it can be hosted in the cloud or on-premise when added security is needed.

API and integrations

TestFairy integrates with a lot of popular tools for bug tracking, communication, and continuous integration. As for security, they integrate with Google, Okta, Onelogin and other popular tools. Moreover, they have a REST API, upload API, and Webhooks.

Pricing

TestFairy’s pricing is not available on their website and you will need to contact them for a quote.

Pros

Video recording of sessions.

In the cloud or on-premise.

Tester groups and group permissions.

Support for SAML and LDAP, and compatible with SSO services.

Cons

No free tier or startup pricing.

No access to crowdtesters.

Bottom Line

TestFairy is not a good match for indie developers and small companies who will not need some of its features. For medium to large companies focused on security, TestFairy can be a suitable solution provided they don’t need access to crowdtesters.

Appaloosa

Appaloosa is an enterprise app store that can also be used for beta app distribution on Android and iOS. Using their service, you can upload your beta build and distribute it to your testers through your own private app store.

Testers are directed to your private app store through a link that you can send by email. Appaloosa does not offer any in-app feedback, bug reporting, or crash reporting tools, but testers can submit feedback to your app store page.

With their enterprise focus, Appaloosa supports SAML, OAuth, and SAML SSO solutions to enhance security. Appaloosa also offers push notifications and self-updating apps, which can be useful to make sure that all your testers are on the latest build. The push notifications also allow you to engage your testers throughout your beta test in order to maximize feedback.

Tester management and the ability to organize them into groups with different authorizations and permissions is also supported to enable you to control who has access to your app.

API and integrations

Appaloosa integrates with most of the popular tools for bug tracking, communication, continuous integration, and SSO. In addition, they provide a RESTful API but the integrations and APIs are available starting with the enterprise plan.

Pricing

Appaloosa’s free tier is limited to one app with five versions and only group-based deployment.

Their Plus tier costs €1.49/user/month (minimum 20 users and maximum 100) for 5 apps with 10 versions each and doesn’t add features.

The Enterprise plan costs €2.49/user/month (minimum 100 users) for unlimited apps, versions, and users. This plan also unlocks all of Appaloosa’s features except for SSO and a custom domain URL.

The Custom plan unlocks the full potential of Appaloosa but you will have to contact them for a quotation.

Pros

Auto-updating apps.

SSO authentication.

Tester groups and group permissions.

Cons

No crash or bug reporting capabilities.

Most useful features are restricted to Enterprise plan.

Bottom Line

Appaloosa is another enterprise-focused tool that is not suitable for smaller teams. Some medium-sized companies might find their offering suitable if their test has specific needs like a private app store or added security. Big companies might find this tool valuable especially if they also want to use a private app store for their company’s applications.

DeployGate

DeployGate is a beta distribution tool that was spun off from Japanese social media platform Mixi. It is an interesting tool that offers pretty extensive features for beta testing, especially for a new arrival.

DeployGate provides you with fully customizable landing pages that you can share with your testers to download the app. The landing pages can be password protected or open depending on your needs. Moreover, the tool creates tester accounts when they access the app for the first time without the use of emails. Every tester is assigned a unique ID related to his device with which the account will be created.

Testers can be organized into groups that can be granted different permissions according to their roles. You can use these groups to selectively push different builds and apps to different groups. DeployGate even supports testing of more than one build at once.

When it comes to feedback and reporting, DeployGate has a slightly different approach than the other tools we have looked at. While they do offer an SDK that enables crash reporting and basic analytics, they don’t provide a way for in-app bug reports and user feedback. Instead, they give each of your apps a private forum page that includes the respective testers and allows you to communicate with your testers there.

A neat feature that they offer is their command line interface tool that automates adding UDIDs, updating provisioning profiles, building, and deploying your app.

API and integrations

DeployGate fully integrates with Slack, HipChat, and ChatWork. They also offer a RESTful API so you can integrate the tools of your choice.

Pricing

DeployGate offers a very limited “Personal Free” tier that although offers unlimited apps and testers, is restricted to two distributions/app and one week of log retention.

The “Personal Pro” tier costs $15/month and introduces unlimited apps and testers, 5 private apps, 10 distributions/app and 3 months log retention.

Business tiers start at $99 for the startup tier and goes up to $999 for the “large business tier”.

Pros

Auto tester account creation

Tester groups and group permissions

Test more than one build of the same app simultaneously

CLI tool for auto UDID acquiry, provisioning profile management, and deployment

Cons

No in-app feedback or bug reporting

Limited third-party integrations

Bottom Line

DeployGate is an interesting tool that could be capable to help small to medium teams accelerate their beta test and deployment. The tool is still quite new and they will presumably be iterating on it and adding new features but we can’t be certain about its future yet. DeployGate can handle beta testing on a large scale too, but with the absence of analytics, in-app feedback, bug-reporting, and push notifications, it will need to be assisted by other tools that can handle this.