Technical Note TN2431

This guide covers recommended procedures for app testing. Though the importance of testing apps and app updates before submission or deploying to Enterprise users is common knowledge, this guide highlights specific areas that are frequently overlooked. Additionally, strategies are provided to debug general problems that might occur in your app's distribution build.

Before submitting your app or its update to the App Store or deploying to Enterprise users, apps must be tested in a simulated customer environment in order to guard against failures that result from untested circumstances. This guide covers the general process to do so including common areas that new apps and app updates frequently miss during testing.

Development build is synonymous with Debug build in the context of this document. It is a Debug build that is code signed by a developer provisioning profile.

Distribution build is synonymous with Release build in the context of this document. It is a Release build that is code signed by one of the various distribution provisioning profiles.

Debug build refers an app bundle created with Xcode's Debug build configuration. Debug is the default build configuration used when running an app on a device through Xcode.

App Testing used within this guide refers to all testing of the app’s Release build before it is submitted for App Store review or distributed to Enterprise users. This includes new apps and updates to existing apps.

Release build refers to an app bundle created with Xcode's Release build configuration. Release is the build configuration used when archiving an app in Xcode, by default. Since archives are the recommended way to distribute all apps, this document uses "Release build" to refer to a distribution build created from an Xcode archive that is code signed by an Ad Hoc, Enterprise, or App Store provisioning profile (the latter including TestFlight builds).

Customer Environment within the context of this document, refers to your app's end users – their computer or device on which your app runs, and its hardware constraints and other unique features that could affect your app, such as the strength of its network connection, memory availability, and disk space.

The issue of forgetting to test the app-update scenario before going live is one of the common customer-facing issues covered in this guide; see section: App update process . If you're experiencing a different customer-facing issue in the Release build, the next section lists other common problems that are frequently first-detected in customer environments.

To test an app in a specific state, save the app's container on the device whose state you want to preserve, and restore it on the testing device using the steps in:

Files saved through NSFileManager to locations defined by NSPathUtilities, such as NSDocumentDirectory, and NSCachesDirectory.

An app's container that is saved through Xcode includes data that was created through use of the following APIs:

Important: On macOS, the filesystem is more readily accessible for the purposes of testing the app's behavior with specific files on disk. On iOS, tvOS and watchOS, this is done by viewing, downloading and restoring app containers through Xcode's Devices Window.

Unless this is the a new application, the majority of your customers will be upgrading from a previous release. Test that upgrading the app on top of a prior installation of your app works smoothly and, if you have made changes to your file format, that existing data is successfully migrated, or continues to be supported in some other fashion. This recommendation generally applies to all platforms, including macOS.

Keychain data previously created by your application may not be deleted when the app is removed from the device. To clear its keychain, an application can call the SecItemDelete API with a query that matches all existing items. You might find it more convenient to prepare a small utility application that you can quickly install on your test device for this task. This utility application must have the same bundle identifier (and App ID prefix ) as your real application to modify its keychain.

If your application uses a shared app group container to store common data files, you should also remove all other applications in that app group from the testing device. Shared containers are not deleted until all applications in the app group are removed from the device.

You can delete an application and its container from the iOS home screen or using Xcode's Devices window .

Incidentally creating a dependency on the presence of previously-created data while iteratively developing an app is a common problem. Xcode's app install process is optimized for development, but is slightly different than how iTunes and the App Store install apps. Prior to distributing a new version of your app, thoroughly test launching from a clean installation – one that is free from any data that might have been created by an older version of your app.

watchOS : Apple Watch App (on the paired iPhone) > My Watch tab > General > About> "Version" and "Model".

For problems reported by testers or Enterprise users, the OS and device type should be collected by the user in the following ways varying by platform:

Crash logs that happen for your App Store customers or TestFlight testers are available in Xcode; the following guide walks through obtaining these types of crash logs App Distribution Guide > Analyzing Crash Reports .

Crash logs that are generated in App Review will be attached to the rejection letter.

For crashing Issues, the OS version and specific device type can be found at the top of a crash log. As an example, see the OS and device in the example crash log shown in Analyzing Crash Reports section of TN2151 - Understanding and Analyzing iOS Application Crash Reports .

The following examples illustrate ways to get this important information:

When a new OS release is imminent, carefully plan which devices you’re going to take to the new OS release and which devices you want left on the old OS release; upgrade the latter before the new OS is released.

Maximize your deployment target (and thus minimize the spread of OS releases you need to cover).

Try to maintain a spread of test devices (32- vs 64-bit, memory sizes, processor speed, GPU, iPad vs compact devices like iPhone and iPod touch, OS releases).

It can be difficult to acquire and maintain configuration of all the combinations of device type and OS versions your app supports, but this section helps offer a reasonable strategy to maximize your ability to do so.

If an issue is reported against your app's release build, pay close attention to the OS version and device type it occurred on when trying to reproduce the issue in your local development environment.

It is possible that a particular app issue only reproduces on a specific version of the OS or device type. Therefore, testing all the OS versions and device types your app supports is crucial to ensuring a good customer experience.

Important: Only use Ad Hoc distribution to test your Release build if there is a good reason you cannot use TestFlight. Ad Hoc testing requires the app to be re-signed an additional time during submission, and the resulting switch in provisioning profile introduces the opportunity for the submitted build to behave differently than the Ad Hoc version. TestFlight solves this problem by submitting the same build that was tested. Other benefits of using TestFlight are listed above.

Note: TestFlight offers a testing environment that most closely matches the real world environment of a customer's device. For apps built with Bitcode, distributing with TestFlight is the only way to test the final build of your app that is created by the App Store.

Testing your Release build can be done in the following ways varying by platform:

Release build testing involves a unique workflow in Xcode that is different from Building and Running connected to the debugger. The difference is important because they can, when properly facilitated, reveal problems that weren't visible during development. The largest source of issues found in the customer environment are due to lack of thoroughly testing the Release build, and respecting the large number of fundamental differences between development and distribution builds.

Should a customer facing issue be reported in your app's Release build, use the app testing procedure to debug the issue in your local development environment. This section also focuses on increasing chances of reproducing a customer facing issue as that is key to confirming the problem is solved after app changes are made to address an issue.

The recommended app testing procedure follows. Because there are differences between the app's Debug build (ran during development) and the Release build (the app Xcode optimizes for submission), the full app testing procedure covered by this guide should be used to maximize the chances of catching problems that could otherwise surface in the customer environment.

Common causes of customer facing issues

Sometimes issues that surface only in the customer environment can relate to the build configuration that was used to build the app (Debug versus Release) or, the code signing profile used to sign the app (Developer versus Distribution). Other times, the issue only reproduces when disconnected from the debugger, running on an macOS guest account, or ran within differing network or memory-availability conditions. This section lists common causes of release build differences; often times, solutions or workarounds can be inferred once specific triggers are identified.

Debugger effects There are a couple monumental differences that result from running your app through Xcode (therefore, attached to the debugger). This section covers the differences that frequently cause problems you should therefore be mindful of. The debugger disables Watchdog timeouts The Xcode debugger disables crashes due to watchdog time outs, and for this reason, crashes due to watchdog timeout are often noticed only in the Release build. The easiest way to check for watchdog timeouts is to disconnect the Lightning cable and run the development build from the home screen. For more information on watchdog timeouts, see: • QA1693 - Synchronous Networking On The Main Thread.

The debugger prevents your app from being suspended Because the debugger prevents apps from being suspended, your app must be launched from the home screen in order to test any background processes your app might implement. For example, NSURLSession allows an app to opt into a background session, however this code will not be properly tested until the app is run disconnected from the Xcode debugger. For more information on this topic, see: Apple Developer Forums > Topic 42353.

Device power iOS may behave differently depending on battery charge level, how and when the battery was last fully charged, and whether or not the device is currently being charged. Therefore, to confirm your app behaves properly across the myriad of differences that can result from power level, you should: Test the app unplugged from a power source. Test the app in Low Power Mode. Steps to enable Low Power Mode on iOS: Navigate to Settings > Battery. Switch on Low Power Mode. Note: The battery indicator turns yellow to indicate the device is in Low Power Mode.

Note: An example is how the factors above change the behavior of Core Location. Core Location always provides whichever accuracy the client app has requested but, depending on the factors above as well as the accuracy requested by other apps on the system, it might return much higher accuracy than was directly specified. Unplugging your device from a power source and enabling Low Power Mode is one of the only ways to ensure an app using Core Location behaves correctly in all conditions. Important: This section does not attempt to list specific heuristics that trigger OS differences due to power level, but impresses that testing your app in varying power levels is an important consideration, regardless.

Build configuration Exercise to determine if an issue is Build Configuration related. What configuration was used in the build that failed? Set the Run task Build Configuration to the configuration used to build the failing app. If the issue does not reproduce, the problem is likely not related to the build configuration. If the issue does reproduce, now set the Run task Build Configuration to the opposite configuration used to build the failing app. If the issue does not reproduce, this confirms the issue is related to the build configuration. Check each build setting that is different across the configurations for potential causes of the problem. Notes on Build Setting differences: In Xcode's Build Settings almost every setting has the capability of being set differently for each build configuration. The default build configurations are Debug and Release, and by default, most build settings are the same. As an example, if Xcode were using a different Info.plist file for release builds, there is a good chance the Info.plist build setting differs across the two build configurations. Because the Info.plist build setting is the same across the two build settings by default, a change to this setting must have been made explicitly by the developer of this project. Figure 2 Example build setting that shows how something as crucial as the Info.plist could be set differently for Release builds.

Compiler optimizations A build setting that is different by default across the two build configurations is Optimization Level. This build setting controls compiler optimizations, which are code optimizations that are made by the compiler at build time. Compiler optimizations are on by default for Release builds because they result in a more performant app, however, you should be mindful of differences in behavior that can occur if you do not follow language rules. For example, in Objective-C, the following article covers a hypothetical example of a Release build difference that results from compiler optimizations: • LLVM PROJECT BLOG - What Every C Programmer Should Know About Undefined Behavior. For a Swift example, see the optimization note within: • The Swift Programming Language > Declarations > In-Out Parameters: "As an optimization, when the argument is a value stored at a physical address in memory, the same memory location is used both inside and outside the function body. The optimized behavior is known as call by reference; it satisfies all of the requirements of the copy-in copy-out model while removing the overhead of copying. Write your code using the model given by copy-in copy-out, without depending on the call-by-reference optimization, so that it behaves correctly with or without the optimization." Figure 3 The default Optimization Level build settings. Important: Different build settings define code optimizations across Objective-C and Swift because they have different compilers. Use the following steps to confirm whether compiler optimizations causing the difference in your Release build: Reproduce the issue by running the app's Release build. Temporarily set this build setting's Release value equal to the value of Debug (which is None [-O0] ). Then, build and test a new Release build of the app. If the issue does not reproduce, the issue is related to a compiler optimizations. Important: As a workaround in this case, you can submit a Release build of your app that is built with optimizations turned off, by setting Release to None [-O0] .

Code-signing provisioning profile Exercise to determine if an issue is Provisioning Profile related • Was a distribution or development provisioning profile used to code sign the failing app? Create an installable .ipa of the app file using the steps in App Distribution Guide > Distributing Your App Using TestFlight (iOS, tvOS, watchOS) and code sign the app with the same App Store profile that was used to sign the problematic app. When creating the app, use the process in How to test the same build that generated a crash log to ensure you choose the right app archive. If the issue is not crash-related, refer to the Version column of the Archives Organizer to make sure the correct archive is chosen. If the issue does not reproduce with the newly created TestFlight build, the problem is not related to the code signing provisioning profile. If the issue does reproduce, now grab the development signed version of this same app as it resides on disk within the .xcarchive file. Control-click on the archive in Xcode's > Window menu > Archives, and choose "Show in Finder" Control-click on the .xcarchive in Finder and choose "Show Package Contents" Find the .app within the ./Products/Applications folder Note: The app within the .xcarchive is signed with developer profile if you're using default Xcode build settings, or if you've properly reverted to the default build settings using the steps in QA1814 - Setting up Xcode to automatically manage your provisioning profiles. For the purposes of this test, you perform the steps on new archive built using the recommended settings if they were not used to create the initial archive. Install this developer-signed version of the same app using Xcode's > Window menu > Devices, with the steps in Devices Organizer Help > Installing Apps on a Device If the issue does not reproduce with the developer signed version of the app, this confirms the issue is related to provisioning profile. Check the entitlements of each profile for differences that could potentially cause the issue using the steps in TN2415 - Entitlements Troubleshooting > Inspecting a profile's entitlements. The best way to confirm which entitlements are enabled on a build submitted to the App Store is to view build details in iTunes Connect. For more information, see: • iTunes Connect Developer Guide > Viewing App Information > Viewing Build Details section.

It's possible that one of the provisioning profiles used during the signing of either the development or release build are out of sync with the current state of enabled capabilities. If it's the distribution profile that is out of sync, the issue produced as a result will only be present in the distribution build and not exhibited in the development build.

User privileges Testing your app on a Guest Account is an essential step to simulate the customer environment for an macOS app. Mac App Review facilitates review with minimally-privileged accounts for the purpose of catching bugs that developers and testers might have missed testing the app on an Admin account. For example, ensuring the guest account login runs your app smoothly ensures there are no privileges problems or missing files or folders that had been preconfigured within the development environment.

App update process The following are important notes that relate to issues that can occur during the app update process: Building and Running a pending app update through Xcode does not properly test the case of installing an app update on top of an existing app downloaded from the app store.

App Review devices typically do not contain an existing installation of your app from the App Store and therefore, the review process does not test the case of updating an existing customer installation.

An example issue that could only be caught while testing an app update is if your app has changed a file format and forgets to check for or support existing customer files in the previous format. The app might crash assuming that all files in the users Documents directory are in the new file format.

Test your app update using the steps in TN2285 - Testing iOS App Updates

Network conditions Debug builds are often tested within an isolated network, whereas Release builds run in varying network conditions that your users are exposed to. The result is an issue can appear to be related to the apps Release build, but may in fact relate only to the network that either build runs under. To alleviate these kinds of issues: Use the Network Link Conditioner This tool runs on your iOS device or tvOS device simulating slow or unreliable internet connections. The Network Link Conditioner can be activated using Settings app > Developer > Network Link Conditioner on the testing device.

Test in IPv6-only networks It's possible for an issue to surface in an IPv6-only or IPv4-only network, and not the other, so it's a good idea to test your app in both. Since IPv4-only networks are still widely in use, it's more common that an app issue relating to DNS64/NAT64 go undetected before submission.

Follow the steps in the definitive guide: Networking Overview > Designing for Real-World Networks.

Memory availability Another important consideration involves understanding there are different amounts of memory available to various devices and a given customer's instantiation of your app. This section recommends you walk through the process of minimizing your app's memory footprint as the best way to ensure your app avoids memory warnings or termination due to memory depletion in the customer environment. In addition, minimizing memory usage using Instruments can sometimes be a quick exercise that can yield surprising performance gains. To do so, follow the steps in: • TN2434 - Minimizing your app's Memory Footprint Note: In addition, simulating memory warnings using the iOS Simulator is the easiest way to test how your app recovers from memory warnings. For example, if your implementation of -didReceiveMemory warning causes other bugs, you'll want to reconsider how the app responds to low memory conditions.

Data edge-cases Apps that rely on customer-supplied data face the challenge of anticipating uncommon situations while processing that data. The wider the range of variations in customer supplied data your app accepts, the more responsibility there is to detect and gracefully handle unsupported cases. These kinds of considerations during testing are referred to here as data edge cases. The following are example data edge cases: If your app loads user-defined images, test the loading of an excessively large image. Because many images are compressed file formats, loading an image in memory can require significantly more space than is required to store it on disk. If an excessively large image opens normally in your app, does your app implement a size limit at a certain point larger than that?

If your app loads user Contacts, how does the app behave with an extremely large amount of Contacts? The extreme case of one or no contacts should also be tested. Important: Your app is responsible for detecting and gracefully handling cases where the user has supplied data that could risk the stability of your app.