Making marketing decisions based on an analysis of Big Data can be risky if not done properly, because data seldom reveal the causal links between correlated events. Take the case of one large retailer we studied. The company noticed that customers who purchased perishables also tended to purchase large-screen TVs. Based on this observation, the company made a significant investment in marketing activities directed at increasing purchases of perishables, in the hope that this would trigger more TV purchases. But while they sold more perishables, they didn’t manage to shift any more TVs, and the profits from selling extra perishables weren’t enough to cover the marketing investment.

Insight Center The New Tools of Marketing Sponsored by Percolate How leading companies connect with customers.

To avoid falling into traps like this, a growing number of companies are supplementing their data analysis with controlled field experiments that test whether the correlations thrown up by analysis of historical data truly represent causal relationships. But field experiments can also be misused: for example managers may make decisions based on findings that become stale by the time actions are taken, or they may ignore side effects of their actions that are not carefully considered during the experiments. So how do firms know when to use them? The decision depends on the answers to four questions:

Will our experiments really scale?

In determining whether it makes sense to conduct an experiment, you need to be confident that the testing environment is similar to the environment in which the insights will be deployed. Although a direct mail test involving a small sample of customers should scale well to larger populations, in-store interventions, in which employees change how they interact with customers, do not always scale well. With a small experiment, monitoring compliance with the experimental instructions is easy, particularly when the staff used to implement the intervention are specially trained, supervised and supplied as an additional resource. However, when the change is rolled out and regular employees apply the new process with less supervision, then outcomes observed in the experiment often fail to replicate.

How significant are cross-product effects?

If there are many and large cross-product effects in a category you are analyzing, you will need to conduct a large number of experiments in order to account for all of the effects at work. But if cross-product effects are small, our research shows that the number of experiments required to make pricing or other marketing decisions may grow slowly with the number of products in a category. This allows firms to obtain meaningful estimates from a realistic number of experiments, even in categories where the number of complementary or substitutable products is large.

How much precision do you need?

There are many reasons why a product’s environment may change: the time of the year, the level of prior exposure, intervening customer experiences, and differences in communications details can all lead to changes in market outcomes. As a result, the results from a field experiment may not predict future outcomes. Our research suggests that the difference between present and future responses depends upon the level of aggregation in the data. In general, experiments establishing aggregate predictions for large groups of customers are more reliable than those targeted at small, tightly defined groups. Additionally, we find directional findings are safer than absolute ones. So the experimental finding that a free trial is more profitable than a price discount will be safer than the finding that 2% of customers will respond to the free trial and 1.5% will respond to the price discount. In many cases, this directional guidance is sufficient to make good business decisions.

Can you use data from a field experiment to target customers?

Field experiments offer a source of data that can be exploited by a wide range of segmentation methods. In principle, the combination of field experiment data and these segmentation methods allows firms to tailor their marketing decisions to different customer segments. For example, consider a retailer facing a customer acquisition problem; the retailer wants to segment customers and target each customer segment with an optimal promotion mailing. Harnessing the power of both machine learning and field experiments, the retailer can use an initial field experiment as a source of data for the segmentation methods, which can then be used to choose an optimal policy.

Big Data provides a window into real-time changes in the business environment, customer needs, and customer behavior. But the dangers of misusing data may outweigh the benefits — bad data-driven business decisions occur much more often than managers realize. Carefully applied field experiments can help to change this equation.