The paradox of choice seems relevant in our modern consumer culture. Though it goes by many names it was popularised by Barry Schwartz in his 2004 book ‘The Paradox of Choice – Why More Is Less’. The concept as was initially proposed is straightforward: when presented with a large number of options, consumers are overwhelmed and thus are less likely to make a decision and, when they do, are less satisfied with their choice.

When there is too much choice

Research into the effects of overwhelming choice on people started in earnest at the beginning of the 21st century. theoretically proposed the idea and first empirically tested this phenomenon. Iyengar and Lepper looked at whether presenting participants with a large number of choices demotivated them to make a purchase or perform a related task and whether they were happy with their decision. The authors ran 3 studies looking at the effect of greater choice in 2 domains: food (specifically jam and chocolates) and university assignments.

Mm.. Food

The 2 studies looking at the effect of choice on food examined whether having a larger number of options impacted purchasing behaviour and satisfaction ratings. Study 1 used a real world setting of a supermarket with a selection of jams on display. Motivation was measured by the participants’ initial attraction to the booth and whether they bought any jam or not. 60% of the 242 participants who passed the extensive-selection display of jams (which contained 24 jams) stopped at the stall. 40% of the 260 participants stopped at the limited-selection display of jams (which contained 6). Of the 145 who were intrigued by the 24 jams 30% bought a jam, whereas only 3% of the 104 who saw the 6 jams did the same.

In Study 3, 67 participants were presented with either 6 different chocolates (limited-choice condition) or 30 (extensive-choice condition) and choose which chocolate they would like, after which they were allowed to taste it. There were 33 participants in the limited-choice condition and 34 participants in the extensive-choice condition. 67 participants were in a control group, where they could choose a chocolate but were given another chocolate they didn’t choose to taste . Participants could then choose either $5 or $5 worth of chocolates as reimbursement for their time. The relevant dependent variables were the participants’ satisfaction with their choice (coded as 3 questions relating to their experiences of enjoyment and 2 to regret regarding their choice on a 7-point scale) and their purchasing behaviour (coded as whether they took the chocolates or not).

After collapsing all 5 questions related to satisfaction into 1 composite number the researchers found participants in the limited-choice group were more satisfied than those in the extensive-choice group (M=6.28, SD = 0.54 versus M=5.46, SD = 0.82). They also found these participants were more likely to choose chocolates as compensation (48% versus 12%).

Possible problems

Whilst statistically significant, these results are based on a potentially meaningful mistake: analysing ordinal data like it is interval or ratio. One of the assumptions of an Analysis of Variance (ANOVA) is metric data i.e. data has equal gaps between each unit. Ordinal data (like the Likert scale used in this experiment) are not metric and thus this assumption is violated. Many have argued these tests are robust to such violations, but there is strong evidence this isn’t the case . This can greatly distort the results and lead to incorrect inferences. However, you would have to reanalyse the raw data using a non-metric model to see if there are differences in results. This analysis of ordinal data using a metric model is very common in psychology .

Another common theoretically risky behaviour is assuming the philosophical idea of operationalism. Operationalism holds that theoretical constructs e.g. motivation or satisfaction, are identical to the measurement of said construct . This philosophical stance was rejected as deeply flawed by philosophers of science over 50 years ago but is still the pervading view in psychology . If the assumption that the measurement is the same as the construct is violated, this undermines the validity of the conclusion reached as there is an unknown amount of measurement error . Given that the authors apparently subscribe to this belief without exploring other models, the conclusions could be unjustified. There are also questions about whether taking chocolates as compensation is a suitable proxy for ‘purchasing behaviour’. Is choosing chocolate as a reimbursement for the participant’s time the equivalent of spending their money in a store? Whilst none of these criticisms prima facie invalidate the results, they do raise concerns about the validity of the conclusions. Additional analyses testing these assumptions and exploring alternatives would be enough to refute them.

You don’t understand my words, but you must choose

Study 2 used 193 first year Stanford psychology students to see whether being presented with more or fewer essay titles would affect their motivation to write an extra-credit essay. Motivation was measured by the percentage of students in the different groups who completed the essay and the quality of those essays .

Of the 70 students assigned to the limited-choice condition (6 essay titles), 74% completed the assignment. Of the 123 students assigned to the extensive-choice condition (30 essay titles), 60% submitted an essay. The quality of the essays were statistically significantly different, with the limited-choice condition essays being rated as being of higher quality content (p<0.05) and form (p<0.03). P-values near the statistical significance threshold are weaker evidence for a phenomenon than smaller values and as such you should be cautious when interpreting this as evidence for a difference. Compounding this issue is the inappropriate analysis method. The researchers tested for a difference in content and form (ordinal data) using a metric model. As explained earlier, this is risky and can distort results. The participants were also explicitly told their performance on the assignment was irrelevant to the receipt of the two points. Thus, an external motivator was likely removed and only internal motivation remained, which may have differed between the groups.

Large-scale paradox of choice

explored the paradox of choice in relation to contributions to employee’s 401(k) plans . Using a regression model which controlled for various individual and plan characteristics e.g. age, number of employees, they found that as the number of funds offered by a plan increased, the lower the percentage chance of the employee participating in a 401(k). For every ten funds added, a 1.5-2% drop in participation was seen. This real world example is arguably more persuasive because of the greater external validity. However, the lack of control means a causal relationship cannot be established. This is why both experimental and and evidence from more naturalistic environments is often needed to support a claim.

Do options overwhelm us?

Whilst these individual studies may pique interest in the phenomenon, they aren’t in and of themselves enough to convince us of the choice overload hypothesis. tried to provide a more comprehensive answer. The authors performed a meta-analysis of all the studies they could find exploring this phenomenon (including unpublished work). They included 63 results across a variety of domains including food and dating across a number of different items e.g. mobile phones, chocolates, and pens. To standardise the results, they transformed all of the results into Cohen’s d (weighted by sample size). The mean effect size was 0.02 (95% confidence interval=-0.09 to 0.12). There was high heterogeneity in the results, ranging from d=-1.89 to d=1.21, with larger sample sizes typically finding smaller effect sizes . They fitted a meta-regression to the results to see what impact different moderators e.g. different dependent variables, publication status, had on the estimate.

There is one possible problem with the mean effect size: converting results to Cohen’s d assumes the data is normally distributed, continuous, and the variance between the groups is homogeneous. Ordinal data isn’t normally distributed nor continuous. Of the 63 results, 29 converted the results from the original papers (which came from ordinal data) into Cohen’s d. Common effect size estimates like Cohen’s d are sensitive to departures from normality and homogeneity . Thus, there is a risk of distorting results when using tests which have their assumptions violated . Therefore, our confidence in the mean effect size is reduced. Rerunning the analyses using a non-parametric equivalent would allow readers to know if the violations of assumptions had an impact on the final estimate. The other results were binary data converted into Cohen’s d e.g. Study 1 from Iyengar and Lepper (2000), but the authors didn’t explain which transformation they used. When looking at the results, they identified some feasibly important preconditions for choice overload to occur. Unfortunately, the data didn’t give a clear picture as to which of these were important.

Moderating factors

A more recent attempt to summarise the literature was published by . They set out to better explain the variation in results by positing 4 moderating factors. These moderating factors were: the difficulty of the decision task; the complexity of the choice set; consumers’ preference uncertainty (how well can the consumers evaluate the choices and whether they have a clear preference); and consumers’ decision goal (whether consumers aim to minimise the cognitive load of the decision. Each of these moderators had a statistically significant effect on choice overload, with estimates in the meta-regression model ranging from 0.32 (preference uncertainty) to 0.56 (decision goal). Their model accounted for 68% of the residual variances of the reported effect sizes, compared with 36% of the model.

Distribution difficulties

However, this meta-analysis suffers from the same issues identified above. 32 of the 99 results had ordinal data converted into Cohen’s d without any apparent consideration of the underlying distribution. There are also questions about how the authors transformed the binary results into this effect size. The authors used 2 transformations to see whether they produced similar results which they should be commended for. But the methods they used have been criticised. The first was an arcsine transformation which has been harshly criticised because of its inferiority to other methods .

The other method of transformation was a log-odds ratio. This test assumes an underlying continuous trait exists with a logistic distribution and homogeneous variance . It may be reasonable to assume a logistic distribution for the dichotomised continuous variable (classifying the number of items seem into “small” or “large” categories). But it is not reasonable to assume whether they made a purchase or not (which forms a binomial distribution) is a latent continuous trait with a logistic distribution . The paper the authors cite to justify their use of a log-odds ratio explicitly state none of the effect indices they examine (which includes a log-odds ratio) is suitable to estimate the population effect size with non-normal distributions . As with the points raised earlier, they aren’t enough to negate the results. But they expose potential uncontrolled errors which could undermine the conclusions.

Measuring what is missing

The authors of the meta-analysis decided to only include published studies. Thus, they are missing “grey literature”. Grey literature refers to unpublished papers or those published in less mainstream avenues, such as journals . Meta-analyses that exclude grey have been found to report effect sizes a third larger than meta-analyses that don’t .

The authors did test for publication bias by looking at the funnel plot and testing whether the residual effect size estimates could be predicted by the standard errors . If the funnel plot is symmetrical and the effect sizes cannot be predicted by the standard error , this is taken as evidence for the absence of publication bias. Factoring in all results reported in the meta-analysis, the checks for publication bias didn’t report any evidence of said bias. But as highlight, different tests of publication bias have differing strengths in the face of questionable research practices, heterogeneity, etc. They recommend authors run a variety of methods and compare the results. Depending on the plausible conditions of the literature, different tests may produce more accurate results and therefore different conclusions may be warranted.

Conclusion

The evidence regarding the paradox of choice hypothesis is equivocal. Whilst there is evidence for the phenomenon, it suffers from potentially serious flaws. The summaries of the literature provide different conclusions and each have issues which may undermine the validity of the conclusions. In my opinion, what is needed is a large-scale, preregistered, multi-site study with an agreed upon (by both proponents and sceptics) design. Something akin to the preregistered replication of ego-depletion . If operationalism is rejected, then the difference between the measure and the construct should be modelled. This will allow readers to see how variation in the construct is reflected in changes in the scores and whether this affects the conclusions. The data should be analysed using more theoretically appropriate statistical analyses. This can then be contrasted with more typical analysis plans to see for any differences. As with many results in psychology, the results aren’t clear cut and there are legitimate questions about the validity of the methods. As our field reckons with its past, perhaps new data will come to light answering questions around the paradox of choice. Until such a time, I’m going to remain sceptical.

References

{5421944:KDZ3P8QW};{5421944:CFUWNXWQ};{5421944:VD8XETGZ};{5421944:B8YAYQXS};{5421944:5LSA4RWL};{5421944:7BMPFF3L};{5421944:5ITNRVRK};{5421944:BXUZW6GB};{5421944:245MPM65};{5421944:VD8XETGZ};{5421944:UPT8QFKI};{5421944:HHLU3FGU};{5421944:245MPM65};{5421944:EFIGAML9},{5421944:U3F2CQ7D};{5421944:TKGPB2WN};{5421944:NLBN7F6L};{5421944:8JTL9NF8};{5421944:UMLZ7HJC};{5421944:4PPNIY8A};{5421944:EWC7J273} apa default asc no 8237 Psychological Methods, 8(4), 448–467. Sánchez-Meca, J., Marín-Martínez, F., & Chacón-Moscoso, S. (2003). Effect-size indices for dichotomized outcomes in meta-analysis.(4), 448–467. https://doi.org/10.1037/1082-989X.8.4.448 Ecology, 92(1), 3–10. Warton, D. I., & Hui, F. K. C. (2011). The arcsine is asinine: the analysis of proportions in ecology.(1), 3–10. https://doi.org/10.1890/10-0340.1 Wilson, E., Underwood, M., Puckrin, O., Letto, K., Doyle, R., Caravan, H., Camus, S., & Bassett, K. (2010). The arcsine transformation: has the time come for retirement? Hasselblad, V., & Hedges, L. V. (1995). Meta-analysis of screening and diagnostic tests. Psychological Bulletin, 117(1), 167–178. Perspectives on Psychological Science, 11(4), 546–573. Hagger, M. S. (2016). A Multilab Preregistered Replication of the Ego-Depletion Effect.(4), 546–573. https://doi.org/10.1177/1745691616652873 Advances in Methods and Practices in Psychological Science, 2(2), 115–144. Carter, E. C., Schönbrodt, F. D., Gervais, W. M., & Hilgard, J. (2019). Correcting for Bias in Psychology: A Comparison of Meta-Analytic Methods.(2), 115–144. https://doi.org/10.1177/2515245919847196 Where is the evidence: realising the value of grey literature for public policy and practice. https://apo.org.au/node/42299 Lawrence, A., Houghton, J. W., Thomas, J., & Weldon, P. (2014). Conn, V. S., Valentine, J. C., Cooper, H. M., & Rantz, M. J. (2003). Grey literature in meta-analyses. Nursing Research, 52(4), 256–261. Nature Human Behaviour, 2(1), 6. Benjamin, D. J. (2018). Redefine statistical significance.(1), 6. https://doi.org/10.1038/s41562-017-0189-z Psychometrika, 71(3), 425–440. Borsboom, D. (2006). The attack of the psychometricians.(3), 425–440. https://doi.org/10.1007/s11336-006-1447-6 Theory & Psychology, 11(1), 35–44. Bickhard, M. H. (2001). The Tragedy of Operationalism.(1), 35–44. https://doi.org/10.1177/0959354301111002 The logic of modern physics. New York : Macmillan. Bridgman, P. W. (1927).. New York : Macmillan. http://archive.org/details/logicofmodernphy00brid A Call for Greater Use of Nonparametric Statistics. https://eric.ed.gov/?id=ED471346 Leech, N. L., & Onwuegbuzie, A. J. (2002). American Psychologist, 55(1), 79–88. Schwartz, B. (2000). Self-determination: The tyranny of freedom.(1), 79–88. https://doi.org/10.1037/0003-066X.55.1.79 Journal of Consumer Psychology, 25(2), 333–358. Chernev, A., Böckenholt, U., & Goodman, J. (2015). Choice overload: A conceptual review and meta-analysis.(2), 333–358. https://doi.org/10.1016/j.jcps.2014.08.002 Journal of Personality and Social Psychology, 79, 995–1006. Iyengar, S., & Lepper, M. (2001). When Choice is Demotivating: Can One Desire Too Much of a Good Thing?, 995–1006. https://doi.org/10.1037/0022-3514.79.6.995 How Much Choice is Too Much? Contributions to 401(k) Retirement Plans. Oxford University Press. Sethi-Iyengar, S., Huberman, G., & Jiang, G. (2004).. Oxford University Press. https://www.oxfordscholarship.com/view/10.1093/0199273391.001.0001/acprof-9780199273393-chapter-5 Journal of Consumer Research, 37(3), 409–425. Scheibehenne, B., Greifeneder, R., & Todd, P. M. (2010). Can There Ever Be Too Many Options? A Meta-Analytic Review of Choice Overload.(3), 409–425. https://doi.org/10.1086/651235 Journal of Experimental Social Psychology, 79, 328–348. Liddell, T. M., & Kruschke, J. K. (2018). Analyzing ordinal data with metric models: What could possibly go wrong?, 328–348. https://doi.org/10.1016/j.jesp.2018.08.009

5421944 {5421944:M7WKTFLU} items 1 apa default asc http://psychbrief.com/wp-content/plugins/zotpress/ Multivariate Behavioral Research, 47(2), 201–223. Ruscio, J., & Mullen, T. (2012). Confidence Intervals for the Probability of Superiority Effect Size Measure and the Area Under a Receiver Operating Characteristic Curve.(2), 201–223. https://doi.org/10.1080/00273171.2012.658329