- Experimentation and Incrementalism: The Impact of the Adoption of A/B Testing
This paper studies how the adoption of experimentation as a selection method shapes the direction of innovation. The spread of A/B testing, or digital randomized experiments, has made experimentation one of the most common methods organizations use to evaluate and select internally generated ideas. The strength of experimental evidence can be a force for radical innovation. By providing seemingly irrefutable evidence, well-designed and well-executed experiments can disabuse people of their false beliefs and generate breakthrough discoveries. However, the adoption of experimentation can also result in incrementalism, whereby firms focus on minuscule yet reliable improvements. I argue that these divergent outcomes can be explained by the incentives driving the people who design and implement experiments. The incentives of managers in established firms may lead them to use experiments in a way that undermines the pursuit of novelty while encouraging the search for incremental improvements. I investigate the relationship between experimentation and innovation in the context of US newspaper websites and their adoption of A/B testing. Using a historical archive of US newspaper websites and a novel computational method, I find that the adoption of A/B testing decreases the likelihood of radical change and increases the on-average similarity of websites over time.
- Organizational vs. Crowd Selection: Implications for Exploration and Exploitation
with Jesper B. Sorensen , (R&R at Organization Science)
This article examines the interaction between the variation and selection stages of organizational innovation. Most organizations rely on internal selection mechanisms, whereby creators submit their ideas to managers or experts within their firm for approval. Recent years have seen a growth in the uses of crowd-based selection mechanisms, whereby external audiences choose among ideas. While prior work has compared crowds and expert in terms of which kinds of ideas they select, we examine how these alternative selection mechanisms might influence the idea-generation process. We argue that internal selection generally lowers exploration by reducing the variation in ideas because creators with a clear conception of selection criteria constrain their search for ideas. We use two separate innovation tournaments to compare the two selection mechanisms on the variety of ideas generated. The findings are consistent with the claim that internal selection reduces variation. The results have implications for both the theory and practice of organizational innovation.
- How Do Firms React to Success and Failure: Evidence from a Regression Discontinuity
This article examines the causal effect of success and failure on future search behavior. Iteration based on frequent collection of market feedback has gained immense popularity as a method for innovation. Even as the literature has acknowledged this transformation, we have learned surprisingly little about how such feedback influences firms’ search for innovation. This paper investigates firms’ reaction to feedback using a unique data set consisting of 40,053 A/B tests (digital randomized controlled trials) conducted by 2,106 unique teams from 1,360 different firms. Using a sharp regression discontinuity around the 0.1 p-value (the default p-value for determining statistical significance in the particular software used to run the A/B tests), I compare experiments that had results just above or just below the 0.1 p-value. I demonstrate that failure causes teams to shrink their search, implementing fewer experiments and using fewer metrics to measure outcomes. Moreover, the difference between treatment and control is smaller in experiments following a failure, suggesting that the ideas implemented are relatively similar to the current version of their product.