Illustration by Agnes Lee

Illustration by Agnes Lee

If a running shoe made you 25 percent faster, would it be fair to wear it in a race? What about 10 percent? Or 2 percent? The Nike Zoom Vaporfly 4% — a bouncy, expensive shoe released to the public one year ago — raises these questions like no shoe in recent distance running history.

Nike says the shoes are about 4 percent better than some of its best racing shoes, as measured by how much energy runners spend when running in them. That is an astonishing claim, an efficiency improvement worth almost six minutes to a three-hour marathoner, or about eight minutes to a four-hour marathoner.

And it may be an accurate one, according to a new analysis by The New York Times of race data from about 500,000 marathon and half marathon running times since 2014.

Using public race reports and shoe records from Strava, a fitness app that calls itself the social network for athletes, The Times found that runners in Vaporflys ran 3 to 4 percent faster than similar runners wearing other shoes, and more than 1 percent faster than the next-fastest racing shoe.

We found that the difference was not explained by faster runners choosing to wear the shoes, by runners choosing to wear them in easier races or by runners switching to Vaporflys after running more training miles. Instead, the analysis suggests that, in a race between two marathoners of the same ability, a runner wearing Vaporflys would have a real advantage over a competitor not wearing them.

How the Nike Vaporflys compare with other popular running shoes ... When we use a statistical model, based on runners’ ages, genders, race histories and other information, to measure the effect of shoes

When we compare changes in race times among groups of runners who ran the same pairs of races

When we measure finish times after runners switch to new shoes

When we see how common a personal record is when switching shoes

The advantages for runners wearing Vaporflys were consistent for slower racers and fast ones; for men and women; for runners on their second marathon or their fifth.

The shoes’ effect when we restrict our analysis to ... Analysis type Statistical

models Shoe

switchers Matched

pairs Chance

of P.R. Everyone 4% 4% 4% 68% Men 4% 4% 5% 69% Women 3% 2% 3% 65% Marathons only 4% 3% 4% 63% Half marathons only 5% 5% 4% 80% Fastest runners 2% 2% 4% 65% Fast runners 4% 4% 4% 68% Slower runners 5% 7% 6% 65% Enthusiastic runners 4% 4% 5% 59% Logged training miles 4% 3% 4% 68% fastest runners are those whose average time meets the fast runners are those whose average time is within 25 percent of those standards or faster; slower runners are those whose average time exceeds 25 percent above those standards. Enthusiastic runners are runners with at least four completed marathons or four completed half marathons in the data. Logged training miles is the subset of runners who logged training miles in Strava in the four months before the race date. The subset ofare those whose average time meets the 2018 time-qualifying standards for the New York marathon;are those whose average time is within 25 percent of those standards or faster;are those whose average time exceeds 25 percent above those standards.are runners with at least four completed marathons or four completed half marathons in the data.is the subset of runners who logged training miles in Strava in the four months before the race date.

The Vaporflys — which retail at $250 a pair — were widely released to the public by Nike last summer. Unlike most running shoes, they have a carbon-fiber plate in the midsole, which stores and releases energy with each stride and is meant to act as a kind of slingshot, or catapult, to propel runners forward. Compared with typical training shoes, the Vaporflys are believed to wear out quickly: Some runners have said they lose their effectiveness after 100 miles or so.

The apparent effectiveness of the shoes highlights an issue that has vexed sports officials for decades: how to determine which technological advances constitute an unfair competitive advantage. Golf officials barred the use of certain balls that fly straighter, the N.F.L. barred the use of a sticky substance that helped players catch the ball, and swimming officials barred high-tech suits that were said to have enhanced buoyancy and speed.

The swimsuits were believed to lower times by as much as 2 percent — comparable to the apparent advantage derived from Nike Vaporflys relative to the next-best-performing popular shoes in our data.

The International Association of Athletics Federations, track’s governing body, has rules about shoes, but they are vague: “Shoes must not be constructed so as to give athletes any unfair assistance or advantage.” It does not specify what such an advantage might be.

The rules also state that shoes “must be reasonably available to all in the spirit of the universality of athletics.” The Vaporflys sell out quickly; on the secondary market, a pair can cost $400 or more. Nike’s newest version of the shoe, the Elite Flyprint, was sold to a limited number of runners in London for the 2018 London Marathon at a cost of £499, or about $650.

When asked whether the shoes conform to track and field’s rules, a spokesman for Nike wrote in an email that the shoe “meets all I.A.A.F. product requirements and does not require any special inspection or approval.”

Yannis Nikolaou, a spokesman for the I.A.A.F., said that while it’s accurate to say that the Vaporflys are legal, it’s actually more accurate to say there is no evidence they shouldn’t be.

“We need evidence to say that something is wrong with a shoe,” he said. “We’ve never had anyone to bring some evidence to convince us.”

Where these estimates come from

An ideal experiment to measure how much shoes matter for race performance might involve a series of marathons on a variety of courses, with runners randomly assigned different running shoes.

That experiment does not yet exist, but something like it happens around the world almost every weekend, when tens of thousands of amateur runners compete in races and upload their race data — collected on smartphones or satellite watches — to Strava.

This data usually includes statistics like a runner’s overall time, splits for every mile and a map of the runner’s route. In about one-third of the races on Strava, athletes reported data on the shoes that they wore.

Here’s what those race reports look like for three runners of the 2017 New York Marathon who finished near the three-hour mark:

No shoes reported Nike Zoom Vaporfly 4% Brooks Launch No shoes reported Nike Zoom Vaporfly 4% Brooks Launch No shoes reported Nike Zoom Vaporfly 4% Brooks Launch

Based on profiles from more than 700 races in dozens of countries since 2014, The Times compiled results from about 280,000 marathon and 215,000 half marathon completed races. (These race records are public, but The Times obtained permission from every athlete named in this article.)

Below, we describe the four ways we measured the shoes’ effect.

Measuring shoe effects using statistical models

Pros of this approach: Tries to control for race conditions, weather, gender, age, pre-race training and a runner’s previous race times. Cons of this approach: Still not a randomized controlled trial.

We know a lot about the runners in our data set, including their age, gender, race history and, in some cases, how much training they’ve done in the months before a race. We also know about the races themselves, including the distribution of runners’ times and the weather that day. We can put all of this information into a model to try to estimate the change in runners’ time from their previous races.

After controlling for all of these variables, our model estimates that the shoes account for an expected improvement of about 4 percent over a runner’s previous time. Including the uncertainty around the estimates, the Vaporflys are a clear outlier, one of the only popular shoes we can really say makes any difference at all.

Estimated change in race time, compared with a previous result, when switching shoes Best guess 95% confidence interval Show more shoes ↓ Some shoe names have been shortened.

This model estimates the effect of wearing Vaporflys compared with the effect of wearing any other shoes. It estimates the expected change in race time from a runner’s previous race given the runner’s age, gender, race history, training miles (for those athletes whose training logs are public) and the weather conditions on race day.

There are several statistical approaches one could take with this data set — and we tried several of them. None are perfect, but every way we tried, the effect of the shoes was more or less consistent: whether we included training miles or omitted them; whether weather data was included or ignored; or whether we modeled the change in time after switching shoes or the change in time from a runner’s average.

Of course, these observations do not constitute a randomized control trial. Runners choose to wear Vaporflys; they are not randomly assigned them. One statistical approach that seeks to address this uses something called propensity scores, which attempt to control for the likelihood that someone wears the shoes in the first place. We tried this, too. Our estimates didn’t change.

Comparing groups of runners who completed the same two races

Pros of this approach: Follows athletes of similar ability who ran in identical conditions. Cons of this approach: Runners could save their special shoes for when they expect to have a fast race.

Tony Kharitonov and Marios Athineos don’t know each other — in fact, they live on opposite coasts — but they have something in common: They both failed to get the time they wanted in the 2017 Boston Marathon. But they failed together, just nine seconds apart, two minutes on the wrong side of three hours.

That race was one of the hottest in years, with temperatures approaching 80 degrees, and both runners were off pace after the course’s famous Heartbreak Hill, around Mile 21, and slowed considerably in the race’s final miles.

Running a marathon in less than three hours is a sign of an elite runner, and Boston is one of the world’s most prestigious races. In April, the men returned to run it again — and both switched to a different pair of shoes. Mr. Kharitonov wore Altra Solstice shoes, and Mr. Athineos wore Vaporflys.

This time, the results were different. Mr. Athineos, a 42-year-old researcher based in San Francisco, finished with 33 seconds to spare, at 2:59:26. But Mr. Kharitonov, 49, a software engineer in Millburn, N.J., could not stay warm, and once again could not maintain his pace. For the fifth year in a row, he failed to break three hours at the Boston Marathon, finishing in 3:07:17.

Perhaps not surprisingly, given the conditions, neither runner identified shoes as a factor in this year’s result. And, given what we know about how different the races were, it may seem unhelpful to compare times between them at all. But there is still meaningful information in events like these in aggregate.

Instead of directly comparing performances in the two races, we can compare the net change of runners who switched to Vaporflys with the net change of similar runners who did not. Of the 1,275 runners in our data who ran Boston both years, 52 switched to Vaporflys in 2018, as Mr. Athineos did. On the whole, those runners’ times improved more than that of the group of runners who did not.

Change in performance, Boston 2017 to Boston 2018 On average, there wasn’t much of a difference in race times among runners who ran Boston in 2017 and 2018. Change in performance, Boston 2017 to Boston 2018, among runners who switched to Vaporflys But 85 percent of runners who switched to Vaporflys got faster.

This specific comparison may be too extreme for any study of shoes, and perhaps it is. But Boston 2017 and Boston 2018 are just one pair of races, and there are hundreds of pairs of races in our data in which large groups of runners ran the same two races and in which a subset of them switched shoes. When we compare all these pairs of races among all kinds of popular shoes, we see that, on average, runners who switched to Vaporflys improved their times more than similar runners who switched to any other kind of popular shoe.

Average change among shoe-switchers compared with non-switchers Best guess 95% confidence interval Show more shoes ↓ Some shoe names have been shortened.

Following runners as they switch to a new kind of racing shoe

Pros of this approach: Accounts for runners of varying skills over several races. Cons of this approach: Runners could save Vaporflys for when they expect to be faster than normal, or Vaporfly wearers could be different in some way from other kinds of runners.

The runners who use Strava are an enthusiastic bunch: About one in four have uploaded data for two or more marathons, and about one in 10 have uploaded data for three or more. The Strava data allows us to follow these repeat racers over time, and, crucially, as they change shoes.

Glenn Kasin, a 49-year-old doctoral student at the Southern California University of Health Sciences, is one of these avid runners. He has uploaded five marathons to Strava since November 2015, and he wore a different pair of shoes in every race. In January, at the Houston Marathon, he decided to race in Vaporflys.

New York, 2015 Austin, 2016 Chicago, 2016 Boston, 2017 Houston, 2018 Saucony Triumph New Balance Vazee Pace Hoka One One Clifton New Balance 1400 Nike Vaporfly 4% 3:35:22 3:13:30 3:30:23 3:13:50 2:56:00 New York Austin Chicago Boston Houston 2015 2016 2016 2017 2018 New Balance Vazee Pace Hoka One One Clifton New Balance 1400 Nike Zoom Vaporfly 4% Saucony Triumph 3:35:22 3:13:30 3:13:50 3:30:23 2:56:00

In Houston, Mr. Kasin improved on his Boston time by about 18 minutes, an improvement owing to much more than a pair of shoes. He attributed his result, a 2:56:00, to many things: perfect weather, a fast course and the culmination of two consecutive years of a training regimen that included, on average, about 50 miles of running per week. But he said he also thought the shoes made a difference.

“Everything aligned, and then you throw that shoe on top of it,” he said. “The pace felt so easy that you felt like questioning: What’s going on here? When is it not going to feel easy?”

Mr. Kasin is a single data point, but in our data there are almost 4,000 runners just like him — men and women who have uploaded results for five marathons or more. When we aggregate the change in race times for runners the first time they switch to a new pair of shoes, runners who switched to Vaporflys improved their times more than runners who switched to any other kind of popular shoe.

Median change in race time when runners switch shoes Best guess 95% confidence interval Show more shoes ↓ Includes only runners who switch from one shoe to another, rather than switching from not reporting shoes at all to reporting them. Some shoe names have been shortened.

Measuring the likelihood of a personal best

Pros of this approach: A measure of race performance most runners know by heart. Cons of this approach: Doesn’t account for race conditions, increased training miles or aging. Runners who switch to Vaporflys could be different from other runners.

Race times are, in many ways, a crude way to measure performance. A marathon is not like a 400-meter dash or a 100-meter freestyle swim: One marathon may be hilly or feature sharp turns; others may be flat and straight. Weather, too, is important, with higher temperatures typically resulting in much slower times.

And yet race times are how runners qualify for prestigious races, like the Boston Marathon, and most runners know their personal best times by heart, regardless of whether the race they ran was flat or hilly, on a hot day or a cold one.

We can follow the runners in our data with this measure in mind, testing whether a runner’s fastest time is more likely when he or she switches to the Vaporflys, or any other kind of shoe.

Consider two fast runners in our data: Stephanie Andre, 35, a freelance writer from Bixby, Okla., and Amanda Hicks, 34, a product director in Washington, D.C. Both women have uploaded several marathons to Strava and both switched to the Vaporflys for a recent marathon. Mrs. Andre ran the 2017 Chicago Marathon and Ms. Hicks the 2018 Boston Marathon.

By objective standards, both women had excellent races. Mrs. Andre placed 22nd among women in Chicago with a 2:41:50, fast enough to qualify for the 2020 U.S. Olympic team trials, and Ms. Hicks finished in Boston with a 3:15:31, among the top 5 percent of all female finishers that day.

Here are the two women’s marathon race histories as they appear in our data, with races in the Vaporflys highlighted:

Stephanie Andre set her P.R. ... ... but Amanda Hicks did not.

Someone can run a personal best for all kinds of reasons unrelated to shoes. A runner may train more, execute a better strategy on race day or run an easier course. Regardless, we found that runners who switched to Vaporflys were more likely to run their fastest race than runners who switched to almost any other kind of popular shoe.

Likelihood of a personal record when switching to … Best guess 95% confidence interval Show more shoes ↓ *Among runners who have run two or more marathons or half marathons. Some shoe names have been shortened.

Putting them all together

None of these approaches are perfect, but they all point to a similar conclusion. Wherever we look for evidence that shoes matter in a marathon or half marathon, we find Vaporflys at or near the top of that list. More than that, the shoes tend to be outliers among all popular shoes, suggesting that there is something happening in races with Vaporflys that is not happening in races with nearly any other kind of popular shoe.

Nothing is certain, of course, and we describe some reasons for continued skepticism below. But from our perspective, the most honest answer we can give to the question “Do Vaporflys really make runners faster?” is a qualified yes.

Reasons to remain skeptical

The data is self-reported — and self-typed. All of our analysis comes from runners who publicly tell Strava what shoe they’re wearing during a race, which they type in manually. The race records included about 33,000 distinct shoe descriptions, and our analysis depends on standardizing those records. (Even for a shoe as heavily marketed as the Vaporflys, Strava users found 147 different ways to spell the name.) We’ve done our best to identify misspellings, remove shoe records that are clearly incorrect and consolidate different variations of the same basic shoe. But there’s no perfect way to do so.

More problematic is when runners do not identify their shoes with enough precision for us to categorize them. Wherever possible, we’ve attempted to identify the shoe to its most specific name, differentiating between shoes within a shoe line (or “franchise”). But not all runners were this detailed. This was most common among Adidas shoes. Adidas, for example, lists more than a dozen different subfranchises belonging to its “Boost” franchise. When runners did not specify to this level of detail, we simply identified the parent franchise of a given shoe. This could mask high-performing shoes within a franchise.

The shoe names published here reflect a bespoke attempt at standardizing names, but we also tried an algorithmic approach. While the ordering of the shoes changed somewhat, the general pattern did not.

These estimates compare Vaporflys only with other popular shoes. Perhaps the greatest limitation of this data set is that for us to make meaningful conclusions, it requires that large groups of people to run in a pair of shoes. This necessarily omits less popular shoes that, for all we know, may be faster than the shoes mentioned in this article. This analysis includes only the 50 most popular shoes — those with roughly 500 races or more in our data.

There still may be something we’re missing that’s not captured in the data. It’s possible that runners wear Vaporflys only when they know they are going to run faster, or that the act of wearing Vaporflys correlates with other things that indicate a runner is going to run faster. We acknowledge that it’s impossible to control for every factor that affects marathon performance. But we have some evidence that it really is the shoes.

First, we identified training mileage in the months leading up to a race for runners whose training logs are public, an important indication that a runner is getting more serious in his or her race preparations. Training miles are a variable we included in our statistical approach, and when we matched pairs of runners who ran in the same groups of races, we tried to match not just race time but also training regimens, where possible.

Second, the effect of the shoe was persistent across different tiers of athletes — for three-hour marathoners and four-hour marathoners alike. Faster runners improved their race times at about the same rate that slower runners did.

Finally, if runners raced in Vaporflys only when they thought they might have their best race, we might expect to see fewer of them on days with suboptimal weather. But we do not see that. In the 2018 Boston Marathon, in particular — a race with conditions few runners would prefer — no shoe was more popular among the runners in our data who reported what they wore.

Reasons to believe these estimates

This is a large sample of runners. There’s nothing random about people so enthusiastic about running that they upload information about all their races and shoes to a social media site. But we believe the data set is broadly representative of competitive runners. Among the races known as the World Marathon Majors — the largest and most competitive races in the world — about one in six finishers uploaded race data to Strava and made it public. Not counting the Tokyo Marathon, about one in five did:

Race Strava records in our data Share of all finishers Berlin 2017 6,399 15% Chicago 2017 6,245 14% New York 2017 7,961 16% Tokyo 2018 1,536 5% London 2018 10,102 25% Boston 2018 6,294 24% Total 38,537 16%

These runners are a little faster than a much larger sample of runners that other researchers have examined, but they are broadly similar.

In all, about 600 different athletes reported wearing Vaporflys in 825 marathons and half marathons. They were the 32nd-most popular shoe in this data.

The speed increases associated with Vaporflys are consistent with the increases in a Nike-funded study by University of Colorado researchers published in the journal Sports Medicine. That experiment measured the energy output of 18 male elite runners on a treadmill at various speeds and concluded that, on average, runners in Vaporflys exerted 4 percent less energy than they did in other racing shoes. They estimated such a savings would translate into a performance improvement of around 3 percent for a three-hour marathoner. Our findings were similar but showed a smaller effect. We found Vaporflys to be 1 percent faster than the next-fastest running shoe in our data (the Nike Streak), but our sample included a much wider range of athletes, and included men and women.

Runners who improved their performance in Vaporflys and then switched to other shoes got slower. There were very few runners in our data who met this criterion: We counted just 24 such races. But these runners finished about 7 percent slower on average than they did while wearing the Vaporflys. (Of the 24 races, 17 races were slower, and 7 were faster.)