If a running shoe made you 25 per cent faster, would it be fair to wear it in a race? What about 10 per cent? Or 2 per cent? The Nike Zoom Vaporfly 4% — a bouncy, expensive shoe released to the public one year ago – raises these questions like no shoe in recent distance running history.
Nike says the shoes are about 4 percent better than some of its best racing shoes, as measured by how much energy runners spend when running in them. That is an astonishing claim, an efficiency improvement worth almost six minutes to a 3-hour marathoner, or about eight minutes to a 4-hour marathoner.
And it may be an accurate one, according to a new analysis by the New York Times of race data from 500,000 marathon and half marathon running times since 2014.
Using public race reports and shoe records from Strava, a fitness app, it was found that runners in Vaporflys ran 3 per cent to 4 per cent faster than similar runners wearing other shoes, and more than 1 per cent faster than the next-fastest racing shoe.
The difference was not explained by faster runners choosing to wear the shoes, by runners choosing to wear them in easier races, or by runners switching to Vaporflys after running more training miles. Instead, the analysis suggests that, in a race between two marathoners of the same ability, a runner wearing Vaporflys would have a real advantage over a competitor not wearing them.
The advantages for runners wearing Vaporflys were consistent for slower racers and fast ones; for men and women; for runners on their second marathon or their fifth.
The Vaporflys – which retail at €250 a pair! — were widely released to the public by Nike last summer. Unlike most running shoes, they have a carbon-fiber plate in the midsole, which stores and releases energy with each stride and is meant to act as a kind of slingshot, or catapult, to propel runners forward. Compared with typical training shoes, the Vaporflys are believed to wear out quickly – some runners have said they lose their effectiveness after 100 miles or so.
The apparent effectiveness of the shoes highlights an issue that has vexed sports officials for decades: how to determine which technological advances constitute an unfair competitive advantage. Golf officials barred the use of certain balls that fly straighter and swimming officials barred high-tech suits that were said to have enhanced buoyancy and speed. The swimsuits were believed to lower times by as much as 2 per cent – comparable to the apparent advantage derived from Nike Vaporflys relative to the next-best-performing popular shoes, according to the data.
The International Association of Athletics Federations (IAAF), track’s governing body, has rules about shoes, but they are vague: “Shoes must not be constructed so as to give athletes any unfair assistance or advantage.” It does not specify what such an advantage might be.
The rules also state that shoes “must be reasonably available to all in the spirit of the universality of athletics.” Nike’s newest version of the shoe, the Elite Flyprint, was sold to a limited number of runners for the 2018 London Marathon at a cost of £499 pounds (about €560).
When asked whether the shoes conform to track and field's rules, a spokesman for Nike wrote in an email that the shoe "meets all IAAF product requirements and does not require any special inspection or approval". Yannis Nikolaou, a spokesman for the IAAF, said that while it's accurate to say that the Vaporflys are legal, it's actually more accurate to say there is no evidence they shouldn't be. "We need evidence to say that something is wrong with a shoe," he said. "We've never had anyone to bring some evidence to convince us."
Where these estimates come from
An ideal experiment to measure how much shoes matter for race performance might involve a series of marathons on a variety of courses, with runners randomly assigned different running shoes.
That experiment does not yet exist, but something like it happens around the world almost every weekend, when tens of thousands of amateur runners compete in races and upload their race data – collected on smartphones or satellite watches – to Strava. This data usually includes statistics like a runner’s overall time, splits for every mile and a map of the runner’s route. In about one-third of the races on Strava, athletes reported data on the shoes that they wore.
Based on profiles from more than 700 races in dozens of countries since 2014, the New York Times compiled results from about 280,000 marathon and 215,000 half marathon completed races.
The Irish Times' Ian O'Riordan tried the Nike Zoom Vaporfly 4% (or, at least, he thought he did) last year.
How the shoes’ effect was measured: Part 1 (statistical models)
We know a lot about the runners in the data set, including age, gender, race history and, in some cases, how much training they had done in the months before a race. We also know about the races themselves, including the distribution of runners’ times and the weather that day. All this information can be put into a model to try to estimate the change in runners’ time from their previous races.
After controlling for all of these variables, the model estimates that the shoes account for an expected improvement of about 4 per cent over a runner’s previous time. Including the uncertainty around the estimates, the Vaporflys are a clear outlier, one of the only popular shoes that makes any difference at all.
There are several statistical approaches one could take with this data set. None are perfect, but every way tried, the effect of the shoes was more or less consistent: eg whether it included training miles or omitted them; or whether weather data was included or ignored.
Of course, these observations do not constitute a randomised control trial. Runners choose to wear Vaporflys – they are not randomly assigned them.
How the shoes’ effect was measured: Part 2 (different runners, same races)
Tony Kharitonov and Marios Athineos don't know each other – in fact, they live on opposite coasts of the US. But they have something in common: They both failed to get the time they wanted in the 2017 Boston Marathon. But they failed together, just nine seconds apart (3:02:04 and 3:02:13, respectively).
That race was one of the hottest in years, with temperatures approaching 80 degrees, and both runners were off pace after the course’s famous Heartbreak Hill, around Mile 21, and slowed considerably in the race’s final miles.
Running a marathon in less than three hours is a sign of an elite runner, and Boston is one of the world's most prestigious races. In April, the men returned to run it again — and both switched to a different pair of shoes. Kharitonov wore Altra Solstice shoes, and Athineos wore Vaporflys.
Once again, weather was a factor. The 2018 race featured pouring rain, strong winds, low temperatures and a big rise in dropouts.
This time, the results were different. Athineos, (42), a researcher based in San Francisco, finished with 33 seconds to spare, at 2:59:26. But Kharitonov, (49), a software engineer in Millburn, New Jersey, could not stay warm, and once again could not maintain his pace. For the fifth year in a row, he failed to break three hours at the Boston Marathon, finishing in 3:07:17.
Perhaps not surprisingly, given the conditions, neither runner identified shoes as a factor in this year’s result. And, given what we know about how different the races were, it may seem unhelpful to compare times between them at all. But there is still meaningful information in events like these in aggregate.
Instead of directly comparing performances in the two races, we can compare the net change of runners who switched to Vaporflys with the net change of similar runners who did not. Of the 1,275 runners in our data who ran Boston both years, 52 switched to Vaporflys in 2018, as Athineos did. On the whole, those runners’ times improved more than that of the group of runners who did not.
This specific comparison may be too extreme for any study of shoes, and perhaps it is. But Boston 2017 and Boston 2018 are just one pair of races, and there are hundreds of pairs of races in our data in which large groups of runners ran the same two races and in which a subset of them switched shoes. When we compare all these pairs of races among all kinds of popular shoes, we see that, on average, runners who switched to Vaporflys improved their times more than similar runners who switched to any other kind of popular shoe.
How the shoes’ effect was measured: Part 3 (switching racing shoe)
The runners who use Strava are an enthusiastic bunch: About 1 in 4 have uploaded data for two or more marathons, and about 1 in 10 have uploaded data for three or more. The Strava data allows us to follow these repeat racers over time, and, crucially, as they change shoes.
Glenn Kasin, (49), a doctoral student at the Southern California University of Health Sciences, is one of these avid runners. He has uploaded five marathons to Strava since November, 2015, and he wore a different pair of shoes in every race. In January, at the Houston Marathon, he decided to race in Vaporflys.
In Houston, Kasin improved on his Boston time by about 18 minutes, an improvement owing to much more than a pair of shoes. He attributed his result, a 2:56:00, to many things: perfect weather, a fast course and the culmination of two consecutive years of a training regimen that included, on average, about 50 miles of running per week. But he said he also thought the shoes made a difference.
“Everything aligned, and then you throw that shoe on top of it,” he said. “The pace felt so easy that you felt like questioning: What’s going on here? When is it not going to feel easy?”
Kasin is a single data point, but in the data there are almost 4,000 runners just like him – men and women who have uploaded results for five marathons or more. Aggregating the change in race times for runners the first time they switch to a new pair of shoes shows that runners who switched to Vaporflys improved their times more than runners who switched to any other kind of popular shoe.
How the shoes’ effect was measured: Part 4 (personal best)
Race times are, in many ways, a crude way to measure performance. A marathon is not like a 400-metre dash or a 100-metre freestyle swim. One marathon may be hilly or feature sharp turns – others may be flat and straight. Weather, too, is important, with higher temperatures typically resulting in much slower times.
And yet race times are how runners qualify for prestigious races, like the Boston Marathon, and most runners know their personal best times by heart, regardless of whether the race they ran was flat or hilly, on a hot day or a cold one.
We can follow the runners in the data with this measure in mind, testing whether a runner’s fastest time is more likely when he or she switches to the Vaporflys, or any other kind of shoe.
Consider two fast runners in our data: Stephanie Andre, (35), a freelance writer from Bixby, Oklahoma, and Amanda Hicks, (34), a product director in Washington, DC. Both women have uploaded several marathons to Strava and both switched to the Vaporflys for a recent marathon. Andre ran the 2017 Chicago Marathon and Hicks the 2018 Boston Marathon.
By objective standards, both women had excellent races. Andre placed 22nd among women in Chicago with a 2:41:50, fast enough to qualify for the 2020 US Olympic team trials, and Hicks finished in Boston with a 3:15:31, among the top 5 per cent of all female finishers that day. Andre set her personal record, but Hicks did not.
Someone can run a personal best for all kinds of reasons unrelated to shoes. A runner may train more, execute a better strategy on race day or run an easier course. Regardless, the data found that runners who switched to Vaporflys were more likely to run their fastest race than runners who switched to almost any other kind of popular shoe.
Putting them all together
None of these approaches are perfect, but they all point to a similar conclusion. Wherever we look for evidence that shoes matter in a marathon or half marathon, we find Vaporflys at or near the top of that list. More than that, the shoes tend to be outliers among all popular shoes, suggesting that there is something happening in races with Vaporflys that is not happening in races with nearly any other kind of popular shoe.
Nothing is certain, of course, and we describe some reasons for continued scepticism below. But from our perspective, the most honest answer we can give to the question, “do Vaporflys really make runners faster?” is a qualified yes.
Reasons to remain sceptical
All the analysis comes from runners who publicly tell Strava what shoe they’re wearing during a race, which they type in manually. The race records included about 33,000 distinct shoe descriptions, and the analysis depends on standardizing those records. (Even for a shoe as heavily marketed as the Vaporflys, Strava users found 147 different ways to spell the name.)
More problematic is when runners do not identify their shoes with enough precision to be categorised.Adidas, for example, lists more than a dozen different subfranchises belonging to its "Boost" franchise. When runners did not specify to this level of detail,the parent franchise of a given shoe was chosen. This could mask high-performing shoes within a franchise.
Perhaps the greatest limitation of this data set is that, for meaningful conclusions to be made, it requires that large groups of people run in a pair of shoes. This necessarily omits less popular shoes that may be faster than the shoes mentioned in this article. The analysis includes only the 50 most popular shoes.
Also, it is, of course, possible that runners wear Vaporflys only when they know they are going to run faster, or that the act of wearing Vaporflys correlates with other things that indicate a runner is going to run faster.
Reasons to believe
This is a large sample of runners. There's nothing random about people so enthusiastic about running that they upload information about all their races and shoes to a social media site. Among the races known as the World Marathon Majors – the largest and most competitive races in the world – about 1 in 6 finishers uploaded race data to Strava and made it public. Not counting the Tokyo Marathon, about 1 in 5 did. These runners are a little faster than a much larger sample of runners that other researchers have examined, but they are broadly similar. In all, about 600 different athletes reported wearing Vaporflys in 825 marathons and half marathons. They were the 32nd-most popular shoe in this data.
The speed increases associated with Vaporflys are consistent with the increases in a Nike-funded study by University of Colorado researchers published in the journal Sports Medicine. That experiment measured the energy output of 18 male elite runners on a treadmill at various speeds and concluded that, on average, runners in Vaporflys exerted 4 percent less energy than they did in other racing shoes. They estimated such a savings would translate into a performance improvement of around 3 per cent for a 3-hour marathoner. The findings here were similar, but showed a smaller effect. The New York Times analysis found Vaporflys to be 1 per cent faster than the next-fastest running shoe (the Nike Streak), but the sample included a much wider range of athletes, and included men and women.
Finally, runners who improved their performance in Vaporflys and then switched to other shoes got slower. There were very few runners in the data who met this criteria: just 24 such races. But these runners finished about 7 percent slower on average than they did while wearing the Vaporflys. (Of the 24 races, 17 races were slower, and seven were faster.)
– New York Times
