There's a bit more spread among keepers, where we have more data per keeper. Rimando is clearly very good. Donovan Ricketts is more likely bad. Jake Gleeson isn't great either. The Timbers sure know how to pick 'em. Given his recent playoff heroics it's affirming to see Zach Steffen pop up highly without that playoff data included.

To conclude, and summarize my main point again, because of small sample sizes it's very difficult to determine between good and poor penalty shooters only using the data that’s available from games. This is quite a lot of math to make a fairly intuitive point, but I already had half an article written with the math, and I’m going to come back to the method at a later date.

With that in mind:

After Atlanta United lost to the Columbus Crew on penalties last Thursday, there was a good amount of discussion around Tata Martino's admission that he did not have his team practice penalties in the game's lead-up. Martino's critics mostly argued that practice makes perfect, and that improving a team's ability there would pay huge dividends in that situation. Those on the other side maintained that penalties are mostly a crapshoot, especially given the pressure and exhaustion of the situation.

Both sides have some merit. It is certainly difficult to simulate that amount of pressure in practice, but committing penalties to muscle memory can go some way towards making that unnecessary.

Given the discussion I began this piece with, I think there's an even greater benefit to practicing penalties, though, and that's to figure out who is good at taking them. As I showed, small sample sizes make it very hard to pick out the most proficient shooters. To determine a difference with 80% probability between a true 80% penalty taker and a true 70% penalty taker, we would need to observe 231 shots from each shooter. That's a pretty substantial, maybe unrealistic, difference in ability, but a team still is never going to be able to measure it if they're only using data from actual games. If a team instead spends practice getting those reps- they'd need about 3 a day over three months in the example just given- the team can improve its knowledge of its players.

Former Philadelphia 76ers VP of Basketball Strategy Ben Falk recently talked about doing this exact thing in basketball, with three-pointers. In Philadelphia under Sam Hinkie, the coaching staff would track each player's three-point shooting percentage during practice. If a player wanted to be allowed to shoot threes in a game, he would have to hit them at a certain rate in practice.

Using practice data to assess a player's ability can, of course, can be applied to any measurable skill where in-game sample sizes may be too small - penalty kicks, free kicks, finishing, headers...

It's occasionally argued that players can't really improve their ability to shoot penalties through practice. It's something they either have, or they don't. They can either withstand the pressure, or they can't. I'm wholly unqualified to evaluate that point. But if a team can instead focus on picking the right penalty takers, it can improve its odds of winning a shootout by, say, 5%, or its odds of scoring a single penalty by 2%, and that's functionally the same thing as improving that skill individually.

There are some aspects of an in-game penalty scenario that practice can't perfectly replicate- pressure and exhaustion being the two biggest. But coaches generally have to be able to simulate these anyway - basketball coaches at every level run end-of-game drills, after all - so I do think there are ways around that.

An interesting further wrinkle is that penalties taken in practice are a repeated game. The same shooter will face the same keeper many times. This isn't the case in a live game. However, given the proliferation of data at the professional level, I would argue that we shouldn't think of an in-game PK as a one-off scenario either. Ideally opposing keepers and shooters have watched enough film or reviewed enough notes to have a good idea of each other's tendencies.

I should note lastly that it's not clear whether training time spent on penalties gives a club a greater return on investment than time spent practicing something else - set pieces, for example. Penalties certainly occur with low frequency, but when they do occur, it can be in a very high leverage situation, like, say, the knockout round of the MLS playoffs.

So there are some difficulties that mean using practice time to get more data on a team's penalty takers isn't necessarily perfect. But I think there are conditions under which it generally makes sense.