In Part I of this series I started off by pointing out an inconvenient truth in the friction between tape grinders and stat nerds, and that is that when it comes down to it and it’s time to decide on what is true, we’ll be talking about a number. We can start out by talking about a running back’s vision, or balance, or whatever, but eventually when it’s time to grade the graders, or evaluate the evaluators, we’ll be talking about rushing yards, or yards per carry, or some number.
In this post I want to move the discussion ahead a little by posing a question. In the false dichotomy that I’m presenting where the stat nerds are on one side and the tape grinders are on the other, how would each side go about improving their process? So let’s say that on a systemic level, you wanted to improve player eval among scouts, how would you do that? I’m not talking about improving the eval of an individual scout. Individuals can get better with more experience. I’m talking about improving the system. Because I think you could argue that since scouting is what it’s always been, which is a collection of opinions among individuals, that from a systemic standpoint it would have a difficult time improving. Whatever errors are built into the process are likely to sustain.
Except that’s not exactly true. It would certainly be possible to improve on scouting as a whole, but the improvements would for the most part start to look more like data analysis. If you were a team interested in reducing your draft misses and increasing your draft hits you might go back and figure out what seems to be working. But the problem with that is that the more careful you are about figuring out what worked and what didn’t work, the more it would resemble data analysis. It would start to include a lot of isolating of variables (even if they’re not box score stats) and backtesting to check the validity of your assumptions.
That leads me to the other side of the question I originally posed because the stat nerds have ways to improve what they’re doing. They can improve the models they use and they can also increase the amount of data that they analyze. For examples of both improving models and increasing datapoints, you can look at baseball. Bill James’ early work was impressive given that he was basically clipping box scores from newspapers, but it also doesn’t resemble the analysis that’s going on today. Baseball has both improved upon early models with new stats like BABIP and FIP, and it’s increased the amount of data that’s being collected with PitchFX data. The same thing has also happened in basketball, where the models continue to evolve, and the amount of data being collected has gone parabolic with the introduction of the SportVU system. Actually, these things are sort of already happening in football, even if we don’t have player tracking yet. PFF is tracking non-box score statistics and Mike Clay is doing the math on things like what percent of passes a QB should complete at various depths of target. So it’s already on the way in football.
There are two related positions that I see sometimes associated with the tape grinders who also happen to be anti-advanced stats (it’s a subset at this point, as I think a lot of the younger guys are open to the idea that data can add to their process) and which I find problematic. The first position is “Well, you can’t do that with numbers.” and the second position is “Well, you can’t do that with numbers yet… maybe someday we’ll have the ability to analyze that.” These two positions are bros in terms of dismissing the value in advanced stats before anything has even been tried. They’re also essentially religious positions because no amount of evidence to the contrary would change the minds of their adherents. But the problem is that the road to having a football equivalent of a SportVU system is not paved with “Well, there’s no point in trying numbers.” That road will be forged by getting as far as we can with conventional statistics and then turning to optical player tracking because there’s no more wood to chop in the box score.
I actually think there’s a huge place for scouting in the move to improve player evaluations, even if I think that left on its own scouting would pretty much just keep bumping along. Think about how many hours the draftnik community is spending watching college tape from February to April . What if they came up with some standards that they could all agree on and just started recording the data, instead of leaving it in informal player notes? They already have PBP databases that are available. They could build on top of those DBs by tagging each play with “clean pocket” or “good blocking” or “missed tackle” or “hurried throw” or “2nd read”, as well as record things like air yards and amount of time the line gave the QB to throw. Then what if between April and September they all went back and decided to collect data on one college game per week? How long would it be before they had a PFF quality historical database that they could backtest against? So basically take the PBP dbs that already exist and then improve them with crowdsourcing among other draftniks.
I know that there are probably tape grinders reading that thinking “I don’t need to collect more data, I already watch… I know what I’m looking for.” But I guess that’s sort of the problem. It’s not that player eval right now is bad. It’s that in order to improve it has to become more like data analysis. I have more thoughts on this that I will throw out in Part III.