Recently, a pair of film-based evaluations of wide receiver prospects Laquon Treadwell and Sterling Shepard caught my eye. ProFootballFocus and NFL.com’s Matt Harmon both seem to agree that Shepard is great while they seem to disagree drastically on Treadwell.
This reminded me of a question I asked myself last year regarding Kevin White and Breshad Perriman. It also seems to confirm one of my biggest concerns about film-based scouting, even film-based methods that aim to be as objective as possible like the ones PFF and Harmon use.
Can You See Age on Film?
That was the question I was asking myself last year. The single biggest difference between White and Perriman was their age, a factor that was in the younger Perriman’s favor. Yet that didn’t stop White from being drafted 19 picks earlier in the 2015 NFL Draft.
So why was that? I think the simplest explanation may simply be that White had “better film.” But we know that it’s important to age-adjust production for prospects and NFL players. Specifically, we need to hold older prospects to a higher standard. With that in mind, if we expect older players to be more productive, to quite literally be better, wouldn’t we expect them to also have “better” film? I think we have to at least be open to that possibility.
This brings us to Sterling Shepard. PFF ranked him as the best WR in the nation. Harmon, who aims to objectively break down WR film with his Reception Perception series, called Shepard, “…bar-none the best route runner coming out of college this year.” So even as someone who is relatively skeptical of the utility of film-based scouting, I took note of the high praise.
The wet blanket here is that Shepard is fairly old. Shepard was 22.9 years old at the end of his final collegiate season. So while Shepard may be more refined than most, you might expect that just based on his age. I should mention that Shepard was PFF’s fifth-ranked WR going into the 2015 season, but that was based off his work in the 2014 season when he was 21.9 years old, older than some NFL rookies.1
Let’s compare that to Laquon Treadwell. PFF’s Sam Monson recently argued that Treadwell is not a top WR prospect. Monson says that Treadwell is only their 13th-ranked WR in this draft class. But nowhere in his article does he acknowledge Treadwell’s age at all. Treadwell is the second youngest WR in this entire class. Now if we should be age-adjusting production, shouldn’t we probably be age-adjusting film? I think so. I think it has to be considered that they are likely being far too critical of Treadwell and far too generous to Shepard due to a failure to properly calibrate their expectations based on age.
You may be wondering what Harmon thinks of Treadwell. Well, that raises another concern…
How Objective Can You Really Be?
While PFF dislikes Treadwell, Harmon absolutely loves him, comparing him favorably to Alshon Jeffery, Allen Robinson, and Dez Bryant. They don’t just differ in their overall evaluations but even in the specifics. PFF cites an inability to separate as one of Treadwell’s two biggest weaknesses. Meanwhile, Harmon thinks Treadwell is better at gaining separation than he is generally given credit for and doesn’t hesitate to call him the best WR in the class.
If PFF and Harmon are ostensibly doing the same thing, trying to objectively break down film on a play-by-play basis, why are they getting such disparate results? I don’t have a truly satisfying answer. One possible explanation is that their methods are inherently so subjective that it doesn’t matter if you try to translate them to objective results. Another possibility is that while they’re seemingly trying to describe the same thing, such as “separation,” what that thing means to them is fundamentally different. If that’s the case then it presents a fundamental communication problem, as readers are likely understanding what the writers are communicating in a fundamentally different way than the writers intend. The one thing I do know is that if they’re getting wildly different results then I don’t know whose evaluation I should put more stock in, which makes it difficult to put much stock in either.
I really don’t mean for this to seem like a hit piece. On a purely selfish level, good predictive film-based scouting methods would be another tool to use to win at fantasy football. I’ll take all the help I can get there. On a more selfless level, I was excited when PFF first announced they would be evaluating college players and I want them to succeed. Likewise, I really enjoy Harmon’s work and I don’t think NFL.com could have hired a better guy.
I do believe that I should at least try to offer potential solutions to the problems mentioned. One thing that I think both PFF and Harmon could do is take the logical next step and test their numerical findings for predictive value or incorporate them into a larger predictive model. If they could show predictive value it would alleviate a lot of my concerns. I personally would be less concerned with disagreements so long as both models were shown to be significantly predictive. This would also give them a relatively simple means of incorporating age. Realistically, I think it would be more feasible to incorporate their numerical findings into a formula that incorporates age than it would be to meaningfully apply age to their current frameworks.
- This also raises the possibility of confirmation bias since they already thought Shepard was good, but I’m not particularly concerned about that. (back)