Welcome to the 24th installment of the “The Wrong Read,” an article series that reflects on recent podcast episodes, pushing the ideas discussed on the podcasts to their logical conclusions and offering some further thoughts on the topics broached by the guests and hosts. Now that the Super Bowl is well behind us, it’s prospect season. You should pay special attention to rookie running backs.
Baker Mayfield is rightly a RotoViz favorite among 2018 QB prospects. Mayfield’s 2016 AYA of 12.3 is the second highest since 1956. The only player to have a season with a higher AYA: Mayfield, in 2017, with 12.9. To give you a sense of how insane that is, no other FBS QB has put up more than 12 adjusted yards per attempt in a season in over 60 years. Mayfield has done it twice.
The latest episode of RotoViz Radio ended with a discussion about the 2018 QB class and specifically the possibility that the Browns could select Mayfield with one of their top-four picks. Although guest Matthew Freedman was skeptical they would make such a move, all agreed that Mayfield should be in consideration for the first pick.
From a statistical standpoint, it’s hard to argue that the Browns shouldn’t consider Mayfield at No. 1 overall. But what do Mayfield’s absurd college statistics really tell us? Specifically, what do they tell us about what kind of NFL player Mayfield might be? Before trying to answer this question, it’s worth asking whether Mayfield’s statistical profile is even a reliable part of his resume. That question sounds ridiculous, so let me explain what I mean.
Do Mayfield’s Stats Need Qualification?
One criticism draft analysts occasionally bring up regarding Mayfield’s dominance is that he plays in the Big 12, where defenses are weak and point totals are high. His AYA is a product of his offense-centric conference, so this criticism goes. If this were the case, we would expect to see high AYA figures across the board in the Big 12. However, it turns out the Big 12 is only slightly above average in AYA. Since 2010, the Big 12 ranks fourth in average AYA, behind both the Pac 12 (the conference in which other top QB prospects Sam Darnold and Josh Rosen play) and the SEC (a conference which is known for strong defensive play).
And this is despite the fact that several of the other top AYA seasons have been put up by Big 12 QBs like Robert Griffin III, Bryce Petty, and fellow 2018 prospect Mason Rudolph.
What’s more likely to inflate Mayfield’s AYA is the era in which he plays. Average AYA numbers have been steadily rising for years. It’s probably no accident that the two best player-seasons ever according to AYA took place in the last two years.
This fact somewhat relativizes Mayfield’s achievement from a historical perspective. But it doesn’t change his performance compared to his contemporaries. So we should not discount Mayfield’s statistical dominance because of the conference he plays in. And even though he plays during a pass-friendly era, he is still doing something unprecedented.
The Predictability of NFL AYA
Even so, there is a question as to whether whatever skill college AYA picks up is translatable to the NFL. Does a high college AYA signal a high NFL AYA? I ran some quick regressions to try to get at this question:
Don’t pay too much attention to the r-squared or the overall model—my aim wasn’t to build a model, but just to test the significance of various college statistics. Notice instead the t-values and p-values (the column labeled “Pr(>|t|)”) for the three variables I chose. A good rule of thumb1 is that a significant predictor requires an absolute t-value above 2 and a p-value below 0.05.2
The variable named College AYA is a player’s career AYA in college, while Best AYA and Final AYA are just what they sound like (and keep in mind that for many players these last two are the same). None of the three versions of college AYA is a statistically significant predictor for NFL career AYA. College career AYA comes the closest, but it still doesn’t meet the threshold for statistical significance. Visualizing this makes it readily apparent:
The r-squared here is 0.018—in other words, it’s random. I also tested unadjusted passing yards per attempt, which Mayfield also excels in. Because YPA doesn’t account for touchdowns or interceptions, it ought to be more predictable than AYA.
One issue with all of these metrics is that there is so much that goes into determining an NFL QB’s YPA and AYA, such as the offensive scheme, his weapons, and his offensive line. And all of those things affect a college QB’s stats as well. It’s extremely difficult to use these metrics to isolate a QB skill that translates to excellence in the same statistic at the NFL level.
Mayfield’s dominant YPA and AYA figures are impressive, but it’s by no means clear that we can predict this same level of success in the NFL. My point is largely the same as what Freedman said on RotoViz Radio: Mayfield does have within his range of outcomes a mediocre NFL QB. It’s not impossible for him to fail.
The Predictiveness of College AYA
Now, don’t misinterpret me: I’m not saying college YPA and AYA are meaningless. RotoDoc has found that a player’s final college season AYA is a significant predictor of NFL success in his QB model. He defines success as having an NFL season with an AYA of 7.0 or higher while starting at least eight games. So there is some connection between college AYA and NFL AYA. Mayfield’s final season AYA of 12.9 should give him a nice boost in RotoDoc’s 2018 model. So what’s going on here?
A further problem with translating college stats to the NFL is the issue of sample size. If you’ll allow a brief aside, I promise we’ll eventually make our way back to Mayfield. Among QBs who have thrown at least 100 passes in the last 15 years, Deshaun Watson has the highest career AYA at 8.40, just fractions ahead of Aaron Rodgers’ 8.36 (including playoffs). However, Watson has only attempted 204 passes, while Rodgers has attempted over 5,000. So it’s much more likely that Rodgers’ AYA is a reflection of his true ability than Watson’s is of his.
Relying on some work Danny Tuccitto has done at Intentional Rounding, we can calculate true AYA numbers by regressing each player’s observed AYA toward league average based on the number of passes that player has attempted.3 The difference this makes is massive: Rodgers’ 5,493 career attempts mean his 8.36 career observed AYA is pretty close to his true AYA of 8.11. But Watson’s true AYA dips to 7.12, because the much smaller sample of Watson’s pass attempts means his observed AYA is not as reliable as Rodgers’.
This turns out to be an important transformation: not only is true AYA (probably) a more accurate reflection of a player’s ability, it’s also (probably) more predictive of the future, and (definitely) more predictable from college stats.4 Here’s a simple linear regression that shows (to a degree) the usefulness of college AYA:
The r-squared is not good, but we’ve met our thresholds for statistical significance. The point is that although there doesn’t appear to be a direct connection between college AYA and observed NFL AYA, college AYA is a reliable indicator of some QB talent that’s translatable to the NFL. A machine-learning model like RotoDoc’s is better able to discern that signal and turn it into a useful predictor.
All of this is to say that despite the lack of a direct translation from college AYA to NFL AYA, Mayfield should still probably be the top QB in this class. By the numbers he’s arguably the best college QB ever. That seems significant.
- It’s more complex than this, but for our purposes this will do. (back)
- Testing the variables as a part of a multiple linear regression does change their significance slightly compared to a simple linear regression with only one predictor, but not enough to make a difference. The output of the multiple linear regression gets us close enough and makes the same point. (back)
- Actually, we have to regress YPA, touchdown rate, and interception rate individually, since they stabilize at different rates, and then recombine them into AYA. (back)
- “Predictable” is a weird word here—true AYA is not, after all, a real stat, at least not in the sense that it’s directly measuring anything that happens on the football field. It’s imputed based on league averages. So to say it’s predictable is somewhat disingenuous. The point is that whatever skill true AYA measures is something that college AYA seems to pick up on. (back)