The folks here at RotoViz have been arming readers with all sorts of incredibly valuable information with regard to advanced metrics, similarity scores, and how those can allow you to arbitrage ADP. It made me wonder just how (in)efficient ADP actually is at predicting fantasy performance.
Raw ADP (i.e. overall selection order) reflects two things 1) the player’s predicted positional finish and 2) the relative scarcity of each position with respect to the others. In a world with perfect foresight, raw ADP would start with the highest scoring player at the scarcest position and work its way down from there. The best player from the second scarcest position is selected when his relative advantage over others at his position is greater than that of the next player down in the scarcest position, and so on and so forth.
ADP does tend to look like that (RBs at the top, followed by top-shelf WRs, Gronk, and then the QBs a little later), but my hunch was that ADP tends to be largely based upon a player’s performance in the previous year with a few other factors layered on top of that (change of team, new positional competition, offensive scheme/QB change, perceived injury probability, etc.) rather than a projection of the upcoming year.
The problem with the underpinning of ADP being the player’s prior year performance is that a good chunk of that tends to be touchdowns, and we know how those tend to regress. Of course, the amount of the “chunk” depends on the position we’re talking about. Here’s a breakdown of the percentage of standard fantasy scoring for all starters (top 12 QB, 36 WR, 24 RB, 12 TE) that comes from TDs vs. yards for each position from 1999 – 2012 (numbers courtesy of pro-football-reference.com):
|WR TD||WR Yd||TE TD||TE Yd||RB TD||RB Yd||QB TD||QB Yd|
|99 – 12 Avg||29.0%||71.0%||31.9%||68.1%||28.0%||72.0%||49.2%||50.8%|
|1 Std Dev||7.5%||7.5%||11.0%||11.0%||8.7%||8.7%||4.7%||4.7%|
In case you were wondering, the percentage breakdowns for each position remain remarkably consistent through time.
With between 1/3rd and 1/2 of a player’s fantasy production in any given season coming from a volatile source like touchdowns, basing the following year’s ADP on his previous year’s numbers would seem ill-advised at best and downright stupid at worst.
Before measuring just how much of a player’s ADP in a given year (call it Year N) is determined by his performance in the previous year (call it Year N-1) I first wanted to control for some of the variability in relative positional value from year-to-year. 2012 is a great example where more QBs went higher than ever in drafts because of the insane-o numbers that so many QBs put up in 2011. To do that, I decided to look at a player’s rank within his position in a given year both for his fantasy production and for his ADP, instead of his overall rank.
I also wanted to control a bit for the role that injuries may play in a player’s fantasy performance in a given year, so I decided to measure performance based on fantasy points-per-game (PPG) rank rather than on Total Points (TP) positional rank. For example, Tony Romo played 6 games in 2010 before he got pancaked by Michael Boley and missed the remainder of the season. He finished with a TP rank of QB 30, but on PPG basis he finished as QB 9. I think it’s more likely that his 2011 ADP would be based on his PPG rank than on his TP rank. I decided to use a minimum of 4 games as the cutoff so that some backup player who put up one or two crazy games and then vanished into oblivion didn’t get included in this analysis. By doing this, I run the risk of leaving out a stud player who got injured in a season and played less than three games (Jamaal Charles 2011), but I figured that scenario would be the exception to the rule.
A couple other things to note before I present the results:
● The ADP data is courtesy of myfantasyleague.com. Their data starts in the 1999 season. There may be other sites with more users and potentially “more accurate” ADP data, but MFL’s was the most readily accessible and covered a large enough timeframe that I thought it was an appropriate measurement of ADP.
● I used positional PPG ranks as my starting point and then captured that player’s ADP in the following season. This means I’m excluding rookie ADP in all cases, since they have no Year N-1 NFL season upon which to base their Year N ADP.
● Likewise, if a top-tier veteran decides to retire in Year N-1 (Shannon Sharpe in 2003) he is excluded from this analysis because he has no Year N ADP.
There are any number of ways to slice up the data, but here’s the way I did it. Feel free to question or debate my methods. I felt like this was the most relevant way to capture the data and present my findings, but I’m open to suggestions.
1 Because most leagues start only 1 QB, 2 RBs, 3 WRs, and 1 TE per team I decided to examine ADP on only the top 12 QBs, 24 RBs, 36 WRs, and 12 TEs on a PPG basis. I felt like scoring as the 18th best QB or the 32nd RB probably doesn’t mean that much to next year’s ADP, so I focused on the top-tier finishers.
2 Take Player’s Year N-1 positional PPG rank.
3 Find Player’s Year N positional ADP rank.
4 Do a linear regression of those two numbers to find their correlation.
5 I did several iterations of this exercise whittling down the population each time, because I was a little skeptical of the ADP data from waaay back in 1999. MFL shows the number of times each player was drafted and for the earlier years those numbers are only in the hundreds, leaving a lot more room for outliers to skew ADP. It looks like the participation on their site took a pretty big leap in 2005 and then again in 2007 (multiple thousands of drafts), so as you’ll see I reran the numbers on the slightly smaller (but potentially more accurate) ADP samples.
Here are the results:
DAMNIT! Not nearly as explanatory as I would have thought. Still, it does explain some of where a player’s ADP comes from. Now, for a comparison, let’s see how a player’s observed PPG positional rank in Year N-1 correlates to his Total Points positional rank in the following year. This should answer the question: “Is ADP reflective of prior year performance because that variable is predictive of the following year’s performance?” (I used Total Points positional rank in the following year because that’s how your fantasy team will actually score. PPG gets you nothing unless a player does it for a whole or most of the season)
Well that’s a resounding NO! But maybe, just maybe, the differences between prior year PPG positional rank and ADP are accurately capturing those differences from one season to the next…we’ll attempt to answer the question “Perhaps ADP doesn’t match prior year performance because drafters are accounting for all of the factors that will make the player’s performance different from one year to the next?” That’s actually just a statement with a question mark, but you get the point…Here’s the correlation between ADP and Total Points positional rank at the end of that same year:
Certainly better than blindly using PPG from the previous year to predict a player’s production, but still not great.
Ultimately this is a long-winded way of confirming that ADP is pretty inefficient at predicting a player’s performance in a given year because it’s at least partially based upon a player’s performance in the previous year, which actually has very little relationship at all to current year performance. ADP is indeed very ripe for arbitrage and something that you can use to your advantage when you pair it with the predictive tools here at RotoViz. Happy sniping…