The following is a repost from FantasyDouche.com. Original post date: April 19, 2012.
Last night I suggested on Twitter that the Jimmy Johnson Chart has another problem that people seem to overlook. Everyone knows that NFL talent evaluators often disagree on the rankings of prospects. That’s actually to be expected. Reasonable people can often disagree. But it does present a problem for the Jimmy Johnson Chart. The Jimmy Johnson Chart says that the universe of NFL abilities is one made up of a very few elite performers. The falloff from the top picks is steep.
But if reasonable experts often disagree as to the rankings of prospects, then the talent curve is not shaped like the Jimmy Johnson Chart says it is. The Jimmy Johnson Chart says that the talent curve is steep. But disagreement among experts implies that the talent curve is flatter (flatter like the kind of flat I proposed the other day).
Let’s take this year’s wide receiver class as an example.
Below I’ve listed the top three wide receiver rankings of three NFL talent evaluators (media, not team employed). None of them share the same ordering of prospects. That strikes me as reasonable. Josh Norris has Kendall Wright as the top WR, Mike Mayock has Blackmon, and Bleacher Report’s Matt Miller has Alshon Jeffery on top.
I can anticipate an objection where someone might say “Yes, but this year’s wide receiver class is an outlier and usually talent isn’t that flat. Usually it follows a steeper falloff as shown in the JJ Chart.”
But the problem with the JJ Chart is how steep the falloff is. The JJ Chart isn’t believable based on what we can observe of the range of human abilities. The first overall pick value is 6 standard deviations above the average pick value. That implies the existence of individuals with other-worldly football ability. If individuals like that actually existed, they would be so obvious in talent that there would be no debate as to their ability or where they might rank among their peers. Six standard deviations implies almost super human ability. I can’t think of a single individual that has entered the draft without debate as to their abilities. Peyton Manning wasn’t even the clear cut #1 selection. He turned out to be better than Ryan Leaf, but there was no consensus pre-draft. Even Calvin Johnson, who is a physical freak and was an accomplished wide receiver in college, is only about 3.2 standard deviations above the other receivers in his class in terms of receiving yards. That’s actually really good. It’s ridiculous actually. But it’s nowhere near 6 standard deviations and he’s the the closest thing I can think of to an other-worldly individual in terms of football ability.
Here’s a graph which shows how many standard deviations above average each pick in the draft has averaged in the past 20 years (wide receivers only). There are no receivers who are 6 SDs above average, and when you then average it for each pick, the numbers come down to earth even more (Keyshawn Johnson and Carlos Rogers pull the numbers down while Larry Fitzgerald, Andre Johnson, and Calvin Johnson push the numbers up).
Now let’s look at that same trendline when shown along with what the JJ Chart says we should expect from each pick relative to the average. Again, the JJ Chart starts at 6 SDs above average. But the expectation for actual wide receiver performance says that the very top is closer to 2 SDs over average.
The problem with the JJ Chart is that it asks us to subscribe to a view of human abilities that doesn’t actually exist. When it does that, it misprices the value of the assets that it purports to be a price sheet for.
Teams that use the Jimmy Johnson Chart to buy picks are operating under a view of football abilities that isn’t supported by reality. Teams that use the Jimmy Johnson Chart to sell picks are taking advantage of the teams willing to buy on those terms.