I don’t really have an opinion on whether Johnny Manziel (hereafter “JFF”) will be a good NFL QB or not. The QB position is extremely difficult to project I think. NFL teams have a tough time scouting QBs. Media scouts make enough mistakes in their evals that if they actually ran teams, the mistakes would be enough to get them fired. Even the leading statistically based projection systems will have problems baked into their methods because there have been so few QBs to accurately judge whether the system even works or not. There probably aren’t enough QBs to actually hold out a test set from the training set and even know whether your model works on out of sample observations. So evaluating QBs is really difficult.
But in the case of JFF’s draft stock I do think there’s something interesting going on. Everyone points to Russell Wilson as a reason that it’s safe to draft JFF, presumably because the two players are similar size. But Wilson fell to the third round in his draft based on his size while JFF is likely to go top 5 in his draft. NFL evaluators applied the results of their years of watching football to Wilson, decided that he was worth drafting but also that a discount was in order due to his size. Then Wilson hit. But in JFF’s case they haven’t simply added the Wilson observation to their model and modified it. They threw out all of the information that they had that led them to discount Wilson. It’s a totally new model and one in which size is not a variable at all.
I’m sure that some people will point to the shared hand size that JFF and Wilson have and say that’s the reason that they’re both safe. Maybe that’s even true. But if most QBs fit into a certain mold based on their size, and then you have one QB that doesn’t really fit that mold, are you really scrapping all of the size consideration that was originally in your model?
It’s pretty common to see people claim credit for calling Russell Wilson even if their original evaluation said something like “if Wilson was 6’4” he would be a lock first rounder” and rarely said “Wilson isn’t 6’4” and to me that’s totally irrelevant”. I think the distinction between those two statements is meaningful. It matters because I tend not to think that one observation should be enough to nullify all of your size related concerns. To be clear, I don’t personally have size related concerns. I’m just talking about the way that QB evaluation is being exposed as a “cat on a hot stove” process.
Again, I don’t really have an opinion on whether JFF will be a good pro or not. He might be, he might not be. I’ll probably end up owning him in fantasy because he’ll pick up some rushing yards as long as he’s playing. But I do find it interesting that size as a component of QB attributes went from requiring a two round discount in the NFL draft, to requiring absolutely no discount.
To put it another way, the Seahawks drafted Russell Wilson where they couldn’t really go wrong. Whoever drafts JFF will be dealing with an entirely new set of probabilities. They’ll have a player of the size that has historically required a discount to draft and they will have gotten no discount. What if the historical discount was accurate as to the probabilities at play?