Introduction
Hello,
We are halfway through the 2023 NFL season, and I wanted to do a quick look into some metrics pertaining to the QB position.
The two main ones are Completed Air Yards (“CAY”) and Success Rate (“Succ%”)
Completed Air Yards are the amount of yards traveled past the line of scrimmage minus Yards After Catch. Success Rate measures if a play gains at least 40% of yards required on 1st down, 60% of yards required on 2nd down, and 100% on 3rd or 4th down.
I decided to look at CAY because it controls for Yards After Catch, and may be more indicative of a QB’s passing abilities. For instance, if a QB throws a slant for 5 yards, but it goes for a 50 yard TD, that was less a consequence of the QB and more a consequence of the receiver and blocking by teammates. Granted, football is a team sport, and even just looking at that is not a comprehensive look at the greatness of a QB; however, that coupled with Succ% gives some insight I think.
Again, not comprehensive, and that is not really what I am trying to go for anyway, but this is more so just to share some interesting findings.
Methodology
All data was taken from Pro Football Reference. I only chose QBs with a minimum of 97 pass attempts. The different metrics I looked at include CAY, CAY/PA, CAY/Cmp, and Succ%.
After copying all the data, I found the Z-score for each individual and their corresponding metric. Then, I did a two-tail test to see which Z-scores were statistically significant. I set alpha to 0.05.
Findings
The table above is the dataset pertaining to each individual QB with their corresponding metrics.
This table shows the Z-scores for each individual QB and their corresponding metric. The areas highlighted in green are Z-scores that were determined to be statistically significant.
Tables 3-5 just show the graph version of the datasets from tables 1 and 2.
Z-Score Discussion
The only areas that were found to be statistically significant were the following:
P.J. Walker - CAY (lower end of distribution)
P.J. Walker - CAY/PA (lower end of distribution)
Joe Burrow - CAY/Cmp (lower end of distribution)
Brock Purdy - CAY/PA (upper end of distribution)
Brock Purdy - Succ% (upper end of distribution)
Matthew Stafford - CAY/Cmp (upper end of distribution)
The higher the success rate, obviously the better, and with Purdy his success rate is statistically significant. Some QB’s that are close to the upper end (but for whom we would not reject the null hypothesis for) in terms of success rate only really include Allen, Tua, and Goff. On the lower end, Watson, Walker, Wilson and Jones are close, but again, do not cross that critical value of -1.959964.
Limitations
This is not all-encompassing. As far as assessing the QBs, Completed Air Yards and success rate are only part of the equation. There are non-football stat related things to keep in mind, such as personality, experience, age, teammates, and coaches. Taking those into account can help explain why the data is the way it is.
Additionally, a QB can make a great play not through the air. They could very well rush for a big gain or even a TD. This analysis only views the QB position through the prism of passing.
Conclusion
Overall, I think this is interesting, but incomplete. Further inquiry can be made to give a more comprehensive view of the QB position. However, with the metrics we are examining here, I think it is a decent start. Another analysis of this, with more metrics and at the end of the season, would probably be more insightful.
You must log in or register to comment.
This is interesting. While I think people work too hard to take YAC out of the equation (it can 100% be a QB skill based on context), I like this. What I’ve noticed about my team is that we’re at our best when Burrow can pick apart teams playing cover 2 by finding short-intermediate holes in zones. Not only is very good at reading a play, but he’s great at looking off defenders trying to read him and making an accurate pass. This also just makes our team better.
Burrow (and the offense in general, but mostly Burrow) gets a lot of shit for not scoring a lot of points, but we win a ton of games because the defense is good and when Burrow is operating at his best, they rarely have to be on the field. Couple that with our defense allowing a ton of yards, but coming up clutch, and you don’t get many high scoring games at all.
Because YAC can sometimes be a WR stat that inflates one QB’s production because he has better WRs compared to another who’s WRs may not be gaining those extra yards on the same exact type of throw.
It’s a blend of QB/receiver skill/scheme/defense these things are hard to measure. It’s pretty obvious that incredibly accurate QBs like Drew Brees or Joe Burrow have massively helped their receivers with excellent accuracy, not just getting the ball to the receiver, but hitting them in stride so they can maximise their YAC. If PFF ever charted games from the 80s I’m sure they’d see the same thing with Joe Montana.
Right lol. I even said it can be a QB stat based on context, but the QB can be a huge part of YAC. Sometimes it is a guy like Ja’Marr Chase getting the pass and just making something out of nothing… but usually it does have to do with accuracy, decision making, anticipation, and timing. That’s why the best QB’s lead in YAC every year. Patrick Mahomes is always near the top.
First off, thanks for putting this together. Interesting stuff to dig into.
For the two-tailed z-scores does that mean looking for scores greater than/less than +/- 1.96 or am I thinking of something else?
On the success rate numbers, are those only for pass attempts or do they include things like QB scrambles? I noticed that Mahomes has a decent success rate but is below average in these passing metrics. Could that be due to his ability to pick up yards/first downs with his legs? I’m assuming success rate doesn’t distinguish between air yards and YAC, so for somebody like Mahomes (Ridder has the opposite thing going with bad success but decent CAY) can we attribute the success to YAC or is that an unjustified leap?
1.) If alpha is set to 0.05, for two-tail, you find the upper and lower critical values (all you need is to find one and use symmetry to find the other). So after doing the math, the closest value on the Z chart is 1.959964. Anything between -1.959964 and +1.959964 is not statistically significant, so we therefore fail to reject the null hypothesis.
2.) Success Rate is only looking at passing. There is a separate Succ% for rushing
3.) I think Mahommes is having a down year based off his previous seasons. Could be due to Kelce injury, strength of opponent, and that flu game. May not explain all of it, but maybe some.
4.) Not sure I understand your last question. A 6 yard gain on 3rd and 10 would not be a successful play, and if it were accomplished with his legs it would not count toward the Succ% above
The removal of YAC as a way to determine QB skill is pretty dicey. Short passes that go for a lot of YAC aren’t necessarily easy throws. Oftentimes the windows are small both in terms of surface area and temporal length. Being able to create space on short passes with a QB’s eyes is also a skill.
Additionally, while saying YAC is largely a function of receiver play and good blocking may have some truth to it, so does CAY. A receiver like Davante Adams or Megatron who can reliably gain separation deep will inflate a CAY just like how a receiver like AJ Brown or Deebo Samuel can inflate YAC.
Offensive line pass blocking skill also systematically biases CAY values. Poor pass blocking lines or average lines who have to face formidable defensive lines will necessarily have game plans to neutralize this issue. You will see a lot of slide protections and screens, which is a choice made somewhat independent of QB skill.
These analyses are interesting, however I would caution using these numbers as a way to rank QBs against each other when there is this amount of noise in the data. This is a theme true for many football statistics, not just CAY.
However, I do see practical value on using CAY on a case by case basis. If there are high CAY values despite conditions being hostile towards it, as determined qualitatively on film (poor pass blocking line play, game plans reliant on shorter throws for more YAC, etc…), then I would argue there is a strong indication of a QB’s deep ball skill. Vice versa also applies for poor deep ball skill.
Additionally, while saying YAC is largely a function of receiver play and good blocking may have some truth to it, so does CAY. A receiver like Davante Adams or Tyreek Hill who can reliably gain separation deep will inflate a CAY just like how a receiver like AJ Brown or Deebo Samuel can inflate YAC.
I see what you mean, and maybe you’re right, but idk if CAY would be inflated to the same extent. A screen pass is sometimes thrown behind the line of scrimmage which means the CAY is 0, but the passing yards (and YAC) could go for many yards. Idk if that’s a useful way of looking at how well a QB throws the football. Not to say it isn’t useful at all, it obviously can increase a teams EPA/Play or per dropback for the QB.
Offensive line pass blocking skill also systematically biases CAY values. Poor pass blocking lines or average lines who have to face formidable defensive lines will necessarily have game plans to neutralize this issue. You will see a lot of slide protections and screens, which is a choice made somewhat independent of QB skill
Thanks, forgot to list this as one of the limitations at the end. I remember thinking that after I listed “coach”.
These analyses are interesting, however I would caution using these numbers as a way to rank QBs against each other when there is this amount of noise in the data. This is a theme true for many football statistics, not just CAY.
Yeah, and like I mentioned, this is only part of the equation. A more comprehensive look can (and should) be taken into consideration down the line, probably at the end of the season when there is a larger sample size.
The comprehensive evaluation already exists. It’s called film study. Luckily we can access it for free on youtube from former NFL quarterbacks such as JT OSullivan and Kurt Warner. There are also people like Greg Cosell and Brian Baldinger at NFL Network.
Ultimately any omnibus number that tries to rank a QB’s overall play against the other, even if it has a number of variables feeding into an equation with an effect size as an output, will strip valuable context from the discussion. A QB on one team may look terrible in one place but amazing in another or vice versa (see Tua, Geno Smith, and Watson.) Some QBs are exceptional in some areas but severely lack efficacy in others (See Cam Newton). To put it in statistical terms, football is a gnarly mess of difficult to measure interaction effects.
I would advocate for qualitatively assessing traits and only using quantitative assessments as supplements. With any statistical analysis we need to understand the context in which the numbers exist or we’re prone to misinterpretation.
Chasing down a rabbit hole for a single way to rank quarterbacks in 2023, with our currently measured variables, is unlikely to yield a practically valuable outcome.
PFF does a good job for qualitative analysis. Additionally, and I said this in the beginning of the post, but this wasn’t meant to be a comprehensive evaluation.
It is not possible for PFF to do a good qualitative analysis the way they are doing it. They are assigning an omnibus number to a multifaceted construct with serious myriad interaction effects. Their rankings are asking a question that’s inherently leading us away from useful information.
My beef isn’t just about the evaluation. The stats are the stats and I trust that you’ve done them appropriately. It’s that the entire approach, even downstream. Even if you added every single variable available on football reference, we still wouldn’t have truly actionable information. Good outcomes come from schemes that fit with good player traits and the stats on PFR do a poor job at measuring both those things.