No Sean McVay in PFF’s Top-10 coach rankings is just wrong

Data points don’t connect, sorry

A new and unique way of looking at data is not an easy task. For most of the mainstream data, the field of new perspectives has been picked over. That, in turn, incentivizes new ways of looking at the data, sometimes with disastrous results. It’s the age-old adage of putting the cart before the horse, a descriptive term for trying to jump to conclusions that really are not supported by data. Or, as we will see, letting data drive us down a blind alley.

A more current example is that of Pro Football Focus’s 2022 NFL Head Coach Rankings by Connor McQuiston. So how does he stumble in the effort? Well in his own words, he tries to eliminate all other factors and create an ‘average team roster’:

"Using this idea in ranking the current NFL head coaches, we try to do two things: 1) properly account for a team’s talent level, and 2) predict something less volatile than wins. We do this by creating a multilevel model where the fixed effects are the salaries of each starter on both sides of the ball, including an indicator for if the player is a rookie, and the target is points scored or allowed in a season. A starter is defined as the player who took the most snaps at their position, filtering out key injuries. This was altered for several positions, such as the top three players being considered for wide receiver, cornerback, safety and linebacker, as well as the top two players qualifying for guard, tackle, edge and interior defender. – methodology used in PFF 2022 NFL Head Coach Rankings"

In the end, the ‘telling point’ is setting a mean of points scored and allowed, then defining the difference between what the offense scores in terms of points in a season, and then how many points the team allows in a season.

Wrong conclusion? Wrong method

See the problem? In the effort to quantify an objective output, he infuses a very subjective method. How did he eliminate running up the score, versus teams that sat starters to avoid scoring differentials? How did the author account for player injuries? How did the author account for the average age of a roster? Randomly restating some factors, while ignoring other factors does not sanitize the conditions of the population.

Rather, the exercise may actually exaggerate the inherent bias. We know that QB Tom Brady won while playing for HC Bill Belichick. Did he falter with the Tampa Bay Buccaneers when he signed with them? No. In fact, he won a Super Bowl with the Buccaneers.  What about QB Jared Goff’s ability to win games with the LA Rams. How did he perform with the Detroit Lions? How did the 0-3 postseason track record of QB Matthew Stafford respond to playing for Sean McVay?

Goff and the Detroit Lions were 3-13-1. Stafford and the LA Rams were 12-5 and were 4-0 in postseason play. And that is only good for ranking HC Sean McVay at11th place?