clock menu more-arrow no yes

Filed under:

Pac-12 & Computer Formulas: Accuracy Through Week 3

New, 7 comments

How are FPI and SP+ doing so far? (Also, promise this article is shorter than the last one)

NCAA Football: Arizona State at Michigan State Mike Carter-USA TODAY Sports

(As one kind reader pointed out, the previous breakdown of FPI and SP+ rankings for the conference was Way. Too. Long. So today, I'll just cover one part and keep it relatively simple. Thanks for reading!)

Tracking Computer Predictions' Accuracy: an Intro

There are many computer-formula-based rankings out there to track the quality and success of college football teams. Two of the main ones – both of which are now part of ESPN's resources – are the Football Power Index (FPI) and Bill Conley's SP+; they track (and rank) teams' quality overall, as well as for offense, defense, and special teams. Importantly, these formulas were used for preseason predictions for each FBS team, game-by-game for the entire season: both formulas publish a team's chance of winning (or losing) in any given game, and SP+ also puts forward the anticipated margin-of-victory (or loss). Each algorithm also updates its predictions as the season goes on: FPI re-calculates every game left on the schedule, while SP+ publishes updated predictions for both chance-of-winning and predicted point margin. Updated FPI rankings can be found here, and both details (e.g. offense/defense/special teams) and chances-of-winning can be found by clicking a given team's name on that list (so our Golden Bears would be here). Updated SP+ rankings pop up on ESPN on Sundays, and the link is easy to find through Bill Conley's Twitter account. Bill also puts up the updated win chance and margins for upcoming Friday/Saturday games on Wednesday or so - here’s last week's, for reference.

I've been curious: which formula is more accurate at predicting outcomes? Well, this little series will track who's been better over the course of the season. Enjoy!

The Schedule so Far (Post-Week 3)

First, a refresher on how the teams in the Pac 12 have done so far. The full schedule of game results – and margins of victory – are below.

A table showing the game outcomes for every Pac 12 game through week 3.
Pac-12 Schedule & Results through Week 3
Margins-Of-Victory by Game, with Totals and Averages Overall and in-Conference

A few things stand out: first of all, only 4 teams are undefeated (Cal, Wazzu, ASU and Utah), while 5 are 2-1 (Oregon, Washington, Arizona, Colorado & USC), 2 are 1-2 (Furd and OSU) and UCLA is the sole win-less team. Only 2 teams have a season-long negative point margin: Furd at 33 total (11 per game) and UCLA at 53 total (17.7 per game). Cal (21 points/7 per game) has the fewest points of the undefeated teams, a full 5.7 PPG behind ASU (38 total/12.7 PPG), while Utah (67 total/22.3 PPG) and WSU (100 total/33.3 PPG) are much higher. Oregon (97 total/32.3 PPG) is actually in 2nd place thanks to its blowout wins over Nevada and Montana (negating its relatively close loss to Auburn), and UW is close to Utah at 64 points in the green (21.3 PPG). The rest of the teams with positive point margins are hovering between 6.3 and 12.7 PPG, showing little room for error going forward.

The Preseason and Weekly Predictions: FPI & SP+

How well did the computer formulas do? First, we've got to revisit exactly what they predicted. The following tables show the preseason predictions (percentages for both FPI and SP+, plus point margins for SP+). On the right side is the likely wins in the season and in-conference: the "overall" and "Conf W-L” are the sum of each game's percent chance, while the "Raw W-L” and “Raw Conf” track how many games are either over 50% (“W”) or under 50% (“L”). After the first trio of preseason predictions, you’ll find the week-of predictions for each game.

FPI Preseason Projections: game percentages and total wins
SP+ Preseason Projections: game percentages and total wins
SP+ Preseason Projections: point margins per game

I've done a full breakdown of this in prior blogs, but long story short: both formulas saw the Bears with a losing season, while there was a HUGE disagreement on the quality of UCLA (FPI was much more confident in the Bruins, and has also been proven wrong). As many expected, Washington and Oregon would be battling for the North – and SP+ even favored the Huskies in every game. Meanwhile, Utah was almost guaranteed the South – though ASU could be a dark horse with the "raw" wins and losses. UW would have the overall largest point margin per SP+, and of the 4 teams with a negative overall margin, Cal would be closest at -1.8 PPG over the course of the season (although a tough Pac 12 schedule would make it -5.3 PPG in-conference).

Weekly Predictions so Far…

The computer formulas publish weekly updates – both have percentage likelihood, and SP+ adds point margin. Here are the week-by-week projections through Week 3.

FPI Weekly Predictions – chance of winning
SP+ Weekly Predictions – chance of winning
SP+ Weekly Predictions – point margin

A couple interesting things come through… After week one, both formulas knocked the Bears down a notch for their Battle in Seattle (a 4.1% drop for FPI and a 5% drop for SP+), and SP+ put us as a full 20.7-point underdog the week of the game. Furd tumbled downward from their preseason projections for both week 2 & 3, with FPI giving the biggest hit pre-USC and both knocking off over 20% pre-UCF. Although both formulas had Arizona as the favorite against Texas Tech in the preseason, they put the Wildcats as an underdog the week of the game; so turns out the early projections were right. Finally, ASU had a HUGE upset – almost as big as Cal @ UW – per both the preseason and weekly projections (and the Spartans were favored even more by Saturday morning than they were in August). Given FPI's favoritism toward UCLA, the projected likelihood of a win v Oklahoma was, from highest to lowest: preseason FPI (38%), week-of FPI (20.3%), preseason SP+ (13%), and week-of SP+ (8%). Bill Conley already gets brownie points on that one. Finally, turns out that SP+’s preseason margin for Cal v North Texas (8 points) was better than the week-of margin of 12.

Accuracy – Pure Wins and Losses

So, it's time to look at the accuracy of the formulas, just with projected wins and losses, through the course of the season so far. The first couple tables will show whether the preseason projections were right, and the second duo will show the accuracy of projections for each week. At the bottom is the accuracy for each game so far: unlike last week, it doesn't duplicate games when 2 conference teams play each other, so it's just the accuracy game-by-game.

FPI's Accuracy from the Preseason – Pure Wins & Losses
SP+ Accuracy from the Preseason – Pure Wins & Losses
FPI Accuracy Week-by-Week – Pure Wins & Losses
SP+ Accuracy Week-by-Week – Pure Wins & Losses

First, the obvious: SP+ was more accurate than FPI for predicting games, both in the preseason (27-7 for SP+ vs. 25-9 for FPI) and with weekly predictions (27-7 vs. 26-8). Both computers got the same 7 games* wrong in the preseason, while FPI also got UCLA-Cincinnati and USC-Furd wrong. Week-of switches made both formulas correctly predict a Colorado win over Nebraska after the first week of play, but incorrectly turned Arizona from a preseason favorite over Texas Tech to an underdog by Week 3 – so that was a wash as far as levels of accuracy were concerned. FPI's appropriate switch in week 2, putting Furd as a preseason favorite over USC to making them an underdog, ultimately improved its record from 25-9 up to 26-8 – but it was still behind SP+ overall. Looks like Bill Connelly's formula is in the lead so far.

*Arizona-Hawaii; Washington-Cal; Colorado-Nebraska; UCLA-SDSU; ASU-MSU; Colorado-Air Force; and USC-BYU.

Changes Going Forward: Updated FPI Projections

Finally, we can look at how the first 3 weeks influenced FPI's projections going forward. The next couple tables show the updated season projections, with existing games as either 100% (win) or 0% (loss). It also includes updated total wins and losses over the season, both as the sum of game chances in percent, and in the "raw" wins and losses – both sets of numbers are listed for overall records and in-conference.

Updated FPI Season Projections – Post-Week-3
FPI Change – Updated v Preseason – Post-Week-3

First of all, the Bears are the biggest winner for the entire season in terms of overall wins and losses, jumping up 1.6 wins from 4.9 to 6.5 (hooray for winning records!) and are tied with WSU for a "raw" jump of 2 games. The Bears have also moved up 1.2 games in-conference – which is 2nd behind USC's 1.3 games and #1 in the north – and are tied with 2 other teams (WSU and USC) for a 2-game "raw" in-conference jump. Granted, a lot of that is because of the largest upset among the conference against UW, but it's still encouraging. Meanwhile, UCLA is a dumpster-fire with every single game moving downward and a season-record drop of 4.2 games – over 2.5 times next in line, Stanfurd – with a record 6-game "raw" drop (it's hilarious that FPI had projected them as a 7.7-win team, favored in 9 games). Good times!

So – that's all for now. Keep an eye out later this week for an update on the conference rankings for both FPI and SP+, including all offensive and defensive performance. Go Bears!