With the end of the BCS and the rise of the college football playoff, coaches and commissioners have been paying more attention to strength of schedule, a critical component of the new playoff selection process. During last month's media days, coaches and commissioners from the Power 5 conferences addressed the discrepancy between eight-game conference schedules and nine-game schedules. While the ACC, SEC, and Big Ten play 8-game schedules, the Pac-12 and Big 12 play 9-game schedules. This may give teams from one scheduling format an advantage in qualifying for the upcoming playoff system. Stanford's David Shaw has strong words for members of the 8-game conferences:
"I've been saying this for three years now: I think if we're going to go into a playoff and feed into one playoff system, we all need to play by the same rules," Shaw said on the teleconference. "Play your conference. Don't back down from playing your own conference. It's one thing to back down from playing somebody else. But don't back down from playing your own conference."
By playing only eight conference games, these teams get an extra non-conference game. The problem here is that these extra non-conference games are usually played against pitiful teams. Let's look at the non-conference schedules of the teams who won their divisions in the 8-game conferences last season. These are the kinds of teams who will be competing for one of the playoff spots.
SEC
- Missouri: Murray State (FCS, 6-6), Toledo (7-5), Indiana (5-7), Arkansas State (8-5)
- Auburn: Washington State (6-7), Arkansas State (8-5), Western Carolina (FCS, 2-10), Florida Atlantic (6-6)
- Michigan State: Western Michigan (1-11), South Florida (2-10), Youngstown State (FCS, 8-4), Notre Dame (9-4)
- Ohio State: Buffalo (8-5), San Diego State (8-5), California (1-11), Florida A&M (FCS, 3-9)
- Duke: North Carolina Central (FCS, 5-7), Memphis (3-9), Troy (6-6), Navy (9-4)
- Florida State: Nevada (4-8), Bethune-Cookman (FCS, 10-3), Idaho (1-11), Florida (4-8)
Among these six division winners' 24 non-conference opponents, there are only two bowl-eligible teams from the Power 5 conferences. That's 8.3% of all opponents. Furthermore, none of the teams played more than one Power 5 opponent. In fact, half of them did not play any Power 5 opponents (although Notre Dame is certainly a worthy foe on Michigan State's otherwise flaccid non-conference schedule).
These teams won 23 of 24 of their non-conference games, with Michigan State's loss to Notre Dame as the only exception. In total, they defeated these teams 1062-330.
With such weak nonconference schedules, these teams' strength of schedule ratings should suffer, right? That's a point Pac-12 Commissioner Larry Scott kept reiterating during Pac-12 Media Days.
"Bottom line, the establishment of a playoff is a great development for the Pac‑12. I think it's going to be great news for us and our teams and the access we have to compete on the field for national championships more frequently. The other thing I'm particularly excited about in terms of the shape of the playoffs is the importance that strength of schedule will play. We know that there will be continued controversy and debate, but the clear statement has been made that strength of schedule is going to be a determining factor in figuring out which of the four teams ought to be competing in that playoff."
"With the most competitive nine-game conference schedule in the nation, our champion will be incredibly well-positioned in this first-ever historic college football playoff" - Larry Scott
Larry Scott believes that the 9-game conference schedule will give the Pac-12 and Big 12 an extra game against a quality foe, which will give them strength-of-schedule advantages over their counterparts in the SEC, ACC, and Big Ten.
I not so easily convinced.
Sure, it's a compelling argument when taken at face value. Yes, Michigan State will be penalized slightly for playing a 1-win Western Michigan team and a 2-win South Florida team. It seems they would be better off playing (and not losing to, of course) another team like Notre Dame with 8 regular-season wins. But that carries risk. Do you know what would be easier than playing an 8-win team? Playing 8 conference opponents whose records are 1 win better thanks to an extra non-conference game. Beat an 8-win team or beat 8 teams with slightly better records--the benefit to the strength of schedule is roughly the same over the course of the season.
That is exactly what is wrong with maintaining the discrepancy between the 8-game conference schedule of the SEC, ACC, and Big Ten and the 9-game schedule of the Pac-12 and Big 12. If some teams have the opportunity to get an extra win in non-conference play, the penalty for playing a weak opponent will be washed out because most of their conference opponents will have inflated their records by one win. That is how you get conferences like the ACC with eleven bowl-eligible teams. Can anyone legitimately challenge Florida State or Duke's strength of schedule under these circumstances? They had to battle through a conference with eleven bowl-eligible teams! Their conference schedule is so tough that they had to play some weaker teams during non-conference play to balance things out. I'm sure you have heard this argument before from other conferences.
So which argument is correct? Does the 9-game conference schedule give the Pac-12 and Big 12 an advantage? Or do the better records in the ACC, Big Ten, and SEC give them an advantage? We could debate this all day.
In fact, coaches and commissioners have debated this all year and we still don't know which system gives its teams an advantage. I want concrete answers. I am a man of science and we're going to solve this dilemma with science. Hold onto your butts.
Busting the Pac-12's 9-Game Schedule Superiority Myth
The Pac-12 currently plays a 9-game conference schedule. What would happen to its teams' strength of schedule ratings if they had played an 8-game conference schedule last season? That is the question we are going to answer today.
Because I cannot (yet) go back in time and watch multiple iterations of the 2013 season unfolding with 8 conference games and with 9 conference games, I'm going to use the next best option: simulation.
Here's what we're going to do. We will simulate a full season under both an 8-game and a 9-game conference format and compare the final strength-of-schedule ratings. The only difference between the 8- and 9-game schedules is that in the 8-game schedule I replaced a confrence opponent with a weak nonconference opponent.
I simulate the 2013 season 10,000 times using data from last year's games to predict the outcome of each game again. I don't expect the results from the season to be identical to last season's results, so I am allowing some variation in the outcomes (hence replaying each game, including conference games). Stanford will not always win 10-regular season games in these simulations, nor will Cal always win only 1. However, last year's performances strongly influence the outcome of the simulated games. If you would like to read more about the simulation process, please see the appendix. I aim to keep this part as free from math and technical details as possible. In each of these 10,000 simulated seasons, I simulate all 54 conference games as well as the nonconference slate. At the end of each season, I keep track of each team's total number of wins and its final strength of schedule.
With these results from the original schedule in hand, I consider a scenario in which the Pac-12 has 8 conference games. To do this, I eliminate one of the conference games, replace it with another non-conference game, and run another set of simulations. To match the most likely possible schedule, I remove a Pac-12 North team at random from each Pac-12 South team's conference schedule, and vice versa. This way each team still plays every foe in its division. Because one of the biggest issues with 4-game nonconference schedules is weak scheduling, I give each Pac-12 team an extremely weak opponent. These are equal in weakness to the team's worst OOC opponent: Oregon gets another Nicholls State-level team, UCLA gets another New Mexico State, and so on.
Using this new, randomly generated 8-game conference schedule in each iteration, I run through the season 10,000 more times. Now I simulate 48 conference games, the original set of nonconference games, and the new, additional nonconference games. Once again, I calculate each Pac-12 team's number of wins and strength of schedule at the end of the season.
Now we can return to Larry Scott's argument: does the Pac-12 really enjoy better strength of schedule ratings with a 9-game schedule?
Strength of Schedule Results
Below I list the average strength-of-schedule ratings for each team in our sets of 10,000 simulated seasons. Again, the only difference between the normal (9-game) schedule and the 8-game schedule is that a non-division conference game is replaced, at random, with a low-level nonconference opponent.
Avg. SoS (9 games) | Avg. SoS (8 games) | Difference in SoS with 8-game schedule | |
Cal | 0.6260 | 0.6419 | +0.016 |
Oregon | 0.5925 | 0.6143 | +0.022 |
OSU | 0.6116 | 0.6386 | +0.027 |
Stanford | 0.6176 | 0.6434 | +0.026 |
Washington | 0.6089 | 0.6282 | +0.019 |
WSU | 0.6227 | 0.6528 | +0.030 |
Arizona | 0.6055 | 0.6302 | +0.025 |
ASU | 0.6096 | 0.6299 | +0.020 |
Colorado | 0.6178 | 0.6539 | +0.036 |
UCLA | 0.6119 | 0.6361 | +0.024 |
USC | 0.5997 | 0.6239 | +0.024 |
Utah | 0.6352 | 0.6465 | +0.011 |
Wow. Every single team has a better strength of schedule under the 8-game schedule (and those differences are all statistically significant by an enormous margin). Despite playing one fewer quality opponent, teams earn a great advantage during the 8-game schedule because conference opponents have, on average, more wins. This incrementally improves the strength of schedule over the course of the season.
Are these differences meaningful, however? Sure, SoS climbs about .02-.03 points, but that's not a big difference, is it? Actually, that's a surprisingly important difference. Looking over the 2013 NCAA strength of schedule rankings shows that an increase of .02 is roughly equal to climbing 10-15 positions in the strength of schedule rankings. An increase of .03 can see teams jumping up to 30 spots in the rankings.
This means that teams playing an 8-game schedule can expect, on average, a sizable strength of schedule advantage over teams with 9-game conference schedules.
How could this have happened? How is it that these teams earn such a boost in strength of schedule if their non-conference schedules are weaker? Take a look at the differences in average number of regular-season wins in 9-game and 8-game conference schedules:
Avg. Wins (9 games) | Avg. Wins (8 games) | Difference | |
Cal | 3.226 | 4.056 | +0.830 |
Oregon | 8.258 | 8.883 | +0.625 |
OSU | 6.632 | 6.492 | -0.139 |
Stanford | 8.635 | 9.241 | +0.606 |
Washington | 7.710 | 8.050 | +0.339 |
WSU | 6.239 | 6.201 | -0.037 |
Arizona | 6.741 | 7.207 | +0.466 |
ASU | 9.386 | 9.844 | +0.458 |
Colorado | 4.305 | 3.638 | -0.667 |
UCLA | 7.649 | 8.538 | +0.888 |
USC | 7.680 | 7.967 | +0.287 |
Utah | 5.286 | 7.022 | +1.736 |
Generally these Pac-12 teams have more wins under an 8-game conference schedule. A few exceptions exist, however. Oregon State's struggles against nonconference foes last year harmed them when they added an extra nonconference game. Washington State and Colorado both had their greatest successes against foes outside their divisions, so under the new schedule they each lose out on a winnable non-divisional game. Otherwise, everyone benefits from the 8-game schedule. These benefits are shared throughout the conference. When nearly every conference team has a better record, thanks to an easier nonconference schedule, wins over those conference foes look better. As a result, teams are rewarded over the course of the season for beating slightly better conference foes. Beating another New Mexico State does not hurt your strength of schedule as much when most of your conference opponents have stronger records thanks to an extra nonconference game.
Conclusion: Without Parity in Conference Schedules, the Playoff is a Farce
I'm not interested in arguing that one conference schedule is better than another; that is not the point of this exercise. I am interested, however, in demonstrating that these teams are on unequal footing in the post-BCS era. As long as we have discrepancies between the number of nonconference games, we will have teams whose strength-of-schedule ratings are systematically biased upward. Until we achieve nationwide parity in conference schedules, this playoff system is relying on an inherently flawed metric. What was the point of overhauling the BCS if we are going to replace it with a broken system?
Appendix
Why do you use simulations?
I could have kept last year's records intact. In such a situation I would still have to remove one of the conference games from the schedule in order to create the 8-game schedule. Removing the same conference game each time could introduce spurious effects. Would the change in strength of schedule be due to the removal of that non-divisional game instead of the new non-conference game? For example, if we remove UCLA from Cal's conference schedule, the Bears' strength of schedule would probably decrease because UCLA is a strong foe. I don't want this to bias our results. In order to eliminate this bias, I remove a conference game at random (again, removing a game against a non-divisional foe). Cal, for example, could have UCLA, USC, Colorado, or Arizona disappear from its schedule. To implement this random selection on a massive scale, I use simulation.
Since I have already gone down the rabbit role of using simulation to create the new, 8-game conference schedule, it was trivial to add the outcome of each game to the simulations. Furthermore, I would not expect the outcome of all 54 (or 48, in the 8-game schedule) conference games and all 36 (or 48, in the 8-game schedule) non-conference games to be identical if the 2013 season happened again. Accordingly, I simulate the outcome of every single game, which leads us to our next question...
How do you determine who wins each game?
I use the results of last season's game to inform the results of the simulated games. I take the score of each game from last season and apply the Pythagorean projection (I used the version modified for football).
As I'm doing this on a game-by-game basis, I do not multiply the result by 16 from the above equation. Using the UCLA-Nebraska game as an example, UCLA would have an 83.00% chance of defeating Nebraska, as UCLA handily won 41-21 last season. Cal meanwhile would only have a 62.18% chance of defeating Portland State again after the Bears squeaked by with a 37-30 victory.
I use these probabilities to inform the simulated outcome for each game. Using the percentage, I draw either a 1 (representing a win) or a 0 (representing a loss). The likelihood of drawing a 1 is equal to the win probability from the above calculation.
For each season, I go through each team's schedule and determine whether it won or lost each of its 12 regular-season games based on the above probabilities. To avoid entering the bizarro world where Cal draws a 0 (a loss) for its game against Arizona and Arizona also draws a 0, I introduce dependency in these outcomes such that if Cal draws a 0, Arizona must draw a 1. This way we don't have more or fewer wins than possible in each simulated season. That would corrupt the strength-of-schedule calculations at the end of season and render this whole exercise meaningless.
Once I have simulated each of the conference games, I go ahead and simulate the nonconference games. Because we don't have any complex dependency among those outcomes, those games are easier to simulate. Once I have simulated all 12 games for each of the Pac-12's teams, I add up the total number of wins for each team. With every team's final record, I can finally calculate each team's strength of schedule rating.
How do you calculate strength of schedule?
I use the NCAA's longstanding formula for calculating strength of schedule.
Here OR is the opponents' records and OOR is opponents' opponents' records. As the final records vary from simulated season to simulated season, these opponents' records and opponents' opponents' records may vary, so I recalculate both of those variables before calculating strength of schedule. To be honest, calculating the OOR was really tedious (keeping track of opponents' opponents' records for each season is rather complicated), but I wanted these SoS calculations to be as accurate as possible. Accordingly, I painstakingly calculate all these variables for each team in all 10,000 simulated seasons.
Did Cal ever go undefeated in your simulated seasons?
Of course not. Don't be silly. Cal's best record was 8-4 and it only happened 4 times.