Championships Aren’t Won On Paper. But What If They Were?

You probably missed this as baseball’s postseason was coming to an end last week — but congratulations are in order to the Houston Astros. Why? Because Houston finished the 2019 MLB season with the No. 1 Elo rating in Major League Baseball, of course.

I mean, sure, the Washington Nationals just beat the Astros to win the World Series in seven games. But still, the Astros were your official Elo champs for the 2019 season. (Somehow I doubt the Astros will throw a parade or put up a banner for the honor.)

Because Elo takes a long view of the entire season, being the best in it is a pretty good proxy for being the best team in the league “on paper.” And it’s actually not uncommon for Elo’s Paper Champion and the team that wins the World Series not to be one and the same. Including Houston this year, it’s happened 28 times — or in a whopping 52 percent of seasons — since the late 1960s, when MLB expanded to a division-based playoff format. Simply put, baseball is a sport in which the best team doesn’t always win. (Or even if it does, maybe we don’t always know who the best team is anyway.)

Other sports have their own Paper Champs. (Although none happen anywhere near as frequently as in MLB.) I went back to the start of the Super Bowl era in 1966¹ and looked at the other sports for which we keep Elo — the NFL, NBA, college football, and men’s and women’s college basketball. Using the versions of our Elo that contain no adjustments for trades or players being in and out of the lineup,² I found each case where the champion at the end of the season³ was not the team that finished atop the Elo leaderboard. Across all of these sports, these Paper Champs come up more frequently than you might think:

Since 1966, all but seven seasons⁴ (2013, 2009, 2005, 1998, 1989, 1979 and 1967) contained at least one Paper Champion across these five sports. Some years featured a lot more: In 2011, for instance, there were four Paper Champs — the Yankees in MLB, the Patriots in the NFL, and Ohio State (men’s) and UConn (women’s) in college basketball. (The only champs that actually led in Elo were the Dallas Mavericks in the NBA and Alabama in college football.)

In general, there is about a 50-50 chance that a given baseball season would produce a Paper Champ and somewhere between a 20 and 30 percent probability each of the other sports will as well.

What does all of this mean? Well, it could be that Elo is broken. Even though it is calibrated to be the best predictor for a team’s next game — given its recent form, long-term expectations, wins and losses, scoring margin, opponent quality and game locations — maybe there are certain aspects of each sport’s postseason that aren’t captured by the algorithm. (This is the “Billy Beane’s Shit Doesn’t Work In The Playoffs” theory.)

A fundamental challenge of forecasting is the balancing act between considering a large amount of information — some of which may be of less relevance than others — and a more specific one that is more relevant, but also more prone to factors such as random variance in a small sample. Our Elo models attempt to straddle this divide, but it’s impossible to find the perfect mix of information that works in every single case.

Then again, maybe the real issue is that playoff systems are too small of a sample to determine the best team. Perhaps the best we can do is be content believing the champion was simply one of the top teams in a given season, nothing more.

Still another way of reconciling Paper Champs to postseason reality, though, is to consider that most of these actual champions vanquished their on-paper rivals head-to-head along the way. When there was a Paper Champ in baseball, for instance, 46 percent of the time that team lost directly to the eventual champion in a postseason game or series. (See: Nationals over Astros.) In the NBA, that number was 60 percent; in men’s college basketball, 79 percent; and in the NFL, a whopping 91 percent. Although Elo still wasn’t convinced that the matter was settled afterward, the Paper Champ at least had a chance to make its case on the field or court.

And in most of the cases where things weren’t settled head-to-head, you only have to zoom out a little to find a path of head-to-head superiority between the actual champ and the paper one. Like in the 2017 NCAA Women’s Tournament, when UConn finished as Paper Champ … but lost in the national semis to Mississippi State, which then lost to South Carolina in the title game. Almost every disagreement between Elo and the official championship can be settled either directly head-to-head or in this manner — with the exceptions of a few pre-wild-card MLB seasons (in which the Paper Champ didn’t even make the postseason at all) and a number of older college football campaigns that underscored just how broken the sport’s pre-playoff system truly was.

In 2007, famously one of the weirdest college football seasons ever, USC was Elo’s choice, while LSU prevailed in the BCS. To find a head-to-head path that put LSU over USC, even if you open up the possibilities to include the regular season,⁵ you needed to follow a trail of four games: LSU beat Ohio State, which beat Washington, which beat Stanford, which beat … USC.

But at least the BCS existed by then. Before it came along, the 1970s and ’80s often required even more ludicrous daisy-chaining of head-to-head results to reconcile the championship. In 1976, the path from actual champ Pitt to Paper Champ USC required a string of five games. And in 1983, the chain went like this: Actual champ Miami beat Notre Dame, which beat Boston College, which beat Clemson, which tied Georgia, which beat Texas, which beat Auburn, Elo’s Paper Champion. No wonder college football fans clamored for a proper playoff (even if the one they have now could also probably stand to be expanded).

But even with the perfect playoff system, you can never really avoid Paper Champs. Random variance and matchups — plus a million other factors — will always cause teams to play better or worse than they look on paper. And would we really want it any other way? If we look at who would have benefited most over the past half-century if Elo perfectly aligned with actual championships, the rich would mostly have gotten richer:

Although I feel bad for the Orioles and Indians, who would have won multiple extra championships if Elo had been perfectly predictive, other teams that have won plenty of real titles — Alabama football, UConn women’s basketball, the Patriots, the Yankees, the Spurs, etc. — “should” have even more under Elo.

Oftentimes, it’s the unpredictability of sports that make them great — just ask the Nationals. So we’re sorry, Astros. Although the first-ever Elo Championship pennant would be something to behold.