Does finishing sixth rather than seventh matter? Regression discontinuity

The football question

After 33 games the Scottish Premiership splits in two. The top six play each other for the title and Europe; the bottom six play each other to stay up. Sixth and seventh can be separated by a point, or by goal difference, yet one ends up in the top half and the other doesn't.

Clubs talk about "top-six status" as if it carries over: better crowds for the big games, more money, players who want to stay. So does finishing sixth at the split, rather than seventh, make a club better the following season?

What happens over the last five games is already covered: the split myth shows the side in seventh takes more points than the side in sixth after the split, because it plays weaker teams. This part asks about the season after.

The concept

The first part of this series needed a fair stand-in for the season that didn't happen. A regression discontinuity finds one wherever a sharp line divides clubs.

The clubs just above the line and just below it are nearly alike. One got a point more, or a better goal difference, which is mostly luck. If they differ afterwards, the line is the likeliest reason. The side in seventh is the counterfactual for the side in sixth: the same kind of club, on the other side of the line.

The catch is the word just. The further from the line you look, the less alike the clubs are. Compare the whole top six with the whole bottom six and you're comparing Celtic with the bottom club, which tells you nothing about the line. How far either side you go is called the bandwidth.

What the data shows

Here are 25 splits since 2000/01 (there was none in 2019/20, which stopped early), 24 of them with a following season to look at. For each window, the clubs above the line and below it, their points a game before the split, and their points a game the next season:

Right at the line, there's no real difference. The sides that were sixth and the sides that were seventh both averaged 1.27 points a game the following season, a gap of +0.01 with a luck margin of ±0.16. Being in the top six didn't carry over.

Widen the window and a "top-six effect" seems to appear. The gap grows from +0.01 for sixth against seventh, to +0.06, to +0.10, and to +0.49 a game for the whole top six against the whole bottom six, well outside its luck margin of ±0.10. Someone who compared the two halves of the table would announce that the top six sets you up for next season.

But look at the "before" column. The wider the window, the bigger the gap was before the split, too: from about 0.1 a game at the line to 0.72 for the halves. The wide windows aren't measuring the line. They're measuring how much better the better teams were, and better teams stay better.

Are sixth and seventh really alike?

The method only works if the clubs either side of the line are alike, so it's worth checking. Before the split, sixth averaged 1.32 points a game and seventh 1.23. That's about three points over 33 games: close, but not identical. Sixth was a little better, which, if anything, should have made it better next season too. It wasn't.

This balance check is the regression discontinuity's version of Part 3's parallel trends: the test that the comparison is fair before you trust what it says.

Why it matters

Lines are everywhere in football. Promotion and relegation places, the top four for Europe, a points threshold for a bonus, an age cut-off for academy groups. Each one invites the same question: did being over the line change anything? And each one invites the same mistake: comparing everyone above it with everyone below.

Regression discontinuity is one of the cleanest methods for cause and effect outside a real experiment, because the clubs either side of the line chose almost nothing about which side they landed on. Its weakness is the one shown here: near the line there aren't many clubs. With one pair a season, this data can only rule out a large effect.

Limitations

• Only 24 pairs. The luck margin for sixth against seventh is ±0.16 points a game, so an effect smaller than that, about six points over a season, can't be ruled out.
• Some clubs drop out. Hibernian were seventh at the split in 2013/14 and went down through the play-off, so they have no next season to compare. That removes a club below the line that did badly, which flatters the bottom side slightly.
• Points a game isn't everything. Prize money, European places and season-ticket sales near the line aren't in the data, so the line might matter in ways the next season's points don't show.

Reproduce the analysis

Save the Premiership results files (SC0) from football-data.co.uk in one folder; the download script fetches them under the names the snippet expects. It needs only Python 3 and takes under a second:

# From Football Data Science by Bryan McGuire. Free to use with credit.
# https://www.footballdatascience.co.uk/learn/regression-discontinuity-split
import csv
from collections import Counter
from math import sqrt
from statistics import mean, pstdev

def season(y):
    with open(f"SC0_{y % 100:02d}{(y + 1) % 100:02d}.csv", encoding="latin-1") as f:
        return [r for r in csv.DictReader(f) if r.get("FTR") in ("H", "D", "A")]

# for each season: points a game over the whole season, and the table after 33 games (where the split comes)
full, split, before = {}, {}, {}
for y in range(2000, 2026):
    played, pts, gd, gf, total, games = Counter(), Counter(), Counter(), Counter(), Counter(), Counter()
    for r in season(y):
        h, a, hg, ag = r["HomeTeam"], r["AwayTeam"], int(r["FTHG"]), int(r["FTAG"])
        for team, p, scored, conceded in ((h, {"H": 3, "D": 1, "A": 0}[r["FTR"]], hg, ag), (a, {"H": 0, "D": 1, "A": 3}[r["FTR"]], ag, hg)):
            total[team] += p
            games[team] += 1
            if played[team] < 33:
                pts[team] += p
                gd[team] += scored - conceded
                gf[team] += scored
            played[team] += 1
    full[y] = {t: total[t] / games[t] for t in total}
    if y != 2019:  # 2019/20 was stopped before the split
        split[y] = sorted(pts, key=lambda t: (pts[t], gd[t], gf[t]), reverse=True)  # as the split myth orders them
        before[y] = {t: pts[t] / 33 for t in pts}

def average(values):
    return mean(values), pstdev(values) / sqrt(len(values))

print(f"{len(split)} splits; clubs still in the Premiership the next season are compared")
for width, label in ((1, "6th v 7th"), (2, "5th-6th v 7th-8th"), (3, "4th-6th v 7th-9th"), (6, "top six v bottom six")):
    groups = []
    for side in (slice(6 - width, 6), slice(6, 6 + width)):
        clubs = [(y, t) for y, table in split.items() if y + 1 in full for t in table[side] if t in full[y + 1]]
        groups.append(([before[y][t] for y, t in clubs], [full[y + 1][t] for y, t in clubs]))
    (pa, _), (pb, _) = average(groups[0][0]), average(groups[1][0])
    (na, sa), (nb, sb) = average(groups[0][1]), average(groups[1][1])
    print(f"{label:21} ({len(groups[0][1])} v {len(groups[1][1])}): before the split {pa:.2f} v {pb:.2f}, "
          f"next season {na:.2f} v {nb:.2f}, gap {na - nb:+.2f} (luck margin ± {2 * sqrt(sa ** 2 + sb ** 2):.2f})")

gone = [(y, t[6]) for y, t in split.items() if y + 1 in full and t[6] not in full[y + 1]]
print("7th at the split but not in the Premiership the next season:", ", ".join(f"{t} {y}/{(y + 1) % 100:02d}" for y, t in gone))

Further reading

• Regression discontinuity design, Wikipedia. The method, from Thistlethwaite and Campbell's 1960 study of scholarship winners, and how to choose the bandwidth.
• Scottish Premiership, Wikipedia. The league's format, including how the split works.