Bottom at Christmas, new manager by March. Did the change make it worse? Counterfactuals

The football question

Since 2000/01, 26 sides have been bottom of the Scottish Premiership on Christmas Day. Seven of them changed manager between Christmas and the end of March. Not one of those seven got off the bottom. Of the 19 that kept their manager, eight did.

So does sacking the manager when you're bottom make things worse?

It looks that way. But the table above can't tell you, and working out why is the starting point for everything in this series.

The concept

Every club bottom at Christmas has two possible futures. In one, the board changes the manager. In the other, it doesn't. Each future ends somewhere in the table. These are the club's two potential outcomes.

The effect of the change, for that club, is the difference between them: where it finished with the new manager, against where it would have finished without one. The trouble is that a club only ever lives one of the two. The other is a counterfactual, the season that didn't happen.

Since the missing season can't be watched, the only way forward is to find other clubs that can stand in for it: a comparison group. Everything depends on choosing it well.

A made-up example where we can see both

Here are four made-up clubs, each bottom or near it at Christmas, with both of their possible futures shown: points after Christmas without a change, and with one. In this made-up world, a change helps every club by exactly 2 points. The two that were sinking fastest change their manager; the two that were steadying keep theirs.

The numbers in bold are the only ones anyone would ever see. The changers took 14 and 15, an average of 14.5. The keepers took 20 and 23, an average of 21.5. Compare the two and the change seems to cost 7 points, when in fact it gained every club 2.

The comparison is unfair because the clubs weren't alike to begin with. Boards change managers when results are bad and getting worse, so the changers would have done worse than the keepers whatever happened. Comparing them mixes up the effect of the change with the reason for it.

What the real seasons show

The real Premiership has the same problem. Here are the 26 sides bottom on Christmas Day, split by whether they changed manager between Christmas and the end of March:

Now look at how each group was doing in the weeks between Christmas and the change (or, for the keepers, the end of March):

The clubs that changed were already doing about half as well. That's why they changed. And the clubs that kept their manager include the ones whose results had picked up, which is partly why their boards kept faith. The comparison group was picked, in effect, by what happened after Christmas. The same trap turned up in do substitutions work?: never let what happens next decide who goes in the comparison group.

A fairer look

A better comparison is with keepers whose results after Christmas were just as bad. Six of the 19 kept their manager while taking no more points a game than the best of the changers (0.78). One of those six got off the bottom. Against none of the seven changers, that's no clear gap at all. Once the clubs are more alike, most of the apparent damage done by the sacking disappears.

That still isn't the counterfactual. The six and the seven may differ in other ways, from budgets to injuries, and the numbers are small. But it shows the direction: the sharper the comparison, the less the change seems to matter. The site's new manager bounce makes a careful version of this comparison across 80 changes, matching each club that changed with one in the same position that didn't.

Why it matters

Football is full of counterfactual claims dressed up as facts. "The new manager kept them up." "The signing won us the league." "We'd have lost if we'd not changed the shape." Each one says what would have happened otherwise, and none of them can be checked by looking at what did happen.

The rest of this series is about finding fair stand-ins for the season that didn't happen: drawing out what causes what, using moments when something changed for one group and not another, a real coin toss, and matching like with like. The site has already met some of these traps. Do football stats mislead? shows five of them, and collider bias shows how choosing who to look at can create a pattern on its own. And for how often being bottom at Christmas ends with finishing bottom, whoever is in charge, see bottom at Christmas, down in May?

Limitations

• Small numbers. Seven changers and 19 keepers. A single club going the other way would change the picture, so treat the gaps as an illustration, not a finding.
• Finishing bottom isn't always relegation. The rules changed over the years (a play-off, seasons when the champions below couldn't come up), so this looks at the table, not who went down.
• Not every season was ordinary. Gretna in 2007/08 were in administration, and 2019/20 was cut short, with the table settled on points a game.
• Only changes in the site's manager-changes data count, and only between Christmas and the end of March. A change made before Christmas, or too late to matter, isn't counted as one.
• The two windows differ in length. Changers' form runs up to their change, keepers' to the end of March. That's fair to the question "how were they doing when the board decided?", but it isn't a perfect like-for-like.

Reproduce the analysis

Save the Premiership results files (SC0) from football-data.co.uk and the site's manager-changes-spfl.csv in one folder; the download script fetches the results files under the names the snippet expects. It needs only Python 3 and takes about a second:

# From Football Data Science by Bryan McGuire. Free to use with credit.
# https://www.footballdatascience.co.uk/learn/counterfactuals-new-manager
import csv
from collections import Counter
from datetime import datetime

def season(s):
    with open(f"SC0_{s}.csv", encoding="latin-1") as f:
        games = [r for r in csv.DictReader(f) if r.get("FTR") in ("H", "D", "A")]
    for r in games:
        r["date"] = datetime.strptime(r["Date"], "%d/%m/%Y" if len(r["Date"]) == 10 else "%d/%m/%y")
    return games

def table(games):
    """Teams in league order: points, then goal difference, then goals scored (as the bottom-at-Christmas myth)."""
    pts, gd, gf, played = Counter(), Counter(), Counter(), Counter()
    for r in games:
        h, a, hg, ag = r["HomeTeam"], r["AwayTeam"], int(r["FTHG"]), int(r["FTAG"])
        pts[h] += {"H": 3, "D": 1, "A": 0}[r["FTR"]]
        pts[a] += {"H": 0, "D": 1, "A": 3}[r["FTR"]]
        gd[h] += hg - ag
        gd[a] += ag - hg
        gf[h] += hg
        gf[a] += ag
        played[h] += 1
        played[a] += 1
    return sorted(played, key=lambda t: (pts[t], gd[t], gf[t]), reverse=True)

def points(games, team):  # points a game for one team across these matches
    won = [{"H": 3, "D": 1, "A": 0}[r["FTR"]] if r["HomeTeam"] == team else {"H": 0, "D": 1, "A": 3}[r["FTR"]]
           for r in games if team in (r["HomeTeam"], r["AwayTeam"])]
    return sum(won) / len(won)

with open("manager-changes-spfl.csv", encoding="utf-8") as f:
    changes = list(csv.DictReader(f))

rows = []
for y in range(2000, 2026):
    games = season(f"{y % 100:02d}{(y + 1) % 100:02d}")
    christmas, deadline = datetime(y, 12, 25), datetime(y + 1, 3, 31)
    bottom = table([r for r in games if r["date"] < christmas])[-1]
    label = f"{y}/{(y + 1) % 100:02d}"
    made = sorted(datetime.fromisoformat(c["date"]) for c in changes  # a change between Christmas and the end of March
                  if c["season"] == label and c["club"] == bottom and christmas <= datetime.fromisoformat(c["date"]) <= deadline)
    until = made[0] if made else deadline  # changers: up to the change; keepers: up to the end of March
    rows.append({"season": label, "club": bottom, "changed": bool(made), "off_bottom": table(games)[-1] != bottom,
                 "form": points([r for r in games if christmas <= r["date"] < until], bottom)})

for r in rows:
    print(f"{r['season']} {r['club']:14} {'changed' if r['changed'] else 'kept   '} "
          f"{'got off the bottom' if r['off_bottom'] else 'finished bottom'}, {r['form']:.2f} points a game after Christmas")
print()
for label, group in (("changed manager by the end of March", [r for r in rows if r["changed"]]),
                     ("kept their manager", [r for r in rows if not r["changed"]])):
    print(f"{label}: {len(group)}, got off the bottom {sum(r['off_bottom'] for r in group)}, "
          f"{sum(r['form'] for r in group) / len(group):.2f} points a game from Christmas to the change (or the end of March)")

# a fairer look: keepers whose results after Christmas were no better than the best of the changers'
worst = max(r["form"] for r in rows if r["changed"])
alike = [r for r in rows if not r["changed"] and r["form"] <= worst]
print(f"\nkept their manager with {worst:.2f} points a game or fewer: {len(alike)}, got off the bottom {sum(r['off_bottom'] for r in alike)}")

Further reading

• Rubin causal model, Wikipedia. The potential outcomes idea used here, from Donald Rubin and Jerzy Neyman, and why only one outcome is ever seen.
• Selection bias, Wikipedia. How the way a group is chosen can make it look different for reasons that have nothing to do with the thing being studied.
• Causal inference, Wikipedia. The wider field this series introduces, with its main methods.