Did VAR change Scottish football? Difference-in-differences

The football question

VAR came to Scottish football on Friday 21 October 2022, with Hibs against St Johnstone at Easter Road. Only the Premiership got it. The Championship, the league below, never did.

Since that night, Premiership matches have averaged 3.71 yellow cards, against 3.66 before. So VAR changed nothing about bookings?

Not so fast. Football changes every season anyway: new instructions to referees, new rules on dissent, new players. A before-and-after comparison can't separate VAR from everything else that changed at the same time. We need the counterfactual: the Premiership's seasons without VAR.

The concept

The first part of this series said that every causal question needs a stand-in for the season that didn't happen. Difference-in-differences finds one in a group that went through the same years without the change.

The Championship had the same calendar, the same referees' body and the same rule changes, but no VAR. So its change over the same years shows what would have happened anyway. VAR's effect is whatever the Premiership changed beyond that:

$$\begin{aligned} \text{effect} &= \Delta_\text{Prem} - \Delta_\text{Champ} \\ \Delta &= \text{after} - \text{before} \end{aligned}$$

What the comparison says

Here are both leagues, before and after 21 October 2022, from 2017/18 (when the Championship files start recording cards and fouls), leaving out 2020/21, which was played without crowds. That's 1,775 Premiership matches and 1,397 Championship matches.

The Premiership's yellow cards rose by 0.05 a game. The Championship's rose by 0.67. So the difference-in-differences says VAR saved 0.62 yellow cards a game, with a luck margin of ±0.28. Fouls tell the same story: down 1.86 in the Premiership, unchanged in the Championship, so VAR cut fouls by 1.87 a game (±0.78). Both are outside their luck margins. Goals went up 0.20 more in the Premiership, but that's inside its margin of ±0.23.

That would make a fine headline: VAR calmed Scottish football down. It doesn't survive the next check.

The check: did the leagues move together before?

The method rests on one assumption, called parallel trends: without VAR, the two leagues would have kept changing in step. They don't have to be equal, only to rise and fall together. If they were already drifting apart before VAR, the "effect" is just the drift.

The gap hardly moved when VAR arrived. The Premiership's lead in yellow cards was between +0.14 and +0.69 a game before VAR, and +0.19 and +0.04 in the first two seasons with it. Then, in 2024/25, Championship bookings jumped to 4.18 a game and stayed high (4.28). Fouls show the same thing: the Championship's jumped in the same season. Almost all of the "VAR effect" comes from one season in the league that didn't get VAR, nearly two years after VAR started.

Something changed in the Championship that season. It might have been how matches were refereed or how cards were recorded; the results files can't say. Whatever it was, it wasn't VAR in the Premiership.

A placebo test

There's a second check. Pretend VAR arrived in summer 2019 instead, and run the same comparison using only matches from before the real date. Nothing happened in 2019, so a sound comparison should find nothing.

The yellow cards and goals pass. The fouls don't, quite: a pretend VAR "added" 1.10 fouls a game, right at the edge of its margin. The two leagues' fouling was already drifting apart before VAR existed. Another reason not to trust the fouls result.

Why it matters

Difference-in-differences is one of the most used methods in economics and policy: it's how researchers have studied minimum wages, smoking bans and new laws, whenever one place changes and a similar one doesn't. Its strength is that it allows for everything that changed for both groups. Its weakness is the assumption it can't prove: that the two groups would have moved together. The comparison table alone looks convincing. The season-by-season chart is what tells you whether to believe it.

The same goes for any "since X came in" claim in football, from a new manager to a rule change: find the group that didn't get it, and check they were moving together before. For the bookings themselves, and whether referees treat home and away sides alike, see do referees favour the home side?

Limitations

• The results files don't record what VAR does most. Penalties awarded, goals disallowed and red cards upgraded or overturned are where VAR acts, and none of them are counted separately. A real effect there could be hidden from this data.
• The Championship isn't a perfect twin. It has different clubs, smaller crowds and, after 2024/25, clearly something else going on. A better comparison might be another league that never got VAR, if its data were as complete.
• Short history. The Championship files only record cards and fouls from 2017/18, so there are four seasons before VAR to check the trends on, one of them cut short in 2019/20.
• Red cards are too rare to compare reliably in so few seasons, so they're left out.

Reproduce the analysis

Save the Premiership (SC0) and Championship (SC1) results files from football-data.co.uk in one folder; the download script fetches them under the names the snippet expects. It needs only Python 3 and takes under a second:

# From Football Data Science by Bryan McGuire. Free to use with credit.
# https://www.footballdatascience.co.uk/learn/difference-in-differences-var
import csv
from datetime import datetime
from math import sqrt
from statistics import mean, pstdev

VAR = datetime(2022, 10, 21)  # Hibs v St Johnstone, the Premiership's first VAR match
LEAGUES = {"SC0": "Premiership", "SC1": "Championship"}  # the Championship never got VAR
STATS = {"yellow cards": ("HY", "AY"), "goals": ("FTHG", "FTAG"), "fouls": ("HF", "AF")}

# every match in both leagues from 2017/18, when the Championship files start recording cards and fouls
matches = []
for code in LEAGUES:
    for y in range(2017, 2026):
        if y == 2020:
            continue  # 2020/21 was played without crowds
        with open(f"{code}_{y % 100:02d}{(y + 1) % 100:02d}.csv", encoding="latin-1") as f:
            for r in csv.DictReader(f):
                if r.get("FTR") not in ("H", "D", "A") or not r.get("HY"):
                    continue
                d = datetime.strptime(r["Date"], "%d/%m/%Y" if len(r["Date"]) == 10 else "%d/%m/%y")
                m = {"league": code, "season": y, "after": d >= VAR}
                m.update({stat: int(r[h]) + int(r[a]) for stat, (h, a) in STATS.items()})
                matches.append(m)

print(", ".join(f"{name} {sum(m['league'] == code for m in matches):,} matches" for code, name in LEAGUES.items()))

def summary(group, stat):  # average a match, and its standard error
    values = [m[stat] for m in group]
    return mean(values), pstdev(values) / sqrt(len(values))

for stat in STATS:
    print(stat)
    change = {}
    for code, name in LEAGUES.items():
        (b, sb), (a, sa) = (summary([m for m in matches if m["league"] == code and m["after"] == after], stat)
                            for after in (False, True))
        change[code] = (a - b, sqrt(sa ** 2 + sb ** 2))
        print(f"  {name:12} {b:5.2f} -> {a:5.2f}, change {a - b:+.2f}")
    did = change["SC0"][0] - change["SC1"][0]
    margin = 2 * sqrt(change["SC0"][1] ** 2 + change["SC1"][1] ** 2)
    print(f"  difference-in-differences {did:+.2f} (luck margin ± {margin:.2f})")

# the check: did the two leagues move in step? season by season, Premiership minus Championship
for stat in ("yellow cards", "fouls"):
    print(f"\n{stat} a match: season, Premiership, Championship, gap")
    for y in range(2017, 2026):
        if y == 2020:
            continue
        p, c = (mean(m[stat] for m in matches if m["league"] == code and m["season"] == y) for code in LEAGUES)
        print(f"  {y}/{(y + 1) % 100:02d}  {p:5.2f}  {c:5.2f}  {p - c:+.2f}")

# a placebo: pretend VAR arrived in summer 2019 and use only matches from before the real date
placebo = [dict(m, after=m["season"] >= 2019) for m in matches if m["season"] < 2022 or not m["after"]]
print("\nplacebo, a pretend start in 2019/20:")
for stat in STATS:
    ch = {}
    for code in LEAGUES:
        (b, sb), (a, sa) = (summary([m for m in placebo if m["league"] == code and m["after"] == after], stat) for after in (False, True))
        ch[code] = (a - b, sa ** 2 + sb ** 2)
    print(f"  {stat}: {ch['SC0'][0] - ch['SC1'][0]:+.2f} (luck margin ± {2 * sqrt(ch['SC0'][1] + ch['SC1'][1]):.2f})")

Further reading

• Difference in differences, Wikipedia. The method, its parallel-trends assumption, and Card and Krueger's famous study of the minimum wage in New Jersey and Pennsylvania.
• Video assistant referee, Wikipedia. How VAR works, which decisions it can review, and where it's used.