Case study
Mitigating Dialect Bias in Toxicity Detection
Feb 2026 – May 2026
The problem
Toxicity classifiers are trained on data where African American English is labelled offensive more often than Standard American English saying comparable things, and a model fitted to that data learns the correlation rather than the offence. The result is a classifier that flags benign AAE text as toxic. That failure is worse than an ordinary error, because the people whose speech is disproportionately removed are the people the moderation system is supposed to protect.
The obvious fix is more data, and it does not work. Annotation imbalance is not noise that averages out with volume; it is a signal the model is rewarded for learning, and every example labelled the same way reinforces it. The intervention has to change what the model may represent or where its boundary sits, so we tested one of each.
What I did
We built a toxicity classifier on the Davidson hate speech dataset, tagged each tweet with a dialect label using the pretrained TwitterAAE model, and measured false positive and false negative rates separately for each group. Three baselines set the scale of the problem, on balanced data:
| Model | F1 | FPR (AAE) | FPR (SAE) | Gap |
|---|---|---|---|---|
| XGBoost on BERT embeddings | 0.840 | 0.370 | 0.137 | 0.233 |
| ToxicBERT | 0.848 | 0.481 | 0.191 | 0.290 |
| Fine-tuned BERT | 0.961 | 0.074 | 0.035 | 0.039 |
ToxicBERT is the one to sit with: a model pretrained specifically for toxicity detection, scoring the second-best F1 of the three and carrying the largest disparity of any baseline we ran. A purpose-built toxicity model, amplifying the bias it exists to catch.
My own contribution was the adversarial arm. It trains the XGBoost predictor jointly with an adversary that tries to recover the dialect group from the model’s margin and the true label, and conditioning on the label is the choice that matters: it targets Equalized Odds rather than demographic parity, so the model is pushed toward equal error rates instead of equal flag rates. I paired it with per-group thresholds tuned to minimise the FPR gap subject to a floor on F1. Teammates ran the other branches, reweighting and a post-hoc vector-scaling calibration.
What came out
The combination worked. AAE false positives halved from 0.370 to 0.185, the gap fell from 0.215 to 0.033, and overall F1 cost 0.008 — a rounding error against the disparity it removed. The division of labour between the two pieces turned out to be clean: adversarial training aligns the score distributions, and threshold tuning converts that alignment into fair decisions. Neither does the job alone, since the same trained model at a shared 0.5 threshold still leaves AAE false positives above 0.77.
Neither knob behaved like a dial you turn up. Fairness is not monotonic in the coupling strength: small values leave the baseline gaps, 0.15 to 0.25 collapses both at F1 around 0.82, and past 0.6 it destabilises — false positives spike and F1 drops. Adversary capacity is worse; it is a bathtub. Weak is noisy, strong settles into equilibrium, and the middle is the worst cell of the sweep, with AAE false positives at 0.52 — above the baseline the exercise was meant to fix. Capacity explains about 1% of the variance in F1 against the coupling strength’s 54%, so it governs whether training converges rather than where the tradeoff sits. Two different kinds of knob, and treating them as one wastes a lot of sweep.
Two further findings mattered more than any single number.
The gains did not survive distribution shift. Carrying the trained model and its tuned thresholds straight over to HateXplain, the 0.033 gap became 0.43 and F1 fell from 0.823 to 0.52. Per-group thresholds are fitted to one score distribution, and when the distribution moves they stop meaning what they meant — the bias reappears at close to its original size.
A stronger adversary closed the gap in the wrong direction. Conditioning it on BERT embeddings let it see dialect cues a margin-only adversary cannot, and the optimal coupling strength duly fell sixtyfold — but it equalised by raising SAE false positives to meet AAE rather than by lowering AAE, landing at a higher mean FPR overall. Equalized Odds is a parity criterion, and flagging both groups aggressively satisfies it exactly as well as flagging both conservatively. So the weaker adversary wins.
What I’d do differently
I would fix the metric before fixing the model. We spent our effort on the optimiser and inherited Equalized Odds without asking whether parity was what we actually wanted, and the BERT adversary result is what happens when a criterion is satisfied in the wrong direction. Stating the objective as a bound on AAE false positives, rather than as equality between two rates, would have ruled that solution out by construction instead of catching it in analysis afterwards.