Case study
HaluGuard: Hallucination-Aware Context Selection for Code Generation
Feb 2026 – May 2026
The problem
A language model completing code inside a repository needs context from other files: the type signature it is calling against, the import that makes a name resolve, the class it is subclassing. Retrieval-augmented generation supplies that context by embedding the query and returning the most similar chunks, which is a reasonable default and the wrong objective.
Similar code is not necessarily helpful code. A function that looks like the one being written scores highly because it shares surface structure, while the file that actually determines whether the completion is correct — the module defining the symbol, the signature fixing the argument order — may share almost no tokens with the query. Optimising for similarity retrieves the former and misses the latter, and the model then produces something syntactically valid that calls an API which does not exist. So the project asked a different question: instead of what context resembles this query, it selects for what context stops the model hallucinating.
What I did
The team built three components around that reframing; my work was the data and the evaluation side. I processed the RepoBench data the system trains and evaluates on — 8,033 queries, roughly eleven candidate chunks each with one correct — and supported the contrastive training setup for the selector. That selector scores a chunk by how well it prevents hallucination rather than how closely it matches, and we compared seven architectures over precomputed embeddings, most trained with a masked listwise cross-entropy that treats selection as ranking the gold snippet against every valid candidate. A rule-based routing layer sits alongside it, boosting different context types for different failure modes: missing imports want dependency snippets, identifier errors want definitions, API misuse wants typed signatures.
I also implemented the retrieval baselines and the evaluation harness, which is where the comparison actually lives. Retrieval ran against embedding similarity, token overlap, edit distance and a random control, scored by Recall@k and split into easy and hard queries so we could see how each degrades as the repository grows. Generation was scored by exact match, edit similarity and CodeBLEU, bracketed by two bounds worth having: gold-only as an oracle, and full-context as the everything-in-the-prompt alternative.
What came out
On hard queries the similarity baselines barely function. Recall@1, against a random control:
| Method | R@1 | R@3 | R@5 |
|---|---|---|---|
| Ensemble | 0.648 | 0.797 | 0.849 |
| DualEncoderDeep | 0.644 | 0.795 | 0.856 |
| UniXcoder (cosine) | 0.186 | 0.410 | 0.579 |
| CodeBERT (cosine) | 0.079 | 0.211 | 0.345 |
| Random | 0.063 | 0.190 | 0.320 |
| Jaccard | 0.056 | 0.205 | 0.326 |
CodeBERT similarity retrieves the right chunk 7.9% of the time against random’s 6.3%, and Jaccard overlap does worse than random. That is a stronger result than “learned ranking helps” — on the hard split, ranking code by resemblance to the query is close to not ranking it at all, which is the premise of the project stated as a measurement.
Downstream generation is where it gets complicated. The learned selector reaches 0.290 exact match against 0.265 for stuffing the whole repository into the prompt and 0.200 for no context — real, but a tenth of the retrieval improvement. Two details in that table are worth more than the headline.
The best retriever is not the best generator. Ensemble wins retrieval and DualEncoderDeep wins generation, so picking a system on retrieval quality alone would have picked the wrong one.
The oracle loses. Gold-only, which hands the model exactly the ground-truth snippet and nothing else, scores 0.250 — below full context and well below the learned retriever. Having the right snippet is not the same as having enough to use it, and a framing that treats retrieval as finding one correct chunk is measuring the wrong target. Neither finding is visible from retrieval metrics alone.
What I’d do differently
The evaluation measured retrieval quality and generation quality separately, which was the right decomposition but left the link between them unmeasured — and that link is where the interesting number lives. A tenfold retrieval gain producing a 0.025 improvement in exact match is either evidence that most retrievals land on queries the model already completed correctly, or evidence that one good snippet is not what a completion needs. The aggregates cannot tell those apart. I would pair every retrieval result with the generation outcome for the same query from the start, so the mismatch is something the evaluation reports rather than something you infer afterwards from two separate tables.
The other limit is scale: generation was scored on 200 samples, enough to see the gap but not enough to characterise it. The code is at Sparshg3011/HaluGuard.