Accuracy isn't enough: calibrating a tool-selection model

An agent router doesn't just need the right answer — it needs to know when it might be wrong, so it can fall back or ask a clarifying question.

Expected Calibration Error

Bucket predictions by confidence and compare average confidence with actual accuracy in each bucket:

def ece(confs, correct, n_bins=10):
    bins = np.linspace(0, 1, n_bins + 1)
    err = 0.0
    for lo, hi in zip(bins[:-1], bins[1:]):
        m = (confs > lo) & (confs <= hi)
        if m.any():
            err += m.mean() * abs(confs[m].mean() - correct[m].mean())
    return err

What I saw

	Before	After LoRA
Accuracy	50.6%	100%
ECE	0.220	0.002

Lessons

• Shuffle candidate order in the training data, or the model learns "the answer is usually first".
• A random split shares tools between train and test — the real test is held-out tools. That's next.

Code: Fine-tuning-Decision_model