Migrating a catalogue with confidence-tiered AI classification

Overview

Moving to Square meant remapping a legacy EPOS catalogue of thousands of products into a category hierarchy that did not correspond to the old one. Done by hand, this is weeks of tedious, error-prone work. Done by asking a language model about every product, it is expensive and produces confident answers to questions where confidence is not warranted.

Neither approach is good. What the problem actually needs is a way to spend effort in proportion to difficulty.

---

Key challenges and solutions

Challenge 1: Most products are easy and a few are genuinely hard

The bulk of a catalogue maps cleanly. A minority is ambiguous. Treating every product with the same expensive process wastes money on the easy ones and does not help the hard ones.

Our solution: confidence-tiered routing. Every product and category is embedded once with text-embedding-3-small and compared by cosine similarity. Matches above 0.93 are auto-accepted. Matches between 0.65 and 0.93 — the ambiguous band — escalate to GPT-4o for adjudication with the candidate categories in context. Anything below 0.65 is rejected outright for human review rather than guessed at.

This means the frontier model is invoked only where it can actually add judgement, which is a small fraction of the catalogue.

Challenge 2: Re-running the migration should not re-cost it

Migrations are never run once. You run them, inspect the output, adjust the thresholds, and run again.

Our solution: a cached embedding store, so re-runs pay for embedding only what changed. Iterating on the thresholds became cheap enough to actually do properly rather than something to avoid.

Challenge 3: The generated hierarchy could contain cycles

Programmatically assembling a deep category tree can produce cycles — a category that is transitively its own ancestor — which Square will reject on import, often unhelpfully.

Our solution: a dedicated cycle-detection and repair pass over the hierarchy before anything is submitted, so structural integrity is verified rather than discovered at import time.

---

What we did

• Embedding-based matching
• Confidence-tiered routing
• Cached embedding store
• Hierarchy integrity checks
• Python & scikit-learn

---

The design in numbers

0.93

Auto-accept confidence threshold

0.65

Floor below which nothing is guessed

1 of 3

Tiers that call a frontier model

Cached

Embeddings, so re-runs stay cheap

---

Conclusion

This project is a small one, but it is the clearest illustration of how we think about cost. "Use AI" is not a single decision with a single price. Routing work by difficulty — cheap embeddings for the obvious, a frontier model for the genuinely ambiguous, and an honest refusal below a confidence floor — is what makes AI economics work at scale.

The threshold that matters most is the bottom one. A system that declines to answer when it does not know is worth considerably more than one that always has an answer.

---