Catalogue AI: classifying new SKUs and cleaning 182 000 product cards

Two problems had to be solved together: classifying every new SKU into one of about 1 400 leaf categories, and finding duplicate cards that exact string matching cannot see. Classification is a large multi-class problem with a long tail, multilingual descriptions and inconsistent supplier attributes. Duplicates differ in word order, units, abbreviations and language, so a name match finds almost nothing; even a shared GS1 barcode (GTIN) only catches the subset of duplicates where suppliers filled the field correctly. Both models had to plug into the existing item-onboarding process without a new tool to learn, and a human always had to confirm.

The category manager sees three ranked suggestions with probabilities and confirms with one key. Corrections go back into the training set the same night.

It was built as two models behind one confirmation interface: a classifier that returns the top-3 category codes with calibrated probabilities, and a duplicate finder that embeds every card semantically and searches for near neighbours. Approximate nearest-neighbour search is the standard way to compare hundreds of thousands of embeddings quickly; the FAISS library used here was published by Meta researchers in 2017 and remains the reference implementation. Every human confirmation feeds a weekly retraining run.

• Classifier. A gradient-boosted model over text embeddings (multilingual sentence transformers) and structured attributes (composition, brand, unit, storage), trained on the history of assigned codes. A confidence threshold decides whether the top-1 is pre-selected or the manager must choose.
• Duplicate finder. Cards are embedded (name + attributes + barcode + description); FAISS produces candidate pairs, a pairwise model scores them, and candidates are grouped and ranked. Same-EAN pairs are surfaced first as certain duplicates.
• Confirmation UI. A merge-review screen shows both cards with attribute completeness and stock coverage, proposes which card to keep, and records the decision. Regular reruns catch new duplicates as they are entered.
• Retraining loop. Confirmations and corrections are collected daily; the classifier retrains weekly (Airflow) with an automatic accuracy gate before deployment.

The table compares the rule-based approach the client had tried (keyword rules and exact matching) with the learned models that replaced it. Rules are transparent but brittle: each new supplier spelling needs a new rule. Learned models cost more to set up and need labelled history, but they generalise to spellings and languages nobody wrote a rule for.

Accuracy is measured on the items the category managers actually confirmed — the honest metric — and improved every week as corrections fed back into training.

Category managers process duplicate groups in a dedicated queue; most decisions take seconds because the evidence — same EAN, attribute completeness, where the stock is — is on the screen.

The stack is Python end to end: scikit-learn and LightGBM for the classifier, Sentence Transformers for multilingual embeddings, FAISS for approximate nearest-neighbour search, FastAPI inference services, PostgreSQL for cards and decisions, and Airflow for weekly retraining and scheduled duplicate scans. The Vue confirmation interface is embedded in the existing item-onboarding tool through its API, so category managers never leave the screen they already use.

This approach is for any company whose product master is maintained by people and fed by many suppliers: grocery and DIY retail, pharmacy chains, spare-parts distributors, marketplaces. Clean master data is the foundation half of an AI roadmap stands on — demand forecasting, substitutions for out-of-stock items and conversational BI all depend on one card per product. McKinsey (2023) puts the annual value of generative AI in retail at roughly 400–660 billion dollars, and most of that value assumes clean catalogue data. Regulation is pushing the same way: the EU Ecodesign Regulation (2024) introduces a Digital Product Passport that will require consistent product identifiers and attributes per item.

It is the project we recommend starting with when the data is ready and the business needs a visible result in weeks.