A confidence score is the matcher's own estimate that two listings are the same product. It is useful for deciding what to do with a pair, and it is not evidence that the pair is correct. Use it to route work, not to close the question.
Full automation loses because the cost of a mistake is lopsided. A missed match costs you one row of visibility. A wrong match feeds a real number from the wrong product into a pricing decision, and there is nothing downstream that catches it.
What the number actually means
A score can have two different properties, and teams routinely assume both when they have only the first.
- Ranking: pairs with higher scores are more likely correct than pairs with lower scores. Almost any scoring function gets this roughly right.
- Calibration: out of every hundred pairs scored 0.90, about ninety are correct. Very few scoring functions have this property unless they were explicitly fitted for it.
If your score only ranks, then 0.90 means nothing on its own and the threshold you chose because it sounded confident is arbitrary. Checking calibration is cheap: take the pairs in each score band, review a sample by hand and compare observed accuracy against the band. If the 0.90 band comes out at 72 per cent correct, the score is a rank, not a probability, and you should stop reading it as one.
The arithmetic of a threshold
Suppose ten thousand candidate pairs and a threshold that auto accepts eight thousand of them at 97 per cent precision. Three per cent of eight thousand is 240 wrong matches sitting in your data, each one a competitor price attached to the wrong product.
Raise the threshold so that only six thousand auto accept at 99.5 per cent precision and you are left with 30 wrong matches, plus two thousand pairs pushed into review. Whether that trade is worth making depends on one comparison: what one wrong match costs against what one hour of review costs.
Work it through with your own figures. If a wrong match leads you to cut a price by 3 per cent on a product turning over 2,000 euro a month, that error costs 60 euro a month while it stands, and thirty of them cost 1,800 euro a month. A reviewer deciding a pair every fifteen seconds handles 240 an hour, so two thousand extra pairs are about eight hours of work once, plus a trickle afterwards. Put your numbers into those three lines and the decision usually makes itself.
Three bands, not one threshold
- Auto accept: high score, at least one hard identifier agreeing, and no contradiction on a blocking attribute such as pack size or vintage.
- Review: everything in the middle, plus everything with a high score and a contradiction. High text similarity with a mismatched quantity belongs here, not in accept.
- Auto reject: low score with no identifier agreement. Store the rejection, because it is a label.
Blocking attributes matter more than the score itself. Define, per category, the fields where disagreement forces review no matter how confident the model is. In pharmacy that is strength and pack count. In construction it is dimension and grade. In wine it is vintage and bottle size.
Designing a queue people actually clear
Review queues fail for human reasons rather than technical ones. They are dull, unbounded and ordered wrongly.
- Order by money at risk, not by score. A 0.62 on a product doing 40,000 euro a year outranks a 0.88 on a product doing 400.
- Batch by competitor. Somebody who has just read twenty pages from one shop reads the twenty first in seconds.
- Show both listings side by side with the disagreeing fields highlighted. Do not make a person open two browser tabs.
- One keystroke per decision: yes, no, or not enough information. The third option is data, not failure.
- Cap the queue. If it grows faster than it is cleared, the thresholds are wrong and willpower will not fix that.
- Record who decided and on what basis. A decision without a reason cannot be audited or reused.
Every decision is a training label
The queue is not overhead. It is the only dependable source of ground truth you will ever have about your own catalogue, and keeping it costs nothing extra.
- Rejections are worth more than acceptances. Near misses are exactly what a scorer gets wrong, and a set of confirmed near misses is the hardest data to buy.
- Freeze a golden set of a few hundred hand-checked pairs, positive and negative, and never let the matcher train on it. Score every change against it.
- Re-score the whole catalogue after a rule change and diff the result. A change that improves the golden set while flipping two thousand live matches needs a human look before it ships.
PriceRoom keeps the review decision as a first class record for this reason: matchers improve, competitor pages change, and you need to know which pairs a person actually confirmed.
When automation is fine
None of this argues against automation. It argues against unsupervised automation at the boundary.
- Two valid GTINs that agree, with pack count and quantity also agreeing: accept automatically.
- A pair a reviewer confirmed this quarter, where neither page's identity fields have changed: keep it and do not re-review.
- A competitor page that has been deleted: unmatch automatically and tell somebody.
The middle band is where judgement is needed, and the middle band is usually small. That is the whole argument. You are not hiring a team, you are giving one person a well ordered hour a day.
The honest version
A confidence score makes the size of your doubt explicit so you can decide what to do about it. A system with no scores is not more confident, it is only less honest about where it will be wrong.