Track competitor prices by starting with a small matched set of 50 to 200 products, collecting from public pages at a polite rate, storing every observation with a timestamp and the evidence behind it, and reviewing matches before anybody is allowed to act on a number. Breaking something means one of three things: getting blocked, breaking a rule you are bound by, or breaking trust in the data.
The third happens most often and gets noticed last. Everything below is arranged to prevent it.
What breaking something means here
- Technical breakage: your requests get blocked, rate limited, or served a different page than a human sees. Symptom, coverage falls and nobody says why.
- Legal and contractual breakage: you ignore a site's terms, collect from behind a login, or hold personal data you have no reason to hold.
- Analytical breakage: everything runs and the numbers are wrong. A pack size changed, a match drifted, a promotional price got stored as a list price. This is the expensive one.
Step 1: choose the products before you choose the tool
Do not start with the whole catalogue. Start where a pricing decision is worth money.
- Your top 50 by gross profit contribution, not by revenue
- The 20 products your sales team gets challenged on most often
- Anything where you suspect you are cheapest and cannot explain why
A hundred products you understand beats ten thousand you do not.
Step 2: pick competitors you actually lose to
Ask sales which names appear in lost deals. Ask customer service which sites customers quote back at them. That list is usually 3 to 6 companies, and it is rarely the list the board assumes.
A competitor whose prices you cannot act on does not belong in the first version. Neither does one you never meet in a real deal.
Step 3: match by hand first, then verify
Match the first hundred pairs manually. It is dull, and it will teach you more about your data than any dashboard.
Record why you believe each pair is a match: identical manufacturer part number, identical GTIN, or same title plus same pack size plus same image. Then have somebody who did not make the matches check twenty at random. If more than two of the twenty are wrong, your matching rule is not good enough to automate yet.
Step 4: collect politely
- Read robots.txt and honour it. If it disallows you, that is a decision for your legal team, not for a config file.
- Rate limit per domain, not globally. One request every two to five seconds to a single site is defensible. Twenty in parallel is not.
- Send an honest user agent with a contact address. Sites that would have blocked you sometimes email instead.
- Avoid peak trading hours. Nobody minds a crawler at 04:00.
- Back off on 429 and 503 responses rather than retrying immediately, and stop entirely after repeated refusals.
- Never log in, never bypass a paywall, never accept terms on the site's behalf.
Do the sums before you worry. 200 products across 5 competitors is 1,000 page reads. At one request every three seconds per domain, each competitor sees 200 requests over 600 seconds, so ten minutes of light traffic per site per day. Nobody notices that load.
Step 5: store the observation, not just the number
Overwriting yesterday's price with today's destroys the only asset you are building.
- Price, currency and tax treatment
- Timestamp in UTC, plus the local time the site displayed
- Availability, and any delivery threshold shown
- The URL, and a stored copy or hash of the page section you read
- Which collection method produced it, since methods differ in reliability
- Whether the value was parsed confidently or inferred
Step 6: measure the collection before you trust it
Two numbers belong on the same screen as every price report.
- Coverage: of the matched pairs you expect, what share returned a price in the last cycle. If it is 71 per cent, say 71 per cent. Do not quietly report the average of the 71.
- Freshness: the median age of the observations behind the report. A three day median on a daily job means roughly a third of your collections are failing.
A price report without those two numbers is a confidence trick, including when you are the one running it.
When a page breaks, and it will
Sites redesign. Parsers fail. The failure mode to design against is silence: a parser returning nothing looks identical to a competitor with no price.
- Alert on a drop in coverage per competitor, not on individual errors
- Hold any price that moves more than a set percentage for review instead of publishing it
- Label the last known good value as stale rather than showing it as current
The review step you should not automate away
Give one person a daily queue of changed prices and broken matches, capped at what can be cleared in fifteen minutes. Everything else can be automated later.
That queue is the reason the data stays true, and it is the first thing teams delete when they get busy. Deleting it is how a monitoring project quietly becomes a monitoring theatre.