Status: Planned. No findings are published yet.
When someone asks an AI-assisted search product to recommend a local business, the answer can look authoritative. It is harder to tell whether that answer is stable, whether another product would name the same companies, or which sources the system will cite.
BizFacts is designing a repeatable study to measure those outputs.
The study will ask four practical questions:
- Which local businesses appear in recommendation responses?
- How often do identical tests produce the same business set?
- How much do different products overlap?
- Which websites and source domains are cited?
The purpose is to document what the products return under defined conditions and give business owners a firmer basis for judging whether visibility is repeatable or merely a one-off result.
The study at a glance
| Study element | Planned design |
|---|---|
| Cities | New York, Los Angeles, Chicago, Houston, Phoenix, Philadelphia, San Antonio, San Diego, Dallas, and Fort Worth |
| Categories | Plumbers, dentists, auto repair shops, and hair salons |
| Tracked businesses | 10 businesses per category in each city |
| Maximum tracking sample | 400 business-location-category records; fewer if an OpenStreetMap cell has fewer than 10 eligible records |
| Prompts | 4 exact formulations for every city-category pair |
| Repetition | 3 weekly observations for each prompt-product combination |
| Products | ChatGPT, Gemini, Claude, and Perplexity consumer web products |
| Planned responses | 1,920 if all four products remain eligible at fieldwork |
| Analysis | Tracked-business coverage, all-output recommendation frequency, repeatability, product overlap, and citation review |
The cities are the ten largest incorporated places in the Census Bureau’s Vintage 2025 estimates. This deliberately produces a large-city pilot rather than a nationally representative sample. The four categories are fixed in advance because they represent distinct local services and have established OpenStreetMap tags that make the starting lists reproducible.
How the tracking sample will be built
The study has two distinct populations. The tracking sample is the fixed set of local businesses selected before any prompts run. The recommendation output is every business the products actually name, whether or not it is in that sample. Tracking-sample measures will answer whether the products find a predefined cross-section of businesses; output measures will describe what the products choose to recommend.
Before fieldwork, BizFacts will use the OpenStreetMap Overpass API to download every named feature in a bounding box around each city with the relevant tag:
- plumbers:
craft=plumber; - dentists:
amenity=dentist; - auto repair shops:
shop=car_repair; - hair salons:
shop=hairdresser.
The geographic boundary will be the incorporated-place polygon in the Census Bureau’s 2025 TIGER/Line Place shapefile. The sample manifest will record each city’s five-digit Census place GEOID. A point or the centroid of a mapped way or relation must fall inside that polygon to qualify.
The Overpass query template is fixed below. The four tag lines will be queried together using the bounding coordinates of the relevant TIGER/Line polygon, then spatially filtered against that polygon:
[out:json][timeout:180];
(
nwr["craft"="plumber"]({south},{west},{north},{east});
nwr["amenity"="dentist"]({south},{west},{north},{east});
nwr["shop"="car_repair"]({south},{west},{north},{east});
nwr["shop"="hairdresser"]({south},{west},{north},{east});
);
out center tags;The substituted query, bounding coordinates, TIGER/Line files, Overpass response, download timestamps, and OpenStreetMap attribution will be archived with the study. A record will be eligible when it has a business name, falls inside the city polygon, is not tagged closed, disused, or abandoned, and includes an address, website, or phone number that can support entity matching.
Duplicate map records will be combined when their mapped centers are within 25 meters and they share either a normalized name, website hostname, or phone number. The retained record will be the one with the most matching identifiers, with the lowest OpenStreetMap element ID breaking a tie. Separate locations of the same chain will remain eligible and will be treated as separate establishments. Records outside the boundary, without a name, without a matching identifier, or explicitly marked inactive will be excluded.
For each city-category cell, the fields city|category|name|address|osm-element-type|osm-element-id will be Unicode-normalized with NFKC, converted to lowercase, trimmed, and have internal whitespace collapsed to one space. BizFacts will encode bizfacts-local-ai-pilot-v1| plus that string as UTF-8, calculate a SHA-256 hash, sort the lowercase hexadecimal hashes in ascending byte order, and select the first ten records. The script, source extracts, exclusions, and final sample will be published before AI collection begins.
If an OpenStreetMap cell contains fewer than ten eligible records, BizFacts will include every eligible record and report the smaller denominator. It will not fill the cell from search-engine results or AI recommendations.
Products, prompts, and field dates
The product set is fixed as the U.S. consumer web versions of ChatGPT, Gemini, Claude, and Perplexity. A product will be included only if it is publicly available in the United States and can return a web-backed answer on the sample-freeze date. An unavailable product will be reported as excluded rather than replaced after collection starts.
Tests will use new study-only accounts where sign-in is required, with no prior conversation history and location permissions disabled. Each product’s default general-purpose model will be used, with web search enabled when the interface provides that control. The visible product and model labels will be recorded because defaults can change.
Every city-category pair will receive these four exact prompts:
Recommend a [category] in [city].What are the best [plural category] in [city]?Which [category] in [city] should I consider?I need a [category] in [city]. What local businesses should I look at?
The category labels will be plumber, dentist, auto repair shop, and hair salon, with the corresponding regular plurals in prompt 2.
The sample and product list will be frozen before the first prompt. Fieldwork will begin on the first Monday at least seven calendar days later. The same test set will run once on that Monday and again on the following two Mondays, creating three weekly waves. Exact dates and any documented outage rescheduling will be added to the protocol before results are published.
Within each wave and product, prompts will be ordered by the SHA-256 hash of bizfacts-prompt-order-v1|wave|product|city|category|prompt-id, using the same text-normalization and sorting rules as the business sample. This fixes the order without letting a researcher choose it after seeing results.
Why a repeated test is necessary
A single recommendation is only one observation. AI outputs may change across products, model versions, dates, account states, locations, and repeated runs of the same prompt.
The fixed prompts do not name any tracked business. Running them in three weekly waves will allow the report to distinguish a company that appears once from one that appears consistently.
How one observation will work
Where the product permits it, a tester will begin in a fresh session and submit one of the fixed prompts. The study log will capture:
- the exact prompt and response;
- product name and visible model or version label;
- date, time, and language;
- whether the tester was signed in;
- the location supplied in the prompt;
- any product location setting visible to the tester;
- whether web search or a related retrieval control was enabled;
- businesses mentioned and their order;
- URLs and domains cited or linked;
- whether each citation resolves and supports the nearby statement;
- refusals, caveats, or expressions of uncertainty.
The fixed hash order above will determine prompt order. A model-label change will be recorded but collection will continue. If a product changes its default model or web-retrieval setting during the three waves, observations under the new configuration will form a separate product-version stratum and will not be combined with the earlier configuration for repeatability measures. A temporary outage will be retried once within 48 hours and marked as rescheduled. If a product becomes unavailable for the rest of fieldwork, its missing observations will be reported; it will not be silently replaced and the other products will not be restarted.
The draft protocol schemas are available now: observations, business mentions, and citations. They contain headers only—not study results. Separate files prevent a response with several businesses and several citations from being flattened into ambiguous duplicate rows.
Matching a recommendation to a real business
A business will count as a match only when the response can be connected to a specific company using at least two identifiers, such as:
- name and website;
- name and address;
- name and phone number;
- name and location.
Franchises and companies with multiple locations will be coded at both the brand and establishment level when the response provides enough information.
Ambiguous names will remain unresolved rather than being assigned by guesswork. Two reviewers will code ambiguous matches independently. If they disagree, a third reviewer will adjudicate; the two original decisions and the final decision will remain in the audit file.
Every named recommendation will be captured, including businesses outside the tracking sample. Resolved recommendations will receive stable establishment and brand IDs. Unresolved names will remain in the mention file with an unresolved status, but they will not be treated as tracked-business matches or included in set-similarity calculations.
What the final report will measure
| Measure | What it will show |
|---|---|
| Tracked-business coverage | For each sampled business, valid observations in its city-category cell that mention it, divided by all valid observations in that cell |
| Recommendation frequency | For every resolved business returned, valid observations in its city-category cell that mention it, divided by all valid observations in that cell |
| First-position rate | For every resolved business, valid observations in which it is the first clearly ordered recommendation, divided by all valid observations in its city-category cell; an unordered answer contributes no first-position mention but stays in the denominator |
| Repeat consistency | Pairwise comparisons of the resolved establishment-ID sets for the same product version, prompt, city, and category across waves; pairs with one or two empty sets are reported separately and excluded from the percentage |
| Product overlap | Jaccard similarity of resolved establishment-ID sets for the same city, category, prompt, and wave; pairs with one or two empty sets are reported separately and excluded from the average |
| Citation-domain share | Unique (observation_id, canonical_url) pairs for a domain divided by all unique (observation_id, canonical_url) pairs |
| Citation support rate | Citation occurrences that resolve and support the associated claim divided by every captured citation occurrence; the pilot uses a full review rather than a sample |
A valid observation is a scheduled prompt for which the complete response was captured. A refusal or a complete answer with no business recommendation is valid and remains in rate denominators as an empty result. A technical outage, truncated capture, or accidental protocol deviation is invalid, excluded from denominators, and retried once under the rescheduling rule. Missing citations, unordered answers, refusals, and unresolved businesses receive explicit codes rather than disappearing from the analysis.
This first pilot will not attempt to explain visibility using review counts, rankings, business age, or directory coverage. Those comparisons would require additional sampling choices and are better treated as follow-up studies. Website, address, and phone data will be used only to resolve business identities.
What the platforms document
OpenAI documents ChatGPT search and the OAI-SearchBot crawler. Google describes website eligibility for its AI features through ordinary Search requirements. Perplexity documents crawlers used for retrieval and indexing. Anthropic documents web search with citations.
Those sources establish that the products can retrieve current web information and present citations in relevant experiences. They do not document a local-business formula in which a specified number of reviews, links, mentions, or directory listings produces a recommendation.
A citation will be recorded as a source shown in the response. The study will not treat it as a documented ranking factor.
Quality and publication commitments
Before results are published, BizFacts will:
- define and retain the complete prompt set;
- record the products and visible model labels tested;
- report field dates, exclusions, refusals, and missing-data counts;
- use two reviewers for ambiguous entity matches;
- review every captured citation occurrence for resolution and support of the associated claim, with a second reviewer checking all disputed cases;
- publish formulas or analysis code;
- release raw data or a privacy-safe summarized dataset;
- label findings outside the preregistered analysis as exploratory.
Percentages and rankings will not be published until collection and review are complete.
Limits of the study
The pilot will cover four categories and ten cities, so it will not represent every industry or local market. Account state, inferred location, personalization, retrieval availability, and product changes may not always be fully observable.
Because the study is observational, it can identify patterns and associations but not establish what caused a recommendation. Recommendation presence is also not a judgment of business quality or customer suitability.
When fieldwork and quality review are complete, this page will be updated with collection dates, the final sample, findings, analysis files, data, and a revision log.