The Unstable Answer: A 332-Market Study of ChatGPT Law Firm Recommendations
12AM Agency Research – 748 Recorded ChatGPT Prompt Runs Across Personal Injury, Family & Medical Malpractice
Original Research · Published August 24, 2026 · Version 1.1 · Research lead: Robert Portillo, Co-CEO, 12AM Agency · Engine studied: ChatGPT only, July 27–30, 2026 · Methodology · Key statistics · Observation ledger: CounselIndex
From anecdotes to evidence
What Did the 332-Market ChatGPT Study Measure?
| Outcome | Markets | Share |
|---|---|---|
| Shared zero of three firms | 72 of 231 | 31.2% |
| Kept the same first-named firm | 65 of 231 | 28.1% |
| Agreed on all three firms | 15 of 231 | 6.5% |
| Mean Jaccard overlap | — | 0.26 (median 0.20) |
Source: 12AM Agency ChatGPT sweep, July 27–30, 2026. Both phrasings run same-day per market.
Does Rephrasing Change ChatGPT's Law Firm Recommendations?
In this sweep, yes — most of the time. ChatGPT did not behave like a ranking. It assembled each answer fresh, and the answer moved when the question moved.
We ran the family-law sweep twice against all 231 markets on the same day: "best divorce lawyers in X" vs "best family law attorneys in X." In 72 of 231 markets (31.2%) the two three-firm answers shared zero firms. Only 65 of 231 (28.1%) kept the same first-named firm, and just 15 of 231 (6.5%) agreed on all three. The mean overlap of the two answer sets was 0.26 (Jaccard; median 0.20). In plain English: on average, only about a quarter of the combined firm names showed up in both answers.
- A single "ChatGPT recommends us" screenshot is an anecdote, not a position
- Any single-number "AI share of voice" — how often a firm appears across a defined set of prompts — is meaningless without that prompt set disclosed
What We Set Out to Measure
Replace vendor screenshots with recorded, dated, reproducible observations of which firms ChatGPT named when a consumer asked for a lawyer.
To do that, the study needed to:
- Record the verbatim consumer prompt, market, response order, and date for every run
- Cover every state plus DC — 51 statewide and 281 metro markets
- Test answer stability across prompt phrasings
- Match results against a pre-defined 13,053-firm practice-matched denominator to measure who never appears
— 12AM Agency Research Team, 2026
How Was the Law Firm AI Visibility Study Conducted?
| Pillar | Detail |
|---|---|
| 748 prompt runs | July 27–30, 2026, one engine (ChatGPT). 744 usable; the four excluded runs returned no capturable firm names. |
| 332 unique markets | 51 statewide + 281 metro, all 50 states plus DC. |
| 2,232 firm-mentions | Top three named firms captured per answer — 1,497 distinct firm entities after normalization, from 1,565 raw name strings. |
| Re-derived twice | Every figure re-derived from raw sweep files before publication, once at publication and once under a stricter entity-normalization robustness check. Claims that failed re-derivation were retired, not shipped. |
| Practice area | Runs | Markets | Prompt phrasings | Repeat design |
|---|---|---|---|---|
| Personal injury | 224 | 224 | 1 | Single pass |
| Family & divorce | 462 | 231 | 2 | Every market run under both phrasings, same day |
| Medical malpractice | 62 | 57 | 2 | Partial repeat |
Two units, stated plainly. 1,497 distinct firm entities were captured across all answers. 438 of them matched the pre-defined 13,053-firm practice-matched denominator (business-profile records in the swept practice areas and geographies) used for the rarity rates. The remaining entities fell outside that measured universe — firms not present in the source business-profile export, or names that could not be matched to a unique record. Rates such as 3.36% and 96.6% use only the matched denominator. The full entity-resolution protocol and exclusion categories publish in the methodology appendix.
What we will not claim: this study measured ChatGPT only, during four days in July 2026. It should not be generalized to Gemini, Perplexity, Google AI Overviews, or future ChatGPT behavior. And with one to two prompts per market, no firm's "probability of being named" is estimable — we don't publish one.
What the Study Found About ChatGPT's Law Firm Recommendations
Finding 1: Naming Was Rare in This Sweep
| Key number | What it measures |
|---|---|
| 96.6% | Of the 13,053 practice-matched firms, never appeared in a single recorded answer |
| 438 firms (3.36%) | Named anywhere in their practice sweep |
| 362 firms (2.77%) | Named in their own state |
| 1,497 | Distinct firm entities captured across all answers |
Within the measured universe, appearing in a ChatGPT recommendation was rare — far rarer than most firms assume.
Finding 2: No Firm Dominated This Sweep
| Key number | What it measures |
|---|---|
| 1,146 of 1,497 (76.6%) | Named firms that appeared in exactly one observed market |
| 6 of 332 | Most markets held by any single firm |
| 18.3% | Share of all 2,232 mentions held by the top 100 firms |
No dominant national firm emerged in this sample. In the markets we observed, visibility was earned market by market.
Finding 3: The Strongest Association We Measured — Review Volume
| Key number | What it measures |
|---|---|
| 1.4% → 15.0% | Naming rate rose every bucket: 1.4% (0 reviews), 2.3% (1–50), 6.0% (51–200), 7.6% (201–500), 15.0% (501+) |
| χ² = 191.2, df 4, p = 3×10−40 | Unadjusted. The pattern is extremely unlikely to be random in this dataset. It does not establish causation |
| 61,409 firms | Separate 12AM/CounselIndex census analysis showing a consistent direction |
Firms in higher review-count buckets were named at higher rates. The study does not show that adding reviews causes ChatGPT to name a firm — firm size, market, and other unmeasured variables could contribute. It shows review volume was the strongest single association we measured.
Share of firms named, by review volume
| Reviews | Named rate |
|---|---|
| 0 | 1.4% |
| 1–50 | 2.3% |
| 51–200 | 6.0% |
| 201–500 | 7.6% |
| 501+ | 15.0% |
n = 13,053 practice-matched firms · χ² = 191.2, df 4, p = 3×10−40 · unadjusted association, not causal.
Key statistics from this study (quote with attribution)
| Statistic | Finding | Attribution |
|---|---|---|
| 31.2% / 28.1% | In 231 family-law markets tested twice on the same day in ChatGPT, 72 of 231 answer pairs (31.2%) shared no firm, and only 65 of 231 (28.1%) kept the same first-named firm. | 12AM Agency, July 2026 |
| 96.6% | In 12AM Agency's July 27–30, 2026 ChatGPT sweep, 438 of 13,053 practice-matched U.S. law firms (3.36%) were named at least once; 96.6% never appeared. | 12AM Agency, 2026 |
| 76.6% | Of 1,497 distinct firm entities captured, 1,146 (76.6%) appeared in only one observed market, and no firm appeared in more than 6 of 332 markets. | 12AM Agency, 2026 |
| 52.1% | 52.1% of metro-level ChatGPT answers (281 of 539 pairs) shared no firm with their own statewide answer. | 12AM Agency, 2026 |
| 2.3% → 15.0% | Naming rates rose from 2.3% in the 1–50-review bucket to 15.0% in the 501+ bucket — an unadjusted association, not a causal estimate. | 12AM Agency, 2026 |
What These Findings Mean for a Law Firm
1. Rarity cuts both ways
96.6% of measured firms never appeared. That is the threat. The implication: the answer set in most markets is thin, so a firm that builds the signals machines can read is not fighting an entrenched incumbent — in this sample, no such incumbent existed.
2. Instability demands measurement
With 31.2% of markets sharing zero firms between phrasings, one screenshot proves nothing — including a competitor's. The implication: track your market with repeated, dated, multi-phrasing observations before spending on optimization.
3. The record is the asset
Review volume showed the strongest association we measured. Causation is not established, but the implication is practical: documented, machine-readable evidence at volume is the rational place to invest while the research matures.
The AI Visibility Diagnostic includes your market's recorded answers and a signal-by-signal scorecard.
Visibility Was Earned Market by Market
1,146 of 1,497 named firms appeared in exactly one observed market and none held more than 6 of 332. In this sweep there was no national default for ChatGPT to fall back on — local signals decided local answers.
Screenshots Aren't Rankings
With 31.2% of markets sharing zero firms between two phrasings of the same question, any single screenshot or undisclosed "AI share of voice" number is an anecdote. Only repeated, dated, multi-phrasing measurement counts as evidence.
Review Volume Showed the Strongest Gradient
Firms with 501+ reviews were named at 6.5× the rate of firms with 1–50 in this sweep. An association, not proof of cause — but no other signal we measured came close, and it replicated directionally in our separate 61,409-firm census analysis.
Measurement Is the First Step
Because the answer set moved with phrasing, the honest starting point is measurement: recorded, dated observations of your own market, repeated across phrasings — before a dollar goes to optimization. That standard is why this study exists.
When Was the Study Run and Verified?
A four-week arc from raw prompt runs to published, re-verified findings.
July 27–30, 2026 — The Sweep
- 748 recorded ChatGPT prompt runs across 332 markets
- Personal injury, family & divorce, medical malpractice
- Five consumer prompt phrasings, verbatim, dated
August 2026 — Verification
- Every headline figure re-derived from the raw sweep CSVs; claims that failed re-derivation were retired
- Results matched against the 13,053-firm practice-matched denominator
- Independent robustness check under stricter entity normalization confirmed direction and magnitude of all headline findings
August 2026 — Publication
- Findings published ungated, limitations stated plainly, version 1.1
- Observation ledger publishes at CounselIndex, the public index we operate
- Companion research: a 61,409-firm census and a blinded 400-firm study, both by 12AM/CounselIndex
— Robert Portillo, Founder & Co-CEO, 12AM Agency
Questions This Study Can Answer
Does ChatGPT recommend law firms?
Yes. In this July 2026 sweep, ChatGPT returned named law firm recommendations for consumer prompts like "best divorce lawyers in [city]" in nearly every one of 748 recorded runs — 744 of 748 answers named specific firms.
How often did ChatGPT name a given law firm?
Rarely. Of 13,053 practice-matched U.S. firms, 438 (3.36%) were named at least once anywhere, and 362 (2.77%) were named in their own state. 96.6% never appeared in any recorded answer.
Does changing the prompt change which lawyers ChatGPT recommends?
Usually. Across 231 family-law markets run under two phrasings on the same day, 31.2% of answer pairs shared zero firms and only 28.1% kept the same first-named firm. 52.1% of metro answers also shared no firm with their own statewide answer.
Are Google reviews associated with ChatGPT recommendations?
In this dataset, yes — higher review volume was associated with higher naming rates, from 1.4% at zero reviews to 15.0% at 501+ reviews. This is an unadjusted association; the study does not establish that reviews cause naming.
Did this study test Gemini, Perplexity, or Google AI Overviews?
No. This sweep measured ChatGPT only, July 27–30, 2026, and its findings should not be generalized to other engines or dates. Multi-engine measurement is planned in our ongoing observation ledger.
How can a law firm measure its own AI visibility?
The same way this study did: recorded, dated observations of real consumer prompts in your market, repeated across phrasings. Our AI Visibility Diagnostic runs that measurement for your firm and scores the signals associated with being named.
In this sweep, ChatGPT behaved less like a ranking and more like a fresh draw from the public record it can read about each market.
The firms that kept reappearing across phrasings were the ones with deep documented footprints — and review volume was the single strongest association we could verify.
The measured takeaway: the answer set was rare, unstable, and evidence-linked — which makes disciplined measurement, then the public record, the assets worth building.
Request Your Firm's Diagnostic
You’ll get your market’s recorded ChatGPT answers, your firm’s signal-by-signal score on the 100-point model behind this research, and the prioritized fix list.