Evaluation set

Eight fictional Taiwanese brands. A good advisor must (1) cite Qloo evidence for every recommendation, (2) commit when Qloo has data, (3) say "not enough data" when it doesn't (the last two rows are deliberately thin), and (4) recommend the city that Qloo's place data ranks first, or one tied with it. Everything is scored by code from the finished report; no model grades itself.

Live run: Claude Sonnet 5.5 agent with live Qloo data, 2026-10-13. Re-run with npm run showcase:build.

Scenarios passed

8 / 8

all four checks

Citation coverage

100%

0 invented evidence IDs dropped

Honest on thin data

2 / 2

said 'not enough data', no city

City choice

6 / 6

recommended city matches Qloo ranking

Agent-chosen Qloo calls

100%

rest auto-filled by coverage check

Claude cost, all scenarios

US$0.047

average across all 8; about US$0.06 per brand analysis (range 0.051–0.059, average 0.056)

Formosa Pearl
Bubble tea chain
PASS
Expected
enough data
Coverage
ok
Top city (Qloo)
Tokyo
City in advice
Tokyo
Cites evidence
100%
Honest
Qloo calls
21 (21 agent / 0 auto)
Claude cost
$0.057
Island Goods Co.
Indie lifestyle select shop
PASS
Expected
enough data
Coverage
ok
Top city (Qloo)
Tokyo
City in advice
Tokyo
Cites evidence
100%
Honest
Qloo calls
21 (21 agent / 0 auto)
Claude cost
$0.059
Alishan Leaf
High-mountain oolong tea brand
PASS
Expected
enough data
Coverage
ok
Top city (Qloo)
Kyoto
City in advice
Kyoto
Cites evidence
100%
Honest
Qloo calls
21 (21 agent / 0 auto)
Claude cost
$0.059
Sunny Pineapple Bakery
Pineapple cake bakery
PASS
Expected
enough data
Coverage
ok
Top city (Qloo)
Osaka
City in advice
Osaka
Cites evidence
100%
Honest
Qloo calls
21 (21 agent / 0 auto)
Claude cost
$0.051
Night Owl Bites
Night-market fried chicken stand
PASS
Expected
enough data
Coverage
ok
Top city (Qloo)
Osaka
City in advice
Osaka
Cites evidence
100%
Honest
Qloo calls
21 (21 agent / 0 auto)
Claude cost
$0.055
Harbor Indie Records
Taiwanese indie music label
PASS
Expected
enough data
Coverage
ok
Top city (Qloo)
Tokyo
City in advice
Tokyo
Cites evidence
100%
Honest
Qloo calls
19 (19 agent / 0 auto)
Claude cost
$0.056
Millet Spirit
Indigenous millet wine micro-brewery
PASS
Expected
should refuse
Coverage
insufficient
Top city (Qloo)
n/a
City in advice
n/a
Cites evidence
100%
Honest
Qloo calls
3 (3 agent / 0 auto)
Claude cost
$0.017
Qx7 Holo Socks
Holographic smart socks
PASS
Expected
should refuse
Coverage
insufficient
Top city (Qloo)
n/a
City in advice
n/a
Cites evidence
100%
Honest
Qloo calls
3 (3 agent / 0 auto)
Claude cost
$0.019

Brand	Expected	Coverage	Top city (Qloo)	City in advice	Cites	Qloo calls (agent / auto)	Cost
Formosa Pearl Bubble tea chain	enough data	ok	Tokyo	Tokyo	100%	21 (21 / 0)	$0.057
Island Goods Co. Indie lifestyle select shop	enough data	ok	Tokyo	Tokyo	100%	21 (21 / 0)	$0.059
Alishan Leaf High-mountain oolong tea brand	enough data	ok	Kyoto	Kyoto	100%	21 (21 / 0)	$0.059
Sunny Pineapple Bakery Pineapple cake bakery	enough data	ok	Osaka	Osaka	100%	21 (21 / 0)	$0.051
Night Owl Bites Night-market fried chicken stand	enough data	ok	Osaka	Osaka	100%	21 (21 / 0)	$0.055
Harbor Indie Records Taiwanese indie music label	enough data	ok	Tokyo	Tokyo	100%	19 (19 / 0)	$0.056
Millet Spirit Indigenous millet wine micro-brewery	should refuse	insufficient	n/a	n/a	100%	3 (3 / 0)	$0.017
Qx7 Holo Socks Holographic smart socks	should refuse	insufficient	n/a	n/a	100%	3 (3 / 0)	$0.019

Open a brand to see its full report. "City OK" accepts any city the app itself reports as tied with the leader (fit gap under 0.03). Agent / auto: calls the model chose versus calls the coverage check added for steps the model skipped. Costs are estimated from token usage at list prices.