Brand Context API: Behind the Scenes

Why "just scrape the page and feed it to an LLM" for brand context works in the demo and falls apart the moment you ship — and what you're actually signing up to build.

The version that works

You want structured brand context for a domain: who the company is, how they position themselves, their voice, their products, their market. The obvious build is one you could sketch on a whiteboard in two minutes — fetch the homepage, hand the HTML to a frontier model, ask for JSON back.

Point it at apple.com and it's great. Point it at a dozen other household names and it's great. This is usually the exact moment a "let's just build it" decision gets made: on a demo, with brands the model already has memorized.

The version that doesn't

Now point it at a domain the model has never heard of — which is to say, almost all of them.

The model has no priors to fall back on. All it has is what you handed it, and what you handed it is raw HTML: nav bars, cookie banners, script tags, analytics blobs, a footer full of links. The model burns its limited attention on that noise, you pay for every junk token, and the answer comes back vague, generic, or quietly wrong.

The trap is that it still demos fine. You don't find out it's broken until you're running real traffic across the real web — every CMS, every template, every language, brands nobody's heard of. At that point it's not a prototype anymore; it's a thing that's wrong in production.

We measured how much the model actually does

We wanted to know how much of the final result really comes from the model versus everything around it. So we ran the experiment: we took our production system and stripped out everything upstream of the model — all the fetching, page selection, and cleaning — and gave the model only the schema and the domain. Then we scored the output the same way we score production.

Quality collapsed to essentially zero. Across every model we tried.

The takeaway is uncomfortable if you're planning to build this: the part you can stand up in an afternoon — "scrape the page, call an LLM" — is the part that contributes almost nothing on its own. The 90% that makes the output actually good is the part the demo conveniently hides.

What that 90% has to do

Get the content at all. Crawling sounds like the boring box in the diagram. At scale it's a product of its own, because the real web fights back — anti-bot defenses, Cloudflare, robots rules, redirect chains, timeouts, and pages that return empty while pretending they succeeded. We evaluated the available off-the-shelf crawlers; each one broke in a different way at the breadth and volume we needed, so we ended up building our own.

And a "better" crawler isn't automatically better for you. When ours started succeeding on pages the old one had been quietly skipping, output quality actually dropped — the system was suddenly being fed content it had never been tuned for. You don't catch that kind of regression by eyeballing a few demos. You catch it with a real evaluation harness, which is itself a thing you have to build before you can trust any change you make.

Figure out which pages matter. The homepage alone isn't enough, and you can't crawl an entire site for every request. Something has to reliably pick the handful of pages that actually carry brand signal and ignore the ones that don't.

Clean it. Strip the scaffolding so the model spends its attention on signal instead of noise. This is where most of the quality and most of the cost difference actually lives.

None of these are edge cases. At scale, the edge cases are the job — they're most of your traffic.

And it's never done

Say you get it working. It doesn't stay working for free. Models change. The web changes. Your own crawler regresses in subtle ways. Without an eval in place, you won't notice until a customer does.

Meanwhile the model — the part everyone fixates on — turns out to be the cheap, swappable, easy piece. Once you're above a quality floor, picking one is a cost decision, not a quality decision, which makes it a recurring optimization chore rather than a one-time choice. You'll be re-running that comparison every time something new ships.

What you'd actually be reimplementing

For reference, here's the surface area. You hand us a domain:

1curl \
2  -H 'Authorization: Bearer YOUR_KEY' \
3  https://api.brandfetch.io/v2/context/apple.com

…and you get back structured brand context — identity, positioning, voice, products, market, customer types:

1{
2  "meta":        { "domain": "...", "canonical_name": "..." },
3  "identity":    { "summary": "...", "tagline": "...", "category": "..." },
4  "positioning": { "summary": "...", "value_propositions": ["..."] },
5  "voice":       { "summary": "...", "tone_attributes": ["..."] },
6  "products":    [ { "name": "...", "summary": "...", "category": "..." } ],
7  "market":      { "segments": ["..."], "customer_types": ["..."] }
8}

Defining that shape is the easy part. Filling it in reliably, for a domain nobody has ever heard of is the entire game — and it's the part that doesn't fit on a whiteboard.

Build vs. Buy

You can absolutely build the afternoon version. It will demo beautifully. What you can't build in an afternoon — or a quarter — is the part that makes it reliable across the whole web, and then keep it reliable as the web and the models keep moving underneath you.

That's the actual work. It's most of the work. It's why this is an API instead of a blog post with a prompt in it.

If you'd rather spend your roadmap on your product instead of on a crawler and an eval harness, start here.