# Routing a prompt between Gemini Nano and the cloud

Source: https://tpiros.dev/blog/hybrid-ai-in-the-browser

Chrome ships a language model, and so does Edge. The [Prompt API](https://developer.chrome.com/docs/ai/prompt-api) hands you a session against it, and the [AI SDK has a community provider](https://ai-sdk.dev/providers/community-providers/browser-ai) that wraps it. So the code asking a local model for an answer is the same code that asks a datacentre.

What I built on top of that is a single page running on Vite. A prompt box, three tiers behind it, and a router that picks one and tells you which rule fired.

# Two tiers, one call site

```ts
// on your machine. no network, no cost
streamText({ model: browserAI('text'), prompt })

// in a datacentre. bigger, smarter, metered
streamText({ model: google('gemini-3.5-flash-lite'), prompt })
```

The function, the options and the streaming interface are identical. Only the model changes.

The cloud half lives in a Vite dev-server middleware so the API key never reaches the browser. The on-device half needs no server at all.

# What browserAI() actually is

[`@browser-ai/core`](https://ai-sdk.dev/providers/community-providers/browser-ai) is a community provider for the AI SDK, and it ships no model of its own. It wraps the browser's global `LanguageModel`, which is the Prompt API, and hands the SDK a `LanguageModelV4` implementation on top.

Which model answers depends on the browser. Chrome backs it with Gemini Nano, and the provider docs list Phi-4-mini for Edge. The package itself names neither. When the API is missing it says only that it "requires Chrome or Edge browser with browser AI capabilities".

Two siblings sit alongside it at the same version:

```bash
npm i @browser-ai/core             # whatever model the browser ships
npm i @browser-ai/web-llm          # open models over WebGPU
npm i @browser-ai/transformers-js  # Hugging Face models via Transformers.js
```

That third package matters for machines with no native model, which is most of them. They can pull down a small open model over WebGPU and still answer locally, instead of handing the request to a datacentre. This demo sends them to the cloud anyway, because 2GB is already a lot to ask, but the option is sitting right there.

# The World Cup problem

Ask a model whose training stopped in 2025 who won the last World Cup and it says Argentina, 2022, with total confidence. The 2026 tournament finished two months ago. Spain won it.

Escalating to the bigger model does nothing. The cloud model has more parameters and the same training cutoff, so both tiers give you the same wrong answer.

The instinct here is to tell the model what day it is, or to instruct it to flag stale training data. Both leave the answer exactly where it was.

The date changes nothing, and the model doesn't even hedge. Telling it to flag stale training data only works where it already feels uncertain. The question about software versions gets a refusal. The football match doesn't, though both answers are equally out of date.

Attaching Google Search costs one line and about 380ms.

For anything past the training cutoff you need retrieval, and parameter count has nothing to do with it. So the same cloud model with search attached is a third tier.

Search is the model's choice rather than yours, and it occasionally answers from memory instead. A stale answer looks identical to a fresh one, so the source count goes on screen. `0 sources` on a grounded answer means it never searched.

# The rules

Eight rules across three tiers. First match wins, and the page highlights whichever one fired.

1. **No network.** Run on-device. If there's no on-device model either, nothing can answer, so the page says that instead of firing a request at a tier that can't run.
2. **Prompt too long.** Use the cloud. Nano gets about 9,216 tokens, covering input and output together. The check trips at three quarters of that, estimating tokens as characters over four.
3. **Needs current facts.** Use the cloud with search grounding.
4. **No on-device model.** Use the cloud. Nano needs a supported machine and a 2GB download.
5. **Save-Data is on.** Run on-device and don't escalate. The user asked you not to spend their bandwidth.
6. **Otherwise.** Try on-device first.
7. **The on-device answer didn't hold up.** Escalate.
8. **The cloud stalled.** Hand the question back to Nano.

Six of the eight are cheap boolean checks. Rules 3 and 7 each cost a whole inference, and they're the two worth explaining, so they get a section each below.

Rule 3 runs before rule 4 on purpose. The freshness check also decides between the two cloud tiers, so it matters just as much on a machine with no Nano at all.

Rule 5 leans on the least known signal of the lot. Save-Data is a preference the user sets, surfaced to JavaScript as `navigator.connection.saveData` and to servers as a `Save-Data: on` request header. Switching it on is someone telling you they're paying for their bytes, on a metered connection or a tight tariff, and would rather you sent fewer of them.

That makes it a clean fit for the on-device tier, since an answer that never leaves the machine costs no bytes at all. It's also why rule 5 blocks escalation. Quietly upgrading to the cloud because the local answer looked weak spends the bandwidth the user just asked you to save.

Testing it is fiddly these days. Chrome's Lite mode, the main user-facing way to switch it on, was removed in Chrome 100 back in 2022. DevTools can emulate the header from the Network conditions panel, though that covers the header a server would see, and I haven't confirmed it flips the JavaScript property the router actually reads.

Nano spends more of its time routing than answering. It writes the prose, but it also classifies the question before the request goes out and judges the answer once it comes back. Both of those jobs belong to the router.

# The question the classifier asks

Rule 3 needs to fire before the request, which means classifying the question rather than judging the answer. The phrasing matters more than the model does.

Asking a model whether it needs newer information scores 4/6, and it misses the World Cup specifically. The model believes the last one was 2022, so from where it sits no newer information is required. The question asks it to notice a gap in its own knowledge, which it has no way to do.

Asking whether the answer *could have changed* scores 6/6, and 12/12 on a harder set with adversarial cases. The winning framing asks about the world instead of about the model.

```ts
const { object } = await generateObject({
  model: session('classify'),
  schema: z.object({ needsCurrentInformation: z.boolean() }),
  prompt: `Could the correct answer to this question have changed in the last two
    years? Answer true for anything about current events, latest releases, sports
    results, prices, or who currently holds a position. Answer false if the question
    is pinned to a specific past date or year, because settled history does not
    change. Question: "${prompt}"`,
});
```

Nano runs this when it's available, and the cloud runs the identical question when it isn't. Same schema, same prompt, different model, which is the swap this whole demo is about, applied to the router itself.

The clause about pinned dates is there for Nano. Flash-lite already reads "who was the prime minister in 1983" as settled history and answers from memory. Nano matches on "prime minister" and reads the question as being about who holds the job now, so it goes to the web. That costs a search and about 1200ms to answer something it already knew. Both models get the sentence, because the prompt has to work for the weaker one.

# The question the judge asks

Rule 7 fires when an on-device answer doesn't hold up, which needs a definition of "doesn't hold up" that doesn't require knowing the truth.

Confidence scores from a 3B model are noise, and sampling repeatedly just returns the same stale fact. Comparing both tiers and escalating on disagreement falls over when they agree and are both wrong.

What works is a reading-comprehension task:

```ts
const { object } = await generateObject({
  model: session('judge'),
  schema: z.object({
    answeredTheQuestion: z.boolean(),
    admitsNotKnowing: z.boolean(),
  }),
  prompt: `Question: "${question}"\n\nAnswer: "${answer}"\n\nDoes the answer give
    specific information that addresses the question? Set admitsNotKnowing to true
    if it declines, says it lacks information, or is too vague to be useful.`,
});
```

"Does this text answer that question, or dodge it" is well within a small model's range. Across six test cases it caught the explicit refusal, the vague waffle, and the technically-on-topic-but-useless answer.

It also scores "Argentina won in 2022" as a good answer, because by every measure it applies, it is one: specific, fluent and unhesitating. It's also false. Nothing post-hoc catches that, which is why rule 3 exists as well. It catches stale knowledge before the request goes out, where rule 7 only catches hedging after.

# What each tier costs

Measured on a machine that runs all three.

| phase | on-device | cloud | cloud + search |
| --- | --- | --- | --- |
| route | 530-940ms | 530-940ms | 530-940ms |
| first token | 90-113ms | 480-575ms | 1076-1563ms |
| answer | 1155-2702ms | 35-184ms | 115-560ms |
| check | ~1500ms | not run | not run |

On-device reaches first token roughly fifteen times faster than the cloud, then generates roughly fifteen times slower. Both halves matter. Local wins anything short and interactive, and loses anything long.

Those numbers set the timeouts. The cloud tier aborts if no token arrives in 1500ms; the grounded tier gets 4000ms because it waits on a web search first. Set a budget below your healthy time-to-first-token and it fires on every request, which tells you nothing.

Reasoning effort moves that cloud column more than anything else. Gemini 3 controls it with `thinkingLevel`, which takes `minimal`, `low`, `medium` or `high`, and defaults to `medium`. There is no zero, and the `thinkingBudget` parameter from Gemini 2.5 is rejected outright.

On 3.6-flash the default reaches first token in 3259ms, against 942ms at `minimal`. That gap swamps the network latency the demo exists to show, so the cloud tier runs `gemini-3.5-flash-lite`, which defaults to `minimal` and lands at 481-575ms.

# Speculative local execution

The freshness classifier is a whole inference, and blocking on it costs about 570ms before a single token appears. On-device work is free, so the answer and the classifier start together:

```ts
if (canRunLocally) speculative = localStream(prompt);

if (await needsFreshFacts(prompt, canRunLocally)) {
  await speculative?.return(undefined);   // wrong guess, bin it
  route = { tier: 'grounded', rule: 'fresh-facts', ... };
}
```

Throwing away a local answer costs nothing, whereas the 570ms it saves is 570ms of an audience watching a blank box.

# Behaviours the docs don't mention

Each of these three shaped the code above.

**Node buffers headers until the first write**, which means `fetch()` doesn't resolve until the model has spoken, and a slow model becomes indistinguishable from a slow network. One line separates them:

```js
res.flushHeaders();
```

Headers go from blocking to 2-25ms. Any timeout that's meant to detect a stalled network needs this, or it's measuring the model instead.

Flushing early costs you the status code, though. It goes out before you know whether the model will fail, so a 200 carrying an error as body text reads to the client as an answer, and an expired key looks exactly like a successful reply. Failures ride back on the same NUL-delimited trailer as the source count, and the client rethrows them:

```js
} catch (error) {
  res.write(`${TRAILER}${JSON.stringify({ error: error.message })}`);
}
```

**The AI SDK provider caches one session per model instance** and never re-reads its options:

```js
async getSession(options) {
  if (this.session) {
    return this.session;   // options ignored from here on
  }
  ...
}
```

Share one instance between a structured call and a freeform one and the JSON `responseConstraint` from the first leaks into the second, so asking a question returns the classifier's schema instead of prose. A system prompt added later is silently dropped for the same reason. One instance per role fixes it, since each role always passes the same options.

**Chrome unloads Nano when nothing holds a session open.** The next request then pays a full model reload, which shows up as a routing spike of 7 to 9 seconds on the first question after a pause. Keeping one warm session per role flattens it and takes on-device time-to-first-token from 279-425ms down to 90-113ms.

A reused session accumulates history against that 9,216 token window, so each one gets recycled at 60% usage.

# The failure states

In a room full of laptops, most machines land in one of these, so they get the same attention as the happy path.

The status strip reads in plain words rather than the API's vocabulary: `ready`, `not downloaded`, `downloading` with a live percentage, `won't run here`, `no browser support`. The last two explain themselves. A visitor whose machine can't run Nano should be told it needs 22GB of free disk and either 4GB of VRAM or 16GB of RAM, so they don't assume the page is broken.

An escalation path that falls back to a model that isn't there produces an empty stream and logs `first token 0ms, total 5ms`, which reads as success. So the fallback checks availability before promising anything.

A stall and a failure also carry different messages. An expired key and a slow network both end up on-device, but you fix them differently.

# Try breaking it

Start with offline mode. Open DevTools in Chrome, go to the Network panel, and pick `Offline` from the throttling dropdown. Ask something and rule 1 fires immediately, with the status strip flipping to offline as you watch, because DevTools drives `navigator.onLine` and the `offline` event for real.

Then try a bad connection instead of no connection at all. In that same dropdown, add a custom profile with roughly 2000ms of latency. Now the cloud request starts, hangs, and gets abandoned partway through while Nano picks up the answer. The fetch genuinely gets aborted, so you're watching the real thing rather than a staged one.

If you want to go further and route on connection quality, you'll hit a wall. `navigator.connection` is read-only. It'll tell you `effectiveType`, `rtt`, `downlink` and `saveData`, and fire a `change` event when any of them move, but nothing in your code can set them. DevTools throttling only partly drives `effectiveType`, and the only way to force it is the `force-effective-connection-type` flag in `chrome://flags`, which needs a browser restart. That's why the router sticks to `navigator.onLine`, a measured timeout, and `saveData`.

# What it adds up to

118 lines of routing logic, five cheap checks and two model-driven ones, in front of three tiers that answer the same question. One is free, one is metered, one is metered and waits on a web search first.

Nano is fast now, 90ms to first token on a warm session, and small, 9,216 tokens covering input and output together. The router exists to use the first number and work around the second.