Why local AI does not always need the largest LLM: model routing explained

Choosing a local language model often starts with size: parameters, memory and leaderboard position. Yet a working day contains different tasks. Finding a contract clause, transcribing a meeting and preparing a complex explanation are different computing problems.

A more useful question is which tool can handle each stage adequately, and how the system knows when to stop or choose another route. A model router supplies selection rules that connect quality, latency, cost and data restrictions. This architecture does not assume that a small model can always replace a large one.

“Review the meeting recording and prepare assignments in light of the contract” involves transcription, document retrieval, reading relevant conditions, matching participants and drafting tasks.

A scanned contract needs text recognition. A document with a usable text layer may be better served by a parser, avoiding unnecessary OCR errors. Totals should be calculated from verified numbers by dependable software rather than guessed by a language model.

Employees do not need every component's name. They need sources, outcomes and limitations. The system designer does need to know where facts enter the process and how errors can pass between stages.

Select tools for individual stages, then evaluate the route by its final accepted result.

A router chooses an execution path according to task type, required quality, available resources and data policy. It may use explicit rules, a classifier or a combination. Another large language model is not an essential requirement.

For example, a file containing text goes to a parser; audio goes to speech recognition; a question about an instruction goes through authorised knowledge retrieval and then a language model. A total goes through a calculation routine.

The route also needs authority boundaries. A request to prepare a reply does not authorise sending it. Switching models must not widen document access or silently transmit information to an external service.

OCR converts images of text into machine-readable text and may recover page structure. ASR transcribes speech. Embeddings represent passages numerically for similarity-based retrieval. A reranker re-evaluates retrieved candidates against a query. A language model generates and explains an answer from the supplied context.

Qwen3 Embedding and Qwen3 Reranker are separate models for text representation and ranking. They illustrate why search components and conversational generators have different roles. The official description was published on 5 June 2025; it is used here as an architectural example, not a claim about current leaderboard leadership. Qwen · Qwen3 Embedding and Reranking ↗

The separate official Qwen3-ASR project addresses speech recognition. Its existence does not mean every AI Office configuration includes it or guarantees quality on your recordings. Model choice, language, noise, terminology and hardware require individual validation. QwenLM · Qwen3-ASR official repository ↗

Components need not come from one vendor. Compatible data formats, licensing, versions and maintenance matter more than a long list of models nobody can operate or diagnose reliably.

A narrow task with a clear input and verifiable output can justify testing a compact candidate: classifying an inquiry, extracting a few fields or drafting a short template-based response. That is a testable hypothesis, not a property shared by every small model.

Use company terminology, abbreviations, product names and unusual documents. A model can answer general questions well while confusing almost identical catalogue entries. Public benchmark averages cannot replace examples from the actual workflow.

Judge the accepted result. Faster generation can still make the process slower if it creates more corrections. Once a compact candidate passes the quality threshold, a larger one should justify additional resources through practical value.

Measure concurrent use too. A quick isolated response does not describe behaviour while the system indexes an archive and transcribes another meeting.

Ambiguous questions, difficult multi-source comparisons and writing under many constraints can benefit from a stronger candidate. Size alone does not establish superiority for the task. Compare candidates on the same examples and evaluation rules.

Sometimes retrieval is the problem. If the system missed the current contract, a larger model may explain an outdated one more convincingly. If OCR misread a digit, additional reasoning cannot guarantee recovery of the original information.

Locate the quality loss before raising the computing budget. Inspect extraction, source freshness, context coverage, instructions and evaluation criteria. A larger model is one possible remedy, rather than an automatic first step.

Consider a hypothetical proposal workflow. The system extracts requested products, searches the catalogue and validates mandatory fields. An ordinary case produces a draft. Multiple incompatible matches or a missing unit of measure produces a clarification request.

A stronger local model might help interpret a difficult phrase. It must not invent missing commercial terms. Unresolved ambiguity remains a reason to stop even when an expensive external service is available.

Escalation rules should use observable signals: a missing field, conflicting sources, failed format validation or a task outside the agreed class. A model saying “95% confident” is not itself a calibrated probability of correctness.

Limit retries and total expenditure. Repeatedly trying other models can increase cost without adding evidence. Each route needs a defined endpoint: a result, clarification, human review or refusal.

External models can help with authorised tasks in a local-first architecture. They should not become an automatic fallback for every local failure. If documents must stay inside the company, a local outage does not cancel that requirement.

Specify eligible data, destination, permission and spending limits before transmission. An anonymised question or public text may be suitable in some cases; a short summary can still reveal sensitive commercial information in others. Removing names does not guarantee that the remaining information is harmless to disclose.

The audit trail should identify the route used. Users need not study model names, but the company should be able to inspect external transfers. If cloud use is prohibited, the system should explain the constraint and offer an allowed next step.

Specialisation distributes work without creating free resources. Several models can occupy memory simultaneously, and loading or unloading introduces delay. Queues, concurrency limits and background schedules become part of the design.

Capacity planning includes weights, working memory, context, indexes, OCR, system services and expected traffic. Validate actual memory demand and speed on the chosen software stack. A memory figure alone does not prove that a department's workload is supported.

Separate urgent user work from background jobs where useful. Archive indexing need not compete equally with a manager answering a customer. Base scheduling on the company's working pattern rather than assuming nights are always idle.

Compare a simple baseline with the proposed route on the same tasks. Include normal cases, difficult documents, missing information and requests that should be rejected.

Measure accepted-result rate, end-to-end time including review, cost per task, latency under load and human handoff frequency. Check separately that restricted data stays off prohibited cloud routes and that permissions survive every stage.

Error analysis should identify the failing stage. Poor retrieval, transcription mistakes and weak explanations require different fixes. “The AI failed” does not tell you whether to change a model or repair the data.

Recheck affected workflows when a component changes. A new embedding model may require index rebuilding; a changed output format may affect the next stage. Version records and a route back to a validated configuration matter alongside new capabilities.

For AI Office, routing is an architectural way to match tools to business work. Specific local language models, OCR, ASR, embeddings and permitted cloud connections are defined and tested within the project. Naming a component is not a promise that it is already installed or performs identically on every hardware setup.

Start with one route: a document or recording enters the system, passes through the necessary stages and produces a verifiable draft with sources. The pilot reveals where simple processing is enough, where a stronger tool helps and where a person must decide.

The company is buying a useful workflow. Model size matters insofar as it helps that workflow meet quality, control and cost requirements.