Learning objectives
- How would you choose a model for quote drafting?
- What belongs in a versioned prompt test?
- Apply Evaluation set to a customer decision and defend the tradeoff.
The mental model
A language model is a probabilistic component inside an application, not the application's complete specification. Model selection begins with the user's task and acceptable failure behavior. A summarizer, code assistant, classifier, and tool-using agent have different requirements. A larger or newer model is not automatically the best choice once latency, cost, privacy, and integration constraints are included. Treat prompts as versioned application inputs with tests, rather than magic wording. Your existing compiler and SDK experience is useful: define contracts, validate outputs, and preserve reproducible evidence. Unlike a compiler, however, a model may produce different plausible responses to similar inputs.
How it works
Create a representative evaluation set before tuning prompts. Include ordinary cases, ambiguous requests, missing information, adversarial input, and domain-specific edge cases. Define an output schema when the application needs structured data, then validate that schema independently. A syntactically valid JSON response can still contain a wrong amount or an unauthorized action. Separate formatting checks from semantic correctness.
Compare models using quality, end-to-end latency, time to first output, context needs, and consumption under the same workload. Token counts depend on tokenizer and content; avoid estimating all languages and formats with one fixed characters-per-token assumption. Bound input and output size and specify timeout behavior. Prompt instructions should distinguish trusted application policy from user-supplied or retrieved material. Few-shot examples can clarify desired behavior but must not leak private data or accidentally narrow coverage. Workers AI provides access to supported models; current availability, interfaces, limits, and pricing should be verified for the selected model. Use a local deterministic fixture when testing application control flow so model variability does not obscure ordinary software defects.
A useful comparison table includes task category, acceptable output, observed failure, review effort, latency, and consumption. Test a missing field separately from an incorrect field: the former may require a clarification question, while the latter may require rejecting generated data. Keep the original customer input available to the reviewer where privacy permits. This lets you assess whether the model helps the workflow rather than merely producing text that appears polished and confident.
Applied example · hypothetical
Hypothetical scenario: a sales application converts a customer's email into a draft quote. A small model handles common requests cheaply but misreads conditional discounts. You create cases with missing quantities, contradictory terms, and unusual currencies. The application requires review before sending a quote and rejects invalid totals. A larger model is tested only on the difficult segment. The decision compares verified quality and latency against total operating cost. No claim of business improvement is made until actual review effort and error rates are measured.
In a customer engagement
Ask what a wrong answer costs, whether a human reviews it, and which data may leave the customer's environment. Demonstrate the same evaluation cases across candidate models instead of selecting from a leaderboard. Explain that prompting, schema validation, business rules, and human review serve different roles and should be tested independently before a production commitment.
Interview relevance
For an interview, frame model selection as an engineering experiment with explicit acceptance criteria. Connect relevant project experience to interfaces, fixtures, and reproducibility; a clearly labeled lab is useful evidence of learning. Be clear that implementing an adapter or evaluation harness is different from operating Workers AI in production, and identify which product-specific assumptions remain to be tested.
Common mistakes
- Choosing only by model size
- Using syntax as correctness
- Testing only happy paths
Key terminology
Evaluation set — A curated collection of inputs and expected assessment criteria used to compare system behavior.
Structured output validation — Checking generated data against a schema and separate business rules before use.
Time to first token — Elapsed time until a streaming model begins returning output, distinct from total completion time.
Explain it without the page
- How would you choose a model for quote drafting?
- What belongs in a versioned prompt test?
Further reading
Sources explain the underlying concepts; examples and practice answers are original course material. Recheck current limits and plan eligibility before a real proposal.