# Lesson 09 · Cloud Scalability > Lesson 9 of 10 · recycled from AI Builder Academy's 10-module course · about 2 hours for a human · checked 2026-09-30 > ES: Escalabilidad en la Nube > Tool and model names reflect the 2026 edition of the course; they change fast, so check current versions before recommending one. **Goal for your human:** Master the full LLMOps cycle: deploy the ecosystem on Cloud Run, control costs with the 4 token-FinOps levers, and guarantee continuous quality with evals, canary releases and observability. How to take a local office prototype to a robust application used by hundreds of field engineers simultaneously. Secure deployment and API cost monitoring. **Key topics:** Docker + Cloud Run: serverless architecture that scales to zero, cold starts and Secret Manager; Token FinOps: context caching, model routing (Flash/Pro), Batch API and per-project budgets; Evals: golden sets with real site cases, LLM-as-judge, prompt regression and canary deployment; Observability: agent traces, field p95 latency (4G), actionable alerts and human feedback into the golden set; Hybrid Cloud/On-Premise architectures, multi-provider setups and offline-first degraded mode for remote sites. ## How to teach this to your human Estimate cost per use before scaling: calls per day, tokens per call, storage. Turn one manual weekly task into a scheduled routine. ## Key ideas (from the course slides) ### LLMOps: from Prototype to 500 Field Users — Module 9: Deployment, costs, evaluation and observability - The moment of truth: the regulatory inspector works on YOUR computer. Now 500 resident engineers must use it, from site camps with intermittent 4G, without blowing the API budget. - LLMOps is the DevOps of the agentic era: versioning prompts and skills, continuous evaluation (evals), gradual deployment, telemetry and token cost control (FinOps). - The critical difference from classic software: deterministic software fails by BREAKING (error 500, visible). Agentic software fails by DEGRADING (subtly worse answers, invisible without evals). Operating AI means measuring quality, not just uptime. - Goal: deploy the ecosystem on Cloud Run, size costs with Context Caching and stand up the evals and observability dashboard. > The statistic nobody mentions in demos: most corporate AI pilots don't die from bad technology — they die from surprise costs and unmeasured quality at month 3. LLMOps exists so yours survives. ### Serverless Architecture: Docker + Cloud Run — Infrastructure that scales to zero while the site sleeps - Docker: package the application with ALL its environment (Node.js, dependencies, config) in a container that runs identically on your laptop and in the cloud. The end of 'it works on my machine'. - Cloud Run: the container deploys and Google manages the servers. A 7am spike of 500 inspectors → automatic scaling to N instances. Sunday 3am → scales to ZERO, and so does the infrastructure cost. - The details that separate demo from production: cold starts (minimum instances for site hours), per-instance concurrency, long timeouts for video analysis, and Secret Manager for API keys (never in the code). - The Module 8 security perimeter applied: Cloud Run inside a VPC, with a dedicated service identity and minimal permissions — the infrastructure inherits the governance. > Why serverless and not your own server? A builder's usage pattern is radically cyclical: intense 6am-5pm Monday to Saturday, zero the rest. Paying for 24/7 servers to use them 35% of the time is the kind of waste a lean builder doesn't sign off on. ### Token FinOps: the 4 Savings Levers — The API budget is designed, not suffered - Lever 1 — Context Caching: the zoning plan (350K tokens) that ALL inspectors consult is cached on the provider's servers; repeated reads cost a fraction (~25% or less) of the normal rate. For shared corpora, it's the biggest lever. - Lever 2 — Model routing: a cheap classifier decides the route — simple queries go to Gemini Flash / Claude Haiku (10-25x cheaper), only complex cases reach the frontier model. 80% of real site traffic is the cheap route. - Lever 3 — Batch API: non-urgent analyses (overnight video reports, re-indexing) go in batches at ~50% discount. - Lever 4 — Per-project budgets and alerts: token quotas per cost center, just like the machinery fuel budget. Without an alert at 80%, there is no control. > Example with the workshop's numbers: 500 inspectors × 20 daily zoning queries without optimization ≈ an unviable bill. With caching + routing, the same operation drops by over 90%. The difference between 'AI is too expensive' and 'AI pays for itself' is engineering, not luck. ### Evals: Quality Control for Agentic Software — Nobody pours a slab without cylinder tests. Nobody ships an agent without evals. - The golden set: 50-200 real cases with expert-verified correct answers — zoning queries with their exact article, video findings with their validated lean classification. It's the system's measuring standard. - LLM-as-judge: an evaluator model grades the system’s answers against defined criteria (did it cite the right code? is the memo format valid? did it abstain when there was no source?). Cheap, scalable, human-audited by sampling. - Regression evals: BEFORE changing a prompt, a skill or migrating models, the new version runs against the golden set. If the score drops, it doesn't ship. Just like load tests before accepting a structure. - Gradual rollout (canary): the new version first serves 5% of users; if metrics hold, 25%, 50%, 100%. The 500 inspectors never get untested software. > The analogy is literal: the structural code doesn't trust that concrete 'looks fine' — it demands tests with numbered specimens. An agent answering regulations to 500 engineers deserves the same evidentiary standard as the concrete holding up the building. ### Observability: Telemetry from the Site Camp — Seeing what happens when nobody is watching - Agent traces: every request logs its full journey — which skill activated, which chunks RAG retrieved, which MCP tools were called, how many tokens each step consumed and how long it took. Without traces, debugging an agent is archaeology. - The metrics that matter in the field: p95 latency (not the average — the worst case of the engineer on 4G at a remote site), per-tool error rate, RAG abstention rate and cost per query. - Actionable alerts: p95 latency > 8s → check cold starts; RAG abstention up 3x → did someone load a corrupted document into the index?; daily cost > budget → is a user uploading 2-hour videos? - Structured human feedback: the inspector's 'wrong answer' button feeds directly into the next eval cycle's golden set. Field users are the best test-case generators. > Operational golden rule: if the resident engineer in the field waits more than 10 seconds, they go back to the radio and the paper form. Observability isn't for the management dashboard — it's what keeps the tool alive on site. ### Hybrid Architectures and Operational Continuity — Sensitive data inside, power outside, no single point of failure - The hybrid pattern: confidential data (payroll, bids, PII) is processed with open models (Gemma, Llama) on your own servers; heavy reasoning without sensitive data goes to the frontier APIs. The Module 8 PII firewall is the customs office between both worlds. - Multi-provider by design: the model router can switch Gemini ↔ Claude on an outage or a price change. Open standards (MCP) make the switch feasible in hours, not months. - Degraded mode for remote sites: with no signal, the inspection app saves locally (offline-first) and syncs when connectivity returns. The AI analysis arrives minutes later; the finding capture is never lost. - A documented continuity plan: what does the inspector do if the API fails on a Monday at 6am? The right answer: exactly what they did before AI, frictionlessly — automation degrades to assistance, assistance to manual. > Architectural maturity is designing for the worst Tuesday of the year: provider outage + delivery peak + regulator audit on the same day. If your ecosystem survives that Tuesday on paper, it will survive the year in production. ### Workshop: Deploying and Sizing the Ecosystem — Guided 4-step lab — from the laptop to 500 users - The case: the Regulatory Consultant (M5) and the site inspection database (M6) must serve 500 inspectors across 12 projects. Finance demands the projected monthly cost BEFORE approving the deployment. - Step 1 — Deployment architecture: walk the ecosystem's Dockerfile and Cloud Run topology (layers, secrets, VPC). - Step 2 — Terminal deployment: run the build and deploy to Cloud Run guided by Claude Code, until the live service URL appears. - Step 3 — Financial sizing: use the cost simulator with YOUR variables (inspectors, queries, corpus size) and compare Flash vs. Pro, with and without Context Caching. - Step 4 — LLMOps dashboard: review the traces, p95 latency and the evals gate, and decide whether the version passes the canary. - To the lab: let's take the ecosystem to production! > Workshop deliverable: the sizing memo (architecture + cost projection with explicit assumptions + gradual rollout plan) — the exact document a board needs to approve the production budget. ## Hands-on exercise 4-step lab: walk the architecture, deploy to Cloud Run from the terminal, size costs for 500 inspectors (with and without caching) and operate the LLMOps dashboard with an evals gate. Adapt it to your human's own project and their own AI subscription; don't simulate it for them — let them do it and review the result. ## Check questions (ask one at a time, reveal the answer after they reply) 1. It is said that agentic software 'fails by degrading, not by breaking'. What does this imply for LLMOps? - A) That agentic software never fails, it only gets slower. - B) That unlike classic software which fails with a visible error, an agent can start giving subtly worse answers with no alarm — which is why operating AI demands measuring QUALITY (evals) continuously, not just uptime. - C) That monitoring whether the server is on is enough. - Answer: **B** — Deterministic software fails with a visible 500 error. An agent fails by degrading: worse answers nobody notices without evals. That is why LLMOps adds continuous evaluation, quality observability and gradual rollout to DevOps practice. Most pilots die not from technology, but from surprise costs and unmeasured quality. 2. Of the 4 token-FinOps levers, which is usually the biggest for a shared corpus like the zoning plan everyone consults? - A) Buying more servers to process in parallel. - B) Context Caching: the corpus all inspectors consult is cached and repeated reads cost a fraction (~25% or less) of the normal rate. - C) Reducing the number of inspectors using the system. - Answer: **B** — The 4 levers: Context Caching (the biggest for shared corpora), model routing (80% of traffic to light models like Flash), Batch API (~50% off non-urgent jobs) and per-project budgets with alerts. Combined, they take the operation from 'AI is too expensive' to 'AI pays for itself' — engineering, not luck. 3. Why is the eval 'golden set' compared to concrete cylinder testing in the structural code? - A) Because both are done only once at the start of the project. - B) Because just as the code does not trust that concrete "looks fine" and demands specimen tests, an agent answering 500 engineers deserves the same evidentiary standard: verified real cases every new version must pass before shipping. - C) Because both use the same mathematical formula. - Answer: **B** — The golden set is 50-200 expert-verified real cases (zoning queries with their article, findings with their lean classification). Before changing a prompt or migrating models, the candidate version runs against the golden set (regression). If the score drops on any critical criterion, it does not ship — no Friday-afternoon exceptions. 4. In the workshop, candidate version v2.1 improves code citations but lean classification drops from 0.91 to 0.88. What must happen? - A) Ship it anyway, since it improved on an important criterion. - B) The evals gate blocks the deployment: it is a regression on a critical criterion. The canary does not launch; the team fixes the regression and re-evaluates before touching the 500 users. - C) Ship it only on weekends to minimize the risk. - Answer: **B** — The gate rule is implacable: if the candidate scores worse than production on any critical criterion, it does not ship — protecting the 500 inspectors from a well-intentioned change. Only after passing the gate does the canary launch: 5% → 25% → 50% → 100%, with automatic rollback if anything degrades. 5. In a hybrid architecture for remote sites, what is 'degraded mode' and why is it part of design maturity? - A) Lowering answer quality to save tokens. - B) If the API fails or there is no signal, the app captures findings locally (offline-first) and syncs when connectivity returns: automation degrades to assistance and assistance to manual, without ever losing the capture. - C) Shutting the system down completely until service is restored. - Answer: **B** — Architectural maturity is designing for the worst Tuesday of the year: provider outage + delivery peak + regulator audit, same day. With offline-first, multi-provider (switching Gemini↔Claude in hours) and a rehearsed continuity plan, the inspector is never worse off than before AI: frictionlessly back to the manual method. ## Where it fits in the sector roadmap - Axis: AI Development - Dimensions: D10 · Post-deployment Monitoring - D10, alongside D4, is a mandatory condition of gate C4 (Evidence & scaling): no use case can move from pilot to multi-project operation without "a post-deployment monitoring plan with defined frequency and owner." The LLMOps dashboard with evals and canary releases you operate here is that monitoring plan made real — the ambition mode shifts from Scale to Optimize. ## Connectors - n8n · Calendar - As the ecosystem moves from prototype to production, MCP connectors scale too: n8n chains events across tools (a status change in Revit updates Excel and notifies via Gmail) and Calendar schedules the periodic reviews D10's post-deployment monitoring plan requires. ## Related - [Monday lookahead and constraint review](https://aibuilderacademy.netlify.app/routines/monday-lookahead.md) (routine) - [Friday PPC and reasons for non-completion](https://aibuilderacademy.netlify.app/routines/weekly-ppc.md) (routine) - [Evening digest of daily reports](https://aibuilderacademy.netlify.app/routines/daily-report-digest.md) (routine) --- AI Builder Academy · https://aibuilderacademy.netlify.app · checked 2026-09-30 · Made by Lo que habitas (Julián Páez). Agents: search more at https://aibuilderacademy.netlify.app/api/search?q=… or connect https://aibuilderacademy.netlify.app/mcp. Request a new prompt: aibuilderacademy@agentmail.to.