Module 1: Building and Deploying AI Applications

LLM Foundations

Ng lists what you need to understand about the model itself: how it tokenizes input and generates output, when to reach for a multimodal model, what to put in the context window, and how to reason about cache hits, knowledge cutoff, reasoning effort, sampling parameters, and tool calling. That understanding is what lets you choose the right model or mix of models, and know when to reach for fine-tuning or self-hosting.

The failure mode: you pick the biggest model for everything, pay ten times what you need to, and still get caught out when it cannot see last month's product change because of its knowledge cutoff.

The practical version of this skill is a budget. Every call has a cost in tokens and a cost in seconds, and both scale with what you put in the context window. Prompt caching cuts the cost of a repeated prefix. Reasoning effort trades latency for quality. Sampling temperature trades consistency for variety.

You do not need to memorize the arithmetic. You need to know which lever to pull when a stakeholder says it is too slow, or too expensive.

You have this skill when: you can look at a latency or cost complaint and name the specific lever you would change first.

Grounding Models With Data

An LLM only knows what is in its context. Ng's point in the second letter is that RAG with vector search was an early answer to that problem and the menu has grown since. You now choose between putting data in the prompt and letting the model retrieve it on demand with tools, and you choose the representation that fits your data and your queries: a vector index, a knowledge graph, or a semantic layer over structured records. You also turn messy documents into inputs the model can use, and build pipelines that keep that data clean and fresh.

The failure mode: you embed every document in the company, retrieve the top five chunks, and the model answers confidently from a policy that was replaced eight months ago.

Think about it from the query side first. If the question is what does our refund policy say, a vector index over docs works. If the question is what plan is this customer on and what did they buy, that is a database query, so give the model a tool that runs it rather than a pile of embedded records. Most real applications need both.

You have this skill when: you can look at a question and say which retrieval method answers it, and why the other two would not.

Building Agentic Systems

Ng draws the range clearly. At one end, a workflow: a fixed sequence of LLM calls you designed. At the other, an agent harness where the model decides its own next step in a loop. Your job is to choose the architecture, pick the tools the model can call, decide the memory setup, manage context over long sessions, and know when a task genuinely needs multiple agents. Then you harden it with guardrails and an answer for risks like data exfiltration.

The failure mode: you build a fully autonomous agent for a task that was three predictable steps, then spend a month debugging why it sometimes skips step two.

Anthropic's guidance on this is blunt and Ng agrees with it. Start with the simplest thing that works. If you can write the flow as code with LLM calls inside it, write it as code. Reach for an agent loop only when the path through the task cannot be known in advance.

You have this skill when: you can defend, for any AI feature, why it is a workflow or why it needs an agent, in one sentence each.

Evaluation-Driven Development: Building the Eval Set

Ng is explicit that this is the skill that separates the best builders. He calls driving a disciplined evals and error analysis loop the most important trait of someone great at building AI systems, and says it is hard to master because the right approach changes by project and by stage.

The failure mode is the one nearly every team hits: you change a prompt, it feels better on the three examples you tried, you ship it, and the thing you did not test gets worse.

An eval is a set of inputs with a way to score the outputs. That is all it is. The skill is in choosing what to measure. Ng's method is to look at real traces and outputs, do exploratory analysis on them, and combine that with product judgment to decide what matters. For Triage, did it pick the right product area matters more than was the reply eloquent.

You have this skill when: you refuse to change a prompt without a number to compare before and after.

Evaluation-Driven Development: Error Analysis and Choosing Your Judge

Once you have a score, the second half of the skill is knowing what kind of eval to run and how to read what fails. Ng lists three options: deterministic code-based checks, an LLM as judge, and a human in the loop. He also says you need to evaluate your evals, because a judge that scores the wrong thing steers the whole project the wrong way.

Use code when the answer is checkable. Did the classifier output one of the five allowed labels. Did the reply include the ticket number. Use an LLM judge when the answer is a quality call, and spot-check that judge against a human on a sample. Use humans for the cases where being wrong is expensive.

Error analysis is the part most teams skip. Take the failures, sort them into buckets by cause, and count. That is what makes progress systematic rather than random. If 60 percent of Triage's misses are billing tickets tagged as account tickets, you do not need a better model. You need three examples of billing tickets in the prompt.

You have this skill when: your first move after a bad score is to bucket the failures, not to swap the model.

Operating in Production

Ng names three things that make operating AI software different from operating normal software: unpredictability, cost, and latency. So you need observability that shows how the system performs on real usage, drift detection, and a fast response to model failures and security incidents like prompt injection. Regression testing and CI need statistical evals rather than pass or fail tests, calibrated to how bad a mistake would be.

The failure mode: the model provider ships a silent update, your classification accuracy drops 15 points, and you find out from a customer three weeks later.

Observability for AI means logging the full trace of every call: the prompt, the retrieved context, the output, the tool calls, latency, and tokens. Without that you cannot do error analysis on production data, which means the eval loop from 1.4 stops working the day you launch.

You have this skill when: you can answer how is it doing in production with a number from the last 24 hours, not a feeling.

Machine Learning Foundations

Ng says every engineer he knows who is good at building with LLMs also understands machine learning and deep learning at some depth, and that many applications still need a classic trained model, whether you train it or someone else did. The concepts he calls out as still essential are bias and variance, error analysis, and engineering your data.

The failure mode: you use an LLM at two cents a call to do something a 5MB classifier does in a millisecond, and you cannot explain why its accuracy moves around from day to day.

You do not need to derive backpropagation. You need three ideas. Bias means your model is too simple to capture the pattern, so more data will not help and a better model will. Variance means it memorized the training data, so more data or a simpler model will help. And the data you feed in sets the ceiling for everything downstream. These three are exactly why the error analysis in 1.5 works, because you are diagnosing which problem you have before you pick a fix.

You have this skill when: you can look at an AI failure and say whether it is a data problem, a model capacity problem, or an overfitting problem.