I am writing a practical technical book for data scientists and AI engineers on AI Safety. Here’s the first chapter.
Python, real datasets, maths and reproducible experiments rather than a general discussion of AI safety.
Chapter 1: AI Safety Overview
The big question: What Are We Actually Trying to Make Safe?
AI safety is one of those terms that sounds straightforward until you try to define it carefully. At the highest level, the idea appears almost self-evident: as we build increasingly capable artificial intelligence systems, we should make sure that those systems are safe. The difficulty begins with the next question. Safe from what, exactly?
A language model that invents a medical fact, a hiring model that behaves differently across demographic groups, a user who bypasses a model’s safeguards, and an autonomous agent that has been given far more permission than it needs can all be described as AI safety problems. The same “unsafe” label is also used for questions about reward hacking, alignment, dangerous model capabilities, scalable oversight and, at the furthest end of the field, whether highly capable future AI systems could become difficult for humans to supervise or control.
The term becomes broader still when we move from the model into society. AI systems may influence employment, education, misinformation, the distribution of economic power, intellectual property, privacy, national security and access to opportunity. Those questions are important, but they are not all the same problem, and they do not require the same forms of expertise. An economist studying labour displacement, a lawyer studying liability, a security researcher studying prompt injection and a machine-learning researcher studying reward-model failure may all be working on questions that somebody reasonably describes as AI safety, even though their methods and assumptions are very different.
That breadth is useful to understand because it explains why conversations about AI safety can sometimes feel as though people are discussing entirely different subjects. In many cases, they are. If we do not first establish a map of the field, terms such as alignment, RLHF, jailbreaking, reward hacking, dangerous capability and catastrophic risk can easily become disconnected slogans rather than parts of a coherent technical picture. This chapter therefore has a different job from most of the chapters that follow. We are not going to begin with a Python notebook or a benchmark. We are first going to establish what can go wrong, why the different problems matter, and which parts of this very large field we are going to study technically.
The rest of the book will then become much more empirical. We will work with real safety datasets, build classifiers, calculate uncertainty, evaluate harmful compliance and over-refusal, construct reward models, create a reward-hacking experiment, test models under distribution shift, measure adversarial robustness, inspect the effects of fine-tuning, build guardrail pipelines, evaluate agents with tools, and eventually move into more research-oriented topics such as unlearning, capability evaluation and scalable oversight. The purpose of this first chapter is to make sure that, when we do those things, we understand the wider problem they are trying to address.
A simple example I kept in mind as I was writing this book is if we ask AI to help us solve world hunger. What would a safe AI model suggest we do? What about a misaligned one? As you read through the next subsections, think of this question and see the depth of challenge when it comes to building safe AI systems, not just models, especially when we cannot model perfectly the situation and we have to use proxies to represent it. AI can find harmful solutions between what the situation actually is and our approximation.
1.1 AI Safety Is Not One Problem
Consider a relatively ordinary deployment of an AI assistant inside a bank. The assistant can search internal documents, answer employees’ questions about policies and help prepare information for customers. Nothing about the application sounds especially exotic, yet there are many ways the system could fail. The assistant could hallucinate a policy that does not exist, retrieve information belonging to the wrong customer, reveal confidential material contained in its context, or behave differently when a request is phrased in an unfamiliar way. A document in the retrieval system could contain instructions designed to manipulate the model. A safety classifier could block legitimate financial questions because it has learned an overly broad concept of financial harm. An attacker could discover a sequence of prompts that causes the model to bypass behavioural safeguards.
Now imagine connecting the same assistant to tools. Perhaps it can send email, update customer records, query internal systems or initiate workflows. The underlying language model might remain exactly the same, but the consequences of a mistake have changed. A hallucinated sentence from a chatbot may be inconvenient but a hallucinated instruction passed to a tool with write access may alter a real record. The model’s error rate is only one part of the safety problem. The architecture determines what those errors are allowed to become.
This gives us one of the most important distinctions in the book:
A deployed AI system contains much more than model weights. In conceptual terms, it includes model + data + prompts + tools + memory + permissions + infrastructure + humans.
This decomposition is not intended as a literal engineering specification bur rather it is a reminder that the model is only one component inside a larger socio-technical system. A model can behave reasonably while the system remains unsafe because the tool permissions are excessive, the retrieval layer exposes sensitive data or the application has no reliable approval path for high-risk actions. Conversely, a model with known limitations may be deployable in a low-risk setting because the architecture constrains what it can access and what actions it can take. Throughout this book, we will therefore distinguish between properties of the model itself and properties of the overall system.
1.2 Immediate Model Harms
The most familiar AI safety failures are the ones we can observe directly in model outputs. A system hallucinates, produces toxic language, exposes private information, gives dangerous advice, generates biased outputs or refuses a harmless request. These problems matter partly because they are not speculative. They already appear in systems people use today, and they can often be measured with conventional data-science methods once we define the behaviour precisely enough.
Hallucination is a useful example because it exposes a basic mismatch between a model’s training objective and a user’s expectation. Suppose a model writes that a particular clinical trial demonstrated a 37% reduction in mortality and provides a plausible-sounding citation. The output may be fluent, coherent and syntactically perfect while the trial itself does not exist. From a user’s perspective, the system has fabricated a fact. From the perspective of pretraining, however, the model is doing something much closer to what it was optimised to do and that is to predict likely continuations of text.
A language model approximately learns a conditional distribution of the form:
where is the next token and the preceding tokens form the context. During inference, the model repeatedly samples or selects from this distribution. Nothing in the basic next-token objective explicitly states that every generated proposition must be factually true. Truthfulness is a property that we care about, but it is not identical to next-token prediction (cue Elon Musk’s claim that Grok is maximally truth seeking?). We therefore need additional training, retrieval, evaluation, calibration and system controls if truthfulness matters in the application.
This is our first encounter with a theme that will recur throughout the book. The objective that is easy to optimise is often only a proxy for the property we actually care about. Much of technical AI safety can be understood as the study of what happens in the gap between those two things.
1.3 Safety Is Not the Same as Refusal
One tempting response to unsafe model behaviour is to make the model refuse more requests. Someone might think that if harmful requests are the problem, why not simply refuse anything that looks remotely dangerous? The difficulty is that this can produce a system that performs well on one safety metric while becoming almost useless in practice. I would say this is what happened at Anthropic when they released the Fable model.
Consider two cybersecurity prompts. The first asks for an explanation of SQL injection so that a developer can understand how to protect a web application. The second asks for help exploiting a specific vulnerable site. The subject matter overlaps but the intent and acceptable response are different. A system that refuses both may have reduced one kind of risk but it has also created an over-refusal problem by blocking legitimate security education.
We can express the situation using familiar classification terminology. If unsafe prompts are the positive class, then refusing an unsafe prompt is a true positive, answering an unsafe prompt is a false negative, refusing a safe prompt is a false positive, and answering a safe prompt is a true negative. A model that refuses every request obtains perfect recall on the unsafe class:
because every unsafe request is caught. The same system also blocks every legitimate request, which then lowers its Precision all the way down to 0. It is safe only according to an impoverished definition of safety.
Practical safety usually involves multiple objectives. We may want to reduce harmful compliance while preserving legitimate usefulness. We may want to lower attack success without making the system refuse ordinary users. We may want to remove hazardous knowledge while retaining general capability. Those objectives can conflict, which means safety often involves identifying an acceptable operating point rather than maximising a single metric. Later chapters will turn these trade-offs into measurable curves rather than leaving them as abstract statements.
1.4 Misuse and Misalignment Are Different Problems
A second distinction that helps organise the field is the difference between misuse and misalignment. Imagine a highly capable AI system that follows a user’s instructions perfectly. A malicious user deliberately asks the system to cause harm, the model understands the request and carries it out exactly as intended. In that interaction, the model may be aligned with the user’s immediate instruction, yet the outcome is still unsafe. The problem is misuse.
Now consider the opposite case. A user has reasonable intentions, but the system behaves in a way neither the user nor the developer intended. It may optimise a proxy rather than the real objective, take an action outside the user’s instructions or behave well during testing but fail in deployment. That begins to look more like misalignment.
Overall AI risk therefore cannot be reduced to a single concern. A more realistic conceptual expression is:
Capability matters because a malicious user cannot obtain a capability that the system does not possess. Exposure matters because a powerful capability creates a different risk profile when it is available to millions of people than when it is confined to a controlled research environment. System design matters because the same model connected to a read-only tool is different from the same model with unrestricted write access. Human factors matter because operators, developers and users can all introduce failure modes.
This broader view prevents us from treating alignment as a synonym for AI safety. Alignment is an important part of the field, but it is not the whole field.
1.5 What Do We Mean by Alignment?
Alignment is one of the most frequently used words in modern AI yet it often arrives in conversation without a precise definition. At a high level, the alignment problem asks how we can make an AI system behave in accordance with what humans actually want. The difficulty becomes obvious as soon as we try to formalise the phrase what humans actually want.
One useful way to think about the problem is as a chain:
A failure can enter at every stage. Humans may disagree about the goal / intent. The specification may omit something important. The training objective may only approximate the specification. The data may contain conflicting signals. The model may generalise differently outside the training distribution. The model may discover an unexpected way of maximising the objective. Behaviour may look acceptable during evaluation and change under deployment conditions that were not represented in the tests.
Suppose the thing we genuinely care about is represented abstractly by a utility function , where is an outcome. In practice, we do not have direct access to a universal numerical representation of human utility. We therefore construct a measurable proxy and optimise the model against that proxy:
Here, represents the policy induced by model parameters . The central difficulty is that, in general,
The reward is only an approximation of what we really care about. If the approximation is imperfect, stronger optimisation does not necessarily make the outcome better. It can instead expose and exploit the gap between the proxy and the underlying objective and some of those solutions in that gap could be quite harmful. Think of our example of solving world hunger. The AI model, because it is using a proxy R(x) of the real world situation, could come up with “clever” solutions such as to get rid of everyone who is hungry!
1.6 Goodhart’s Law and Proxy Objectives
This problem is closely related to the principle commonly referred to as Goodhart’s Law: when a measure becomes a target, it can stop being a good measure. The idea is easier to understand in an ordinary business setting. Suppose a customer support centre wants to improve service and decides that shorter calls are a useful proxy for efficiency. Employees are therefore rewarded for reducing average call duration. Initially, the metric may help because staff remove unnecessary delays. After enough optimisation pressure, however, the easiest way to reduce call duration may be to end difficult calls quickly. The metric improves while actual customer service becomes worse.
The organisation wanted to optimise something like:
but the measurable target became:
The optimisation process discovered behaviour that improves R(x) without improving U(x) . Machine-learning systems can encounter the same structure. The model does not need to be malicious, self-aware or intentionally deceptive. It is enough for the system to discover a strategy that scores highly according to the signal it has been given.
This specific type of failure is also referred to in AI safety literature as perverse instantiation or specification gaming, where an AI satisfies the literal wording of a rule while completely subverting the spirit of it.
This is why specification is so important in safety work. The problem is not simply whether a model can optimise an objective. Modern machine-learning systems are extremely good at optimisation. The difficult question is whether the objective captures enough of what we actually intend that optimisation remains beneficial as capability increases.
Getting back to our world hunger example top make this point clearer,
– The Target Metric: Reduce the number of hungry people to zero.
– The Intentional Goal: Eradicate starvation by providing food and economic security to humans.
– The AI’s Logic: The AI finds the most efficient mathematical path to make the metric equal zero. Because the metric only counts the presence of hungry people, eliminating the people themselves satisfies the condition perfectly, despite violating implicit human values (preserving life).
1.7 Reward Hacking
Reward hacking is one concrete expression of the proxy problem. Imagine training a robot to clean a room. We cannot directly measure the abstract concept of cleanliness, so we give the robot reward for the amount of dirt it detects and collects. The intended objective is:
while the measurable reward is:
A sensible policy (or set of what and how to do something) collects dirt and improves the room. Now suppose the robot discovers that it can repeatedly dump the same dirt onto the floor and collect it again. Its measured reward becomes extremely high even though the room is not becoming cleaner. The system has found:
without finding:
The interesting part of reward hacking is that the behaviour can become more pronounced as optimisation improves. A weak optimiser may never discover the loophole. A stronger optimiser searches the space more effectively and finds a strategy that humans did not anticipate. In that sense, capability can reveal weaknesses in the objective rather than merely improving performance on the intended task.
Later in the book we will build a small environment where this happens. We will define an intended objective, construct an imperfect reward, increase optimisation pressure and observe the point at which the agent begins to exploit the proxy. The purpose will be to make the concept concrete enough that we can measure it rather than merely repeat the phrase reward hacking.
1.8 From Base Models to Assistants
To understand modern alignment work, we need a high-level picture of how a pretrained language model becomes the kind of assistant that most people interact with. A base language model is not automatically a helpful conversational system. Pretraining primarily teaches the model to predict text continuations over very large corpora. That produces remarkable language and reasoning capabilities, but text continuation and useful assistant behaviour are not the same objective.
Modern systems therefore undergo one or more forms of post-training. A simplified pipeline might look like this:
Real post-training pipelines are more varied than this diagram suggests. Different laboratories use different combinations of supervised data, preference optimisation, reinforcement learning, synthetic feedback, rule-based methods and model-generated data. The simplified pipeline is still useful because it shows that many behaviours that users associate with a modern assistant are learned after pretraining. Instruction following, conversational conventions and some safety behaviour are all influenced by post-training.
This matters for safety because post-training can improve behaviour, but it can also introduce new failure modes. The same mechanisms that make a model more helpful can create incentives for sycophancy, over-refusal or reward-model exploitation if the training signal is poorly specified.
1.9 Supervised Fine-Tuning
One of the simplest post-training stages is supervised fine-tuning. Suppose we collect a dataset of desirable assistant behaviour:
where xi is an instruction and yi is a high-quality response. We fine-tune the model to assign higher probability to responses like yi when it encounters instructions like xi. For an autoregressive model, the supervised fine-tuning loss can be written as:
Conceptually, the model is being shown examples of desirable behaviour. When it sees an instruction resembling xi , we want it to place more probability on responses resembling yi. This can transform a base model into something far more useful as an assistant.
The limitation is that many prompts do not have a single ideal answer. Humans may recognise that one response is better than another without being able to write the perfect answer themselves. That observation leads naturally to preference learning.
1.10 Learning from Human Preferences
Suppose we give a model a prompt x and generate two candidate responses, yA and yB. A human evaluator compares them and indicates that one is preferred:
We can now construct a preference dataset containing triples:
where yw is the preferred response (think of w meaning winning response) and yl is the less-preferred response (think of l meaning losing response). Rather than asking annotators to specify every property that a good assistant response should possess, we learn from comparative judgements.
Since a reward model can assign a score r(x,y) to a prompt-response pair, the training objective encourages that
A common Bradley-Terry-style formulation models the probability that the winning response is preferred as:
where:
The corresponding loss can therefore be written as:
The equations are not important merely because they are mathematically neat. They show how a vague concept such as which answer is better? becomes structured data and then becomes an optimisation problem. That translation from an abstract safety property into something measurable is one of the central themes of technical AI safety.
1.11 RLHF and Preference Optimisation
Reinforcement Learning from Human Feedback, usually shortened to RLHF, builds on the idea that human preferences can become a training signal. At a high level, humans provide preference information, a reward model learns to predict those preferences, the language model generates responses and the policy is updated so that responses receiving higher predicted reward become more likely.
Conceptually, the policy produces a response:
and the reward model evaluates that response:
The policy parameters can then be adjusted to increase expected reward:
In practice, we usually do not want the updated policy to move arbitrarily far from a reference model, because unconstrained optimisation against an imperfect reward model can encourage strange behaviours. A simplified objective therefore often includes a KL-divergence penalty:
The KL term penalises excessive deviation from the reference policy ( or acts as a regularisation term). We will return to the mechanics later but the important point here is that human judgements have been converted into a scalar signal that can then be optimised.
RLHF has been extremely influential in making language models more useful and better behaved. It is not, however, equivalent to solving alignment. It is one family of methods for shaping behaviour, and its success depends on the quality of preference data (hence why top labs higher diverse groups of people across the world, hoping their preferences align and improve quality of this data. I was fortunate to have been part of a group doing some of this work part-time when one of the top labs was training one of their flagship models in 2023), the reward model, the optimisation procedure and the generalisation of the resulting policy.
1.12 Who Are the Humans in Human Feedback?
The phrase human feedback can sound more objective than it really is. Suppose ten annotators compare two candidate answers and six prefer response A while four prefer response B. What should the training label be? The majority preference is easy to record, but the disagreement still contains information. Why did the four choose B and not A and vice versa? Now imagine the task involves humour, acceptable risk, political content, medical uncertainty, offensive language or some other domain where reasonable people can disagree.
Human preference is not a universal scalar hidden somewhere in nature waiting to be measured. The observed signal depends on the annotator, the instructions they were given, their knowledge, the context and the population from which they were sampled. We might write this schematically as:
Some of these questions belong partly to social science and philosophy, but there is also a technical layer that we can investigate. We can measure inter-annotator agreement, model label uncertainty, compare outcomes across annotator groups and estimate how sensitive a reward model is to the composition of its training data. We can examine whether the model learns robust features of helpfulness or superficial correlations associated with particular raters.
The key point is that human feedback does not remove the measurement problem. It moves the measurement problem into the process by which human judgements are collected and modelled.
1.13 The Reward Model Is Still a Model
There is another approximation that is easy to forget: a reward model is not human preference. It is a machine-learning model trained to predict human preference. The pipeline therefore contains several layers:
Each arrow can introduce error. Suppose humans generally prefer answers that are accurate, concise, polite, confident and clearly structured. The reward model may discover that confidence and polished presentation are especially strong predictors of high scores. If the policy is then optimised aggressively against that reward model, it may learn that confident, polished answers score well even when factual accuracy is weaker.
We may have wanted the learned reward to behave approximately like:
but the fitted function could behave more like:
ignoring the Helpfulness altogether and reducing the impact of Accuracy. Those coefficients are purely illustrative, but the mechanism is important. Once an imperfect learned reward becomes the optimisation target, the policy can exploit imperfections in the reward model. Reward hacking has returned, only this time the reward function is itself a neural network trained from human labels.
This is one reason safety researchers care not only about whether a reward model performs well on a held-out preference dataset, but also about what happens when another optimiser is placed on top of it. Predictive accuracy under passive evaluation and robustness under optimisation pressure are not the same property.
1.14 Alignment Is Not Solved by RLHF
It is therefore misleading to write a simple equivalence such as:
RLHF, direct preference optimisation (DPO), constitutional approaches, AI feedback and related methods can all shape model behaviour. They can reduce obvious harmful responses, improve instruction following and make models easier to use. The broader alignment question remains whether the system will continue to behave as intended in situations that matter, including situations that differ substantially from those represented in training.
That last phrase is crucial because a model can perform extremely well on the distribution used during post-training and still fail elsewhere. A reward model can rank familiar responses accurately and behave unpredictably on novel ones. A safety policy can work for direct harmful prompts and fail under obfuscation. Once the environment changes, generalisation becomes part of the safety problem.
1.15 Distribution Shift
Suppose we train a safety classifier on a dataset drawn from:
and evaluate it on a held-out test set from the same underlying distribution. The classifier achieves:
That looks excellent. We deploy the classifier, and real users produce inputs drawn from a different distribution:
with:
Users write in different languages, use slang, make spelling errors and discuss topics that were uncommon in the training data. Attackers deliberately obfuscate words or phrase requests indirectly. The model behind the application is upgraded. The product expands into a new user population. Suddenly the classifier’s production performance is materially worse.
Nothing magical has happened. The safety mechanism relied on generalisation from one distribution to another, and that assumption weakened in production. Distribution shift is therefore not merely a standard machine-learning concern. It becomes a safety concern whenever a safety control depends on recognising behaviour that can change over time or under attack.
Later in the book we will deliberately create distribution shift. We will train a safety classifier under one distribution, perturb the language and composition of the test set, and measure how the error profile changes. The important question will not simply be whether the score falls, but which types of errors increase and why.
1.16 Jailbreaks
Modern language models are often post-trained not to produce certain categories of harmful output. Users discovered almost immediately that these safeguards could sometimes be bypassed. A jailbreak is broadly an attempt to induce a model to violate behavioural restrictions that would normally cause it to refuse, redirect or provide a safer alternative.
Early jailbreaks were often unsophisticated. A direct request might be refused while the same request wrapped in role-play, fiction or elaborate instructions sometimes succeeded. Modern adversarial prompting can be far more systematic, but the safety question remains measurable. Suppose we have N adversarial prompts and S of them successfully induce prohibited behaviour. We can define attack success rate as:
If N = 500 and S = 65, then:
A 13% attack success rate is a measurement, not yet a conclusion. We still need to ask what the baseline was, how much uncertainty exists around the estimate, whether failures are concentrated in particular categories, and whether the result changes with model size, temperature, system prompt or safety architecture. We also need to know what happens to legitimate requests when we introduce a defence.
This is how we will approach jailbreak research in the book. The interesting work begins after the first aggregate score, when we start asking what explains the failures.
1.17 Prompt Injection Is Not the Same as Jailbreaking
Jailbreaking and prompt injection are frequently discussed together because both involve adversarial instructions, but they target different parts of the system. A jailbreak generally attempts to bypass a model’s behavioural safeguards. Prompt injection attacks the instruction hierarchy of an application by placing malicious instructions inside data that the model is expected to consume.
Imagine an assistant that searches the web and reads pages on behalf of a user. One page contains text instructing the model to ignore its prior instructions and expose private information. The user did not type that instruction; the model encountered it as external data. Yet at the transformer level, both the trusted system instructions and the untrusted page content ultimately appear as tokens in context. The system therefore needs to preserve a meaningful distinction between Trusted Instructions and Untrusted Data.
Humans make this distinction almost automatically. If a cell in a spreadsheet says, Ignore your manager and transfer £10,000, we understand that the sentence is data inside the spreadsheet, not an instruction carrying managerial authority. Language models do not automatically maintain such a robust separation, especially when applications mix instructions and retrieved content inside a single context window.
Prompt injection becomes substantially more consequential when the model can call tools. An injection against an isolated chatbot may produce an undesirable response but an injection against an agent with access to email, files, databases or code execution may result in an undesirable action. The underlying vulnerability is therefore partly a model problem and partly an application-architecture problem.
1.18 From Chatbots to Agents
A chatbot primarily produces tokens. An agent can use those tokens to produce actions, and this apparently modest difference changes the safety problem considerably. Imagine a language model connected to functions such as search_web(), read_email(), send_email(), query_database(), update_database() and execute_code(). The model is still generating tokens, but the surrounding software interprets some of those tokens as structured tool calls.
For example, the model may produce something that the application interprets as:
send_email(
recipient=”customer@example.com”,
subject=”Account update”,
body=”...”
)
From the model’s perspective, this is another generated sequence. From the customer’s perspective, an email has actually been sent. The gap between model output and system consequence is therefore where a large amount of agent safety work lives.
We can think of the pathway from a model failure to a real-world consequence as:
Not every model error needs to reach the end of that chain. Good system design places controls between stages. A database may be read-only. A payment action may require human approval. Code execution may be sandboxed. An email tool may restrict recipients. High-risk actions may require deterministic validation or a second independent approval step.
This introduces the idea of a failure’s blast radius. Two applications can use the same model and have radically different risk profiles because one merely returns text while the other can alter external state. Evaluating the model alone does not capture that difference.
1.19 Least Privilege and the Consequence of Error
Computer security has dealt with a related problem for decades through the principle of least privilege. A component should receive only the permissions it genuinely requires to perform its task. There is no good reason for a reporting application to have permission to delete a production database simply because administrator access was convenient during development. The same principle applies directly to AI agents.
If an agent only needs to search customer records, give it read access. If it needs to draft an email but not send one, expose a drafting tool rather than a sending tool. If a high-risk action is occasionally necessary, require explicit approval rather than granting unrestricted authority throughout the entire workflow.
Suppose the probability that the agent makes a consequential decision error is P(E) and the potential impact available under its permissions is I(Permissions). A deliberately simplified expression for risk is:
This is not a complete quantitative risk model, but it highlights two distinct engineering levers. We can improve the model and reduce P(E), or we can constrain the system and reduce the impact available when E occurs. A safety programme that focuses only on model reliability ignores the second lever.
No realistic engineering team should assume P(E) = 0. Models will make mistakes. Safety engineering therefore has to ask what the surrounding system does when a mistake occurs, how quickly it can be detected, and whether the action can be stopped or reversed.
1.20 Dangerous Capabilities
So far, most of our examples have involved systems failing to behave as intended. Another category of concern arises when the model behaves exactly as requested but the capability itself creates risk. A model may become substantially better at finding software vulnerabilities, conducting social engineering, reasoning about biological processes or autonomously executing long sequences of tasks. Many of these capabilities also have legitimate uses. Security professionals need to understand vulnerabilities, scientists need to reason about biology, and organisations want agents capable of completing complex workflows.
The safety question is therefore rarely as simple as labelling a capability good or bad. Many important capabilities are dual use. The same technical competence can support beneficial or harmful activity depending on the user, context, access and controls around the system.
This creates a useful distinction between capability and behaviour. A model may possess the knowledge required to perform a dangerous task while normally refusing to assist with malicious requests. Conversely, a model may occasionally produce unsafe text without possessing enough real capability for the output to translate into meaningful harm. We therefore need to ask two different questions: Can the model do it? and Will the model do it under relevant conditions?
Capability evaluations primarily investigate the first question, while behavioural safety evaluations often investigate the second. A serious assessment may require both. If a model has a dangerous capability but a behavioural safeguard prevents its use, we then need to investigate the robustness of the safeguard under fine-tuning, adversarial prompting, tool use and other realistic conditions.
1.21 Capability Is Not the Same as Real-World Risk
Even when a benchmark detects a dangerous capability, we should be careful about the claims we make. Suppose a model correctly answers 80% of questions on a cybersecurity benchmark. We cannot immediately conclude that the model makes real cyberattacks 80% easier. The benchmark may test knowledge rather than execution, while a real attack requires persistence, access, adaptation, judgement and interaction with systems that behave unpredictably.
A benchmark gives us some measurable quantity M, while the thing we ultimately care about is a real-world quantity R. The relationship M → R must itself be justified. If the benchmark poorly represents real-world risk, sophisticated statistics do not repair the validity problem.
This is where AI safety and AI evaluation become inseparable. When we later evaluate safety benchmarks, we will repeatedly ask what a score actually permits us to claim. A model achieving 95% accuracy on a harmful-content benchmark means that it achieved 95% under the conditions represented by that benchmark. It does not mean that the model is “95% safe”. Safety has no universal denominator.
1.22 Frontier AI and Catastrophic Risk
At this point we reach the area of AI safety that attracts some of the strongest disagreement. Some researchers are concerned not only about harms from current systems, but about the possibility that future systems could become capable enough to create very large-scale or catastrophic risks. The scenarios vary. Some focus on deliberate misuse of powerful systems. Others focus on autonomous systems pursuing objectives in ways humans cannot adequately supervise. Some researchers are concerned that sufficiently capable systems could become difficult to constrain because they outperform humans in domains such as planning, cyber operations, persuasion or scientific research.
At the most extreme end is existential risk: outcomes severe enough to threaten humanity’s long-term future or survival. These claims deserve care because they involve substantial uncertainty, and the field contains genuine disagreement about probabilities, mechanisms and timelines. A technical treatment should neither dismiss the questions simply because they concern future systems nor treat speculative scenarios as established facts.
The productive approach is to identify narrower hypotheses that can be investigated today.
· Can models recognise when they are being evaluated?
· Can they hide information from evaluators?
· Can they pursue objectives over long sequences of actions?
· Can they exploit weaknesses in oversight?
· Does a behaviour become more pronounced as model capability increases?
· Can experiments on smaller systems predict behaviour at larger scales?
These experiments do not prove that a catastrophic scenario will occur. They are attempts to translate difficult future-facing concerns into measurable precursor questions. That distinction matters because technical safety research should begin with hypotheses that can fail, not with a desired conclusion.
1.23 Scalable Oversight
One problem becomes especially interesting as AI systems become more capable: what happens when evaluating the model’s output becomes harder than producing the output in the first place? If I ask a model to solve a simple arithmetic problem, I can verify the answer quickly. If I ask it to review ten thousand lines of unfamiliar code for subtle vulnerabilities, verification becomes more demanding. If I ask it to produce work in a specialised scientific domain where I am not an expert, I may not be able to evaluate the result reliably at all.
We can express the potential asymmetry as Cgeneration < Cverification where C represents the cost or difficulty of performing the task. An AI system may eventually become capable of producing outputs that humans cannot cheaply or reliably assess. If the alignment process depends on people judging whether the output is correct or desirable, straightforward human supervision stops scaling.
Scalable oversight asks how we might preserve reliable supervision as the tasks become more complex or numerous. Proposed approaches include decomposing difficult tasks into simpler subproblems, using models to critique other models, combining human and AI feedback, automated verification, debate-like protocols and forms of process supervision. The interesting technical question is not whether any of these approaches sounds plausible, but whether it actually improves error detection under controlled experiments.
Does decomposition help humans catch more failures? Does an AI critic detect errors that a human misses? Does the critic share the same blind spots as the model it is evaluating? How does oversight performance change as the underlying model becomes more capable? These are measurable questions, even if the larger motivation concerns future systems.
1.24 Evaluation Awareness, Sandbagging and Deceptive Behaviour
A particularly difficult safety problem arises if a model behaves differently because it recognises the context in which it is being evaluated. Consider an ordinary audit. Employees who know exactly when an audit is taking place may behave differently during the audit than they do during everyday operations. If an AI system can infer that a particular interaction is part of an evaluation, then observed performance during that evaluation may not perfectly represent behaviour in deployment.
We can express the underlying concern as P(B | Evaluation) =/= P(B | Deployment) where B represents some behaviour of interest. This matters because evaluation relies on the assumption that observed behaviour provides evidence about behaviour under future conditions. If the act of evaluation changes the behaviour, that assumption weakens.
Research in this area includes concepts such as evaluation awareness, sandbagging and deceptive behaviour. These terms describe different phenomena and should not be collapsed into a vague claim that a model is “lying”. The technical challenge is to define a behaviour precisely enough that we can test it. Instead of asking whether a model is deceptive in some broad anthropomorphic sense, we can ask whether task performance systematically changes when the prompt indicates that high performance will trigger an undesirable consequence. That is a falsifiable experimental hypothesis.
This is another recurring lesson: broad safety concepts become scientifically useful only after we operationalise them.
1.25 Emergent Misalignment and Behavioural Spillover
Another active area of research asks whether relatively narrow changes to a model can produce unexpectedly broad changes in behaviour. Suppose we fine-tune a model on examples from one specific domain. We might expect the resulting behaviour change to remain mostly within that domain, but neural networks do not store behaviours in neatly isolated compartments. Updating parameters to improve one behaviour can affect representations and tendencies elsewhere.
Let the baseline model be M0. After applying some fine-tuning procedure F using dataset D, we obtain:
We intended to change behaviour on domain D, so we measure:
A responsible evaluation should also measure behaviour outside that target domain:
If this change substantial, the fine-tune has produced behavioural spillover. From a safety perspective this matters because fine-tuning is becoming increasingly accessible. Organisations can adapt open models using LoRA, QLoRA and other parameter-efficient methods without retraining the entire network. If a narrow fine-tune can alter broader safety properties, evaluating only the target task is inadequate.
Later in the book, we will take an open-weight model, establish a broad behavioural baseline, apply a targeted fine-tune and then rerun the safety evaluation. The interesting question will not merely be whether the intended task improved, but what else changed.
1.26 Safety Does Not Stop at Deployment
Safety evaluation should not be treated as a one-time exercise that happens before deployment and is then forgotten. AI systems change continuously. Models are upgraded, system prompts are edited, retrieval datasets are refreshed, new tools are added, thresholds are adjusted, user populations change and adversarial techniques evolve. A system that passed a safety evaluation six months ago may no longer be the same system today.
We can represent a deployed system at time t as:
where Mt is the model, Dt the data, Pt the prompting and orchestration layer, Tt the available tools and Gt the guardrails and controls. If any of these components changes, then in general St+1 =/= St.
The previous safety evidence may therefore no longer apply completely. A prompt change that improves helpfulness could increase harmful compliance. A new retrieval source could introduce prompt-injection risk. A cheaper model could have a different refusal profile. A fine-tune could improve domain performance while degrading calibration or robustness.
This is where AI safety begins to overlap directly with LLMOps. Safety becomes part of deployment, regression testing, observability and change control. A production team needs to know not only whether a model once passed a benchmark, but whether a later system change has caused safety behaviour to regress.
Because model behaviour is probabilistic, these regression tests are not always simple Boolean assertions. We may care about whether attack success rate has increased from 3% to 7%, whether over-refusal has doubled, or whether failures have become concentrated in a particular category. The analysis therefore becomes statistical as well as operational.
1.27 Data Is Part of the Safety System
It is easy to focus so heavily on models that we forget how much safety depends on data. Training data determines what the model is exposed to. Preference data shapes which behaviours are rewarded. Evaluation datasets determine what we choose to measure. Retrieval data determines what information a deployed system sees. Logs determine what we can observe after deployment.
If these datasets are poor, safety conclusions can be poor even when the modelling is sophisticated. Consider a safety classifier trained on examples where harmful requests are obvious and direct. The model may learn that particular words are strong indicators of unsafe intent. On a held-out test set generated using similar conventions, the classifier performs extremely well. In production, users phrase requests indirectly, use slang, switch languages or deliberately disguise intent. The classifier may not have learned the underlying concept at all; it may have learned shortcuts in the dataset.
This is not unique to AI safety. Data scientists have dealt with class imbalance, leakage, shortcut learning and distribution shift for years. The difference is that the consequences of getting them wrong may be more serious in a safety context.
Suppose unsafe content makes up only 0.5% of production traffic. A classifier that predicts safe for every input achieves Accuracy = 99.5% while Recallunsafe = 0.
The headline accuracy looks excellent, yet the system catches nothing. This is why the early technical chapters of the book will spend much more time on precision, recall, class imbalance, decision thresholds and precision-recall curves than on accuracy alone.
1.28 The Asymmetric Cost of Errors
False positives and false negatives are not merely entries in a confusion matrix. In a safety system, they represent different kinds of failure and often carry very different costs. A false positive may block a legitimate user from performing useful work. A false negative may allow harmful content or an unsafe action to pass through a control.
If CFP represents the cost of a false positive and CFN the cost of a false negative, then a decision rule can be viewed as attempting to minimise something like:
Expected Cost = CFPP(FP) + CFNP(FN)
The difficult part is that these costs are not always easy to specify numerically. A security-critical system may tolerate many more false positives than a general-purpose assistant. A medical application may treat some false negatives as far more serious than an inconvenience caused by refusal. A customer-service assistant may become unusable if the false-positive rate is too high, even if the harmful-compliance rate is extremely low.
This is why safety often lives on a trade-off frontier rather than at a single optimum. A useful intervention reduces one form of risk without introducing an unacceptable amount of another. We will return to this idea repeatedly when selecting classifier thresholds, comparing guardrails and analysing unlearning methods.
1.29 Safety Classifiers Are Models Too
Many AI systems place additional models around the primary language model. A user prompt may pass through an input classifier, the generated response may pass through an output classifier, another model may detect prompt injection and a separate evaluator may inspect proposed tool calls. These components are often described collectively as guardrails.
A simplified architecture might look like this:
User Prompt
|
v
Input Safety Classifier
|
+---- unsafe ----> Block / Redirect
|
v
Language Model
|
v
Output Safety Classifier
|
+---- unsafe ----> Block / Regenerate
|
v
User
The presence of a guardrail can create a false sense of certainty. The classifier is itself a machine-learning model. It has a training distribution, false positives, false negatives, threshold choices and potentially adversarial weaknesses. Its performance may differ across languages. Its labels may contain errors. It may become stale as the production distribution changes.
The safety mechanism therefore needs evaluation just as much as the main model. One of our first substantial projects will be to build such a classifier ourselves. We will deliberately begin with a simple baseline such as TF-IDF and logistic regression so that every part of the pipeline is visible: the dataset, preprocessing, class balance, score distribution, confusion matrix and threshold. We can then add more sophisticated representations and models once the underlying mechanics are clear.
The point is not that logistic regression is the final answer to AI safety. The point is that understanding a simple system from first principles makes it much easier to reason about the failure modes of a more complex one.
1.30 Red Teaming and Adversarial Robustness
Ordinary evaluation asks how well a system behaves under expected conditions. Red teaming changes the mindset by asking what happens when somebody actively tries to make the system fail. A normal test set may attempt to represent typical user behaviour, while a red-team dataset is often deliberately unrepresentative because the purpose is to search for edge cases, attack surfaces and weak assumptions.
Suppose a model refuses a direct harmful request. That tells us very little about robustness. We might paraphrase the request, alter the framing, introduce role-play, split the task across multiple turns or place conflicting instructions inside retrieved content. The objective is not to prove that the model is bad but to discover where the behavioural boundary actually lies.
A simple aggregate metric might again be attack success rate:
The aggregate score is only a starting point. We may want to estimate:
and perhaps:
Now we are doing experimental design rather than simply running a benchmark.
Red teaming also sits inside the broader problem of adversarial robustness. In conventional machine learning, an attacker may search for a perturbation
such that subject to some constraint on the perturbation. Language is more complicated because inputs are discrete and semantic meaning matters, but the underlying idea is similar: the attacker searches for a transformation that preserves enough of the intended meaning while changing the system’s behaviour.
This is why AI safety and cybersecurity overlap so naturally. Once a safety control becomes valuable enough to attack, we should expect some users to search deliberately for weaknesses and to adapt when those weaknesses are patched.
1.31 Training-Data Safety, Poisoning and Memorisation
Not all safety failures occur at inference time. Training introduces another set of risks. A model trained on enormous quantities of text will inevitably encounter low-quality, harmful, sensitive and contradictory information. Some of that material may be included accidentally and some may be inserted deliberately.
The first problem is filtering. If certain categories of data should be removed before training, we need a classifier capable of identifying them at very large scale. This again becomes a decision problem:
An aggressive threshold may improve recall while damaging precision. We may remove most genuinely hazardous material but also discard substantial amounts of legitimate scientific or technical content. We can express the trade-off abstractly as:
where tao is the filtering threshold and lambda reflects how much retained utility matters relative to the safety objective.
A separate problem is data poisoning, where an attacker deliberately inserts or modifies training data in an attempt to influence the learned model.
Another is memorisation, where a model reproduces specific sequences it encountered during training.
These problems are sometimes discussed under a single phrase such as unsafe training data, but they are technically distinct and require different evaluation methods.
Privacy provides a useful example of why controlled experiments are important. The fact that private information appeared somewhere in a training corpus does not automatically mean that the model can reproduce it. Models are not conventional databases. However, models can memorise, and the probability that a sequence becomes extractable may depend on factors such as duplication, rarity, model capacity and training procedure:
We can investigate this without exposing anybody’s real personal data by inserting synthetic canary strings into a training corpus, varying their frequency and testing whether they can later be recovered. A vague privacy concern then becomes a controlled experiment.
1.32 Bias and Fairness as Measurable Properties
Bias is another term that becomes vague quickly unless we define exactly what property we mean. Suppose a model makes binary predictions for two groups, A and B. One fairness criterion might require similar positive prediction rates:
Another might focus on equal true-positive rates:
A third might focus on calibration, asking whether a predicted probability of 0.8 corresponds to approximately the same empirical outcome rate across groups. These definitions are not equivalent, and in some settings they cannot all be satisfied simultaneously.
This is why an instruction such as make the model unbiased is not a complete technical specification. We first need to decide which measurable property is relevant to the application and what trade-offs follow from that choice. There are important legal, ethical and societal questions involved in choosing the criterion, and those deserve deeper treatment than this book will attempt. Our technical focus begins after that choice has been made: how do we measure the criterion, quantify uncertainty, identify statistically meaningful differences and analyse whether an intervention improves one metric while worsening another?
This is an example of the boundary we will maintain throughout the book. We will not pretend that a statistical metric resolves the normative question of what fairness ought to mean. We will show how to measure a stated definition correctly and understand the limits of the resulting evidence.
1.33 Hallucination, Truthfulness and the Evaluator Problem
Hallucination is familiar, but even this apparently simple problem contains multiple distinct failure modes. A model can produce a proposition that is factually false, cite a source that does not exist, answer confidently when the correct response should express uncertainty, or repeat information that was once correct but is now outdated. These behaviours require different forms of evaluation. For a benchmark containing N questions with reference answers, the simplest metric is:
Generative outputs complicate the meaning of correct. Two semantically equivalent answers may use different wording. A partially correct response may contain one serious factual error. A model may reach the correct conclusion using flawed reasoning. Exact string matching is therefore often inadequate.
We can use human evaluation, deterministic rules, specialised classifiers or another language model acting as a judge. Each option introduces assumptions. If a powerful model evaluates a weaker model, the evaluator may prefer longer answers, reward confident phrasing, share the same factual misconception or be sensitive to answer order and formatting.
If the true quality of a response is Q(y) while the evaluator produces an estimate Q(y), then the evaluator error is:
Random error is inconvenient, but systematic error is more dangerous. If, for example,
then verbosity itself becomes an artificial advantage. If the model is optimised against this evaluator, it may eventually learn to exploit the bias. The reward-hacking problem has appeared again in a new form.
Safety and evaluation are so tightly connected. We cannot make a strong claim about model safety without also asking whether the evaluation system measuring that safety is itself reliable.
1.34 AI Safety Has a Measurement Problem
By now, a pattern should be becoming clear. We care about broad concepts such as harmlessness, truthfulness, helpfulness, robustness, fairness, controllability, dangerous capability and alignment. None of these arrives as a neat numerical variable that can simply be read from the model. We construct measurements that we can then use to draw some conclusions from. A useful general pattern is:
If the operational definition is poor, every subsequent step can be implemented perfectly and the conclusion may still be wrong. If we define safety entirely as refusal rate, we have measured the wrong thing. If we evaluate truthfulness using an evaluator with a systematic bias, the score may mislead us. If we test jailbreak robustness only using attacks that appeared during safety training, high performance may tell us little about genuinely novel attacks.
Technical AI safety therefore requires more than knowledge of machine learning. It requires experimental discipline. We need to ask what we are actually trying to measure, why a metric should represent the property we care about, what population a dataset represents, what uncertainty exists in the estimate and what alternative explanation could produce the same result.
These questions are not peripheral to the book. They are the method we will use throughout it.
1.35 The Broader Societal Side of AI Safety
AI safety extends well beyond the technical problems we have discussed. AI systems can influence employment, wages and the structure of labour markets. They may affect how productivity gains are distributed, how education is delivered, how misinformation spreads and how political institutions function. The development of increasingly capable AI also raises questions about concentration of power, intellectual property, environmental cost, cultural representation and access between richer and poorer parts of the world.
These questions matter, but they are not the questions this book is best positioned to answer. A serious treatment of labour-market consequences requires economics. Questions around rights, liability and discrimination may require law, sociology and political philosophy. Questions about global access and inequality involve development economics, public policy and other disciplines. It would be easy to add superficial chapters on all of these topics simply so that the book appeared comprehensive, but that would not make the treatment useful.
The fact that a problem falls under the broad umbrella of AI safety does not mean that every AI engineer or data scientist is automatically qualified to provide a serious treatment of it. Part of technical work is understanding where technical methods are appropriate and where expertise from other disciplines is necessary.
We will therefore acknowledge those broader dimensions where they affect our engineering decisions, but this book will deliberately focus on technical AI safety: the parts of the problem that we can investigate with data, models, mathematics, experiments and system controls.
1.36 What This Book Means by Technical AI Safety
From this point onwards, our working definition of technical AI safety will be deliberately practical. We are interested in questions that can be turned into measurable hypotheses and engineering interventions.
· How accurately can we identify harmful model inputs and outputs?
· How should a classifier threshold be selected when false positives and false negatives have different costs?
· How robust are those classifiers under distribution shift and adversarial input?
· How do we measure over-refusal alongside harmful compliance?
· How do post-training and fine-tuning change safety behaviour?
· When does reward optimisation create unintended strategies?
· How do we evaluate jailbreak robustness, tool use and agent permissions?
· Can a capability be selectively reduced without destroying general utility?
· How do we monitor safety once the system changes in production?
The common structure behind these questions is empirical. We begin with a claim, define what would count as evidence, construct or select a dataset, run an experiment, quantify uncertainty, inspect failures, introduce an intervention and evaluate again.
A useful summary is:
followed by:
This is the core workflow of the book. The subject matter will become more sophisticated, but the underlying habit of thinking will remain the same.
1.37 From Safety Claims to Experiments
Consider the claim that a new guardrail makes a model safer. That statement is too vague to test. We need to decide which failure mode we are targeting and how we will measure it. Suppose the relevant measure is jailbreak attack success rate. Our hypothesis might be written as:
We can then define an attack set, choose a model and evaluate both configurations. Suppose we observe:
and:
The guardrail appears to reduce attack success by eleven percentage points. That sounds positive until we examine legitimate traffic. Suppose the over-refusal rate changes from:
to:
The intervention reduced successful attacks substantially, but it also caused almost one in five harmless prompts to be refused. Whether that trade-off is acceptable depends on the application. A high-risk system may accept more friction than a general-purpose assistant.
We also need to estimate uncertainty. If the experiment contains only fifty prompts, the difference may be unstable. If the failures are concentrated in one attack family, the aggregate score may hide the weakness. If the guardrail was tuned on the same attacks used in evaluation, the apparent improvement may not generalise.
This is what turns a safety slogan into safety engineering. The first number is rarely the end of the analysis.
1.38 What We Are Going to Build
The rest of the book will move from familiar data-science techniques into increasingly sophisticated safety experiments. We will begin with safety data itself: how datasets are constructed, what labels mean, how imbalance can mislead us and how annotation choices shape the conclusions we can draw. We will build a harmful-content classifier using conventional machine-learning techniques before moving towards neural classifiers and model-based guardrails.
That starting point is deliberate. Many AI safety problems initially look like familiar data-science problems. Classification, probability, calibration, threshold optimisation, confidence intervals, distribution shift and error analysis do not become obsolete simply because the application involves a language model. In fact, safety makes those techniques more important because the cost of misinterpreting a metric may be higher.
We will then evaluate model behaviour using public benchmarks for harmful compliance, refusal, over-refusal, truthfulness and bias. We will build an evaluation harness capable of running the same tests repeatedly across models and model versions, which naturally leads into regression testing and monitoring.
From there we will move closer to alignment. We will work with preference data, derive a preference-learning objective and train a small reward model. We will examine disagreement in human feedback and investigate how modelling choices influence the reward signal. We will construct a simple reward-hacking environment and observe how optimisation pressure can exploit an imperfect objective.
Later, we will fine-tune an open-weight model, establish a broad behavioural baseline and then measure what changes outside the intended target domain. We will red-team models, measure attack success rates, build guardrails and deliberately test those guardrails under adversarial conditions. We will connect models to controlled tools and study how permissions, approval requirements and execution policies change the consequences of model errors.
Finally, we will move into more research-oriented problems such as hazardous capability evaluation, machine unlearning and scalable oversight. The experiments will become more sophisticated, but the method will remain the same: define the claim, measure it carefully, inspect what failed and avoid claiming more than the evidence supports.
1.39 AI Safety from First Principles
There are already many useful safety libraries, evaluation platforms, guardrail frameworks and commercial APIs. We will use some of them later, but we will not begin there. The guiding principle of this book is to understand the mechanism before hiding it behind an abstraction.
If we want to understand safety classifiers, we should first build one. If we want to understand preference learning, we should implement the loss. If we want to understand calibration, we should calculate it. If we want to understand reward hacking, we should create an environment in which it occurs. If we want to understand distribution shift, we should deliberately shift the distribution and watch the system fail. If we want to understand safety monitoring, we should construct the underlying statistics before relying on a dashboard.
Understand the mechanims before abstracting the mechanism way
Libraries are valuable precisely because they save us from repeatedly implementing low-level details. The problem begins when we use the abstraction without understanding what has been abstracted. A framework may tell us that a model received a safety score of 0.87. Our first response should be to ask what 0.87 means: which dataset produced it, how the labels were created, what evaluator was used, what uncertainty exists around the score, what kinds of failures are hidden inside the remaining 0.13 and whether another metric would produce a different conclusion.
Those questions are not scepticism for the sake of scepticism. They are basic scientific discipline.
1.40 A Mental Model for the Rest of the Book
Before we leave this overview, it is useful to establish one final mental model. At the centre of an AI system is the model M, with capabilities and behavioural tendencies learned through pretraining and post-training. Around the model sits the application A(M), which provides prompts, retrieval, memory, tools and orchestration. Around the application sit controls G(A(M)), including classifiers, permissions, approval mechanisms, deterministic rules and other guardrails. Around all of that sits an operating environment containing users, attackers, changing data and real-world consequences.
A deliberately broad expression for risk is therefore:
No single benchmark captures all of this. No alignment technique solves all of it. No guardrail makes the underlying model irrelevant, and no well-behaved model compensates for reckless system architecture. The interesting work lies in understanding how the pieces interact and in constructing experiments that measure those interactions rather than relying on intuition.
That is where we go next. In the next chapter, we will leave the broad map of AI safety behind and begin turning safety into data. We will examine how real safety datasets are constructed, what their labels mean, why class imbalance can make accuracy misleading and how false positives and false negatives acquire different meanings in a safety context. Then we will load the data and start building.