Machine Learning Basics: Learning from examples instead of rules

Ask a chatbot for a limerick about tax law and you'll have one in seconds. It will rhyme, it will mostly keep the rhythm, and it will mention deductions. That's strange when you consider who built it. No engineer at any AI company wrote instructions for composing limericks. None wrote instructions about tax law either.

Most software doesn't work like this. A spreadsheet adds up a column because a programmer wrote out, step by step, how to add. If the program handles a situation, someone thought of that situation in advance. Timothy B. Lee and Sean Trott describe the difference in their explainer on language models. Conventional software is made by programmers who give computers "explicit, step-by-step instructions." A chatbot is built on a system "trained using billions of words of ordinary language" (Lee and Trott 2023).

The important word there is "trained." The chatbot's abilities weren't written down by anyone. The system picked them up from a very large number of examples. This approach is called machine learning. It's older than chatbots and much wider. It's how a photo app finds pictures of your dog, how a phone turns your speech into text, and how a streaming service picks what to recommend (Royal Society 2017, chap. 1).

Three things follow from building software this way.

• Nobody can point to the place where a skill is stored. There's no line of code for limericks. Lee and Trott go further and say that "no one on Earth fully understands the inner workings" of these systems (Lee and Trott 2023).
• What the system can do depends on what its examples contained. Abilities that were well represented in the examples tend to be strong, and abilities that were rare or absent tend to be weak or missing.
• The system fails differently from ordinary software. A spreadsheet either adds correctly or shows an error. A learned system can be nearly right, right for the wrong reason, or wrong in a way nobody predicted.

This changes the question you should ask when a system surprises you. With ordinary software, the useful question is "What was it told to do?" Somewhere there's an instruction that explains the behavior, and a programmer can find it and change it.

With a learned system, that question has no good answer. The more useful one is "What was it trained on?" If a photo app finds every dog in your library and also labels a fox as a dog, no rule about foxes went wrong. The app's examples probably taught it a pattern that fits dogs and happens to fit foxes too.

You can put that question to any AI tool you use, whether or not its maker tells you the answer. It also explains why two AI products can look alike on the surface and behave very differently underneath.

References

• Lee, Timothy B., and Sean Trott. 2023. "Large Language Models, Explained with a Minimum of Math and Jargon." Understanding AI, July 27, 2023.
  • Free: Understanding AI
• Royal Society. 2017. Machine Learning: The Power and Promise of Computers That Learn by Example. London: The Royal Society.
  • Free: Royal Society report (PDF)

Introduction

Nearly all the software you use does exactly what someone wrote down for it to do. That's a strength, and for some tasks it turns out to be a hard limit.

This reading explains what written rules are, where AI built from rules succeeded, and why rules fail on tasks such as understanding language and recognizing what's in a picture.

Instructions Written in Advance

An algorithm is a fixed sequence of steps that turns an input into an output. A tax calculator is a clear case. It takes your income, checks which bracket it falls in, applies that bracket's rate, and subtracts your deductions. Each step was written by a person who knew the tax rules.

Software built this way has three useful properties. It's predictable, because the same input always gives the same output. It's inspectable, because a programmer can read the steps and see why a result came out as it did. And it's fixable, because a wrong result traces back to a wrong step that someone can correct.

A rule-based system is software whose behavior comes entirely from rules that people wrote. Most everyday software fits this description.

AI Built from Rules

Early work in artificial intelligence often took the same approach. Researchers tried to write down the rules of intelligent behavior directly.

The best-known products of this approach are expert systems. An expert system is a program that stores a specialist's knowledge as a large set of if-then rules. Each rule has the form "if some condition is true, then draw this conclusion or take this action." The first expert system was built at Stanford University in 1965 to analyze chemical compounds, and later ones were applied to fields such as medical diagnosis (Zwass n.d.).

Expert systems work in narrow domains where specialists can say what they know. Inside those domains they can be useful, though they "remain aids to, rather than replacements for, human experts" (Zwass n.d.). Outside them they have nothing to offer, since a rule applies only to the cases its author had in mind.

Where Rules Run Out

Try writing rules for recognizing a cat in a photograph. You might begin with pointed ears, whiskers, and a tail. Then you need rules for a cat seen from behind, a cat curled up asleep, a cat half hidden by a chair, and a cat in poor light. Each rule needs exceptions, and each exception needs exceptions of its own.

Language is the same. Consider a rule that marks a sentence as a complaint if it contains the word "terrible." It will catch "The service was terrible." It will also catch "I had a terrible time choosing, because everything looked so good."

Two problems are at work here.

• The exceptions never end. Language and images vary without limit, so a finite list of rules can't cover every case.
• People can't state the rules they follow. You recognize a friend's face in an instant, and you can't say how you do it. A skill that can't be put into words can't be written into a program by hand.

When a rule-based system meets an input its rules don't cover, it doesn't make a reasonable guess. It gives an error or a wrong answer. This quality is called brittleness: a system works inside the boundary of its rules and fails abruptly outside it. The input that triggers the failure is called an edge case, meaning a situation the rule-writers didn't anticipate.

Conclusion

Written rules give software that is predictable, inspectable, and fixable, and they remain the right tool for tasks people can fully describe. They break down on tasks people can do but can't spell out, which include most of what we do with language and vision. That limit is the main reason AI researchers turned to systems that learn from examples.

Key Terms

• Algorithm: A fixed sequence of steps that turns an input into an output.
• Rule-based system: Software whose behavior comes entirely from rules that people wrote.
• Expert system: A program that stores a specialist's knowledge as a large set of if-then rules.
• Brittleness: The tendency of a system to work inside the boundary of its rules and fail abruptly outside it.
• Edge case: A situation the rule-writers didn't anticipate.

References

• Zwass, Vladimir. n.d. "Expert System." Encyclopaedia Britannica. Accessed October 3, 2026.
  • Free: Encyclopaedia Britannica

Introduction

News stories say that an AI system "learned" to do something. The word suggests a student, and that picture misleads in several ways.

This reading explains what a machine learning system is made of, what its builders decide, and what they leave to the system.

The Three Parts of a Learning System

Machine learning is a way of building software in which the system's behavior comes from examples instead of from rules a person wrote. The Royal Society's 2017 report on the subject describes such systems as "learning from data, rather than following pre-programmed rules" (Royal Society 2017, chap. 1). Every machine learning system has three parts.

1. Training data is the collection of examples the system learns from. An example is one item in that collection, such as one photograph, one email, or one sentence.
2. A model is the part of the system that takes an input and produces a prediction. A prediction is the model's output for a given input: its best guess at the answer.
3. A training procedure compares the model's predictions with the examples and adjusts the model so that its predictions improve.

Take a photo app that finds pictures of dogs. Its training data is a large set of photographs. Its model takes a photograph and predicts "dog" or "not dog." The training procedure runs the model on the examples, checks how often it's wrong, and adjusts it, over and over, until it's wrong less often.

What a Model Holds

A model doesn't keep its training data. A dog-finding model trained on a million photographs doesn't contain a million photographs, and it doesn't look anything up when you give it a new one.

What it holds is a large set of adjustable settings, stored as numbers. Training changes those settings until the model's predictions fit the examples. The result is a compact summary of patterns in the data. A pattern here means a regularity that holds across many examples, such as a combination of shapes and textures that tends to appear in photographs of dogs.

This matters for how you read claims about AI. When a system gets something right, it hasn't found the answer in a store of examples. It has applied patterns to a new input. The same holds for chatbots. Lee and Trott describe them as "trained using billions of words of ordinary language" (Lee and Trott 2023). That text shaped the model's settings, and the model doesn't consult it when it answers.

What the Builders Decide

The people who build a learning system make three kinds of decisions.

They don't decide which patterns the model finds. The training procedure settles that, and the builders learn what it produced by testing the finished model.

Why Learned Behavior Is Hard to Inspect

With rule-based software, a programmer can read the rules and see why a result came out as it did. A trained model has nothing comparable to read. Its behavior is spread across its settings, and no single number means "pointed ears" or "has a tail."

So the builders of a learning system often can't say exactly why it gave a particular answer. They can test it on many inputs and describe how it tends to behave. In most cases they can't trace one output back to one cause, the way a programmer traces a bug to a line of code (Royal Society 2017; Lee and Trott 2023).

Conclusion

A machine learning system is training data, a model, and a procedure that adjusts the model to fit the data. Its builders choose the data, the design, and the goal, and the procedure finds the patterns. The finished model is a summary of those patterns, held as numbers that people can't easily read.

Key Terms

• Machine learning: A way of building software in which the system's behavior comes from examples instead of from rules a person wrote.
• Training data: The collection of examples a system learns from.
• Example: One item in the training data, such as one photograph, one email, or one sentence.
• Model: The part of a machine learning system that takes an input and produces a prediction.
• Prediction: A model's output for a given input: its best guess at the answer.
• Pattern: A regularity that holds across many examples.

References

• Lee, Timothy B., and Sean Trott. 2023. "Large Language Models, Explained with a Minimum of Math and Jargon." Understanding AI, July 27, 2023.
  • Free: Understanding AI
• Royal Society. 2017. Machine Learning: The Power and Promise of Computers That Learn by Example. London: The Royal Society.
  • Free: Royal Society report (PDF)

Introduction

A system that learns from examples needs some way to tell whether its predictions are right. Where the right answers come from is the main thing that separates one kind of machine learning from another, and it explains why chatbots became possible when they did.

This reading compares three kinds of learning: supervised, unsupervised, and self-supervised.

Supervised Learning: People Supply the Answers

In supervised learning, every training example comes with the correct answer attached. That attached answer is called a label. A photograph is labeled "dog" or "not dog." An email is labeled "spam" or "not spam."

During training, the model makes a prediction for each example, and the training procedure compares the prediction with the label. When they differ, the model is adjusted.

Supervised learning works well, and it has a cost. Someone has to produce the labels. For a large system that can mean people labeling a very large number of examples by hand, one at a time (Royal Society 2017, chap. 1). Some tasks have no agreed label at all. It's hard to say what the single correct label would be for a paragraph of an essay.

Unsupervised Learning: No Answers at All

In unsupervised learning, the examples have no labels. The system looks for structure in the data on its own, and the usual result is a set of groups.

A retailer might give a system its customers' purchase records and ask it to find groups of customers who buy similar things. Nobody tells the system what the groups should be. It finds whatever groupings fit the data, and people then decide whether those groupings mean anything.

Unsupervised learning needs no labeling work. Its limit is that the system has nothing to check its results against, so it can find groupings that are real but useless.

Self-Supervised Learning: The Data Supplies Its Own Answers

Self-supervised learning takes a different route. The system builds its own labels out of the data, with no human labeling. Google's machine learning glossary describes it as "using the unlabeled dataset to generate labels" (Google for Developers n.d., "self-supervised learning").

Text makes this easy. Take any sentence, hide its last word, and ask the model to predict the hidden word. The correct answer is already there, because it's the word that was hidden. Every sentence ever written can serve as a training example, and nobody has to label any of them.

This is how language models are trained. They learn "by trying to predict the next word in ordinary passages of text," and Lee and Trott call it a key innovation that such models "don't need explicitly labeled data" (Lee and Trott 2023, "How language models are trained").

The Three Compared

Why Self-Supervision Changed What Was Possible

The amount of labeled data in the world is small, because labeling takes human time. The amount of unlabeled text is enormous.

Self-supervision lets a system learn from all of that text with a clear right answer for every prediction. That removed the labeling bottleneck. Language models could then be trained on far more text than any group of people could label, which is one reason they improved so quickly.

Conclusion

The three kinds of learning differ in where the correct answers come from: from people, from nowhere, or from the data itself. Supervised learning is limited by the supply of labels, and unsupervised learning by the lack of anything to check against. Self-supervised learning avoids both limits for data like text, and it's the method behind today's language models.

Key Terms

• Supervised learning: Machine learning in which every training example comes with the correct answer attached.
• Label: The correct answer attached to a training example.
• Unsupervised learning: Machine learning in which the examples have no labels and the system looks for structure in the data on its own.
• Self-supervised learning: Machine learning in which the system builds its own labels out of the data, with no human labeling.

References

• Google for Developers. n.d. "Machine Learning Glossary." Accessed October 3, 2026.
  • Free: Google for Developers
• Lee, Timothy B., and Sean Trott. 2023. "Large Language Models, Explained with a Minimum of Math and Jargon." Understanding AI, July 27, 2023.
  • Free: Understanding AI
• Royal Society. 2017. Machine Learning: The Power and Promise of Computers That Learn by Example. London: The Royal Society.
  • Free: Royal Society report (PDF)

Introduction

A system that does well on the examples it was trained on hasn't shown much yet. What matters is how it does on inputs it has never seen, and a high training score can hide a poor answer to that question.

This reading explains generalization, the ways it fails, how builders test for it, and how the training data limits it.

Generalization Is the Goal

Generalization is a model's ability to give good predictions on inputs that weren't in its training data. It's the whole reason for training. A spam filter that works only on the emails it was trained on is useless, because tomorrow's spam will be new.

The opposite of generalizing is memorization: getting training examples right by retaining them individually, without capturing a pattern that carries over to new cases. A student who memorizes the answers to last year's exam can score perfectly on last year's exam and fail this year's.

Overfitting

Overfitting happens when a model fits the quirks of its training examples instead of the pattern those examples share. The model looks excellent on its training data and does worse on anything new.

Here is a made-up case. Suppose you train a model to tell wolves from dogs, and every wolf photograph in your training data happens to have snow in the background. The model may learn that snow means wolf. On the training photographs that works perfectly. On a photograph of a dog in a snowy garden, the model says "wolf."

Nothing in training would warn you. The model was asked to fit the examples, and it did. It used a shortcut that the examples happened to allow.

Testing on Held-Back Examples

The standard check is simple. Before training, the builders set some examples aside and never show them to the model. This held-back portion is the test data. After training, they run the model on the test data and compare its score there with its score on the training data.

A model that scores well on both has probably learned something general. A model that scores well on the training data and badly on the test data has overfitted (Google for Developers n.d., "overfitting" and "test set").

The check has a limit. Test data is usually drawn from the same source as the training data. If every wolf photograph in that source has snow in it, the test photographs will too, and the shortcut will pass the test.

The Data Sets the Limits

A model can't learn what its examples don't contain. Two kinds of shortfall are common.

• Gaps. If a kind of case is absent from the training data, the model has nothing to go on. A voice system trained only on adult voices may do poorly with children.
• Skews. If the data over-represents some cases and under-represents others, the model's accuracy follows the same imbalance. This is called data bias: a systematic slant in the training data that the model reproduces. A 2018 study tested commercial systems that classify a person's gender from a photograph of their face. Error rates reached 34.7 percent for darker-skinned women and were at most 0.8 percent for lighter-skinned men. The same study found that two widely used collections of face photographs were about 80 and 86 percent lighter-skinned (Buolamwini and Gebru 2018).

Adding more data helps only if the new data fills the gap or corrects the skew. More of the same data teaches the same patterns.

Learned Systems Fail Differently

A rule-based program fails at a sharp boundary: an input either matches a rule or it doesn't. A learned system has no such boundary. Its accuracy falls off gradually as inputs get less like its training examples, and it falls unevenly, so that the system is strong on some kinds of input and weak on others that look similar to you.

The system also gives no signal when it has moved onto weak ground. It produces an answer either way. That makes its failures harder to predict than a rule-based program's, and harder to notice.

Conclusion

Generalization is what training is for, and overfitting and memorization are the ways it goes wrong. Held-back test data catches many failures, though it misses the ones built into the data source. A model's limits follow the gaps and skews of its training data, and its failures are gradual, uneven, and unannounced.

Key Terms

• Generalization: A model's ability to give good predictions on inputs that weren't in its training data.
• Memorization: Getting training examples right by retaining them individually, without capturing a pattern that carries over to new cases.
• Overfitting: What happens when a model fits the quirks of its training examples instead of the pattern those examples share.
• Test data: Examples set aside before training and never shown to the model, used to check how well it generalizes.
• Data bias: A systematic slant in the training data that the model reproduces.

References

• Buolamwini, Joy, and Timnit Gebru. 2018. "Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification." In Proceedings of the 1st Conference on Fairness, Accountability and Transparency. Proceedings of Machine Learning Research 81:77–91.
  • Free: Proceedings of Machine Learning Research
• Google for Developers. n.d. "Machine Learning Glossary." Accessed October 3, 2026.
  • Free: Google for Developers

Introduction

Spam filtering is a good task for comparing two ways of building software. Everyone knows what the right answer looks like, and the task is harder than it seems.

This example works through one small problem twice. The first attempt uses rules a person writes. The second uses patterns learned from examples. All the emails are invented.

The Problem

Here are the subject lines of ten emails. The task is to sort them into spam and not spam.

Five are spam and five aren't. You can sort them at a glance. The question is how to get software to do it.

Working It Through

Step 1: Write three rules and apply them

Start with three rules that seem sensible.

• Rule A: if the subject contains "free," mark it as spam.
• Rule B: if the subject contains "urgent," mark it as spam.
• Rule C: if the subject contains three exclamation marks, mark it as spam.

Rule A marks emails 1 and 2. Rule B marks 7 and 8. Rule C marks 1 and 9. So the rules mark five emails as spam: 1, 2, 7, 8, and 9.

Compare that with the right answers. Emails 1 and 7 are correctly caught. Emails 2, 8, and 9 are ordinary messages wrongly marked as spam. Emails 3, 5, and 10 are spam that got through. Only emails 4 and 6 were correctly left alone. The rules got four out of ten right.

Step 2: Patch the rules

Each error suggests a fix.

• Narrow Rule A so it needs "free" together with "won." That releases email 2.
• Narrow Rule B so it needs "urgent" together with "account." That releases email 8.
• Drop Rule C. That releases email 9.
• Add Rule D: mark anything containing "prize," "cheap," or "lowest price." That catches emails 3, 5, and 10.

The patched rules now sort all ten emails correctly. They have also created new ways to fail. Rule D will mark a colleague's message that says "Found cheap flights for the conference." The narrowed Rule B will miss "URGENT: confirm your password," because that message has no "account" in it. Every patch fits the ten emails more closely and says less about the next email to arrive.

Step 3: Switch to learning

Now take the other approach. In place of rules, collect a large number of real emails that people have already marked as spam or not spam. Those marks are the labels.

A learning system looks across all the examples for signals that tend to go with each label. It isn't limited to single words. It can pick up many weak signals: combinations of words, an unfamiliar sender, the number of links, unusual capitalization, the time a message was sent. No one signal decides the outcome. The system gives each a weight and combines them into a score for how likely a message is to be spam.

Nobody tells the system which signals to use. It finds the ones that separate the two groups in its examples.

Step 4: Test both on two new emails

Take two emails that neither approach has seen.

The first is ordinary: "Reminder: dentist appointment Tuesday." The patched rules leave it alone, which is correct. A learned filter would very likely give it a low spam score, since nothing about it resembles the spam in its examples.

The second is unusual: "Fr3e pr1ze, c1aim n0w." The rules look for exact words, and the spammer has swapped letters for numbers. No rule matches, so the message gets through.

The learned filter doesn't depend on exact spelling alone. If its examples included disguised words like these, or if other signals such as an unfamiliar sender and a link carry enough weight, it can still give the message a high score. If nothing like it appeared in the examples, the learned filter may miss it as well. Which of these happens depends on the training data, and you can't tell by reading the system.

Step 5: Compare how each fails and how each is fixed

The rules failed at a sharp edge. A message either matched a rule or it didn't, and a trivial change in spelling was enough to get past. Fixing the failure means writing another rule, by hand, after the spam has arrived.

The learned filter fails by degree. Its score for the disguised message could be high, middling, or low. Fixing a failure means adding examples of the new kind of spam and training again. Nobody edits the system directly.

Key Considerations

"Learning" in this example means adjusting weights until the system's scores fit the labeled examples. It doesn't mean the system understands what an email says or why someone would send spam. The Royal Society's report on machine learning puts the general point this way: such systems work "by learning from data, rather than following pre-programmed rules" (Royal Society 2017, chap. 1).

A common mistake is to assume that a learned system contains hidden rules that someone could read off if they looked hard enough. It doesn't. The filter's behavior comes from many weights acting together, and there's no list inside it that says "free plus won means spam." This is why the builders of a learned system test it to find out how it behaves, where the author of a rule-based system can read the rules.

Summary

The same ten emails can be sorted by rules or by learned patterns, and the two approaches differ in where their behavior comes from, how they fail, and how they're corrected.

Check: the patched rules sort all ten original emails correctly and miss the disguised one, which is the brittleness the comparison describes.

References

• Royal Society. 2017. Machine Learning: The Power and Promise of Computers That Learn by Example. London: The Royal Society.
  • Free: Royal Society report (PDF)