Machine Learning Basics: Neural networks and how training adjusts them

News stories about AI often report that a model "has billions of parameters." The figure is offered as a measure of size, like horsepower in a car review, and the story moves on. Few stories say what a parameter is or what it means to have billions of them.

A parameter is a number. A model with billions of parameters is a very long list of numbers, and that list is nearly all there is. Timothy B. Lee, a journalist, and Sean Trott, a cognitive scientist, wrote a plain-language explainer on language models. They describe one model from 2020 that had 175 billion of them (Lee and Trott 2023, "How language models are trained").

When an AI company finishes building a model, what it has is a file. If you could open that file, you wouldn't find sentences, or a list of rules, or a database of facts. You'd find numbers, along with a short description of how they're arranged. A chatbot that can explain a tax form or draft a wedding toast is that file of numbers being run by a computer.

This is hard to square with what a chatbot does. It answers questions about history, so it seems the history must be stored somewhere. In a sense it is, though not in a form anyone can read. Lee and Trott compare the numbers to valves in a huge system of pipes, where each valve setting controls how much flows through one connection. Training is the process of turning those valves. As a model sees more and more examples, they write, its numbers are "gradually adjusted to make better and better predictions" (Lee and Trott 2023, "How language models are trained").

Two things follow from this.

• You can translate the word "learned." When a headline says a model learned to write code or learned to pass an exam, the plain version is that its numbers were adjusted until its outputs on those tasks improved. That says nothing either way about whether the model understands the task. It tells you what physically happened.
• You can see why a model is hard to inspect. A company can publish every one of a model's numbers and still be unable to say why it gave a particular answer. The numbers are all visible, and their meaning isn't written on them.

The translation also helps with claims about size. More parameters means more numbers available to adjust, which gives training more room to capture patterns. A parameter count describes a model's capacity, and its abilities depend on what the training did with that capacity.

The questions worth asking are about the adjusting. You can ask what the numbers were adjusted to do, what examples drove the adjustment, and how anyone checked the result. Those questions apply to chatbots from every company, and they have answers even when the numbers themselves can't be read.

References

• Lee, Timothy B., and Sean Trott. 2023. "Large Language Models, Explained with a Minimum of Math and Jargon." Understanding AI, July 27, 2023.
  • Free: Understanding AI

Introduction

AI systems are often said to be built from "neural networks," and the name suggests an electronic brain. The real thing is plainer. It's a large amount of simple arithmetic arranged in a particular order.

This reading explains the single unit a neural network is built from, how units are arranged in layers, and how far the comparison with brains goes.

One Unit

A neural network is a model made of many simple units arranged in layers, each passing a number forward to the next layer. The unit is called an artificial neuron. It takes in several numbers, weighs them, and passes one number on.

The weighing works like this. Each number coming into a neuron has its own weight, a number that sets how much that incoming number counts toward the neuron's result. A large weight means the incoming number counts for a lot. A weight near zero means it's almost ignored. A negative weight means it counts against. The neuron multiplies each incoming number by its weight, adds up the results, and applies one simple fixed rule to the total, such as replacing any negative total with zero. What comes out is a single number.

Lee and Trott describe neurons the same way, as small mathematical functions that work out a weighted sum of what comes in (Lee and Trott 2023, "The feed-forward step"). One neuron can't do anything interesting. It can only say, in effect, how strongly its particular mix of incoming numbers is present.

Layers

Neurons are grouped into layers. A layer is a group of artificial neurons that all work on the same incoming numbers at the same stage. The numbers produced by one layer become the incoming numbers for the next.

Two more terms describe the ends of this chain. The input is what is given to a model, or to one part of it, in the form of numbers. The output is what a model, or one part of it, produces, also in the form of numbers. Everything a network handles has to be turned into numbers first. A photograph becomes a list of brightness values, one for each dot in the picture.

Here is the path of one input through a small network with three layers. Suppose the network's job is to say which digit, zero through nine, appears in a small image of a handwritten number. This is an example Stephen Wolfram, a computer scientist and the founder of the software company Wolfram Research, uses in his own explanation (Wolfram 2023, "Neural Nets").

1. The image arrives as a list of numbers, one for the brightness of each dot.
2. Every neuron in the first layer takes in all of those numbers, weighs them with its own weights, and passes on one number.
3. Every neuron in the second layer takes in the numbers from the first layer, weighs them, and passes on one number.
4. The third layer has ten neurons, one for each digit. Each takes in the numbers from the second layer and produces one number.
5. The network's answer is the digit whose neuron produced the largest number.

Nothing else happens. The numbers move in one direction, from input to output, and each neuron does the same small calculation. The network's ability to tell a three from an eight comes entirely from the values of its weights.

Networks used in practice follow the same plan at a far larger scale. They have many more layers, and each layer has many more neurons. Wolfram describes one network behind a chatbot as having 175 billion weights (Wolfram 2023, "Inside ChatGPT").

The Link to Brains

The vocabulary comes from biology. A brain's nerve cells are connected to one another, and each cell's activity depends on signals from the cells connected to it, with some connections counting for more than others. Early researchers borrowed that picture.

The borrowing is loose. Wolfram calls neural networks "simple idealizations of how brains seem to work" (Wolfram 2023, "Neural Nets"). An artificial neuron is a short calculation, and a living nerve cell is far more complicated. The way artificial networks are trained also has no agreed counterpart in the brain. A statement that an AI system "works like the brain" describes where the idea came from and tells you little about what the system does.

Conclusion

A neural network is many artificial neurons arranged in layers. Each neuron weighs the numbers it receives and passes one number forward, and each layer's output is the next layer's input. What a given network does depends on the values of its weights. The comparison with brains explains the name and little else.

Key Terms

• Neural network: A model made of many simple units arranged in layers, each passing a number forward to the next layer.
• Artificial neuron: One unit in a neural network, which takes in several numbers, weighs them, and passes one number on.
• Weight: A number that sets how much one incoming number counts toward an artificial neuron's result.
• Layer: A group of artificial neurons that all work on the same incoming numbers at the same stage.
• Input: What is given to a model, or to one part of it, in the form of numbers.
• Output: What a model, or one part of it, produces, in the form of numbers.

References

• Lee, Timothy B., and Sean Trott. 2023. "Large Language Models, Explained with a Minimum of Math and Jargon." Understanding AI, July 27, 2023.
  • Free: Understanding AI
• Wolfram, Stephen. 2023. "What Is ChatGPT Doing … and Why Does It Work?" Stephen Wolfram Writings, February 14, 2023.
  • Free: Stephen Wolfram Writings

Introduction

A chatbot can tell you the boiling point of water, so it's natural to assume that the fact sits somewhere inside the model, the way a sentence sits on a page. It doesn't, and the reason explains a good deal about why these systems are hard to inspect and hard to correct.

This reading explains what parameters are, how the same network can do different jobs, and why what a model has learned can't be found in any one place.

Dials

A parameter is one of the adjustable numbers in a model whose values are set by training. Most of a neural network's parameters are weights. A weight is a number that sets how much one incoming number counts toward an artificial neuron's result, where an artificial neuron is one of the simple calculating units a network is built from.

A useful picture is a wall of dials, one for each parameter. Each dial can be turned up or down, and its position is the parameter's value. Lee and Trott use a similar picture of valves in a system of pipes, each one tightened or loosened to control the flow through one connection (Lee and Trott 2023, "How language models are trained").

Before training, nobody knows where the dials should be set. So the builders set them at random. This is called random initialization: setting a network's parameters to random values before training begins. A freshly initialized network produces nonsense. Wolfram shows a small network run with several random sets of weights, and each set gives a different and useless result (Wolfram 2023, "Machine Learning, and the Training of Neural Nets"). Training then turns the dials, a little at a time, until the outputs are useful.

One Network, Many Jobs

The arrangement of neurons and layers is a design the builders choose. That design, on its own, does nothing in particular. What the network does depends on where its dials are set.

Take two copies of the same network design. Train one on photographs labeled with the animals in them, and train the other on photographs labeled as indoor or outdoor scenes. The two finished networks have the same number of neurons, the same layers, and the same connections. They differ only in their parameter values, and they do different jobs.

So when people speak of "a model," they usually mean the design together with one particular setting of all its dials. Change the settings and you have a different model, even though the design is unchanged.

No Single Dial Holds a Fact

Since everything the network learned is in its parameter values, you might expect to find the dial that stores a given fact. The search fails. What a network has learned is held as a distributed representation: it's stored across many parameters at once, so that no single one holds a fact.

A comparison with a recipe may help. The taste of a soup doesn't live in the salt or in the stock. It comes from all the ingredients in their proportions, and changing one ingredient alters the whole soup a little. In a network, each parameter takes part in a great many of the network's outputs, and each output depends on a great many parameters.

There's a practical reason for this. A network has to handle far more facts and patterns than it could if each one needed its own dial, and sharing dials among many patterns lets it fit more in.

Why You Can't Read or Edit a Model by Hand

Two consequences follow for anyone hoping to look inside a model.

• Reading. A single parameter value, taken alone, tells you nothing. It's a number with no label. Researchers who study the insides of models have to look at patterns across many parameters at once, and the work is slow. Lee and Trott state the current position bluntly: "no one on Earth fully understands the inner workings" of large language models (Lee and Trott 2023).
• Editing. Suppose a model has a fact wrong and you'd like to fix it. There's no entry to correct. Changing the parameters involved in that fact also changes everything else those parameters take part in, so a hand edit can cause damage in places you didn't expect. In practice, builders change a model's behavior by training it further on new examples, which adjusts many dials slightly.

This is a sharp contrast with a database, where a wrong entry can be found and replaced without touching any other.

Conclusion

A model's parameters are adjustable numbers, set at random to begin with and then tuned by training. The same network design does different jobs with different settings. What the network learned is spread across many parameters, which is why a trained model can't be read like a document or corrected like a database.

Key Terms

• Parameter: One of the adjustable numbers in a model whose values are set by training.
• Weight: A number that sets how much one incoming number counts toward an artificial neuron's result.
• Random initialization: Setting a network's parameters to random values before training begins.
• Distributed representation: The storing of what a network has learned across many parameters at once, so that no single one holds a fact.

References

• Lee, Timothy B., and Sean Trott. 2023. "Large Language Models, Explained with a Minimum of Math and Jargon." Understanding AI, July 27, 2023.
  • Free: Understanding AI
• Wolfram, Stephen. 2023. "What Is ChatGPT Doing … and Why Does It Work?" Stephen Wolfram Writings, February 14, 2023.
  • Free: Stephen Wolfram Writings

Introduction

Reports on AI say that a model was "trained to" do something, such as answer questions or recognize faces. Behind that phrase is a number. Training can only push a model toward something that has been turned into a score.

This reading explains how a model's wrongness is measured, how the measurements are combined into one score, and why the choice of what to score matters so much.

Scoring One Prediction

Training works through examples. A training example is one item from the training data together with the right answer for it. For a network that reads handwritten digits, an example is one image and the digit it shows.

The network is given the image and produces its prediction. The prediction is then compared with the right answer. The gap between the two is the prediction error: how far the model's prediction is from the right answer for one example.

The error is stated as a number, and it can be large or small. A network doesn't just name a digit. It gives a strength to each of the ten possible digits. If the right answer is seven and the network put nearly all its strength on seven, the error is small. If it split its strength between seven and one, the error is larger. If it put nearly all its strength on four, the error is large. A confident wrong answer scores worse than a hesitant one.

One Score for Many Examples

A score for a single example says little about the network as a whole. So the errors are combined. The loss is a single score for how wrong a model's predictions are, averaged over many examples. Wolfram describes it as the way to see "how far away we are" from the results the examples call for (Wolfram 2023, "Machine Learning, and the Training of Neural Nets"). Google's machine learning glossary gives a matching account: loss measures how far a model's predictions are from the answers attached to its examples (Google for Developers n.d., "loss").

A high loss means the network is often and badly wrong. A low loss means it's close on most examples. Training is the effort to bring this one number down.

Having a single number is what makes training possible. A procedure that adjusts billions of parameters needs a clear test of whether an adjustment helped, and the test is whether the loss went down.

What You Score Is What You Get

The builders decide what the loss measures. That decision is called the objective: what training is set up to achieve, which is usually the lowest possible loss on some chosen task.

The model gets good at whatever the objective scores, and at nothing else on purpose. Here are two supposed cases.

• A network that sorts job applications is scored on how well it matches past hiring decisions. It will learn to reproduce those decisions, including any unfairness in them. Nothing in the score rewards fairness.
• A system that recommends videos is scored on how long people keep watching. It will learn to hold attention. Nothing in the score rewards accuracy or the viewer's later satisfaction.

In both cases the training works as intended, and the trouble lies in what was chosen as the score. A measurable target often stands in for a goal that's harder to measure, and the model pursues the target.

A Low Score Has Limits

A low loss is a statement about the examples that were scored. It doesn't say how the model behaves on anything else.

A network can reach a very low loss on its training examples by fitting details that are peculiar to those examples. It will then do worse on new inputs. This is why builders also compute the loss on examples the network never trained on. Google's glossary describes this practice of holding examples back to test a model after training (Google for Developers n.d., "test set").

Even a low loss on held-back examples covers only the kinds of cases those examples include. A digit reader that scores well on neat handwriting from one country may do badly on another country's way of writing a seven. The loss never looked at that case, so it can't report on it.

Conclusion

Training needs a number to push down. Prediction error measures the gap for one example, and loss combines the errors over many examples into one score. The objective decides what that score counts, so it decides what the model becomes good at. A low loss reports on the examples that were scored and on nothing beyond them.

Key Terms

• Training example: One item from the training data together with the right answer for it.
• Prediction error: How far a model's prediction is from the right answer for one example.
• Loss: A single score for how wrong a model's predictions are, averaged over many examples.
• Objective: What training is set up to achieve, which is usually the lowest possible loss on some chosen task.

References

• Google for Developers. n.d. "Machine Learning Glossary." Accessed October 3, 2026.
  • Free: Google for Developers
• Wolfram, Stephen. 2023. "What Is ChatGPT Doing … and Why Does It Work?" Stephen Wolfram Writings, February 14, 2023.
  • Free: Stephen Wolfram Writings

Introduction

Training a large AI model is reported to take weeks of computing and a great deal of electricity. The work behind those figures is one simple move, repeated an enormous number of times.

This reading explains that move, the bookkeeping that makes it possible, and what the process leaves out.

Feeling for Downhill

A network's parameters are its adjustable numbers, and its loss is a single score for how wrong its predictions are. Every possible setting of the parameters gives some loss. Training looks for a setting where the loss is low.

Trying every setting is out of the question, because there are far too many. Training uses a local method. Picture yourself on a hillside in thick fog, wanting to reach the valley. You can't see the valley. You can feel which way the ground slopes under your feet, so you take a short step in the downhill direction and then feel again.

Gradient descent is the training method that does this with parameters. It repeatedly works out which way to nudge each parameter to lower the loss, then nudges each one slightly. Height on the hillside stands for the loss, and your position stands for the current parameter settings. Wolfram describes the same idea as following "the path of steepest descent" (Wolfram 2023, "Machine Learning, and the Training of Neural Nets").

One round of this is a training step: the network makes predictions on some examples, the loss is computed, and every parameter is adjusted a little.

Working Out Each Dial's Share

The hillside picture hides a hard problem. A person on a hill has two directions to consider. A network has billions of parameters, and the method needs to know, for each one, whether turning it up or down would lower the loss, and by how much.

Backpropagation is the procedure that supplies this. It works backward through a network, from the output toward the input, and gives each parameter its share of the responsibility for an error. It starts with the error at the output and asks which numbers coming into the last layer contributed to it and how strongly. Then it asks the same about the layer before, and so on back to the first layer.

Lee and Trott describe an algorithm that "walks backwards" through the network to estimate how much each weight should change (Lee and Trott 2023, "How language models are trained"). Google's glossary describes backpropagation as the procedure that carries out gradient descent in neural networks (Google for Developers n.d., "backpropagation").

The result is a direction and a size for every parameter. A parameter that contributed a lot to the error gets a larger nudge. One that barely took part gets a small nudge or none.

Why the Steps Are Small

The size of each adjustment is controlled by the learning rate, the setting that decides how big each adjustment is. Builders choose it, and they keep it small for two reasons.

• The slope is only a local guide. It tells you which way is downhill where you're standing. Take a giant stride and you may land on the far side of the valley, higher than where you started.
• Each step is based on a small batch of examples. A big adjustment to suit one batch would undo what earlier batches taught. Small adjustments let the effects of many examples add up.

Small steps mean that training takes a very large number of them. A single step on a large language model requires, by Lee and Trott's account, hundreds of billions of calculations, and training repeats the step over hundreds of billions of words of text (Lee and Trott 2023, "How language models are trained"). This is the main reason training large models is slow and costly.

Training also has no guarantee of finding the best possible setting. It finds a setting that works well, and builders stop when the loss no longer falls enough to justify the cost.

What Training Doesn't Do

Nobody tells the network what to look for at any point in this process. There's no step in which an engineer explains what a cat looks like or how a sentence is put together. The network gets examples and a loss, and its parameters are nudged in whichever direction lowers the loss.

Wolfram puts it this way: networks can be "trained from examples" to do a task, with no need for anyone to spell out how the task is done (Wolfram 2023, "Neural Nets"). Whatever patterns the network ends up using are the ones the nudging happened to produce. The builders find out what those are, as far as they can, by testing the result.

Conclusion

Gradient descent lowers the loss by repeated small adjustments, each in the locally downhill direction. Backpropagation makes this workable by giving every parameter its share of the error. The steps are small, so training needs a very large number of them, and the patterns the network ends up with are found by the process and not supplied by a person.

Key Terms

• Gradient descent: A training method that repeatedly works out which way to nudge each parameter to lower the loss, then nudges each one slightly.
• Training step: One round in which a network makes predictions on some examples, the loss is computed, and every parameter is adjusted a little.
• Backpropagation: The procedure that works backward through a network and gives each parameter its share of the responsibility for an error.
• Learning rate: The setting that decides how big each adjustment is.

References

• Google for Developers. n.d. "Machine Learning Glossary." Accessed October 3, 2026.
  • Free: Google for Developers
• Lee, Timothy B., and Sean Trott. 2023. "Large Language Models, Explained with a Minimum of Math and Jargon." Understanding AI, July 27, 2023.
  • Free: Understanding AI
• Wolfram, Stephen. 2023. "What Is ChatGPT Doing … and Why Does It Work?" Stephen Wolfram Writings, February 14, 2023.
  • Free: Stephen Wolfram Writings

Introduction

Accounts of AI training usually stay at the level of slogans, such as "the model adjusts its weights to reduce error." The claim is easier to judge once you've watched it happen on something small enough to follow by hand.

This walkthrough traces two training steps on an invented network with three dials. The network, the reviews, and all the numbers are made up for the purpose. The arithmetic is whole numbers only.

The Starting Point

The network's job is to guess whether a short customer review is positive or negative. It looks at three things in a review and has one dial for each.

To make a guess, the network multiplies each count by its dial, adds the three results, and reads the total. A total above zero means "positive." A total below zero means "negative." The further the total is from zero, the more confident the guess.

The starting settings are random, which is how real networks begin. They're also badly wrong. Dial A is negative, so praise words count against a review. Dial B is positive, so complaints count in its favor. Nobody will correct these by hand. Training has to do it.

For scoring, each review has a target. A positive review should total 5, and a negative review should total minus 5. The error on a review is the distance between the network's total and the target.

Walking Through the Training Steps

Step 1: Feed in one review and follow the numbers to a guess

The first review is "Great food, loved it!" It's labeled positive. It has two praise words, no complaint words, and one exclamation mark.

• Dial A: two praise words times minus 1 gives minus 2.
• Dial B: no complaint words times 2 gives 0.
• Dial C: one exclamation mark times 1 gives 1.

The total is minus 1. That's below zero, so the network guesses "negative."

Step 2: Compare the guess with the label and state the error

The label says positive, so the guess is wrong. The target for a positive review is 5 and the network's total was minus 1. The distance between them is 6, so the error on this review is 6.

The error says two things. The total was too low, and it was too low by a lot. Any adjustment should raise the total for this review.

Step 3: Assign blame to each dial

Now trace the error back to the dials, one at a time.

• Dial A contributed minus 2. It pushed the total down when it needed to go up. It gets the most blame, and it should be turned up.
• Dial B contributed nothing, because the review had no complaint words. It took no part in this error, so this review gives no reason to move it.
• Dial C contributed 1. It pushed the right way, though weakly. Turning it up would raise the total further.

This tracing is a hand version of backpropagation, the procedure that works backward from an error and gives each parameter its share of the responsibility (Lee and Trott 2023, "How language models are trained").

Step 4: Nudge each dial slightly

Each dial that took part moves one notch in the helpful direction. Dial A goes from minus 1 to 0. Dial C goes from 1 to 2. Dial B stays at 2.

One notch is a small move on purpose. Dial A is still not positive, so praise words still earn nothing. A single example hasn't fixed the network. It has shifted it a little.

Step 5: Run the review again, then train on a second review

With the new settings of 0, 2, and 2, the first review totals 2: nothing from Dial A, nothing from Dial B, and 2 from Dial C. The guess is now "positive," which is correct. The total is 3 short of the target, so the error has fallen from 6 to 3.

A second review now gets the same treatment. It's "Slow service, bad coffee!" and it's labeled negative. It has no praise words, two complaint words, and one exclamation mark.

• Dial A contributes 0.
• Dial B: two complaint words times 2 gives 4.
• Dial C: one exclamation mark times 2 gives 2.

The total is 6, so the network guesses "positive." The target for a negative review is minus 5, so the error is 11. The total needs to come down.

Blame goes to Dial B, which pushed up by 4, and to Dial C, which pushed up by 2. Dial A took no part. So Dial B moves down one notch, from 2 to 1, and Dial C moves down one notch, from 2 to 1.

Dial C has now been pulled both ways. The first review turned it up, and the second turned it back down. That's the correct outcome for a signal that appears in both kinds of review. An exclamation mark doesn't tell praise from complaint, and over many examples a dial like this would be expected to settle near zero.

With the settings at 0, 1, and 1, the second review totals 3. The guess is still "positive" and still wrong, with an error of 8 where it was 11. The first review now totals 1, with an error of 4. The second step cost the first review a little and helped the second review more.

Key Considerations

Real networks differ from this one in scale and not in kind. A large language model has billions of dials arranged in many layers, and its training runs through an enormous number of examples (Wolfram 2023, "Inside ChatGPT"). The move is the same: make a guess, measure the error, share out the blame, and adjust every dial slightly.

Two simplifications were made here. Real training sizes each nudge by how much blame the dial earned, where this example moved every dial exactly one notch. Real training also adjusts for a batch of examples at once, where this example used one review at a time.

A common mistake is to think the network stores each example it trains on. Look at what remains of the two reviews after training. The network holds three numbers. The words "Great food, loved it!" aren't kept anywhere. What's left of that review is its small effect on Dials A and C, already partly overwritten by the second review. The same is true at full scale. What training leaves behind is the adjusted settings, and those settings mix the effects of all the examples together.

Summary

Two training steps moved three dials from random settings toward useful ones, and the combined error on the two reviews fell at each step.

1. The start row scores both reviews with the random settings. The second review totals 5 there, which is 10 away from its target.
2. The step on review 1 helped review 1 and slightly hurt review 2, because raising Dial C raised the total for both.
3. The step on review 2 reversed that change to Dial C and lowered Dial B. The combined error fell again.
4. The network is still wrong about review 2. Many more steps would be needed before Dial A turns positive and Dial B turns negative.

References

• Lee, Timothy B., and Sean Trott. 2023. "Large Language Models, Explained with a Minimum of Math and Jargon." Understanding AI, July 27, 2023.
  • Free: Understanding AI
• Wolfram, Stephen. 2023. "What Is ChatGPT Doing … and Why Does It Work?" Stephen Wolfram Writings, February 14, 2023.
  • Free: Stephen Wolfram Writings