Why Does AI Need So Much Computing Power?

You type a short question into an AI chatbot:

Why is the ocean salty?

A few seconds later, several paragraphs appear.

From the user's side, the job seems tiny. A sentence went in; a sentence came out.

So why are AI companies building data centres filled with thousands of specialised processors? Why do powerful AI models need GPUs, enormous amounts of memory, high-speed networking and so much electricity just to work with text, images or video?

The main reason is scale.

A modern AI model may contain an enormous collection of numerical values called parameters. To produce an answer, the system repeatedly combines the input with those parameters through large mathematical operations. Training is even more demanding because the model must perform these calculations again and again across huge datasets while continually adjusting those parameters.

And raw calculation is only part of the problem.

The system also has to move vast amounts of data between processors and memory, keep many accelerators synchronised and serve potentially thousands of users at once.

What looks like a simple chatbot response is the visible result of a large computational machine working underneath.

What Does "Computing Power" Actually Mean?

It is easy to imagine computing power as simply having a faster processor.

For AI, several resources matter at the same time.

Arithmetic performance

AI models perform huge numbers of operations, particularly matrix multiplications.

A matrix is essentially a rectangular collection of numbers. Neural networks repeatedly multiply and combine large matrices as information passes through their layers.

Specialised AI hardware is designed to perform these calculations extremely quickly.

Memory capacity

The model's parameters must be stored somewhere.

Larger models can require tens or hundreds of gigabytes just for their weights, before accounting for other working memory.

Memory bandwidth

It is not enough to store the parameters.

They have to be moved quickly between memory and the processor.

An accelerator may be capable of extraordinary arithmetic performance but still sit idle if data cannot arrive quickly enough.

Communication between processors

Very large models may not fit on one accelerator.

Training may therefore be distributed across hundreds or thousands of devices that constantly exchange information.

Electricity and cooling

Every calculation consumes energy, and much of that electrical energy ultimately becomes heat.

Large AI systems therefore need power delivery and cooling infrastructure as well as processors.

So when people say AI requires enormous computing power, they usually mean an entire system of compute, memory, data movement and networking, not simply a fast chip.

The Model Itself Can Be Enormous

A neural network learns by adjusting numerical parameters.

You can think of these parameters as millions or billions of adjustable values that determine how information is transformed as it moves through the network.

Large language models use Transformer architectures containing many layers of these operations. Google describes modern LLMs as models containing very large numbers of parameters and notes that larger Transformers have historically produced improved performance, although larger models also require more resources.

Consider a hypothetical model with 70 billion parameters.

If every parameter were stored using a 16-bit number, each would require two bytes.

The weights alone would therefore occupy roughly:

70 billion × 2 bytes = 140 GB

That does not include:

• temporary calculations
• attention caches
• input and output data
• training gradients
• optimizer states
• communication buffers

A model can therefore exceed the memory capacity of a single accelerator even before it starts doing useful work.

Every Layer Performs Large Amounts of Mathematics

Parameters are not merely sitting in memory waiting to be looked up.

They participate in calculations.

When text enters a Transformer, each token is represented numerically and passed through many successive layers.

Inside those layers are operations involving:

• attention
• matrix multiplication
• nonlinear transformations
• normalization
• other numerical processing

The original Transformer architecture was designed to make much of this computation parallelizable, which was one reason it could make effective use of modern accelerators.

But parallelizable does not mean cheap.

Imagine a model containing dozens of large layers.

One token must pass through those layers.

Then another token.

Then another.

Now multiply that work by a long prompt, millions of users and potentially billions or trillions of training tokens.

Small calculations become enormous through repetition.

Attention Becomes Expensive as Context Grows

One of the defining mechanisms in Transformers is self-attention.

Attention lets the model relate different parts of the input to one another.

In a sentence such as:

The trophy didn't fit inside the suitcase because it was too large.

the model needs to consider relationships between it, trophy, suitcase and the surrounding words.

In standard full self-attention, every token can interact with every other token.

If you double the number of tokens, you do not merely double the number of possible token-to-token relationships.

For basic full attention, that relationship matrix grows approximately with the square of the sequence length.

That is one reason long context windows can become computationally and memory intensive.

Modern architectures can reduce this burden using approaches such as sliding-window, chunked or other specialised attention mechanisms, so not every current model pays the full quadratic cost across its entire context.

But longer context still means more information has to be stored and processed.

Training Is Much Harder Than Simply Running the Model

There are two very different stages in AI:

training

and

inference.

Inference is what happens when you use a trained model.

Training is how that model acquired its parameters in the first place.

During training, the system repeatedly performs a process similar to this:

Input data

↓

Forward pass through the model

↓

Generate prediction

↓

Compare prediction with target

↓

Calculate error

↓

Run backward pass

↓

Calculate gradients

↓

Update parameters

↓

Repeat

The backward pass is especially important.

The system has to determine how millions or billions of parameters contributed to the error so they can be adjusted.

And this does not happen once.

It happens across enormous quantities of training data.

Bigger Models Also Need More Training Data

One might imagine that building a better AI model means simply adding more parameters.

Research has shown that the relationship is more complicated.

Early scaling-law work found predictable relationships between model performance, model size, dataset size and training compute.

Later work from DeepMind on compute-optimal language-model training found that increasing model size without increasing the amount of training data could be inefficient. In its experiments, model size and the number of training tokens needed to grow together for efficient use of a fixed compute budget.

That creates an expensive combination:

larger model

more training data

more training steps

=

far more computation

This is why training a frontier-scale model can be such a different undertaking from training an ordinary machine-learning model on a laptop.

Why GPUs Became So Important

A normal CPU is excellent at performing a wide variety of tasks.

But neural networks contain many calculations that can be performed at the same time.

GPUs were originally designed to process large numbers of graphics operations in parallel. That same parallel structure turns out to be extremely useful for neural-network mathematics.

NVIDIA identifies parallel processing as one of the fundamental reasons GPUs are well suited to AI training and inference.

Imagine multiplying millions of pairs of numbers.

A CPU with a relatively small number of powerful cores may process a limited number of operations simultaneously.

A GPU contains many more execution units designed to handle highly parallel workloads.

Modern AI accelerators go further by adding specialised hardware for matrix multiplication.

Google's TPU architecture, for example, includes dedicated matrix-multiply units, high-bandwidth memory and specialised inter-chip connections designed specifically for machine-learning workloads.

Memory Can Matter as Much as Arithmetic Speed

Here is an unintuitive problem.

Suppose an AI processor can perform calculations faster than its memory can supply the necessary numbers.

The processor has to wait.

That means an AI workload can become memory-bandwidth limited rather than compute limited.

Large language-model inference often requires repeatedly reading vast quantities of model weights.

Even if the mathematical operation applied to each weight is relatively simple, moving the numbers into the processing units takes time and energy.

This is why modern AI accelerators are paired with high-bandwidth memory, commonly abbreviated HBM.

The goal is not only:

calculate faster

but also:

keep the processors continuously supplied with data.

Why Can't One Huge GPU Do Everything?

Eventually, the model becomes too large or training becomes too slow for one device.

The workload then has to be divided.

There are several ways to do that.

Data parallelism

Multiple GPUs each receive different batches of training data.

Each device performs its own calculations, and their gradients are then combined so that all model copies remain synchronised.

PyTorch's DistributedDataParallel system, for example, synchronises gradients across processes during training.

Model sharding

Instead of placing the entire model on every GPU, parts of it can be distributed between devices.

PyTorch's Fully Sharded Data Parallel system can divide model parameters, gradients and optimizer state across devices, reducing the amount each accelerator must store.

Other forms of parallelism

Large systems can also divide work by:

• model layers
• tensor operations
• sequences
• experts in mixture-of-experts architectures

The exact strategy depends on the model.

But splitting work creates another requirement:

the devices must communicate extremely quickly.

Thousands of Fast Chips Are Useless If They Cannot Talk Quickly Enough

Imagine 1,000 workers solving parts of the same problem.

If every worker has to spend most of the day waiting for information from the others, adding workers eventually stops helping.

AI clusters face the same problem.

Distributed training requires processors to exchange:

• gradients
• activations
• parameters
• intermediate results

That makes interconnect bandwidth critical.

Modern accelerator systems therefore include high-speed links between chips and high-performance data-centre networking.

Google's TPU specifications, for example, list both high-bandwidth memory and dedicated inter-chip interconnect bandwidth as core architectural characteristics.

AI performance is therefore partly a networking problem.

Inference Is Cheaper Than Training, but It Is Not Free

Once training is complete, the model's parameters no longer need to be constantly adjusted.

That makes inference much less computationally demanding than training.

But a large model can still be expensive to run.

For a chatbot, the system must first process your prompt.

Then it generates an answer.

And language generation introduces another complication:

tokens are normally generated sequentially.

If the model has produced:

Artificial intelligence requires

it must use that sequence when predicting the next token.

Then the newly generated token becomes part of the context for the following prediction.

So generating 500 tokens involves hundreds of successive model steps.

You cannot simply calculate every word independently at the same time.

Why AI Responses Use a KV Cache

Without optimisation, a Transformer would repeatedly recompute information about all previous tokens every time it generated a new one.

That would waste huge amounts of work.

Autoregressive language models therefore commonly use a key-value cache, usually called a KV cache.

It stores attention-related information from previous tokens so that it can be reused during subsequent generation.

Hugging Face's Transformer documentation explains that caching avoids recalculating these earlier key and value representations and can substantially improve generation performance.

But caching creates another trade-off.

The KV cache consumes memory.

As conversations become longer, that memory requirement can grow substantially and become a bottleneck of its own.

Once again, AI performance becomes a balance between computation and memory.

Serving Millions of Users Changes the Problem Again

A single AI request may be manageable.

A popular AI service has to handle many requests simultaneously.

Suppose thousands of users are asking questions at once.

The provider must decide:

• which requests go to which accelerators
• how several requests can be batched together
• how much memory each conversation requires
• how long each user has already waited
• whether requests use different model sizes
• how much output each user is generating

This becomes a large scheduling problem.

Running AI efficiently at scale therefore involves sophisticated software around the model itself.

The model may be the most visible part of the system, but serving infrastructure determines whether it can be delivered economically to millions of people.

Images and Video Can Demand Even More Work

Text is not the only input modern AI processes.

An image can be divided into many visual patches or represented in a compressed latent form.

Higher-resolution images contain more visual information.

Video adds an entire time dimension.

A system working with video may need to understand:

width × height × frames × time

instead of simply a sequence of text tokens.

Multimodal models may also process combinations of:

• text
• images
• audio
• video

Each additional modality creates more data and potentially more interactions for the model to process.

That is one reason advanced multimodal AI can demand substantial computing resources.

Engineers Spend Enormous Effort Making AI Use Less Compute

The story is not simply that models get larger forever.

A major part of AI engineering is about reducing unnecessary computation.

Lower-precision numbers

Not every operation needs full 32-bit floating-point precision.

Mixed-precision training can perform suitable operations using 16-bit formats while retaining higher precision where necessary.

Smaller numbers also mean less data has to move through memory.

Quantization

For inference, model weights can sometimes be represented with even fewer bits.

Quantization can reduce model size and memory requirements and improve inference speed, although the trade-off with accuracy depends on the model and technique.

Caching

KV caches prevent the model from recomputing previous attention information during generation.

Batching

Multiple user requests can sometimes be processed together, improving accelerator utilisation.

Sharding and parallelism

Large models can be divided across hardware efficiently rather than forcing every device to hold everything.

Better architectures

Researchers continually develop more efficient attention mechanisms, sparse models, mixture-of-experts systems and other techniques intended to use computation more selectively.

So AI models are becoming more capable while engineers simultaneously work to reduce the cost of each useful result.

Does Every AI System Need a Data Centre?

No.

"Artificial intelligence" covers a huge range of systems.

A small image classifier, recommendation model or speech-recognition system can run comfortably on a smartphone.

Modern laptops and phones increasingly include:

• GPUs
• NPUs
• neural-processing accelerators

Smaller language models can also run locally, particularly when they are quantized.

The extraordinary computing requirements usually associated with AI come from a particular combination:

very large models + large datasets + long contexts + demanding modalities + many simultaneous users.

AI itself does not inherently require a warehouse full of GPUs.

Scale does.

Why Not Just Make the Models Smaller?

Researchers do.

But model size is only one dimension.

A small specialised model may outperform a huge general-purpose model on a narrow task.

For example, an AI trained only to classify a small set of manufacturing defects does not need to know how to:

• write poetry
• solve programming problems
• explain physics
• analyse photographs
• translate languages

General-purpose AI models are computationally demanding partly because they are expected to perform an unusually wide range of tasks.

The more capabilities a single system is expected to support, the more difficult it becomes to compress the entire problem into a tiny model without losing useful performance.

The Simplest Way to Understand AI's Compute Problem

AI needs so much computing power because enormous amounts of relatively simple mathematics are repeated at enormous scale.

For training:

Large model

×

Huge training dataset

×

Forward calculations

×

Backward calculations

×

Many training steps

=

Enormous computation

For everyday use:

Large model

×

Prompt length

×

Generated tokens

×

Number of users

=

Large ongoing inference workload

And surrounding both calculations are additional constraints:

memory

memory bandwidth

chip-to-chip communication

electricity and cooling

A chatbot answer may contain only 300 words.

But those 300 words can be the endpoint of an enormous stack of numerical operations running across specialised hardware.

That is the central reason AI requires so much computing power.

The individual mathematical operations are not mysterious.

There are simply an extraordinary number of them, performed on enormous models, repeated across enormous amounts of data, at a scale where even moving the numbers around becomes one of the hardest engineering problems.