Compute & chips
Why it matters
An AI chip is only as fast as the data reaching it. High-bandwidth memory stacks memory chips on top of each other right beside the processor, so the model's parameters can stream in quickly.
Few companies make it, so HBM supply often decides how many AI chips can ship. When a new GPU is announced, its memory size and bandwidth are among the figures that matter most for running large models.
Related terms
- GPU
- A GPU (graphics processing unit) is a chip first built to speed up graphics. Its strength at parallel math also makes it the main chip for training and running AI models, which is why AI clusters are counted in GPUs.
- Parameters
- The adjustable numbers inside a model that training sets; a rough measure of the model's size.
- AI data center
- A building full of GPUs, power and cooling, used to train and run AI models. Colossus is xAI's.