What Is Computer Vision? How AI “Sees” and Interprets Images

Your phone unlocks by recognizing your face. A self-driving car spots a pedestrian stepping off a curb. A medical AI flags a suspicious spot on an X-ray. All of these rely on computer vision — the branch of AI that teaches machines to make sense of images and video.

What Computer Vision Actually Does

Computer vision is the field of AI focused on extracting meaningful information from images and video, the same way natural language processing extracts meaning from text. A photo, to a computer, starts out as nothing more than a grid of numbers — each pixel’s color and brightness. Computer vision is the set of techniques that turn that raw grid of numbers into useful understanding: “this is a cat,” “this is a stop sign,” “this person is 32 years old,” or “this X-ray shows a fracture.”

Common Computer Vision Tasks

  • Image classification: labeling an entire image with a category, like “dog” or “beach.”
  • Object detection: finding and drawing boxes around specific objects within an image, and labeling each one.
  • Image segmentation: identifying the exact pixels that belong to each object, useful for things like a car’s self-driving system distinguishing the road from the sidewalk.
  • Facial recognition: identifying or verifying a specific person’s identity from their face.
  • Optical character recognition (OCR): reading printed or handwritten text out of an image.
  • Pose estimation: mapping the position of a person’s body joints, used in fitness apps and motion capture.

How It Works, at a High Level

Modern computer vision is built almost entirely on neural networks trained on huge labeled datasets — millions of images, each tagged with what it contains. For years, the dominant architecture was the convolutional neural network (CNN), which scans an image in small patches, learning to recognize simple patterns like edges and textures in its early layers, then combining those into more complex shapes and eventually full objects in its later layers.

More recently, the same transformer architecture that powers large language models has been adapted for vision too, in what’s called a vision transformer. And the two fields are increasingly merging: multimodal AI systems like GPT-4o and Gemini process images and text through a shared model, letting you show a picture and ask a question about it in the same conversation.

Real-World Applications

  • Autonomous vehicles: detecting lanes, pedestrians, other cars, and traffic signs in real time.
  • Healthcare: flagging tumors, fractures, or other abnormalities in medical scans for a radiologist to review.
  • Retail and manufacturing: spotting defective products on an assembly line, or tracking inventory on shelves.
  • Security: facial recognition for access control, or detecting unusual activity in camera footage.
  • Agriculture: identifying crop disease or estimating yield from drone or satellite imagery.

Where It Still Struggles

Computer vision systems can be fooled by conditions that barely affect human perception: unusual lighting, partial occlusion, or images deliberately altered in subtle ways designed to trick the model (known as adversarial examples). Facial recognition systems have also been documented to perform less accurately on some demographic groups than others, a form of AI bias tied directly to what the training data did or didn’t include. And like any AI model, a vision system is only as good as the data it was trained on — a self-driving system trained mostly on sunny-day footage will struggle more in heavy snow.

The Bottom Line

Computer vision is what lets machines interpret the visual world the way NLP lets them interpret language. From unlocking your phone to catching manufacturing defects to assisting doctors, it’s quietly embedded in far more products than most people realize — and as multimodal AI models mature, the line between “seeing” and “understanding” is only getting thinner.