So how does image recognition work? A computer treats a photo as a grid of numbers, runs those numbers through layers that learn to respond to more and more complex visual patterns, and ends with a classifier that scores each possible label and outputs the most likely one. A colour photo 248 pixels wide and 400 tall, for example, is 248 × 400 × 3 integers from 0 to 255, one per pixel per red, green and blue channel, or 297,600 numbers in total. What separates modern image recognition tech from older systems is that it learns which patterns in the array matter from labelled examples, rather than having engineers write those patterns down by hand.
What is image recognition and how does it work?
Image recognition is the task of analysing an image and assigning what it contains to one of a set of categories. It sits inside the wider field of computer vision, which covers how computers gain high-level understanding from digital images, and narrowly means classifying a detected object into categories.
A machine does not see an image as a whole. It sees only the pixel values. The practical consequence is that you cannot describe a cat in code; instead you collect many examples of each class and let a learning algorithm work out what each class looks like, which the Stanford CS231n course notes call the data-driven approach.
The pipeline in five steps
- Represent the image as numbers. The photo becomes a height × width × channels array of pixel intensities; LeNet-5, for example, scaled its inputs so the mean was roughly zero and the variance roughly one, which speeds up training.
- Extract low-level features. Early convolutional layers apply small filters across the image and produce feature maps that respond to edges and simple textures.
- Combine them into higher-level features. Deeper layers combine those responses into shapes, object parts and eventually whole objects.
- Score each class. Fully connected layers (or an equivalent head) turn the final features into a score or probability for every label.
- Pick the label, or refuse. The highest score becomes the prediction; a system can also reject inputs whose top two scores are too close, as LeCun et al. did, because in many applications rejection performance matters more than raw error rate.
Their weights are learned from labelled images by minimising a loss that measures how far the output is from the correct label, with gradients computed by back-propagation from the output back to the input.
Why are convolutional neural networks used for image recognition?
Convolutional neural networks (CNNs) are used because they build the structure of images into the network itself, which gives them far fewer free parameters than a generic network fed raw pixels and some tolerance to shift, scale and distortion. A fully connected first layer looking at an image would need tens of thousands of weights for an image of several hundred pixels and a first layer of one hundred hidden units, which demands a larger training set and, worse, has no built-in tolerance for an object shifting a few pixels.
LeCun and colleagues, in their 1998 paper on document recognition, described the three ideas that fix this: local receptive fields (each unit looks at a small patch), shared weights (the same filter is reused at every position) and subsampling (later pooling) to give some tolerance to shift, scale and distortion. Units that share one set of weights form a feature map, and a convolutional layer holds several feature maps so that several features are extracted at each location. Scanning one small unit across the image this way is mathematically a convolution, which is where the name comes from.
Weight sharing has a side effect that matters in practice: it cuts the number of free parameters and narrows the gap between training error and test error. LeNet-1 needed about 100,000 multiply-add operations per image yet had only about 2,600 free parameters. Because all of those weights are learned, a CNN effectively builds its own feature extractor, and this is the property that distinguishes CNNs from traditional machine learning methods that rely on manual feature engineering.
What the network learns has a recognisable shape. Yosinski et al. report that networks trained on natural images tend to learn first-layer features resembling Gabor filters and colour blobs, and that these early features are general across datasets and tasks while later layers become specific to the task.
Can image recognition work without neural networks?
Yes, and for years it did: the earlier recipe was to compute hand-designed features and feed them to a conventional classifier. The clearest example is the 2005 person detector by Dalal and Triggs, which divides an image window into small cells, accumulates a histogram of gradient orientations in each, contrast-normalises the cells over larger blocks, and passes the combined vector to a linear support vector machine (SVM). On person detection, Dalal and Triggs report that HOG reduced false positive rates by more than an order of magnitude relative to the best Haar wavelet-based detector they compared against. Classic OCR worked in a similar way, decomposing characters into lines, closed loops and intersections and comparing those with stored character prototypes. Our explainer on feature extraction for computer vision goes further into these hand-built descriptors.
Learning machines operating directly on pixels proved able to replace hand-crafted feature extraction. In the LeCun et al. digit experiments, plain fully connected networks reached 4.5–4.7% test error, while LeNet-5 reached 0.95% and then 0.8% once trained on distorted copies of the training images. The authors’ conclusion was that increasing the role of learning had consistently improved recognition systems. Accuracy on several computer vision benchmarks has since moved further with deep learning, and the best current algorithms for recognition are based on CNNs.
For open-ended recognition of natural images, we would not start with hand-crafted features.
How much training data does an image recognition model need?
The amount varies from task to task, but the structure of the training data does not: a set of images each labelled with one of K classes, a learning step that fits the model to them, and an evaluation on images the model has never seen. LeCun et al. stress that performance on the training set is of little interest; what matters is the error rate in the field, estimated on a test set kept separate from training.
For a sense of scale at the small end, Kili Technology’s training guide states that effective training requires at least 200 labelled images, combined with augmentation such as rotation, blurring and greyscale conversion to prevent overfitting. At the other end, the ImageNet challenge covers hundreds of object categories and millions of images, and the MNIST handwritten digit set used by LeCun et al. had 60,000 original training patterns.
Three factors move the requirement up or down:
- Augmentation. Generating altered copies multiplies the effective data. LeCun et al. added 540,000 distorted versions of their 60,000 digits and cut test error from 0.95% to 0.8%. Apple describes adding colour jitter, flips, blur, compression artefacts and even synthetic face masks to its face model, introduced progressively as training proceeds.
- Pretraining. Starting from a backbone already trained on a large dataset, for years by default an ImageNet-trained CNN, reuses early features that are general across datasets and tasks.
- Architecture. Roboflow’s comparison notes that CNNs cope with limited data because their built-in assumptions (locality, weight sharing) guide learning, while vision transformers need much larger datasets or strong pretraining.
The check that tells you whether you have enough is a comparison of training accuracy against validation accuracy: a large gap means the model has memorised its training images rather than generalised. Apple notes that a face model must perform consistently across age, gender, ethnicity and skin tone, which starts with diverse and balanced data collection and continues through failure analysis.
Have vision transformers replaced CNNs for image recognition?
No: modern workflows increasingly use both, and the best choice depends on data size, hardware limits and how much global context the task needs. A vision transformer (ViT) splits an image into fixed-size patches, turns each patch into a token with a positional embedding, and uses self-attention so every patch can interact with every other patch from the first layer. Dosovitskiy et al. (ICLR 2021) showed that a pure transformer on image patches, pre-trained on large amounts of data, attains excellent results compared with state-of-the-art convolutional networks.
Then the results start to disagree. Liu et al. write that ViTs quickly superseded ConvNets as the leading classification models, yet their pure-ConvNet ConvNeXt family reached 87.8% ImageNet top-1 accuracy and outperformed Swin Transformers on COCO detection and ADE20K segmentation. Smith et al. at Google DeepMind found that NFNet ConvNets pre-trained at web scale match the reported performance of ViTs at comparable compute budgets. The “Battle of the Backbones” comparison found that CNNs pretrained in a supervised way on large datasets still performed best on most tasks among the models it tested. On the other side, a ViT survey reports very good ViT performance on ImageNet, COCO and ADE20K. We read the weight of evidence as: at equal data and compute, neither family wins by default.
| Factor | CNNs | Vision transformers |
|---|---|---|
| How they see the image | Compose local filters layer by layer into global understanding | Global self-attention across patches from the first layer |
| Small or costly-to-label datasets | Handle limited data well thanks to built-in assumptions | Struggle without much larger datasets or strong pretraining |
| Large data, strong pretraining | Competitive at scale (ConvNeXt, NFNet) | Higher accuracy reported with enough data and compute |
| Inference speed | Fast: convolution is heavily optimised on GPUs, NPUs and mobile chips | Slower: attention cost grows quadratically with patch count |
| Plain detection and segmentation | ImageNet-trained CNNs were the default backbone | Vanilla ViT has difficulties; hierarchical variants reintroduce ConvNet priors |
Hybrids blur the line further: DETR uses a CNN backbone such as ResNet for feature maps and a transformer encoder-decoder for global relationships between objects.
What hardware do you need for real-time image recognition?
Real-time speed depends on both the model architecture and the image size: CNNs run faster, and high-resolution images increase processing time significantly. Convolution is heavily optimised on GPUs, NPUs and mobile processors, which is why CNNs achieve low latency and high throughput, while transformers slow down as resolution (and so the number of patches) grows. LeCun et al. already noted in 1998 that convolutional networks suit hardware implementation because of their regular structure and small weight memory.
Apple reports that its Photos face-embedding network runs end to end on the Apple Neural Engine and completes in under 4 ms on recent iOS hardware. Apple describes trading accuracy against computational cost at every stage of that network, using blocks inspired by MobileNetV3.
Our recommendation would be to fix the latency target and input resolution first, then measure candidate models on the actual target device, because a published accuracy figure says nothing about frames per second on your hardware.
What is the role of Core ML in image recognition?
Core ML is Apple’s framework for running trained machine-learning models on Apple devices, so its role in image recognition is deployment, not learning. A recognition model is trained elsewhere, converted, and then executed on the device’s CPU, GPU or Neural Engine. On-device execution keeps recognition private, which is the reason Apple gives for running its Photos face recognition entirely locally.
Image recognition examples you rely on every day
Reading text (OCR). Optical character recognition converts images of typed, handwritten or printed text into machine-encoded text, from scanned documents, phone photos, street signs or broadcast subtitles. It drives data entry from passports, invoices, bank statements and receipts, and makes printed text searchable and editable. By 1998 a large proportion of commercial OCR systems already used multilayer neural networks trained with back-propagation, and the CNN-based cheque reader described by LeCun et al. had been fielded in US banks since June 1996, reading millions of cheques. Modern engines such as Tesseract and OCRopus use neural networks that recognise whole lines of text instead of single characters. OCR is still not perfect: one study of 19th- and early 20th-century newspaper pages found character-level accuracy for commercial software ranged from 81% to 99%.
Tagging people in photos. Apple Photos computes face embeddings with a deep CNN and clusters face and upper-body embeddings to group the same person across a library without anyone labelling the photos.
Visual search. Jing et al. describe Pinterest’s visual search and report that a small engineering team could build and run a large-scale system with widely available tools, and that recommendations powered by it improved user engagement.
Specialist domains. CNNs have improved disease detection in cardiology, pathology, dermatology and radiology images. In agriculture, an open-source vision transformer model detects strawberry diseases with 98.4% reported accuracy. Applications also extend to autonomous vehicles and security screening. How these pieces fit into larger vision systems is covered in our piece on computer vision and image understanding.
Benchmark accuracy is not field accuracy
ImageNet and MNIST are standard benchmarks for the field. MNIST is a widely used test of handwritten digit recognition, and the ImageNet challenge has run annually since 2010, including comparisons between machine and human accuracy. He et al.’s residual networks, up to 152 layers deep, reached 3.57% ImageNet test error as an ensemble and won the 2015 classification task, and CNN performance on ImageNet is now described as close to human.
LeCun et al. call the error rate in the field, estimated on a test set disjoint from training, the more relevant measure, and the cheque-reading case shows what a real target looks like: the bank’s threshold was reading 50% of cheques with under 1% error and sending the rest to human operators. A recognition system is defined by its error and rejection behaviour on the inputs it will actually see, not by its leaderboard score. When the question shifts from how recognition works to making it hold up on your own cameras, documents or production line, that is the work TechnoLynx’s Computer Vision R&D takes on.
