Understanding Computer Vision and Pattern Recognition

How does face recognition work in practice? Detect, align, embed and match, verification vs identification, SIFT/HOG vs CNNs, and defect detection.

Understanding Computer Vision and Pattern Recognition
Written by TechnoLynx Published on 24 Jul 2024

Here is how face recognition works, in short. A detector finds the face, an alignment step rotates and scales it into a standard crop, a deep network turns the crop into a fixed-length vector (a template, or embedding), and a comparator scores that vector against one enrolled template or a whole gallery, then applies a threshold. That chain is a clear demonstration of pattern recognition, which extracts information from signals using statistical methods and neural networks. Automated inspection ends in the same kind of decision, with pass or fail in place of match or no match.

How does face recognition work, step by step?

Face recognition runs four stages in sequence (detection, alignment, embedding, matching), and each stage hands a narrower, more normalised input to the next. Our walkthrough of how the facial recognition pipeline actually works covers the engineering of each stage and where it fails; this section looks at what each stage contributes as pattern recognition.

Detection and alignment

Face detection identifies the face regions in the image. MTCNN (2016) does this with a cascade of three convolutional networks: a proposal network generates candidate windows, a refinement network rejects most of the false candidates, and an output network places five landmarks (both eyes, the nose and the two mouth corners). Its authors report 16 frames per second on a 2.60 GHz CPU and 99 frames per second on an Nvidia Titan Black GPU. BlazeFace reports 98.61% average precision at 0.6 ms on an iPhone XS, against 97.95% and 2.1 ms for a MobileNetV2-SSD baseline with the same anchor scheme.

Alignment normalises pose and scale using those landmarks. BlazeFace’s six keypoints let the pipeline estimate the face’s roll angle and pass a rotated, centred, scale-normalised crop downstream, so later stages do not need to cope with large shifts and rotations. According to the National Academies (2024), contemporary recognisers work on faces cropped to 112 × 112, 128 × 128 or 256 × 256 pixels, a resolution too coarse to show hair or skin pores.

Embedding: where the recognition actually happens

The embedding stage turns an aligned face into a fixed-length vector. FaceNet (2015) trains a deep convolutional network to map face images into a compact Euclidean space where distance corresponds directly to face similarity, using triplets of matching and non-matching face patches. It needs only 128 bytes per face and reported 99.63% accuracy on the Labeled Faces in the Wild dataset. ArcFace instead adds an additive angular margin to the softmax loss, maximising class separability and enhancing discriminative power.

Training such networks is almost always supervised. The National Academies (2024) notes that strong recognisers are trained on identity-labelled images from millions of people, with labels that must be correct, and that much of a developer’s intellectual property lies in dataset curation, architecture, loss functions and tuning. Templates are not one-way hashes. They can be partly reversed into something resembling the face and can leak attributes such as sex, so they must be protected from disclosure.

The matching stage compares two templates and returns a similarity score, usually computed from Euclidean distance, Manhattan distance or the angle between the vectors. These scores are not probabilities or “percentage matches”: each developer emits scores on its own interval, with its own distribution, so there is no universal value at which a score counts as strong. A threshold tuned for one vendor’s scores therefore tells you nothing about another vendor’s.

Some gallery search algorithms do not work as a series of one-to-one comparisons and scale sublinearly with gallery size; the National Academies (2024) describes one algorithm in NIST’s Face Recognition Vendor Test that searches a 12-million-entry database in a few tens of milliseconds on a commodity CPU.

Face verification and face identification answer different questions

Face verification is a one-to-one check: is this person who they claim to be? Face identification is a one-to-many search: who in this gallery, if anyone, is this? Access control is an authentication scenario of this kind. Identification builds a template from a probe image, searches the enrolled gallery, and a successful search yields the metadata attached to the gallery entry, such as a name.

The two modes are measured with different error rates, defined in NIST’s evaluations.

  Verification (1:1) Identification (1:N, with threshold) Investigation (1:N, no threshold)
Question asked Is this the claimed person? Is this person in the gallery, and who? Which gallery faces look most like this one?
Compared against One enrolled template The enrolled gallery The enrolled gallery
Output Match or no match Candidate(s) above threshold, or none Ranked candidate list for a human reviewer
False-accept measure FMR: share of impostor comparisons at or above the threshold FPIR: share of non-mated searches returning one or more candidates above threshold Not thresholded; a human makes the call
False-reject measure FNMR: share of mated comparisons below the threshold FNIR: share of mated searches not returning the mate above threshold FNIR: share of mated searches where the true mate is not at rank 1

NIST defines FNMR at a threshold set to achieve a specified FMR, and lists an FNMR of 0.025 or less at an FMR of 0.000001 on its Visa-Border dataset. Kairos’s documentation uses the terms false accept rate and false reject rate for these two errors.

Who sets the threshold, and why it moves

The system owner sets the decision threshold, often on the provider’s recommendation, and the right value depends on the image statistics and on the relative cost of a false match versus a false non-match. Raise it and genuine matches start being rejected; lower it and different people start matching. The National Academies (2024) gives access control as the case where low false positives win: almost every attempt comes from the legitimate account holder, and a false positive rate of 1 in 10,000 is typical.

NIST makes the operating choice explicit in its demographic analysis: it fixes each algorithm’s threshold to give an FPIR of 0.002 on one reference group and applies that threshold to every group, because a subject’s group is not usually known at capture. The variation matters. False positive rates can differ by one, two or three orders of magnitude between demographic groups depending on the algorithm, while variation in false negatives is much smaller.

False negatives depend strongly on image quality. With cooperative capture, the National Academies (2024) reports that leading 2023 algorithms searching a 12-million-identity mugshot database return the correct entry in 99.9% of searches. With low-quality probes and a long ageing gap, true matches score as low as false ones and no clear answer exists. Criminal investigations sidestep the threshold altogether: the system returns a ranked list whether or not the subject is enrolled, and a human reviews the candidates in every search. Why uncontrolled camera footage sits at the hard end of that range is covered in our piece on CCTV face recognition in production. For system design, IEEE 2945-2023 specifies a general architecture with functional, performance and security requirements for face recognition systems.

Is pattern recognition in artificial intelligence the same as image recognition?

No. Pattern recognition covers signals in general, and image recognition is one of its applications. Pattern recognition extracts information from signals in general, mainly through statistical methods and neural networks, and a large share of the field applies those methods to images. Optical character recognition, speech recognition and facial recognition are all tasks that pattern recognition underpins. Within the field, statistical methods reason with probability and inference, while structural methods find patterns in spatial or relational arrangements.

Computer vision covers the whole chain from image acquisition to understanding. A vision system acquires images from a sensor, pre-processes them (resampling, noise reduction, contrast enhancement), extracts features, decides which regions matter, and only then runs high-level steps. Image recognition, meaning classifying a detected object into categories, is one of those high-level steps, next to pose estimation, image registration and the final application decision. That final decision takes the form the application needs: match or no match in recognition, pass or fail in automated inspection, or a flag for human review. Put simply, face recognition is a computer vision pipeline that demonstrates pattern recognition.

From SIFT, SURF and HOG to learned CNN features

Classic feature methods and convolutional networks both turn image regions into feature vectors. SIFT and HOG are engineered representations built from oriented-gradient histograms, while a CNN learns its features in training.

SIFT: sparse keypoints with invariance designed in

Lowe’s SIFT (2004) finds keypoints at extrema of a difference-of-Gaussian scale space, refines their location, assigns each one an orientation from local gradients, and describes the surrounding patch relative to that orientation and scale, which makes it invariant to rotation and scale. The descriptor is a 4 × 4 grid of orientation histograms with 8 bins each, giving a 128-element vector. Matching uses the nearest neighbour by Euclidean distance; rejecting matches whose distance ratio to the second-nearest neighbour exceeds 0.8 removes 90% of false matches while discarding under 5% of correct ones. Objects are then confirmed by clustering consistent matches with a Hough transform and fitting affine pose parameters by least squares. The limits are stated plainly in the paper: textured planar surfaces are recognised reliably up to about 50 degrees of rotation in depth, 3D objects only up to about 30 degrees.

SURF: a faster approximation of the same idea

Bay, Tuytelaars and Van Gool’s SURF (2006) uses a Hessian-based detector and integral images for speed, and describes each keypoint with Haar-wavelet responses in 4 × 4 sub-regions, giving 64 dimensions instead of 128. In the authors’ own object recognition test, SURF reached 82.6% and SIFT 78.1%. OpenCV exposes both the 64-element basic and 128-element extended descriptor, and a Hessian threshold that trades keypoint count against strength.

HOG: dense gradients for detection

Dalal and Triggs’ HOG applies gradient-orientation histograms to human detection rather than matching. Its default configuration uses 9 orientation bins, 8 × 8 pixel cells grouped into overlapping 16 × 16 blocks with strong contrast normalisation, a 64 × 128 detection window and a linear SVM. Unlike SIFT, HOG is computed on a dense grid at a single scale with no orientation alignment, so the vector also encodes where each gradient sits within the window. For person detection, it cut false positive rates by more than an order of magnitude relative to the best Haar-wavelet detector of the time.

What CNNs changed

Krizhevsky et al. (2012) trained a network with 60 million parameters, five convolutional layers and three fully connected layers, reaching a top-5 error of 17.0% on ILSVRC-2010 against 25.7% for the best previously published result, which averaged classifiers built on Fisher Vectors. The “CNN Features off-the-shelf” study took a 4096-dimensional feature vector from a pretrained network, put a linear SVM on top, and beat highly tuned systems on all its visual classification tasks. On the Oxford 102 Flowers dataset, reported scores were 55.1 for SIFT (internal), 49.6 for HOG, 72.8 for their multiple-kernel combination and 74.7 for CNN features without segmentation, rising to 86.8 with augmentation. The authors concluded that CNNs should be the primary candidate in essentially any visual recognition task. In face recognition, the National Academies (2024) reports that over the decade after 2014 suppliers largely discarded hand-crafted features for deep CNNs.

  SIFT SURF HOG CNN features
How features are made Designed: gradient histograms at sparse keypoints Designed: Haar-wavelet responses at sparse keypoints Designed: gradient histograms on a dense grid Learned from labelled data
Descriptor size 128 64 (128 extended) Window-dependent dense vector e.g. 4096 from a fully connected layer
Invariance Scale and rotation, designed in Scale and rotation; an upright variant drops rotation for speed Small local shifts and rotations within a bin Learned during training
Interpretability Readable as edge-orientation histograms Readable as wavelet sums Readable as edge-orientation histograms Mostly a distributed code across many filters

Hand-crafted methods have not become obsolete. O’Mahony et al. (2019) argue that classical techniques remain relevant and review hybrid methods that improve performance or handle problems poorly suited to deep learning. Lowe lists view matching for 3D reconstruction, panorama assembly and robot localisation as further uses of SIFT keypoints.

Where pattern recognition goes beyond faces: defects on a production line

Defect detection applies pattern recognition where defect data are hard to obtain. Automated quality control inspects parts or finished products for defects, and wafer manufacturing, where every wafer is measured and inspected, is one of the most prevalent examples. A 2022 survey of surface defect detection points out that defect images are hard to collect and expensive to annotate, and groups methods into traditional ones (statistical, spectral, model-based) and deep learning ones suited to few labels (augmentation, transfer learning, fine-tuning, semi-supervised, weakly supervised and unsupervised).

One answer to “we have almost no defect images” is to learn only what normal looks like. PatchCore (2021) fits a model on defect-free images alone, storing a representative memory bank of patch features from ImageNet-pretrained networks and treating a patch far from anything in that bank as an anomaly. On the MVTec AD benchmark, it reports image-level anomaly detection AUROC of up to 99.6%, more than halving the error of the next best method. The parallel with faces is close: both pipelines embed an image region and decide by distance, one asking “whose face is this?”, the other “does this patch look like any good part we have seen?”. Finding past images like a given one across a large archive is the subject of our guide to content-based image retrieval with embeddings and ANN indexes.

Open-source detection models that look solved in a demo break down under real lighting, real edge cases and real throughput targets. Our guide to industrial CV inspection production reliability covers the artefacts that keep a line-side model reliable. TechnoLynx’s computer vision R&D takes a model from a benchmark score to one that holds up on the production line.

Across faces, flowers and wafer surfaces, the step that decides most of the outcome is the same: choose a representation in which the classes you care about sit far apart, then choose a decision threshold based on what each kind of error costs. The questions still open on a new project are how much labelled data the representation needs and who pays for a false accept.

Back See Blogs
arrow icon