Several uses of artificial intelligence in the beauty industry that shoppers touch run on computer vision, such as virtual makeup try-on and smart mirrors that bundle skin analysis, try-on and product recommendation. Further from the shopper, machine learning also predicts how a formulation will behave and helps perfumers design new fragrances, but that work runs on formula data, not on faces. Beauty recommendation models are often trained assuming the photo faithfully represents the customer’s face, two widely used face datasets skew heavily lighter-skinned, and an industry ranking report finds undertone the hardest dimension for try-on platforms.
Where artificial intelligence in the beauty industry runs today
A 2024 survey of smart mirrors in health and beauty (Fatima et al.) describes demand for AI-driven personalisation in cosmetics, with augmented reality used for product recommendations and colour customisers. The same survey says commercial beauty tech is limited to information delivery and product customisation, through portable skin analysers, curation apps and skin analysis apps that are not yet widespread. Smart mirrors are the attempt to put these together, and products such as HiMirror, Shiseido’s Digital Counseling Mirror, Panasonic’s mirror and ModiFace each offer some of skin analysis, virtual try-on and product recommendation.
The uses, and what breaks each one
| Use | What the system does | Depends on computer vision | Where it breaks outside the demo |
|---|---|---|---|
| Virtual makeup try-on | Tracks facial landmarks, renders the product’s finish and opacity, adjusts the overlay to the lighting | Yes | Foundation shade matching and undertone detection; low light |
| Selfie skin analysis | A CNN extracts skin features (texture, colour, shape) and classifies them | Yes | Small training sets; conditions only a physical examination reveals |
| Foundation shade matching | Estimates skin tone, then measures the colour difference to every shade in the range | Yes | Badly lit selfies shift the estimated tone and scramble the recommendation |
| Smart mirrors | Combine try-on, skin checks and recommendations on one display | Yes | Low optical resolution of acrylic two-way mirrors; limited real-time try-on |
| Search by product photo | Retrieves the product a shopper photographed | Yes | Background, pose and brightness variation; look-alike products; index lag |
| Formulation and fragrance design | Predicts texture, stability and sensitisation risk; proposes fragrance formulas from archives | No | Needs oversight and harmonised standards; small formula deviations change the result |
Five of the six rows depend on computer vision, and for several of them the breaking point is an input image less controlled than the model expected. The sixth is a different engineering problem entirely, which is why it gets only a short section below.
How a shade finder matches your skin tone to a foundation
As the 2023 illumination study describes it, a foundation shade finder estimates your skin tone from a photo, then assesses the colour difference between that estimate and each shade the brand sells to choose the best match. L’Oréal Paris, Maybelline and NARS offer consumer shade-finder tools. The skin-tone estimate going into it is the hard part.
Skin colour is a combination of ethnicity, phototype, undertone, reflection and the light falling on the skin, according to a 2026 note from the testing laboratory ALS. Undertone (the apparent temperature of the skin under light, usually classed as cool, neutral, warm or olive) is why two people with similar skin tones can need different shades. A face photo records skin tone and room lighting mixed together. Separating the skin’s own colour from the colour of the room is, in the words of a 2023 study on illumination assessment for beauty recommendations, still an open problem.
Laboratories handle this with colorimetry and specialised instruments, which give reproducible skin-colour measurements and can be compared against visual panels to find shade gaps in a portfolio. A phone camera in a bathroom has none of that control, so the engineering question for a shade finder is less about the classifier and more about whether the input photo can be trusted at all.
Hair appearance depends on fibre geometry and how light bounces between fibres, so reproducing a specific colour in a renderer has traditionally needed manual tuning by a computer graphics expert; a 2022 paper proposes capturing a physical hair sample with deep inverse graphics and re-rendering it under different styles and lighting.
Can you trust AI skin analysis from a selfie?
One validation study shows where selfie skin analysis holds up. A study of 1,140 women from three ethnic groups, aged 18 to 80, graded nine facial ageing signs from smartphone selfies with a CNN trained on referential skin-ageing atlases. Its results were compared with assessments by 12 trained experts and dermatologists, on selfies and, for a smaller group, in person. The authors report that the system was only weakly influenced by lighting and expression, with coefficients of variation of 10 to 13% for most signs, and that it agreed globally with the experts.
That result covers a specific job: grading nine facial signs against skin-ageing atlases, with ethnicity declared by the user. The smart-mirror survey notes that CNN classification of facial skin patches showed room for better accuracy with a larger dataset, and that a mirror’s analysis may miss conditions a physical examination would catch.
Lighting is the first failure
The 2023 illumination study states the problem directly: product photos are usually taken in good light, face photos are not, and features extracted from a badly lit face can be misleading or simply not comparable with the product’s attributes. When the researchers estimated skin tone from badly lit photos, the average colour difference from the best-lit photo exceeded 10 for five of their six test models, a perceptually noticeable shift. For one model, no shade of a test product fell within distance 5 using well-lit photos, while 11 of its 12 shades did using badly lit ones, a recommendation with no discriminating power at all.
They trained a classifier that labels a face photo as well or badly lit, and built guidance that tells the user, while holding the camera, whether to retake the shot. The classifier is a MobileNet-v2, chosen to run on edge devices, trained in PyTorch on synthetic faces and tested on 674 real photos. We think guiding the capture is the right choice over correcting colours afterwards: rejecting a bad input at capture time avoids colour correction, which the authors describe as prone to bias and unexplainable to the user.
Skin tone is the second, and it compounds the first
Lighting and skin tone are hard to disentangle in a face image. The same study reports that an earlier illumination-assessment model failed on dark-skinned faces, classifying most of them as badly lit, and cites work showing face-quality models favour light-coloured faces. Few studies tackle the issue, largely because no large, labelled dataset is diverse in both skin tones and light patterns.
Research on general facial analysis shows a similar skew. Buolamwini and Gebru (2018) found two benchmark face datasets were 79.6% and 86.2% lighter-skinned subjects. Testing three commercial gender classifiers, they measured error rates of up to 34.7% for darker-skinned women against a maximum of 0.8% for lighter-skinned men. That study measured gender classification, not skin analysis, so its figures do not transfer as numbers. We would still test any face analysis pipeline per skin-tone group, since the datasets studied skewed lighter-skinned and errors were highest for darker-skinned women.
How AR makeup try-on renders a shade, and how close it gets
AR try-on overlays digital makeup on a live camera feed or photo, and a 2026 industry ranking report comparing 20 platforms describes three parts to it: face mapping across hundreds of landmark points, photorealistic rendering of finish, texture and opacity, and lighting correction that adjusts the overlay to the room. The report adds that strong face mapping with weak lighting correction still looks unnatural. Rendering a given product faithfully remains a challenging task. Kips et al. (CVPRW 2021) note that while try-on is widespread, setting a rendering engine’s parameters to reproduce a given cosmetic product remains hard, and they propose learning those parameters from an example portrait wearing the makeup. The smart-mirror survey likewise records colour reproduction as a known limitation of virtual makeup, with CNN-based methods proposed for accurate lipstick colour.
Rendering accuracy differs by product category. The same ranking report scores lip colour as the most accurate category, with the best platforms at 97/100, because the lip border is well defined and contrast is high. Foundation is the hardest, needing skin-tone matching, undertone analysis and texture simulation at once, with an average score of 82/100 across the 20 platforms and undertone detection the weakest dimension for all of them. Mascara averages 78/100. The report also flags low light as an overlooked dimension, since shoppers try products on at home.
Those scores need careful reading. They come from a ranking report that attributes several of its fidelity claims to the vendors themselves; ModiFace’s shade validation against L’Oréal laboratory data, for example, is reported from ModiFace’s documentation. The smart-mirror survey’s own assessment is that the accuracy of these technologies may not yet be sufficient, and that the virtual try-on and the real result can differ. The report itself argues that low accuracy can damage brand perception when the delivered product looks visibly different from the simulation, which is the commercial reason to measure fidelity yourself, not take a score on trust.
Scaling try-on across a large catalogue raises its own problem, getting rendering right for products no one has tuned yet, which we cover in AR beauty try-on at scale and the cold-start problem. How the same AR approach plays out beyond cosmetics is in how AR and AI redefine virtual try-on in e-commerce.
What image-to-product matching for a beauty catalogue costs to build and keep current
Matching a shopper’s photo to a product in a beauty catalogue is harder than it looks for two reasons, according to a 2023 study of beauty product retrieval: real-world photos vary in background, pose, angle and brightness, and the catalogue holds many items that look almost identical but are different products.
Pinterest’s account of its Shop The Look visual shopping system (KDD 2020) covers object detection, visual embeddings, serving infrastructure for real-time inference, and a data-labelling method for training, evaluation and human relevance judgement.
Keeping the index current is a separate task. In Google’s Vision API Product Search, images added or deleted are not reflected in search results until the index is next updated, and an indexTime field shows which index version answered a query. The same documentation marks Product Search as in maintenance mode and points users to Vision Warehouse. For a beauty brand with seasonal launches and discontinued shades, that lag means a shopper can be matched to a product that no longer exists, or miss one that launched last week, so reindexing belongs in the release process, not in an occasional maintenance job. The wider mechanics of recognising products from photos are covered in how an online shopping assistant does visual product recognition.
Beyond the consumer app: formulation, sensitisation and fragrance
AI in beauty also reaches into product development, where the cited work runs on formula and ingredient data. A 2025 review in the journal Cosmetics describes predictive models for surfactants, polymers, fragrances, preservatives and other ingredients, forecasting texture, stability and shelf-life, and using computational toxicology to predict skin sensitisation risks such as allergic contact dermatitis early. The same review stresses transparency, accountability and harmonised standards as conditions for using these models safely.
In October 2018 Symrise and IBM Research announced Philyra, which analyses thousands of historical formulas to propose novel combinations for perfumers, with two fragrances for O Boticário due on the market in 2019. Symrise noted that fragrance demands unusual precision: a minimal deviation from the specified amounts can make or break a perfume.
What a production test set for beauty vision has to contain
The common thread is that models working in demos fail in production on edge cases and lighting conditions. The smart-mirror survey makes a related point in hardware terms, listing consistent lighting and high-resolution cameras as what a fixed mirror offers over varying phone cameras and lighting.
Before trusting a try-on, shade finder or skin-analysis feature, we would want the evaluation set to cover these conditions, each reported separately and not averaged away:
- Lighting: well-lit, partially shadowed, over-exposed and warm indoor light, since badly lit photos shift estimated skin tone noticeably.
- Skin tone: deep and light skin in comparable numbers, with errors broken out per group, because commercial face classifiers have shown sharply different error rates between skin-tone groups.
- Undertone: cool, neutral, warm and olive, since undertone is where try-on platforms are weakest.
- Product category: foundation and mascara tested separately from lip colour, as category accuracy differs widely.
- Instrument ground truth: skin and product colour measured by colorimetry and controlled visual panels, which give objective, reproducible data.
- Catalogue churn: queries for newly added and newly removed products, run after each catalogue update to catch index lag.
- Input rejection: how often the system asks for a retake, and whether it rejects dark-skinned faces as “badly lit” more often than light ones.
The last item is the subtle one. A capture-guidance step that fixes lighting can quietly reintroduce skin-tone bias at the gate, and only a per-group rejection rate will show it. For visual retail beyond cosmetics, see our look at where AI helps in fashion and apparel.
FAQ
Does L’Oréal use AI?
Yes. L’Oréal Paris runs a virtual foundation shade matcher called Match My Shade, and a 2026 ranking report describes the AR try-on platform ModiFace as backed by L’Oréal’s cosmetics laboratories.
How do I use AI for makeup?
Tools include brand shade finders, such as those from L’Oréal Paris, Maybelline and NARS, and virtual try-on, some of it built into smart mirrors. Take the photo in even light with no shadows on your face, because poor lighting shifts the skin tone the app estimates and can change which shade it recommends.
