Virtual try on glasses is a real-time face-tracking loop on the shopper’s live camera feed. A face tracker finds landmarks such as the eye corners, nose bridge and head pose on every camera frame, and a renderer pins an accurate 3D frame model to them, with occlusion and lighting, so the frame stays put as the head moves. Generative clothes try-on works another way: Google’s feature generates a new photo with a diffusion model, and Amazon’s Diffuse-to-Choose takes about 6.4 seconds per image on a cloud GPU, not milliseconds. So the deployment question is less “AR or AI” than which part of the experience has to be interactive, which part can wait, and what happens on the phones that cannot keep up.
How virtual try on glasses works on a phone
An eyewear try-on is a pipeline of three layers: face tracking, accurate 3D frame assets, and rendering that looks believable in the shopper’s own lighting. If any one layer is weak, the experience feels like a filter rather than a shopping tool. Eyewear is unforgiving here, because glasses sit in the middle of the face, where small errors in scale or alignment are obvious straight away.
The tracking layer is available as off-the-shelf components. Google’s MediaPipe Face Landmarker chains three models: a face detector, a face mesh model that estimates 478 three-dimensional landmarks, and a blendshape model that outputs 52 expression coefficients. It also returns transformation matrices for effects rendering. Its detector is BlazeFace short-range, built for inference on mobile GPUs. On Android, ARCore’s Augmented Faces gives a native route: the app selects the front camera, enables AugmentedFaceMode, and gets a trackable face whose pose is known while its state is TRACKING. On iOS the equivalent native path is Apple’s ARKit face tracking.
A 2019 paper on makeup try-on in mobile browsers describes a two-stage face tracker of 607 KB that runs in 20 ms on an iPhone XR, client-side, inside a web page.
Frames need consistent occlusion, correct shadows and reflections that match the room, which is why try-on engines estimate the lighting and shade materials in real time; handling hair overlap and partial occlusion stops the frame looking pasted on. Stable pose matters as much as how good it looks: a frame that jitters by a few pixels as the head turns breaks the “will it fit?” judgement the shopper came for.
Does virtual try-on for clothes use 2D images, 3D body models or live AR?
Image-based try-on takes a photo of a person and a photo of a garment and generates an image of that person wearing it. The foundational VITON network (2017) did this without any 3D information, producing a coarse image and then refining the clothing region so the garment deforms naturally. Diffusion models followed: most early ones masked out the existing clothing and inpainted the new garment, while newer models increasingly skip the mask and avoid errors from bad masks.
3D body and cloth models fit a parametric body to the shopper and simulate the fabric on it, so drape and size can be checked from any angle. Live AR overlay works well for rigid items that track a face, and much less well for fabric that has to fold and stretch in real time.
The broader catalogue questions (sizing, fit recommendation, where AI pays off in apparel) are covered in our piece on where AI actually helps in fashion and apparel.
How do virtual try-on systems handle garment fit, eyewear, and cosmetics differently in the CV pipeline?
Each category stresses a different part of the stack:
| Category | What the CV pipeline must get right | Typical runtime | Main failure |
|---|---|---|---|
| Eyewear | Landmark accuracy, scale, stable head pose, occlusion behind ears and hair | Live, on device | Wrong scale or jitter, visible at once |
| Cosmetics | Fine-level facial alignment for makeup rendering | Live, on device, even in a browser | Lower accuracy on occluded or non-frontal faces |
| Garments | Person and garment segmentation or mask-free replacement, pose and shape change | Generated, in seconds | Bad masks, lost detail and colour |
| Shoes, bags, jewellery | A model trained for that category | Generated | Most dedicated garment models do not cover them |
Beauty brands shipped consumer AR at scale on the phone already in the customer’s pocket, which is the same hardware assumption glasses try-on makes.
How do Google’s and Amazon’s virtual try-on features work, and what do they tell us about the production stack behind them?
Both are generative. Google’s 2023 feature uses diffusion with a pair of images instead of a text prompt: the garment image and the person image each go to their own U-Net, and the two networks exchange information through cross-attention to produce a photoreal image of the person wearing the garment. The research behind it, TryOnDiffusion, targets the hard case of warping a garment across a large change in body pose and shape while keeping its detail. Google later extended this to the shopper’s own full-length photo, using a custom image generation model for fashion that understands how materials fold, stretch and drape, across billions of items in its Shopping Graph.
Amazon has published two routes. Amazon Science’s Diffuse-to-Choose is a latent-diffusion inpainting model that inserts any product into a personal scene from a single 2D reference image, adding a secondary U-Net encoder whose “hint signal” preserves fine product detail. Amazon reports about 6.4 seconds per image on a single g5.xlarge instance with an NVIDIA A10G and 24 GB of GPU memory. On AWS, retailers can use Nova Canvas, which takes a source image of a person and a reference image of a product and uses masking to confine generation to the right region.
The AWS reference architecture shows what the production stack looks like around the model: customer and product images in S3, Lambda for requests, DynamoDB for job status, Step Functions orchestrating the call to Nova Canvas through Bedrock, SQS absorbing traffic peaks, and WebSocket connections through API Gateway to push results back. Read that design for what it implies: generation is a queued job with progress feedback, not an interactive frame loop.
For teams building their own generative tier, what Lambda cloud GPUs offer a retail AR stack is covered separately.
Generative try-on versus AR overlay: GPU cost, latency and quality
Face tracking runs in milliseconds and generation in seconds.
| Classical AR overlay (glasses, makeup) | Generative try-on (diffusion) | |
|---|---|---|
| Where it runs | Phone GPU or browser | Server GPU (A10G or A100-class in published figures) |
| Per-result latency | 20 ms for a makeup try-on face tracker on an iPhone XR (2019 paper) | About 6.4 s on an A10G (Amazon); about 11 s (CatVTON) to 46 s (OOTDiffusion) on A100-class hardware (FASHN tests, June 2025) |
| Frame-rate target | 30 FPS, at least 15 FPS on most devices (Snap Lens guidance) | Still images per request, no frame loop |
| Memory | Small models: 607 KB tracker | Under 8 GB of VRAM (CatVTON) up to 28 GB or more (OmniTry) |
| Quality strength | Live, stable as the head moves, any angle | Photoreal drape and fabric on the shopper’s body |
| Asset needed | Accurate 3D asset for each frame design | One 2D product image |
Can generative try-on run on the phone instead? Research has run diffusion on a phone GPU, but in seconds, not at interactive speed. Google’s ML Drift engine ran Stable Diffusion 1.4 (512×512, 20 iterations, FP16) in 8.97 s on a Qualcomm Adreno 750 GPU. ML Drift’s memory planning cut runtime activation memory from 4.31 GB to 387 MB. Stable Diffusion 1.4 is a general image model, not a try-on model; the memory side of that limit is the subject of our piece on on-device memory (DRAM) for AR try-on rendering.
Every generative figure here is in seconds, so a sub-200 ms response budget leaves only the overlay path. Generative output can still feature in the experience, as a “see it on your photo” result that arrives a few seconds later with clear progress feedback.
What does cloud rendering and streaming cost in network and latency terms compared with full on-device rendering for try-on, and how do you decide the split before a campaign launches?
Streaming rendered frames from a server needs a stable, high-bandwidth connection. NVIDIA’s CloudXR documentation recommends 200 Mbps of downstream bandwidth (100 Mbps minimum), 20 to 30 ms from pose to frame received (100 ms maximum, and that only for non-interactive content), jitter of 1 ms (4 ms maximum) and zero packet loss (1% maximum). It requires Wi-Fi clients to use the 5 GHz or 6 GHz band and does not support high-latency links such as VPNs. Those are requirements for headset streaming, not phone try-on, but they show the order of what a streamed interactive frame loop asks of a shopper’s connection.
The ML Drift authors list user privacy, lower latency, offline operation and lower server costs as the reasons on-device execution still matters when servers have far more compute. On-device execution brings its own costs: heat, battery and operator support that varies across mobile processors.
We would decide the split before launch with three checks:
- Classify each interaction by its latency need. Head-tracked placement and frame swapping are interactive and stay on device; “see this on my photo” can be asynchronous and go to the cloud.
- Size the cloud tier for the peak, not the average. The AWS design uses queues and serverless orchestration because retail traffic fluctuates; with several seconds of GPU time per image, campaign peaks set the GPU count.
- Profile the device mix the campaign targets, not the team’s own phones, and set the tiers below from that.
For teams building their own render or generation hosts, the host-side choices are covered in Linux OpenCL for GPU rendering pipelines, and the interconnect decision when a render tier grows past one box in active optical cable versus copper in a GPU render tier.
How does a production virtual try-on app assign per-device rendering tiers across the mix of phones shoppers actually use, and what triggers the fallback path?
A tiered try-on app can pick a rendering tier at launch from the device’s capabilities and demote itself at runtime when measurements show it falling behind.
TechnoLynx’s GPU audit checks the device-tier rendering budget a tiered try-on relies on before the campaign goes live. The retail side of that engineering work is described under AI engineering for retail.
The static part has precedent. Facebook’s Device Year Class library maps an Android device’s RAM, CPU core count and clock speed to the year that combination counted as high end, and apps branch on the result (advanced animation, simple animation, or none). Its published mappings stop at 2016, so the idea carries over but the table does not; we would rebuild the equivalent from GPU family, RAM and measured startup benchmarks on the actual device list.
A workable three-tier scheme for glasses looks like this (assumptions, not measurements):
| Tier | Assigned when | Renders |
|---|---|---|
| Full | Recent GPU, enough RAM, thermal headroom | Full material shading, reflections, occlusion, 30 FPS target |
| Reduced | Mid-tier GPU or warm device | Simpler shading, no reflections, lower tracking resolution |
| Fallback | Old device, tracking unsupported, or sustained failure | 3D product viewer or a still-photo try-on |
A 3D product view (rotating the frame in space) and the face-based try-on can work together on one product page.
The runtime triggers are where tiers earn their keep:
- Frame rate below the floor. Snap’s guidance is to target 30 FPS and stay above 15 FPS on most devices; a sustained drop under the chosen floor demotes one tier.
- Thermal state. Android’s ADPF Thermal API lets an app monitor the device’s thermal state and reduce workload before it becomes unsustainable, so the app can demote itself before the workload becomes unsustainable.
- Operator fallback at model load. When a mobile GPU cannot run an operator, inference falls back to the CPU; one 2024 study measured latency increases of 93% and 126% across two frameworks when this happened. Unsupported-operator ratios vary by processor combination, reaching about 48% in one Xiaomi 9 configuration, so a model that runs well on one GPU family may quietly fall back on another.
- Lost tracking. If the face is not in a TRACKING state, the frame has no known pose to attach to.
- Slow activation. Snap requires an ad Lens to activate in under 650 ms on its benchmark device and recommends keeping Lens size under 8 MB; lower RAM use means faster loading and stabler running on lower-end devices.
The same study warns that maximising inference resources can starve the UI in AR apps. A try-on that tracks perfectly while the “add to basket” button stutters has the wrong budget.
Where do virtual try-on deployments break down: fit accuracy, cloth draping, thermal throttling or the content pipeline?
Phone heat builds within minutes of camera use. A 2019 MobiCom study found test smartphones easily passed 45°C, with temperature rising quickly in camera scenarios; video chat took only 6 minutes to pass the thermal pain threshold. A live try-on is a camera, a tracker and a renderer running together, so it sits in that camera-heavy class of workload. When phones get hot, governors cut clock frequency: one analysis of commercial platforms measured performance losses of up to 34% in popular Android apps, with one app’s frame rate falling from 35 to 23 FPS. Another study found that GPU inference latency rises as temperature climbs, because DVFS steps in to prevent overheating. A tier tested on a cool device misses the throttling that rising temperature brings.
Fit accuracy fails visibly on eyewear because of where glasses sit. On garments, generative models fail on detail and masks: CatVTON’s authors note detail loss and colour discrepancies at 512×384 and say an inaccurate mask can significantly degrade results. Cloth draping is what Google’s fashion model was built to understand, and TryOnDiffusion’s two-UNet design carries expensive computational costs.
Glasses try-on needs accurate 3D frame assets alongside face tracking, and a weak asset layer makes it feel like a filter. Before choosing a model, check the quality and quantity of the product data and the process that produces more of it, set expectations away from near-perfect accuracy, and design for faults: a fallback tier, and a human review path for assets and results that fail checks.
Are open-source virtual try-on models on GitHub ready for production?
The open-source try-on models in FASHN’s June 2025 tests took about 11 seconds (CatVTON) to 46 seconds (OOTDiffusion) per image on A100-class hardware, far from a sub-200 ms budget.
In that try-on vendor’s comparison, CatVTON, IDM-VTON and OOTDiffusion were all non-commercial, so none could ship in a product without a separate licence. Treat that comparison as a vendor’s view of its competitors.
On the engineering side, CatVTON was the strongest and fastest of the three tested. It concatenates garment and person images spatially, drops the ReferenceNet and text encoder, trains only 49.57M parameters, and needs just the garment image, the person image and a mask, with no pose estimation or human parsing. FASHN reports it running in under 8 GB of VRAM.
The split the evidence supports is a tracked on-device overlay for the interactive part and a later generative result, with a queue, socket progress feedback and peak-sized GPUs, as in the AWS design.
For the wider picture, see immersive XR for customer engagement; the problems virtual reality can solve outside retail are a separate discussion. For glasses, the open question is not whether the technology works but whether it still works on the fifth minute, on the phone your median shopper owns, in their kitchen lighting.
