CASE STUDY
A Generative Approach to Anomaly Detection (Under NDA)
TechnoLynx partnered with an SME known for validating user-supplied data as they expanded into anomaly detection. Their specific focus was to identify cases where data appeared to “drift” away from expected norms, which could suggest errors, unexpected changes, or even more serious concerns like security threats.
The Challenge
As the client’s projects grew in scope and complexity, they needed anomaly detection that could work in real time and handle highly unbalanced data, where normal samples vastly outnumber anomalies. They also wanted a more advanced approach that could identify anomalies without requiring large labelled datasets.
Detect anomalies in real time.
Work effectively with unbalanced datasets where anomalies are rare.
Use an unsupervised approach due to limited labelled anomaly data.
Identify drift and other deviations from expected norms (global outliers / point anomalies).
Image by Freepik
Project Timeline
From problem framing to a Proof-of-Concept (PoC) integrated into the client’s internal toolkit
Problem framing
Defined the anomaly detection target: spotting drift and deviations from expected norms in user-supplied data, including cases that could indicate errors or security threats.
Selected an unsupervised strategy to avoid dependence on labelled anomalies and to fit the client’s unbalanced data reality.
Approach selection (unsupervised)
Model development (generative + neural)
Built an imaging-based anomaly detection system leveraging modern neural networks and custom ML models, extending established auto-encoder methods with diffusion models.
Combined density-based techniques with neural networks and optimised for performance so anomalies could be flagged as they occur.
Real-time optimisation
Handover and integration readiness
Delivered a PoC that the client integrated into their internal toolkit, enabling their team to take over further development and productisation.
The Solution
TechnoLynx delivered a tailored, imaging-based anomaly detection system built with custom code in PyTorch. The approach models the underlying “normal” data distribution, then flags outliers and drift using a combination of generative methods (including diffusion models) and density-based techniques.
Unsupervised anomaly detection
An unsupervised approach allowed the system to learn what “normal” looks like and detect deviations, even when labelled anomaly examples are scarce or unavailable.
Generative models for precision
Auto-encoders (VAEs/AAEs) and diffusion models were used to model complex data distributions and improve the precision of anomaly identification.
Density + neural methods for real-time flagging
Density-based detection helps surface points in sparse regions (often anomalies), paired with neural networks and performance optimisation to support real-time detection.
Technical Specifications
100% custom code built primarily in PyTorch.
Variational auto-encoders (VAEs), adversarial auto-encoders (AAEs), and diffusion models integrated into an imaging-based anomaly detection system.
Unsupervised learning focused on modelling the normal distribution to detect global outliers / point anomalies and drift without requiring labelled anomalies.
Optimised for real-time anomaly detection using efficient algorithms and a combination of neural networks and density-based methods to process large volumes quickly and accurately.
The Outcome
The client was highly satisfied with the PoC outcome, which expanded their anomaly detection capabilities, especially in scenarios without labelled examples. They integrated the solution into their internal toolkit, and their in-house team has since taken over productisation and ongoing development on top of the foundation TechnoLynx delivered.
Key Achievements
Expanded anomaly detection capability using an unsupervised approach suited to rare-anomaly, unbalanced data.
Built an imaging-based system combining neural networks with density-based methods.
Enabled real-time flagging of deviations from expected distributions.
Delivered a PoC integrated into the client’s toolkit, supporting internal productisation.
Want to Improve Fraud Detection Resilience?
Let’s discuss how stronger data pipelines, better accuracy measurement, and ML-driven approaches can reduce risk from rare outlier cases.
