An end-to-end machine learning pipeline is a chain of stages: data validation, feature engineering, training, validation, registration, deployment and monitoring. Only some of that chain is ordinary software delivery. Code promotion and integration testing reuse the CI/CD you already run in Jenkins or GitLab. Data validation, model training, model validation and a model registry are added on top, with ML-specific orchestrators handling the model layer.

The wider definitional contrast between the two disciplines is covered in our MLOps vs DevOps overview. What follows is the stage mechanics: what each stage produces, which of your existing tooling carries over, how far reproducibility can really go, and where pipelines break once they reach production.

How does an end-to-end machine learning pipeline actually work?

An end-to-end machine learning pipeline works as three layers. DevOps CI/CD tools handle code promotion, ML orchestrators manage model training and deployment, and a monitoring layer feeds production signals back into retraining. In practice Jenkins, GitHub Actions or GitLab CI keep doing integration tests and code promotion, while an orchestrator such as Airflow or Kubeflow runs the training and deployment graph. The usual integration pattern joins the layers: a code merge triggers the training pipeline, the resulting model is registered in a model registry, and validation and deployment run as later automated stages.

What counts as a change also differs. In an ML system a build can be triggered by a code change, a data change, a model change, or any combination of the three. Google’s MLOps architecture guidance spells out what follows from that. CI tests data, schemas and models as well as code. CD ships a training pipeline that in turn deploys a prediction service. Continuous training (CT) is a property unique to ML systems. At what Google calls level 1, you no longer deploy a trained model. You deploy a whole training pipeline that runs recurrently to serve the model.

The ML Test Score paper offers a useful analogy. Training is like compilation where the source is both code and training data, so data needs testing the way code does, and a trained model needs the production practices a binary gets, such as rollbacks and monitoring.

What are the stages in a machine learning pipeline diagram, and what does each produce?

Each stage of an ML pipeline produces an output that the next stage uses. The table below is the diagram in extractable form. For each stage it shows the output, the part your existing CI/CD already covers, and the part that is genuinely new.

Stage	What it produces	Reuses existing CI/CD	Needs a new MLOps artefact
Data ingestion and validation	A validated data batch, computed statistics and any schema anomalies	Scheduling and triggering of the job	A data schema versioned in source control alongside code, plus dataset versions
Feature engineering	Transformed features from versioned feature code	Unit tests of feature logic in CI	Shared feature definitions, for example a feature store serving training and serving alike
Training	A trained model plus its parameters, metrics and artefacts	Building packages and container images; smoke tests that training converges and produces no NaNs	An experiment-tracking run with data checksums, pinned libraries and committed hyperparameter configs
Evaluation and validation	A pass or veto decision against the current production model on held-out data	The test-stage runner that fails the build	A fixed held-out set, agreed thresholds and the baseline comparison
Registration	A versioned model linked to the run that produced it	The same development, staging and production promotion model	A model registry entry with aliases and tags
Deployment	A deployed model prediction service	Environment promotion, canary and rollback	A pinned model version and an alias that can be moved back to the previous version
Monitoring	Statistics on live performance and a trigger to rerun the pipeline or start a new experiment	Infrastructure metrics and alert routing	Drift and training-serving skew checks on inputs and predictions

An empirical study of AutoML tool usage (2022) treats hyperparameter optimisation as a stage of its own, and 40.9% of AutoML function calls fell there, ahead of model training at 11.5% and model evaluation at 11.2%. The diagram’s real value is that every arrow carries a named artefact. When something degrades, you can ask which handover went wrong instead of debating the whole system.

Which pipeline stages reuse existing DevOps CI/CD tooling, and which need new MLOps artefacts?

Jenkins and similar CI/CD tools keep handling code promotion and integration testing in an ML retraining pipeline. What MLOps engineering adds on top is data validation, model training, model validation and registry steps, run by ML-specific orchestrators. Databricks’ MLOps reference assigns Jenkins, GitHub Actions and GitLab CI the code promotion and integration-testing layer.

The fastest way to see your own gap is to audit your existing jobs stage by stage against a reference checklist, marking each item satisfied, partial or missing. A practical reference set, drawn from Databricks’ MLOps CI/CD checklists:

• Training data is versioned, with checksums logged to the experiment tracker.
• Training code is pinned to specific library versions in a lockfile.
• Hyperparameters live in configuration files committed to source control.
• Training runs log metrics, parameters and artefacts through experiment tracking.
• Integration tests run training on a representative data subset.
• Automated model validation gates run before production promotion.
• The model registry stores every promoted version with metadata and lineage.
• Monitoring alerts on drift thresholds, and CI/CD pipelines can trigger retraining automatically or manually.

Generic build-and-test jobs cover the code-promotion items; the data, model and registry items are the steps MLOps adds on top. If the audit shows only a few missing capabilities (versioning, a registry, a drift trigger), bolting point tools onto Jenkins is the cheaper route. If it shows broad gaps across scheduling, orchestration and multi-stage dependencies, an orchestrator such as Kubeflow or Airflow running alongside Jenkins starts to earn its place.

Two caveats apply. The audit is only as complete as the reference checklist you use. And a drift-triggered retrain depends on a separate monitoring layer that closes the loop back to retraining.

Is a DVC workflow with simple CI enough for a small team, or does Kubeflow pay off?

For a team of about three people running two related model families, such as crop detection and yield estimation, our judgement is that DVC plus the CI you already have is enough for now. Kubeflow Pipelines’ orchestration overhead pays off later, once branching, concurrency or team size outgrow what CI scripts can manage cleanly. TechnoLynx can run that assessment with you. TechnoLynx catalogues your current retrain frequency, branch count and team bandwidth against those signals, sets up DVC-tracked dataset versioning and CI triggers for both retraining paths, confirms that runs reproduce and roll back from DVC versions, and documents the route to orchestration if you cross the line later.

DVC covers more than file versioning. Thoughtworks’ CD4ML write-up describes it storing large files outside the Git repository while tracking their versions, recording the dependency graph and commands of the pipeline so it can be reproduced in other environments, and integrating with Git branches so experiments can coexist. dvc repro rebuilds the stage graph much as make does, running only the stages whose dependencies changed and caching outputs along the way.

Treat these signals as judgement rather than rules:

• Pipelines with several concurrent or conditional branches that CI scripts express badly.
• Steps that need distributed or parallel execution across machines.
• Retraining frequent enough that nobody can watch each run by hand.
• A team large enough that pipeline logic held in CI scripts stops being maintainable.

The YOLOv8 crop-detection and yield-estimation models can share one pipeline definition only if their preprocessing, training and evaluation stages are similar enough to parameterise. If they are not, give them separate DAGs. Forcing two dissimilar models into one definition to save tooling effort creates fragile coupling. Migrating from DVC plus CI to Kubeflow later also has a real switching cost, which belongs in the decision now rather than as a surprise afterwards. If you are still comparing tools, our look at the pros and cons of MLOps tools covers that ground.

A model that ships inside firmware should get its own CI stage, decoupled from the firmware release train. That stage should build, tag and publish the model to a registry, with the firmware pipeline consuming a pinned model version. A registry such as MLflow tracks each version and its lineage for exactly this kind of independent rollback. The pin alone does not make releases safe: without a versioned input and output contract between model and firmware, rolling the model back can still break firmware that expects a particular interface.

Where should the model-accuracy gate sit in a Jenkins or GitLab pipeline?

The model-accuracy gate belongs after training and before any promotion or deploy stage. It should veto the retrained model unless it performs better than the currently deployed model on the test data. TechnoLynx builds the evaluation harness and regression suite that a gate like this runs, as part of the Production AI Monitoring Harness.

Your existing code tests cannot do this job, and the reason is structural. ML system behaviour is learned from data rather than specified in code, so unit and integration tests of the code are not enough on their own. A green CI check shows that code compiles, tests pass and data contracts hold. It does not show that the model is fit for production. Model performance is non-deterministic, and the gate uses threshold tests against a known performance baseline. Production AI reliability is its own discipline for this reason: models regress in ways unit tests cannot catch, and the artefacts that catch those regressions are evaluation harnesses, regression suites and release gates, not dashboards.

The metric follows from the cost of each kind of error. For a safety-relevant collision-risk classifier, recall on near-miss incidents (equivalently, the false-negative rate) should decide pass or fail, with precision-recall AUC as a secondary, threshold-independent sanity check. Google’s guidance adds that the comparison should hold across data segments, not only in aggregate. Its Rules of ML note that many teams that deploy continuously check AUC on held-out data before exporting a model.

On the tolerance, the ML Test Score paper suggests two complementary checks. Loose thresholds against a validation set catch slow degradation over many versions, and tighter thresholds against the previous model’s predictions catch sudden drops. CD4ML calls the same idea threshold tests or ratcheting against a known baseline. The tolerance should be small and explicitly agreed, and revisited as retraining history builds up. None of this works unless the held-out set stays fixed and representative across cycles. Change it, and the comparison with the baseline means nothing.

Wire the gate’s outcome into promotion state. MLflow’s registry lets you tag a version validation_status:pending while it is checked, and a gated state machine moves a version from candidate to validated, staged, canary and production, with rollback by reassigning the champion alias. Data validation and drift detection need gates of their own. Our piece on where reliability gates belong at each stage of an ML pipeline places them.

How does the monitoring stage differ from application observability, and why does it matter for silent data drift?

Continuous monitoring, as one MLOps paper defines it, watches the IT system for problems such as compliance issues and security threats; MLOps extends it to production data and model performance. In practice that means checking whether production inputs still resemble familiar conditions, as a proxy when ground-truth labels are unavailable. The distinction matters because ML systems tend to fail silently on dataset shift: the endpoint keeps returning well-formed predictions while their quality decays. Zinkevich’s Rules of ML gives a concrete case: a joined table stops updating, the system adapts and decays gradually, and a feature populated in 90% of examples can suddenly drop to 60%.

Tracking true model quality directly is the preferred detector when labels allow it. When feedback delays prevent that, input drift becomes the proxy signal. Evidently AI recommends separating the two groups of checks: verify data quality first (completeness, valid ranges), then test distributions for statistical shift, since a broken pipeline can look like drift. Detection methods include summary feature statistics, hypothesis tests and distance metrics.

A drift alert is an investigation signal, not a licence to promote. A retraining candidate raised by drift should go through the same CI and CD gates as any code-driven change, and drift alone should never trigger automatic promotion. TechnoLynx treats this monitoring stage as reliability engineering rather than an extension of observability tooling, because the deliverable is a check that catches silent quality regression, not a dashboard. Which signals deserve an alert is a question of its own, covered in our guide to performance metrics to track in production.

What makes a pipeline reproducible end-to-end when several stages produce non-deterministic artefacts?

An ML pipeline is reproducible end-to-end when its release process is reproducible, even though individual training outputs may not be bitwise identical. CD4ML states exactly that split: model outputs can be non-deterministic and hard to reproduce, while the process of releasing ML software stays reliable and reproducible through automation. In practice that means every input to a stage is pinned and recorded: a data snapshot or version, feature code under version control, a fixed feature order, documented hyperparameters, and a container image matching production. DVC records the hashes of changed dependencies and outputs in dvc.lock, and its --no-run-cache flag forces a stage to rerun rather than being restored from cache, which is useful when the stage command is non-deterministic.

Bitwise reproducibility has hard limits, and framework documentation states them. The PyTorch 2.14 documentation states that completely reproducible results are not guaranteed across PyTorch releases, individual commits or platforms, nor between CPU and GPU executions, even with identical seeds. cuDNN convolution benchmarking can choose different algorithms on later runs, even on the same machine. Floating-point accumulation order differs across SDPA backends, so their results do not match bitwise either. Sculley et al. (NeurIPS 2015) list randomised algorithms, non-determinism in parallel learning, reliance on initial conditions and interactions with the external world as obstacles to strict reproducibility.

Determinism can be bought, at a price. torch.use_deterministic_algorithms() switches to deterministic implementations where they exist and raises an error where none does, and deterministic operations are often slower. In Chen et al.’s 2022 study, patching for hardware determinism added between 24% and 114% to training time in their experiments. The same study found software randomness caused larger variance than hardware non-determinism, though the hardware share was not negligible. Our judgement is to turn determinism on where it pays: regression tests, debugging and audit reruns. For production training, define “reproduced” as same lineage and metrics within an agreed tolerance.

Where does an end-to-end pipeline typically break in production, and how does a clear stage map shorten diagnosis?

Data errors degrade a pipeline’s model irrespective of the algorithm used. Three documented failure points are schema skew at ingestion, training-serving skew between pipelines, and unstable upstream data dependencies. Schema skew means unexpected features, missing features or features with unexpected values, and Google’s guidance is to stop the pipeline when it appears. Training-serving skew most commonly comes from different code paths generating training and serving data. The ML Test Score paper calls the test for it perhaps the most important and least implemented of its rubric. Unstable dependencies arise when another team or another model owns an input signal, and even “improvements” to it can damage the consuming model in ways that are costly to diagnose.

A stage map turns a degraded model into a bounded question. Breck et al. (MLSys 2019) describe a missing-features alert that led to a backend storing features in an unexpected format: diagnosis and fix took two days, where similar issues had taken months. Lineage metadata helps debug such issues. TensorFlow’s ML Metadata can trace back what data went into a model and find every model trained from a given dataset, so models built on bad data can be marked.

A diagnostic procedure for a degraded model, by stage

1. Confirm which model version is serving and pull its registry lineage to the training run, data and parameters.
2. Check data quality on recent inputs (completeness, types, ranges) against the versioned schema before looking at distributions.
3. Compare training and logged serving feature statistics for skew.
4. Check upstream tables and signals for staleness or ownership changes.
5. Rerun the evaluation for the serving version on data gathered after its training cutoff.
6. Roll the model back by moving its alias to the previous version while the investigation continues.

A wider catalogue of components and failure points is in our MLOps pipeline components and failure points piece.

How do three teams move off cron jobs and notebooks onto one pipeline without a big-bang cutover?

Three teams retraining on ad-hoc cron jobs and notebooks should standardise on one component set (data versioning, experiment tracking, a model registry and CI/CD for models) and move onto it one team at a time. Each team’s old cron job keeps running until the new pipeline’s outputs match it.

TechnoLynx supports this kind of migration through the Production AI Monitoring Harness, which builds the evaluation, regression and drift checks that show when a team’s new pipeline matches its old cron job, and the first step is an audit of each team’s existing jobs.

Existing Jenkins and Feast investment does not need replacing. Jenkins keeps the code-promotion layer while an orchestrator takes training, and a feature store serving both training and serving remains the guard against training-serving skew. Self-hosted Kubeflow on Kubernetes suits teams with cluster operations skill. Each migrated team should leave a working, usable pipeline behind it rather than value that arrives only when all three are done, because migrations that pay off only at the end lose momentum. Two caveats: the parallel run keeps both systems in operation for its duration, and adoption by one team alone defeats the purpose, so set a deadline for the rest. Teams starting from no operational ML at all may find our note on MLOps for organisations that have never operationalised a model a better starting point.

Should two model families share one CI/CD pipeline and retraining trigger?

Two architecturally different model families run by two data scientists should share one CI/CD pipeline framework but keep independent, model-specific drift thresholds for raising a retraining candidate. One framework means one set of stages to maintain. Separate triggers respect the fact that a crop detector and a yield estimator drift for different reasons.

A drift-triggered candidate fits a small team better than a fixed unattended schedule or purely manual triggers. Continuous training is designed to retrain on new data or when monitoring shows drift has pushed accuracy below acceptable thresholds, but each candidate still passes the same gates. The caveats are practical. The pattern assumes reliable drift detection already exists for both model types. Shared tooling can hide architecture-specific failures if monitoring is not model-aware. And two reviewers may struggle when both models drift in the same week.

Can the deploy pipeline block a model whose feature tables hold undisclosed personal data?

A deploy pipeline can block such a model by embedding a compliance check as a CI/CD gate before promotion. TechnoLynx’s AI Governance and Trust work covers gates like this.

Who can help us add versioning, gates and drift monitoring to the Jenkins pipeline we already run?

TechnoLynx builds the eval, regression, drift and slice-metric harness that sits alongside the CI/CD you already run. The hard part of the build-versus-buy question is the reliability layer: gates whose thresholds hold up, regression suites and drift checks that catch silent decay. When choosing a partner, ask for a stage-by-stage audit of your own jobs before any tooling proposal, and for deliverables that work in your pipeline rather than slides. The engagement for this is the Production AI Monitoring Harness, which starts with that audit of your existing jobs.

A team with one model, infrequent retrains and engineers who already know DVC and MLflow can usually close the gap itself. A team that wants the operations taken off its hands entirely may be better served by a managed MLOps platform.

FAQ

Can MLflow be used in MLOps?

Yes. MLflow’s Model Registry provides versioning, aliases, tags and lineage from each model version back to the run that produced it, which covers the registration stage of an ML pipeline.

What does “pipeline” mean in machine learning?

Google’s Rules of ML defines the pipeline as the infrastructure around a learning algorithm: gathering data, putting it into training files, training one or more models and exporting them to production.