{"slug":"mlops-production-pipeline","title":"Production MLOps: From Jupyter Notebook to Scalable Model Serving","description":"Building an end-to-end MLOps pipeline with Kubeflow, MLflow, Seldon Core, and automated drift detection for production ML systems","date":"2025-04-05","author":"Elaph Hilful","thumbnail":"/blog-images/mlops-pipeline.svg","tags":["MLOps","Kubernetes","Machine Learning","Kubeflow","AIOps"],"category":"MLOps","featured":false,"readTime":6,"htmlContent":"

The gap between training a model in a Jupyter notebook and serving it reliably in production is enormous. Most ML models never make it to production — not because they don't perform well, but because organizations lack the infrastructure to deploy, monitor, and maintain them at scale.

\n

This guide covers building a production MLOps pipeline that serves 15+ models with sub-hour deployment times and automated drift detection.

\n

\"Production

\n

The MLOps Maturity Gap

\n

Here's what most ML teams deal with:

\n\n

Most organizations are stuck between Level 0 and Level 1. Our pipeline operates at Level 3.

\n

Pipeline Architecture

\n

Stage 1: Data Versioning with DVC

\n

Every ML experiment must be reproducible. DVC (Data Version Control) tracks datasets alongside code:

\n
# dvc.yaml - Pipeline definition\nstages:\n  prepare:\n    cmd: python src/prepare.py\n    deps:\n      - src/prepare.py\n      - data/raw/\n    params:\n      - prepare.split_ratio\n      - prepare.seed\n    outs:\n      - data/processed/train.parquet\n      - data/processed/test.parquet\n\n  train:\n    cmd: python src/train.py\n    deps:\n      - src/train.py\n      - data/processed/train.parquet\n    params:\n      - train.learning_rate\n      - train.epochs\n      - train.batch_size\n    outs:\n      - models/model.onnx\n    metrics:\n      - metrics/train_metrics.json:\n          cache: false\n\n  evaluate:\n    cmd: python src/evaluate.py\n    deps:\n      - src/evaluate.py\n      - models/model.onnx\n      - data/processed/test.parquet\n    metrics:\n      - metrics/eval_metrics.json:\n          cache: false\n    plots:\n      - metrics/confusion_matrix.csv\n
\n

Stage 2: Training Orchestration with Kubeflow

\n

Kubeflow Pipelines provides scalable, reproducible training on Kubernetes:

\n
# pipelines/training_pipeline.py\nfrom kfp import dsl, compiler\nfrom kfp.dsl import Input, Output, Dataset, Model, Metrics\n\n@dsl.component(\n    base_image=\"python:3.11-slim\",\n    packages_to_install=[\"pandas\", \"scikit-learn\", \"mlflow\"]\n)\ndef train_model(\n    train_data: Input[Dataset],\n    hyperparams: dict,\n    model_artifact: Output[Model],\n    metrics: Output[Metrics],\n):\n    import pandas as pd\n    from sklearn.ensemble import GradientBoostingClassifier\n    from sklearn.metrics import accuracy_score, f1_score\n    import mlflow\n    import joblib\n\n    # Load data\n    df = pd.read_parquet(train_data.path)\n    X = df.drop('target', axis=1)\n    y = df['target']\n\n    # Train with hyperparameters\n    model = GradientBoostingClassifier(\n        learning_rate=hyperparams['learning_rate'],\n        n_estimators=hyperparams['n_estimators'],\n        max_depth=hyperparams['max_depth'],\n    )\n    model.fit(X, y)\n\n    # Log metrics\n    predictions = model.predict(X)\n    accuracy = accuracy_score(y, predictions)\n    f1 = f1_score(y, predictions, average='weighted')\n\n    metrics.log_metric('accuracy', accuracy)\n    metrics.log_metric('f1_score', f1)\n\n    # Save model\n    joblib.dump(model, model_artifact.path)\n\n    # Log to MLflow\n    with mlflow.start_run():\n        mlflow.log_params(hyperparams)\n        mlflow.log_metrics({'accuracy': accuracy, 'f1': f1})\n        mlflow.sklearn.log_model(model, \"model\")\n\n\n@dsl.component(\n    base_image=\"python:3.11-slim\",\n    packages_to_install=[\"evidently\", \"pandas\"]\n)\ndef validate_model(\n    model: Input[Model],\n    test_data: Input[Dataset],\n    baseline_data: Input[Dataset],\n    validation_result: Output[Metrics],\n) -> bool:\n    \"\"\"Validate model against quality gates before promotion.\"\"\"\n    import pandas as pd\n    from evidently.test_suite import TestSuite\n    from evidently.tests import (\n        TestAccuracyScore,\n        TestF1Score,\n        TestColumnDrift,\n    )\n\n    current = pd.read_parquet(test_data.path)\n    reference = pd.read_parquet(baseline_data.path)\n\n    # Run data quality and model performance tests\n    suite = TestSuite(tests=[\n        TestAccuracyScore(gte=0.85),  # Must beat threshold\n        TestF1Score(gte=0.80),\n        TestColumnDrift(column_name='prediction'),\n    ])\n\n    suite.run(reference_data=reference, current_data=current)\n    results = suite.as_dict()\n\n    passed = all(test['status'] == 'SUCCESS' for test in results['tests'])\n    validation_result.log_metric('validation_passed', int(passed))\n\n    return passed\n\n\n@dsl.pipeline(name=\"ml-training-pipeline\")\ndef training_pipeline(\n    dataset_path: str,\n    hyperparams: dict = {\n        'learning_rate': 0.1,\n        'n_estimators': 200,\n        'max_depth': 5,\n    },\n):\n    # Step 1: Prepare data\n    prepare_task = prepare_data(dataset_path=dataset_path)\n\n    # Step 2: Train model\n    train_task = train_model(\n        train_data=prepare_task.outputs['train_data'],\n        hyperparams=hyperparams,\n    )\n\n    # Step 3: Validate model\n    validate_task = validate_model(\n        model=train_task.outputs['model_artifact'],\n        test_data=prepare_task.outputs['test_data'],\n        baseline_data=prepare_task.outputs['baseline_data'],\n    )\n\n    # Step 4: Deploy only if validation passes\n    with dsl.Condition(validate_task.output == True):\n        deploy_model(\n            model=train_task.outputs['model_artifact'],\n            model_name=\"fraud-detector\",\n            namespace=\"ml-serving\",\n        )\n\n\n# Compile and submit\ncompiler.Compiler().compile(training_pipeline, 'pipeline.yaml')\n
\n

Stage 3: Model Serving with Seldon Core

\n

Seldon Core provides production-grade model serving with A/B testing and canary deployments:

\n
# seldon-deployment.yaml\napiVersion: machinelearning.seldon.io/v1\nkind: SeldonDeployment\nmetadata:\n  name: fraud-detector\n  namespace: ml-serving\nspec:\n  predictors:\n    # Canary: New model version (10% traffic)\n    - name: canary\n      traffic: 10\n      componentSpecs:\n        - spec:\n            containers:\n              - name: model\n                image: ghcr.io/myorg/fraud-model:v2.1.0\n                resources:\n                  requests:\n                    cpu: \"2\"\n                    memory: \"4Gi\"\n                  limits:\n                    cpu: \"4\"\n                    memory: \"8Gi\"\n      graph:\n        name: model\n        type: MODEL\n        implementation: SKLEARN_SERVER\n        modelUri: s3://models/fraud-detector/v2.1.0\n        parameters:\n          - name: method\n            type: STRING\n            value: predict_proba\n\n    # Default: Current stable version (90% traffic)\n    - name: default\n      traffic: 90\n      componentSpecs:\n        - spec:\n            containers:\n              - name: model\n                image: ghcr.io/myorg/fraud-model:v2.0.0\n                resources:\n                  requests:\n                    cpu: \"2\"\n                    memory: \"4Gi\"\n      graph:\n        name: model\n        type: MODEL\n        implementation: SKLEARN_SERVER\n        modelUri: s3://models/fraud-detector/v2.0.0\n\n  # A/B test configuration\n  annotations:\n    seldon.io/ab-test: \"true\"\n    seldon.io/canary-promote-threshold: \"0.95\" # Promote if 95%+ accuracy\n
\n

Stage 4: Drift Detection and Auto-Retraining

\n

Models degrade over time as data distributions shift. We detect this automatically:

\n
# monitoring/drift_detector.py\nfrom evidently.metrics import (\n    DataDriftTable,\n    DatasetDriftMetric,\n    ColumnDriftMetric,\n)\nfrom evidently.report import Report\nfrom prometheus_client import Gauge, start_http_server\nimport schedule\nimport time\n\n# Prometheus metrics for Grafana dashboards\ndrift_score = Gauge(\n    'model_data_drift_score',\n    'Current data drift score',\n    ['model_name', 'feature']\n)\nprediction_drift = Gauge(\n    'model_prediction_drift',\n    'Prediction distribution drift',\n    ['model_name']\n)\n\ndef check_drift(model_name: str, reference_data, current_data):\n    \"\"\"Run drift detection and expose metrics.\"\"\"\n    report = Report(metrics=[\n        DatasetDriftMetric(),\n        DataDriftTable(),\n    ])\n\n    report.run(\n        reference_data=reference_data,\n        current_data=current_data,\n    )\n\n    results = report.as_dict()\n\n    # Update Prometheus metrics\n    dataset_drift = results['metrics'][0]['result']['dataset_drift']\n    drift_share = results['metrics'][0]['result']['drift_share']\n\n    prediction_drift.labels(model_name=model_name).set(drift_share)\n\n    # Per-feature drift scores\n    for col_name, col_data in results['metrics'][1]['result']['drift_by_columns'].items():\n        drift_score.labels(\n            model_name=model_name,\n            feature=col_name,\n        ).set(col_data['drift_score'])\n\n    # Trigger retraining if drift exceeds threshold\n    if dataset_drift or drift_share > 0.3:\n        trigger_retraining(model_name)\n\n    return dataset_drift\n\n\ndef trigger_retraining(model_name: str):\n    \"\"\"Trigger Kubeflow pipeline for model retraining.\"\"\"\n    import kfp\n    client = kfp.Client(host='http://kubeflow-pipelines:8888')\n    client.create_run_from_pipeline_package(\n        'pipeline.yaml',\n        arguments={'dataset_path': f's3://data/{model_name}/latest/'},\n        run_name=f'auto-retrain-{model_name}-{int(time.time())}',\n    )\n\n\n# Run drift check every 5 minutes\nschedule.every(5).minutes.do(\n    check_drift,\n    model_name='fraud-detector',\n    reference_data=load_reference_data(),\n    current_data=load_current_data(),\n)\n
\n

GitOps for ML Models

\n

Every model deployment is tracked in Git:

\n
# argocd/ml-models/fraud-detector/kustomization.yaml\napiVersion: kustomize.config.k8s.io/v1beta1\nkind: Kustomization\nresources:\n  - seldon-deployment.yaml\n  - hpa.yaml\n  - monitoring.yaml\n\nimages:\n  - name: ghcr.io/myorg/fraud-model\n    newTag: v2.1.0 # Updated by CI on model promotion\n\npatches:\n  - target:\n      kind: SeldonDeployment\n      name: fraud-detector\n    patch: |\n      - op: replace\n        path: /spec/predictors/0/traffic\n        value: 10  # Canary percentage, gradually increased\n
\n

Results

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
MetricBefore MLOpsAfter MLOps
Model deployment time3-5 days< 1 hour
Models in production315+
Drift detection timeNever (manual)< 5 minutes
Experiment reproducibility0%100%
A/B test capabilityNoneAll models
Rollback timeHours (manual)< 2 minutes (GitOps)
\n

Lessons Learned

\n
    \n
  1. \n

    Version everything: Data, code, models, and infrastructure. If you can't reproduce it, you can't debug it.

    \n
    2. \n
  2. \n

    Separate model training from serving: Training is batch-oriented and GPU-heavy. Serving is latency-sensitive and CPU-heavy. Different scaling patterns require different infrastructure.

    \n
    3. \n
  3. \n

    Drift detection is non-negotiable: A model that worked great at deployment can silently degrade. Statistical tests on incoming data distributions are your early warning system.

    \n
    4. \n
  4. \n

    Canary deployments for models: Never switch 100% traffic to a new model version. Start at 5-10%, monitor key metrics, and gradually promote.

    \n
    5. \n
  5. \n

    Treat ML infrastructure as a platform: Data scientists shouldn't need to know Kubernetes. Provide simple abstractions (submit training job, deploy model, check metrics) that hide infrastructure complexity.

    \n
    6. \n
\n

The MLOps Maturity Roadmap

\n

Start here and progress iteratively:

\n
    \n
  1. Foundation (Month 1-2): MLflow for experiment tracking, DVC for data versioning
    2. \n
  2. Automation (Month 3-4): Kubeflow pipelines for training, model registry for promotion
    3. \n
  3. Serving (Month 5-6): Seldon Core with canary deployments and A/B testing
    4. \n
  4. Monitoring (Month 7+): Drift detection, auto-retraining, SLOs for model performance
    5. \n
\n

The goal isn't to automate everything at once — it's to systematically remove the friction between \"model works in notebook\" and \"model works in production.\"

"}