MLOps and LLMOps in Production

Frame the lifecycle of a production ML system and set up reproducible project structure.

• The ML lifecycle and where projects fail after the notebook stage
• MLOps versus LLMOps: shared practices and key differences
• Project structure, environments, and dependency pinning
• Data and model versioning with DVC
• Reproducibility: seeds, configs, and data snapshots
• Roles and handoffs between data science, engineering, and operations

Hands-on lab: Convert a provided notebook into a structured Python project with pinned dependencies and DVC-versioned data, then reproduce a training run from a clean clone.

Record experiments systematically and manage model versions through a registry.

• Logging parameters, metrics, and artifacts with MLflow
• Comparing runs and selecting candidates
• Model signatures and packaging formats
• Registry stages, aliases, and approval workflows
• Linking models to code commits and datasets
• Metadata needed for audits and rollbacks

Hands-on lab: Run a hyperparameter sweep tracked in MLflow, register the best model, and promote it through staging with a written approval note.

Automate testing and deployment, and expose models through dependable serving patterns.

• Testing ML code: data checks, unit tests, and model quality gates
• Building pipelines with GitHub Actions
• Containerizing a model with Docker
• Batch, real-time, and streaming serving patterns
• Building an inference API with FastAPI
• Canary and blue-green releases with rollback plans

Hands-on lab: Build a pipeline that tests, containerizes, and deploys a model API, including a quality gate that blocks a deliberately degraded model.

Watch live systems for technical and statistical problems and decide when to retrain.

• Operational metrics: latency, errors, throughput, resource use
• Data drift, concept drift, and prediction drift
• Drift reports with Evidently
• Dashboards and alerts with Prometheus and Grafana
• Delayed ground truth and proxy metrics
• Retraining triggers and safe automated redeployment

Hands-on lab: Simulate drifted traffic against your deployed model, produce a drift report, configure an alert, and document the retraining decision.

Apply operational discipline to LLM applications, where outputs are open-ended and costs scale with usage.

• Building evaluation datasets and rubrics for LLM tasks
• Automated checks and LLM-as-judge, with known weaknesses
• Tracing prompts, retrieval steps, and tool calls with LangSmith or Langfuse
• Prompt and model versioning with regression tests
• Token usage, caching, model routing, and rate limits
• Guardrails against prompt injection and unsafe output

Hands-on lab: Instrument a retrieval-based assistant with tracing, build a 30-case evaluation set, compare two prompt versions, and set a per-request cost budget.

Add the controls organizations need for responsible operation and bring the full stack together.

• Model cards, documentation, and audit trails
• Access control, secrets management, and data residency
• Risk tiers and approval processes for AI systems
• Incident response and post-incident review for ML systems
• Overview of frameworks such as the NIST AI RMF and the EU AI Act
• Architecture trade-offs: managed platforms versus self-hosted stacks

Hands-on lab: Complete your capstone pipeline, write its model card and runbook, and defend your architecture and monitoring choices in a peer design review.