{"title":"What is ML System Design?","description":"","content":"A model with 95% test accuracy can still fail in production.\n\nLatency spikes, stale predictions, or broken data pipelines aren’t problems with the model itself. They’re failures in the system around it.\n\nML system design focuses on that broader picture. It’s about building reliable pipelines, serving infrastructure, and monitoring, so the model works correctly in real-world conditions, not just in offline evaluation.\n\n---\n\n# The Model is Just the Beginning\n\nModel selection, hyperparameter tuning, and improving accuracy on a held-out test set are important work. But they represent a surprisingly small fraction of what it takes to run ML in production.\n\nA widely cited observation from Google's engineering team puts it bluntly: ML code, the part that trains and runs the model, accounts for roughly 5-10% of the total code in a production ML system. The other 90% is everything else. Data collection and validation, feature extraction and management, serving infrastructure, monitoring, configuration, testing, and the glue code that ties it all together.\n\nHere's what a production ML system actually looks like:\n\n\n```mermaid\nflowchart TD\n subgraph Data[\"Data Layer\"]\n DC[Data
Collection]:::primary\n DV[Data
Validation]:::primary\n FE[Feature
Engineering]:::primary\n end\n\n subgraph Training[\"Training Layer\"]\n TP[Training
Pipeline]:::orange\n EV[Evaluation]:::orange\n EX[Experiment
Tracking]:::orange\n end\n\n subgraph Serving[\"Serving Layer\"]\n MD[Model
Deployment]:::teal\n INF[Inference
Service]:::teal\n FS[Feature
Store]:::teal\n end\n\n subgraph Operations[\"Operations Layer\"]\n MON[Monitoring]:::red\n LOG[Logging]:::red\n AL[Alerting]:::red\n end\n\n DC --> DV --> FE\n FE --> TP --> EV\n EV --> MD --> INF\n INF --> MON\n FS --> INF\n FE --> FS\n MON --> AL\n MON --> DC\n EX --> TP\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef red fill:#ff8787,stroke:#000,color:#000\n```\n\n\nThe model lives inside the training box. Everything outside it, data pipelines, serving systems, monitoring, is what makes it useful in the real world.\n\nA model can achieve state-of-the-art accuracy and still fail if it cannot serve predictions under 100 ms, handle high traffic, or adapt when user behavior changes. At that point, it is not a production system. It is just a research experiment.\n\nThis is the key mindset shift. ML system design is not about choosing the right algorithm. It is about designing the entire system that makes the model work reliably at scale.\n\nConsider YouTube's recommendation engine. Yes, there's a deep learning model at its core. \n\nBut the system also includes data pipelines that process billions of user interactions daily, a feature store that serves hundreds of features per request in single-digit milliseconds, a multi-stage ranking pipeline that narrows millions of candidate videos down to a few dozen, A/B testing infrastructure that evaluates whether a new model actually improves watch time, and monitoring systems that detect when recommendations go off the rails.\n\nThe model is one component among many, and arguably not even the hardest one to get right.\n\n---\n\n# The ML System Lifecycle\n\nEvery production ML system goes through four major phases. Understanding this lifecycle matters because each phase comes with its own challenges, tools, and failure modes.\n\n\n```mermaid\nflowchart LR\n A[Data]:::primary --> B[Training]:::orange\n B --> C[Serving]:::teal\n C --> D[Monitoring]:::red\n D -->|Feedback| A\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef red fill:#ff8787,stroke:#000,color:#000\n```\n\n\nThese phases form a continuous loop, not a one-time pipeline. In production, you don't build a model and walk away. You collect data, train, deploy, monitor, and then use what you learn from monitoring to improve the next iteration. This loop runs indefinitely for as long as the system is in production.\n\n### Data\n\nEverything starts with data. You need to collect it, clean it, validate it, and transform it into features the model can use. This is usually the most time-consuming part. In practice, most of the work in ML is data work.\n\nThe challenge is not just getting data. It's getting the right data, in the right format, at the right time, and making sure it stays reliable. A recommendation system, for example, depends on user click logs, watch histories, search queries, and item metadata. Each comes from different sources, arrives at different times, and brings its own issues such as missing values, duplicates, delays, or schema changes.\n\nFeature engineering turns raw data into signals the model can learn from. A raw timestamp is not useful on its own, but \"hours since the user's last session\" or \"day of the week\" might be highly predictive. Feature engineering is where domain knowledge meets data, and it's often where the biggest accuracy gains come from.\n\n### Training\n\nTraining is more than calling `model.fit()`. You need to split data correctly (avoiding data leakage), select appropriate evaluation metrics, track experiments so you know which configuration produced which results, and validate that the new model is actually better than what's currently in production.\n\nProduction training pipelines run on a schedule. A fraud detection model might retrain every few hours to catch emerging patterns. A recommendation model might retrain daily or weekly. The pipeline needs to be automated, reproducible, and resilient to failures like missing data or infrastructure issues.\n\n### Serving\n\nThis is where the model meets users. Serving is the process of taking a trained model and making its predictions available to applications, whether that's a search results page, a fraud alert, or a content feed. It needs to happen fast (often under 100ms), reliably (99.9%+ uptime), and at scale (thousands to millions of requests per second).\n\nThere are two main serving patterns. Batch inference precomputes predictions for all users or items ahead of time, stores the results, and serves them from a lookup table. It's simple and cheap, but predictions can be stale. Real-time inference runs the model on demand when a request comes in. It's fresh but more complex and expensive. Most production systems use a combination of both.\n\n### Monitoring\n\nMonitoring is the phase most teams underinvest in, and it's where things silently break. Unlike traditional software, where a broken feature usually throws an error, a degrading ML model can keep serving predictions that are technically valid but increasingly wrong.\n\nYou need to monitor the model's prediction quality over time (are click-through rates dropping?), the distribution of input data (has user behavior shifted?), and operational health (latency, throughput, error rates). When monitoring detects a problem, it feeds back into the data phase, triggering a retraining cycle or a deeper investigation.\n\nThe following table summarizes each phase:\n\n\n| Phase | Key Activities | Common Tools | What Goes Wrong |\n|-------|---------------|--------------|-----------------|\n| Data | Collection, cleaning, validation, feature engineering | Spark, Airflow, dbt, Feast | Schema changes, missing data, stale features, data leakage |\n| Training | Model selection, hyperparameter tuning, experiment tracking, validation | PyTorch, TensorFlow, MLflow, W&B | Overfitting, training-serving skew, unreproducible results |\n| Serving | Model deployment, inference, feature retrieval, scaling | TFServing, Triton, KServe, Redis | High latency, scaling bottlenecks, stale batch predictions |\n| Monitoring | Prediction quality tracking, drift detection, alerting | Prometheus, Grafana, Evidently | Silent model degradation, undetected data drift |\n\n\n---\n\n# How ML System Design Differs from Model Building\n\nModel building starts with a clean dataset, tries different algorithms, optimizes a metric, and submits the best result. It's a valuable skill, but it solves a different problem than ML system design.\n\nModel building asks: \"Given this data, what's the best model?\" ML system design asks: \"How do I build a reliable, scalable system that uses ML to solve a business problem?\"\n\nThe difference shows up in what you think about:\n\n\n| Aspect | Model Building | ML System Design |\n|--------|---------------|-----------------|\n| Goal | Maximize accuracy on a test set | Solve a business problem end-to-end |\n| Data | Given to you, already cleaned | You design how to collect, process, and store it |\n| Features | Provided or manually engineered | You decide what features to build and how to serve them |\n| Evaluation | Offline metrics (AUC, F1, RMSE) | Offline metrics + online metrics (CTR, revenue, user engagement) |\n| Latency | Not a concern | Critical constraint (often <100ms) |\n| Scale | Thousands of rows on a laptop | Millions of requests per second in production |\n| Failure modes | Low accuracy | Stale data, training-serving skew, cascading failures, silent degradation |\n| Iteration | One-time competition submission | Continuous improvement over months and years |\n| Cost | Free compute (Kaggle kernels) | Real infrastructure cost (GPU hours, storage, serving) |\n\n\n**A concrete example:** in a Kaggle competition for click-through rate prediction, you might spend days engineering features from the provided dataset and testing different gradient-boosted tree configurations. You never think about where the data comes from, how fast the model needs to run, what happens when the data distribution shifts, or how to roll out the model safely.\n\nIn an ML system design interview for the same problem, you'd need to address all of those questions. How do you collect click logs at scale? How do you compute features in real-time vs batch? How do you serve predictions in under 50ms? How do you detect when user behavior shifts and your model's accuracy degrades? How do you run A/B tests to validate that a new model actually improves revenue? The model architecture might take 5 minutes of the conversation. The system around it takes the other 40.\n\n---\n\n# What Interviewers Look For\n\nML system design interviews evaluate a different set of skills than coding interviews or traditional system design interviews. Understanding what interviewers are looking for helps you focus your preparation in the right areas.\n\nInterviewers evaluate three core competencies:\n\n### **Systems thinking**\n\nCan you design a complete pipeline from data ingestion to model serving? Do you understand how components interact and where bottlenecks emerge? When you propose a candidate retrieval stage, do you think about what feeds it data and what consumes its output?\n\n### **Trade-off analysis**\n\nThere are no perfect solutions in ML system design. Every decision involves trade-offs. Batch inference is cheaper but produces stale predictions. Complex models are more accurate but harder to serve at low latency. Real-time features are powerful but expensive to compute. Interviewers want to see that you can articulate both sides and make a reasoned choice for the given context.\n\n### **Practical awarenes**\n\nDo you know what actually matters in production? Can you anticipate operational challenges like data drift, training-serving skew, or cold-start problems? Have you thought about how to monitor the system after deployment?\n\n\n| Do This | Avoid This |\n| ------------------------------------------------ | ----------------------------------------- |\n| Start by clarifying requirements and constraints | Jump straight to model architecture |\n| Discuss data collection and feature engineering | Assume clean data is available |\n| Consider latency, throughput, and cost | Ignore operational constraints |\n| Propose metrics tied to business outcomes | Only mention offline accuracy metrics |\n| Address failure modes and monitoring | Treat deployment as the finish line |\n| Explain trade-offs for each design decision | Present one approach as obviously correct |\n| Build the architecture incrementally | Draw a complex diagram all at once |\n\n\nOne common misconception is that you need to know the latest model architectures or be able to derive backpropagation on a whiteboard. In practice, what matters far more is the ability to design a system that can reliably deliver ML predictions at scale.\n\nA simple approach like logistic regression, combined with a well-thought-out data pipeline, serving layer, and monitoring, often leads to a stronger solution. In contrast, introducing a cutting-edge transformer adds little value if the training, deployment, and operational aspects are unclear.\n\nThe real signal is not how advanced the model is, but how well the entire system works end to end.\n\n---\n\n# Quiz","pageType":"ml-system-design"}

What is ML System Design?

Get Premium