{"title":"Common ML Algorithms","description":"","content":"The previous chapter introduced classification and regression, the two core tasks in supervised learning. But identifying the task is only the starting point. It does not tell you which algorithm to choose.\n\nTake fraud detection as an example. You could build it using logistic regression, a gradient-boosted tree, or a neural network. Each option comes with very different tradeoffs in training cost, serving latency, interpretability, and accuracy.\n\nIn practice, most production ML systems rely on a small set of algorithm families. Knowing when to use each one is one of the most practical and high-impact skills in ML system design.\n\n---\n\n# Linear Models\n\nLinear models are the oldest and simplest family, yet they remain one of the most widely used in production. The idea is straightforward: take a weighted sum of input features, optionally pass it through a sigmoid function for classification, and output a prediction.\n\nLogistic regression applies a sigmoid to the weighted sum to produce a probability between 0 and 1. Linear regression skips the sigmoid and outputs the weighted sum directly as a continuous value. In both cases, the model learns by adjusting weights to minimize a loss function on training data.\n\n\n```mermaid\nflowchart LR\n A[Input
Features]:::primary --> B[Weighted
Sum]:::orange\n B --> C{Task?}:::teal\n C -->|Classification| D[Sigmoid]:::orange\n D --> E[Probability
0 to 1]:::green\n C -->|Regression| F[Direct
Output]:::green\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n```\n\n\nSo why do these simple models persist when more powerful alternatives exist? The reasons are practical.\n\n#### **Speed**\n\nLinear models train in minutes on billions of examples and serve predictions in microseconds. Google's early ad click-through rate prediction system ran on logistic regression at massive scale, handling billions of predictions per day.\n\nWhen your system needs to score thousands of candidates per request with single-digit millisecond latency budgets, microsecond inference matters.\n\n#### **Interpretability**\n\nEach feature gets a weight, and you can directly read which features push predictions up or down. Credit scoring models at banks use logistic regression partly because regulators require explanations for why a loan was denied.\n\nYou can point to the specific feature weights: high debt-to-income ratio contributed -0.8, short credit history contributed -0.3. Try explaining a 200-tree gradient-boosted model to a regulator.\n\n#### **Stability**\n\nSmall changes in training data produce small changes in predictions. There are no wild swings from random initialization or hyperparameter sensitivity. This predictability matters when models run in high-stakes production systems.\n\n\n| Property | Logistic Regression | Linear Regression |\n|----------|-------------------|-------------------|\n| Task type | Classification | Regression |\n| Output | Probability (0-1) via sigmoid | Continuous value |\n| Training speed | Minutes on billions of rows | Minutes on billions of rows |\n| Serving latency | Microseconds | Microseconds |\n| Interpretability | High (feature weights) | High (feature weights) |\n| Key limitation | Linear decision boundary | Assumes linear relationship |\n| Best use | Baseline classifier, regulated domains | Baseline predictor, simple relationships |\n\n\nThe fundamental limitation is expressiveness. Linear models can only learn linear decision boundaries. If the relationship between features and the target is nonlinear (and it usually is), the model underfits. \n\nYou can partially fix this by manually engineering nonlinear features: polynomial terms, interaction features, bucketed numericals. Google's ad system used logistic regression with billions of manually engineered feature crosses. But this pushes complexity from the model into the feature engineering pipeline, which brings its own maintenance burden.\n\nThis is the \"feature engineering tax.\" Linear models are cheap to run but expensive to feed. Every nonlinear pattern you want the model to capture requires an engineer to identify it, build it, and maintain it. As the number of patterns grows, this approach doesn't scale.\n\nDespite this limitation, linear models serve a critical role: they're the baseline every production team should start with. If a logistic regression achieves 90% of the performance you need with 10% of the complexity, that's often the right trade-off. You always want to know how much accuracy the fancier model actually buys you.\n\n---\n\n# Tree-Based Models\n\nTree-based models work fundamentally differently from linear models. Instead of learning a weighted sum, they learn a series of if-else decision rules that recursively split the data.\n\nA single decision tree asks questions like \"Is the transaction amount greater than $5,000?\" and \"Is the merchant category 'electronics'?\" at each node, splitting the data until it reaches a leaf node with a prediction.\n\nThis is intuitive and highly interpretable. But single decision trees have a serious problem: they overfit badly. A deep tree memorizes the training data instead of learning generalizable patterns.\n\nTwo techniques fix this, and the difference between them matters.\n\n**Random forests** train many trees in parallel, each on a random subset of the data and features, then average their predictions. This reduces overfitting through diversity: individual trees make different errors, and averaging cancels them out. Random forests are solid, reliable, and hard to misconfigure. But they're not state-of-the-art.\n\n**Gradient-boosted trees** (GBTs) take a different approach. Instead of training trees independently, they train trees sequentially. The first tree makes predictions. The second tree focuses on the errors the first tree made. The third tree focuses on the remaining errors. Each successive tree corrects the mistakes of all previous trees. The final prediction is the sum of all trees' contributions.\n\n\n```mermaid\nflowchart TD\n A[Single Decision Tree]:::red --> B[Problem: Overfits badly]:::red\n B --> C{Solution?}:::orange\n\n C --> D[Random Forest]:::teal\n D --> D1[Train 100+ trees
in parallel on random
data subsets]:::teal\n D1 --> D2[Average predictions
to reduce overfitting]:::teal\n\n C --> E[Gradient-Boosted Trees]:::primary\n E --> E1[Train trees sequentially,
each correcting errors
of the previous ones]:::primary\n E1 --> E2[Sum predictions
for final output]:::primary\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef red fill:#ff8787,stroke:#000,color:#000\n```\n\n\nThe gradient boosting mechanism is worth understanding in more detail, since it's the algorithm behind XGBoost, LightGBM, and CatBoost, the three most popular implementations.\n\n\n```mermaid\nflowchart LR\n A[Training
Data]:::primary --> B[Tree 1]:::orange\n B --> C[Residual
Errors]:::red\n C --> D[Tree 2]:::orange\n D --> E[Residual
Errors]:::red\n E --> F[Tree 3]:::orange\n F --> G[...]:::teal\n G --> H[Final Prediction
= Sum of All Trees]:::green\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef red fill:#ff8787,stroke:#000,color:#000\n```\n\n\nEach tree is typically shallow (depth 4-8), so it captures a simple pattern. But hundreds of these shallow trees stacked together can model complex nonlinear relationships. The learning rate controls how much each tree contributes, preventing any single tree from dominating.\n\nGBTs dominate tabular data in production, and understanding why is important for making algorithm choices.\n\n#### **They handle mixed feature types natively**\n\nTabular data has categorical columns (merchant category, device type), numerical columns (transaction amount, user age), and often missing values. GBTs handle all of these without preprocessing. You don't need to one-hot encode categoricals or impute missing values. The tree splits naturally partition both types.\n\n#### **They capture feature interactions automatically**\n\nA tree that splits on \"transaction amount > $5,000\" at the root and \"merchant category = electronics\" at the next level has learned an interaction: high-value electronics purchases behave differently. Linear models need these interactions engineered manually.\n\n#### **They train fast and serve fast**\n\nTraining on a dataset with millions of rows and hundreds of features takes minutes to hours on CPU. Serving requires traversing a few hundred shallow trees, which completes in sub-millisecond time. No GPU needed.\n\nAirbnb used XGBoost as the core of their search ranking system for years. Uber's ETA prediction runs on gradient-boosted trees with features like distance, traffic conditions, and time of day. Fraud detection systems at companies like Stripe and PayPal rely heavily on GBTs because the data is inherently tabular (transaction metadata) and the models need to serve in real-time.\n\nIn Kaggle competitions, GBTs win the vast majority of tabular data challenges. This isn't just an academic observation. It reflects a genuine performance advantage that carries over to production.\n\n\n| Property | Decision Tree | Random Forest | Gradient-Boosted Trees |\n|----------|--------------|---------------|----------------------|\n| Number of trees | 1 | 100-1000 (parallel) | 100-1000 (sequential) |\n| Overfitting risk | High | Low | Medium (needs tuning) |\n| Training speed | Fast | Moderate (parallelizable) | Moderate (sequential) |\n| Interpretability | High | Low | Low |\n| Accuracy (tabular) | Low-medium | Medium-high | High |\n| Key tuning params | Max depth, min samples | Number of trees, max features | Learning rate, number of trees, max depth |\n| Production usage | Rare standalone | Moderate | Very common |\n\n\nGBTs do have limitations. They extrapolate poorly: if the training data contains transaction amounts from $1 to $10,000, the model has no principled way to handle a $100,000 transaction. They also struggle with very high-dimensional sparse data (like raw text represented as bag-of-words with millions of vocabulary terms), where neural approaches have an advantage.\n\n---\n\n# Neural Networks\n\nNeural networks stack layers of artificial neurons, where each neuron computes a weighted sum of its inputs, adds a bias, and passes the result through a nonlinear activation function. The power comes from depth: multiple layers can learn hierarchical representations, capturing increasingly abstract patterns.\n\nThree architectures cover the major use cases in production ML.\n\n### Feedforward Networks (MLPs)\n\nA multi-layer perceptron (MLP) is the simplest neural network: fully connected layers stacked on top of each other. Input features pass through hidden layers, each applying a learned transformation, and produce an output.\n\nMLPs on their own aren't special for tabular data. They rarely outperform gradient-boosted trees. Where they shine is as a component in larger systems. YouTube's Wide & Deep architecture combines a linear model (the \"wide\" path for memorization) with an MLP (the \"deep\" path for generalization). \n\nThe MLP takes dense feature vectors, including pre-computed embeddings, and learns complex interaction patterns that the linear component misses. This architecture and its variants power ranking systems across Google's products.\n\n### CNNs for Images\n\nConvolutional neural networks exploit a key property of images: spatial locality. A pixel's meaning depends on its neighbors. Convolutional filters slide across the image, detecting patterns like edges, textures, and shapes at progressively higher abstraction levels. Early layers detect edges. Middle layers detect parts (eyes, wheels). Deep layers detect objects.\n\nThis hierarchical feature learning is what makes CNNs transformative. Before CNNs, image classification required hand-engineered features (color histograms, edge detectors, texture descriptors). An engineer had to decide which visual patterns mattered. CNNs learn these features directly from raw pixels, and they learn better features than humans design.\n\nGoogle Photos uses CNNs for image search and organization. Pinterest's visual search lets users find products by photographing them, powered by CNN-based visual similarity. Content moderation systems at every major platform use CNNs to detect policy violations in uploaded images and video frames.\n\n### RNNs and LSTMs for Sequences\n\nRecurrent neural networks process data one step at a time, maintaining a hidden state that carries information from previous steps. This makes them a natural fit for sequential data: text (word by word), time series (timestep by timestep), and user event sequences (action by action).\n\nStandard RNNs struggle with long sequences because gradients either vanish (becoming too small to learn from) or explode (becoming too large and destabilizing training) as they propagate through many timesteps. LSTMs (Long Short-Term Memory networks) fix this with a gating mechanism that controls what information to keep, forget, or output at each step.\n\nBefore transformers took over NLP, LSTMs were the dominant architecture for text processing. They still appear in production systems for time-series forecasting and session-based recommendation, where the sequential nature of user behavior is the primary signal. Netflix has used recurrent architectures to model viewing sessions, predicting what a user wants to watch next based on their recent activity.\n\nTransformers have largely replaced RNNs for text and are expanding into vision and audio. They process all positions in parallel (rather than sequentially) and use attention mechanisms to capture long-range dependencies. This chapter won't deep-dive into transformers, but they represent the current frontier for sequence modeling and increasingly for other data types.\n\n\n| Architecture | Best for | Input type | Key strength | Key weakness | Example system |\n|-------------|----------|-----------|-------------|-------------|---------------|\n| Feedforward (MLP) | Ranking, combined features | Engineered feature vectors | Learns complex interactions | Needs feature engineering input | YouTube deep ranking |\n| CNN | Image/video understanding | Raw pixels | Learns spatial features from data | Compute-intensive training | Google Photos, Pinterest |\n| RNN/LSTM | Sequences, time series | Ordered sequences | Captures temporal patterns | Slow training, struggles with long sequences | Netflix session modeling |\n| Transformer | Text, multimodal | Tokens, patches | Parallelizable, long-range context | Very compute-intensive | Google Search, ChatGPT |\n\n\n\n```mermaid\nflowchart TD\n NN[Neural Networks]:::primary\n\n NN --> MLP[MLP]:::orange\n NN --> CNN[CNN]:::teal\n NN --> RNN[RNN / LSTM]:::orange\n NN --> TF[Transformer]:::teal\n\n MLP --> MLP1[Tabular data +
engineered features]:::green\n CNN --> CNN1[Images, video,
spatial data]:::green\n RNN --> RNN1[Time series,
event sequences]:::green\n TF --> TF1[Text, multimodal,
long sequences]:::green\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n```\n\n\nThe common thread across all neural architectures: they learn feature representations from raw data. You don't need to manually engineer \"is there a face in this image\" or \"does this sentence express negative sentiment.\" The network discovers these patterns on its own. The trade-off is that this learning requires large datasets (often hundreds of thousands to millions of examples), expensive compute (GPU training for hours or days), and the resulting models are difficult to interpret and debug.\n\n---\n\n# When to Use What\n\nChoosing an algorithm isn't about which one is \"best\" in the abstract. It's about which one fits the constraints of your specific system.\n\nThe single strongest signal is **data type**. This one factor drives the decision more than anything else.\n\nFor **tabular data** (structured rows and columns: user attributes, transaction metadata, sensor readings), the path is well-established. Start with logistic regression or linear regression as a baseline. If you need better accuracy, move to gradient-boosted trees. XGBoost or LightGBM will almost always outperform the linear baseline, often significantly. \n\nNeural networks for tabular data rarely outperform well-tuned GBTs, and they're harder to train, serve, and debug. Research papers occasionally show neural networks matching GBTs on tabular benchmarks, but in practice the operational overhead isn't worth marginal gains.\n\nFor **unstructured data** (images, text, audio, video), neural networks are the only practical choice. You can't hand-engineer features from raw pixels or text tokens at the scale production systems require. A CNN will extract visual features that no amount of manual feature engineering could match. A transformer will capture language patterns that bag-of-words models miss entirely.\n\nBeyond data type, four secondary factors shape the decision:\n\n#### **Interpretability**\n\nRegulated industries (banking, healthcare, insurance) often require model explanations. Linear models provide inherent interpretability through feature weights. GBTs can be explained post-hoc using tools like SHAP, but the explanations are approximations. Neural networks are the hardest to interpret. If a regulator asks \"why was this loan denied?\", your answer needs to be specific and defensible.\n\n#### **Training budget**\n\nLinear models train in minutes on CPU. GBTs train in minutes to hours on CPU. Neural networks train in hours to days on GPU. If you're iterating quickly on a new problem and need to test dozens of feature combinations, the fast feedback loop of linear models or GBTs is valuable. If you have a mature system where 0.1% accuracy improvement justifies GPU costs, neural networks become viable.\n\n#### **Serving latency**\n\nLinear models serve in microseconds. GBTs serve in sub-millisecond time. Small MLPs take 1-5ms. CNNs and transformers need 10-100ms and often require GPU inference. A system with a 5ms total latency budget can't afford a transformer in the serving path unless it pre-computes predictions in batch.\n\n#### **Data volume**\n\nLinear models can learn useful patterns from thousands of examples. GBTs need tens of thousands. Neural networks typically need hundreds of thousands to millions before they outperform simpler alternatives. With a small dataset, a complex model just memorizes the training data.\n\n\n```mermaid\nflowchart TD\n A[What type of
data?]:::primary\n A -->|Tabular| B[Start with
linear model]:::green\n B --> C{Need better
accuracy?}:::orange\n C -->|Yes| D[Gradient-Boosted
Trees]:::teal\n C -->|No| B1[Ship it]:::green\n D --> E{Still not
enough?}:::orange\n E -->|Rarely| F[Try MLP, but expect
diminishing returns]:::red\n E -->|Usually sufficient| D1[Ship it]:::green\n\n A -->|Unstructured| G[Neural Network]:::teal\n G --> H{Image/video?}:::orange\n H -->|Yes| I[CNN or
Vision Transformer]:::green\n H -->|No| J{Text/sequence?}:::orange\n J -->|Yes| K[Transformer
or RNN]:::green\n J -->|Other| L[Task-specific
architecture]:::green\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef red fill:#ff8787,stroke:#000,color:#000\n```\n\n\nOne principle ties this together: **start simple and upgrade when measured improvements justify added complexity.** Every production team should know how well a logistic regression or simple GBT performs on their problem before reaching for neural networks. If the simple model gets you 90% of the way, the remaining 10% may not be worth the operational burden of training, serving, and debugging a deep network.\n\n\n| Factor | Linear Models | Gradient-Boosted Trees | Neural Networks |\n|--------|--------------|----------------------|----------------|\n| Tabular data | Good baseline | Best choice | Rarely outperforms GBTs |\n| Image data | Not applicable | Not applicable | Best choice (CNN) |\n| Text data | Bag-of-words baseline | With TF-IDF features | Best choice (Transformer) |\n| Sequence data | Not applicable | With lag features | Best choice (RNN/Transformer) |\n| Training time | Minutes | Minutes to hours | Hours to days |\n| Serving latency | Microseconds | Sub-millisecond | Milliseconds (GPU often needed) |\n| Minimum data needed | ~1,000 examples | ~10,000 examples | ~100,000 examples |\n| Interpretability | High | Medium (SHAP) | Low |\n| Feature engineering effort | Heavy (manual) | Moderate | Minimal (learns from raw data) |\n\n\n---\n\n# Why Two Worlds Exist\n\nThe split between \"GBTs for tabular, neural nets for unstructured\" isn't arbitrary. It reflects a fundamental difference in how these data types are structured.\n\nTabular data comes with pre-defined, semantically meaningful columns. \"Transaction amount,\" \"merchant category,\" \"user age\" are already meaningful features. Tree-based models exploit this structure directly. A decision tree split on \"transaction amount > $5,000\" is a natural, axis-aligned partition that matches how the data is organized. Trees are essentially learning a set of rules over pre-defined features, and that's exactly what tabular data provides.\n\nNeural networks don't have this structural advantage on tabular data. Their gradient-based optimization has to learn which feature boundaries matter from scratch, without the inductive bias that axis-aligned splits provide. \n\nThe heterogeneous nature of tabular features (some categorical, some numerical, some ordinal, some with missing values) also creates challenges for the uniform tensor operations that neural networks rely on.\n\nUnstructured data is the opposite situation. Raw pixels, text tokens, and audio waveforms carry no pre-defined semantic meaning. Pixel 1,247 in an image means nothing on its own. The meaning emerges from spatial patterns across thousands of pixels. No human can manually specify the features that distinguish a cat from a dog at production quality. \n\nNeural networks learn these hierarchical representations automatically: edges combine into textures, textures into parts, parts into objects. Trees fundamentally can't do this because a decision tree can't split on \"this region of the image contains a face.\"\n\nIn practice, production systems often combine both approaches. A recommendation system might use a neural network to generate embeddings from item images and text descriptions (extracting features from unstructured data), then feed those embeddings as input features into a gradient-boosted tree alongside tabular features like item popularity, user tenure, and time of day.\n\n\n```mermaid\nflowchart LR\n subgraph Unstructured\n A1[Images]:::primary --> B1[CNN]:::orange\n A2[Text]:::primary --> B2[Transformer]:::orange\n end\n\n B1 --> C[Learned
Embeddings]:::teal\n B2 --> C\n\n subgraph Tabular\n D1[User features]:::primary\n D2[Item features]:::primary\n D3[Context features]:::primary\n end\n\n C --> E[Combined
Feature Vector]:::teal\n D1 --> E\n D2 --> E\n D3 --> E\n\n E --> F[Gradient-Boosted
Tree]:::orange\n F --> G[Final
Prediction]:::green\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n```\n\n\nThis hybrid pattern is common at companies operating at scale. \n\n---\n\n# Quiz","pageType":"ml-system-design"}

Common ML Algorithms

Get Premium