{"title":"Classification vs Regression","description":"","content":"The previous chapter covered the four learning paradigms. Within supervised learning, the most common paradigm in production, every prediction task falls into one of two categories: **classification** or **regression**. This choice affects your model architecture, loss function, evaluation metrics, and serving infrastructure.\n\nChoosing the wrong framing does not just lead to a suboptimal model, it means optimizing for the wrong objective from the start.\n\n---\n\n# Classification\n\nClassification predicts a discrete category. Given an input, the model assigns it to one of a fixed set of classes.\n\nBinary classification is the simplest and most common variant: two possible outcomes. Is this email spam or legitimate? Is this transaction fraudulent or genuine? Will this user click on this ad or not? These yes/no decisions power some of the highest-impact ML systems in production.\n\nBut classifiers don't actually output a hard yes or no. They output a **probability**, a score between 0 and 1 representing the model's confidence that the input belongs to the positive class. A fraud detection model might output 0.87 for a suspicious transaction, meaning \"87% chance this is fraud.\" The system then applies a **threshold** to convert that probability into a decision.\n\n\n```mermaid\nflowchart LR\n A[Input
Features]:::primary --> B[Classification
Model]:::orange\n B --> C[Probability
Score: 0.87]:::teal\n C --> D{Threshold
≥ 0.5?}:::primary\n D -->|Yes| E[Positive
Class: Fraud]:::red\n D -->|No| F[Negative
Class: Legit]:::green\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef red fill:#ff8787,stroke:#000,color:#000\n```\n\n\nThe threshold is a design decision, not a fixed rule. Setting it at 0.5 seems natural, but it's rarely optimal. Consider fraud detection: a false negative (missing real fraud) costs the company money and erodes user trust.\n\nA false positive (blocking a legitimate transaction) frustrates the user but causes less harm. To catch more fraud, you lower the threshold to 0.3, which flags more transactions as fraudulent. You catch more actual fraud, but you also block more legitimate transactions.\n\nThis is the **precision-recall tradeoff**. Lower thresholds increase recall (catching more positives) at the cost of precision (more false alarms). Higher thresholds increase precision at the cost of recall. The right threshold depends on the business context, not on mathematics.\n\n\n| System | Threshold Strategy | Why |\n|--------|-------------------|-----|\n| Fraud detection | Low threshold (~0.3) | Missing fraud is costly, false alarms can be reviewed manually |\n| Email spam filter | Moderate threshold (~0.5) | False positives (good email in spam) are annoying but recoverable |\n| Medical diagnosis | Very low threshold (~0.1) | Missing a disease can be life-threatening, follow-up tests confirm |\n| Content moderation | Varies by severity | Illegal content: low threshold. Borderline content: higher threshold + human review |\n\n\nThis threshold tuning is one reason why classification systems need **calibrated probabilities**. If the model says 0.7, that should mean roughly 70% of such predictions are actually positive. Poorly calibrated models make threshold selection unreliable.\n\n---\n\n# Multi-Class Classification\n\nBinary classification handles two outcomes. Multi-class classification extends this to N mutually exclusive categories, where each input belongs to exactly one class.\n\nLanguage detection is a clean example. Given a text snippet, the model predicts one of 100+ languages. Product categorization works the same way: an e-commerce listing gets assigned to one category from a taxonomy of thousands. Image classification labels a photo as one of many object types.\n\nThe key word is **mutually exclusive**. Each input gets exactly one label. A product can't be both \"Electronics\" and \"Kitchen Appliances\" in a strict multi-class setup. If it can, that's a multi-label problem (covered in the next section).\n\nMechanically, multi-class models output a probability distribution across all classes using a **softmax** function. The probabilities sum to 1, and the class with the highest probability becomes the prediction.\n\n\n```mermaid\nflowchart LR\n A[Input
Text]:::primary --> B[Multi-Class
Model]:::orange\n B --> C[Softmax
Probabilities]:::teal\n C --> D[\"English: 0.82
French: 0.11
Spanish: 0.04
Other: 0.03\"]:::green\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n```\n\n\nMulti-class classification gets interesting at scale. A few real-world examples:\n\n\n| System | Number of Classes | Challenge |\n|--------|------------------|-----------|\n| Google Search query intent | ~10 categories | Classes are ambiguous, queries can blend intents |\n| YouTube video categorization | ~15 top-level, ~100 fine-grained | Category taxonomy changes over time |\n| Amazon product classification | ~10,000 leaf categories | Extreme class imbalance, some categories have millions of examples, others have dozens |\n| Google Vision API | ~20,000 labels | Hierarchical categories, \"dog\" is also \"animal\" and \"pet\" |\n\n\nWhen the number of classes grows into the thousands, standard softmax becomes expensive because it computes a score for every class on every prediction. Production systems at that scale use techniques like **hierarchical softmax** or **sampled softmax** to reduce computation. A later chapter on algorithms covers these techniques in more detail.\n\nOne subtlety worth noting: multi-class problems sometimes have classes that aren't equally important. In a content safety system with categories like \"safe\", \"mildly offensive\", \"hate speech\", and \"illegal content\", misclassifying illegal content as safe is far worse than the reverse. The evaluation metrics and loss function weights need to reflect these asymmetric costs.\n\n---\n\n# Multi-Label Classification\n\nMulti-label classification allows each input to have **multiple labels simultaneously**. A movie can be both \"Action\" and \"Comedy.\" A news article can cover \"Politics,\" \"Economics,\" and \"Healthcare.\" A social media post can violate multiple content policies at once.\n\nThe architectural difference from multi-class is fundamental. Multi-class uses softmax, which forces probabilities to sum to 1 (the classes compete with each other). Multi-label uses **independent sigmoid functions**, one per label. Each label gets its own probability between 0 and 1, and each label has its own threshold.\n\n\n```mermaid\nflowchart TD\n A[Input:
Movie Description]:::primary --> B[Multi-Label
Model]:::orange\n B --> C1[\"Action: 0.91 ✓\"]:::green\n B --> C2[\"Comedy: 0.78 ✓\"]:::green\n B --> C3[\"Romance: 0.12 ✗\"]:::red\n B --> C4[\"Horror: 0.03 ✗\"]:::red\n B --> C5[\"Sci-Fi: 0.85 ✓\"]:::green\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef red fill:#ff8787,stroke:#000,color:#000\n```\n\n\nThis distinction has real consequences for system design:\n\n\n| Aspect | Multi-Class | Multi-Label |\n|--------|------------|-------------|\n| Output layer | Single softmax | Independent sigmoids |\n| Number of predictions | Exactly one class | Zero or more labels |\n| Probabilities | Sum to 1 | Independent (each 0-1) |\n| Threshold | One (argmax, or top-k) | One per label |\n| Evaluation metrics | Accuracy, macro/micro F1 | Per-label precision/recall, subset accuracy |\n| Training complexity | Standard cross-entropy | Binary cross-entropy per label, or ranking losses |\n| Label correlation | Implicit (softmax competition) | Must be modeled explicitly if needed |\n\n\nThe \"label correlation\" row matters more than it might seem. In multi-label problems, labels are often correlated. Movies tagged as \"Horror\" are more likely to also be tagged as \"Thriller\" than \"Comedy.\"\n\nContent that violates \"hate speech\" policies often also violates \"harassment\" policies. Naive multi-label models with independent sigmoids ignore these correlations. More sophisticated approaches model label dependencies using techniques like label-aware attention or graph neural networks over the label taxonomy.\n\nSystems like YouTube's video tagging, Google's image labeling, and Stack Overflow's question tagging all use multi-label classification. At YouTube's scale, a single video might receive 5-15 tags from a vocabulary of thousands, and the system needs to handle tag correlations, tag hierarchy, and evolving tag vocabularies.\n\n---\n\n# Regression\n\nRegression predicts a continuous numeric value. Instead of assigning inputs to categories, the model outputs a number on a continuous scale.\n\nRegression powers many of the ML systems people interact with daily. Uber's ETA prediction estimates arrival time in minutes. Zillow's Zestimate predicts home values in dollars. Google Ads' bid optimizer predicts the expected revenue from showing a specific ad. YouTube's watch-time prediction estimates how many seconds a user will watch a video (a key signal for ranking).\n\n\n```mermaid\nflowchart LR\n A[Input
Features]:::primary --> B[Regression
Model]:::orange\n B --> C[Continuous
Output: 23.7 min]:::teal\n C --> D[Post-Processing
Round, Clip, Format]:::green\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n```\n\n\nUnlike classification, regression outputs don't have a natural decision boundary. There's no threshold to tune. But regression has its own design considerations:\n\n#### **Bounded vs. unbounded outputs**\n\nSome predictions have natural bounds: a probability score must be between 0 and 1, a rating between 1 and 5, delivery time can't be negative. Others are theoretically unbounded: revenue predictions, price estimates, or time-to-event.\n\nBounded outputs often benefit from output transformations (sigmoid for 0-1, clipping for hard bounds). Unbounded outputs need careful handling of outliers during training, since a few extreme values can dominate the loss function.\n\n#### **Point predictions vs. uncertainty estimates**\n\nA model that predicts \"delivery in 23 minutes\" is useful. A model that predicts \"delivery in 23 minutes, 95% likely between 18 and 31 minutes\" is more useful. Many production regression systems output prediction intervals or full probability distributions rather than single point estimates.\n\nUber shows an ETA range for exactly this reason, a single number creates false precision.\n\n\n| System | Output | Bounded? | Uncertainty Needed? |\n|--------|--------|----------|-------------------|\n| Uber ETA prediction | Minutes | Yes (> 0) | Yes (shown as range) |\n| Zillow home valuation | Dollars | Yes (> 0) | Yes (Zestimate range) |\n| YouTube watch time | Seconds | Yes (0 to video length) | No (used for ranking only) |\n| Google Ads revenue | Dollars per impression | Yes (≥ 0) | Sometimes (for bidding confidence) |\n| Weather temperature forecast | Degrees | No (can be negative) | Yes (confidence intervals) |\n| Credit scoring | Score (300-850) | Yes | No (but calibration matters) |\n\n\nRegression evaluation uses different metrics than classification. Instead of accuracy or precision, regression models are evaluated by how far off their predictions are: mean absolute error (MAE), mean squared error (MSE), or root mean squared error (RMSE).\n\nThe choice between these depends on how much you want to penalize large errors. MSE penalizes large errors quadratically, making it sensitive to outliers. MAE treats all errors linearly. A later chapter covers loss functions and evaluation metrics in detail.\n\n---\n\n# Classification vs Regression: A Comparison\n\nThe two task types differ across several dimensions:\n\n\n| Dimension | Classification | Regression |\n|-----------|---------------|------------|\n| Output type | Discrete categories | Continuous values |\n| Model output | Probabilities per class | Single numeric value (or distribution) |\n| Decision boundary | Threshold converts probability to class | No threshold needed |\n| Key metrics | Accuracy, Precision, Recall, F1, AUC | MAE, MSE, RMSE, R-squared |\n| Calibration | Probability calibration (Platt scaling, isotonic regression) | Prediction interval calibration |\n| Error types | False positives, false negatives | Over-prediction, under-prediction |\n| Business impact of errors | Asymmetric (missing fraud vs. false alarm) | Proportional to magnitude (off by $10 vs. $10,000) |\n\n\n### When the Boundary Blurs\n\nThe line between classification and regression isn't always clean. Several common scenarios sit in the gray area:\n\n#### **Ordinal classification**\n\nStar ratings (1-5 stars), severity levels (low/medium/high/critical), and customer satisfaction scores (1-10) have a natural ordering. You could treat these as multi-class classification (ignoring the order) or as regression (treating them as continuous). Neither is perfect. \n\nMulti-class classification ignores that predicting 4 when the answer is 5 is better than predicting 1. Regression assumes equal intervals between values, which is rarely true (the difference between 1-star and 2-stars isn't the same as between 4-stars and 5-stars).\n\n#### **Discretized regression**\n\nSometimes a regression problem is easier to solve as classification. Predicting exact house prices is hard. Predicting price buckets ($0-200K, $200K-400K, $400K-600K, etc.) is easier and might be sufficient for the use case. Uber's surge pricing model doesn't predict an exact multiplier. It predicts a bucket (1x, 1.2x, 1.5x, 2x), which is effectively classification over a set of multiplier tiers.\n\n#### **Classification with probability outputs used as scores**\n\nClick-through rate prediction is technically binary classification (clicked or didn't), but the output probability itself is the valuable signal. Ad ranking systems use the predicted click probability directly as a score to rank ads, not as a yes/no decision. The model is a classifier by architecture, but the system uses it as a regression model in practice.\n\n---\n\n# Framing the Problem: Classification or Regression?\n\nThe same real-world problem can often be framed either way. The right framing depends on what the system needs to do with the prediction.\n\n\n| Problem | Classification Framing | Regression Framing | Better Choice |\n|---------|----------------------|-------------------|---------------|\n| Will a user watch this video? | Binary: watch/skip | Predicted watch time (seconds) | **Regression.** YouTube ranks by predicted watch time, not binary will-watch. The magnitude matters. |\n| Will a user click this ad? | Binary: click/no-click | Predicted click probability | **Classification.** But the probability score is used as a ranking signal, so calibration matters. |\n| How long will delivery take? | Buckets: <15 min, 15-30 min, 30-45 min, 45+ min | Predicted minutes | **Regression.** Users need specific ETAs, not ranges. Classification loses information. |\n| Is this content toxic? | Binary: toxic/safe, or multi-class severity | Toxicity score (0-1) | **Classification.** The action (remove, warn, allow) is discrete. A severity score adds nuance. |\n| What rating will a user give? | Multi-class: 1-5 stars | Predicted rating (continuous) | **Regression.** Predicting 3.7 vs 3.2 affects ranking. But consider ordinal approaches. |\n| Will this machine fail? | Binary: fail within 30 days / won't fail | Predicted days until failure | **Regression** (time-to-event). Knowing \"failure in 5 days\" is more actionable than \"will fail.\" |\n\n\nA few principles guide the decision:\n\n\n```mermaid\nflowchart TD\n A[What does the
downstream system need?]:::primary\n A --> B{Is the action
discrete?}:::orange\n B -->|Yes: approve/reject,
show/hide| C[Classification]:::green\n B -->|No: rank, estimate,
set a value| D{Does magnitude
matter?}:::orange\n D -->|Yes: how much,
how long, how many| E[Regression]:::teal\n D -->|No: just rank
relative order| F[Classification with
probability as score]:::green\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n```\n\n\n#### **Start with the downstream action**\n\nIf the system needs to make a binary or categorical decision (block this content, approve this loan, route this ticket), classification is natural. If the system needs a magnitude (how long, how much, how many), regression fits better.\n\n#### **Consider what information matters**\n\nBinary classification collapses information. \"This user will engage\" throws away whether they'll engage for 5 seconds or 50 minutes. If that difference matters for ranking or business decisions, regression preserves it.\n\n#### **Think about evaluation**\n\nClassification is easier to evaluate with stakeholders. \"We catch 95% of fraud\" is intuitive. \"Our RMSE is 3.2 minutes\" requires more context to interpret. If you need to communicate model quality to non-technical stakeholders, classification metrics are more accessible.\n\n#### **Consider label availability**\n\nRegression labels need to be continuous and accurate. If your labels are noisy (user ratings are subjective, delivery times have measurement error) or naturally discrete (click/no-click), forcing regression can hurt performance. Classification is more robust to label noise because it only needs to get the category right, not the exact value.\n\nOne more real-world pattern worth knowing: many production systems use both. Netflix predicts whether you'll watch a show (classification) AND how much you'll enjoy it (regression) AND how long you'll watch (regression). Different predictions feed different parts of the system.\n\nThe home feed ranking uses predicted watch time. The \"match score\" percentage uses the enjoyment prediction. Content acquisition decisions use aggregate engagement classification models.\n\n---\n\n# Quiz","pageType":"ml-system-design"}

Classification vs Regression

Get Premium