Probability Estimates from Linear Regression Analysis

What Linear Regression Actually Gives You

Linear regression outputs raw numbers. Not probabilities. Not percentages. Just continuous values that can swing anywhere from negative infinity to positive infinity, depending on your input data. That's the bitter truth nobody tells you upfront. When you run a linear regression, you're getting predicted values on whatever scale your target variable lives on. Sell 100 units or 1,000,000. Temperature in Kelvin or Celsius. The model doesn't care about probability—it just draws a line through your data and tells you where that line lands for any given input. This matters because most real-world decisions need probability estimates. Will this customer convert? Is this transaction fraudulent? Should I approve this loan? These questions demand answers between 0 and 1, not arbitrary real numbers.

The Core Problem with Raw Linear Outputs

Linear regression predictions can exceed 1 and drop below 0. That's not a bug—it's just math working as designed. The model optimizes for minimizing squared error, not for producing valid probability distributions. Consider a model predicting loan default. Your linear regression might output -0.3 for one applicant and 2.7 for another. What does -0.3 even mean? Is that a negative default? The model has no built-in mechanism for handling this. Three fundamental issues emerge: Unbounded predictions. Values can land anywhere on the real number line. No probabilistic interpretation. A prediction of 50 doesn't mean 50% chance—it means something 50 units above the baseline. Heteroscedasticity issues. Prediction errors tend to cluster differently at different parts of the scale.

Methods to Extract Probability Estimates

You have several paths forward. Each has tradeoffs.

Logistic Regression: The Proper Solution

Logistic regression exists specifically because linear regression fails at probability estimation. It models the log-odds of an outcome using a sigmoid function that naturally constrains outputs between 0 and 1. The sigmoid function transforms any real number into a probability. A raw prediction of 0 becomes 0.5. Large positive values approach 1. Large negative values approach 0. If you're building a binary classifier and need probabilities, use logistic regression from the start. Don't retrofit linear regression to do a job it wasn't designed for. Implementation is straightforward:

import numpy as np
from sklearn.linear_model import LogisticRegression

model = LogisticRegression()
model.fit(X_train, y_train)

probabilities = model.predict_proba(X_test)[:, 1]

That's it. The predict_proba method gives you exactly what you need—clean probability estimates that sum to 1 across classes.

Converting Linear Predictions

Sometimes you're stuck with a linear regression model. Maybe it's legacy code. Maybe you've already built extensive validation around it. In these cases, you can apply post-hoc transformations. Linear to probability conversion methods:

Min-max scaling: Map predictions to [0,1] based on observed training range. Simple but fragile—it breaks if new data falls outside historical bounds.
Logit transformation: Apply the inverse sigmoid function to convert bounded predictions back to probabilities. Requires predictions to stay strictly between 0 and 1.
Platt scaling: Fit a logistic regression on the raw linear predictions to learn an optimal transformation. Works well when linear predictions have a monotonic relationship with true probability.
Isotonic regression: Non-parametric calibration that monotonically maps predictions to probabilities while preserving rank order.

Platt scaling tends to work best in practice. The sklearn library makes implementation trivial:

from sklearn.calibration import CalibratedClassifierCV
from sklearn.linear_model import LinearRegression

# Train your linear model
lr = LinearRegression()
lr.fit(X_train, y_train)

# Calibrate using Platt scaling
calibrated = CalibratedClassifierCV(lr, method='sigmoid', cv='prefit')
calibrated.fit(X_train, y_train)
probabilities = calibrated.predict_proba(X_test)[:, 1]

Probit and Complementary Log-Log Models

Logistic regression uses a symmetric sigmoid, but sometimes your data doesn't follow that pattern. Probit regression uses the normal cumulative distribution function instead, producing slightly different probability curves. Complementary log-log (cloglog) models are useful when probabilities approach 0 or 1 asymmetrically. These models handle cases where the outcome has natural boundaries that logistic regression struggles with.

Comparing the Methods

Method	When to Use	Pros	Cons
Logistic Regression	Building from scratch, binary outcomes	Native probabilities, interpretable coefficients	Requires model retraining
Platt Scaling	Retrofitting existing linear models	Post-hoc solution, preserves ranking	Needs held-out calibration data
Isotonic Regression	Large datasets, non-monotonic relationships	Flexible, no distributional assumptions	Can overfit, requires more data
Min-Max Scaling	Quick fixes, bounded known ranges	Simple, fast	Fragile, breaks with new data ranges
Probit/Cloglog	Asymmetric probability distributions	Handles boundary effects better	Less common, harder to interpret

Getting Started: Practical Implementation

Here's a complete workflow for extracting probability estimates from regression analysis: Step 1: Assess your situation Are you starting fresh or working with an existing linear model? Fresh starts mean logistic regression. Existing models mean calibration. Step 2: Prepare your data Binary outcomes need to be encoded as 0/1. Continuous targets need conversion to binary classification if you want probabilities. Decide your threshold—what constitutes a "positive" outcome? Step 3: Train or calibrate For new models:

from sklearn.linear_model import LogisticRegression

model = LogisticRegression(solver='lbfgs', max_iter=1000)
model.fit(X_train, y_train)

# Get probabilities directly
probs = model.predict_proba(X_test)[:, 1]

For existing models:

from sklearn.calibration import CalibratedClassifierCV

calibrated_model = CalibratedClassifierCV(
    estimator=your_linear_model,
    method='sigmoid',  # or 'isotonic'
    cv=5
)
calibrated_model.fit(X_train, y_train)
probs = calibrated_model.predict_proba(X_test)[:, 1]

Step 4: Validate calibration Use reliability curves (calibration plots) to check if your probabilities match actual outcomes. Divide data into bins, plot predicted probability against observed frequency. Step 5: Test decision threshold Probabilities are only half the battle. Choose your classification threshold based on your cost function. A medical screening test might use 0.1. Fraud detection might use 0.7. The optimal threshold depends entirely on what happens when you're wrong.

Common Mistakes That Kill Your Probabilities

Treating linear predictions as probabilities. Just because a number falls between 0 and 1 doesn't mean it's calibrated. A prediction of 0.7 should mean 70% accuracy across all such predictions. Linear regression makes no such guarantee. Ignoring class imbalance. Logistic regression handles imbalanced data worse than linear regression handles continuous outcomes. You may need class weights, oversampling, or different evaluation metrics. Calibrating without cross-validation. Calibration on training data leads to overfitting. Always use held-out data or k-fold cross-validation when applying transformations. Forgetting about feature scaling. Logistic regression coefficients are sensitive to feature scales. Standardize your inputs before fitting.

When Each Approach Wins

Use logistic regression when you're building a binary classifier from scratch and need interpretable probabilities. The coefficients tell you how much each feature pushes the odds toward one outcome. Use calibration techniques when you have an existing regression pipeline and can't retrain. Platt scaling works well for most situations. Isotonic regression handles more complex mappings but needs more data. Use alternative link functions when logistic regression systematically miscalibrates at the extremes. If your model predicts 0.9 but outcomes only happen 60% of the time, the sigmoid assumption might be wrong. The bottom line: if you need probabilities, use a model designed for probabilities. Retrofitting linear regression works, but it's always a compromise. The proper solution exists for a reason—use it.