Linear regression from scratch: what's behind .fit()

It's easy to call LinearRegression().fit(X, y) without thinking about what happens inside. The problem is that when a model doesn't converge or produces strange coefficients, without that understanding there isn't much to diagnose.

The cost function

For a simple linear regression $\hat{y} = wx + b$ , the mean squared error over $n$ observations is:

J(w, b) = \frac{1}{n} \sum_{i=1}^{n} (\hat{y}_i - y_i)^2

Gradient descent

At each step, we update the parameters in the direction that reduces the error the most:

w \leftarrow w - \alpha \frac{\partial J}{\partial w}, \qquad b \leftarrow b - \alpha \frac{\partial J}{\partial b}

python

def gradient_descent(x, y, w, b, lr, epochs):
    n = len(x)
    for _ in range(epochs):
        y_hat = w * x + b
        dw = (2 / n) * sum(x * (y_hat - y))
        db = (2 / n) * sum(y_hat - y)
        w -= lr * dw
        b -= lr * db
    return w, b

Why it matters

If $\alpha$ (the learning rate) is too large, the error can oscillate or diverge instead of going down. If it's too small, training converges, but very slowly. Most of the "the model isn't learning" cases I've seen get solved by looking at this curve rather than by switching algorithms.