Gradient Descent in Linear Regression — 2

Recommend to read the first blog here.

How Gradient Descent works

We have the function whose value we have to minimize. Say we have to minimize the value of F(X) = X² + 6*X + 8

X = X — (Learning Rate) * F`(X=0)

Why do we take derivative?

To get the direction of where is minimum. If we are on left side of minimum value of parabola then the derivate is negative and negative sign before derivate will make the new value of X go right and thus more closer to minimum value. Vice versa for right side.

We do this for I iterations. Try to keep learning rate small but not too small.

Matrix form

Analytical form becomes quite difficult at times. Having the matrix, system behaves like a normal 1 d equation with matrix calculus rules being used.

We can write error as

Error(θ) = (Σ ((Data * θ — Selling_Price) ^ 2) ) / N

Where Data is matrix of size N * M which contains the M feature values of N rows of data., θ is a matrix of size M * 1 , And Selling_Price is the matrix of size N * 1which contains actual selling price of house.

In gradient descent, we do Error(θ) = Error(θ) — ( Learning Rate) * Error`(θ)

To get the new value, we need derivative of Error(θ). Also known as gradient.

Data` * (Data * (θ) - Y)

Where Data` Represents transpose of Data.

def gradient_descent(X, Y, theta, learning_rate, iterations):
    for iteration in range(iterations):
        loss = X.dot(theta) - Y
        m = len(X)
        gradient = X.T.dot(loss) / m
        theta = theta - learning_rate * gradient
    return theta

Here X is Data, Y is Selling Price.

Final Answer is to get coefficients or θ.

def coefficients_by_gradient_descent(data, target):
    Y = target.reshape(-1, 1)
    X = data.values
    theta = np.ones((len(data.columns), 1))
    return gradient_descent(X, Y, theta, LEARNING_RATE, ITERATIONS)

Initially fill all theta values with 0

Complete IPython note book for Boston housing dataset can be seen below. Run it here.

from sklearn import datasets, linear_model
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import numpy as np

LEARNING_RATE = 0.05
ITERATIONS = 10000


def normalise(data):
    data = (data - data.mean()) / data.std()
    return data
  
def coefficients_by_linear_regression(data, target):
    lm = linear_model.LinearRegression()
    data = normalise(data)
    lm.fit(data, target)
    coefficients = lm.coef_
    return coefficients

def gradient_descent(X, Y, theta, learning_rate, iterations):
    for iteration in range(iterations):
        loss = X.dot(theta) - Y
        m = len(X)
        gradient = X.T.dot(loss) / m
        theta = theta - learning_rate * gradient
    return theta
  
def coefficients_by_gradient_descent(data, target):
    Y = target.reshape(-1, 1)
    X = data.values
    theta = np.ones((len(data.columns), 1))
    return gradient_descent(X, Y, theta, LEARNING_RATE, ITERATIONS)

def coefficients_by_matrix_inverse_formula(data, target):
    Y = target.T.reshape(-1, 1)
    X = data.values
    return np.linalg.inv(X.T.dot(X)).dot(X.T).dot(Y)

def linear_regression(data, target):
    print_coefficients(data, target)
  
def logistic_regression(data, target):
    median = np.median(target)
    target[target <= median] = 0
    target[target > median] = 1
    print_coefficients(data, target)

def print_coefficients(data, target):
    print('Coefficients by Gradient Descent:')
    print(coefficients_by_gradient_descent(data, target))
    
    print('\n\nCoefficients by Matrix Inverse Formula')
    print(coefficients_by_matrix_inverse_formula(data, target))
    
    print('\n\nCoefficients by Scikit Learn')
    print(coefficients_by_linear_regression(data, target))

def main():
    raw = datasets.load_boston()
    data = pd.DataFrame(raw['data'], columns = raw['feature_names'])
    data = normalise(data)
    target = raw['target']
    print ('Linear Regression: \n')
    linear_regression(data, target)
    
    print ('\n\nLogistic Regression: Bonus homework \n')
    logistic_regression(data, target)
    
main()

Logistic Regression

Linear regression is used for converting linear regression to classification problem using sigmoid function and probability.

You may contact the author at me@abhinavrai.com