Gradient Descent in Linear Regression

I am a beginner in Machine Learning and faced a hard time in understanding what these big terms are in our Data Science module at Go-Jek Bootcamp 3 . Most of the material on the web was too advanced as a beginner and I didn’t have time to watch the videos. I will try to explain these things in as basic way as possible.

Linear Regression

Its a way to model the data. We have data. And we want to predict some thing based on this data. Linear Regression is one way to do it. Say we have the data for house selling price based on the house features (house_size, garage, condition, year_built, has_swimming_pool, etc) [Dataset] and we want to get the sale price of a house. We can model this linearly — as shown below.

Selling Price = Theta1 * [Feature1] + Theta2 * [Feature2] + Theta3 *             [Feature3] + …… + ThetaM * [FeatureM]

Feature values are numerical values. Theta are the coefficients. If we can tune the value of theta according to the data we have, we can just plug in the value for features for new data and we get the sale price for house. So our job is to tune these parameters now.

This is for y = mx+c. For one feature. For n features its an n dimension space with n-1 th dimension as correlating to line here

Gradient Descent

Its an algorithm to find the minimum of the function. We have to come down from the hill, to the lowest value. We take the path with highest slope and come down.

Can it be of any help in Linear Regression?

We can convert Linear Regression to error function. And then use this gradient descent to minimize this error function. As we minimize the error function, our coefficient of linear regression(theta) will get tuned.

How to convert Linear Regression to error function?

Data Size = N

HouseQuality=2, Garage=1, Size=15, ..... , Sale Price=20000
HouseQuality=5, Garage=1, Size=10, ..... , Sale Price=26000
HouseQuality=1, Garage=0, Size=27, ..... , Sale Price=22000
HouseQuality=3, Garage=1, Size=20, ..... , Sale Price=30000
HouseQuality=2, Garage=0, Size=10, ..... , Sale Price=12000
HouseQuality=1, Garage=0, Size=7, ..... , Sale Price=10000
.
.
.
.
HouseQuality=4, Garage=2, Size=25, ..... , Sale Price=40000

Fix a value of thetas. Initially let them be all one.

For each row in the data, get the estimated selling price.

Selling Price = Theta1 * [Feature1] + Theta2 * [Feature2] + Theta3 *             [Feature3] + …… + ThetaM * [FeatureM]

Now subtract this from actual selling price. This is our error for one row in data. Square this error. (So that it is magnified)

Now do this for all N rows and add this squared error. Divide by N to get mean squared error for current value of theta.

This is our error function. If we can minimize this value by gradient descent, then we are done.

Follow the second part here.

You may contact the author at me@abhinavrai.com