Journey to the Home for Data Science

First step towards Data Science

Getting started with kaggle competitions

Kaggle is an AirBnB for Data Scientists — this is where they spend their nights and weekends. It’s a crowd-sourced platform to attract, nurture, train and challenge data scientists from all around the world to solve data science, machine learning and predictive analytics problems. — Zeeshan-ul-hassan Usmani

Introduction

Kaggle is the best place to Get Started if you want to get your hands dirty in Data Science. If you want then you can read all the What and Why at: https://www.kaggle.com/getting-started/44916. Trust me it is all worthy.

One main thing what makes Kaggle so awesome is, you can find the dataset without any hassle of data gathering and a notebook to write and run your code in the cloud. You can also use the Google’s Colab Jupyter Notebook with their GPU & TPU to write and run your code. They are free.

You can easily signup to the platform and get started with the getting started competitions. I am going to show you how to predict the House Prices by getting into House Prices: Advanced Regression Techniques competition. It’s gonna be awesome, let’s get started.

Getting Started

Lets move to the Notebook Section and start a New Notebook on Kaggle:

Notebook on kaggle

After you start a Notebook you will get a nice Notebook Platform with a list of datasets availabe at the left side.

Kaggle’s Notebook

Now before starting to write some code, we have to understand the Machine Learning Pipe Line

ML Pipeline

ML pipeline

though kaggle makes it easy for us by providing the dataset, it is still challenging to process and clean the data and extract the relevant features form it and identify the outliers in dataset. This whole process is called Data Wrangling and it is a part of Feature Engineering.

Lets see what type of data we have got; we are going to use Pandas: Data analysis library for analyzing the dataset. As you can see in the Notebook Pandas and numpy library is already imported. We have got two dataset files: train.csv and test.csv another sample_submission.csv is just a sample of how our submission should look like.

Code Part

lets see the head data of train and test set.

train and test set

as you can see that we have 81 columns in train set and 80 columns in test set; it is because, in test set the label column(SalePrice) is not given. We have to predict the SalePrice for the given features by training a model based on the train set. We also should have a domain knowledge in the field so that it makes us easier to know what we are doing. Example: for this project we are predicting a SalePrice for the House, and domainly speaking features that most matters are: House Condition, How is the location, What features are there in house and so on..

We have given a set of features and a label and we are predicting a continuous value for a given features not a Categorical value; so this is a clear case of Regression problem not a Classification Problem.

Classification and Regression

now we are going to play with the train set (train.csv) to check the skewness of SalePrice data.

normally distributed data makes it easier to analyse

why normally distributed data? you can check here.

Skewness of the SalePrice data

as we can see that the SalePrice is right(Positive) skewed, we have used matplotlib and seaborn to visualize the data.

now it’s time for data transform. We can use log to transform the data into normally distributed data:

log transform of the data

we have used a log with base 10, now the data in normally distributed. To understand why we used log to transform the data, watch this video.

Logarithms — Real Life Applications

after that we can check our probability plot:

Probability Plot

It looks nice, we can move forward.

checking the correlation

Now we have to check for the highly correlated datas. First we are extracting all the numerical data typed features and checking the correlation of those features with respect to the SalePrice.

These first five features are highly correlated to the Sale Price; meaning- these are the values which are linearly correlated and highly affects the SalePrice.

as we can see GarageArea is also highly correlated with the sale price lets visualize the data:

visualizing garage area data

those few points which are higher then 1200 in x-axis are the outliers for our data. Those few datas can affect the model so we have to eliminate them.

removing outlier data points

simply we removed them by taking datas less then 1200. 😄 Just a simple approach.

converting string data to numerical

in our dataset we have Street feature which has two unique values: Pave and Grvl. This is also a useful feature but it is in String data type; so we are creating a funcction mapStreet to convert these data into 0 for Pave and 1 for Grvl.

Sale price with respect to Sale Condition

Now it’s time to see the SalePrice of the house with respect to the SaleCondition. And from the image above we can see that Partial Sale Condition are known to be sold a High Sale Price so, for the ease we group the Partial SaleCondition as 1 and other SaleCondition as 0.

now to make our dataset better we can either drop or use another approach to get rid of all the NAN or null values. For now we are just dropping it and keeping only the numerical values on train and test set.

Now we log transform the SalePrice and split the train.csv dataset in two parts: y for labels and x for features and again split those into train set and a test set by using scikitlelarns train_test_split method.

Finally! now we are ready to train the model.

First off, we start by importing an Regression Modal library from Sci Kit learn and giving our training set to train. You can use any appropriate modal for the task. For now we are using Linear Regression.

we can train our data by just calling a .fit() method on the model. It trains our modal and we can check the score by using .score() method.

currently we have accuracy of around 84%. It’s not that bad, but you can improve the accuracy by tweaking some data and from better feature engineering.

now it’s time to prepare a submission file and submit it on Kaggle.

preparing a submission file

we are using same model to predict the label for the features in test.csv file and saving all the outputs with an ID in the submission.csv file. Now you can see your submission file and submit it to the kaggle’s submit section and see yourself in a public leaderboard.

kaggle leaderboard of Hose Price prediction

Final Thoughts

Kaggle competitions is a good place to get your hands dirty with real world datasets. I hope you have learned something. You can keep exploring other datasets on kaggle and carry on to the journey of Data Science by doing exciting projects. Keep learning and stay curious.