Setup

Our setup is pretty simple. We create a new folder and run this command inside that folder to create a package.json file. Let’s cook the recipe to make our food delicious.

npm init -y

Before we start cooking, let’s collect the ingredients for our recipe. Add Axios and Cheerio from npm as our dependencies.

npm install axios cheerio

Now, require them in our `index.js` file

const axios = require('axios');

const cheerio = require('cheerio');

Make the Request

We are done with collecting the ingredients for our food, let’s start with the cooking. We are scraping data from the HackerNews website for which we need to make an HTTP request to get the website’s content. That’s where axios come into action.

Our response will look like this —

<html op="news">

<head>

<meta name="referrer" content="origin">

<meta name="viewport" content="width=device-width, initial-scale=1.0">

<link rel="stylesheet" type="text/css" href="news.css?oq5SsJ3ZDmp6sivPZMMb">

<link rel="shortcut icon" href="favicon.ico">

<link rel="alternate" type="application/rss+xml" title="RSS" href="rss">

<title>Hacker News</title>

</head>

<body>

<center>

<table id="hnmain" border="0" cellpadding="0" cellspacing="0" width="85%" bgcolor="#f6f6ef">

.

.

.

</body>

<script type='text/javascript' src='hn.js?oq5SsJ3ZDmp6sivPZMMb'></script>

</html>

We are getting similar HTML content which we get while making a request from Chrome or any browser. Now we need some help of Chrome Developer Tools to search through the HTML of a web page and select the required data. You can learn more about the Chrome DevTools from here.

We want to scrape the News heading and its associated links. You can view the HTML of the webpage by right-clicking anywhere on the webpage and selecting “Inspect”.

Chrome DevTools to inspect the HTML

Parsing HTML with Cheerio.js

Cheerio is the jQuery for Node.js, we use selectors to select tags of an HTML document. The selector syntax was borrowed from jQuery. Using Chrome DevTools, we need to find selector for news headlines and its link. Let’s add some spices to our food.

First, we need to load in the HTML. This step in jQuery is implicit since jQuery operates on the one, baked-in DOM. With Cheerio, we need to pass in the HTML document. After loading the HTML, we iterate all the occurrences of the table row <tr> to scrape each and every news on the page.

The Output will look like —



{

title: 'Malaysia seeks $7.5B in reparations from Goldman Sachs (reuters.com)',

link: '

},

{

title: 'The World Through the Eyes of the US (pudding.cool)',

link: '

},

.

.

.

] title: 'Malaysia seeks $7.5B in reparations from Goldman Sachs (reuters.com)',link: ' https://www.reuters.com/article/us-malaysia-politics-1mdb-goldman/malaysia-seeks-7-5-billion-in-reparations-from-goldman-sachs-ft-idUSKCN1OK0GU' },title: 'The World Through the Eyes of the US (pudding.cool)',link: ' https://pudding.cool/2018/12/countries/' },

Now we have an array of JavaScript Object containing the title and links of the news from the HackerNews website. In this way, we can scrape the data from various large number of websites. So, our food is prepared and looks delicious too.

Conclusion

In this article, we first understood what is web scraping and how we can use it for automating various operations of collecting data from various websites.

Many websites are using Single Page Application (SPA) architecture to generate content dynamically on their websites using JavaScript. We can get the response from the initial HTTP request and can’t execute the javascript for rendering dynamic content using axios and other similar npm packages like request. Hence, we can only scrape data from static websites.

Don’t miss out on part 2: “how to scrape data from dynamic websites”.

Feel free to comment and ask me anything. You can follow me on Twitter and Medium. Thanks for reading! 👍