I love web scraping. Not only has it become a big industry in recent years, it’s also a lot of fun. It can help you easily find interesting insights using data that already exists. Today, I’m going to be building a very basic web scraper that can search through DuPont Registry (a marketplace for expensive cars, homes, and yachts); for this scraper, I’ll be focusing on the car section.

I will be using two relevant gems: Nokogiri and HTTParty. Nokogiri is a gem which allows you to parse HTML and XML into Ruby objects. HTTParty, on the other hand, simplifies the process of pulling raw HTML into your Ruby code. These two gems will work together in our scraper.

It’s important to note that although Nokogiri and HTTParty are cool, they are more like sprinkles on top. The core skills for building a program with this are things like defining and initializing a class and its instance(s), iterating through hashes and arrays, and building helper methods.

Getting Started

If you have not already, make sure that you have your essential gems installed and required in your code. Also, I like to use byebug or pry so that I can stop the code if needed and take a look at what’s going on!

require 'nokogiri' require 'httparty' require 'byebug' require 'pry' class Scraper

#we will be adding code here shortly

end

Initializing an Instance

Now, let’s think about our Scraper class. What kind of methods should it have? What does this class need to keep track of?

We will need to decide this before defining how our new scraper is initialized. I’ve already given this a little thought, and here’s what I came up with:

Let’s break this down. Each instance of a Scraper class should know what make and model it’s supposed to be looking at and which URL it needs to visit to find its data. It would also be nice if these instance variables were readable, so I’ve added an attr_reader to each of them.

DuPont Registry’s URL paths are fairly straightforward, so we can pull in the template literals from ‘make’ and ‘model’ to generate our destination URL and then save that to our instance variable of @url. The .sub(“ “, “ — “) at the end of the URL string is just a method that replaces whitespace with two dashes.

Time to add a little more functionality. We need our scraper instance to actually, well, scrape. Let’s start building out some instance methods.

I want to get started on these helper methods early before our #scrape method becomes a novel. The #parse_url method is taking in a URL as an argument, calls on HTTParty to pull in raw HTML, and then Nokogiri is taking that unparsed page and… umm… parsing it. Just like that, we can take an entire webpage and transform it into a usable Ruby object! We then save that entire object into a local variable inside of #scrape called parsed_page. Now is a pretty good time to set a binding.pry and take a look at what we actually have before we go any further.

Let’s go ahead and run this code in our terminal using the bash command ruby scraper.rb (this assumes that you’re coding in a file titled scraper.rb, if your file has a different name then adjust accordingly). Make sure that you are in the correct directory.

Now, you are (or should be) in a pry session, so we can start typing Ruby code into our terminal. First, let’s create a new Scraper instance:

bentley = Scraper.new("bentley", "continental GT")

Cool, we have a new instance of a Scraper that is going to be looking for Bentley Continental GTs at https://www.dupontregistry.com/autos/results/bentley/continental--gt/for-sale. We can confirm this by typing the following into our terminal:

Great! Now let’s get down to business. We need to call our #scrape method on our Scraper instance of bentley so that we can hit our second binding.pry and take a look at what we got:

bentley.scrape

We are now prying inside of our #scrape method

What I’m interested in at this point is what exactly parsed_page is since theres a lot of Ruby gem trickery going on. Let’s call on the parsed_page local variable.

pry(#<Scraper>)> parsed_page

Whoa! Okay. That’s a lot of stuff. Clearly it’s doing something, I’m just not sure what to do with all of this yet. Mine looked something like this:

Nokogiri parses HTML into something that looks like this

This may look different if you are scraping a different website.

Before we get too deep into trying to figure out how to scrape our bits of data from this huge object, let’s look at what I’m trying to get to so we can reverse engineer it. I’d like a hash that looks something like this:

Cool, so rather than trying to pull out every piece of data, we’re just looking for these five pieces of data: year, make, model, price, and link. But wait! We already know our make and model from when we instantiated our scraper instance, so we’re really only looking for 3 data points.

Dealing With HTML

Now that we know what we’re looking for, let’s look through the HTML of the page we want to scrape to find our paths. First, let’s see if we can find the container which includes all of our relevant car info. We’ll need to visit that listings page and open up developer tools.

Looks like each car listing is stored in a separate div called ‘searchResults’

Let’s save each listing in a variable that we can use later. We’ll use the .css method from Nokogiri to build this array.

cars = parsed_page.css('div.searchResults') #creates an array where each element is parsed HTML pertaining to a different listing

Now, let’s just try to find the price.

The price is contained in a span called ‘cost’

We’ll have to play around with this in order to pull out the text we need. It’s best to do this in a pry or byebug session. I was able to pull out the price using the following code:

car.css('.cost').children[1].text.sub(",","").to_i #looks in an instance of car for the cost div

#looks in its children at an index of 1 (this is where it lives)

#converts it to text

#gets rid of the comma

#converts the string into an integer

We’ve got price, now let’s get year. In the same fashion, we’ll need to poke around with developer tools to find our string that contains the year of the car. In my case, this ended up being a string that looked like “2017 Bentley Continental GT V8 S” so I decided to just steal the first 4 characters and convert it into an integer:

car.css('a').children[0].text[0..4].strip.to_i #looks in an instance of car for the <a> tag

#looks in its children at an index of 0 (this is where it lives)

#converts to text

#takes the first 4 characters (ie "2017")

#strips any whitespace if it exists

#converts to integer

Last but not least, lets get our link to the car’s page.

car.css('a').attr('href').value #looks in an instance of car for the <a> tag

#gets the attribute of <href>

#pulls out is value (as a string)

So, we have all of our JQuery-esque stuff figured out. As you read on, you’ll see how these are implemented in the helper method create_car_hash.

Finishing Up The Scraper

From here, it’s pretty easy to finish up our scraper. Lets get our helper methods set up so that our #scrape method will have everything it needs.

I ended up making 4 different helper methods:

create_car_hash creates a hash based on the values we need get_all_page_urls does what it sounds like! Collects all urls into an array which will allow us to factor in pagination get_number_of_pages is pretty self explanatory — this will also help us handle pagination build_full_cars gives us an array of car hashes from page 2 and beyond

Let’s put it all together in the #scrape method so that we can get what we want. At the end, this is what my code looks like:

It may seem a little weird to build our hashes for the first page and the rest of the pages separately (see line 32), and you could probably consolidate this, but the first page’s URL is a little different and is important for getting our initial data set up.

Let’s run this really quick to make sure it does what we want it to do.

ruby scraper.rb

bentley = Scraper.new("bentley", "continental gt")

bentley.scrape

Our output is massive! I had to truncate it a bit. This is what I got:

If we call .length on this array, we indeed get 120, which matches our number of listings. Perfect!

There are still many methods we would want to build into this to make it useful. For example, we could create a method that returns the link of the cheapest Bentley currently listed, or one that calculates the average asking price. You could even beautify this code quite a bit as its far from perfect. However, I’m going to stop here for today before this article becomes War and Peace! Here’s a link to the GitHub repository if you’d like to use it for inspiration for your own web scraper.