This is an aggregated tutorial on how to automate a repetitive task such as periodically looking up prices online. We will use a simple python script for scraping a website, Google Sheets for storing data, Slack for sending out notifications, and AWS Lambda/Cloudwatch for scheduling and executing jobs.

The biggest upside of this stack is that maintenance cost is minimal; no need to pay for a database or server, no need to pay a monthly subscription fee for scraping services, while simultaneously taking full advantage of free services like Google Sheets and Slack (free-tier).

While many tutorials exist for each component, I’ve yet to come across one that combines everything. I hope this serves as a launch pad for users wanting to automate tedious tasks throughout their day to free up time for more interesting things!

Outline

Simple scraper Using Google Sheets as a DB Sending Slack notifications Serverless cron using AWS Lambda and CloudWatch Events

One task I would liked to have automated when I was graduating from college was looking up prices for an apartment I was interested in. While the prices weren’t super volatile day to day, they would suddenly drop when new rooms opened up and resulted in a race of who could send in a down payment to the leasing office first. If I had the current tools at my disposal, I could have saved a couple hundred dollars a month on rent, or at least freed myself from the headache of having to check the site every day.

First and foremost, all code examples are located in the github repo below:

Make sure to reference this while going through this tutorial. Let’s begin!

1. Simple scraper

The simple python script uses Beautiful Soup to scrape this listing site, and prints out the minimum price. You can test out the script by running the following command (make sure to set up a virtualenv and install the necessary libraries):

python apartment_scraper.py

2. Using Google Sheets as a DB

While using Google Sheets may not be the most efficient or scalable solution for storing data, it’s an easy way to share data with non-tech people. In my example, I would have shared the apartment data with my girlfriend at the time (now my wife) and gotten her feedback as well.

It’s also free to use!

Let’s jump right in.

A. Create a new Google APIs project

Head over to https://console.developers.google.com/apis/dashboard and create a new project.

B. Enable Google Sheets API for the newly created project

C. Create a service account