Anytime we find ourselves repeating tedious actions on the web with the same sequence of steps, this is a great chance to write a program to automate the process for us. With selenium and Python, we just need to write a script once, and which then we can run it as many times and save ourselves from repeating monotonous tasks (and in my case, eliminate the chance of submitting an assignment in the wrong place)!

Here, I’ll walk through the solution I developed to automatically (and correctly) submit my assignments. Along the way, we’ll cover the basics of using Python and selenium to programmatically control the web. While this program does work (I’m using it every day!) it’s pretty custom so you won’t be able to copy and paste the code for your application. Nonetheless, the general techniques here can be applied to a limitless number of situations. (If you want to see the complete code, it’s available on GitHub).

Approach

Before we can get to the fun part of automating the web, we need to figure out the general structure of our solution. Jumping right into programming without a plan is a great way to waste many hours in frustration. I want to write a program to submit completed course assignments to the correct location on Canvas (my university’s “learning management system”). Starting with the basics, I need a way to tell the program the name of the assignment to submit and the class. I went with a simple approach and created a folder to hold completed assignments with child folders for each class. In the child folders, I place the completed document named for the particular assignment. The program can figure out the name of the class from the folder, and the name of the assignment by the document title.

Here’s an example where the name of the class is EECS491 and the assignment is “Assignment 3 — Inference in Larger Graphical Models”.

File structure (left) and Complete Assignment (right)

The first part of the program is a loop to go through the folders to find the assignment and class, which we store in a Python tuple:

# os for file management

import os # Build tuple of (class, file) to turn in

submission_dir = 'completed_assignments' dir_list = list(os.listdir(submission_dir)) for directory in dir_list:

file_list = list(os.listdir(os.path.join(submission_dir,

directory)))

if len(file_list) != 0:

file_tup = (directory, file_list[0])



print(file_tup) ('EECS491', 'Assignment 3 - Inference in Larger Graphical Models.txt')

This takes care of file management and the program now knows the program and the assignment to turn in. The next step is to use selenium to navigate to the correct webpage and upload the assignment.

Web Control with Selenium

To get started with selenium, we import the library and create a web driver, which is a browser that is controlled by our program. In this case, I’ll use Chrome as my browser and send the driver to the Canvas website where I submit assignments.

import selenium # Using Chrome to access web

driver = webdriver.Chrome()

driver.get(' # Open the websitedriver.get(' https://canvas.case.edu'

When we open the Canvas webpage, we are greeted with our first obstacle, a login box! To get past this, we will need to fill in an id and a password and click the login button.

Imagine the web driver as a person who has never seen a web page before: we need to tell it exactly where to click, what to type, and which buttons to press. There are a number of ways to tell our web driver what elements to find, all of which use selectors. A selector is a unique identifier for an element on a webpage. To find the selector for a specific element, say the CWRU ID box above, we need to inspect the webpage. In Chrome, this is done by pressing “ctrl + shift + i” or right clicking on any element and selecting “Inspect”. This brings up the Chrome developer tools, an extremely useful application which shows the HTML underlying any webpage.

To find a selector for the “CWRU ID” box, I right clicked in the box, hit “Inspect” and saw the following in developer tools. The highlighted line corresponds to the id box element (this line is called an HTML tag).

HTML in Chrome developer tools for the webpage

This HTML might look overwhelming, but we can ignore the majority of the information and focus on the id = "username" and name="username" parts. (these are known as attributes of the HTML tag).

To select the id box with our web driver, we can use either the id or name attribute we found in the developer tools. Web drivers in selenium have many different methods for selecting elements on a webpage and there are often multiple ways to select the exact same item:

# Select the id box

id_box = driver.find_element_by_name('username') # Equivalent Outcome!

id_box = driver.find_element_by_id('username')

Our program now has access to the id_box and we can interact with it in various ways, such as typing in keys, or clicking (if we have selected a button).

# Send id information

id_box.send_keys('my_username')

We carry out the same process for the password box and login button, selecting each based on what we see in the Chrome developer tools. Then, we send information to the elements or click on them as needed.

# Find password box

pass_box = driver.find_element_by_name('password') # Send password

pass_box.send_keys('my_password') # Find login button

login_button = driver.find_element_by_name('submit') # Click login

login_button.click()

Once we are logged in, we are greeted by this slightly intimidating dashboard:

We again need to guide the program through the webpage by specifying exactly the elements to click on and the information to enter. In this case, I tell the program to select courses from the menu on the left, and then the class corresponding to the assignment I need to turn in:

# Find and click on list of courses

courses_button = driver.find_element_by_id('global_nav_courses_link') courses_button.click() # Get the name of the folder

folder = file_tup[0]



# Class to select depends on folder

if folder == 'EECS491':

class_select = driver.find_element_by_link_text('Artificial Intelligence: Probabilistic Graphical Models (100/10039)') elif folder == 'EECS531':

class_select = driver.find_element_by_link_text('Computer Vision (100/10040)') # Click on the specific class

class_select.click()