Very simple Python script for extracting most common words from a story

Simple Python script without the use of heavy text processing libraries to extract most common words from a corpus.

What is the most used word in all of Shakespeare plays? Was ‘king’ more often used than ‘Lord’ or vice versa?

To answer these type of fun questions, one often needs to quickly examine and plot most frequent words in a text file (often downloaded from open source portals such as Project Gutenberg). However, if you search on the web or on Stackoverflow, you will most probably see examples of nltk and use of CountVectorizer. While they are incredibly powerful and fun to use, the matter of the fact is, you don’t need them if the only thing you want is to extract most common words appearing in a single text corpus.

Below, I am showing a very simple Python 3 code snippet to do just that — using only a dictionary and simple string manipulation methods.

Feel free to copy the code and use your own stopwords to make it better!

import collections

import pandas as pd

import matplotlib.pyplot as plt

%matplotlib inline # Read input file, note the encoding is specified here

# It may be different in your text file

file = open('PrideandPrejudice.txt', encoding="utf8")

a= file.read() # Stopwords

stopwords = set(line.strip() for line in open('stopwords.txt'))

stopwords = stopwords.union(set(['mr','mrs','one','two','said'])) # Instantiate a dictionary, and for every word in the file,

# Add to the dictionary if it doesn't exist. If it does, increase the count.

wordcount = {} # To eliminate duplicates, remember to split by punctuation, and use case demiliters.

for word in a.lower().split():

word = word.replace(".","")

word = word.replace(",","")

word = word.replace(":","")

word = word.replace("\"","")

word = word.replace("!","")

word = word.replace("â€œ","")

word = word.replace("â€˜","")

word = word.replace("*","")

if word not in stopwords:

if word not in wordcount:

wordcount[word] = 1

else:

wordcount[word] += 1 # Print most common word

n_print = int(input("How many most common words to print: "))

print("

OK. The {} most common words are as follows

".format(n_print))

word_counter = collections.Counter(wordcount)

for word, count in word_counter.most_common(n_print):

print(word, ": ", count) # Close the file

file.close() # Create a data frame of the most common words

# Draw a bar chart

lst = word_counter.most_common(n_print)

df = pd.DataFrame(lst, columns = ['Word', 'Count'])

df.plot.bar(x='Word',y='Count')

An example of the code output and plot of the 10 most frequently used words in the corpus. The text is ‘Pride and Prejudice’ and you can see the familiar names of Elizabeth and Mr. Darcy! :)