Introducing FitBERT

Use BERT for smart string interpolation without deep learning experience

tl;dr Qordoba is open sourcing FitBERT, a library to make it easy for anyone who knows Python to use BERT (or other fancy deep learning NLP models) for string interpolation given a list of options.

If you follow NLP news, even peripherally, you have probably heard of BERT, Google’s very large masked language model that shattered benchmarks when it came out in October 2018. If you’ve used BERT, you probably used it with the fantastic HuggingFace library, Transformers. The Transformers library makes it easy to use BERT (and other similar models like GPT2 and XLNet) without rewriting or training these large models from scratch, using PyTorch or TensorFlow2.

However, most people who write code do not use deep learning frameworks, or any machine learning at all. Even if you do know how to use these frameworks, sometimes you want to quickly interact with BERT, and are willing to cede some flexibility in exchange for ease of use. It is for these cases that we wrote FitBERT.

Thanks to NodoBird https://www.instagram.com/nodobird/ for letting us use this drawing

What does FitBERT mean?

FitBERT stands for “Fill in the Blanks, BERT”. Because of the training objective for BERT (masked language modeling), it is very good at filling in the blanks. In fact, that is one of the two tasks that BERT is trained on (the other is next sentence prediction, and the RoBERTa paper showed that performance on downstream tasks actually improves if you only train on the fill in the blank task).

What does FitBERT do?

After instantiating a FitBert instance, you pass it a string with a blank to fill in, and a list of candidates to go in that blank. Let’s look at some code:

Here, in 5 lines of code (excluding comments), we have loaded a BERT model, given it a string with a blank to fill in (indicated by ***mask*** ), given it a list of options to choose from, and asked the model to rank those options.

Want to use FitBERT to correct a potentially-incorrect word? There is a convenience method for that:

Using BERT to correct grammatical errors in 3 lines!

For Software Engineers

Every software engineer has templated a string. And, in every programming language I know of, this is an easy task. However, it can get complicated. For example, you need to tell the user that their order has shipped, so you do something like confirmation_message = "Your {order.quantity} {order.name} is on its way!"

Then you quickly realize that this sometimes generates messages like "Your 6 burritos is on its way!" Hmm… I guess we can just write a quick is_or_are = lambda quantity: "are" if quantity > 1 else "is" , then change the confirmation message to confirmation_message = "Your {order.quantity} {order.name} {is_or_are(order.quantity)} on its way!" . And if this is the only message you have to worry about, you’ve made the right call.

But maybe you get a feature request. “Hey, the order messages aren’t fun enough! Can you add one of these adjectives, picked at random, between the quantity and the name: delicious, sweet, hot, artisanal” and something doesn’t feel right, but you do the ticket (this is not the optimal version of this code, just an example):

import random # application code def a_or_an(word):

vowels = ["a", "e", "i", "o", "u"]



# some words don't follow the pattern!

if word == "horchata":

return "an"

return "an" if word[0] in vowels else "a" def is_or_are(quantity):

return "are" if quantity > 1 else "is" adjective_list = [

"delicious",

"hot",

"sweet",

"artisanal"

] confirmation_message = f" Your {order.quantity} {random.choice(adjective_list)} {order.display_name} {is_or_are(order.quantity)} on {its_or_their(order.quantity)} way! "

And soon enough, bug report:

The customer complained because we said their sodas would be hot, and sodas should not be hot

Sigh. Ok, so, each order has a category , and we can probably figure out which adjectives go with which categories… But yeah, it’s getting complicated.

Why don’t we leave the adjectives to BERT?

What’s happening here? Well, BERT is trained on an enormous collection of written English. Specifically, it is trained to look at a sentence with a blank, and output an ordered list of every possible word that could fill in that blank, and a score indicating how likely that word is. So FitBERT has BERT do exactly this, then looks at the intersection of that list with the options you have provided.

For Researchers

Just because FitBERT is easy to use doesn’t mean it can’t be used in serious research!

FitBERT is based on Yoav Goldberg’s “Assessing BERT’s Syntactic Abilities”, in which the author assesses “the extent to which the recently introduced BERT model captures English syntactic phenomena.” Here is how to reproduce two experiments from this paper with FitBERT:

While Huggingface’s Thomas Wolf published a supplement to “Assessing BERT’s Syntactic Abilities” comparing GPT (the original) and BERT, to my knowledge this hasn’t been repeated for RoBERTa and DistilBERT. As you can see from the above code sample, with FitBERT, it would be a fairly small project to compare those.