The other day I put an app in production with (gasp) no automated tests. I’m careful to say no automated tests, because of course I had been testing it manually throughout the ~12 hours it took me to write the initial version of the app. In lieu of tests, I made sure to log copious amounts of information, which I then collected from my deployed app into Papertrail.

For a single day’s work this was acceptable. But I knew that once I had a version in production, pushing out even the smallest, seemingly benign change would be an extremely dicey proposition without some automated regression tests in place.

I had also deliberately written the app in a deliberately, uh, “un-factored” fashion. Building the app was an exploratory process, and I wanted my design to evolve out of use rather than impose design assumptions up-front.

An element in choosing to write the app this was way was the knowledge that I have the tools available to me to refactor the design incrementally. But in order to do that, I needed tests to assure me that I hadn’t broken anything. Those tests needed to be extremely high-level, so that they could continue running unchanged even as I heavily refactored the app’s internals.

So I set out to write an initial smoke test, one that would put the app through its paces in a “happy path” scenario. Because this was a Sinatra app, I needed to do a fair amount of setup to get this smoke test running. I’d done this before in other Sinatra apps, so a lot of it was a matter of cribbing off of those apps.

Since some of these steps were non-obvious, and because getting over the “first test” hump can be a pretty daunting prospect, I thought I’d document the process.

RSpec :testing group to my Gemfile , and added a dependency on the RSpec gem. group :test, :development do gem "rspec" end First off, I needed a testing framework, and as a matter of habit I use RSpec. So I added agroup to my, and added a dependency on thegem. Then I ran rspec –init to populate my project with starter spec/spec_helper.rb and .rspec files. I left these mostly unchanged. My project has a top-level environment.rb file which sets up basic stuff like Bundler and Dotenv. Every test would need this environment set up consistently, so I added a line to the top of spec/spec_helper.rb: require File.expand_path("../../environment", __FILE__) I also made one change to .rspec. By default, RSpec configured this file to enable Ruby’s warnings mode. Sadly, my project includes some gems which are not warningfree. So I removed this line from the file: --warnings

The test spec/features directory, so I created it. Inside it, I created the file spec/features/smoke_spec.rb . It was time to write a test. I like to keep high-level tests in adirectory, so I created it. Inside it, I created the file I wrote the file incrementally, but I’m going to paste in its final contents here, somewhat elided. RSpec.describe "happy path", feature: true do let(:example_ipn_data) { { "payer_email" => "johndoe@example.org", # ... } } specify "a customer buying a book" do test_github_team_id = ENV.fetch("GITHUB_TEAM_ID") test_github_login = ENV.fetch("GITHUB_TEST_LOGIN") test_github_uid = Integer(ENV.fetch("GITHUB_TEST_UID")) expect(db[:tokens]).to be_empty expect(db[:users]).to be_empty client = Octokit::Client.new(access_token: ENV.fetch("GITHUB_APP_TOKEN")) client.remove_team_member(test_github_team_id, test_github_login) expect(client.team_member?(test_github_team_id, test_github_login)).to be_falsey authorize "ipn", ENV.fetch("IPN_PASSWORD") post "/ipn", example_ipn_data expect(last_response.status).to eq(202) auth_hash = { provider: "github", uid: test_github_uid, info: { name: "John Doe", } } OmniAuth.config.mock_auth[:github] = OmniAuth::AuthHash.new(auth_hash) open_last_email_for("johndoe@example.org") click_first_link_in_email expect(client.team_member?(test_github_team_id, test_github_login)).to be_truthy email = open_last_email_for("johndoe@example.org" expect(email).to have_body_text(%r(https://github.com/ShipRise/rfm)) end end I’ll be going over this code one piece at a time over the course of this post. A few notes to start out with: First, I’m using the new monkeypatch-free syntax from RSpec 3.0 to declare a test: RSpec.describe "happy path", feature: true do Second, in that last line you can see some RSpec metadata: feature: true . There’s nothing special about the name feature , it’s just a tag that I picked. I’ll be using this later to target some RSpec configuration to only apply to example groups with this tag.

Feature spec helper You might have noticed that the test does not reference the spec_helper.rb file directly. Instead, it has this: require "feature_spec_helper" One of the practices advocated by the RSpec team is to limit the number of dependencies that have to be loaded before every test. One way they advise for doing this is to keep spec/spec_helper.rb file minimal, only requiring libraries in it which are required by every test. If some tests need more than the baseline level of support, they can require a specialized spec helper instead. I opted to follow this advice, and have my feature specs require a specialized spec/feature_spec_helper.rb. Initially, this file just referenced the main spec helper: require "spec_helper" I made various additions to this file as I implemented the test.

Environment variables The test starts out by fetching some volatile and/or sensitive data from the system environment. I keep configuration info like this in environment variables to make it easy to change; for ease of deploy to Heroku; and to make it easier to set up remote Continuous Integration without committing sensitive information to my Github repo. test_github_team_id = ENV.fetch("GITHUB_TEAM_ID") test_github_login = ENV.fetch("GITHUB_TEST_LOGIN") test_github_uid = Integer(ENV.fetch("GITHUB_TEST_UID")) I use ENV.fetch to grab all of the variable values my test needs. The test will fail if any of these variables are missing, so I want to know right away if there’s an unset variable. ENV.fetch will raise a KeyError if it doesn’t find the variable, letting me know exactly what var I forgot to add to my .env file or to my CI configuration. The last variable is expected to be an integer. I don’t just want to ensure that the variable is set; I want to be assured that it is a valid integer as well. So rather than using the lenient #to_i > method, I use the Integer() conversion function. This function will raise an exception if the value it is given can’t be sensibly interpreted as an integer.

Database Since this is my very first test, I am making no assumptions. Thus, the next stanza of my test does a sanity check that the database tables I’m interested in start out empty. This is important, because if I later check for the existence of some record, that check is meaningless unless I also know that the record didn’t exist at the beginning of the test. expect(db[:tokens]).to be_empty expect(db[:users]).to be_empty These lines also force me to set up database integration for my tests. I’m using Sequel as the database layer for this project. In order to make the database connection available to my tests, I add some code to the spec/feature_spec_helper.rb file. The first line requires a file named db.rb. This file lives at lib/db.rb in my project, and is responsible for setting up a global DB constant which refers to a Sequel database connection. require "db" Next I define a module for database-related test helper methods. It defines a db attribute. module DbSpecHelpers attr_accessor :db end And then I update the RSpec config to include this module into any example tagged with feature: true . I also add a before hook to initialize the db attribute with the value of the global DB constant. RSpec.configure do |c| # ... c.include DbSpecHelpers, feature: true c.before feature: true do self.db = ::DB end # ... end I could avoid all this by referencing the DB constant directly in my tests. But I’m not sure I’m going to keep using a global constant, and I like to be able to override the default DB for individual tests.

Test setup Next up in my test, I do some setup. The software that is being tested is supposed to add a user to a Github team when they purchase a product. In order to meaningfully test this behavior, I have to first be sure that the test user isn’t in the team to begin with. So I use Octokit to get the scenario started in the right state. Since I’m not certain if an exception will be raised on failure, I verify that the test user is not a team member after making the change. client = Octokit::Client.new(access_token: ENV.fetch("GITHUB_APP_TOKEN")) client.remove_team_member(test_github_team_id, test_github_login) expect(client.team_member?(test_github_team_id, test_github_login)).to be_falsey This setup illustrates a general rule in tests: it’s better to enforce a particular state of affairs before the test than to try to return things back to a “clean slate” state after the test. Tests can often be interrupted in the middle, especially when you’re first writing them. A robust test makes sure everything is where it is expected to be before getting started.

Rack::Test Now that my setup is done, I need to kick off the test proper. The first step is to simulate a product purchase. To do this, I need to hit a webhook action with some simulated IPN data. In order to simulate a POST to my Sinatra app, I need to set up Rack::Test. First, I add it to my Gemfile. group :test, :development do # ... gem "rack-test" # ... end Then I bundle install. Next I require rack/test in the spec/feature_spec_helper.rb. In order to use Rack::Test , I need my Sinatra app to be accessible, so I require that as well. It’s found in a top-level project file called app.rb. # ... require "rack/test" # ... require File.expand_path("../../app", __FILE__) I create a helpers module for Rack::Test-enabled examples. module RackSpecHelpers include Rack::Test::Methods attr_accessor :app end The app attribute is used by Rack::Test to find the Rack app to be tested. I add some RSpec config to include and configure this new module in feature tests. RSpec.configure do |c| c.include RackSpecHelpers, feature: true c.before feature: true do self.app = Sinatra::Application end # ... end Now I can simulate an authorized IPN POST using Rack::Test helper methods. authorize "ipn", ENV.fetch("IPN_PASSWORD") post "/ipn", example_ipn_data expect(last_response.status).to eq(202)

OmniAuth Shortly this test is going to be simulating a user clicking a link in an email. But before that can happen, I need to do a little more setup. The link is going to redirect them to a Github OAuth authentication page. My test can’t automate actions performed on sites outside of the Sinatra app being tested, so I need a way to fake out this authentication step. Thankfully, I’m using OmniAuth, and OmniAuth has built-in support for faking authentication. First off, I go over to my spec/feature_spec_helper.rb and add this line: # ... OmniAuth.config.test_mode = true # ... This should be pretty self-explanatory. Next, in my smoke test I fake up an authentication hash. Normally OmniAuth calls the app back with auth data sourced from the authentication provider. But in this test, it’s just going to call the app back with the auth data that I give it. auth_hash = { provider: "github", uid: test_github_uid, info: { name: "John Doe", } } I tell OmniAuth to call back with this fake data the next time the app triggers authentication. OmniAuth.config.mock_auth[:github] = OmniAuth::AuthHash.new(auth_hash)

EmailSpec and Capybara I’m almost ready to have the app simulate a user opening their email and answering an invitation. But again, I can’t test systems outside my own, which means I need a way capture emails that the system sends. To do this, I’ll use EmailSpec. EmailSpec in turn depends on Capybara to simulate a human clicking on email links, so I’ll need to add that to the project as well. I’ll start with Capybara. I add it as a project dependency, adding EmailSpec at the same time while I’m at it. group :test, :development do # ... gem "email_spec" gem "capybara" # ... end Then I update the spec/feature_spec_helper.rb file to set up Capybara/RSpec integration. I require the library, tell Capybara what app to use, and include some helper modules into feature example groups. require "capybara/rspec" # ... Capybara.app = Sinatra::Application # ... RSpec.configure do |c| # ... c.include Capybara::DSL, feature: true c.include Capybara::RSpecMatchers, feature: true # ... end Now for EmailSpec. I require the library, and include some helper modules into feature example groups. Then I add a before hook to feature specs, resetting the delivery bin. If I neglected to do this, emails from one test might “bleed” over into others. require "pony" require "email_spec" # ... RSpec.configure do |c| # ... c.include EmailSpec::Helpers, feature: true c.include EmailSpec::Matchers, feature: true c.before :each, feature: true do reset_mailer end # ... end Notice that I require the pony library before email_spec. EmailSpec monkeypatches the two most common Ruby mail-sending libraries, ActionMailer and Pony, to send their email into a fake test delivery bin instead of to a mail server. It automatically detects which library to patch, based on what is available. I use Pony in my app, so I make sure that the pony library is loaded before email_spec. I’m finally able to add some lines to the test, simulating a user opening their email and clicking on a link they find there. open_last_email_for("johndoe@example.org") click_first_link_in_email

Assert In theory, the user clicking on the link in the welcome email should trigger a series of actions: they should authenticate with Github, which will be simulated as we saw earlier. Then the app will add them to a Github team. Finally, it will send them an email welcoming them to the team. The rest of the test asserts that this is, in fact, what happens. First, that the user has been added to a Github team: expect(client.team_member?(test_github_team_id, test_github_login)).to be_truthy And second, that they have a welcome email in their inbox: email = open_last_email_for("johndoe@example.org") expect(email).to have_body_text(%r(https://github.com/ShipRise/rfm))

DatabaseCleaner expect(db[:tokens]).to be_empty expect(db[:users]).to be_empty All this works… exactly once. The second time I run the test, it fails at the beginning, where I checked that the database starts out empty. In order to make sure the database starts out clean every time, I’m going to add DatabaseCleaner to the project. I’ve written about DatabaseCleaner before, so I’m not going to go into detail about this part. But for the record, here’s the configuration. First, I need to add it to the Gemfile (and then bundle install). group :test, :development do # ... gem "database_cleaner" # ... end And then I add DatabaseCleaner setup to spec/feature_spec_helper.rb. # ... require "database_cleaner" # ... RSpec.configure do |c| # ... c.before :suite do DatabaseCleaner[:sequel, {connection: ::DB}].strategy = :transaction DatabaseCleaner[:sequel, {connection: ::DB}].clean_with(:truncation) end # ... c.before feature: true do DatabaseCleaner[:sequel, {connection: ::DB}].start end c.after feature: true do DatabaseCleaner[:sequel, {connection: ::DB}].clean end # ... end Notice that I am specific about which database connection to use: DatabaseCleaner[:sequel, {connection: ::DB}]

Files Gemfile : group :test, :development do gem "rspec" gem "rack-test" gem "email_spec" gem "database_cleaner" gem "capybara" end Here is full addition made to And here is the completed spec/feature_spec_helper.rb : require "spec_helper" require "db" require "rack/test" require "pony" require "email_spec" require "database_cleaner" require "capybara/rspec" require File.expand_path("../../app", __FILE__) Capybara.app = Sinatra::Application OmniAuth.config.test_mode = true module DbSpecHelpers attr_accessor :db end module RackSpecHelpers include Rack::Test::Methods attr_accessor :app end RSpec.configure do |c| c.include RackSpecHelpers, feature: true c.before feature: true do self.app = Sinatra::Application end c.before :suite do DatabaseCleaner[:sequel, {connection: ::DB}].strategy = :transaction DatabaseCleaner[:sequel, {connection: ::DB}].clean_with(:truncation) end c.include DbSpecHelpers, feature: true c.before feature: true do extend DbSpecHelpers self.db = ::DB DatabaseCleaner[:sequel, {connection: ::DB}].start end c.after feature: true do DatabaseCleaner[:sequel, {connection: ::DB}].clean end c.include EmailSpec::Helpers, feature: true c.include EmailSpec::Matchers, feature: true c.before :each, feature: true do reset_mailer end c.include Capybara::DSL, feature: true c.include Capybara::RSpecMatchers, feature: true end As the app grows I will probably begin to split this setup out into multiple files.