Using Gmail with IMAP to Receive Email in Rails

The number one article suggestion thus far is how to receive email in a rails app. Ask and you shall receive. What I am about to present is just a first run at it and I am not going to promise that it scales. :) That said, I fooled around with it a bit and found it relatively easy. I will post more on the topic as I play more with receiving email, but what follows is enough to at least get you up and running. The email address I used for this is actually a Gmail address, so using this method does not involve postfix or some other mail server configuration.

The Project

The project that I added email receiving functionality to was YardVote, a weekend project by my friends at Collective Idea. YardVote allows people to record a “yard” with the political signs that were present in said yard and displays them on a google map. I was immediately enamored with the idea but found entering signs on my iPhone to be a bit tedious. Giving that everyone seems to be interested in receiving email, that it would be handy for yardvote, and that yardvote was open source, I decided to take a crack at it.

The Process

The path to success in summary form was this:

Check for new emails Create new or update existing location from each email Archive emails that have been processed to avoid duplicates Turn process of checking and processing email into daemon

bin/mail_receiver.rb

For whatever reason, I created a bin directory and a mail_receiver.rb script inside of that. Actually, it was for a reason. I think bin seems like a good place for stuff like this. I could have used the script directory, but I figured I would leave that for Rails and plugins. I will over comment the code below to help with any parts of it that may raise questions.

# default rails environment to development ENV['RAILS_ENV'] ||= 'development' # require rails environment file which basically "boots" up rails for this script require File.join(File.dirname(__FILE__), '..', 'config', 'environment') require 'net/imap' require 'net/http' # amount of time to sleep after each loop below SLEEP_TIME = 60 # mail.yml is the imap config for the email account (ie: username, host, etc.) config = YAML.load(File.read(File.join(RAILS_ROOT, 'config', 'mail.yml'))) # this script will continue running forever loop do begin # make a connection to imap account imap = Net::IMAP.new(config['host'], config['port'], true) imap.login(config['username'], config['password']) # select inbox as our mailbox to process imap.select('Inbox') # get all emails that are in inbox that have not been deleted imap.uid_search(["NOT", "DELETED"]).each do |uid| # fetches the straight up source of the email for tmail to parse source = imap.uid_fetch(uid, ['RFC822']).first.attr['RFC822'] # Location#new_from_email accepts the source and creates new location location = Location.new_from_email(source) # check for an existing location that matches the one created from email source existing = Location.existing_address(location) if existing # location exists so update the sign color to the emailed location existing.signs = location.signs if existing.save # existing location was updated else # existing location was invalid end elsif location.save # emailed location was valid and created else # emailed location was invalid end # there isn't move in imap so we copy to new mailbox and then delete from inbox imap.uid_copy(uid, "[Gmail]/All Mail") imap.uid_store(uid, "+FLAGS", [:Deleted]) end # expunge removes the deleted emails imap.expunge imap.logout imap.disconnect # NoResponseError and ByResponseError happen often when imap'ing rescue Net::IMAP::NoResponseError => e # send to log file, db, or email rescue Net::IMAP::ByeResponseError => e # send to log file, db, or email rescue => e # send to log file, db, or email end # sleep for SLEEP_TIME and then do it all over again sleep(SLEEP_TIME) end

Location#new_from_email

The only piece of code that you might need to help what I showed above make sense is the Location#new_from_email method.

class Location < ActiveRecord::Base def self.new_from_email(source) attrs, email = {}, TMail::Mail.parse(source) # set signs attribute equal to subject in proper form attrs[:signs] = email.subject.blank? ? '' : email.subject.downcase.strip.titleize # set street equal to the body with email signatures stripped attrs[:street] = parse_address(email.body) # create new location from the attributes new(attrs) end def self.parse_address(body) body.split("



").first end end

Location#new_from_email Specs

While working on the new_from_email method, I created an emails directory inside fixtures that had several ways an email could be sent to YardVote. Then, I created a few specs to make sure that new_from_email was actually working.

describe Location do describe "#new_from_email" do it "should be subject" do location = Location.new_from_email(email_fixture(:red_no_subject)) location.should have_error_on(:signs) end it "should require body" do location = Location.new_from_email(email_fixture(:red_no_body)) location.valid? location.errors.full_messages.should include("We can't find a precise enough address match.") end it "should set street to email body" do location = Location.new_from_email(email_fixture(:red)) location.street.should == '1600 Pennsylvania Ave. Washington, D.C.' end it "should set sign equal to subject" do location = Location.new_from_email(email_fixture(:red)) location.signs.should == 'Red' end it "should work with mixed case subject" do location = Location.new_from_email(email_fixture(:red_mixedcase_subject)) location.signs.should == 'Red' end it "should work with upper case subject" do location = Location.new_from_email(email_fixture(:red_uppercase_subject)) location.signs.should == 'Red' end it "should work with multi line body" do location = Location.new_from_email(email_fixture(:red_address_two_lines)) location.valid? location.to_location.to_s.should == "1600 Pennsylvania Ave Nw

Washington, DC 20006" end it "should work with multi line body and email signature" do location = Location.new_from_email(email_fixture(:red_address_two_lines_sig)) location.valid? location.to_location.to_s.should == "1600 Pennsylvania Ave Nw

Washington, DC 20006" end end end

bin/mail_receiver_ctl.rb

Now that I could check for new email and process that email, it was time to daemonize the script. The benefit of daemonizing is that you get nice and easy start, stop and restart commands, along with a PID. The PID makes monitoring your script possible, which is needed because it is bound to crash or begin sucking up to much memory.

require 'rubygems' require 'daemons' dir = File.dirname(__FILE__) Daemons.run(dir + '/mail_receiver.rb')

What, you wanted more? Now you can run commands like ruby bin/mail_receiver_ctl.rb start to start your script and ruby bin/mail_receiver_ctl.rb stop to stop it. Very handy.

config/mail.god

The whole setup was running fine for a while, but sure enough, a few days later, I noticed that a few of my emails were still sitting in the inbox. This was because the script had crashed. I fiddled around with god a little bit and came up with the following script, mostly stolen from Ryan Bates Railscast on god.

RAILS_ROOT = File.expand_path(File.join(File.dirname(__FILE__), '..')) God.watch do |w| # script that needs to be run to start, stop and restart script = "ruby #{RAILS_ROOT}/bin/mail_receiver_ctl.rb" # attaching rails env to each script line to be sure the daemon starts in production mode rails_env = "RAILS_ENV=production" w.name = "mail-receiver" w.group = "mail" w.interval = 60.seconds w.start = "#{script} start #{rails_env}" w.restart = "#{script} restart #{rails_env}" w.stop = "#{script} stop #{rails_env}" w.start_grace = 20.seconds w.restart_grace = 20.seconds w.pid_file = "#{RAILS_ROOT}/log/mail_receiver.pid" w.behavior(:clean_pid_file) w.start_if do |start| start.condition(:process_running) do |c| c.interval = 10.seconds c.running = false end end w.restart_if do |restart| restart.condition(:memory_usage) do |c| c.above = 100.megabytes c.times = [3, 5] end restart.condition(:cpu_usage) do |c| c.above = 80.percent c.times = 5 end end w.lifecycle do |on| on.condition(:flapping) do |c| c.to_state = [:start, :restart] c.times = 5 c.within = 5.minute c.transition = :unmonitored c.retry_in = 10.minutes c.retry_times = 5 c.retry_within = 2.hours end end end

Conclusion

Two hours, three files and a few extra methods later, YardVote could receive email. Before too long, I will be adding custom email addresses per person into an application such as Flickr and Highrise. I will be sure to post on that here, but, for now, I hope this helps people get started. Also, you can see the code actually implemented in the app on github.

I think we need to get more knowledge share on how to do things like this in Rails. If you have added receiving emails into your Rails app, how did you do it? Post a comment or link to an article that you wrote or used. Below are a few of the articles I have seen around the interwebs.

Receiving Email in Rails links