View this email in your browser Important Security Notification From Mandrill



We discovered evidence on March 10 that automated attempts were made against Mandrill's internal logging servers in an effort to use them in a botnet. Analysis of the impacted servers, including network traffic logs and files present on the servers, indicates that these attempts were unsuccessful. There are no signs that the servers were targeted to access the data stored on them.



Upon further investigation, we found that the opportunity for this attack stemmed from a firewall change we made on February 20 in order to more granularly control access to some of Mandrill's servers. Parts of Mandrill's infrastructure are hosted with Amazon Web Services (AWS), and we use EC2 Security Groups to control access. One change was made to a security group that contained more servers than we intended to affect. As a result, a cluster of servers hosting Mandrill's internal application logs was made publicly accessible instead of allowing internal-only access.



As soon as we discovered this vulnerability on March 10, we took several immediate steps to mitigate the potential impact: Impacted servers were disabled and taken offline, with log files and data backed up for further investigation. These servers will not be used in any way going forward.

In accordance with internal security protocols, we updated all of Mandrill's internal credentials, including SSH keys and our own API keys/passwords for external services.

The firewall rules in use for all of our AWS security groups were audited and updated to ensure they were correctly limiting access and that no other servers were impacted by the earlier change.

We're also making changes to our internal processes for managing security groups and firewall rules. That includes placing all security group rules into version-controlled configuration files with a tool to apply those rules via the EC2 API. This will allow us to more rigorously vet any rules or rule changes in advance, including more automated testing and robust audit logs for any proposed and approved changes. We're implementing additional regular, automated audits of our security groups as well.



There's no evidence that any customer data was queried or exported, but unfortunately, we can't completely rule out the possibility of access. So, we're being paranoid and letting you know the worst-case scenario. Although it's extremely unlikely, if we assume the attackers were able to access information stored on the servers when the firewall rules were changed, the following data about your Mandrill account could have been accessed: Internal logs with basic log data about emails sent between February 6 and March 10. These logs include sender address, recipient address, and subaccount used (if any), but do not include custom metadata or message content. We're writing to let you know that we recently discovered a security vulnerability in Mandrill's infrastructure that you should be aware of. At this time, we're confident that no customer data was compromised as a result of the vulnerability, but we feel it's our responsibility to let you know exactly what happened and what we're doing about it.We discovered evidence on March 10 that automated attempts were made against Mandrill's internal logging servers in an effort to use them in a botnet. Analysis of the impacted servers, including network traffic logs and files present on the servers, indicates that these attempts were unsuccessful. There are no signs that the servers were targeted to access the data stored on them.Upon further investigation, we found that the opportunity for this attack stemmed from a firewall change we made on February 20 in order to more granularly control access to some of Mandrill's servers. Parts of Mandrill's infrastructure are hosted with Amazon Web Services (AWS), and we use EC2 Security Groups to control access. One change was made to a security group that contained more servers than we intended to affect. As a result, a cluster of servers hosting Mandrill's internal application logs was made publicly accessible instead of allowing internal-only access.As soon as we discovered this vulnerability on March 10, we took several immediate steps to mitigate the potential impact:We're also making changes to our internal processes for managing security groups and firewall rules. That includes placing all security group rules into version-controlled configuration files with a tool to apply those rules via the EC2 API. This will allow us to more rigorously vet any rules or rule changes in advance, including more automated testing and robust audit logs for any proposed and approved changes. We're implementing additional regular, automated audits of our security groups as well.There's no evidence that any customer data was queried or exported, but unfortunately, we can't completely rule out the possibility of access. So, we're being paranoid and letting you know the worst-case scenario. Although it's extremely unlikely, if we assume the attackers were able to access information stored on the servers when the firewall rules were changed, the following data about your Mandrill account could have been accessed: At this time, you don't need to make any changes to your Mandrill account. We realize that notifying you may be an overreaction given the evidence, but we wanted you to be aware of the issue. Finally, we are deeply sorry for our error. We're committed to making the necessary changes to prevent something like this from happening again. Keeping our customers' data secure is integral to everything we do at Mandrill, and we're using this incident to identify and prioritize improvements to our internal systems and processes.



If you have any questions, you can reply directly to this email or reach us at security@mandrill.com.