By now we had made an important discovery:

Analyzing traffic using only HTTP headers isn't enough.

We needed to go deeper. We had a Runtime Application Self-Protection (RASP) tool gave us user context, so we built on that.

Lock a user when they behave like a bot

When a user fails to log in 7 times in 1 minute, lock their account and send them an automated email.

Our runtime tool actually wasn't able to do this on its own. We needed to use an open source library (or own implementation) for the actual act of locking an account. We ended up forgetting about using the RASP tool and hooking into our own user context. It took a few weeks, but we ultimately shipped it and felt pretty good about ourselves.

Then we learned that people forget their password all the time. That reality coupled with the fact that bots spray and pray was causing thousands of accounts to get locked, and people were getting frustrated by the process of unlocking their account – especially when it happened more than once. It also scared a lot of people into thinking their account had been hacked.

This was definitely not going to work, but we had at least addressed the problem of IP rotation at this point: even if an attacker used a new IP address every request, we would still be able to tie their activity to one specific email address, and act accordingly.

Send a user an email whenever they log in from a new IP address

This was a great way to empower the user, but it doesn't actually stop the attack. Further, identity theft is special in that we would see the thief delete the warning email from their account. This occurred commonly enough that we knew that we weren't done. At this point, we finally decided to bring in the big guns...

Evaluate user activity on a risk score

We were hit up by one of the hottest CDN application firewall companies. They offered us their brand new service that used machine learning to automatically detect bot traffic. We were thrilled, and I imagine they were too as the tool cost us $60,000 per year to start.

Even before the tool went live, we could see issues with the simulated results it was producing. There is such a thing as good bot traffic (e.g. Google), and it was unfortunately detecting a lot of it. Also, there was little we could do to whitelist our good bots because attackers were using virtually identical HTTP headers. To make matters worse, there was no way for us to tune the algorithm. We could eye certain traffic as legitimate, but there was no way to tell the tool that.

We had the machine, but not the learning.

We searched for other tools but came up with the same results. We were shocked, and begged them to get out of our contract within a month. Close call.

User login from new IP address gets an email; requires confirmation; sends SMS code

Gmail notably does this, and it's pretty good! We finally felt satisfied with our security, but holy cow did we come a long way from where we started. In order to stop IP rotation while still giving our users a positive experience, we had to utilize a RASP that knew what email was being tried per login attempt instead of just IP address, then add this entire product flow thereafter. How am I going to advocate for doing this at every job? It must have cost us over $100,000 in experimentation and person-power!

And from a user experience perspective...