Motivation

Like most reasonable product-loving application developers, I too enjoy the process of building something and not configuring something. The last thing I wanted to touch was DevOps. For years, I had been the most loyal follower of The Heroku God.

When I decided to switch to AWS Beanstalk, I spent about 3 whole days to get to a successful deploy. Then I mess up something. Then stuff gets so messed up as I trial and error I was so angry I deleted the whole setup to start again.

I spent the past 2 days going through the hell of climbing the Beanstalk of AWS (with new problems!), maturing every link on my Google Search to purple, talking to AWS Support for hours, and debugging from 8am to 11pm.

I’m writing this article because I have endured so much unnecessary pain for the past 2 days, but I have also learned a lot, and I believe the notes I have taken throughout this whole process could be useful for many others.

Not everyone is like Gilfoyle, after all.

Expectation

As a relative beginner in DevOps (I’ve only had minimal experience in Digital Ocean during my time at Fullstack Academy), getting into a deployment environment like AWS feels like learning how to code all over again.

As opposed to writing application logics, you’re configuring settings. You will start to appreciate the huge complicated system that your tiny codebase sits on top with. You will get exposed to lots of new things.

Spectrum

From left: Digital Ocean, AWS EC2, AWS Beanstalk, Heroku, Firebase

So what is AWS Beanstalk? Is it like Heroku? Well, not really.

From the spectrum of deployment environments, we have two ends. On one end, we have the really API-intensive and manual services like Digital Ocean and AWS EC2. On the other, we have services like Firebase and Heroku that basically reduce the need of configurations close to zero.

AWS Beanstalk sits in the middle. It already sets up a lot of the things for you, but you’re still using an AWS EC2 (and some other services), and you still have to do some configurations.

Getting Started

I have a Node.js app with a PostgreSQL database. You might have other stacks but I think the high level notes I have here should still be able to help.

For folks with my stack, there are two tutorial videos I found really helpful:

On YouTube

Source here.

2. On a Linkedin course. (You need to pay, but first month is free)

I actually encourage you to follow those two tutorials then use my notes as a reference for the things you’re about to do. I’ve made many mistakes even following those tutorials. You can follow them and my notes.

High Level Overview

Source here.

This is an Elastic Beanstalk application. From the top, you have the logo of Route 53 (awesome name). It’s where you have your domain registered and DNS configured.

When people visit your domain, they are directed to the Load Balancer. It’s one of first setups of Elastic Beanstalk. Think of it as the gate to your application.

Then you have your EC2 Instances. These are basically your servers. Instead of manually setting them up, Beanstalk gives you a PaaS-like way to set them up easily within the web console.

Notice your EC2s are wrapped around by the Security Group red line? Think of Security Groups like the security guards of your app. Only information from certain sources are allowed to pass through.

On the side, you also have your database. It’s also surrounded by Security Group because it only wants data to pass through from your Beanstalk app!

Getting Started

Let’s start from the basics.

After you have installed all the necessary setups like Python and EB CLI, you can run your first command at your application path:

eb init

This creates the Application. After choosing the options following the command, you now have an Application on AWS Beanstalk. Congrats.

Tips: If you are using WebSockets in your application, I was told by AWS Support that you need to choose Application Load Balancer in the setup. The default option Classical Load Balancer does not support WebSockets.

Then, you create what is called an Environment.

eb create

This is where you app will reside. You can have a bunch of these environments for testing and staging purposes. One is enough at this point though.

Database

While it’s creating an environment for you, it’s wise to start getting your database setup, which also takes time. For PostgreSQL, I had to set up a RDS, which stands for Relational Database Service.

There are 2 ways to set up a RDS.

1. Directly through your AWS Beanstalk console

Which is incredibly stupid because only one environment will be associated with your database. If you terminate an environment, there it goes with it. That’s what I did the first time because I didn’t want to configure anything.

2. Independently through AWS RDS

Which is the wise way because it can work with multiple environments. This is recommended by AWS. I did this on the second time but it requires some work to connect the database to your Beanstalk app.

Now go to AWS homepage and search for a service called AWS RDS. Then follow the steps and set up an RDS instance.

Tip: After you have your RDS, you would need to save the endpoint of that database somewhere for your configuration later on. Remember, don’t include your port number in the endpoint! (i.e. don’t include “:5432” in your endpoint)

Configurations

Now your environment is probably be created. It’s going to show an ugly red message of degraded on the web console, but don’t get discouraged. You’re supposed to get that. You haven’t configured anything yet! I’m with you.

Let’s go into Software Configuration. There are many things here, but the key ones are as follow:

1. Container Options:

Specify your node version & node command, which for me is:

npm run start-prod

If you don’t put anything there, it will run this in your app:

npm start

Tip: npm install is automatically run prior to your command when you put your app in Elastic Beanstalk, so you don’t need that in your command.

2. Environment Properties

This is where you connect your RDS, but since we’re here, let’s put the other necessary stuff too.

NODE_ENV = production NPM_CONFIG_PRODUCTION = true

Then add your other environment properties. If you have Google OAuth for example, you’d need to put your GOOGLE_CLIENT_ID and etc.

Connecting to your RDS

Lastly, let’s connect your RDS. In the same page, add another environment property called RDS_CONNECTION_URL and its value as follows:

RDS_CONNECTION_URL = postgres://[user]:[password]@[endpoint]/[db]

[user] & [password] are values you input when you were setting up your DBS. These are actually arbitrary values, according to all tutorials I’ve watched.

[endpoint] is the value you copied from your RDS earlier.

[db] is the database name you give to your RDS.

Tip: The database name is different from your RDS instance name! Also, change the variable within your app with whatever ORM you’re using. For example, I’m using Sequelize, so I have this in my app:

new Sequelize(process.env.RDS_CONNECTION_URL || [localhost]) { ...

Now you have your config for DB set up from your AWS Beanstalk. Awesome!

Security Groups

There’s just one last thing to do. Remember that Security Group thing? You have to tell your security guards who can pass into your building, so to speak.

This is probably the most confusing part, so get prepared.

After numerous painful experiences, here is what I recommend you follow:

Part 1: Get the Beanstalk Security Group value

Go to AWS homepage Go to your Beanstalk console Go into Configuration Go into Instance Copy down that value in “EC2 security groups” (label it Beanstalk)

Tip: I opted in a “new design” at one point and the security groups column shows a bunch of values with one selected by a radio button. That’s the value you want. “New design” in AWS doesn’t mean it’s better. It’s just new. Still confusing AF.

Part 2: Get the RDS Security Group value

Go to AWS homepage Go into your RDS console Go into your Instances Go into your App’s RDS Instance Copy down that value in the box “Connect”, column “Security group” (label it RDS)

Tip: You will see multiple lines in this column, but they all have the same value. It will probably be something like “rds-launch-wizard” or “rds-launch-wizard-1” followed by your Security Group ID.

Part 3: Configure Security Group in EC2

Go to AWS homepage Go into your EC2 console Go into Security Groups (scroll down on the left to find it) Find the Security Group with your RDS Group Name Click the “Inbound” tab below Click “Edit” Add a Rule, with Type “PostgreSQL” and Source using your Beanstalk Security Group ID. Click “Save”

Tip: You can double check if you have the right source by looking at the other Security Groups. There should be one Security Group with the name of your environment and Description that says “SecurityGroup for ElasticBeanstalk…” Check the Group ID and compare it with the value if you have for Beanstalk.

Voilà

Phew. Woah, what the f*ck happened?

Okay let me stop swearing to express how painful this is, but to congratulate you in completing the configurations for your first Elastic Beanstalk app!

Almost every tutorial skips directly to Part 3. And if you’re doing this for the first time, it’s definitely doable by just going to the EC2 Security Group settings directly.

But if you’re like me who messes up things and do things again and again… your EC2 console would be flooded with security groups. I was pretty much overwhelmed, looking at all those security groups and changing Inbound rules at random, which messed up things further.

It’s just all f*cked up that way.

So lessons learned the hard way, follow all 3 parts! 🙂

Application Side

I also have some notes on proper deployment from the application side. Even though I was on the phone with AWS Support for hours, we were discussing issues on the application nearly half of the time. For this, I really appreciate their incredible customer service and patience.

Tip: I’m on the business support plan, and the severity of my case was the highest one, which is production service down. Also, always select the option for them to call back. I once waited for 2 hours on an email reply. All they gave me was cr*p.

The following notes might be plain obvious to some, but it definitely wasn’t to me and at least one software engineer I talked to.

This is how your code runs, right? You have your codebase, you npm install and then you webpack or grunt , whatever you’re into, then you have code files that nobody can understand that actually run your application, locally or on AWS.

Now, we know AWS Beanstalk runs npm install for you, so library compilation definitely happens on AWS, but, what about your application?

Webpack and Grunt normally belong to devDependencies in your package.json so they are not supposed to be run on AWS. So it seems like you’d have to compile it locally, then deploy it to AWS.

It really troubled me because that supposedly tells me to not gitignore my compiled application file ( bundle.js in my case) which means I would have to upload my entire compilation to GitHub if I were to commit my changes.

Confused and petrified by this whole thing, I once tested by moving all my Webpack related libraries to dependencies and it worked, but that was idiotic.

Enter .ebignore

I’m going to paste a screenshot of a StackOverflow answer that saved me:

That was the AHA moment for me. I would still compile my application code locally, and I would deploy that compiled file to AWS, but I would still gitignore my compiled file.

In my .ebignore file, I have two lines:

node_modules

secrets.js

Basically, don’t deploy the node_modules folder (because AWS Beanstalk compiles the npm libraries) and don’t deploy the environment secrets I have.

Your .gitignore file stays the same, but what gets ignored by AWS Beanstalk is now different with the presence of .ebignore .

On .ebextensions

If you’re surfing on StackOverflow for the AWS Beanstalk questions you have, there are always these magic pill answers that tell you to put an extension file.

Sometimes it works. Sometimes it doesn’t. Sometimes it f*cks you up.

I’m going to paste this AWS Documentation, which I saw after I have deleted my whole Beanstalk Application to start again because I was so frustrated.