Step-by-step guide to modern & secure CI setup

For modern application development, it is very important to deliver changes to the online environment as quickly as possible. This allows product owners to see and test the integrity of evolving products and provide feedback to the development team more quickly, saving a lot of money on improvements.

Today we are going to show you easy professional steps to setup awesome continuous integration flow using DroneCI, GitHub and Slack!

Motivation and tools

We mentioned reasons why CI is a must-have for Agile methodology and Scrum tasks management. When you have automated CI flow you can maintain environments tied to repository branches. For example, one of the most elite and most advanced usages is branch sandboxing. If you have a branch called feature/analytics-page and your web app is hosted on domain myapp.io, you will be able to deploy this branch to analytics-page.myapp.io with git push to this branch. Even if you don't want to bring branches online you might apply this approach to run unit/e2e tests on a branch to allow/deny merge.

No matter which deploy system you use – whether it be Docker, Serverless framework, Ansible or plain script with Secure Shell command – you can make it run automatically each time you push changes to your git repository.

To use the full power, we recommend to split environments to e.g. dev, staging, and live. This will allow developers to always push locally tested changes to dev (or make pull-requests from branches to dev), and then progressively move changes to live.

To make this compatible with most projects and technologies, in this post we will stick to:

- GitHub – repository hosting provider

- Drone – worker with a web dashboard which will run the build script for each of your projects. This must be hosted on some cheap Virtual Private Server (VPS), which we will call the build server.

- Slack – work messenger which we will use to see notifications about the build process (vuild started/build Finished).

Drone on VPS server

To host Drone we need to get a VPS with SSH access and public IP. To provision our build server we will use Docker, which allows very quick setup of all required software and provides the ability in the future to recover a build server state with all software installed very quickly.

We will use the simplest Amazon EC2 t2.micro instance (1 GiB RAM). It is free for new AWS accounts for 1 year and affordable after 1 year at ~10$/month (price may slightly vary depending on region). Instead of Amazon EC2, you can use any VPS hosting service or just follow our steps.

1. Sign In/Sign Up into AWS console https://aws.amazon.com/. For better ping select an AWS region closer to you, e.g. I used Europe Frankfurt .

2. Go to Services → EC2 → Key Pairs (find in the left menu) . Click Create key pair . Use pem format. That key will be used to connect to the EC2 instance later on.

3. Go to Services → EC2 → Instances and click Launch Instance .

4. Using search select any Ubuntu Server 64-bit x86 image. I used Ubuntu Server 18.04 LTS .

5. Select an Instance Type . I use t2.micro .

6. Click Next: xxxx couple of times until you hit the Configure Security Group page. There you will need to add SSH , HTTP and HTTPS using the Add rule button so the ports are available for connection.

7. Click Review and Launch . When you click on Launch you will be prompted to choose or create a new key pair. Choose the key pair you already created, hit checkbox and click on Launch . Instance will start booting up and you can create and assign static IP to it.

8. Go to Services → EC2 → Elastic IPs and click Allocate Elastic IP address and then Allocate .

9. Click on the checkbox on the left side of the IP item you've just created, and click on Actions → Associate Elastic IP address to associate the IP address to EC2 instance.

10. Choose an instance and its private IP and click Associate . Great! You have static IP mapped to your instance. Please note the IP address as we will need it. For me, it is 3.126.16.204 .

Now we have to get a domain for our drone instance.

Option 1: Use your subdomain as it is nice and secure (with free Let's Encrypt https certificate).

You might want to configure the subdomain in your company/own domain for a build server. This automatically generates a built-in Let's Encrypt SSL certificate (feature included in Drone container) so your Drone will work over https .

For this, you only have to get a domain, such as mycompany.com and then in the DNS setup of your domain just add one A record with a name drone , which will point to the noted IP – in my case 3.126.16.204 . So your drone portal will be hosted on drone.mycompany.com .

For example, I will use drone.devforth.io , which is hosted in CloudFlare DNS (without CloudFlare proxy).

Option 2: Less secure (http) using Amazon public domain instead of own subdomain.

If you don't have your own domain or access to its DNS settings, the simplest solution will be using the built-in Public DNS domain of your instance provided by Amazon. Go to Services → EC2 → Instances find yours in the list, click on it and find Public DNS (IPv4)

For me, this domain will be ec2-3-126-16-204.eu-central-1.compute.amazonaws.com . To check you should see the noted IP in this domain. This endpoint can be used with the HTTP URL scheme.

Install Docker & Docker Compose to EC2 Machine

1. Using bash command line, navigate to the directory with a drone-kp.pem key, move it to home (or another preferred dir) and change permission to the file by running the following commands in a terminal:

mv drone-kp.pem ~/drone-kp.pem chmod 700 ~/drone-kp.pem

On Linux/MacOS you can use a native terminal emulator, while on Windows 10 we recommend using WSL Ubuntu (or at least Git Bash).

2. Connect to EC2 instance with SSH:

Instead of ec2-3-126-16-204.eu-central-1.compute.amazonaws.com , use your domain. Now you should see the Ubuntu prompt in a terminal, so you are inside of it!

3. Now we need to install docker software:

sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl gnupg-agent software-properties-common

4. Add docker GPG key for repository:

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

5. Add docker repository:

sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $( lsb_release -cs ) stable"

6. Install docker-ce :

sudo apt-get install docker-ce docker-ce-cli containerd.io

7. Install docker-compose :

sudo curl -L "https://github.com/docker/compose/releases/download/1.25.4/docker-compose- $( uname -s ) - $( uname -m ) " -o /usr/local/bin/docker-compose && sudo chmod +x /usr/local/bin/docker-compose

Create GitHub application for Drone

During an initial Drone setup, we will define a connection to our GitHub account. This account should then have access to repositories of projects which we will build inside of Drone (directly or via GitHub organizations).

1. Go to Developer settings.

2. Click Register a new application .

3. Set Homepage URL to a domain on which Drone will be hosted.

If you have a domain it will be: https://drone.mycompany.com

If you use Amazon public domain it will be something like http://ec2-3-126-16-204.eu-central-1.compute.amazonaws.com (in this case you can use HTTP only later to access your Drone; HTTPS certificate can not be generated for *.amazonaws.com subdomains) Set Authorization callback URL to Homepage URL + /login . See an example:



4. After you create an application you will see fields Client ID and Client Secret . These fields will be needed later.

Deploy Drone to EC2 machine using Docker

1. Clone our compose template and copy .env.sample to .env :

git clone https://github.com/devforth/drone-compose-template.git cd drone-compose-template/ cp .env.sample .env

2. Open .env in any CLI editor (e.g. nano .env ) and fill all variables:

3. Now run sudo docker-compose up --build -d and after it is finished you should be able to go to your domain (e.g. https://drone.mycompany.com or http://amoazon-dns-domain).

TIP: To debug drone output you might want to start it without -d flag TIP: To stop containers run sudo docker-compose down

4. Upon going to Drone website you should be redirected to GitHub OAuth authorization. After you complete authorization you should see your repositories (it can take some time to sync up with GitHub).

Drone CD/CI example project configuration

As an example, I will show you the building of a very simple project with one sh file which does an echo. You can just create a fork into your account from this one.

Go to Drone, Synchronise repository list, find your repo and Activate it by clicking Activate .

Now in repo create a configuration file .drone.yml with the following content:

kind : pipeline name : default type : docker steps : - name : build image : node commands : - sh build . sh

This runs our sh file. Simply push the file and Drone will automatically start building your project! You will notice it working by seeing the yellow drone favicon in the browser tab. To see status you can check the Activity Feed tab.

For more examples, you can visit the Drone documentation page.

Slack Integration

First, we have to add a webhook to the desired channel so Drone can send messages to it: simply visit https://app.slack.com/services/new/incoming-webhook, choose your workspace in the top right corner, select desired channel and click on Add . Copy webhook URL.

Slack webhook is a sensitive value because it can be used to post into a Slack channel. That's why it has to be stored inside of Drone and not in a repository, as it minimizes the number of people who can access it. In Drone these sensitive variables are called Secrets. Go to project settings and add slack_webhook .

Now you need to add a step after the build step to .drone.yml :

- name : slack-end image : plugins/slack settings : webhook : from_secret : slack_webhook username : Drone template : > {{repo.name}}/{{build.branch}} - #{{build.number}} {{uppercasefirst build.status}} after {{since build.started}} (<{{build.link}}|Open>)

As you can see to use Secret you just have to specify from_secret: <secret name> . This works for all values inside of .drone.yml .

You can customeize your message using template parameters. For more parameters and template references visit the plugin documentation page.

After the build, you should see a message in the desired channel.

If you want to receive a message at the start of build, you can also add a similar step as the first and slightly change the template. Then the whole .drone.yml config will look like this:

kind : pipeline name : default type : docker steps : - name : slack-begin image : plugins/slack settings : webhook : from_secret : slack_webhook username : Drone template : > {{repo.name}}/{{build.branch}} - Started #{{build.number}} "${DRONE_COMMIT_MESSAGE}" by {{build.author}} (<{{build.link}}|Open>) - name : build image : node commands : - sh build.sh - name : slack-end image : plugins/slack settings : webhook : from_secret : slack_webhook username : Drone template : > {{repo.name}}/{{build.branch}} - #{{build.number}} {{uppercasefirst build.status}} after {{since build.started}} (<{{build.link}}|Open>)

The build script

In our case, we have a simple bash script build.sh in the test repo:

echo "Building..." echo "The current branch is ${ DRONE_BRANCH } " echo "The current commit hash is ${ DRONE_COMMIT_SHA } " echo "Building done."

DRONE_BRANCH and DRONE_COMMIT_SHA are environment variables substituted by Drone which will have a branch on which build was triggered. If you look into build step details you will see an output of this script:

Having the DRONE_BRANCH variable is enough to implement most of the flows, including branches sandboxing. If you want more variables check Drone documentation

Troubleshoot

If you can't see project of repository which belongs to organisation you have to configure third-party access