I wanted to set up a prometheus machine for me to monitor random stuff, but I was always postpone that because I didn’t want to use SSH port-forwarding, firewalls, create a VPC and/or setup an OpenVPN server or anything like that.

I just wanted something simple to maybe authenticate with github and go on.

Looking into some random GitLab wiki (I don’t remember which one specifically), I found about oauth2_proxy, and it seemed like a good idea.

Since this is a simple setup, I used docker-compose and rsync to set up all the environment.

The general idea is quite simple:

all things but nginx listen on 127.0.0.1 only;

only; nginx listens on 80 and proxy_forward s to oauth2_proxy and the other services: / forwards to prometheus; /grafana forwards to grafana; /alertmanager forwards to alertmanager; all of the above authenticate using proxy_forward and nginx’s auth_request directive.

and s to oauth2_proxy and the other services:

So, let’s get this thing started!

First one is prometheus itself:

# docker-compose.yml version : '3' services : prometheus : image : prom/prometheus ports : - "127.0.0.1:9090:9090" command : - '--web.external-url=http://m.carlosbecker.com/' # content hidden for the sake of brevity

The important parts are:

listen on 127.0.0.1:9090 - so it won’t be exposed to the world;

- so it won’t be exposed to the world; the web.external-url with the final URL: this is required for external links to work properly.

We will also need alertmanager to route the alerts, right? Let’s do it:

# docker-compose.yml version : '3' services : # content hidden for the sake of brevity alertmanager : image : prom/alertmanager ports : - "127.0.0.1:9093:9093" command : - '--web.external-url=http://m.carlosbecker.com/alertmanager/' # content hidden for the sake of brevity

The important parts are:

listen on 127.0.0.1:9093 - so it won’t be exposed to the world;

- so it won’t be exposed to the world; the web.external-url with the final URL: this is required for external links to work properly.

We also need to add alertmanager to prometheus.yml :

# prometheus.yml alerting : alertmanagers : - path_prefix : /alertmanager/ static_configs : - targets : - alertmanager: 9093 scrape_configs : - job_name : 'alertmanager' metrics_path : /alertmanager/metrics static_configs : - targets : [ 'alertmanager:9093' ]

Note that we change the path_prefix in the alerting section and also the metrics_path in the scrape_configs section.

Everyone likes dashboards, and grafana is DOPE for doing that. Let’s add it as well:

# docker-compose.yml version : '3' services : # content hidden for the sake of brevity grafana : image : grafana/grafana: 5.1.3 restart : always user : "0" volumes : - /local/path/to/grafana.ini:/etc/grafana/grafana.ini:ro ports : - "127.0.0.1:3000:3000" # content hidden for the sake of brevity

Here things get a little more tricky: we also have a granafa.ini config file. Here are its contents:

; grafana . ini [server] protocol = http domain = m . carlosbecker . com root_url = % ( protocol )s: // % ( domain )s /grafana/ [users] allow_sign_up = false auto_assign_org = true auto_assign_org_role = Admin [auth . proxy] enabled = true header_name = X - Email header_property = email auto_sign_up = true

We have a couple of important things to look at here. On the docker-compose.yml file:

listen on 127.0.0.1:3000 - so it won’t be exposed to the world;

- so it won’t be exposed to the world; mount the grafana.ini config file;

And in the grafana.ini file:

In the server section:#

The root_url defines the /grafana/ suffix in the root.

This is needed because otherwise, even with proxy_pass on nginx, grafana keeps trying to redirect to / , as mentioned on the beggining, prometheus will leave on / . This config file fixes that;

In the users and auth.proxy sections:#

Those sections tell grafana to auto-create an user when someone authenticates through the proxy (using the X-Email header) and to give this user the Admin role and assing to the organization. It also disables the sign up form.

This way we don’t need to login on both oauth2_proxy and grafana: You log in within oauth2_proxy and everything just works!

Finally, let’s require authentication using oauth2_proxy. I’m using GitHub for that, so, the first thing I did was to create a new GitHub app. You can find the details on the oauth2_proxy README.

Once I had the client ID and secret, it was pretty straightforward:

# docker-compose.yml version : '3' services : # content hidden for the sake of brevity proxy : image : caarlos0/oauth2_proxy ports : - 127.0.0.1 : 4180 : 4180 command : - '-client-id=123' - '-client-secret=456' - '-provider=github' - '-github-org=caarlos0-m' - '-email-domain=*' - '-cookie-secret=foo bar 1234' - '-cookie-secure=false' - '-upstream=http://nginx:80' - '-http-address=0.0.0.0:4180' - '-redirect-url=http://m.carlosbecker.com/oauth2/callback' - '-set-xauthrequest=true' # content hidden for the sake of brevity

The important things here are:

listen on 127.0.0.1:4180 - so it won’t be exposed to the world;

- so it won’t be exposed to the world; upstream is set to the nginx container;

is set to the nginx container; http-address is set to listen on 0.0.0.0 so we can expose the service to the host ( oauth2_proxy listens on 127.0.0.1 by default);

is set to listen on so we can expose the service to the host ( listens on by default); redirect-url must be the same as the one informed while creating the GitHub app;

must be the same as the one informed while creating the GitHub app; client-id , client-secret and provider are the GitHub oauth2 settings;

, and are the GitHub oauth2 settings; github-org is the required org an user needs to be member of to be allowed in;

is the required org an user needs to be member of to be allowed in; email-domain could be an additional email domain filter - for me the org filter is enough;

could be an additional email domain filter - for me the org filter is enough; set-xauthrequest is set to true se we can pass through the user and email headers - grafana uses this header to auto-create an user and log it in;

is set to true se we can pass through the user and email headers - grafana uses this header to auto-create an user and log it in; cookie-secure is set ot false due the lack of https. I’ll manage to add let’s encrypt anoother and will create a new post.

Finally, the last part: nginx!

First, let’s create a nginx config file:

# nginx.conf server { listen 80 ; server_name m . carlosbecker . com; location /oauth2/ { proxy_pass http: // proxy:4180; proxy_set_header Host $host ; proxy_set_header X - Real - IP $remote_addr ; proxy_set_header X - Scheme $scheme ; proxy_set_header X - Auth - Request - Redirect $request_uri ; } location = /oauth2/ auth { proxy_pass http: // proxy:4180; proxy_set_header Host $host ; proxy_set_header X - Real - IP $remote_addr ; proxy_set_header X - Scheme $scheme ; proxy_set_header Content - Length "" ; proxy_pass_request_body off; } location /alertmanager/ { auth_request /oauth2/ auth; error_page 401 = /oauth2/sig n_in; auth_request_set $user $upstream_http_x_auth_request_user ; auth_request_set $email $upstream_http_x_auth_request_email ; proxy_set_header X - User $user ; proxy_set_header X - Email $email ; auth_request_set $auth_cookie $upstream_http_set_cookie ; add_header Set - Cookie $auth_cookie ; proxy_pass http: // alertmanager:9093 /alertmanager/ ; } location /grafana/ { auth_request /oauth2/ auth; error_page 401 = /oauth2/sig n_in; auth_request_set $user $upstream_http_x_auth_request_user ; auth_request_set $email $upstream_http_x_auth_request_email ; proxy_set_header X - User $user ; proxy_set_header X - Email $email ; auth_request_set $auth_cookie $upstream_http_set_cookie ; add_header Set - Cookie $auth_cookie ; proxy_pass http: //g rafana:3000 / ; } location / { auth_request /oauth2/ auth; error_page 401 = /oauth2/sig n_in; auth_request_set $user $upstream_http_x_auth_request_user ; auth_request_set $email $upstream_http_x_auth_request_email ; proxy_set_header X - User $user ; proxy_set_header X - Email $email ; auth_request_set $auth_cookie $upstream_http_set_cookie ; add_header Set - Cookie $auth_cookie ; proxy_pass http: // prometheus:9090 / ; } }

Most of it is based on the oauth2_proxy README examples.

Now, let’s add it to the docker-compose.yml file:

# docker-compose.yml version : '3' services : # content hidden for the sake of brevity nginx : image : nginx ports : - 80 : 80 volumes : - /local/path/to/nginx.conf:/etc/nginx/conf.d/m.conf:ro # content hidden for the sake of brevity

Yeah, this one is pretty simple! Now we finally expose one service - on the port 80 , and we mount our config file to the /etc/nginx/conf.d folder.

nginx will be the only thing facing the internet for real, and it will route traffic to the right places.

So that’s it! I’m running this on http://m.carlosbecker.com - which, given that everything is working as expected, you won’t be able to access, and it is working great.

I hope this is useful to you somehow, enjoy!