We’re always searching new ways of implementing CI/CD at Eurotux, and in this post I’ll describe one of those by leveraging 3 components that we are using in our customers:

Gitlab

Terraform

Openshift

Application

The application we wanted to deploy is Fess Enterprise Search Server , (“Fess is Elasticsearch-based search server”) and we use it to scan our internal Wiki server and allow our teams to have a “google”-like search engine for that wiki. Fess supports all sorts of targets (file servers, web sites, databases) and it supports several authentication methods such as BASIC/DIGEST/NTLM/FORM (keep this in mind for the next minutes).

We use OpenShift which is a container based orchestration, so the first thing to do is to create a container for the application. Fortunately Fess already provides a base container image which I’ll use as the base container for the project and I will only improve on that. The first thing is to create a Dockerfile :

FROM docker.io/codelibs/fess:latest RUN perl -i -p -e "s/crawler.document.cache.enabled=true/crawler.document.cache.enabled=false/" /etc/fess/fess_config.properties ADD logo-head.png /usr/share/fess/app/images/logo.png ADD osdd.xml /usr/share/fess/app/WEB-INF/orig/open-search/osdd.xml ADD logo-head.png /usr/share/fess/app/images/logo-head.png RUN apt-get update && apt-get -y install libjson-perl COPY entrypoint.sh /usr/share/fess/run.sh ADD insert.sh /insert.sh # We are using oc 3.9 because the later ones require libcrypto.so.10 (see https://github.com/openshift/origin/issues/21061) RUN wget https://mirror.openshift.com/pub/openshift-v3/clients/3.9.63/linux/oc.tar.gz && tar zxf oc.tar.gz -C /usr/bin && rm oc.tar.gz ADD fess_config.properties /etc/fess/fess_config.properties

I did do some customization (like changing the logo to our company one), changing the entrypoint and adding the oc (openshift client) command. As one can easily understand, our internal wiki is password protected. It is a form-based username/password (you see why it is great that Fess supports form-based authentication) and I only need to provide the Fess server the username and password to access the wiki.

The entrypoint is changed so that when the container starts, will get the username and password from OpenShift secrets (that’s why the container installs the oc command), update the Fess server configuration and start indexing the wiki. As this is a stateless service, I don’t need to worry about saving state and using Persistent Volumes. If the container dies or gets redeployed, the search engine will re-index our wiki. This keeps this project simpler and cleaner. Here is a snippet of the insert.sh script:

if [ -z "$WIKIUSER" ]; then export WIKIUSER="`oc get secret wikiuser --template='{{.data.username}}' | base64 -d`" fi if [ -z "$WIKIPASS" ]; then export WIKIPASS="`oc get secret wikiuser --template='{{.data.password}}' | base64 -d`" fi curl -XPOST "http://localhost:9200/.fess_config.web_authentication/web_authentication" -H 'Content-Type: application/json' -d " { \"webConfigId\" : \"$CONFIGID\", \"updatedTime\" : 1509224726193, \"hostname\" : \"wiki.eurotux.com\", \"password\" : \"$WIKIPASS\", \"updatedBy\" : \"admin\", \"createdBy\" : \"admin\", \"createdTime\" : 1509224726193, \"protocolScheme\" : \"FORM\", \"username\" : \"$WIKIUSER\", \"parameters\" : \"encoding=UTF-8\

login_method=POST\

login_url=https://wiki.eurotux.com/Special:UserLogin\

login_parameters=username=\${username}&password=\${password}&auth_id=1&deki_buttons%5Baction%5D%5Blogin%5D=login\" }"

Terraform

We use Terraform to bootstrap the infrastructure required for the deployment of this application, which is responsible for the following:

OpenShift Project (Namespace)

Secrets (wiki username and password)

Granting permissions to the container default service account to access the secret (so that the container can fetch that info)

Granting the gitlab runner service account to edit this namespace objects (so that the deployment pipeline can deploy to this namespace)

Adding the anyuid scc to the deployer service account. The Fess container runs several services (actually this is an anti-pattern in the container world), and requires to run as root inside the container (later on it changes the uid to another)

Unfortunately, the terraform kubernetes provider is somewhat lacking in features comparing to others (like aws or azure provider). Because of that, I use a mix of internal resources like the kubernetes_namespace and null_resource as a wrapper to the oc command:

# Create namespace resource "kubernetes_namespace" "search" { metadata { annotations { name = "search-engine" } labels { owner = "npf" } name = "${var.namespace}" } lifecycle { # because we are using openshift, we have to ignore the annotations as openshift does add some annotations ignore_changes = ["metadata.0.annotations"] } }

# This container requires root, so we need to allow anyuserid resource "null_resource" "add-scc-anyuid" { provisioner "local-exec" { command = "oc -n ${kubernetes_namespace.search.id} adm policy add-scc-to-user anyuid -z deployer" } provisioner "local-exec" { command = "oc -n ${kubernetes_namespace.search.id} adm policy remove-scc-from-user anyuid -z deployer" when = "destroy" } }

As you can see, I use local-exec to spawn the oc command when there isn’t support for those features in the kubernetes terraform provider. As a result of a terraform apply :

Gitlab

At Eurotux we are using an internal gitlab server to house all our projects. As so, we make extensive use of its’ CI/CD capabilities. To implement the CI/CD I’ve created a .gitlab-ci.yml file to describe the pipeline:

image: $CI_REGISTRY/docker/base-builder stages: - review - staging - production - cleanup variables: OPENSHIFT_SERVER: https://oshift.install.etux:8443 OPENSHIFT_DOMAIN: oshift.install.etux .deploy: &deploy tags: - kubernetes before_script: - ci-bootstrap script: - "oc -n $CI_PROJECT_NAME get services $APP 2> /dev/null || oc -n $CI_PROJECT_NAME new-app fess --name=$APP --strategy=docker" - "oc -n $CI_PROJECT_NAME start-build $APP --from-dir=fess --follow || sleep 3s && oc -n $CI_PROJECT_NAME start-build $APP --from-dir=fess --follow" - "oc -n $CI_PROJECT_NAME get routes $APP 2> /dev/null || oc -n $CI_PROJECT_NAME create route edge --hostname=$APP_HOST --insecure-policy=Redirect --service=$APP" ...... ...... staging: <<: *deploy stage: staging tags: - kubernetes variables: APP: staging APP_HOST: $CI_PROJECT_NAME-staging.$OPENSHIFT_DOMAIN environment: name: staging url: http://$CI_PROJECT_NAME-staging.$OPENSHIFT_DOMAIN only: - master production: <<: *deploy stage: production tags: - kubernetes variables: APP: production APP_HOST: $CI_PROJECT_NAME.$OPENSHIFT_DOMAIN when: manual environment: name: production url: http://$CI_PROJECT_NAME.$OPENSHIFT_DOMAIN only: - master

The pipeline will create a review application when working on a git branch other than master so that I can review and fix things. When a merge (or a commit for that matter) occurs in master , it will deploy automatically to staging and then I can press play to deploy to production . Here is an example of the pipeline:

Here is a snippet of the pipeline running:

After that, i can browse to https://search.oshift.install.etux/ and I’m presented with the search engine webpage:

OpenShift

As you’ve figured out by now, all of this is running in our testing OpenShift cluster. We are using the 3.11 version of OpenShift, which features monitoring using Prometheus and Grafana (later on, I will detail some other interesting features, such as integration with Keycloak). OpenShift automatically provides some Grafana dashboards so that you can see what are the usage patterns:

One of the interesting things that these dashboards present is the lifecycle of the application (starting new containers and stopping the older ones).