Security needs to adapt to increasingly fast continuous delivery in the container/Kubernetes world, and that means security as code, argued Mateo Burillo. At RebelCon.io 2019 he presented how to implement a DevSecOps process with continuous security.

Humans cannot reasonably review several builds a day; we need a security team that is able to think about security requisites, restrictions, and best practices and automate them using tools that natively understand microservices. To take full advantage of agility and responsiveness of DevOps, security must play an integrated role in the full cycle of your apps. Security should be integrated into the pipeline, said Burillo, using the security as code approach.

Burillo mentioned the benefits of implementing security as code:

The first and most important automation: you need to scale and speed up your security process to catch up with the microservices/containers world.

Visibility: any team member can access and discuss the security rules, eliminating know-how silos.

Traceability: now you have versioning and ownership for every change you make.

Containers are black boxes, which is good for operations but bad for security, said Burillo. It may take days or weeks before breaches are detected. It’s like CSI but then without a corpse, as the container is probably gone after the breach has happened.

The most obvious step to do at build time to build security into the product is integrating your CI/CD tool with an image scanner. People often associate container image scanning with Common Vulnerabilities and Exposures (known as CVEs lists or just "vulnerabilities") and vulnerability scanning, but that’s only the start, said Burillo. There is so much more you can do with an advanced image scanner: from compliance checks (NIST, PCI, MITRE), discovering leaked credentials, implementing your own application-level security practices…

There are several container-specific security compliance standards and a nice set of security capabilities provided by Kubernetes out-of-the-box, so avoid reinventing the wheel, Burillo suggested.

The most common use case nowadays is the integration of the image scanning software with your CI/CD pipeline, be it Jenkins, AWS CI/CD, Bamboo or any other. Burillo mentioned another use case that is maybe more innovative, but not less necessary: doing regular image checks for your runtime containers. What happens if a new 0-day vulnerability is discovered after the image is declared "clean" by the CI/CD, he questioned, or if the container is attacked, modifying the original conditions?

Burillo mentioned that image scanning is necessary but not enough; you also need runtime security. Traditional Linux security tools do not provide container runtime visibility; he suggested modernizing the toolset.

InfoQ spoke with Mateo Burillo, technical writer at Sysdig, after his talk at RebelCon.io 2019.

InfoQ: What can be done to bring agility to the security world?

Mateo Burillo: I think the main concept to adopt isn’t too different from the changes experienced by the IT infrastructure or QA: implementing security as code. Security as code implies that your rules need to be versioned and kept in a repository. Your security ops team will push to that repository and your environment should be able to dynamically pull and enforce those rules. Let me use an example to help clarify this concept: Implementing image scanning with Anchore, for instance, your container images will always be scanned to detect CVEs. Then, your security team decides that you don’t want any container running as root or leaking AWS credentials. The security team modifies the scanning policy and pushes the new version to the internal repository

Your Anchore engine is scripted to detect, download and apply this update

Your CI/CD tool, let’s say Jenkins, is in charge of building the image. It can be easily integrated with Anchore and one of the steps in the build pipeline runs this image scanning verification. Only the images that get approved by Anchore actually get signed by Jenkins and pushed to the internal registry

Kubernetes is configured, using an Admission Controller, to refuse to run any image that has not been signed by your CI/CD tool In the example above, the security team only needs to design high-level security goals; everything else is automated.

InfoQ: What can be done to ensure that applications that are running inside containers are operating in a secure way?

Burillo: Being aware that there is no such thing as 100% guaranteed security in IT, there are several things that we can do to get as close as possible: Image scanning, to detect known vulnerabilities and follow security best practices that are more relevant to your IT sector and the kind of applications you deploy. For example: You know that your application only needs to mount /web/html from an external volume; any other file system operation in the Dockerfile should raise a red flag You have a licence incompatibility with GLPv3; any software package declaring that license needs to be reported

Verifying your containers against the container-specific security compliance frameworks. Not reinventing the wheel is one of the golden rules in security, that’s why we have standards, to name a few: NIST SP 800-190 application container security guide Center for Internet Security (CIS) Kubernetes Benchmark Payment Card Industry Data Security Standard or PCI DSS

You need deep visibility and traceability to understand how your containers are built and how they operate during runtime. You cannot stop the security threats you don’t see.

Implementing runtime security rules and automatic remediation. Containers are simple machines, which is a huge advantage during runtime because you can easily predict their behaviour and immediately detect if there is something suspicious going on. Let’s use that feature to make our IT security better.

InfoQ: What security features does Kubernetes have, and what’s your advice applying them?

Burillo: There are really many of them and the best (worst?) part is that the list is continuously growing: RBAC, admission webhooks, pod security policies, network policies, automatic cert rotation, resource quotas… My advice here is to think like a statistician. There is a danger of going overdrive and just implementing as many rules as you can imagine. Collect data from past attacks, from your developer’s experience, from notorious security breaches in the ecosystem, from security-oriented Kubernetes newsletters and blogs. And then think: what is the next security rule that will maximize security and minimize the complexity I need to introduce? There is a nice example of a vulnerable Kubernetes cluster we exposed to the Internet as a honeypot. The cluster is intentionally flawed, but the attack, attack detection and forensics are real. These security breach drills will help you identify and block actual active attacks in your ecosystem. Bear in mind that they are always mutating and adapting; security is an arms race.

InfoQ: How can we detect anomalies in software that is running in containers?