Automation pipelines as a security enabler

Let’s consider automation pipelines from a security perspective. Pipelines can be a security enabler. Secure code in a developer’s machine, can result as insecure code running in production. Especially when there is manual intervention in the process. Automation pipelines can mitigate that risk. We must ensure that code can be promoted to production only via the pipeline and in doing so, we greatly minimize the attack surface.

However, a pipeline that can be compromised does not provide much security assurance. Marc van Lint, a colleague and automation expert, preaches that our pipelines are as important as our application code. This is especially true when it comes to security. A lot of concepts from DevOps are already in place to help us with this. A pipeline should be defined as code, persisted in a secured repo and versioned. That way, we have good visibility of all the steps for our code reaching production, only authorised users can make changes to it (least privilege principle) and any changes can be traced to specific individuals.

After ensuring our pipeline is secure, we have to make it useful. On a very high level, a pipeline is mainly about testing and promoting to different environments. A good approach when it comes to pipelines is to do the cheap and quick first. There are a lot of tools at our disposal that we can use out of the box. Static application security testing (SAST) and dynamic application security testing (DAST) are a must have in our pipelines. Then there are compliance tests, for example for PCI DSS. OWASP maintains a list of security testing tools[1], evaluate and pick those you need. Then of course tests we defined ourselves. Unit tests, integration tests, regression tests, performance test, failure tests. The list is long. Tests are a confidence builder. With each test we can be more certain that our code is secure. And also important, we generate supporting evidence (more on this later).

The above tools are used on code already in our repo or the application running in one of our environments. But we can start earlier. Tools like Talisman[2], a pre-push hook to Git which can catch suspicious files committed such as authorisation tokens and private keys before we even commit. And now that we are on the topic “keeping secrets” is a challenging topic and often processes in place are insecure. This is a good place for our pipelines to help. Generating certificates, following password policies and keeping secrets, secret should be automated. This ensures security by default. Pipelines again to the rescue!

Now about promoting through environments. The binaries are built once and they are used for all stages (test, acceptance, production). The exact same mechanism is used for release to each environment. Every commit gets built, tested and released to production right? Not so fast. We want to be quick with the release, true, but we cannot do it at the expense of security. For promoting there are a few security principles coming to place.  Separation of duties and asset classification being included. I will start with the latter because it might not be that obvious. In my view our test environment (and our code in it) can be classified as a lower value asset compared to the production environment. Vulnerability in the test environment is not as a big threat as vulnerability in production. Don’t get mad at me yet, I will explain. As stated before, we want our automation pipeline to be fast with as less intervention as possible while staying secure. A commit cannot reach production just because it passed all tests. It is possible that backdoors or vulnerabilities can slip through. There might be some processes necessary in place in your company before releasing to production. So it could be fine for a change to move to the test environment without review of the changes but there needs to be code review and documented approval for going to acceptance. Then promoting from acceptance to production, cannot be done just by approval of developers, but approval from product owners, managers and change requests must be part of it and automated. Thus the separation of concerns. All this of course observable in our pipeline and documented through the process.

So after putting all this effort on securing our pipeline and embed security tests in it and use it to secure our release processes, what is next? We want proof! Any artefact running in production should be accompanied with supporting evidence for the security measures taken. This can be very helpful for audits. Thankfully, having our pipelines as described that is an easy win,  provided that our tasks in the pipeline have associated comprehensive logging.  

We ensured that the pipeline is the only way to get code to production. The pipeline is observable and versioned. The code is tested for functionality, vulnerabilities, compliance, performance, failure and reports of these tests are generated. The 4 eyes principle is used with appropriate approvals before promoting to different environments. All these are documented, grouped, persisted and ready for audits for each artefact in production currently or in the past. Pipelines are neither the beginning nor the end of our security journey. But they can and should be a very important asset.

[1] https://www.owasp.org/index.php/Appendix_A:_Testing_Tools

[2] https://github.com/thoughtworks/talisman