Automation pipelines as a security enabler

Let’s consider automation pipelines from a security perspective. Pipelines can be a security enabler. Secure code in a developer’s machine, can result as insecure code running in production. Especially when there is manual intervention in the process. Automation pipelines can mitigate that risk. We must ensure that code can be promoted to production only via the pipeline and in doing so, we greatly minimize the attack surface.

However, a pipeline that can be compromised does not provide much security assurance. Marc van Lint, a colleague and automation expert, preaches that our pipelines are as important as our application code. This is especially true when it comes to security. A lot of concepts from DevOps are already in place to help us with this. A pipeline should be defined as code, persisted in a secured repo and versioned. That way, we have good visibility of all the steps for our code reaching production, only authorised users can make changes to it (least privilege principle) and any changes can be traced to specific individuals.

After ensuring our pipeline is secure, we have to make it useful. On a very high level, a pipeline is mainly about testing and promoting to different environments. A good approach when it comes to pipelines is to do the cheap and quick first. There are a lot of tools at our disposal that we can use out of the box. Static application security testing (SAST) and dynamic application security testing (DAST) are a must have in our pipelines. Then there are compliance tests, for example for PCI DSS. OWASP maintains a list of security testing tools[1], evaluate and pick those you need. Then of course tests we defined ourselves. Unit tests, integration tests, regression tests, performance test, failure tests. The list is long. Tests are a confidence builder. With each test we can be more certain that our code is secure. And also important, we generate supporting evidence (more on this later).

The above tools are used on code already in our repo or the application running in one of our environments. But we can start earlier. Tools like Talisman[2], a pre-push hook to Git which can catch suspicious files committed such as authorisation tokens and private keys before we even commit. And now that we are on the topic “keeping secrets” is a challenging topic and often processes in place are insecure. This is a good place for our pipelines to help. Generating certificates, following password policies and keeping secrets, secret should be automated. This ensures security by default. Pipelines again to the rescue!

Now about promoting through environments. The binaries are built once and they are used for all stages (test, acceptance, production). The exact same mechanism is used for release to each environment. Every commit gets built, tested and released to production right? Not so fast. We want to be quick with the release, true, but we cannot do it at the expense of security. For promoting there are a few security principles coming to place.  Separation of duties and asset classification being included. I will start with the latter because it might not be that obvious. In my view our test environment (and our code in it) can be classified as a lower value asset compared to the production environment. Vulnerability in the test environment is not as a big threat as vulnerability in production. Don’t get mad at me yet, I will explain. As stated before, we want our automation pipeline to be fast with as less intervention as possible while staying secure. A commit cannot reach production just because it passed all tests. It is possible that backdoors or vulnerabilities can slip through. There might be some processes necessary in place in your company before releasing to production. So it could be fine for a change to move to the test environment without review of the changes but there needs to be code review and documented approval for going to acceptance. Then promoting from acceptance to production, cannot be done just by approval of developers, but approval from product owners, managers and change requests must be part of it and automated. Thus the separation of concerns. All this of course observable in our pipeline and documented through the process.

So after putting all this effort on securing our pipeline and embed security tests in it and use it to secure our release processes, what is next? We want proof! Any artefact running in production should be accompanied with supporting evidence for the security measures taken. This can be very helpful for audits. Thankfully, having our pipelines as described that is an easy win,  provided that our tasks in the pipeline have associated comprehensive logging.  

We ensured that the pipeline is the only way to get code to production. The pipeline is observable and versioned. The code is tested for functionality, vulnerabilities, compliance, performance, failure and reports of these tests are generated. The 4 eyes principle is used with appropriate approvals before promoting to different environments. All these are documented, grouped, persisted and ready for audits for each artefact in production currently or in the past. Pipelines are neither the beginning nor the end of our security journey. But they can and should be a very important asset.

[1] https://www.owasp.org/index.php/Appendix_A:_Testing_Tools

[2] https://github.com/thoughtworks/talisman  

Devs will just dev! The Cloud Foundry promise

“Every company is a technology company” said Peter Sondergaard and evidence of this is all around us. But it was not so easy becoming a technology company, the entry barriers were high. Besides developing their business propositions, companies had to develop, maintain and operate the platform on top of which their businesses (i.e. applications) run. Until Cloud options and "X as a Service” models became available. It started mainly with Infrastructure as a Service (IaaS) offerings but like everything, it keeps evolving.

Rise of DevOps culture, automated pipelines, container technologies, microservices, all contributed to an improved situation. And all these are still evolving and getting increasingly popular. But still, businesses have to deal with things outside the development of their specific business propositions. There is still operational load to carry. And the load seems to be moved now to the hands of developers. Cloud Foundry helps eliminating this operational load, and the need of building platforms and utility components that have no relation to your business propositions. Cloud Foundry makes possible to develop only what contributes to your bottom line and it takes care of the rest. It allows developers, to just develop! 

Cloud Foundry (CF) is an open source platform for hosting cloud native applications. When running on CF you only have to manage your applications and data while CF takes care of the rest. At the same time it allows you to choose your underlying infrastructure, be it AWS, Openstack or you own. CF does not limit you on the kind of applications you can run on it. It supports many languages such as Java, Go, NodeJS, Ruby and others but most importantly, since it is open source, it allows the community to develop and provide what might be missing. The Cloud Foundry platform itself has many implementations and certified providers include Pivotal, IBM, Atos and SAP. As a user, you gain from the abstraction. You interact with all these platforms in the same way. If you know how to run your applications in one of them, you can do it in all of them. The above can be summarized as Cloud Foundry being a language agnostic, multi-vendor, multi-cloud environment for running cloud native applications.

Let’s see what you get from Cloud Foundry, before showing how it delivers:

• Always on: it ensures that your system has the resources you specify 24/7
• Easy elasticity: scale up, down in and out. Fast and possibly without downtime
• Language Agnostic: run your apps in the language and frameworks that suit you
• Out of the box functionality: common needs in any application are provided and can be used so you do not have to built it yourself
• Distributed tracing: distributed applications can be difficult to analyze logs and tracing but CF helps with that too
• Guards against cascading failure: the failure of one component will not bring others of your system down
• No vendor lock-in: multi-vendor, multi-cloud, you can switch where you run your applications any time. No changes in your actual application code necessary
• Enhanced security: ways that makes your application more secure will be discussed later
• You only need to care about your code! No need to manage the platform, just describe what you want and CF takes care of it for you

You can easily see how Cloud Foundry delivers the above by examining the platform itself. Each component has a specific purpose. 

Every Cloud Foundry provider has their own UI for interacting with an instance, but all CF implementations share the same API. Cloud Foundry Command Line Interface (CLI), is a terminal that allows you to communicate with the CF instance in which you want to run your apps (target). Underneath, the CLI makes a series of REST calls to your target. Through the CLI you can connect to the platform, and instruct Cloud Foundry to deploy your application, scale it, bind it to services and more. 

To show how easy it is to deploy and manage an application via the CLI and Cloud Foundry, let’s say that we want to deploy a Java app called myapp, to run on 2 instances, with 1G memory per node. This is simply done by:

cf push myapp -b java_buildpack -i 2 -m 1G -p ./myapp.jar --random-route

That’s it! Our application will run with the specifications instructed, will be accessible on a random auto-generated endpoint and CF will make sure that it stays at this desired situation at all times.

The Cloud Controller, is your interface with Cloud Foundry. It provides the API that listens to the requests made by the CLI and kick-starts the processes for deploying, running, scaling, monitoring and managing your applications. The Cloud Controller is in charge of keeping track of your desired state of your applications, meaning, what was specified for each app when it was “pushed”. It keeps track of such metadata on the Cloud Controller Database. It also stores the uploaded application (the jar in the above example) on the Blob Store.

The Router, is the first component (after the load balancer) that receives incoming requests to a given CF instance and directs the request either to the Cloud Controller or to the application listening on the endpoint specified in the request. Applications go on and off, their number of instances change, as well as the URIs (called routes) on which they are “listening”. The Router takes care to direct each call to the appropriate receiver while all these events are taking place. It achieves that using a routing table which maps the routes to applications. The routes are emitted via the nats message bus. An application can be bound to zero or more routes. A route can be mapped to zero or more applications. This gives us great flexibility, for example to easily make a new release with no downtime. Additionally, when handling requests, the router adds headers for enabling distributed tracing. It allows us to follow a logical request within the platform and all the internal services called in order to handle it.

Diego takes care of the lifecycle and health of your applications using a set of subcomponents with a specific purpose (see picture below). Its Garden component creates containers for the apps as instructed and keeps track on their actual state. Diego contains one or more virtual machines called Cells and the containers for the applications run in them. Knowing the actual state of your application and in cooperation with Cloud Controller (who knows the desired) it ensures that the health of the applications are as expected. Bulletin Board System (BBS) monitors the real time state of your applications and periodically compares it with the desired state (received via nsync). If for some reason one of the instances of your application crashes, Diego Brain discards it and fires up another container with your application automatically. The containers are a made of the operating system, information about the system environment and the application droplet (will be explained in next section). When about to create new instance of an application, Cell Rep performs auction on which cell to host the container. Diego opts for deploying instances of the same applications across different cells to increase resiliency. If enabled also across multiple availability zones. Always on, no need for late night calls to Ops.

Cloud Foundry components. source 

Chances are that your applications need runtime dependencies to execute such as JRE. When staging your applications, external dependencies are added into them, so your application can work. These runtime dependencies are called Buildpacks. There are buildpacks for different kind of languages and different buildpacks for each language. When pushing the application it is possible to define the buildpack you want to use, else Cloud Foundry will try to identify the appropriate buildpack. The application along with the buildpack generate a droplet which is a binary executable that runs inside a container in a Diego cell. The droplet is also stored in the Blob Store.

Applications generate logs and so are the components of Cloud Foundry (e.g., Cloud Controller, Diego, Router). A lot of metrics are generated through the applications lifecycle which are crucial for operations. Loggregator, as the name suggests, is in charge of aggregating all the logs and metrics from your applications (as long as you direct them to standard out and standard error). The logs are made available in real time via CLI (temporarily) but they can also be streamed onto external tools that persist, index and search them such as Splunk. It is important to note that logs can be lost in the Loggregator and this is by design. Logs can be dropped if in danger of the Loggregator becoming bottleneck.

Most applications have common needs, such as persisting data. Such needs can be covered in CF with the use of services. Services can be databases, message buses or specific applications you built in order to do something needed in your system only. Services should be considered as resources which can be bound to applications. Cloud Foundry maintains a list of readily available services which can be found in its marketplace. These services are called managed services. When you want to make a service available in the marketplace, you need to implement the Service Broker API. All services are accessible through this standardized interface. When an application requests to be bound to a service, the service broker is responsible for creating the service instance and passing the credentials to the application as environment variables. 

A special kind of services are the so called route services. These are services that are bound to routes and not applications. Their main usage is for cross cutting concerns, for example checking certain headers and perform some validations before the call reaches applications. We could create an application that does that, create a route service out of it, and place it to the routes of applications we select. Then when a request is received by the router, if the route is bound to a route service, first goes through the route service and then back to the router and eventually to the app. 

One can guess that there are a lot of things going on in terms of security. A detailed description is out of the scope, but we should briefly mention the highlights. Cloud Foundry is a platform, used by many tenants so there is a need for authentication and authorization of users. This way, there can be a segregation between orgs (a logical separation of tenants in a CF instance) and spaces (a logical division of parts of orgs). Users can only perform actions permitted by their assigned role. Cloud Foundry’ s User Account and Authentication (UAA) takes care of that. It is an OAuth2 provider for the platform but UAA can be used as a service itself by your applications. Besides that, there are other reasons that enhance your security when running in CF. Your applications are behind the Load Balancer and the Router, and that decreases the network surface exposed for potential attacks. Also, the easiness of “throwing away” and replacing the containers in which your apps run, makes applying security patches simple and fast. Containers are isolated from each other and all traffic to your application is encrypted. Standardized buildpacks can put you at ease about what your application is running on. 

It should be clear that Cloud Foundry is a strong enabler for developers. It takes care of operational load and makes sure that our systems just work. It allows developers to focus on what matters most: the applications or better, the business proposition to customers.