CI/CD at Cleo

For a lot of engineers, continuous integration and continuous delivery are extremely helpful but slightly mysterious things which make developer life much easier. You press a button and the CI/CD elves take care of everything for you in the background, and other than the initial finger that you had to lift to click “merge” for your code there is no more effort required. Deploys are straightforward, features are unleashed and life is good.

Cleo’s CI/CD process is streamlined, fast and creates a lot of developer happiness, so we figured we would like to lift the hood on it so we can help spread the love.

‍

What is continuous integration and continuous delivery?

As a software engineer who has tinkered around the edges of devOps here and there, my understanding of CI/CD has always been quite top level. Which is why I was keen to research more about Cleo’s deployment pipeline. As I discovered, at Cleo it’s a pretty magical process whereby code is automatically built, tested and then deployed to our staging and production environments with one click of a button.

We have a couple of important aims for our continuous integration and delivery process, namely to:

• improve developer happiness and productivity
• find and squash bugs more easily
• easily and quickly get features and updates to our users

‍

TL;DR - What CI/CD looks like at Cleo

While CI/CD is a huge area that has many different services to choose from and ways to implement, at Cleo (in line with our engineering principles) we keep things simple and use three tools for our deployment pipeline:

• GitHub
• CircleCI
• Heroku

In a nutshell, when we merge pull requests on Github, we have a seamless integration with CircleCI which automatically kicks off our build and testing processes, before triggering an automatic deployment to Heroku if everything is peachy.

We also have CI/CD set up for PR checks, so if you’re pushing code for review you can be reasonably confident that your code isn’t going to break tests or collapse production.

Our CI/CD is really clear, fast and efficient, so let’s dive into our pipeline a bit more 👀

‍

More about CircleCI

CircleCI is a popular devOps tool that manages continuous integration and delivery. We have integrated it with GitHub, so engineers can automatically kick start jobs when they push code. On average, our build and deploy pipeline takes around eight minutes, which is very speedy for a large monolithic Rails app.

We have our CircleCI pipeline configured so that there are two main parts in the process, called deploy and release. The deploy step instructs Heroku to build the application, while other checks run in parallel such as linters, unit tests for Ruby and TypeScript, integration tests and any other tests we have configured. While our app build happens in Heroku, all these parallel steps run independently in containers on the CircleCI platform. If all checks pass and the app build is successful, the pipeline will automatically run the release step which deploys our app on Heroku.

An interesting thing about our CircleCI configuration is that the app build runs independently of our other steps, which makes it fast. And because all of our checks are run in parallel rather than sequentially, this also really speeds up the pipeline. If one step fails then the entire process will fail, and we will never run the release.

‍

How we use Heroku

Heroku lives at the end of our deployment process, but it’s obviously a vital element because it’s where our apps live. This platform as a service is popular with start ups because it takes a lot of configuration pain away, leaving engineers with a simple interface to work with.

Our production app is built with Rails; Heroku understands how to build and manage Rails apps out of the box, which is great when you want to get going quickly.  Heroku also has a bunch of nice and accessible features like no downtime deployments, multiple dynos (which is the Heroku term for a container), autoscaling and integrations with third party tools, all with very easy configuration.

Saying that, we do have a few custom elements with our setup on this service that relates to how we configure CircleCI so it’s not quite as simple as it seems at a first glance.

When CircleCI gives Heroku the thumbs up, it will go ahead and initiate a new build and deployment of the production app. We actually use two Heroku applications to do this - one is purely for the build, and the other is to deploy and run the first built container.

This pattern is a bit unusual - often you would see a build and deploy happening in one container - but we do it for a couple of reasons. Firstly, it is not possible to tell Heroku to only build the app, but not deploy it.  This approach allows us to build the app in parallel with tests, but skip releasing it to production. Secondly, we use a fair number of buildpacks in our monolithic application, which slows down build and deploy time. So to make things faster, we have this custom way of deploying.

The other interesting thing to learn about our CI/CD and Heroku configuration is that at Cleo we don’t have an intermediate step before code makes its way into the real world - we deploy to staging and production environments in parallel. This makes the deployment pipeline super speedy, but because we understand that mistakes can happen, it’s easy for us to follow up if something does accidentally get broken.

Often we can tell something’s up very quickly due to the alerting and monitoring systems we have in place, but hey, sometimes users tell us too. It’s straightforward for us to revert a problematic PR, which triggers a new build process. This allows us to avoid freezing deployments whilst we’re working on a fix.  As well as this option, we can also roll back deploys in Heroku so the app will immediately jump back to the previous release. Either way, users shouldn’t experience any major issues while we fix whatever has fallen over.

‍

How do we look after our CI/CD?

When Cleo was just getting started we didn’t have any devOps engineers, but as we’ve grown we now have a five-strong team of devOps engineers in our Platform Engineering squad. They manage and maintain our deployment pipeline, as well as a very long list of other important things related to our engineering infrastructure.

At Cleo we are keen for our engineers to get stuck in and learn about different areas of engineering to help their career growth and development. You don’t need to have a devOps background - if you are a product engineer you’re welcome to learn more about Cleo’s infrastructure, and through opportunities like pairing, on-call support, talks and knowledge-sharing (or even writing blog posts!), there is no shortage of ways to get involved with CI/CD or other areas of devOps in the business.

‍

To wrap up, here are the key takeaways about Cleo’s CI/CD:

• We use a custom pattern for CircleCI and Heroku with separate build and deployment jobs to help speed things up. We want fast, automated deploys for engineer and user happiness and our production pipeline typically takes less than eight minutes.
• We use Heroku because it has a low barrier to entry and is easily configured.
• We don’t have a separate staging step in our deployment, because we find that pushing straight to production is quick and easy to implement, and if we have any problems then we can deploy fixes quickly or roll forward. We have robust alerting and monitoring that helps us spot problems, which we will definitely talk about in another blog post…
• We have CI/CD checks for all our PR pushes and when we merge into main, so you can feel reasonably confident about pushing working production code
• We have a five-strong devOps team of engineers, but everyone is welcome to get involved with this area and we encourage engineers to learn about our infrastructure via pairing, knowledge sharing and talks.