Iterating Our Product. It’s as Easy as A/B/C Testing

Building an app for millions of people brings a whole host of different challenges. The most exciting challenge is that no two users are the same. 

Because no two users are the same, it makes sense for us to analyse our users’ interactions with every piece of design that gets released. Capturing user journey data allows us to see the real impact of what we build. 

We’re not just talking about the impacts for our business. We’re talking about improving our users’ financial health. We want to put all of our users into a better place with their finances– that’s why we come to work every day. 

Oh, did I mention? At Cleo we love data. And good vibes. But also, data.

So, we A/B test almost every change we make to Cleo products. 

‍

What is A/B Testing?

‍

When we have a new feature that we think our users will love, we start by making a small but material change to Cleo, then roll it out to a subsection of users to see how the change affects them. 

If the change is overwhelmingly positive, and we see that our users are in fact loving the feature, we’ll increase access to all users. Then we then begin building on it, to create a first class user experience.

Testing doesn’t stop there. When a feature is more complicated or has multiple possible impacts, then we’ll not just roll out an A/B test. We sometimes have C, D and E tests. At this point we’re basically making alphabet soup.

Admittedly, the possibilities are endless, but we do try to keep our tests focussed to keep the learnings from data valuable. 

The upside to running many tests is that results constantly pour in from tests across every aspect of our business, meaning Cleo is always improving. 

‍

What Does This Look Like Day-to-Day?

‍

My squad is focussed on delivering great value to our customers through one of our most popular products, the Cash Advance. With Cash Advance, we are committed to helping our users out when they really need financial support the most, without trashing their credit rating 🙌

Recently we were able to A/B test a new feature which helped our users when it came to managing their cards and bank accounts. To our users, this feature meant they saw a single ‘Add Card’ button rather than a multiple card or account options screen. 

We wanted to know if this new process was making it easier for our users to add a card or account. The results did show a great impact for users, by consistently achieving an increase in card validity.

‍Why does card validity matter? When our users have a valid card registered with Cleo, we can continue supporting them with their Cash Advance when they need it. A valid card or account is essential for users to receive an advance.

The new flow saw an increase in the number of successful card and account linkings for our users. We also saw an increase in subscribers staying on the platform and continuing to gain the vital financial support that Cleo offers. Great results all around.

With the positive results coming in so quickly, we were able to rapidly ramp up access to this feature to all of our users. 

Today, we continue to analyse where we can add more value to our users through more small iterations which are then A/B, and occasionally C and beyond, tested.

‍

‍

Doing Good Feels Good

‍

Our commitment to data driven product engineering is one of the reasons I love working at Cleo. 

‍

Every month you are developing product features which are actually being designed by our users and how they use the app. As a squad, we are delivering valuable change every day and making an impact to millions of users. It’s a great way to feel when you log-on and log-off each day.

If you want to know more about how we build our product at Cleo then head over to our building cleo blog.