Iterating Our Product. It’s as Easy as A/B/C Testing
It’s just a button. Or is it?
This is some text inside of a div block with Cleo CTA
CTASigning up takes 2 minutes. Scan this QR code to send the app to your phone.
We like to think we treat Data Scientists differently. The biggest difference is that they sit within cross-functional squads. They’re an integral part of each squad’s mission, working alongside engineers, designers, product analysts, copywriters etc. Every squad is different but they’re all led by one of our intrepid Product Managers.
It’s a well-established set up for Developers, but not always the case for Data Science. But in practice, we’ve found this way of working gives us significant influence on scoping jobs and increasing the impact we have on our audiences.
We’re a relatively young company, growing and evolving but still focused on one main outcome: improving the world’s financial health. At a squad level though, we’re given the autonomy to define our OKRs before executing them.This level of ownership right from the start makes all the difference for me as a Data Scientist.
As a squad, we also assess our current challenges, what data and tracking we have, what our modelling capabilities are (and could be!) then map out how to get there.
We see our users’ problems, get a full picture from front and back end, then have meaningful discussions about where we can balance the use of modelling, user value, and user experience.
We own our modelling all the way from initial idea and experimentation to deployment (using tools like Sagemaker and Metaflow). Being engaged with everyday discussions around the usage of our models; it’s way better than being secluded in our own walled garden.
One of our three company values is ‘iterate with data’ - so our output and impact is valued here. I’ve also appreciated the opportunities to learn from colleagues in different disciplines through close collaboration, too.
I recently spoke with a Product Manager who had left a startup using the more traditional, “data science only” squad approach. She immediately noticed the advantages of an integrated model and having my colleagues and I involved from the very start.
We’re able to help frame the problem, seeing what really needs to be solved before modelling begins. This cross-pollination of ideas means we’re more likely to end up with something that really answers our users’ needs rather than creating a high performing model on paper with a narrow framing.
Similarly, a team member formerly employed by a big investment firm highlighted that a separate data science team can become a support function - and one that has to constantly demonstrate value to internal stakeholders rather than delivering what’s most impactful for the customer. This is sidestepped by being integrated in squads and aligned to the overall mission.
Another joiner from the tech industry said that a separate DS section, “worked well for blue sky thinking, but a lot of the code was exploration – aka big experiments with mainly offline evaluation.” At Cleo we strive to be full stack data scientists and want to avoid falling into the trap of over engineered experiments and delayed deployment.
Every approach has its tradeoffs, and there’s plenty of issues we have to be vigilant about. Being integrated in squads can lead to Data Scientists sharing less knowledge. To combat this, we operate a separate chapter, where Data Scientists and Product Analysts get together regularly to knowledge share, discuss broader issues or what they’re currently working on.
Another challenge is fitting DS work – which can start from an open-ended investigation or take unexpected turns – into agile sprints. Our tickets aren’t always as straightforward as an engineer’s both in terms of sizing and the general flow during a weekly sprint.
It’s an imperfect science, and you can tell by the variety of implementations we’re trying across the business. Do we assign points to the tickets? Should we have a separate board or ceremonies?
But as the cliche goes, the frameworks exist to serve us, we don’t exist to serve the framework. (Sorry – had to slip it in somewhere.)
So we make these compromises with our eyes open and weighed against the benefits we see elsewhere.
No approach is a silver bullet, and the solution that works for us is highly context dependent. But we’re at an exciting stage for growth and product development. Working this way gives us the best chance to have an outsized impact.
Overall, being a Data Scientist at Cleo means keeping a holistic view of the problem space, our mission, and how our solutions can help make a difference for our users through the product.
This keeps the job varied, yet focused, and ultimately compelling.
Like our approach to data science? Have a look at our careers page to see our open roles. Author: Laurence Mandal
It’s just a button. Or is it?
Getting Cleo’s testing game tight.
Last year, our Heroku Postgres database was fast approaching its 3TB storage limit, the maximum size that Heroku supports at time of writing. To make sure our community of 4 million people could continue to work on their financial health with Cleo, it was essential to migrate away from Heroku as quickly and smoothly as possible.