On Thursday October 6th we were lucky to be able to host our most recent Tech event at Dojo’s incredible (and award winning) office space in Paddington, joined by 5 speakers from a new age of data team structures - Analytics Engineering.
With 360° views of London, a rooftop garden, a fully stocked bar as well as their very own barista, Dojo showcased their jaw dropping working environment that doubled as our event space for the evening. For someone coming from a teaching background and is used to a few windows in a cold room with a broken kettle, it really was the height of luxury. As well as the impressive space, Dojo also showed off their vibrant culture and impressive work within the Analytics Engineering space that evening.
Dojo’s two esteemed Analytics Engineering Leads - George Apps & Emanuela Ciotti - kicked off the night and set the standards for the rest of the talks. Dojo have been on an Analytics Engineering journey where not only the size of their Analytics Engineering & Product Analytics teams grew but they also experienced a significant growth in their user base. This led to an unmanagebale quantity of data, or as it was termed, too many chefs spoil broth. Something had to change.
It’s probably a common occurrence for companies (especially pre-growth) to think their Data is in order, always scalable, unproblematic and so on – well George & Emanuela were here tell us that is not the case. They explained how a single depository is quite simply not enough for a growing data driven organisation.
Down with the Monorepo – the title of their talk – explained how to combine multiple DBT depositories together to build a global lineage service and help with a Data Mesh style approach.
An Example of a problem that arose was the overwriting of each other’s code - and judging by the amount of sighs, smirks & side eyes in the room when Emanuela announced it, this is not an uncommon problem. They also said there were issues with ownership and a lack of accountability in this. DAGs was another issue – as the scale of their DAGs increased, there were orchestration issues meaning tasks and tickets were being assigned incorrectly and as a result, jobs which were indispensable to the maintenance of accurate data, were simply not being done!
So naturally, being a forward-thinking business who puts data at the heart of what they do, they had to change, but what did they change to?
They underwent a data transformation – one that took 9 months to get to where they are today! They decentralised their data using DBT and created an in-house solution fully designed to solve their own specific business needs - this can be described as a Data Mesh approach. A data mesh is the creation of a specific infrastructure & architecture to suit your needs.
The difference between the old & the new:
- They went from One Repository to having many repositories, which enabled them to work more independently, faster and have a wider range of access to handle the increase in Data.
- Tenuous Ownership was switched to Clear Domain Ownership, making people accountable for their own specific areas.
- Concise DAGs – Orchestration in Airflow allowed for a solution regarding tickets & jobs to be done
- Mindset of Data as a Product (Rosie’s talk later went into the importance of this point)
Now, I think it is fair to say that Dojo’s talk was the most technical of the evening and I do strongly advise clicking the link below to check out George’s and Emanuela’s talk. It goes into detail on the inner workings of how and why they made changes to their data team.
For the lay person however, it was fascinating to begin to understand the relationships between the repo’s and how even in a business as large and impactful as Dojo, you will encounter unmanageable situations that need to be tackled as a team (and it's safe to say from the 'wooing' culture in the room that night, they are one incredibly supportive team!).
We are so proud to partner with Dojo, not just as a recruitment provider but also on events like this where we can really help educate the wider community and ourselves so our thanks must go out to them again as a company and of course, the speakers George & Emanuela as well as Dave Simpson who worked tirelessly alongside us to make sure this event happened as smoothly & successfully as it did!
Please connect with me below for career advice, who’s hiring at the moment as well as invaluable market insights. 👇
