On Tuesday 11th June 2019, Burns Sheehan partnered with Skyscanner to host our latest tech event, Software Engineering @ Scale. At their amazing London offices just off Tottenham Court Road, we welcomed a full house of 70 Technologists into their purpose-built amphitheatre to listen to a couple of talks from two of our clients – Skyscanner and Zopa.
Prior to organising the event we had gone out to our community to ask what they would be interested in hearing and we had a wide range of topics and technologies, but a consistent theme was on how to solve problems around performance and scalability of platforms. Therefore, we spoke with our network to find two speakers who could share their experiences of ‘scale’ from both a software engineering and a DevOps perspective – Petya Afanasiev, Engineering Manager at Skyscanner and Ehsan Ashouri, Senior SRE at Zopa.
Petya had been on the Skyscanner stand at the Lead Developer Conference earlier in the day and he was first to speak on Big Data Analytics on a Consumer-Facing Application; a journey through multiple re-architectures of a large Skyscanner product to keep up with the ever-increasing load.
Skyscanner are the world’s leading travel search site with around 80 million monthly users and 2000 partners to integrate with so they have significant challenges of scale! Petya gave a very interactive talk about how they deal with 100,000s of price updates per second, 1000s requests per second from travellers all deployed to a globally unified dataset. He discussed 4 iterations of architecture used on the platform starting with a SQL Database and asking the audience if they thought the different designs worked or not. It was a hugely enjoyable discussion with plenty of ideas and questions from the audience and surprises that certain things worked or didn’t. The SQL design did work initially but would struggle to scale at the rate they were growing; iteration 3 with heavy use of AWS S3 buckets and Spark was a partial success but with deduplication and aggregation of the differentials taking too long they tweaked this in iteration 4 to reduce the time and this is how they are currently running the platform… until the next iteration!
Next up was Ehsan from Zopa to discuss Zopa @ Scale - An overview of how Zopa delivers its infrastructure services. Zopa has been operating for nearly 15 years having started as a P2P Lender and they are now a bank who lent out £1bn in 2018 so their needs have significantly changed!
Ehsan explained how they have evolved from a Divergent/Convergent model building out their infrastructure manually and using config management to a Congruent (or immutable) infrastructure which reduces the amount of “toil” their engineers must contend with. This shift has meant it is easier to solve problems of scaling whilst also reducing the cost associated with the Divergent/Convergent model. This process has it challenges including the significant time it takes to implement them and persistent data; however, the trade-off is a far less time spent operating the systems and more time available to build and provision new systems. It was a very informative presentation of how Zopa are overcoming their challenges around scale now they are a digital bank.
This event was a huge success and I would like to thank @Peter Afanasiev and @Ehsan Ashouri for sharing their experiences as well as @Ed Ross and @Fleur Hoey for their help in setting up and running this meet up. We look forward to seeing you at the next one!
