5 May 21 Data
Here in the Burns Sheehan Data Team, a huge part of our role is understanding not only the candidates and clients that we work with, but also the latest trends in the wonderful world of data.
At this point, it’s fair to say that almost all industries have borne their part of the recent Covid-19 – related economic downturn, which was reflected in a decrease in hiring and job postings across the board, especially in the middle of 2020. We, like every recruitment company, felt the weight of this as many of our clients chose to freeze or postpone future hires in the face of an uncertain economic future. However, it’s no secret that the technology industry has proven to be relatively resilient to the many lockdowns and containment measures which have affected the retail and hospitality sectors so severely. This is perhaps due to the fact that technology forms such a key part of almost every other industry nowadays, or maybe because of the relative ease with which technology workers were able to pivot to full-time working from home.
Many of our clients told us that through their need to be able to effectively monitor their own business health during the crisis, Data still formed a crucial part of their normal business operations, and in many cases these key Data hires were actually prioritised. As a result, the Data market has shown quite remarkable resilience throughout the crisis, once again proving its status as a strong growth industry in its own right, with relatively stable and certainly promising long-term job prospects.
Now, due to the success of vaccine programmes and falling infection rates, technology hiring is seeing a boom as companies begin to reassess their staffing needs, faced with stronger short and mid-term economic prospects. And surely enough, within Data we are seeing a huge resurgence in roles across the whole Data Engineering, Science and Analytics spectrum.
Takeaway: if you’re somebody who is thinking about a change, but you were understandably nervous due to Covid-19 uncertainty – now really is the time to get back on the market.
In terms of the larger data landscape (both pre and post-Covid), one trend we’ve certainly noticed is the rise of the DataOps Engineer. Traditionally, Data Engineering was really about ‘connecting the dots’ in a business’ end-to-end data flow; consolidating data streams from various sources, and ensuring that data quality and transaction standards were met.
But within the finer details of any given role, Data Engineering has now become a byword for so many different processes and functions that the job title itself has ceased to have individual meaning. This term can now imply experience with a whole host of technologies including Python, Scala, Java, NoSQL databases, Kafka, Spark, Data Warehouses and ETL or scheduling tools such as SSIS, AWS Glue or Airflow. On top of all of these, most companies are now operating on either hybrid or fully cloud-hosted systems, meaning Data Engineers need to have a good grasp of these microservice-based systems, usually on AWS, Azure or GCP.
However, in adding this last element to the list of technology needs for a Data Engineer, the industry has unintentionally put pressure on the boundary between those who work in and around these cloud platforms, and those who actually provide and administer them. DevOps Engineers or those with more of a DevOps background, and others who work on Infrastructure-As-Code to maintain these cloud platforms have found themselves increasingly encroached upon by Data Engineering. As companies start hiring for ‘DataOps Engineers’ and/or ‘Cloud Data Engineers’, they will be looking for the traditional ETL and dataflow focus of a Data Engineer, but with the Infrastructure and DevOps knowledge to simultaneously leverage and exploit the benefits of a cloud microservices platform.
We have witnessed this mostly in our conversations with clients, who in many cases are now looking to hire one person across Data and DevOps, rather than two separate specialists. This is sometimes highlighted by an ‘ideal’ Data Engineering candidate having knowledge of more traditionally Devops technologies such as Terraform, Kubernetes, Docker, Jenkins or Ansible.
This isn’t to say that ‘traditional’ Data Engineering roles don’t exist – far from it – but the increase in crossover roles like the above have been quite noticeable, particularly throughout early 2020 and as the market starts to open up again. These DevOps technologies aren’t yet quite essential to a modern Data Engineer, but we think the need for knowledge in these areas will only increase over time, so it’s a great time to start learning and investing time in these!
In many ways, the appearance of Analytics Engineers comes as an antithesis to that of the DataOps roles mentioned above. As Data Engineers become increasingly involved in the highly technical world of DevOps, ETL tools have in many cases become simpler, with the ever-increasing popularity of ETL/ELT-as-a-Service tools such as Fivetran and Stitcher in many smaller companies. These take the usually quite time-consuming business of ETL and data pipelining between various tools and platforms, and automate these on a service basis, removing the need for highly technical Data Engineers who would otherwise build and maintain these systems. However, businesses still have a need for someone who is familiar enough with coding and scripting to oversee and operate these, as well as optimising these processes for the business from a more commercial perspective.
Enter the Analytics Engineer. This is someone who can work with BI, Analytics and Data Science Teams from a technical perspective, ensuring that they have the data they need and more importantly, ensuring that this data is in the right places, the right formats, and managed correctly to be useful to the business. In this respect, this is a hybrid role which encompasses both the ‘harder’ technical skills in SQL coding and data warehousing, as well as the ‘softer’ skills needed in order to effectively liaise with stakeholders across multiple business areas. It’s a great role for ‘jack-of-all-trades’ within the world of data, especially those who love to work across many different areas of a business, and is something that we have noticed to be much more in demand recently from some of our clients and within the broader data market.
As new technology continues to be released and becomes more advanced, we think we’re going to continue to see roles within data evolving and responsibilities changing. The demand for analysts, engineers, and scientists is at an all time high and that’ll only increase.