Senior Data Engineer

CloudFactory is looking for a highly skilled and experienced Senior Data Engineer to join our dynamic team. As a global leader in marrying people and technology to offer a cloud workforce solution, CloudFactory carries out numerous machine learning and core business data processing projects. We are committed to creating economic and leadership opportunities for gifted individuals in developing countries. As a Senior Data Engineer at CloudFactory, you have a critical role in developing and maintaining our data infrastructure to support our growing operations. You will get the chance to work with groundbreaking technologies and work with a diverse and zealous team to increase our data processing capabilities. If you are enthusiastic about making a difference in the world through impactful work, enjoy building relationships, and thrive as part of a cohesive team, CloudFactory is for you. Responsibilities: The successful applicant will convert user requirements into actionable insights by harnessing existing tools and technologies. Your duties will include: -Regular contact with PMs/stakeholders to understand their needs. -Investigation to verify if the necessities can be fulfilled using existing technology and data resources. -Data extraction using Python and other tools like Fivetran. -Data modelling using SQL and the concept of Star and Snowflake schema. -Helping in cost and performance optimization choices within the data warehouse. -Communicating reasoning and logic behind opinions to aid in making optimal decisions. -Writing data quality and other metadata collection systems within data pipelines to promote the observability and recoverability of pipelines. -Writing unit and integration tests to enhance system reliability. -Mentoring and guidance. Tech Stack: Our tech stack includes: -Data extraction and orchestration tools - Fivetran, Prefect -Data modelling tool - DBT -Data Warehouse - Snowflake -Data quality checks - DBT, Elementary -Data visualisation - Tableau and Quicksight -Data catalogue and lineage - Atlan -Source code management - Github -Infrastructure as Code - Terraform -Programming Language - Python -AWS services (ECS, S3) Interview Process: Candidates will be asked to complete an initial coding and analytical thinking test, followed by an interview with a Senior Product Manager where they will discuss previous projects, including approaches taken and results accomplished. Successful candidates will then meet with the Engineering Director before a final interview with the CTO. The process is expected to take 1-2 weeks. Requirements: The ideal candidate should have: -Minimum 4 years of experience as a Data Engineer. -Expertise in SQL in the context of building and managing an enterprise data warehouse. -Expertise in data orchestration and related tools in the context of transporting data. -Competence with OLAP systems and related use cases. -Competence in extracting data from source systems using Python. -Deep comprehension of and competence in using MPP architectures. -Competence in modelling data using Kimball, Inmon or Data Vault. -Competence in writing unit tests and integration tests. -Understanding of microservices and event-driven systems. -Excellent communication skills, able to turn complex data into understandable insights and narratives.