Data Engineer

#ouroneteam

At Kaizen, our decisions are driven by the invaluable insight we derive from the expansive data at our disposal. Our vision is to transition from the Lake House architecture to a more Data Mesh architecture that emphasizes on treating data as a precise product with well-detailed agreements, decentralized ownership, and domain-focused design.

We're seeking to improve our data platform by focusing on data democratization, governance, and observability, with the goal of having every piece of data perfectly accessible and governed.

Our well-equipped Data Platform team is leading this transition, focusing on the design and expansion of forthcoming Data Products. By joining our team, you'll have an opportunity to be a part of the innovative journey that puts Kaizen at the forefront of data-driven innovation.

#ourmission

Our Lake House infrastructure has been a foundational pillar, seamlessly supporting a diverse cohort from BI, Data Science and ML, to our innovative product squads. As we steer towards the next frontier characterized by Data Products and insights from a Data Mesh paradigm, we anticipate transformative outcomes. At the heart of our mission, we strive to ensure the steadfastness of our pipelines, the agility of our infrastructureand the meticulousness of our design, while upholding the highest standards of data governance.

We're on the hunt for data engineers skilled in data ops, with a keen sense for software engineering best practices and a product-focused mindset. It is the responsibility of our team to design and create utility tools to simplify complexity. By doing so, we provide other teams with ready-to-use solutions, alleviating them from the tiny details and reducing cognitive load.

The expertise of Kaizen in distributed computing further enhances our commitment. This wealth of knowledge is ready to be exploited, showcasing our relentless dedication to championing a data-centric culture throughout the organization.

#ourTechStack

Spark, Scala, Python, Azure Databricks, Delta, Apache Airflow, Azure DevOps, Terraform Grafana and more

Responsibilities:

• Contribute to developing a scalable and safe Data Platform aligned with the Data Mesh paradigm.
• Design, implement and refine tools for orchestration, observability, data governance, and quality to high engineering standards.
• Deploy and manage our products following CI/CD best practices within the ‘you build it, you run it’ culture.
• Maintain up-to-date knowledge of emerging technologies and industry best practices to facilitate the continuous growth of the data platform
• Promote a culture of documentation, thereby ensuring that our products are user-friendly and easy to maintain.

Requirements:

• Spark Proficiency: A solid understanding of Apache Spark, with an ability to write and optimize effective code for data processing tasks.
• Software Engineering: You should have a strong software engineering background and be skilled at designing, writing, and deploying production-grade code that aligns with our data-first vision.
• Data Pipeline Expertise: Hands-on experience with data pipeline orchestration frameworks ensuring smooth data flow and integration across different sources and platforms.
• DevOps and DataOps Familiarity: Practical experience in DevOps, with a preference for candidates familiar with DataOps practices, ensuring continuous integration, delivery, and efficient data operations.
• End-to-End Data Solutions: Proficiency in designing, planning, and delivering comprehensive data pipelines, from sourcing and ingestion to processing and serving.
• Technology Evaluation: A curious mindset, capable of understanding, evaluating and potentially integrating new technologies that can advance Kaizen's data-driven approach.
• Data Governance and Security: Awareness of the importance of data governance, focusing on ensuring data assets are protected, accessible, and compliant with regulatory standards.

Depending on your level of experience, not all of the above are a must to get a successful start in the role. We are looking for engineers skilled in some of our domains and are dedicated to continuous improvement. Rest assured, there will be plenty of opportunities to learn and grow!