Senior Big Data Engineer

#ouroneteam

At Kaizen, our dedication to the unparalleled worth of data guides all our decisions. Proceeding from our robust Lake House architecture, we foresee a revolutionary move towards a Data Mesh architecture. This vision centers on considering data as a product with clear cut contracts, distributed ownership, and domain-focused design.

In order to further enrich our data platform, we're particularly interested in data democratization, governance, and observability. The ideal situation? Every piece of data being readily available, monitored, and controlled.

Our Data Platform team is suitably equipped to spearhead this shift, by leading the development and augmentation of the next generation Data Products. By becoming a member of our team, you'll be part of an effort that places Kaizen at the forefront of data-driven creativity.

#ourmission

Our Lake House infrastructure has been a cardinal feature, effortlessly empowering a diverse group from BI, Data Science and ML to our inventive product teams. As we navigate towards the upcoming phase characterized by Data Products and insights from a Data Mesh model, we are expecting transformative results. Central to our mission, we strive to ensure the faithfulness of our pipelines, the agility of our infrastructure and the precision of our designs, at the same time maintaining the highest standards of data governance.

In the course of this ambitious path, we're seeking data engineers adept in data ops, with a sharp sense for software engineering optimal practices and a product-oriented mentality. A key task of our team is to architect and create utility tools that remove complexity. In so doing, we provide other teams with plug-and-play solutions, sparing them from the detailed minutiae and reducing cognitive burden.

Further cementing our commitment is Kaizen's renowned expertise in distributed computing. This knowledge base is poised to be utilized, displaying our unyielding devotion to fostering a data-centric culture throughout the corporation.

#ourTechStack

Spark, Scala, Python, Azure Databricks, Delta, Apache Airflow, Azure DevOps, Terraform Grafana and more

Responsibilities:

• Contribute to the creation of a scalable and secure Data Platform in line with the Data Mesh paradigm;
• Create, implement and enhance tools and services for orchestration, observability, data governance and data quality to high engineering standards
• Implement and manage our products applying CI/CD best practices within the ‘you build it, you run it’ culture
• Stay updated on emerging technologies, methodologies and industry best practices to facilitate constant enhancement of the data platform
• Promote a culture of documentation, where the technical documentation is kept up-to-date with our products and ensures their ease of use and maintenance

Requirements:

• Spark Proficiency: Proven knowledge in Apache Spark, being able to write and optimize effective code for data processing activities.
• Software Engineering: Robust software engineering background, skilled in developing, writing and implementing high-standard code that synchronizes with our data-first vision.
• Data Pipeline Expertise: Hands-on experience with data pipeline orchestration frameworks, guaranteeing smooth data flow and integration across different sources and platforms.
• DevOps and DataOps Knowledge: Practical experience in DevOps, with a preference for candidates familiar with DataOps practices, for seamless integration, delivery, and effective data operations.
• End-to-End Data Solutions: Proficiency in designing, planning, and executing comprehensive data pipelines, from sourcing and ingestion to processing and serving.
• Technology Evaluation: An inquisitive mindset, with the capacity to understand, evaluate and potentially assimilate new technologies that can propel Kaizen's data-driven approach forward.
• Data Governance and Security: An understanding of the significance of data governance, with an emphasis on ensuring that data assets are protected, accessible, and compliant with regulatory standards.