Data Engineer (Databricks)

Job Description:

As an Azure Data Engineer, you will have a crucial role in the design, development, and maintenance of data solutions on the Azure platform. Your main duties will involve constructing and optimizing data pipelines, ensuring data quality, and implementing data processing and transformation logic. To succeed in this role, you need expertise in Azure Databricks, Python, SQL, Azure Data Factory (ADF), and /Scala. Your work will enable data-driven decision-making and insights for our organization.

Key Responsibilities:

1. Designing and Developing Data Pipelines:

- Design and execute scalable and efficient data pipelines using Azure Databricks, PySpark, and Scala.

- Oversee data ingestion, data transformation, and data loading processes.

2. Data Modeling and Database Design:

- Formulate and execute data models to facilitate efficient data storage, retrieval, and analysis.

- Work together with relational databases, data lakes, and other storage solutions on the Azure platform.

3. Data Integration and Orchestration:

- Utilize Azure Data Factory (ADF) for orchestrating data integration workflows.

- Handle data movement across various data sources and goals, including scheduling and monitoring data pipelines.

4. Data Quality and Governance:

- Implement data quality checks, validation rules, and data governance processes.

- Ensure data accuracy, consistency, and adherence to relevant regulations and norms.

5. Performance Optimization:

- Optimize data pipelines and queries to increase system performance and minimize processing time.

- Adjust SQL queries, optimize data transformation logic, and use caching techniques when required.

6. Monitoring and Troubleshooting:

- Continuously observe data pipelines and pinpoint performance issues.

- Actively troubleshoot issues related to data ingestion, processing, and transformation.

- Cooperate with cross-functional teams to resolve data-related problems.

7. Documentation and Collaboration:

- Document data pipelines, data flows, and data transformation processes.

- Collaborate closely with data scientists, analysts, and other stakeholders to understand their data needs and provide data engineering support.

Requirements

5+ years of experience in a Data Engineer role.

- Proficiency in Python or Scala programming.

- Exceptional problem-solving skills in a big data environment.

- Experience in building 'big data' data pipelines, architectures, and data sets from various business applications.

- Familiarity with message queuing, stream processing, and scalable 'big data' data stores.

- Enterprising spirit, commitment to delivering high-quality outcomes.

- Degree in Computer Science or a related field, such as Applied Mathematics.

- Experience in a technical customer-facing role (preferably a solutions engineer) or willingness to undertake such role.

- Profound expertise in Azure Databricks, Python, SQL, ADF, PySpark, and Scala.

- Proficiency in designing and developing data pipelines and processes.

- Solid understanding of data modeling concepts and database design principles.

- Knowledge of data integration and orchestration using Azure Data Factory.