Lead Data Engineer

Company Description

When you're with us, you're running with the best. We've spent decades assisting marketers from the world's leading brands in personalizing experiences for millions of individuals with our state-of-the-art technology, solutions, and services. Epsilon's top-tier identity provides brands with a transparent, privacy-protected view of their consumers, which can be utilized across our suite of digital media, messaging, and loyalty solutions. We process over 400 billion consumer interactions every day and own numerous proprietary technology patents, including real-time modeling languages and advancements in consumer privacy. Thanks to the dedication of every employee, Epsilon India is now a Great Place to Work-Certified™. Epsilon has also been consistently recognized as an industry leader by Forrester, Adweek, and MRC. As a core element of Publicis Groupe, Epsilon is a global company with more than 8,000 employees worldwide. For additional information, visit epsilon.com/apac or our LinkedIn page.

Job Description

Organization Objective/Purpose:

This position is in the Engineering team under the Digital Experience organization. We orchestrate the initial stages of the customer experience by personalizing offers and content. We are currently seeking a highly motivated, intelligent software engineer.

You will join a team focused on building solutions, pipelines using the latest software engineering design principles, and tech stacks. You will also be expected to identify, design, and implement improvements, including redesigning infrastructure for better scalability, optimizing data delivery, and automating continuous integration and deployment processes/pipelines.

You're also expected to collaborate with various stakeholders, apply scientific rigor to design and create high-quality software.

You must possess excellent verbal and written communication skills and be comfortable working within a larger company's entrepreneurial, startup-like environment.

Brief Description of Role:

Develop solutions for Epsilon that will deliver high-quality personalized recommendations across different channels to our customers.

Work with the Data Science team to ensure seamless integration and support of machine learning models.

Build the infrastructure necessary for optimal extraction, transformation, and loading of data from a vast range of data sources using SQL and AWS big data technologies.

Develop end-to-end (Data/Dev/MLOps) pipelines based on a deep understanding of cloud platforms, AI/ML lifecycle, and business problems to ensure solutions are delivered efficiently and sustainably.

Collaborate with other team members to ensure the high quality of deliverables.

Learn and implement the latest design patterns in software engineering.

Qualifications

Data Management

• Experience with both structured and unstructured data, Hadoop, Apache Spark, or similar technologies.
• Good understanding of Data Modeling, Data Warehouse, Data Catalog concepts and tools.
• Experience with Data Lake architectures, and combining structured and unstructured data into unified representations.
• Ability to identify, join, explore, and examine data from multiple disparate sources and formats.
• Ability to distill large quantities of unstructured or formless data and put it in a form that can be analyzed.
• Ability to address data imperfections such as missing values, outliers, inconsistent formatting, etc.
• Ability to manipulate large datasets (millions of rows, thousands of variables).

Software Development

• Ability to write code in programming languages such as Python/NodeJs, PySpark, and Shell script on Linux.
• Familiarity with software development methodology such as Agile/Scrum.
• Inclination to learn new technologies, stay updated with the latest technologies within the cloud architecture, and drive your organization to adapt emerging best practices.

Architecture and Infrastructure

• Architectural design experience on AWS.
• Architectural design for applications with high transactional volume.
• Experience in delivering software with AWS EC2, S3, EMR/Glue, Lambda, Data Pipeline, CloudFormation, Redshift etc.
• Good knowledge of working in UNIX/LINUX systems
• Experience in designing and building large scale enterprise systems

Qualifications

• Bachelor’s Degree in Engineering and related field with 10+ years of similar experience
• Tech Stack: Python or NodeJS, PySpark, Microservices, Docker, Serverless Frameworks & Databricks.
• Hands-on experience building ETL workflows/data pipelines
• Experience in relational and non-relational databases and SQL (NoSQL is a plus).
• Experience with Cloud technologies (AWS or Azure)
• Experience in Designing and building APIs for high transactional volume
• Experience building Data and CI/CD/MLOps pipelines
• Familiarity with Airflow and MLFlow tools
• Familiarity with automated unit/integration test frameworks
• Experience working on AdTech or MarTech technologies is an added advantage
• Knowledge of machine learning algorithms and concepts and implementation will be a plus
• Good written and spoken communication skills, team player.
• Strong analytic thought process and ability to interpret findings

The candidate should possess strong business acumen, interpersonal and communication skills, and the ability to work independently in addition. He/she should be able to communicate findings and explain how techniques work in a manner understood by all stakeholders, both technical and non-technical.