Senior Data Engineer - AWS/Scala (Remote/Work From Home)

IQVIA's Digital Enablement Center of Excellence is looking for a Senior Data Engineer who can work remotely from anywhere in Canada. Our vision is to revolutionize customer engagement, bring about next-generation omnichannel storytelling, an integrated measurement suite, and real-time campaign optimization. All these are fuelled through a comprehensive data and platform-agnostic operating system for intelligent healthcare marketing and analytics. We streamline complex workflows, reducing them from days to seconds, and incorporate functionalities, bringing them under one umbrella, thereby allowing brands to focus their time and resources on driving real-world outcomes. As a Senior Data Engineer within the Digital Enablement Center of Excellence, your role will be vital in designing, developing, and maintaining the infrastructure and systems necessary for efficient and effective data processing, storage, and analysis for our AIM, Email and HCN products. Being part of a cross-functional scrum team, you will work closely with other software engineers, data scientists, software testers, and product owners to ensure smooth flow of data across the organization, which, in turn, will facilitate data-driven decision-making and insights. This role calls for a solid understanding of data architecture and programming skills in Spark and Scala, combined with a passion for working with large and complex datasets. Key Responsibilities: 1) Spark Development: Design, build, and maintain scalable and robust data pipelines using Apache Spark. 2) Data Pipeline Development: Design, implement, and optimize data pipelines to extract, transform, and load (ETL) data from various sources into data storage and processing systems. Utilize Spark's data processing abilities to perform complex transformations and aggregations on large datasets. 3) Data Warehouse Management: Develop and manage data warehousing solutions using Spark and Scala to ensure the availability, integrity, and security of data. Design and implement data models, schemas, and indexing strategies to support efficient data retrieval and analytics. 4) Data Integration: Collaborate with cross-functional teams to integrate and consolidate data from multiple sources, including databases, APIs, and external systems. Ensure seamless data integration across different platforms and applications using Spark's connectors and integration capabilities. 5) Data Quality and Governance: Implement data quality checks and validation processes using Spark and Scala to ensure data accuracy, consistency, and reliability. Establish and enforce data governance standards, data security protocols, and data privacy regulations. 6) Performance Optimization: Identify and resolve performance issues in Spark applications, data pipelines, and databases. Optimize Spark jobs by fine-tuning configurations, leveraging caching mechanisms, and applying optimization techniques. 7) Collaboration and Communication: Collaborate with cross-functional teams, including data scientists, and software engineers, to understand their data requirements and provide appropriate Spark and Scala solutions. Clearly communicate complex technical concepts and solutions to both technical and non-technical stakeholders. 8) Emerging Technologies and Trends: Stay informed about the latest advancements in Spark, Scala, big data technologies, cloud platforms, and data management tools. Evaluate and recommend new technologies and approaches that can enhance data engineering capabilities. Qualifications and Skills: 1) Bachelor's or Master's degree in Computer Science, Information Systems, or a related field or equivalent experience. 2) Strong programming skills in Scala and experience with Apache Spark. 3) Proficiency in SQL and experience working with relational databases. 4) Familiarity with big data technologies such as Hadoop, Kafka, or Hive. 5) Experience with data modeling, schema design, and ETL processes. 6) Understanding of data warehousing concepts and experience with data warehouse solutions. 7) Knowledge of cloud platforms such as AWS and experience with cloud-based data services (e.g., S3, EMR, Glue). 8) Strong problem-solving skills and ability to analyze and troubleshoot complex data-related issues. 9) Excellent communication and collaboration skills, with the ability to work effectively in a cross-functional team environment. 10) Attention to detail and a strong commitment to data quality, security, and governance. IQVIA is a leading global provider of advanced analytics, technology solutions and clinical research services to the life sciences industry. We believe in pushing the boundaries of human science and data science to make the biggest impact possible – to help our customers create a healthier world. Learn more at https://jobs.iqvia.com.