Data Engineer

The NewRich Network is seeking a part-time Data Engineer (with an option to go full-time in 6 months) who has experience with data integrations using AWS and Google Cloud technology stack. We are a company wholly dedicated to providing digital information and solutions to our customers worldwide. Our development projects form the backbone of our organization, and so does our development team. We are not mere coders; we are inventors. Our development team is also 100% remote. So if you're looking for a work-from-home opportunity and you have a passion for creating platforms for the new world, we want to hear from you! Requirements: - Design and builds applications that carry out data analysis, transformations, aggregations, and other augmentations on extensive data sets in a spark-based AWS environment (EMR, S3, Glue, Redshift, Athena). - Evaluate pipeline models, tools, and environments, and implement these to push data from our sources through your transformations to our customers. - Work with product management and data research teams to prototype and test new ideas, then take them to production. - Work in a fast-paced, innovate-and-test environment. What You'll Do: - Collaborate with Data architects, Enterprise architects, Solution consultants, and Product engineering teams to gather customer data integration requirements, conceptualize solutions, and build the required technology stack. - Collaborate with the enterprise customer's engineering team to identify data sources, profile and quantify the quality of data sources, develop tools to prepare data and build data pipelines for integrating customer data sources and third-party data sources. - Develop new features and improve existing data integrations with the customer data ecosystem. - Encourage the team to think out-of-the-box and overcome engineering obstacles while incorporating innovative design principles. - Collaborate with a Project Manager to bill and forecast time scale for product owner solutions. More duties include: - Building data pipelines - Reconciling missed data - Acquiring datasets that align with business needs - Developing algorithms to transform data into useful, actionable information - Building, testing, and maintaining database pipeline architectures - Collaborating with management to understand company objectives - Creating new data validation methods and data analysis protocols - Ensuring compliance with data governance and security policies What You Need To Succeed: - 4+ years' experience in Data Engineering - Exceptional communication and interpersonal skills - Bachelor’s degree in Computer Science, Engineering or a related discipline (preferred but not necessary) - 3+ years of experience working on Apache Spark applications using Python (PySpark) or Scala - Experience creating spark jobs that work on at least 1 billion records - Strong knowledge of ETL architecture and standards - Software development experience working with Apache Airflow, Spark, MongoDB, MySQL - Strong SQL knowledge - Strong command of Python - Experience creating data pipelines in a production system - Proven experience in building/operating/maintaining fault-tolerant and scalable data processing integrations using AWS - Experience using Docker or Kubernetes is a plus - Ability to identify and solve problems associated with large-scale data processing workflows in production - Experience with crafting and maintaining unit tests and continuous integration - Passion for crafting intelligent data pipelines that teams love to use - A strong capacity to manage numerous projects is a must Annual Salary: Ranges from 25,000 to 30,000 USD annually.