DataOS Data Engineer

DataOS Data Engineer Description - The DataOS Data Engineer works on cutting-edge advances in scalable solutions for data ingestion, manipulation, and end-to-end integration of data building blocks to support business-driven data products. This role applies developed subject matter knowledge to solve common and complex business issues within established guidelines and recommends appropriate alternatives. It involves work on problems of diverse complexity and scope. May act as a team or project coach, providing direction to team activities and facilitating information validation and team decision-making processes. Judgment is exercised within generally defined policies and practices to identify and select solutions. Ability to handle most unique situations. May seek advice to support complex business issues. Responsibilities - Designs and establishes secure and performant data architectures, enhancements, updates, and programming changes for portions and subsystems of data product pipelines, repositories, or models for structured/unstructured data. Focuses on developing sharable libraries that reduce development time and maintenance. - Analyzes design and determines coding, programming, and integration activities required based on general objectives and knowledge of overall architecture of the product or solution. - Writes and executes complete testing plans, protocols, and documentation for the assigned portion of the data system or component. Identifies, debugs, and creates solutions for issues with code and integration into the data system architecture. - Leads a project team of other data engineers to develop reliable, cost-effective, and high-quality solutions for the assigned data system, model, or component. - Collaborates and communicates with the project team regarding project progress and issue resolution. Supports co-development processes and tools to promote common approaches to solve complex problems. - Represents the data engineering team for all phases of larger and more-complex development projects. - Provides guidance and mentoring to less experienced staff members. Strong technical leadership is required. Knowledge & Skills - Uses data engineering tools, languages (Python is a must. Java Scala is a plus), and frameworks to mine, cleanse and explore large data sets. - Fluent in SQL & cloud-based data systems. Experienced in relational data modeling. - Fluent in complex, distributed, and massively parallel cloud systems (AWS, GCP, AZURE). - Strong analytical and problem-solving skills with the ability to represent complex algorithms in software. - The ability to design data systems/solutions to manage complex data that are highly scalable and performant. - Ability to performance tune Spark code. - Strong understanding of database technologies and management systems. - Strong understanding of cloud-based systems/services. Knowledge to differentiate benefits for big data Lake House vs. Warehouse. - Experience with workflow orchestration tools (Airflow, Jenkins) - Extensive Notebook environment experience (Jupyter, DataBricks) - Experience in collecting requirements from partners and choosing the right technologies to meet end-to-end data flow requirements that are necessary. (Data size, delivery times: hourly, daily, monthly or real-time approaches) - Knowledge in the database architecture testing methodology, including execution of test plans, debugging, and testing scripts and tools. - Excellent written and verbal communication skills in English and the local language. - Ability to effectively communicate product architectures, design proposals, and negotiate options at management levels. Scope & Impact - Collaborates with peers, junior engineers, data scientists, and project team. - Typically interacts with high-level Individual Contributors, Managers and Program Teams. - Leads a project that requires data engineering solutions development. Education & Experience - Bachelor's or Master's degree in Computer Science, Data Engineering, Information Systems, Engineering or equivalent. - Typically, 4-6 years' experience is required. Job - Software Schedule - Full time Shift - No shift premium (United States of America) Travel - Relocation - EEO Tagline - HP Inc. is an equal opportunity employer. We consider applicants without regard to race, color, religion, sex, national origin, veteran or disability status.