Senior Software Development Engineer, Big Data

About the team The Valuation & Housing Trends Data Engineering team at Zillow Group brings innovative analysis, creativity, and excellence to the U.S. housing market. We create, streamline, analyze, and share a full collection of housing metrics and analysis for Zillow’s internal and external customers. Our work helps to contextualize Zillow’s unrivaled living database of all homes and hundreds of millions of consumers, providing insight to regular consumers and offering open-data to hundreds of researchers and policymakers nationwide. You'll be working in a fun, collaborative environment with a high-performance team possessing a diverse set of data engineering and analytical skills. Your work will have significant impact. The data we produce ends up in the country's most reputable publications, is used by government institutions for policymaking and official reports, populates public-facing tools used by millions of Zillow consumers, and underpins the work of hundreds of academic researchers. Small team = big impact. Our engineering teams are highly decentralized, providing the speed and autonomy of a start-up environment while benefiting from the resources of a larger company. About the role We process hundreds of millions of data points to produce macroeconomic housing market indicators, forecasts, and insights. We publish these insights to enable businesses, policymakers, and individuals to make informed decisions about homes. We are also a crucial component of housing market intelligence for our internal businesses, and an essential contributor to tactical and strategic business decisions with customers across the entire organization, including marketing, finance, and business unit ML teams to executives. We are seeking a proficient engineer to work closely with our data science and forecasting teams, and our customers, to help us remodel our rapidly evolving data products and end-to-end data processing and model development framework. You'll have excellent opportunities to learn state-of-the-art ecosystems, and cloud technologies, and work with a group of versatile software developers in a team-oriented, supportive atmosphere. You will: - Build end-to-end data processing systems to ingest raw data and concrete Housing Trends insights. - Build and extend our internal forecasting model development framework. - Architect, build and maintain scalable data products and deliver pipelines built for speed, accuracy, and consistency that will scale as our needs and those of our users grow. - Identify, design, and implement internal process improvements, including automation of manual processes, optimization of data delivery, redesign of infrastructure for greater scalability and performance, etc. - Design and implement LLM based Chatbot tools to aid our customers in better understanding our data and improving their productivity. - Be a trusted partner for Economists, Applied Scientists, and Data Analysts. - Collaborate with multiple teams, including our Data Science, Forecasting, Engineering, and Property Data teams, to efficiently source Zillow’s proprietary datasets for creating industry-leading housing metrics and forecasts. - Translate requirements into development plans and effectively execute on goals. This role is categorized as a Remote position. "Remote" employees do not have a permanent corporate office workplace and instead, work from a physical location of their choice which must be identified to the Company. Employees may live in any of the 50 US States, with limited exceptions. In California, Colorado, Connecticut, Nevada, New York, and Washington the standard base pay range for this role is $159,400.00 - $254,600.00 Annually. This base pay range is specific to California, Colorado, Connecticut, Nevada, New York, and Washington and may not be applicable to other locations. In addition to a competitive base salary, this position is also eligible for equity awards based on factors such as experience, performance, and location. Actual amounts will vary depending on experience, performance, and location. Who you are - Proficient in Python, Spark, SQL, and Hive programming for data analytics and ETL pipeline development. - Experience in building systems on AWS cloud platforms is required. - In-depth knowledge of big data ecosystems and platforms, including Spark, EMR, EKS, Hive, Hadoop, Presto, Kubernetes, and Airflow. - Hands-on experience with Lakehouse technologies like Databricks, Delta Lake, HUDI, Iceberg is a significant advantage. - Familiar with Agile/DevOps software development processes and tools such as GitLab and CI/CD. - Demonstrated experience in building and orchestrating batch and real-time ETL pipelines, handling vast volumes of data from diverse sources. - A good understanding of data science concepts, fundamental machine learning algorithms/libraries, and the development lifecycle is a must-have. - Experience in collaborating with data scientists to conduct feature engineering, deploy machine learning models in production environments, and identify growth opportunities. - Strong data analysis skills to meticulously investigate data issues and provide comprehensive responses to customer queries. - Strong ability in root cause analysis to address operational issues and identify opportunities for improvement. - Good understanding of measuring and ensuring data quality at scale, along with expertise in the necessary tooling for monitoring and optimizing the performance of all data pipelines. - Familiarity with LLM technologies (OpenAI API etc.) is a nice-to-have. - At least 5 years of experience in Big Data Engineering, Machine Learning Engineering, or Data Analytics roles. Get to know us Zillow is reimagining real estate to help make owning a home a reality for more and more people. As the most-visited real estate website in the United States, Zillow® and its affiliates aid movers in finding and securing their home through digital solutions, upper-class partners, and easier buying, selling, financing, and renting experiences. Every month, millions of people visit Zillow Group sites to start their home search, and now they can rely on Zillow to help make it easier to move. The work we do aids people in finding their homes and no matter what job you're in, you will play a critical role in making homeownership a reality for more individuals. Our efforts to streamline the real estate transaction are powered by a robust culture of innovation, a passion for redefining the employee experience, a fundamental commitment to Equity and Belonging, and world-class benefits that include comprehensive medical, dental, vision, life, and disability coverages, as well as parental leave, family benefits, retirement contributions, and paid time off. We're also setting the standard for future work experiences, where our employees are supported in delivering their best work and living a flexible, balanced life. You don't have to just take our word for it. You can read recent reviews on Glassdoor and see recent recognition from multiple organizations, including the 100 Best Companies to Work For, Glassdoor Employees’ Choice Award, Bloomberg Gender-Equality Index, Human Rights Campaign (HRC) Corporate Equity Index, and TIME's 100 Most Influential Companies list. Zillow Group is an equal opportunity employer committed to fostering an inclusive, innovative environment with the best employees. We are committed to equal employment opportunity regardless of race, color, ancestry, religion, sex, national origin, sexual orientation, age, citizenship, marital status, disability, gender identity, or Veteran status. If you have a disability or special need that requires accommodation, please contact us at [email protected]. Applicants who receive job offers from Zillow Group will be asked to sign a Proprietary Rights Agreement which includes confidentiality, intellectual property assignment, customer and employee non-solicitation, and non-competition provisions. If you are contacted for a role at Zillow Group and would like to review a copy of the Proprietary Rights Agreement before receiving an offer, you can request a copy from your Recruiter.