Senior Big Data Engineer

Summary: The Wikimedia Foundation is searching for a Senior Data Engineer to join our Data Engineering team. As a Data Engineer, your responsibilities include creating, maintaining, and enhancing the shared data system that supports research, insights, and new data products, within the Foundation and the Wiki Movement. This involves everything from creating scalable pipeline infrastructure, operating on the event streaming platform to offering scalable data serving solutions. Our team is entirely remote and internationally distributed. We meet in person 1-2 times a year during one of our off-sites (recent ones have been in Berlin, New York, and Copenhagen) or at Wikimania, the annual international conference for the Wiki community. Candidates located in time zones UTC-8 to UTC+2 are welcome to apply. Responsibilities: - Building scalable data pipeline infrastructure, libraries, and processes using Airflow, Spark, Flink, Kafka - Implementing data quality monitoring that alerts the team to possible data issues - Implementing a data governance and lineage solution for all Wikimedia data - Designing the shared data platform that supports use cases for critical aspects of the Wikimedia mission: product analytics, harassment prevention, image classification, bot detection, DDoS attacks flagging and much more. - Improve operational excellence of the data platform Skills and Experience: - 5+ years relevant industry experience - Advanced knowledge of SQL, relational databases, query authoring, preferably in various types (including MariaDB, HiveQL, CassandraQL, Spark SQL, and Presto) - Experience with programming languages such as Python, Scala, and Java - Experience building data pipelines using tools like Airflow, Spark, Gobblin, Yarn - Familiarity with stream processing systems using Kafka, Spark streaming and/or Flink - Excellent written and verbal communication skills - Strong interpersonal and collaboration skills - BS or MS degree, preferably in Computer Science, or equivalent work experience Qualities we value: - Commitment to the mission of the organization and our values - Commitment to our guiding principles - Commitment to diversity, equity, and inclusion - Cross-cultural sensitivity and awareness - Collaborative working experience You might also have: - Experience with Hadoop - Understanding of related disciplines including Machine Learning, Statistics, Privacy and Algorithms - Experience working with site reliability engineers About the Wikimedia Foundation: The Wikimedia Foundation is the nonprofit organization that administers Wikipedia and the other Wikimedia free knowledge projects. Our vision is a world where every single human can freely participate in the sum of all knowledge. We believe everyone has the potential to contribute to our shared knowledge and everyone should have unrestricted access to that knowledge. In addition to hosting Wikipedia and the Wikimedia projects, we develop software experiences for reading, contributing, and sharing Wikimedia content, support the volunteer communities and partners who make Wikimedia possible, and advocate for policies that enable Wikimedia and free knowledge to thrive. The Wikimedia Foundation is a charitable organization that relies on donations. We receive donations from millions of individuals worldwide, with an average donation of about $15. We are an equal opportunity employer that values diversity in its workforce and continually strives for an inclusive and equitable workplace. We encourage applications from individuals of diverse backgrounds. We do not discriminate against any individual based on race, religion, color, national origin, sex, pregnancy or related medical conditions, parental status, sexual orientation, gender identity, age, veteran status, disability or genetic information. This job is remote, with staff and contractors based in more than 50 countries. We set salaries in a way that is competitive, equitable, and consistent with our values and culture. The annual pay range for this US-based position is $105,309 to $163,646, with several particular factors, including the cost of living in the location, being the determinants of the offered pay. For applicants from outside the US, the payment range will be adjusted according to the country of residency. All applicants can ask their recruiter about the specific pay range for their location during the interview process. All qualified applicants requiring assistance or accommodation to complete any step of the application process due to a disability can contact us at [email protected] or +1 (415) 839-6885. More Information: - U.S. Benefits & Perks - Wikimedia Foundation - Applicant Privacy Policy.