Join Our Team as a Sr AI Quality Engineer at Houghton Mifflin Harcourt (HMH)

Who We Are
At HMH, we’re dedicated to transforming education through innovative learning solutions. Our platforms empower millions of learners to unlock their potential and achieve their dreams. By joining our team, you'll contribute to impactful projects that shape the future of education for over 53 million students and teachers worldwide.

Primary Responsibilities

• Develop and execute comprehensive test plans and cases to ensure the functionality, accuracy, and performance of large language models (LLMs) across various use cases.
• Utilize diverse testing techniques—such as input fuzzing, adversarial testing, and bias detection—to guarantee robustness and inclusivity.
• Conduct comparative analyses to evaluate LLM prompt templates and user interactions, identifying the most effective implementations.
• Collaborate with cross-functional teams, including developers, data scientists, and product managers, to refine design, implementation, and testing strategies.
• Lead AI safety and responsibility initiatives, ensuring compliance with ethical AI principles, data privacy standards, and regulatory requirements.
• Measure LLM performance using metrics like precision, recall, BLEU score, and perplexity. Maintain metrics systems to continuously improve quality.
• Perform regression, load, and performance testing using Gatling to ensure scalability and optimal functionality of LLM components.
• Identify, document, and prioritize issues and bugs. Use debugging tools to troubleshoot model predictions and data processing issues.
• Develop test strategies for data processing pipelines, addressing data quality and pipeline scalability concerns.
• Stay updated on developments in AI ethics, NLP, ML research, and quality assurance methodologies, integrating best practices into QA processes.
• Effectively communicate test results and recommendations to stakeholders, providing actionable feedback for improving LLM applications.

What We’re Looking For

We seek experienced and motivated individuals passionate about enhancing their quality craftsmanship, building advanced automated testing suites, and contributing to our quality culture.

Education & Experience

• Bachelor’s degree in computer science or equivalent experience in software quality assurance.

Required Knowledge and Skills

• Proven experience in software quality assurance, particularly with NLP applications or LLMs.
• Strong grasp of NLP and ML concepts, algorithms, and techniques, working with models like BERT or GPT.
• Expertise in creating and executing test plans and scenarios to validate LLM applications and data pipelines.
• Familiarity with NLP-specific testing techniques, including input fuzzing and adversarial testing.
• Understanding of evaluation metrics like precision, recall, BLEU score, and perplexity.
• Excellent analytical and troubleshooting skills for LLM behavior and data processing.
• Ability to prioritize tests based on feature criticality and user impact.
• Strong communication skills for working within cross-functional teams and explaining technical concepts to non-technical stakeholders.
• Self-motivated and proactive, with a passion for quality and continuous improvement in testing methods.

Additional Information
Company: Houghton Mifflin Harcourt
Job Title: Sr AI Quality Engineer