Senior/Staff Research Scientist, SEAL

As the primary data and evaluation associate for cutting-edge AI companies, Scale is pivotal in understanding and protecting large language models (LLMs).

The Safety, Evaluations and Analysis Lab (SEAL) constitutes Scale's novel research initiative that focuses on creating reliable evaluation products and addressing the complex research challenges in evaluation and red teaming.

We are actively looking for skilled researchers to join us as we aim to shape the field for safety and transparency across the entire AI sector. We support collaborations within the industry and the publishing of our research results. Below are some of SEAL's representative projects:

• Design and execute robust evaluation benchmarks. Enhance and ensure evaluation reproducibility and dependability.
• Research and apply advanced model-assisted red teaming methods.
• Develop leading-edge LLM-based automated rating systems using the Scale ecosystem for custom training data.
• Develop rater assist approaches, such as critique modeling, to improve rating quality.

Ideally, you would possess:

• A dedication to our purpose of advancing safe, secure, and reliable AI implementations in the industry.
• Practical experience in conducting collaborative technical research, with fluency in frameworks like Pytorch, Jax, or Tensorflow. You should also be proficient at understanding research literature and quickly transforming new concepts into prototypes.
• A history of published research in machine learning, notably in generative AI.
• At least a three-year experience dealing with intricate ML problems, either in a research setting or in product development.
• Strong verbal and written communication skills to function within a cross-functional team.

Appreciated qualifications:

• Practical experience with open source LLM fine-tuning or involvement in specific LLM fine-tuning projects using Pytorch/Jax.
• Experience in creating evaluations or a history in data science roles associated with LLM technologies.
• Experience in working with cloud technology stack (eg. AWS or GCP) and developing machine learning models in a cloud environment.

Our research interviews are designed to assess candidates' skills in practical ML prototyping and debugging, their understanding of research concepts, and their compatibility with our corporate culture. We will not pose any LeetCode-style queries.

The base for this full-time role at Scale AI in San Francisco, New York, or Seattle, ranges from $200,000 - $370,000. At Scale, compensation packages include base salary, equity, and benefits. The range displayed on each job post indicates the minimum and maximum target salaries for the role, which is determined by location and other factors including job-related skills, experience, interview performance, and relevant education or training. Your recruiter can provide more details about the specific salary range for your preferred location during the hiring process. Scale employees also receive Stock Options that are approved by the board of directors. Benefits include, but are not limited to: Comprehensive health, dental and vision coverage, retirement benefits, a learning and development stipend, and generous PTO. This role may also be eligible for extra benefits like a commuter stipend.