GenAI Research Scientist, Model Evaluation & Safety Team

Company Description

MosaicML, founded in late 2020 by a small team of machine learning researchers, empowers companies to build cutting-edge AI models using their own data. On a business level, MosaicML believes that a company's AI models are as valuable as any of their fundamental intellectual properties. They are dedicated to making high-quality AI models accessible to all. From a scientific perspective, MosaicML is committed to minimizing the cost of training state-of-the-art models and disseminating this information worldwide, fostering innovation and model creation for everyone.

Since July 2023, as the GenAI Team under Databricks, we are passionate about equipping our customers with the ability to solve the world's most difficult problems by providing the leading-edge data and AI platform. We eagerly take on all technical challenges, with our primary aim being to provide top-notch data and AI capabilities to our customers.

Job Description

As a Research Scientist on the GenAI Team at Databricks, your responsibilities include staying abreast of the latest deep learning progress and pushing the scientific boundaries by developing techniques that surpass the current state of the art. You will be part of a cooperative team of researchers with diverse experiences and technical training. Above all, our focus is on our customers. We aim to help them succeed in training large models by embedding our scientific knowledge into our products to facilitate this.

In the area of model evaluation specifically, your work will encompass designing high-performance, science-driven evaluation suites that help our research scientists and customers make crucial decisions when training and deploying state-of-the-art generative models for text, images, and other areas. You'll have the chance to advance the state of the art in evaluation of tool use, code generation, RAG, safety and toxicity evaluation, and model-based evaluation. Furthermore, you will help build and design systems that enable our customers to develop the next generation of evaluations for their custom, domain-specific generative models.

Eligibility Criteria

• You're comfortable working with large-scale language models (LLMs) with tens to hundreds of billions of parameters.
• You have strong skills in ML and text/data processing that encompass both scientific research and engineering.
• You have real-world experience deploying LLM systems and have designed innovative ways to assess system performance.
• You've worked on safety measures, jailbreaking, or red-teaming LLMs.
• You are passionate about getting your work into the hands of real users and, more broadly, about democratizing access to cutting-edge AI technology.
• You're driven by engaging in LLM research that — contrary to the current trend in the field — will be made public.
• You possess excellent communication skills and enjoy working independently on open-ended problems.

This position does not require a PhD. We are open to hiring candidates with bachelor's and master's degrees, recent graduates, and those currently in "research engineer" roles at other companies.

Your responsibilities will include

• Staying current with research literature and thinking beyond the existing state of the art to meet user needs.
• Developing and executing novel methods to assess the capabilities of generative models for text, images, and other domains.
• Meticulously testing these methods, revealing the results of your findings, and implementing those that prove helpful.

Transparency of Pay Range

Databricks is committed to ensuring fair and equitable remuneration practices. The pay range for this role is listed below and represents the base salary range for non-commissionable roles or the on-target earnings for commissionable roles. Actual compensation packages are based on several distinctive factors including but not limited to job-related skills, depth of experience, relevant qualifications and training, and specific work location. Considering these factors, Databricks utilizes the full width of the range. This position's total compensation package may also include eligibility for an annual performance bonus, equity, and the benefits listed above. For more information on which range your place is in, visit our page .