Postdoctoral fellow in LLM auditing at UCSF

[jean feng]

Dear all,

Our group is hiring for a postdoctoral fellow. Please see job description below or at https://www.jeanfeng.com/joining.html

The unprecedented ability of large language models (LLMs) to interpret text data with human-like reasoning is poised to transform many fields. Nevertheless, for LLMs to be safe and effective for use in high-risk domains like healthcare, it is crucial to understand biases embedded in this technology, as it has been shown to vary in performance across subgroups and even discriminate against minorities. This project aims to study and develop red-teaming solutions to audit LLMs and understand the limits of current approaches. Methodologies developed in this project will be tested on real-world clinical data, including unstructured notes. This project is a supplement of our existing PCORI project “Diagnostic Tools for Quality Improvement of Machine Learning-Based Clinical Decision Support Systems” (see project description at https://www.pcori.org/research-results/2022/diagnostic-tools-quality-improvement-machine-learning-based-clinical-decision-support-systems).

RESPONSIBILITIES

We are seeking a highly motivated postdoctoral researcher to join our team. The primary responsibilities are:

• Rigorously analyze and evaluate existing red-teaming algorithms for LLMs
• Develop new statistical methods/frameworks for comprehensive red-teaming of LLMs
• Develop an explanation framework and statistical inference procedures to understand systematic limitations of LLMs
• Write, edit, and publish research manuscripts in collaboration with the team
  

Our predictive analytics team is highly collaborative and includes team members with wide-ranging expertise, including machine learning, biostatistics, computer science, healthcare, and regulation.

QUALIFICATIONS

The position requires at least a PhD degree or equivalent in data science, (bio)statistics, computer science, or another relevant field. We are looking for someone who:

• has experience in training and testing ML algorithms for large datasets
• has experience in natural language processing and working with LLMs
• has experience in methodological development and can perform independent research, with a strong and relevant publication record
• has strong software engineering background (e.g. python, torch, huggingface, git-based workflows, high-performance computing, SQL, spark)
• is able to work collaboratively with a team
  

The position has two years of guaranteed funding. Screening of applicants will begin immediately and will continue as needed throughout the recruitment period. If you are interested, please submit the following materials to jean...@ucsf.edu:

• A cover letter
• A CV summarizing your education and work experience so far
• The names and email addresses of three references
• A code sample on github
• One representative publication

--

Jean Feng

Assistant Professor
Department of Epidemiology and Biostatistics
University of California, San Francisco
https://www.jeanfeng.com/