NAII Seminar: Grounding LLM Evaluations in Social and Cultural Context (Apr 30, 10am)

[Kan Min-Yen]

Dear all,

Prof. Lyle Ungar will be visiting NUS and its AI Institute next week.  The seminar will be broadcast, so you can attend both virtually or in-person.  Please register if you are coming in person.

Hope to see some of you there!

– Min 

(On behalf of NAII, and Kokil Jaidka, his overall visit host)

Dear All, The NUS Artificial Intelligence Institute (NAII) is pleased to invite you to the upcoming seminar by Prof Lyle Ungar on Thursday April 30. Title: Grounding LLM Evaluations in Social and Cultural Context Date: Thursday April 30 Time: 10.00am – 11.30am  Venue: COM2 Executive Classroom (#04-02), 15 Computing Drive, Singapore 117418 Please register for in-person attendance here: https://forms.office.com/r/3ZysJjF87z You can also attend the seminar virtually via Zoom: Zoom Meeting Link https://nus-sg.zoom.us/j/89864008338?pwd=4SyGzON4pEsb9i2fur9qUTmI7Mvh76.1 Meeting ID 898 6400 8338 Password 550475 Abstract: Evaluating LLMs for subjective tasks such as coaching, therapy, and advice giving requires knowing what a "good" response looks like, which depends on cultural and social context. We first demonstrate an interpretable framework for comparing the linguistic cues that LLMs and humans use to signal politeness across languages. We then examine style preservation in cross-cultural translation, finding that LLMs perform worst in non-Western languages, where they systematically bias translations toward stylistic neutrality, a failure invisible to standard metrics. We address this with RASTA, a retrieval-based method that improves culturally appropriate style preservation without degrading content quality. Finally, we introduce the Culturally-Aware Conversations benchmark. Overall,  current frontier LLMs still struggle to converse appropriately across non-Western cultures. This is a joint work with Shreya Havaldar. Speaker biography: Lyle Ungar is a Professor of Computer and Information Science at the University of Pennsylvania, where he also holds secondary appointments in Psychology, Bioengineering, Genomics and Computational Biology, and Operations, Information and Decisions. His group develops natural language processing and explainable AI for psychological and medical research, including analyzing social media to better understand the drivers of physical and mental well-being and building socio-emotionally sensitive AI-based tutors and coaches. Thanks & best regards, NUS AI Institute NUS Artificial Intelligence Institute :: National University of Singapore :: COM3 Building #B1-08, 11 Research Link, Singapore 119391 :: +65 6516 2146 :: na...@nus.edu.sg (E) :: ai.nus.edu.sg (W) :: Company Registration No: 200604346E

 

---

Important: This email is confidential and may be privileged. If you are not the intended recipient, please delete it and notify us immediately; you should not copy or use it for any purpose, nor disclose its contents to any other person. Thank you.