[PennStatNLP] Reading Group - 10/14 - Josh Nguyen: "The nature of LLM common sense"
1 view
Skip to first unread message
Liam Dugan
Oct 14, 2024, 1:43:15 PM10/14/24
to penns...@googlegroups.com
Hello everyone!
Here are the details for today’s reading group meeting. I hope to see you all there.
Also, I'm looking for volunteers to present for the week after next week (10/28). Please let me know if you'll be available to present for that week.
Zoom Link: https://nyu.zoom.us/j/98819790926
Paper(s): Language Models as Agent Models & Role play with large language models & AI-Augmented Surveys: Leveraging Large Language Models and Surveys for Opinion Prediction
Abstract: I will summarize these two uses of LLMs and how they relate to commonsense knowledge, my topic of research. In particular, the nature of common sense — that it varies with many factors such as culture, time and place — does not permit any “ground truth” to anything under its umbrella. Hence, model accuracy with respect to human-prescribed labels does not make sense when we talk about this knowledge. What we propose as an alternative is agreement: common sense is better measured as the degree to which people in some group agree with one another — and not simply as what one benchmark designer determines. There are, in other words, no ground truths but only similarities and differences. I apply the above two uses of an LLM to evaluating its common sense: as (1) an independent agent that participates in society with real humans, and as (2) a simulator of a group of hypothetical humans that collectively construct their reality.
--------
Presenter: Josh Nguyen
Presenter: Josh Nguyen
Date & Time: Monday (10/14) from 4-5pm ET
Zoom Link: https://nyu.zoom.us/j/98819790926
Paper(s): Language Models as Agent Models & Role play with large language models & AI-Augmented Surveys: Leveraging Large Language Models and Surveys for Opinion Prediction
Abstract: I will summarize these two uses of LLMs and how they relate to commonsense knowledge, my topic of research. In particular, the nature of common sense — that it varies with many factors such as culture, time and place — does not permit any “ground truth” to anything under its umbrella. Hence, model accuracy with respect to human-prescribed labels does not make sense when we talk about this knowledge. What we propose as an alternative is agreement: common sense is better measured as the degree to which people in some group agree with one another — and not simply as what one benchmark designer determines. There are, in other words, no ground truths but only similarities and differences. I apply the above two uses of an LLM to evaluating its common sense: as (1) an independent agent that participates in society with real humans, and as (2) a simulator of a group of hypothetical humans that collectively construct their reality.
Best,
Liam
Reply all
Reply to author
Forward
0 new messages