Evaluating natural language ontology definitions from a generative large language model
Robert Stevens
Dear All
One of my final year undergraduate project students is looking at how well a generative large language model can create natural language definitions for terms in the Gene Ontology. her call for people to participate in an evaluation of the outputs is below. I'd be most grateful for any participation.
Thanks
Robert.
Hello all,
I hope that you are well. My name is Asma Alshebli, and I am an undergraduate student at The University of Manchester. For my third-year project, I am evaluating the ability of a Generative Large Language Model (LLM) to produce natural language definitions for the Gene Ontology. Part of my research is evaluating the quality of the definitions produced by publicly available chatbots that rely on LLMs, like ChatGPT. Your expertise would provide valuable insight into this study.
The survey is designed to gather feedback on the clarity, accuracy, and biology coverage of these definitions and will take approximately 20 minutes to complete. All responses will be anonymous and used only for the purpose of this academic research.
Please find the survey link below:
https://www.qualtrics.manchester.ac.uk/jfe/form/SV_cu6CWfiK1TabHNA
If you have any questions please contact me at asma.a...@student.manchester.ac.uk or my supervisor Robert Stevens at Robert....@manchester.ac.uk.
Thank you for your time.
Best regards,
Asma
Professor Robert Stevens
Department of Computer Science
University of Manchester
I may choose to send emails out of working hours, but don’t feel the need to respond out of working hours
Nico Matentzoglu
--
You received this message because you are subscribed to the Google Groups "obo-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to obo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/obo-discuss/LO0P265MB6226E3C53C3A16410556D79DA43D2%40LO0P265MB6226.GBRP265.PROD.OUTLOOK.COM.
Chris Mungall
To view this discussion on the web visit https://groups.google.com/d/msgid/obo-discuss/CAMWhPOfTrS%3Dzv%2B1wYXyrWTF44dFJ5OWQh1vyXszwzDQj_tMemw%40mail.gmail.com.
Jim Balhoff
Robert Stevens
Thanks Chris.
Quite so. One of the things I’d like to try is doing Uberon stuyle (I think it is) – the one that uses a template for nat lang defs along the style of “x is a y that….”. we’ve not tried giving the chat GPT thing instructions to do it this way.
Robert.
Professor Robert Stevens
Department of Computer Science
University of Manchester
I may choose to send emails out of working hours, but don’t feel the need to respond out of working hours
From:
Chris Mungall <cjmu...@lbl.gov>
Sent: Wednesday, April 3, 2024 4:54 PM
To: nicolas.m...@gmail.com
Cc: Robert Stevens <robert....@manchester.ac.uk>; obo-d...@googlegroups.com
Subject: Re: [obo-discuss] Evaluating natural language ontology definitions from a generative large language model
It would be great to compare notes after your survey is completed
I just did it and encourage others too! I think there are some interesting fundamental questions that arise here about definitions, who they are for, why they should be the way they are. I know there have been workshops on definitions at ICBO in the past, it might be good to revisit this as a community some time!
On Wed, Apr 3, 2024 at 9:03 AM Nico Matentzoglu <nicolas.m...@gmail.com> wrote:
Hello Robert!
What a delight to hear from you :)
Sabrina Toro from our group, together with Chris Mungall and a number of colleagues have recently submitted a paper on this subject, preprint here: https://arxiv.org/abs/2312.10904 [arxiv.org]
Perhaps your student may be interested in reviewing the methodology and author list - maybe some of the curators that helped with that paper are willing to help out!
Very fun project, lots of potential, this line of research! Good luck!
Nico
On Wed, 3 Apr 2024 at 15:01, Robert Stevens <robert....@manchester.ac.uk> wrote:
Dear All
One of my final year undergraduate project students is looking at how well a generative large language model can create natural language definitions for terms in the Gene Ontology. her call for people to participate in an evaluation of the outputs is below. I'd be most grateful for any participation.
Thanks
Robert.
Hello all,
I hope that you are well. My name is Asma Alshebli, and I am an undergraduate student at The University of Manchester. For my third-year project, I am evaluating the ability of a Generative Large Language Model (LLM) to produce natural language definitions for the Gene Ontology [geneontology.org]. Part of my research is evaluating the quality of the definitions produced by publicly available chatbots that rely on LLMs, like ChatGPT. Your expertise would provide valuable insight into this study.
The survey is designed to gather feedback on the clarity, accuracy, and biology coverage of these definitions and will take approximately 20 minutes to complete. All responses will be anonymous and used only for the purpose of this academic research.
Please find the survey link below:
https://www.qualtrics.manchester.ac.uk/jfe/form/SV_cu6CWfiK1TabHNA
If you have any questions please contact me at asma.a...@student.manchester.ac.uk or my supervisor Robert Stevens at Robert....@manchester.ac.uk.
Thank you for your time.
Best regards,
Asma
Professor Robert Stevens
Department of Computer Science
University of Manchester
I may choose to send emails out of working hours, but don’t feel the need to respond out of working hours
--
You received this message because you are subscribed to the Google Groups "obo-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to obo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/obo-discuss/LO0P265MB6226E3C53C3A16410556D79DA43D2%40LO0P265MB6226.GBRP265.PROD.OUTLOOK.COM [groups.google.com].
--
You received this message because you are subscribed to the Google Groups "obo-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to obo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/obo-discuss/CAMWhPOfTrS%3Dzv%2B1wYXyrWTF44dFJ5OWQh1vyXszwzDQj_tMemw%40mail.gmail.com [groups.google.com].
Robert Stevens
Hi Jim
We didn’t use any logical axioms in this work. It was only some prompt crafting and a few examples of GO nat lang definitions for some few shot learning.
Our underlying question is whether a Gen LLM (Chat GPT here) is any good on its own at generating nat lang definitions.
Robertft.
Professor Robert Stevens
Department of Computer Science
University of Manchester
I may choose to send emails out of working hours, but don’t feel the need to respond out of working hours
From:
obo-d...@googlegroups.com <obo-d...@googlegroups.com> On Behalf Of
Jim Balhoff
Sent: Wednesday, April 3, 2024 4:58 PM
To: Robert Stevens <robert....@manchester.ac.uk>
Cc: obo-d...@googlegroups.com
Subject: Re: [obo-discuss] Evaluating natural language ontology definitions from a generative large language model
Hi Robert, I’m one of the maintainers of the Gene Ontology, and I was curious which release file your student will be using for her work. I’m asking because the “main” file (go. owl) has many logical axioms stripped out, to simplify it for most
ZjQcmQRYFpfptBannerStart
|
ZjQcmQRYFpfptBannerEnd
Hi Robert,
I’m one of the maintainers of the Gene Ontology, and I was curious which release file your student will be using for her work. I’m asking because the “main” file (go.owl) has many logical axioms stripped out, to simplify it for most users. But there is another release which more closely matches our editors’ file: go-plus.owl. This one contains more logical axioms and references to external ontologies (it also imports some content from those ontologies via a module extraction). Those files are described here: https://geneontology.org/docs/download-ontology/ [geneontology.org]
I just wanted to make sure you are aware, since I have seen several OWL reasoning studies that used only the more limited file, in case the method here will take those axioms into account.
Best regards,
Jim
On Apr 3, 2024, at 8:01 AM, Robert Stevens <robert....@manchester.ac.uk> wrote:
Dear All
One of my final year undergraduate project students is looking at how well a generative large language model can create natural language definitions for terms in the Gene Ontology. her call for people to participate in an evaluation of the outputs is below. I'd be most grateful for any participation.
Thanks
Robert.
Hello all,
I hope that you are well. My name is Asma Alshebli, and I am an undergraduate student at The University of Manchester. For my third-year project, I am evaluating the ability of a Generative Large Language Model (LLM) to produce natural language definitions for the Gene Ontology [geneontology.org]. Part of my research is evaluating the quality of the definitions produced by publicly available chatbots that rely on LLMs, like ChatGPT. Your expertise would provide valuable insight into this study.
The survey is designed to gather feedback on the clarity, accuracy, and biology coverage of these definitions and will take approximately 20 minutes to complete. All responses will be anonymous and used only for the purpose of this academic research.
Please find the survey link below:
If you have any questions please contact me at asma.a...@student.manchester.ac.uk or my supervisor Robert Stevens at Robert....@manchester.ac.uk.
Thank you for your time.
Best regards,
Asma
Professor Robert Stevens
Department of Computer Science
University of Manchester
I may choose to send emails out of working hours, but don’t feel the need to respond out of working hours
--
You received this message because you are subscribed to the Google Groups "obo-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to obo-discuss...@googlegroups.com
.
To view this discussion on the web visit https://groups.google.com/d/msgid/obo-discuss/LO0P265MB6226E3C53C3A16410556D79DA43D2%40LO0P265MB6226.GBRP265.PROD.OUTLOOK.COM [groups.google.com].
--
You received this message because you are subscribed to the Google Groups "obo-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to
obo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/obo-discuss/939C6DC5-FFD4-4C8B-9D32-E8EAC72B35F4%40gmail.com [groups.google.com].
Robert Stevens
Hello again Jim and all
The comment about using axioms in nat lang definition generation reminded me of some old work using rhetorical structure theory to organise axioms when using them for natural language generation of text descriptions of classes. It relies on a fairly rich axiomatisation of a class and uses only the logical axioms.
OntoVerbal: a generic tool and practical application to SNOMED CT
SF Liang, D Scott, R Stevens, A Rector
arXiv preprint arXiv:1312.2798
Other papers with Liang as an author describe the work from other perspectives.
Robert.
Professor Robert Stevens
Department of Computer Science
University of Manchester
I may choose to send emails out of working hours, but don’t feel the need to respond out of working hours
From: obo-d...@googlegroups.com <obo-d...@googlegroups.com> On Behalf Of Jim Balhoff
Sent: Wednesday, April 3, 2024 4:58 PM
To: Robert Stevens <robert....@manchester.ac.uk>
Cc: obo-d...@googlegroups.com
Subject: Re: [obo-discuss] Evaluating natural language ontology definitions from a generative large language model
Hi Robert, I’m one of the maintainers of the Gene Ontology, and I was curious which release file your student will be using for her work. I’m asking because the “main” file (go. owl) has many logical axioms stripped out, to simplify it for most
ZjQcmQRYFpfptBannerStart
|
ZjQcmQRYFpfptBannerEnd
Hi Robert,
I’m one of the maintainers of the Gene Ontology, and I was curious which release file your student will be using for her work. I’m asking because the “main” file (go.owl) has many logical axioms stripped out, to simplify it for most users. But there is another release which more closely matches our editors’ file: go-plus.owl. This one contains more logical axioms and references to external ontologies (it also imports some content from those ontologies via a module extraction). Those files are described here: https://geneontology.org/docs/download-ontology/ [geneontology.org]
I just wanted to make sure you are aware, since I have seen several OWL reasoning studies that used only the more limited file, in case the method here will take those axioms into account.
Best regards,
Jim
On Apr 3, 2024, at 8:01 AM, Robert Stevens <robert....@manchester.ac.uk> wrote:
Dear All
One of my final year undergraduate project students is looking at how well a generative large language model can create natural language definitions for terms in the Gene Ontology. her call for people to participate in an evaluation of the outputs is below. I'd be most grateful for any participation.
Thanks
Robert.
Hello all,
I hope that you are well. My name is Asma Alshebli, and I am an undergraduate student at The University of Manchester. For my third-year project, I am evaluating the ability of a Generative Large Language Model (LLM) to produce natural language definitions for the Gene Ontology [geneontology.org]. Part of my research is evaluating the quality of the definitions produced by publicly available chatbots that rely on LLMs, like ChatGPT. Your expertise would provide valuable insight into this study.
The survey is designed to gather feedback on the clarity, accuracy, and biology coverage of these definitions and will take approximately 20 minutes to complete. All responses will be anonymous and used only for the purpose of this academic research.
Please find the survey link below:
If you have any questions please contact me at asma.a...@student.manchester.ac.uk or my supervisor Robert Stevens at Robert....@manchester.ac.uk.
Thank you for your time.
Best regards,
Asma
Professor Robert Stevens
Department of Computer Science
University of Manchester
I may choose to send emails out of working hours, but don’t feel the need to respond out of working hours
--
You received this message because you are subscribed to the Google Groups "obo-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to obo-discuss...@googlegroups.com
.
To view this discussion on the web visit https://groups.google.com/d/msgid/obo-discuss/LO0P265MB6226E3C53C3A16410556D79DA43D2%40LO0P265MB6226.GBRP265.PROD.OUTLOOK.COM [groups.google.com].
--
You received this message because you are subscribed to the Google Groups "obo-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to
obo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/obo-discuss/939C6DC5-FFD4-4C8B-9D32-E8EAC72B35F4%40gmail.com [groups.google.com].