Test Dataset & Leaderboard Released

[Bohui Zhang]

Dear participants,

Thanks for your interest in our challenge. The test dataset has been released. Submit your prediction files on the CodaLab leaderboard (test) to get your scores and ranking.

Best,

The LM-KBC Team

[BORISTA FONDI FOMUBAD]

Ok thanks.

Le jeu. 18 juil. 2024 à 16:31, Bohui Zhang <bhzh...@gmail.com> a écrit :

Hi Borista,

In general, the rules have been stated on our challenge website and repository. Besides, based on the questions asked by other participants, I would refer to the general stance again: 

• Improving the LLM in general is allowed (e.g., further training on relevant text corpora, etc.) 
• Improving the LLM on problem-specific structured data is not allowed (e.g., downloading relevant relations from Wikidata) 
• Retrieval-augmentation at test time is not allowed (we want to evaluate the knowledge inside LLMs, not their abilities to extract knowledge from other sources).

Hope this helps.

Best,

The LM-KBC Team

On Thu, Jul 18, 2024 at 4:20 PM BORISTA FONDI FOMUBAD <borista...@facsciences-uy1.cm> wrote:

Ok sir, Thanks for your response.

Please are there any other rules?

Le jeu. 18 juil. 2024 à 16:02, Bohui Zhang <bhzh...@gmail.com> a écrit :

Hi Borista,

Thanks for your question. We would not recommend that solution as it is not aligned with our competition.

Best,

The LM-KBC Team

On Thu, Jul 18, 2024 at 3:27 PM BORISTA FONDI FOMUBAD <borista...@facsciences-uy1.cm> wrote:

Dear Organizers,

I hope this message finds you well. My name are Borista Fondi, and I am currently participating in the LM-KBC Challenge @ISWC 2024. I am writing to seek clarification regarding another(an) approach I am considering for the competition.

Specifically, I am interested in understanding if it is permissible to fine-tune a model with the objective of generating SPARQL queries for each subject entity. The idea is that these SPARQL queries would be executed on a Knowledge Graph (KG) to predict the object entities for the given subject entity and relation.

My approach aims to leverage the power of pre-trained language models in generating precise and contextually accurate SPARQL queries, which could then be used to retrieve the relevant object entities from a KG. I don't know if it would be a greate idea, because this method could potentially enhance the accuracy and completeness of the knowledge base construction task.

Could you please confirm if this approach aligns with the competition guidelines? Additionally, if there are any specific considerations or constraints I should be aware of when implementing this method.
Please, your advice would be greatly appreciated.

Thank you for your time and assistance. I look forward to your response.

Ce courriel est confidentiel et ne s’adresse qu’à son destinataire. S’il vous a été transmis par mégarde, veuillez le détruire et nous en aviser aussitôt. / The content of this email is confidential and intended for the recipient specified in the message only. If you received this message by mistake, please delete it and inform us.

--
You received this message because you are subscribed to a topic in the Google Groups "lm-kbc2024" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/lm-kbc2024/Z3h4W7EPdn0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to lm-kbc2024+...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/lm-kbc2024/a1142306-30e2-4815-80cf-d3995daf8900n%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Ce courriel est confidentiel et ne s’adresse qu’à son destinataire. S’il vous a été transmis par mégarde, veuillez le détruire et nous en aviser aussitôt. / The content of this email is confidential and intended for the recipient specified in the message only. If you received this message by mistake, please delete it and inform us.

Ce courriel est confidentiel et ne s’adresse qu’à son destinataire. S’il vous a été transmis par mégarde, veuillez le détruire et nous en aviser aussitôt. / The content of this email is confidential and intended for the recipient specified in the message only. If you received this message by mistake, please delete it and inform us.

[Bohui Zhang]

Ce courriel est confidentiel et ne s’adresse qu’à son destinataire. S’il vous a été transmis par mégarde, veuillez le détruire et nous en aviser aussitôt. / The content of this email is confidential and intended for the recipient specified in the message only. If you received this message by mistake, please delete it and inform us.

--
You received this message because you are subscribed to a topic in the Google Groups "lm-kbc2024" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/lm-kbc2024/Z3h4W7EPdn0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to lm-kbc2024+...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/lm-kbc2024/a1142306-30e2-4815-80cf-d3995daf8900n%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Bohui Zhang]

Hi Borista,

Thanks for your question. We would not recommend that solution as it is not aligned with our competition.

Best,

The LM-KBC Team

On Thu, Jul 18, 2024 at 3:27 PM BORISTA FONDI FOMUBAD <borista...@facsciences-uy1.cm> wrote:

Dear Organizers,

I hope this message finds you well. My name are Borista Fondi, and I am currently participating in the LM-KBC Challenge @ISWC 2024. I am writing to seek clarification regarding another(an) approach I am considering for the competition.

Specifically, I am interested in understanding if it is permissible to fine-tune a model with the objective of generating SPARQL queries for each subject entity. The idea is that these SPARQL queries would be executed on a Knowledge Graph (KG) to predict the object entities for the given subject entity and relation.

My approach aims to leverage the power of pre-trained language models in generating precise and contextually accurate SPARQL queries, which could then be used to retrieve the relevant object entities from a KG. I don't know if it would be a greate idea, because this method could potentially enhance the accuracy and completeness of the knowledge base construction task.

Could you please confirm if this approach aligns with the competition guidelines? Additionally, if there are any specific considerations or constraints I should be aware of when implementing this method.
Please, your advice would be greatly appreciated.

Thank you for your time and assistance. I look forward to your response.

Le mardi 16 juillet 2024 à 15:33:25 UTC+1, Bohui Zhang a écrit :

[Marcelo Machado]

Hey guys,

I'm afraid I didn't find these rules either on the website or on GitHub. Could you direct me to where they are?


Unfortunately, I've been working all this time on the assumption that the only rule would be about the number of parameters, especially since I came to this Challenge inspired by the winning work  from last year's open track. They made use of contexts.

Although our solution could have good results without the context creation strategy, this was not the focus of our implementation. So, if these rules are restrictive, I will need to withdraw my submission.