Creating an LLM to answer implementation/modeling questions

[Andreas Boltres]

[2/2] [will post this as an answer once the first half of the post is approved] 

I created a custom GPT with access to a single .txt context file [2] and custom answering instructions ("ns3_gpt_instructions.txt"). I queried it for two different made-up tasks related to packets and shared links to the chat:

1. Measuring end-to-end delays of individual packets (custom GPT [3a] vs. default GPT 4o [3b])
2. Retrieve the size of a packet's payload when not knowing which headers/trailers exist (custom GPT [4a] vs. default GPT 4o [4b])

Results are, I guess, not satisfactory, as it's evident that some of ns-3's modeling has not been understood by GPT. Also, customizing GPT in the naive way I've done it does not seem to improve the answer quality. Of course, this was a minimal-effort attempt of an LLM amateur, and I did not invest time e.g. to craft vector embeddings of the context or professional instruction sets, nor to thoughtfully design the evaluation questions and interpret the answers in detail. With this experiment and this post in general, I simply wanted to put out an inspiration and starting point for a discussion. 

Cheers, Andreas

[1] early scientific work on this: Lewis, et al. "Retrieval-augmented generation for knowledge-intensive nlp tasks." NeurIPS 2020.

[2] The file was generated using https://repo2txt.simplebasedomain.com, limited to docs, examples, tutorials, and source/header files in src.core/internet/network/flow-monitor.

[3a] https://chatgpt.com/share/6745fbaa-871c-8010-b4d8-4b894624e883

[3b] https://chatgpt.com/share/6745fb4c-09e8-8010-ae1c-d41b841df2a4

[4a] https://chatgpt.com/share/6745fbb9-6ff8-8010-83b8-1a1d69ea2ba4

[4b] https://chatgpt.com/share/6745fbc8-9710-8010-9538-63b8d7031516

Final notes:

• There may be legal aspects to keep in mind especially when forum data shall be included in the context.
• I actually intended to share a link to my custom GPT here, but OpenAI wouldn't let me do so unless I authorized as a "member of the corresponding org". Sorry!
• I am from an ML/Robotics background and only came into computer networking for my PhD research, so eval questions and assumptions may be imprecise.

[Tom Henderson]

Andreas, these are interesting ideas; I wonder whether some kind of
"HOWTO use RAG to improve your ns-3 LLM queries" might be useful to
maintain, starting with your document?

Another recently published direction to consider is to apply a filter on
top of LLM output to fix its errors: https://arxiv.org/pdf/2409.00676

I was also in a recent discussion in which the topic of trying to use
LLMs to generate first drafts of our missing documentation came up
(e.g., missing chapters in the Sphinx documentation). Of course,
maintainers would have to review/correct what the LLMs generate.

- Tom

On 11/26/24 09:29, Andreas Boltres wrote:
> *[2/2] /[will post this as an answer once the first half of the post is
> approved] /*

>
> I created a custom GPT with access to a single .txt context file [2] and
> custom answering instructions ("ns3_gpt_instructions.txt"). I queried it
> for two different made-up tasks related to packets and shared links to
> the chat:
>

> 1. /Measuring end-to-end delays of individual packets/ (custom GPT [3a]

> vs. default GPT 4o [3b])

> 2. /Retrieve the size of a packet's payload when not knowing which
> headers/trailers exist /(custom GPT [4a] vs. default GPT 4o [4b])