Designing an LLM chat assistant for ns-3 users and developers

[Andreas Boltres]

[1/2]

TL;DR:

What if we created and exposed a custom LLM chat assistant that leverages ns-3 knowledge and implementation details to answer programming and modeling questions? Could it accelerate learning and adoption in beginner and intermediate users and thus make ns-3 a more powerful research tool? Could it lift off some of the load from the devs' and maintainers' shoulders of answering unnecessary questions on forums?

---

Hey there,

I've had a fever dream that I wanted to share with the developers and maintainers of this project to comment on and discuss the idea.

After using ns-3 for almost two years in my PhD project, I still find myself browsing the forums in vain for answers to my very specific questions, or hopping in between numerous header and source files in order to understand what is going on. The large amount of questions on the google forums, stackoverflow etc. as well as personal records of fellow PhD students tell me that this is a common experience, and I acknowledge that to some extent it may be unavoidable. Yet, for simple programming tasks and boilerplate code, I can now use Copilot or ChatGPT and a bit of critical thinking on my side. I wondered whether we could transfer the utility brought by LLMs to the ns-3 context by supplying context knowledge of ns-3's implementation, its documentation and, perhaps, Q&A-style information as available from forums like this one.

Using Retrieval-Augmented Generation (RAG) with LLM Chatbots by supplying additional knowledge and turning a conversation internally into a knowledge query has shown to be very effective [1]. As mentioned in the TLDR, in the best (=optimistic?) case, such a chat assistant could greatly increase the speed with which practitioners (and maybe even experts) get things done and running, and developers and maintainers may be faced with far less bad-quality questions if people first turned to an assistant that correctly answered most of the more trivial issues. Technically, creating such a chat assistant should be quite straight forward, as supplying a file containing context information is already easily doable, e.g. via OpenAI's custom GPT functionality (requires their subscription). I assume that you can do the same with open-sourced LLMs by using the variety of AI platforms and tools out there.

[...]

[Gabriel Ferreira]

Not a fever dream. Already in the works :)

https://www.arxiv.org/abs/2408.13781

[Andreas Boltres]

[...]

[2/2]

I created a custom GPT with access to a single .txt context file [2] and custom answering instructions ("ns3_gpt_instructions.txt"). I queried it for two different made-up tasks related to packets and shared links to the chat:

1. Measuring end-to-end delays of individual packets (custom GPT [3a] vs. default GPT 4o [3b])
2. Retrieve the size of a packet's payload when not knowing which headers/trailers exist (custom GPT [4a] vs. default GPT 4o [4b])

Results are, I guess, not satisfactory, as it's evident that some of ns-3's modeling has not been understood by GPT. Also, customizing GPT in the naive way I've done it does not seem to improve the answer quality. Of course, this was a minimal-effort attempt of an LLM amateur, and I did not invest time e.g. to craft vector embeddings of the context or professional instruction sets, nor to thoughtfully design the evaluation questions and interpret the answers in detail. With this experiment and this post in general, I simply wanted to put out an inspiration and starting point for a discussion. 

Cheers, Andreas

[1] early scientific work on this: Lewis, et al. "Retrieval-augmented generation for knowledge-intensive nlp tasks." NeurIPS 2020.

[2] The file was generated using https://repo2txt.simplebasedomain.com, limited to docs, examples, tutorials, and source/header files in src.core/internet/network/flow-monitor. That's because this started as a small experiment for myself, and these src modules are the ones of interest for me at the moment.

[3a] https://chatgpt.com/share/6745fbaa-871c-8010-b4d8-4b894624e883

[3b] https://chatgpt.com/share/6745fb4c-09e8-8010-ae1c-d41b841df2a4

[4a] https://chatgpt.com/share/6745fbb9-6ff8-8010-83b8-1a1d69ea2ba4

[4b] https://chatgpt.com/share/6745fbc8-9710-8010-9538-63b8d7031516

Final notes:

• There may be legal aspects to keep in mind especially when forum data shall be included in the context.
• I actually intended to share a link to my custom GPT here, but OpenAI wouldn't let me do so unless I authorized as a "member of the corresponding org". Sorry!
• I am from an ML/Robotics background and only came into computer networking for my PhD research, so eval questions and assumptions may be imprecise.

[Andreas Boltres]

Sorry for the double post - I realized only after splitting my post that my attempt to append the context file was making the post too long, not the post text itself. I am re-posting Tom's answer from the other thread below, you can delete my other post.

---

On Tuesday, 11/26/24 19:10, Tom Henderson wrote:

> Andreas, these are interesting ideas; I wonder whether some kind of
>"HOWTO use RAG to improve your ns-3 LLM queries" might be useful to
>maintain, starting with your document?
>
>Another recently published direction to consider is to apply a filter on
>top of LLM output to fix its errors: https://arxiv.org/pdf/2409.00676
>
>I was also in a recent discussion in which the topic of trying to use
>LLMs to generate first drafts of our missing documentation came up
>(e.g., missing chapters in the Sphinx documentation). Of course,
>maintainers would have to review/correct what the LLMs generate.
>
>- Tom
>
>On 11/26/24 09:29, Andreas Boltres wrote:
>> *[2/2] /[will post this as an answer once the first half of the post is
>> approved] /*

>>
>> I created a custom GPT with access to a single .txt context file [2] and
>> custom answering instructions ("ns3_gpt_instructions.txt"). I queried it
>> for two different made-up tasks related to packets and shared links to
>> the chat:
>>

>> 1. /Measuring end-to-end delays of individual packets/ (custom GPT [3a]

>> vs. default GPT 4o [3b])

>> 2. /Retrieve the size of a packet's payload when not knowing which
>> headers/trailers exist /(custom GPT [4a] vs. default GPT 4o [4b])