ChatGPT

[micpool]

I think we have generally accepted that the ability of chatGPT to write Applescripts,for QLab is very limited and often counterproductive.

However, ChatGPT has the ability to produce custom chatbots, so I asked this question

How do I create a custom chatbot that is trained on specialised material specifically to answer questions accurately and without hallucinations about QLab including QLab AppleScript.

I won’t paste the full reply here as you can use this prompt as a starting point and refine further but chatGPT does seem to understand the problem and states quite useful objectives. It begins with:

Great question — and given your experience with very confident, very wrong QLab answers, this is exactly the right problem to solve.

Below is the best-practice, battle-tested way to build a chatbot that answers accurately, conservatively, and honestly about QLab and QLab AppleScript, with minimal hallucination.

This is the same approach used in production knowledge assistants.

The Correct Approach (TL;DR)

Do NOT fine-tune a model.

Use Retrieval-Augmented Generation (RAG) + strict prompting + curated QLab docs.

This ensures the bot:

• Uses real QLab properties only
• Says “not supported” when appropriate
• Doesn’t invent AppleScript syntax
• Can be updated as QLab changes

and the end of the reply is this.

What This Gets You

✔ Accurate AppleScript

✔ Honest limitations

✔ No invented properties

✔ Answers that feel like a senior QLab programmer

✔ Easily updatable as QLab evolves

Want help with the 

actual build

?

I can:

• Design the exact document structure
• Provide a minimal Python RAG example
• Harden your Custom GPT even further
• Help you curate a QLab-only knowledge corpus

Say the word and tell me how hands-on you want to be.

This sort of gives the impression that with a few hours work it would be possible for someone with access to large quantities of information about QLab (e.g Figure53) to be assisted by chatGPT to develop a QLab specific custom chatbot, that among other things could write accurate and useful QLab scripts.

Thoughts?

Mic

[Chris Ashworth]

Good morning Mic,

I’m also curious about this question. We have spent much of this month updating our knowledge about the current state of LLMs as a programming tool. (They have come a long way since we looked in to them a year ago.) In that process I had started to think about how we could provide documentation and other information to LLMs to make them better at helping people use QLab.

I’ve created a tracking issue for this idea and added your thoughts. 

While I think your suggestion would mostly involve feeding an LLM documentation, I suppose there’s a possibility it could benefit from seeing code as well. We haven’t yet exposed QLab code to any LLM directly, and are currently evaluating which ones we “trust” enough to do that. We’re also writing down our company expectations and boundaries around using LLMs as a whole. (e.g. Using an LLM to help someone debug existing code or to understand new code is a different risk profile than asking an LLM to write new code.) It’s a rapidly shifting landscape, which I now feel forced to navigate!

We’re currently heavily leaning toward Claude Code as our tool of choice, fwiw, but I will check out custom ChatGPT chat bots too, since that is a tool that end users seem much more likely to use.

best,

Chris

[Paul]

There is certainly some scope in using AI coding and I've seen a couple of articles that "steering" the model could be key skill - saving time for experienced developers to get more done (so potentially for F53 devs in core QL development). The model doesn't understand what it doesn't know, so you need to have that experience of reviewing, testing and debugging code. 

The other Ars article comes to a similar conclusion - that devs won't be typing code but reviewing and correcting it. Not sure how that would play out in a scripting setting.

The other point worth considering is that Claude requires a subscription and while is currently modest (the first article above used £300 worth of credit) that may change if the current AI  "investment bubble" was to burst. so that may change in future and technicians/designers working in cash strapped theatre might not want even modest additional outlay.

[m...@stevensokulski.com]

RAG is incredibly powerful for this sort of thing. Hallucination is still possible, but at least the LLM starts with the right data.

Google’s NotebookLM is the easiest way to do this sort of thing. You can give it the QLab manual, QLab AppleScript dictionary, and the Apple documentation for AppleScript and probably be in a pretty great spot.