Dynamic tracing analyzers in OpenResty XRay automated through Claude 3 Opus

[Aisse Tejas]

Hello all!

When we develop new dynamic tracing analyzers in OpenResty XRay, we usually start by directly copying the C/C++ code from the target programs into our Y/Y++ code, such as CPython, JVM, PHP, etc. Generally, we still need to make minor edits to minimize copying excessive runtime dependency code. Now, even these last bits of manual editing may be automated through Claude 3 Opus.

Just yesterday, we used Claude 3 Opus to rewrite a large C function, and surprisingly, the resulting code was nearly production-quality. The diff patches compared to the original code looked perfect—we were impressed. In contrast, when we assigned the same task to ChatGPT’s GPT-4, the results were disappointing, with some obvious errors. It felt like comparing a professional’s work to that of an elementary school student.

We also saw a post on Reddit where someone mentioned using Claude 3 Opus to refactor a large software project with many files, and he found it significantly better than GPT-4. He commented that ChatGPT often imagines things that don’t exist.

What do you think? Your opinion is always welcome :)