Today – Miltos Allamanis (Google Brain)

[Nadav Timor]

Hi all,

Today we welcome Miltos Allamanis from Google Brain to share his work done while he was at Microsoft Research and published on ICLR 2022 (title and abstract below).

Please register here to receive a link to the meeting (on Zoom).

Best,

Nadav Timor

https://bit.ly/LLMs-for-Code-Seminar

---

Speaker: Miltos Allamanis (Google Brain).

Title: Learning to Complete Code with Sketches.

Abstract: Code completion is usually cast as a language modelling problem, i.e., continuing an input in a left-to-right fashion. However, in practice, some parts of the completion (e.g., string literals) may be very hard to predict, whereas subsequent parts directly follow from the context. To handle this, we instead consider the scenario of generating code completions with “holes” inserted in places where a model is uncertain. We develop Grammformer, a Transformer-based model that guides the code generation by the programming language grammar, and compare it to a variety of more standard sequence models.

We train the models on code completion for C# and Python given partial code context. To evaluate models, we consider both ROUGE as well as a new metric RegexAcc that measures success of generating completions matching long outputs with as few holes as possible. In our experiments, Grammformer generates 10-50% more accurate completions compared to traditional generative models and 37-50% longer sketches compared to sketch-generating baselines trained with similar techniques.