PUP-11232 added # frozen_string_literal: true to some high memory consumption files but even after that, lib/puppet/pops/parser/lexer2.rb still seems to have the most memory allocated.
Proposed investigation direction as stated by Josh Cooper:
I'm guessing the lexer2 allocations are due to the closure scope bound to each lambda? I'm thinking Ruby may be doing more work than is necessary to create the Binding as each token is scanned and the corresponding lambda is called? Or perhaps it's the byteslice calls?
This is due to a ruby bug https://bugs.ruby-lang.org/issues/16906. In MRI, our ThreadLocalSingleton doesn't behave like a singleton, so we create N copies of Puppet::Pops::Parser::Lexer2.
One workaround is to avoid using Thread#thread_variable?