Jira (PUP-11236) Reduce allocated memory in `lib/puppet/pops/parser/lexer2.rb`

Luchian Nemes (Jira)

unread,

Sep 1, 2021, 8:52:03 AM9/1/21

to puppe...@googlegroups.com

Luchian Nemes created an issue

Puppet /

PUP-11236

Reduce allocated memory in `lib/puppet/pops/parser/lexer2.rb`

Issue Type:	Bug
Assignee:	Unassigned
Created:	2021/09/01 5:51 AM
Priority:	Normal
Reporter:	Luchian Nemes

PUP-11232 added # frozen_string_literal: true to some high memory consumption files but even after that, lib/puppet/pops/parser/lexer2.rb still seems to have the most memory allocated.

Proposed investigation direction as stated by Josh Cooper:

I'm guessing the lexer2 allocations are due to the closure scope bound to each lambda? I'm thinking Ruby may be doing more work than is necessary to create the Binding as each token is scanned and the corresponding lambda is called? Or perhaps it's the byteslice calls?

Add Comment

This message was sent by Atlassian Jira (v8.13.2#813002-sha1:c495a97)

Luchian Nemes (Jira)

unread,

Sep 2, 2021, 3:07:03 AM9/2/21

to puppe...@googlegroups.com

Luchian Nemes updated an issue

Puppet /

PUP-11236

Reduce allocated memory in `lib/puppet/pops/parser/lexer2.rb`

Change By:	Luchian Nemes
Attachment:	mem_profile_15614.txt

Add Comment

Ciprian Badescu (Jira)

unread,

Sep 20, 2021, 10:21:02 AM9/20/21

to puppe...@googlegroups.com

Ciprian Badescu updated an issue

Puppet /

PUP-11236

Reduce allocated memory in `lib/puppet/pops/parser/lexer2.rb`

Change By:	Ciprian Badescu
Epic Link:	PUP-11073

Add Comment

Ciprian Badescu (Jira)

unread,

Sep 27, 2021, 9:53:02 AM9/27/21

to puppe...@googlegroups.com

Ciprian Badescu updated an issue

Puppet /

PUP-11236

Reduce allocated memory in `lib/puppet/pops/parser/lexer2.rb`

Change By:	Ciprian Badescu
Sprint:	NW - 2021-10-20

Add Comment

Ciprian Badescu (Jira)

unread,

Oct 6, 2021, 4:10:03 AM10/6/21

to puppe...@googlegroups.com

Ciprian Badescu updated an issue

Puppet /

PUP-11236

Reduce allocated memory in `lib/puppet/pops/parser/lexer2.rb`

Change By:	Ciprian Badescu
Story Points:	3

Add Comment

Josh Cooper (Jira)

unread,

Oct 15, 2021, 3:24:02 PM10/15/21

to puppe...@googlegroups.com

Josh Cooper commented on

PUP-11236

Re: Reduce allocated memory in `lib/puppet/pops/parser/lexer2.rb`

This is due to a ruby bug https://bugs.ruby-lang.org/issues/16906. In MRI, our ThreadLocalSingleton doesn't behave like a singleton, so we create N copies of Puppet::Pops::Parser::Lexer2.

One workaround is to avoid using Thread#thread_variable?