Jira (PUP-11073) Reduce memory usage

[productboard (Jira)]

productboard updated an issue

Puppet / PUP-11073 Reduce memory usage Change By: productboard productboard URL: https://puppet.productboard.com/feature-board/planning/features/8160319 Summary: Reduce Memory Usage memory usage Puppet was written before frozen string literals were added in Ruby 2.3, so by default puppet creates many transient string objects during compilation, which must be garbage collected. We can't simply switch all source files over to using {{frozen_string_literal: true}} because we sometimes create a string in one module/class/file, and the string is mutated somewhere unrelated. So we have to be selective about adding the pragma or calling {{String#freeze}}. The following are prime candidates: {noformat} lib/puppet/concurrent/thread_local_singleton.rb lib/puppet/parser/scope.rb lib/puppet/parser/templatewrapper.rb lib/puppet/pops/model/factory.rb lib/puppet/pops/model/ast.rb lib/puppet/pops/parser/lexer2.rb lib/puppet/pops/parser/slurp_support.rb lib/puppet/pops/types/type_formatter.rb lib/puppet/resource/type_collection.rb lib/puppet/util/tagging.rb {noformat} Add Comment

This message was sent by Atlassian Jira (v8.13.2#813002-sha1:c495a97)

[Beth Glenfield (Jira)]

Beth Glenfield updated an issue

Puppet / PUP-11073 Reduce memory usage

Change By: Beth Glenfield

|Puppet was written before frozen string literals were added in Ruby 2.3, so by default puppet creates many transient string objects during compilation, which must be garbage collected. We can't simply switch all source files over to using \{{frozen_string_literal: true}} because we sometimes create a string in one module/class/file, and the string is mutated somewhere unrelated. So we have to be selective about adding the pragma or calling \{{String#freeze}}.   The following are prime candidates:   {noformat} lib/puppet/concurrent/thread_local_singleton.rb lib/puppet/parser/scope.rb lib/puppet/parser/templatewrapper.rb lib/puppet/pops/model/factory.rb lib/puppet/pops/model/ast.rb lib/puppet/pops/parser/lexer2.rb lib/puppet/pops/parser/slurp_support.rb lib/puppet/pops/types/type_formatter.rb lib/puppet/resource/type_collection.rb lib/puppet/util/tagging.rb {noformat}| Add Comment

[Josh Cooper (Jira)]

Josh Cooper commented on PUP-11073

Re: Reduce memory usage I produced the list above by running the full_catalog benchmark: $ cd ~/work/puppet $ rbenv shell 2.5.8 $ git submodule update --init --recursive $ bundle install --with development $ bundle exec rake benchmark:full_catalog:memory_profile ... Generated memory profile: /home/josh/work/puppet/mem_profile_157882 which shows about 46MB - 3MB = 43MB of string allocations. Note the benchmark uses a very small number of facts: https://github.com/puppetlabs/puppet/blob/a68574e063d74eda66d7e26ad493ad8372a3803d/benchmarks/full_catalog/benchmarker.rb#L40-L75, so memory allocations for real catalog compilations are much worse. $ cat /home/josh/work/puppet/mem_profile_157882 ... allocated memory by class ----------------------------------- 46541290 String ... retained memory by class ----------------------------------- 3179506 String where objects are allocated in: allocated objects by location ----------------------------------- 102870 /home/josh/work/puppet/lib/puppet/pops/parser/lexer2.rb:531 99060 /home/josh/work/puppet/lib/puppet/pops/parser/lexer2.rb:554 56909 /home/josh/work/puppet/lib/puppet/pops/parser/lexer2.rb:708 53340 /home/josh/work/puppet/lib/puppet/pops/parser/lexer2.rb:529 53340 /home/josh/work/puppet/lib/puppet/pops/parser/lexer2.rb:530 53340 /home/josh/work/puppet/lib/puppet/pops/parser/lexer2.rb:552 49530 /home/josh/work/puppet/lib/puppet/pops/parser/lexer2.rb:553 41910 /home/josh/work/puppet/lib/puppet/pops/parser/lexer2.rb:496 38100 /home/josh/work/puppet/lib/puppet/pops/parser/lexer2.rb:497 36270 /home/josh/work/puppet/lib/puppet/pops/parser/lexer2.rb:711 33372 /home/josh/work/puppet/lib/puppet/resource.rb:651 32169 /home/josh/work/puppet/lib/puppet/concurrent/thread_local_singleton.rb:5 30036 /home/josh/work/puppet/lib/puppet/pops/parser/lexer2.rb:737 29592 /home/josh/work/puppet/lib/puppet/pops/parser/lexer2.rb:730 27831 /home/josh/work/puppet/lib/puppet/parser/scope.rb:497 26670 /home/josh/work/puppet/lib/puppet/pops/parser/lexer2.rb:494 26350 /home/josh/work/puppet/lib/puppet/pops/model/factory.rb:84 25549 /home/josh/.rbenv/versions/2.5.8/lib/ruby/2.5.0/pathname.rb:42 25545 /home/josh/work/puppet/lib/puppet/resource.rb:306 23696 /home/josh/.rbenv/versions/2.5.8/lib/ruby/2.5.0/pathname.rb:46 18706 /home/josh/work/puppet/lib/puppet/parser/templatewrapper.rb:85 16540 /home/josh/work/puppet/lib/puppet/parser/resource.rb:221 16453 /home/josh/work/puppet/lib/puppet/resource.rb:418 15260 /home/josh/work/puppet/lib/puppet/resource.rb:687 14454 /home/josh/work/puppet/lib/puppet/pops/model/factory.rb:538 14192 /home/josh/work/puppet/lib/puppet/type.rb:442 13938 /home/josh/work/puppet/lib/puppet/pops/model/factory.rb:779 13633 /home/josh/work/puppet/lib/puppet/resource.rb:642 12618 /home/josh/work/puppet/lib/puppet/pops/evaluator/runtime3_support.rb:321 11376 /home/josh/work/puppet/lib/puppet/resource/catalog.rb:169 11376 /home/josh/work/puppet/lib/puppet/resource/catalog.rb:92 11296 /home/josh/work/puppet/lib/puppet/pops/types/type_formatter.rb:25 10723 /home/josh/work/puppet/lib/puppet/concurrent/thread_local_singleton.rb:8 10615 /home/josh/work/puppet/lib/puppet/pops/visitor.rb:46 10580 /home/josh/.rbenv/versions/2.5.8/lib/ruby/2.5.0/ripper/sexp.rb:33 10333 /home/josh/work/puppet/lib/puppet/resource/type_collection.rb:232 9878 /home/josh/work/puppet/lib/puppet/pops/model/factory.rb:37 9780 /home/josh/work/puppet/lib/puppet/pops/parser/lexer2.rb:458 9756 /home/josh/.rbenv/versions/2.5.8/lib/ruby/2.5.0/erb.rb:873 9521 /home/josh/work/puppet/lib/puppet/pops/types/type_formatter.rb:652 9426 /home/josh/work/puppet/lib/puppet/pops/loader/typed_name.rb:19 9398 /home/josh/.rbenv/versions/2.5.8/lib/ruby/2.5.0/ripper/sexp.rb:106 9353 /home/josh/work/puppet/lib/puppet/parser/templatewrapper.rb:86 9278 /home/josh/work/puppet/lib/puppet/pops/loader/typed_name.rb:29 8982 /home/josh/work/puppet/lib/puppet/pops/visitor.rb:23 8711 /home/josh/work/puppet/lib/puppet/pops/parser/locator.rb:78 8627 /home/josh/work/puppet/lib/puppet/util/tagging.rb:113 8076 /home/josh/work/puppet/lib/puppet/parser/resource.rb:208 7961 /home/josh/work/puppet/lib/puppet/parser/scope.rb:435 7928 /home/josh/work/puppet/lib/puppet/type/file.rb:47 Also the "Allocated String Report" shows where unnecessary allocations are made: Allocated String Report ----------------------------------- 40454 "::" 27831 /home/josh/work/puppet/lib/puppet/parser/scope.rb:497 ... 29980 " " 19966 /home/josh/work/puppet/lib/puppet/pops/parser/lexer2.rb:708 .. 24715 "" 5612 /home/josh/work/puppet/lib/puppet/resource.rb:642 2112 /home/josh/work/puppet/lib/puppet/pops/puppet_stack.rb:33 ... 13639 "_" 9353 /home/josh/work/puppet/lib/puppet/parser/templatewrapper.rb:85 3810 /home/josh/work/puppet/lib/puppet/pops/parser/lexer2.rb:530 ... 10723 ".singleton" 10723 /home/josh/work/puppet/lib/puppet/concurrent/thread_local_singleton.rb:5 ...

Add Comment

[Josh Cooper (Jira)]

Josh Cooper updated an issue

Puppet / PUP-11073 Reduce memory usage

Change By: Josh Cooper Team/s: Coremunity,Night's Watch

Add Comment

[Josh Cooper (Jira)]

Josh Cooper updated an issue

Puppet / PUP-11073 Reduce memory usage

Change By: Josh Cooper Fix Version/s: PUP 7.13.0

Add Comment

This message was sent by Atlassian Jira (v8.20.2#820002-sha1:829506d)