Performance mystery with largish heaps

[Philip Zeyliger]

Hi!

I'm debugging a bit of a performance mystery. In short, piping 1GB from a file to /dev/null is significantly faster when the heap is empty and significantly slower when there's a large object on the heap. I've reproduced this at https://gist.github.com/philz/3e55a1a3377797d1fbc47b10a219ec6f . I've tried node 8 and node 12, on both Mac and Linux. As far as I can tell, based on using "--trace_gc" as well as "perf record" (& friends), Node/V8 are choosing to do a lot more Mark & Sweep collections (as opposed to "Scavenge") in the runs where the heap is more utilized.

I stumbled upon this while debugging a similar-looking problem with S3 upload performance, where a hot path seems to be NodeBIO::TryAllocateForWrite() calling into v8::internal::IncrementalMarking::AdvanceWithDeadline, but there are more variables there.

Many thanks for any insights y'all may have.

-- Philip

$~/node/env-8.15.0/bin//node slow.js
Created random.1gb.bin
Writing to dev null took (ms):  563
Writing to dev null took (ms):  451
Writing to dev null took (ms):  451
Writing to dev null took (ms):  463
Writing to dev null took (ms):  450
[ '/Users/philip/node/env-8.15.0/bin/node',
  '/Users/philip/src/scratch/slow.js' ]
{ total_heap_size: 11354112,
  total_heap_size_executable: 3670016,
  total_physical_size: 9530776,
  total_available_size: 1490917424,
  used_heap_size: 5765264,
  heap_size_limit: 1501560832,
  malloced_memory: 8192,
  peak_malloced_memory: 2890352,
  does_zap_garbage: 0 }
{ leaves: 10000, depth: 2 }
{ total_heap_size: 951926784,
  total_heap_size_executable: 3670016,
  total_physical_size: 848844440,
  total_available_size: 653356752,
  used_heap_size: 812714040,
  heap_size_limit: 1501560832,
  malloced_memory: 8192,
  peak_malloced_memory: 2890352,
  does_zap_garbage: 0 }
Writing to dev null took (ms):  3493
Writing to dev null took (ms):  3650
Writing to dev null took (ms):  3712
Writing to dev null took (ms):  3620
Writing to dev null took (ms):  3494