Executable file size and raco demod

[Dexter Lagan]

Hello there,

  Two things:

- I noticed a doubling of executable file sizes (from 30MB to 70MB for racket/gui with embedded libs, Windows) between Racket 7.9 (non-CS) and Racket 8.0 (CS). Because of this, startup times from network drives also doubled (from 5 to 10s for gui programs when using CS). I had to revert to 7.9 non-CS for now;

- Because of this, I have been trying to reduce file sizes to a minimum. I tried replacing racket/gui by a minimal list of requires to no avail. I tried using the raco demod function to demodularize, but it seems to be broken on recent version of Racket (anything beyond hello world will quit prematurely, racket/gui programs won't run at all). I tried GitHub - bluerider/flattener: Source Code Level Flattener for PLT Racket without success (seems broken as well). I also tried compressing executables with UPX, but it also breaks them.

  Does anybody know of a way to reduce final Racket executable file sizes / flatten / demodularize while keeping gui functionality ?

Dex

[Dexter Lagan]

  I updated to current again, and executable file size has nearly doubled again (120MB vs 70MB). I'd be curious to know if startup time wouldn't be affected by file IO at this point. I'm using 7.9 BC 32 bits in production atm, since it produces the smallest executables (12 MB!).

Dex

[Matthew Flatt]

Hi Dex,

Are you using a snapshot build from the Utah site --- as opposed to a
snapshot for Northwestern or some other build?

I see that the Utah site's compiled code is twice as big as the
Northwestern site's compiled code. It looks like the build process for
Racket at Utah (via Visual Studio) misconfigures the "should compiled
code be compressed?" flag, while the build process used at Northwestern
(via MinGW) configures that setting correctly. The distribution builds
are made in the same way as the Northwestern snapshots.

I'll fix the compilation path that the Utah snapshot uses, but it would
be good to know whether that could be the problem.

Thanks,
Matthew

At Sun, 4 Apr 2021 02:19:28 -0700 (PDT), Dexter Lagan wrote:
> I updated to current again, and executable file size has nearly doubled
> again (120MB vs 70MB). I'd be curious to know if startup time wouldn't be
> affected by file IO at this point. I'm using 7.9 BC 32 bits in production
> atm, since it produces the smallest executables (12 MB!).
>
> Dex
>
> On Wednesday, March 3, 2021 at 8:07:00 PM UTC+1 Dexter Lagan wrote:
>
> > Hello there,
> >
> > Two things:
> >
> > - I noticed a doubling of executable file sizes (from 30MB to 70MB for
> > racket/gui with embedded libs, Windows) between Racket 7.9 (non-CS) and
> > Racket 8.0 (CS). Because of this, startup times from network drives also
> > doubled (from 5 to 10s for gui programs when using CS). I had to revert to
> > 7.9 non-CS for now;
> >
> > - Because of this, I have been trying to reduce file sizes to a minimum. I
> > tried replacing racket/gui by a minimal list of requires to no avail. I
> > tried using the raco demod function to demodularize, but it seems to be
> > broken on recent version of Racket (anything beyond hello world will quit
> > prematurely, racket/gui programs won't run at all). I tried GitHub -
> > bluerider/flattener: Source Code Level Flattener for PLT Racket

> > <https://github.com/bluerider/flattener> without success (seems broken as

> > well). I also tried compressing executables with UPX, but it also breaks
> > them.
> >
> > Does anybody know of a way to reduce final Racket executable file sizes
> > / flatten / demodularize while keeping gui functionality ?
> >
> > Dex
> >
>

> --
> You received this message because you are subscribed to the Google Groups
> "Racket Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email
> to racket-users...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/racket-users/a2a14107-01fb-4f36-b6e1-c02498f35
> 7adn%40googlegroups.com.

[Dexter Lagan]

Hi Matthew,

  It is indeed the one from Utah. I’ll give the other one a try and report back. Thanks for looking into this!

Dex 

[Dexter Lagan]

Looks like it's the opposite. At the moment Utah's is half the size. I'll install the current Utah's and compare generated executables with 8.0 release.

Utah:

Northwestern:

[Dexter Lagan]

Upon opening DrRacket on 8.0.0.13, Utah build:

ptr-set!: cannot install value into non-atomic memory

  value: #<cpointer+offset>

  destination: #<cpointer>

  context...:

   C:\Program Files\Racket-8.0.0.13\collects\ffi\unsafe.rkt:1468:4: loop

   body of "C:\Program Files\Racket-8.0.0.13\share\pkgs\gui-lib\mred\private\gdi.rkt"

[Exited. Close box or Ctrl-C closes the console.]

Dex

[Dexter Lagan]

Windows 10

Snapshot: 20210404-725710e
   Windows
      64-bit x64

Dex

[Bruce O'Neel]

Hi,

I stumbled over that error too.  Apparently Mathew said it would be fixed in the next build.

cheers

bruce

To view this discussion on the web visit https://groups.google.com/d/msgid/racket-users/a2ad2c17-64f6-4ace-b9c2-531ad9f43f64n%40googlegroups.com.

[Matthew Flatt]

The archive size is misleading for two reasons:

* the Northwestern snapshots include a lot more overall content by
including tests, and

* compression on the installer counteracts the mistake where the
content of ".zo" files is not individually compressed; uncompressed
individual ".zo" files end up compressing better as a group.

A new build is ready just now, where DrRacket starts and the size is
changed (i.e., download size is moderately bigger, unpacked size is
much smaller).

> https://groups.google.com/d/msgid/racket-users/5e33921f-c751-40b5-89bc-fadc01bda
> 506n%40googlegroups.com.
>
>
> ------------------------------------------------------------------------------
> [image/png "Utah.png"] [~/Desktop & open] [~/Temp & open]
> .
>
>
> ------------------------------------------------------------------------------
> [image/png "North.png"] [~/Desktop & open] [~/Temp & open]
> .

[Dexter Lagan]

Fantastic. Thanks for the info and the new build! Can’t wait to give it a test drive.

Dex

---

From: Matthew Flatt <mfl...@cs.utah.edu>
Sent: Monday, April 5, 2021 3:18:17 PM
To: Dexter Lagan <dexte...@gmail.com>
Cc: Racket Users <racket...@googlegroups.com>
Subject: Re: [racket-users] Re: Executable file size and raco demod

 

[Dexter Lagan]

  I installed the latest build, and it does start. Generated executable is 80MB vs 32MB for 7.9 BC, about 10MB more than 8.0 release.

 I understand that there's a difference between bytecode and compiled binary size with CS, but I'm not certain if we're talking about the Racket distribution itself, or generated executables. Is there any hope to eventually get closer to BC's executable size with CS? The raco demod option seemed to do miracles on previous versions, but never worked on gracket / gui programs for me.

  Also, somehow touchpad scrolling has degraded since 7.9 BC (which was decent, and even had momentum). If I scroll downwards, then upwards, it keeps going down for another 1 second before switching direction, and momentum is no longer simulated. If I use the scrollbar, it scrolls fast and smooth, with no direction problem.

Happy Easter!

Dex

[Matthew Flatt]

At Mon, 5 Apr 2021 14:29:22 -0700 (PDT), Dexter Lagan wrote:
> I installed the latest build, and it does start. Generated executable is
> 80MB vs 32MB for 7.9 BC, about 10MB more than 8.0 release.

That difference is again related to the compilation paths, but this
time the likely result is that the v8.1 release will be larger in the
way you saw here.

In the cross-compilation path that the release builds use, there was an
unintended layer of compression. (I noticed the size difference before,
but had not yet investigated.) Checking that again, if the extra layer
is used, the 10% or so savings in file size causes a 5% decompression
slowdown for loading. Compressing just the code, meanwhile, makes load
times faster on machines that I tried. So, I think the current
(non-cross) default is still the best choice for most situations, and
I'll adjust cross-compilation to match.

We can make the extra layer of compression an option for someone who
needs to minimize file size. But a more relevant effect may be the
existing configure-time option to compress boot files...


Compressing embedded boot files would save 30 MB, which would make a
big difference in your case. Compressed boot files take 20-40ms longer
to load on a typical machine, which is significant relative to the
minimal load time, but it's very little time in many situations.

Boot-file compression was already an option for the `configure` way of
building on Unix (but the option had rotted; now fixed). I've updated
the MSVC build to recognize a `PLT_BOOTFILE_COMPRESS` environment
variable to enable boot-file compression when building that way. So, it
will be easier to build Racket with compressed boot files --- but you
would have to build Racket yourself.

It may be that compressed boot files make sense as the default on
Windows. I'm starting to lean in that direction, but I'm not sure, and
I haven't changed the default for now.

> I understand that there's a difference between bytecode and compiled
> binary size with CS, but I'm not certain if we're talking about the Racket
> distribution itself, or generated executables. Is there any hope to
> eventually get closer to BC's executable size with CS?

Compressing boot files brings the size of the Racket CS executable/DLL
itself closer to Racket BC --- more like a factor of 2 instead of a
factor of 10 --- and that turns out to be almost half of the size in
your case. But the more code you add in terms of embedded ".zo" files,
the more the size difference will approach a factor of 2 overall; I
don't see a route to big improvements there.

> The raco demod option seemed to do miracles on previous versions, but
> never worked on gracket / gui programs for me.

Restoring `raco demod` has been on my list, and I may get to that soon,
but probably not before v8.1. It may be possible to improve `raco
demod` to support `dynamic-require`d pieces like the platform-specific
backends in `racket/gui`.

> Also, somehow touchpad scrolling has degraded since 7.9 BC (which was
> decent, and even had momentum). If I scroll downwards, then upwards, it
> keeps going down for another 1 second before switching direction, and
> momentum is no longer simulated. If I use the scrollbar, it scrolls fast
> and smooth, with no direction problem.

I don't know what happened there. Version 8.0 included a change for
handling scroll-wheel events, but I don't think that's the same kind of
event as touchpad scrolling. You could try changing `(* wheel-scale
amt)` in "share/pkgs/gui-lib/mred/private/wx/win32/window.rkt" back to
just `amt` to make sure that change isn't the reason.

[Dexter Lagan]

  Thanks for the info, I'll try updating window.rkt and report back.

  What surprises me is that scrolling by keeping the mouse button down while hovering on the scrollbar is fast and smooth, while using a two-finger gesture is erratic and choppy (on top of the direction-change delay). I wonder if there would be a way to map the scrolling gesture (which might be bound to the mouse wheel handler), to the scrollbar instead. If one could do this, the scrolling would smooth no matter which input device is used.

  When it comes to executable file size, I understand that there's a balance between file loading time and decompression time.

Intuitively, for local machines - especially with today's NVMe's - it would make sense to generate larger files, with less compression for best overall performance. However, as soon as one runs an executable file from a network drive (say, over 100Mbits ethernet), file size might be directly correlated to startup time, accounting for network transfer time. I don't know if these ideas are in consideration, and I understand you have other priorities. I'm available to help with benchmarks, debugging and any other task you might find useful.

  As always, I greatly appreciate you taking the time to answer with such detail and care.

Dex

[Dexter Lagan]

  I updated window.rkt, and it fixed the scrolling problem. It's exactly as it was with 7.9. Thanks!

One last thing, I noticed that a small window appears top-left corner of the screen before the full DrRacket window opens since 8.0.0.1x :

I had trouble capturing it, as it only appears a few seconds.

Cheers,

Dex

[Dexter Lagan]

  Also, I caught a seg-fault yesterday, after having left a DrRacket 7.9 BC open for a couple of days on a basic servlet. I'll see if happens again with 8.0.0.x while I use it.

Dex

[Philip McGrath]

On Tue, Apr 6, 2021 at 7:54 PM Matthew Flatt <mfl...@cs.utah.edu> wrote:

In the cross-compilation path that the release builds use, there was an
unintended layer of compression. (I noticed the size difference before,
but had not yet investigated.) Checking that again, if the extra layer
is used, the 10% or so savings in file size causes a 5% decompression
slowdown for loading. Compressing just the code, meanwhile, makes load
times faster on machines that I tried. So, I think the current
(non-cross) default is still the best choice for most situations, and
I'll adjust cross-compilation to match.

…

Compressing embedded boot files would save 30 MB, which would make a
big difference in your case. Compressed boot files take 20-40ms longer
to load on a typical machine, which is significant relative to the
minimal load time, but it's very little time in many situations.

I don't have a problem with the current setup, but I've noticed that other projects (e.g. OpenZFS) have been adding support for Zstd compression, which has an especially wide range of options for trading off compression ratio and (de)compression speed. I don't know if it would be an improvement here—apparently it is both faster and more efficient than gzip but doesn't quite displace LZ4 from its core niche—but it's another direction that could be explored.

-Philip

[Matthew Flatt]

At Wed, 7 Apr 2021 14:08:00 -0700 (PDT), Dexter Lagan wrote:
> One last thing, I noticed that a small window appears top-left corner of
> the screen before the full DrRacket window opens since 8.0.0.1x :

That window's full title turns out to be "Recompiling quickscripts...".
It's shown and hidden too quickly to see on my machine (probably
between GUI sync points).

[Dexter Lagan]

FYI I ended up going to 8.0 BC 32 bits to get the smallest executables for production, while still evaluating current.

I partially fixed my scrolling problem by replacing   

(* wheel-scale amt)

by

(/ (* wheel-scale amt) 2)

in share/pkgs/gui-lib/mred/private/wx/win32/window.rkt.

  There still is a small delay in the direction change while scrolling, but it isn't nearly as objectionnable as it was with * amt.

Cheers,

Dex

[Dexter Lagan]

Hi again,

  After playing around with the gen-wheels procedure and comparing its code with the WM_HSCROLL handling, I was able to get fast, accurate and smooth scrolling for both mouse and touchpad by switching the wheel-steps-mode to 'integer and reducing WHEEL_DELTA by a factor of 4. Here's the updated code :

<pre>

  (define wheel-steps-mode 'integer)

  (define/public (get-wheel-steps-mode) wheel-steps-mode)

  (define/public (set-wheel-steps-mode mode) (set! wheel-steps-mode mode))

  (define/private (gen-wheels w msg lParam amt down up)

    (define AUG_WHEEL_DELTA (/ WHEEL_DELTA 4))

    (let loop ([amt amt])

      (cond

        [((abs amt) . < . AUG_WHEEL_DELTA)

         (case wheel-steps-mode

           [(one integer) amt]

           [(fraction)

            (unless (zero? amt)

              (do-key w msg down lParam #f #f void (/ amt (exact->inexact AUG_WHEEL_DELTA))))

            0.0])]

        [(negative? amt)

         (case wheel-steps-mode

           [(one)

            (do-key w msg down lParam #f #f void 1.0)

            (loop (+ amt AUG_WHEEL_DELTA))]

           [(integer)

            (define steps (quotient (- amt) AUG_WHEEL_DELTA))

            (do-key w msg down lParam #f #f void (exact->inexact steps))

            (loop (+ amt (* steps AUG_WHEEL_DELTA)))]

           [else

            (do-key w msg down lParam #f #f void (/ (- amt) (exact->inexact AUG_WHEEL_DELTA)))

            0.0])]

        [else

         (case wheel-steps-mode

           [(one)

            (do-key w msg up lParam #f #f void 1.0)

            (loop (- amt AUG_WHEEL_DELTA))]

           [(integer)

            (define steps (quotient amt AUG_WHEEL_DELTA))

            (do-key w msg up lParam #f #f void (exact->inexact steps))

            (loop (- amt (* steps AUG_WHEEL_DELTA)))]

           [else

            (do-key w msg up lParam #f #f void (/ amt (exact->inexact AUG_WHEEL_DELTA)))

            0.0])])))

</pre>

On Wednesday, April 7, 2021 at 1:54:24 AM UTC+2 Matthew Flatt wrote:

[Dexter Lagan]

  One precision: this work for Racket 8.0 BC release. There might have been changes made to current.

DrRacket feels as smooth as Sublime Text with this change, a completely different experience. I'd like to make a proper commit if somebody held my hand to do so.

Dex

[Dexter Lagan]

  So in conclusion, only two changes required:

In const.rkt:

(define WHEEL_DELTA                     120)

to

(define WHEEL_DELTA                     30)

In Window.rkt:

(define wheel-steps-mode 'one)

to

(define wheel-steps-mode 'integer)

  The change to const.rkt may affect a lot more code, I'll let you decide if this change should be left local to gen-wheels.

It fixes the direction change delay, as well as slow/erratic scrolling on the Thinkpad's trackpoint.

Dex

[Matthew Flatt]

Are you scrolling in DrRacket's editor or in some other application?

I'm surprised that change to the initial value of `wheel-steps-mode`
has an effect, but maybe my confusion is that I'm thinking of DrRacket.
An editor canvas calls `set-wheel-steps-mode` in its constructor to set
the mode to 'integer, so scrolling in DrRacket is using 'integer mode.
But the default of 'one is meant to keep scrolling behavior the way it
used to be for other windows, including a plain canvas, unless a
program requests otherwise by calling `wheel-event-mode` on the window.

The definition of `WHEEL_DELTA` shouldn't change, since that's a
Windows constant. It's interesting that reducing the scaling of wheel
steps makes scrolling behave better for you, though, and I'm not clear
on why.

> --
> You received this message because you are subscribed to the Google Groups
> "Racket Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email
> to racket-users...@googlegroups.com.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/racket-users/49cf730d-566d-4ee6-ad0e-355b6ad86
> e6dn%40googlegroups.com.

[Matthew Flatt]

At Sat, 10 Apr 2021 03:52:47 -0700 (PDT), Dexter Lagan wrote:
> DrRacket feels

I see that you did say DrRacket, so I remain puzzled. Are you seeing a
value other than 'integer when `gen-wheels` is called in a stock v8.0?


Matthew

[Dexter Lagan]

  By default it’s set to ‘one. When set to ‘one, it’s having trouble detecting direction switch (up/down), especially when using touchpad or track point. The mouse wheel also feels sluggish. Once set to ‘integer, direction switches correctly and accurately, but scrolling is very slow on all input devices. Reducing the delta fixes this and makes it scroll identically to a browser or other editors. Momentum works as well. Considering there’s win32 in une path, is this limited to Windows?

Dex

---

From: Matthew Flatt <mfl...@cs.utah.edu>
Sent: Saturday, April 10, 2021 7:22:40 PM

To: Dexter Lagan <dexte...@gmail.com>
Cc: Racket Users <racket...@googlegroups.com>
Subject: Re: [racket-users] Re: Executable file size and raco demod

[Dexter Lagan]

Yes I’m scrolling in DrRacket. Somehow the scrolling behaviour is different between using the scroll bar, and the mouse wheel or touchpad. I think it’s using a different behaviour on these input devices. I could not find anything in the Windows event defs however.

Dex

---

From: Matthew Flatt <mfl...@cs.utah.edu>
Sent: Saturday, April 10, 2021 7:13:34 PM

To: Dexter Lagan <dexte...@gmail.com>
Cc: Racket Users <racket...@googlegroups.com>
Subject: Re: [racket-users] Re: Executable file size and raco demod

[Matthew Flatt]

At Sat, 10 Apr 2021 17:26:21 +0000, Dexter Lagan wrote:
> By default it’s set to ‘one.

But for an editor, this line changes it to 'integer:

https://github.com/racket/gui/blob/master/gui-lib/mred/private/wxme/editor-canvas.rkt#L237

Or, at least, it should --- and it does in my installation, which I
checked by logging inside `gen-wheels`. That's why I'm confused that
changing the default would do anything.

[Dexter Lagan]

  I'll study all this and get back to you.

Reminds me of my win32 API days, making event loops...

Dex