Improving the compilation process - ideas

[Michał Kozakiewicz]

Hello!

1. I think it would be helpful to add an automatic removal of comments. Because the current result files contains them.

src/module/os.wren.inc

// Generated automatically from src/module/os.wren. Do not edit.
static const char* osModuleSource =
"class Platform {\n"
"  foreign static isPosix\n"
"  foreign static name\n"
"\n"
"  static isWindows { name == \"Windows\" }\n"
"}\n"
"\n"
"class Process {\n"
"  // TODO: This will need to be smarter when wren supports CLI options.\n"
"  static arguments { allArguments[2..-1] }\n"
"\n"
"  foreign static allArguments\n"
"}\n";

2. Is it possible to first compile the script into byte-code and then add it to the result program? The result program will take up less space and will be more difficult to decompile.

3. I think that at the end of the compilation process you can use the strip program. It removes unneeded objects. see: https://en.wikipedia.org/wiki/Strip_(Unix)

[Michel Hermier]

Le lun. 9 juil. 2018 à 03:09, Michał Kozakiewicz <miich...@gmail.com> a écrit :

Hello!

1. I think it would be helpful to add an automatic removal of comments. Because the current result files contains them.

src/module/os.wren.inc

// Generated automatically from src/module/os.wren. Do not edit.
static const char* osModuleSource =
"class Platform {\n"
"  foreign static isPosix\n"
"  foreign static name\n"
"\n"
"  static isWindows { name == \"Windows\" }\n"
"}\n"
"\n"
"class Process {\n"
"  // TODO: This will need to be smarter when wren supports CLI options.\n"
"  static arguments { allArguments[2..-1] }\n"
"\n"
"  foreign static allArguments\n"
"}\n";

Not really since you would end up with wrong line/col count in the stacktrace.

2. Is it possible to first compile the script into byte-code and then add it to the result program? The result program will take up less space and will be more difficult to decompile.

No, not yet. And wrong it would not really take less space, since the script stacktrace will need a full version of the source anyways.

And since source is required (at least in current state) it would not make decompilation harder.

3. I think that at the end of the compilation process you can use the strip program. It removes unneeded objects. see: https://en.wikipedia.org/wiki/Strip_(Unix)

Yes and no. Wren also run on Windows where strip is not available. Anyways considering wren size and the effect of strip, it would not be beneficial unless you are really on a memory stressed system, where you would have other problems running wren.

What are your intensions/aims at trying to make the code smaller? Someone tryed to embedded wren on a microcontroller (if I remember well) so these questions have been discussed somehow.

[Michał Kozakiewicz]

W dniu poniedziałek, 9 lipca 2018 09:06:35 UTC+2 użytkownik Michel Hermier napisał:

Le lun. 9 juil. 2018 à 03:09, Michał Kozakiewicz <miich...@gmail.com> a écrit :

Hello!

1. I think it would be helpful to add an automatic removal of comments. Because the current result files contains them.

src/module/os.wren.inc

// Generated automatically from src/module/os.wren. Do not edit.
static const char* osModuleSource =
"class Platform {\n"
"  foreign static isPosix\n"
"  foreign static name\n"
"\n"
"  static isWindows { name == \"Windows\" }\n"
"}\n"
"\n"
"class Process {\n"
"  // TODO: This will need to be smarter when wren supports CLI options.\n"
"  static arguments { allArguments[2..-1] }\n"
"\n"
"  foreign static allArguments\n"
"}\n";

Not really since you would end up with wrong line/col count in the stacktrace.

You're right, but I think we can do it and protect against this problem.

2. Is it possible to first compile the script into byte-code and then add it to the result program? The result program will take up less space and will be more difficult to decompile.

No, not yet. And wrong it would not really take less space, since the script stacktrace will need a full version of the source anyways.

And since source is required (at least in current state) it would not make decompilation harder.

 

If the language needs an uncompiled script in the executable file, there is no way to do it.

3. I think that at the end of the compilation process you can use the strip program. It removes unneeded objects. see: https://en.wikipedia.org/wiki/Strip_(Unix)

Yes and no. Wren also run on Windows where strip is not available. Anyways considering wren size and the effect of strip, it would not be beneficial unless you are really on a memory stressed system, where you would have other problems running wren.

We can run the strip program when it be available. I never noticed that it would work negatively.

What are your intensions/aims at trying to make the code smaller? Someone tryed to embedded wren on a microcontroller (if I remember well) so these questions have been discussed somehow.

I do not try to use the Wren language on the microcontroller. I just know that the smaller output program is better.

[Michel Hermier]

Le lun. 9 juil. 2018 à 15:35, Michał Kozakiewicz <miich...@gmail.com> a écrit :

W dniu poniedziałek, 9 lipca 2018 09:06:35 UTC+2 użytkownik Michel Hermier napisał:

Le lun. 9 juil. 2018 à 03:09, Michał Kozakiewicz <miich...@gmail.com> a écrit :

Hello!

1. I think it would be helpful to add an automatic removal of comments. Because the current result files contains them.

src/module/os.wren.inc

// Generated automatically from src/module/os.wren. Do not edit.
static const char* osModuleSource =
"class Platform {\n"
"  foreign static isPosix\n"
"  foreign static name\n"
"\n"
"  static isWindows { name == \"Windows\" }\n"
"}\n"
"\n"
"class Process {\n"
"  // TODO: This will need to be smarter when wren supports CLI options.\n"
"  static arguments { allArguments[2..-1] }\n"
"\n"
"  foreign static allArguments\n"
"}\n";

Not really since you would end up with wrong line/col count in the stacktrace.

You're right, but I think we can do it and protect against this problem.

Yes we can, at the cost of quite a lot of complexity.

2. Is it possible to first compile the script into byte-code and then add it to the result program? The result program will take up less space and will be more difficult to decompile.

No, not yet. And wrong it would not really take less space, since the script stacktrace will need a full version of the source anyways.

And since source is required (at least in current state) it would not make decompilation harder.

 

If the language needs an uncompiled script in the executable file, there is no way to do it.

There is always a way, the problem don't really come from the language, but from the implementation choices.

We can separate compilation from runtime, every huge language does that. The problems comes from designing the runtime blob, that add an extra maintenance and annoyance, while we can let everything live in memory with no particular organization.

3. I think that at the end of the compilation process you can use the strip program. It removes unneeded objects. see: https://en.wikipedia.org/wiki/Strip_(Unix)

Yes and no. Wren also run on Windows where strip is not available. Anyways considering wren size and the effect of strip, it would not be beneficial unless you are really on a memory stressed system, where you would have other problems running wren.

We can run the strip program when it be available. I never noticed that it would work negatively.

There is one negative case, that it removes debug symbols. Other than that code base is so small that (for now) it should not work really positively either.

What are your intensions/aims at trying to make the code smaller? Someone tryed to embedded wren on a microcontroller (if I remember well) so these questions have been discussed somehow.

I do not try to use the Wren language on the microcontroller. I just know that the smaller output program is better.

This is wrong in general. Less output means less breadcrumbs or to much fragmentation to easily solve problems.

In addition dedicated code usually produce fonctionnal duplication, to speed specific code (see C++ templates).

The only negative impact is load times. Since we tend to have bigger runtime blobs or have to recreate program memory on each run, it starts to have impact for the user. But both problems can be most of the times solved or ignored since the environment is small (for now at least)

Anyway, even considering we shaved the problem of stack trace and achieved save to disc. I'm not sure the complexity required to parse the blob will be faster than parsing code. Don't get me wrong, I would love to have the VM be more language neutral, and that would be achieved by having blobs. But that extra functionality, might become the new bottleneck, and a huge maintenance burden.

[Michał Kozakiewicz]

We can also use a (python or another) library that is simple to use and has no dependencies. If it exists, of course. What do you think about it?

2. Is it possible to first compile the script into byte-code and then add it to the result program? The result program will take up less space and will be more difficult to decompile.

No, not yet. And wrong it would not really take less space, since the script stacktrace will need a full version of the source anyways.

And since source is required (at least in current state) it would not make decompilation harder.

 

If the language needs an uncompiled script in the executable file, there is no way to do it.

There is always a way, the problem don't really come from the language, but from the implementation choices.

We can separate compilation from runtime, every huge language does that. The problems comes from designing the runtime blob, that add an extra maintenance and annoyance, while we can let everything live in memory with no particular organization.

I understand, thank you for your answer. Forgive me, by writing the language I was thinking about the implementation.

3. I think that at the end of the compilation process you can use the strip program. It removes unneeded objects. see: https://en.wikipedia.org/wiki/Strip_(Unix)

Yes and no. Wren also run on Windows where strip is not available. Anyways considering wren size and the effect of strip, it would not be beneficial unless you are really on a memory stressed system, where you would have other problems running wren.

We can run the strip program when it be available. I never noticed that it would work negatively.

There is one negative case, that it removes debug symbols. Other than that code base is so small that (for now) it should not work really positively either.

To be precise, I'm thinking about an independent Wren interpreter, not a Wren library. The release mode uses the -O3 flag. It can interfere with debugging symbols. There are separate flags for debugging. The -O3 flag improves the program execution time and increases the compilation time. It does not significantly affect the weight of the program. Therefore, we can improve output program by using the strip program. Only for release mode. See: https://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html

What are your intensions/aims at trying to make the code smaller? Someone tryed to embedded wren on a microcontroller (if I remember well) so these questions have been discussed somehow.

I do not try to use the Wren language on the microcontroller. I just know that the smaller output program is better.

This is wrong in general. Less output means less breadcrumbs or to much fragmentation to easily solve problems.

In addition dedicated code usually produce fonctionnal duplication, to speed specific code (see C++ templates).

The only negative impact is load times. Since we tend to have bigger runtime blobs or have to recreate program memory on each run, it starts to have impact for the user. But both problems can be most of the times solved or ignored since the environment is small (for now at least)

Anyway, even considering we shaved the problem of stack trace and achieved save to disc. I'm not sure the complexity required to parse the blob will be faster than parsing code. Don't get me wrong, I would love to have the VM be more language neutral, and that would be achieved by having blobs. But that extra functionality, might become the new bottleneck, and a huge maintenance burden.

I agree with you. I thought about the lower weight of the output program without affecting the functionality.

[Michel Hermier]

If -O3 is enabled with debugging then it is an error. Enabling the optimizer always disturb debugging. So enabling both is an error unless you search for optimizer error. So stripping should not be necessary.

What are your intensions/aims at trying to make the code smaller? Someone tryed to embedded wren on a microcontroller (if I remember well) so these questions have been discussed somehow.

I do not try to use the Wren language on the microcontroller. I just know that the smaller output program is better.

This is wrong in general. Less output means less breadcrumbs or to much fragmentation to easily solve problems.

In addition dedicated code usually produce fonctionnal duplication, to speed specific code (see C++ templates).

The only negative impact is load times. Since we tend to have bigger runtime blobs or have to recreate program memory on each run, it starts to have impact for the user. But both problems can be most of the times solved or ignored since the environment is small (for now at least)

Anyway, even considering we shaved the problem of stack trace and achieved save to disc. I'm not sure the complexity required to parse the blob will be faster than parsing code. Don't get me wrong, I would love to have the VM be more language neutral, and that would be achieved by having blobs. But that extra functionality, might become the new bottleneck, and a huge maintenance burden.

I agree with you. I thought about the lower weight of the output program without affecting the functionality.

--
You received this message because you are subscribed to the Google Groups "Wren" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wren-lang+...@googlegroups.com.
To post to this group, send email to wren...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wren-lang/7cd7c4a6-c946-416d-ba66-4ad10e2293b2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Michał Kozakiewicz]

You are 100% right. But the strip can just help. Such a small addition, but not necessary.

Stripped from 2 [MB] to 484 [KB]. On Windows 7, 64-bits; MSYS2, MinGW 32-bits.

Michal@Michal MINGW32 ~/wren/bin

$ ls -la

together 2032

drwxr-xr-x 1 Michal None       0 07-09 22:59 .

drwxr-xr-x 1 Michal None       0 07-09 22:59 ..

-rwxr-xr-x 1 Michal None 2075593 07-09 22:59 wren.exe

Michal@Michal MINGW32 ~/wren/bin

$ strip.exe wren.exe

Michal@Michal MINGW32 ~/wren/bin

$ ls -la

together 480

drwxr-xr-x 1 Michal None      0 07-09 23:00 .

drwxr-xr-x 1 Michal None      0 07-09 22:59 ..

-rwxr-xr-x 1 Michal None 484366 07-09 23:00 wren.exe

[Michel Hermier]

I did a bit of binary exploration, and on Linux the numbers are a little bit less impressive, but they still show reduction from near 800k to 300k. Exploration seems to show that the debug noise come from libuv, since every wren .o don't have it while libuv.a has it.

I was not able to go deeper why libuv does so, because of lack of time to dig the way build_libuv.py make the archive and try to see why libuv has debug symbols in release mode.

[Michał Kozakiewicz]

A great observation! Earlier, I did not ask myself this question: 'Where does this difference come from?'. 

In my opinion, Unix like systems are more perfect for programmers (today). Hence the difference between Linux and Windows.

Approximate calculation:

Linux:         800KB / 300KB = ~2,67

Windows: 2000KB / 485KB = ~4,12

Still more than 1, because it contains the `debug noise`.

What is the most important thing in all this? That the change from debug to release should speed up the resulting code.

Strip can become completely optional, if it can be done.

I have a lot of time, I will try to look. But I do not know, if I can do it alone.

At the moment the most important thing for me is: https://github.com/munificent/wren/issues/578

[Michel Hermier]

Le mar. 10 juil. 2018 à 15:49, Michał Kozakiewicz <miich...@gmail.com> a écrit :

A great observation! Earlier, I did not ask myself this question: 'Where does this difference come from?'. 

In my opinion, Unix like systems are more perfect for programmers (today). Hence the difference between Linux and Windows.

Approximate calculation:

Linux:         800KB / 300KB = ~2,67

Windows: 2000KB / 485KB = ~4,12

Still more than 1, because it contains the `debug noise`.

What is the most important thing in all this? That the change from debug to release should speed up the resulting code.

Appart from load time, it doesn't speed anything.

I may be mistaken, but what matters most is linking time. When you start to reference a huge nupber of libraries, that needs to gazillion of leaked unusee symbols, that really slow startup time.

That should be what could kill wren performance the most, but technically we should only be linking to live, so it should be good for most usage.

Strip can become completely optional, if it can be done.

It can always be done safely. But you have to keep in mind what you loose.

I have a lot of time, I will try to look. But I do not know, if I can do it alone.

At the moment the most important thing for me is: https://github.com/munificent/wren/issues/578

--
You received this message because you are subscribed to the Google Groups "Wren" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wren-lang+...@googlegroups.com.
To post to this group, send email to wren...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/wren-lang/7fe6d1e6-809f-49d8-8557-d898b95dcc28%40googlegroups.com.

[Michał Kozakiewicz]

I checked the entire build process for libuv v1.22.0 and libwren.

There are two main methods of building libuv, GYP and Autotools. There are differences in flags between them. I will not mention all flags, but only important for this thread.

For the release GYP uses: -DNDEBUG -O3

libuv.a size: 2 270 992 B

After the strip: 405 602 B

For the release Autotools uses: -O2

libuv.a size: 1 544 384 B

After the strip: 337 190 B

Autotools does not set the -DNDEBUG flag.

Releases built for MSYS2 MINGW32 for a 32 bit version both (I used the right flag for GYP.). I know that MSYS2 is not fully compatible (libuv), for these tests it does not bother.

Both libuv.a files contain a lot of debug symbols, despite the release. The flags mentioned above do not affect debug symbols.

The Wren Interpreter in the stand-alone version uses the libuv library. Currently, all unnecessary debug symbols are added to it.

strip --strip-debug libuv.a

The use of the above command eliminates the problem. Removes only debug symbols. In the same way, treated libwren.a library releases about 1KB of rubbish.

I think that you should use the strip program first for libuv.a and then for Wren. 

How will it help?

Removing the debug symbols from the libuv.a library will speed up the Wren build. The program launch time will be improved. We can see this especially when running mass of scripts.

But be careful! Do not use this command:

strip libuv.a

It will remove ALL symbols, even needed! This version can be used only for the resulting program or shared library.

[Michel Hermier]

Le jeu. 12 juil. 2018 à 03:45, Michał Kozakiewicz <miich...@gmail.com> a écrit :

I checked the entire build process for libuv v1.22.0 and libwren.

There are two main methods of building libuv, GYP and Autotools. There are differences in flags between them. I will not mention all flags, but only important for this thread.

For the release GYP uses: -DNDEBUG -O3

libuv.a size: 2 270 992 B

After the strip: 405 602 B

For the release Autotools uses: -O2

libuv.a size: 1 544 384 B

After the strip: 337 190 B

Autotools does not set the -DNDEBUG flag.

Per definition, NDEBUG defines live extra debugging. So you should not have reached the full flags. Unless I'm mistaken debug symbols should mean a -g flags is defined somehow.

Releases built for MSYS2 MINGW32 for a 32 bit version both (I used the right flag for GYP.). I know that MSYS2 is not fully compatible (libuv), for these tests it does not bother.

Both libuv.a files contain a lot of debug symbols, despite the release. The flags mentioned above do not affect debug symbols.

The Wren Interpreter in the stand-alone version uses the libuv library. Currently, all unnecessary debug symbols are added to it.

strip --strip-debug libuv.a

The use of the above command eliminates the problem. Removes only debug symbols. In the same way, treated libwren.a library releases about 1KB of rubbish.

I think that you should use the strip program first for libuv.a and then for Wren. 

How will it help?

Removing the debug symbols from the libuv.a library will speed up the Wren build. The program launch time will be improved. We can see this especially when running mass of scripts.

But be careful! Do not use this command:

strip libuv.a

It will remove ALL symbols, even needed! This version can be used only for the resulting program or shared library.

--
You received this message because you are subscribed to the Google Groups "Wren" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wren-lang+...@googlegroups.com.
To post to this group, send email to wren...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/wren-lang/63e70a95-593e-4b9b-a0b4-b2b3e7374549%40googlegroups.com.

[Michał Kozakiewicz]

W dniu czwartek, 12 lipca 2018 09:20:11 UTC+2 użytkownik Michel Hermier napisał:

Per definition, NDEBUG defines live extra debugging. So you should not have reached the full flags. Unless I'm mistaken debug symbols should mean a -g flags is defined somehow.

You're right! I missed this flag. My bad :(. I am thinking about submitting this to libuv, because it should not be in this configuration.

GYP:

C:\Programy\msys2\home\Michal\libuv-gyp\out\Release\.deps\out\Release\obj.target\libuv\src\version.o.d

cmd_out/Release/obj.target/libuv/src/version.o := cc '-DWIN32' '-D_CRT_SECURE_NO_DEPRECATE' '-D_CRT_NONSTDC_NO_DEPRECATE' '-D_WIN32_WINNT=0x0600' '-D_GNU_SOURCE' '-DNDEBUG' -Iinclude -Isrc  -fvisibility=hidden -g --std=gnu89 -pedantic -Wall -Wextra -Wno-unused-parameter -Wstrict-prototypes -O3 -fomit-frame-pointer -fdata-sections -ffunction-sections  -MMD -MF out/Release/.deps/out/Release/obj.target/libuv/src/version.o.d.raw   -c -o out/Release/obj.target/libuv/src/version.o src/version.c

out/Release/obj.target/libuv/src/version.o: src/version.c include/uv.h \

 include/uv/errno.h include/uv/version.h include/uv/win.h \

 include/uv/tree.h include/uv/threadpool.h

src/version.c:

include/uv.h:

include/uv/errno.h:

include/uv/version.h:

include/uv/win.h:

include/uv/tree.h:

include/uv/threadpool.h:

Autotools:

C:\Programy\msys2\home\Michal\libuv-autotools\Makefile

CFLAGS = -g -O2 -pedantic -g -std=gnu89 -Wall -Wextra -Wno-unused-parameter -Wstrict-prototypes
libuv_la_CFLAGS = -g -O2 -pedantic -g -std=gnu89 -Wall -Wextra -Wno-unused-parameter -Wstrict-prototypes $(am__append_1) $(am__append_14) \

Information about flags:

https://gcc.gnu.org/onlinedocs/gcc/Debugging-Options.html

3.9 Options for Debugging Your Program

To tell GCC to emit extra information for use by a debugger, in almost all cases you need only to add -g to your other options.

GCC allows you to use -g with -O. The shortcuts taken by optimized code may occasionally be surprising: some variables you declared may not exist at all; flow of control may briefly move where you did not expect it; some statements may not be executed because they compute constant results or their values are already at hand; some statements may execute in different places because they have been moved out of loops. Nevertheless it is possible to debug optimized output. This makes it reasonable to use the optimizer for programs that might have bugs.

If you are not using some other optimization option, consider using -Og (see Optimize Options) with -g. With no -O option at all, some compiler passes that collect information useful for debugging do not run at all, so that -Og may result in a better debugging experience.

https://gcc.gnu.org/onlinedocs/gcc/Option-Summary.html

All Debugging Options:

-g  -glevel  -gdwarf  -gdwarf-version 

-ggdb  -grecord-gcc-switches  -gno-record-gcc-switches 

-gstabs  -gstabs+  -gstrict-dwarf  -gno-strict-dwarf 

-gas-loc-support  -gno-as-loc-support 

-gas-locview-support  -gno-as-locview-support 

-gcolumn-info  -gno-column-info 

-gstatement-frontiers  -gno-statement-frontiers 

-gvariable-location-views  -gno-variable-location-views 

-ginternal-reset-location-views  -gno-internal-reset-location-views 

-ginline-points  -gno-inline-points 

-gvms  -gxcoff  -gxcoff+  -gz[=type] 

-gsplit-dwarf 

-fdebug-prefix-map=old=new  -fdebug-types-section 

-fno-eliminate-unused-debug-types 

-femit-struct-debug-baseonly  -femit-struct-debug-reduced 

-femit-struct-debug-detailed[=spec-list] 

-feliminate-unused-debug-symbols  -femit-class-debug-always 

-fno-merge-debug-strings  -fno-dwarf2-cfi-asm 

-fvar-tracking  -fvar-tracking-assignments

[Michel Hermier]

Before submitting to libuv, try to use last version of it, it can be fixed there.

[Michał Kozakiewicz]

W dniu czwartek, 12 lipca 2018 20:00:32 UTC+2 użytkownik Michel Hermier napisał:

Before submitting to libuv, try to use last version of it, it can be fixed there.

 

Of course! Currently, version 1.22.0 is the newest one. All information about it above is just for this version.

[Michał Kozakiewicz]

I created issue for this problem at libuv repository.

https://github.com/libuv/libuv/issues/1924