size optimization report

[Krešimir Špes]

I spent some time trying to compile Ogre wrapper without built in doc
strings, and the results aren't what I expected: I saved 1.4 MB's.

_ogre_.pyd : 25.5 MB

_ogre_optimized_.pyd : 24.1 MB


Roman once mentioned that C++ templates play a significant role in
wrapper size. can you explain? why is that?

[Roman Yakovenko]

On Sun, Feb 15, 2009 at 5:26 PM, Krešimir Špes <kresim...@gmail.com> wrote:
>
> I spent some time trying to compile Ogre wrapper without built in doc
> strings, and the results aren't what I expected: I saved 1.4 MB's.
>
> _ogre_.pyd : 25.5 MB
>
> _ogre_optimized_.pyd : 24.1 MB

Keep going, you have so much things to remove:
* classes and functions you don't use
* if you don't need override virtual functions - make them non virtual
* if you don't need access to protected\private functions exclude them
* it is possible to move all enumerations to Python

> Roman once mentioned that C++ templates play a significant role in
> wrapper size. can you explain? why is that?

C doesn't have overloading, so the linker can put function name only
within dll and everybody will happy.

C++ has overloading, namespace and template - all these information
should be part of the final function name. Of course the name became
bigger - much bigger.

I suggest you to use Google to find better answer to your question.

--
Roman Yakovenko
C++ Python language binding
http://www.language-binding.net/

[Andy Miller]

You can get far more optimization by disabling pre-compiled headers..

However as an overall observation I personally don't believe it's worth while trying to shrink the size of the modules as there are plenty of other tasks that take priority - and my goal with the 'official' python-ogre is to support "everything" possible, hence I will not be disabling classes etc...

If you are concerned about disk usage then use a dll compressor and this will make the modules 'look' smaller (they will still take the same amount of memory).  

Regards

Andy

2009/2/16 Roman Yakovenko <roman.y...@gmail.com>

[Krešimir Špes]

I'm not really concerned about module size, I'm just experimenting,
trying to figure out in depth how python-ogre works so I can
contribute better.

I was thinking of introducing some compile options in the trunk that
would allow easy configuration. eg;

* precompiled header(already exists)
* doc string extraction (boolean)
* private and protected functions wrap (boolean)

if I'm right, these should be put in PythonOgreConfig_xx.py ?

PS: why do precompiled headers take more space in the final binary?

On Feb 16, 2:48 am, Andy Miller <nzmill...@gmail.com> wrote:
> You can get far more optimization by disabling pre-compiled headers..
> However as an overall observation I personally don't believe it's worth
> while trying to shrink the size of the modules as there are plenty of other
> tasks that take priority - and my goal with the 'official' python-ogre is to
> support "everything" possible, hence I will not be disabling classes etc...
>
> If you are concerned about disk usage then use a dll compressor and this
> will make the modules 'look' smaller (they will still take the same amount
> of memory).
>
> Regards
> Andy
>

> 2009/2/16 Roman Yakovenko <roman.yakove...@gmail.com>
>
>
>
> > On Sun, Feb 15, 2009 at 5:26 PM, Krešimir Špes <kresimir.s...@gmail.com>

[Andy Miller]

I'm not sure these changes belong in the trunk yet :)

Having protected etc functions/methods not exposed adds a limitation to the end result without any obvious advantages.. While this might come out of a complete rewrite of the build system (one of the things on my to look at list) I don't want to add options that might cause me to create a less than complete binary (already doing that to myself unintentionally :) )

The way docstrings are generated and final documentation is currently under development and will touch all the code generators so I don't want to change anything there at the moment - I'm looking to make docstrings optional (as you've mentioned) and to generate more usable documentation with Sphinx...

The precompiled header thing is a function of the compiler -- by using precompiled headers it can be much faster and the trade off is in the final generated code size :(

Regards

Andy

2009/2/16 Krešimir Špes <kresim...@gmail.com>

[Roman Yakovenko]

On Mon, Feb 16, 2009 at 11:09 AM, Andy Miller <nzmi...@gmail.com> wrote:
> The way docstrings are generated and final documentation is currently under
> development and will touch all the code generators so I don't want to change
> anything there at the moment - I'm looking to make docstrings optional (as
> you've mentioned) and to generate more usable documentation with Sphinx...

I think, we can join the effort here and convert doxygen format to
reStrucuturedText one. May be not complete but, some subset.

Then this "doc_extractor" could become part of Py++.

[Andy Miller]

Agreed -- I'm rewriting my doc extractor to be more modular so it can cope with variations in the input comments -- also I need to cope with adding comments/docs to a function from various places.

ie If we modify a functions calling sig (functional transformation etc) then I need to track that change so that when the actual doc extractor runs (and I create a function sig for the documentation) that it matches the tweaked version etc etc...

Andy

2009/2/16 Roman Yakovenko <roman.y...@gmail.com>

[Krešimir Špes]

well, imho, precompiled option should be used for testing, but should
be disabled for releases.

protected/private functions should be exposed by default, but there
should be a compile-time option that disables it if someone doesn't
want it. Eg, a linux ogra wrapper is over 60 megs in size, and a few
tweaks like this could go a long way.

While reducing wrapper size certainly is not a priority, it should be
done sooner or later.

On Feb 16, 10:09 am, Andy Miller <nzmill...@gmail.com> wrote:
> I'm not sure these changes belong in the trunk yet :)
>
> Having protected etc functions/methods not exposed adds a limitation to the
> end result without any obvious advantages.. While this might come out of a
> complete rewrite of the build system (one of the things on my to look at
> list) I don't want to add options that might cause me to create a less than
> complete binary (already doing that to myself unintentionally :) )
>
> The way docstrings are generated and final documentation is currently under
> development and will touch all the code generators so I don't want to change
> anything there at the moment - I'm looking to make docstrings optional (as
> you've mentioned) and to generate more usable documentation with Sphinx...
>
> The precompiled header thing is a function of the compiler -- by using
> precompiled headers it can be much faster and the trade off is in the final
> generated code size :(
>
> Regards
> Andy
>

> 2009/2/16 Krešimir Špes <kresimir.s...@gmail.com>

[Roman Yakovenko]

On Mon, Feb 16, 2009 at 2:25 PM, Krešimir Špes <kresim...@gmail.com> wrote:
>
> well, imho, precompiled option should be used for testing, but should
> be disabled for releases.
>
> protected/private functions should be exposed by default, but there
> should be a compile-time option that disables it if someone doesn't
> want it. Eg, a linux ogra wrapper is over 60 megs in size, and a few
> tweaks like this could go a long way.
>
> While reducing wrapper size certainly is not a priority, it should be
> done sooner or later.

I could be wrong, but Python-Ogre development is driven by use cases.
The main Andy's use case is: provide access to **all** Ogre API. What
is your use case? Why the size of the project matter to you ( so much
)?

[Ivan Vučica]

On Mon, Feb 16, 2009 at 13:44, Roman Yakovenko <roman.y...@gmail.com> wrote:

I could be wrong, but Python-Ogre development is driven by use cases.
The main Andy's use case is: provide access to **all** Ogre API. What
is your use case? Why the size of the project matter to you ( so much
)?

Publishers of games for casual market complain about sizes. A lot :)

And it makes no sense if the wrapper library is 50% of game size if it could be 10%. For example, installer of our game that is going gold in a few hours is 90MB in size. Game itself takes 145MB when installed. That is with PythonOgre included. You can see that significant decrease of size of wrapper would mean a significant decrease in overall game size.

--
Regards,

Ivan Vučica

OBJECT Networks :: www.objectnetworks.net
Cateia Games :: www.cateia.com
Zagrebački računalni savez :: www.zrs.hr
iv...@vucica.net :: ivan.vucica.net

[Roman Yakovenko]

2009/2/16 Ivan Vučica <ivu...@gmail.com>:

> On Mon, Feb 16, 2009 at 13:44, Roman Yakovenko <roman.y...@gmail.com>
> wrote:
>>
>> I could be wrong, but Python-Ogre development is driven by use cases.
>> The main Andy's use case is: provide access to **all** Ogre API. What
>> is your use case? Why the size of the project matter to you ( so much
>> )?
>>
>
>
> Publishers of games for casual market complain about sizes. A lot :)
>
> And it makes no sense if the wrapper library is 50% of game size if it could
> be 10%. For example, installer of our game that is going gold in a few hours
> is 90MB in size. Game itself takes 145MB when installed. That is with
> PythonOgre included. You can see that significant decrease of size of
> wrapper would mean a significant decrease in overall game size.

Do you have a good subset definition of what you need?

[Ivan Vučica]

2009/2/16 Roman Yakovenko <roman.y...@gmail.com>

2009/2/16 Ivan Vučica <ivu...@gmail.com>:

> Publishers of games for casual market complain about sizes. A lot :)
>
> And it makes no sense if the wrapper library is 50% of game size if it could
> be 10%. For example, installer of our game that is going gold in a few hours
> is 90MB in size. Game itself takes 145MB when installed. That is with
> PythonOgre included. You can see that significant decrease of size of
> wrapper would mean a significant decrease in overall game size.

Do you have a good subset definition of what you need?

Hm, I'm not sure, so I'll leave this to Krešo.

But, I believe this would be mostly options to throw out as much debugging/documentation information as possible, in specialized builds. Also, looking back to the discussion, I don't think private and protected interfaces need documentation for anything except internal debugging (that is, users of library - that is, programmers) should never access private interfaces, thus they don't need documentation on it in the default build.

Well, those are just my thoughts, as I said I'll leave further discussion to Krešo to avoid further embarrassing myself in case I said something stupid :)

[Roman Yakovenko]

On Mon, Feb 16, 2009 at 3:00 PM, Ivan Vučica <ivu...@gmail.com> wrote:
> 2009/2/16 Roman Yakovenko <roman.y...@gmail.com>

>> Do you have a good subset definition of what you need?
>
> Hm, I'm not sure, so I'll leave this to Krešo.
>
> But, I believe this would be mostly options to throw out as much
> debugging/documentation information as possible, in specialized builds.

I could be wrong, but Python-Ogre release build doesn't contain debug
information. Also, AFAIK MSVC compiler stores debug information
outside of the binary ( executable and .dll ). Same is possible with
gcc. I have SCons custom builder that does this,

> Also, looking back to the discussion, I don't think private and protected
> interfaces need documentation for anything except internal debugging (that
> is, users of library - that is, programmers) should never access private
> interfaces, thus they don't need documentation on it in the default build.

This is just one example: http://en.wikipedia.org/wiki/Template_method_pattern
Ogre has a lot of functionality under "protected" level, which is
available for C++ developers in the derived classes, hence it should
be available to Python developers too.

[Ivan Vučica]

2009/2/16 Roman Yakovenko <roman.y...@gmail.com>

> Also, looking back to the discussion, I don't think private and protected
> interfaces need documentation for anything except internal debugging (that
> is, users of library - that is, programmers) should never access private
> interfaces, thus they don't need documentation on it in the default build.

This is just one example: http://en.wikipedia.org/wiki/Template_method_pattern
Ogre has a lot of functionality under "protected" level, which is
available for C++ developers in the derived classes, hence it should
be available to Python developers too.

Hm, true. I hope you, however, agree with Krešo's:

> protected/private functions should be exposed by default, but there
> should be a compile-time option that disables it if someone doesn't
> want it. Eg, a linux ogra wrapper is over 60 megs in size, and a few
> tweaks like this could go a long way.

[Roman Yakovenko]

2009/2/16 Ivan Vučica <ivu...@gmail.com>:

> 2009/2/16 Roman Yakovenko <roman.y...@gmail.com>
>>
>> > Also, looking back to the discussion, I don't think private and
>> > protected
>> > interfaces need documentation for anything except internal debugging
>> > (that
>> > is, users of library - that is, programmers) should never access private
>> > interfaces, thus they don't need documentation on it in the default
>> > build.
>>
>> This is just one example:
>> http://en.wikipedia.org/wiki/Template_method_pattern
>> Ogre has a lot of functionality under "protected" level, which is
>> available for C++ developers in the derived classes, hence it should
>> be available to Python developers too.
>>
>
> Hm, true. I hope you, however, agree with Krešo's:
>> protected/private functions should be exposed by default, but there
>> should be a compile-time option that disables it if someone doesn't
>> want it. Eg, a linux ogra wrapper is over 60 megs in size, and a few
>> tweaks like this could go a long way.

I think, that you should define your use case better.

As I see it right now, you are asking to do some work without being
sure it will solve your problem.
For example: disabling documentation strings saved you 1.4 Mb - +/- 1%
of the final dll file. It's not worth to add such option, may be in
future.
Same could be true for private and protected functions.

[Krešimir Špes]

I'm not asking for anyone to do the work for me. I'm more than glad to
experiment on my own, I'm meerly sharing my findings with the
community.

Also, size is not that important, I'm doing this research only because
I'd like to know how much size can I remove from the wrapper, and in
that process learn more about pyhton ogre internals.


On Feb 16, 3:08 pm, Roman Yakovenko <roman.yakove...@gmail.com> wrote:
> 2009/2/16 Ivan Vučica <ivuc...@gmail.com>:
>
>
>
> > 2009/2/16 Roman Yakovenko <roman.yakove...@gmail.com>

[Andy Miller]

When I get some time I'll create/update the wiki with my views on the development direction for Python-Ogre so there isn't any confusion -- this by no means stops anyone updating/creating/modifying their own versions (the entire concept of open source)..

I have very clear goals for the primary Python-Ogre build which involves exposing everything possible and making it useful to the widest possible audience, which will mean that there are "capabilities" that will simply not make it into the project if they narrow the scope too much..

As an aside if anyone really needs to reduce disk usage for Python-Ogre you can use UPX to shrink the module sizes.. 

And Linux module size can be dramatically changed by messing with the gcc visibility functionality, however as this varies by gcc version/platform (works on some, not on others) it's something I have currently disabled to ensure things work on the widest possible number of environments..

Regards

Andy

2009/2/16 Krešimir Špes <kresim...@gmail.com>

[Roman Yakovenko]

2009/2/16 Krešimir Špes <kresim...@gmail.com>:

>
> I'm not asking for anyone to do the work for me. I'm more than glad to
> experiment on my own, I'm meerly sharing my findings with the
> community.

Sorry, I didn't mean to hurt you. I tried to understand better your use case.

> Also, size is not that important, I'm doing this research only because
> I'd like to know how much size can I remove from the wrapper, and in
> that process learn more about pyhton ogre internals.

You will be able to reduce the size, by defining a closed subset of
classes, that you use for your game and excluding everything else.

I hope the following link, will help you to understand better the idea
and design behind Python-Ogre:
http://www.language-binding.net/pyplusplus/pyplusplus.html