Re: [wxpython-users] lib.iewin hangs py2exe build
Erez Bibi
Mike Conley wrote:
> Robin got me on the right track here.
> comtypes\gen\_3050F1C5_98B5_11CF_BB82_00AA00BDCE0B_0_4_0.py
> is a 60,000+ file. the .pyc is over 1.4Mb
>
> trying to run a one liner that just imports that file through py2exe
> creates the problem. I suspect that letting it run long enough would
> eventually go to conclusion. Unfortunately we do our build on a
> relatively low power machine and probably can't change that any time
> soon (startups don't have much money).
>
> Does anyone know if there is a technique to tell py2exe to "put this
> in your output, but don't bother to analyze it"? If so, I can manually
> include this file and any relevant dependencies.
>
Here Is an email I just sent the maintainer of py2exe Jimmy Retzlaff
Adding "skip_scan" option to py2exe
===================================
Refer To: py2exe version 0.6.9
Introduction
=============
I'm using py2exe for exe-ing a wxPython application. Recently I had to add
the embedded Internet Explorer module "wx.lib.iewin" to my application. This
module uses "comtypes" which is generating *very* big module that need to be
included in the distribution executable. Scanning this big module takes
about 15 minutes on my computer (Duo Core, 2G RAM). The idea is to tell
py2exe not to scan this module and add any required module manually through
the "includes" option.
Implementation
===============
I added a "skip_scan" option to py2exe, which is a list of modules names
(like: includes, excludes)
Changes in the code:
build_exe.py:
After line 162 in py2exe, insert:
("skip-scan=", None, "comma-separated list of modules not to scan for imported modules"),
Change line 342 in py2exe.create_modulefinder to:
return ModuleFinder(excludes=self.excludes, skip_scan=self.skip_scan)
mf.py:
Change line 87 definition of ModuleFinder.__init__ to:
def __init__(self, path=None, debug=0, excludes=[], replace_paths=[],
skip_scan=[]):
After line 96 in ModuleFinder.__init__ insert:
self.skip_scan = skip_scan
After line 404 in ModuleFinder.scan_code insert:
if m.__name__ in self.skip_scan: return
Usage
======
A strip down example:
setup (
windows = my_win_app,
options = {
'py2exe': {
'includes': ['my_module', <any module "my_module" needs>],
'skip_scan': ['my_module']
}
}
"my_module" will be included in the executable, but will not be scanned for
modules imported from it.
Risks
======
of course I introduced the risk of missing modules in the executable. The
developer will have to test the application, and make sure all the required
modules exists. In my case, it is a risk worth taking as it saves almost 15
minutes from the build process.
Thank you for developing py2exe, it is an excellent tool!
Erez Bibi
Consultant for blackwoodproductions.com
erezbibi AT frontiernet DOT net
Christopher Barker
>> Does anyone know if there is a technique to tell py2exe to "put this
>> in your output, but don't bother to analyze it"? If so, I can manually
>> include this file and any relevant dependencies.
>
> Here Is an email I just sent the maintainer of py2exe Jimmy Retzlaff
great! thanks, that's been a pain for me, too. I hope your patch gets
accepted.
-Chris
--
Christopher Barker, Ph.D.
Oceanographer
Emergency Response Division
NOAA/NOS/OR&R (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception
Andrea Gavana
> Erez Bibi wrote:
>>>
>>> Does anyone know if there is a technique to tell py2exe to "put this in
>>> your output, but don't bother to analyze it"? If so, I can manually include
>>> this file and any relevant dependencies.
>>
>> Here Is an email I just sent the maintainer of py2exe Jimmy Retzlaff
>
> great! thanks, that's been a pain for me, too. I hope your patch gets
> accepted.
Almost time to add another option to GUI2Exe then :-D
Andrea.
"Imagination Is The Only Weapon In The War Against Reality."
http://xoomer.alice.it/infinity77/
http://thedoomedcity.blogspot.com/
Thomas Heller
> Mike Conley wrote:
>> Robin got me on the right track here.
>> comtypes\gen\_3050F1C5_98B5_11CF_BB82_00AA00BDCE0B_0_4_0.py
>> is a 60,000+ file. the .pyc is over 1.4Mb
>>
>> trying to run a one liner that just imports that file through py2exe
>> creates the problem. I suspect that letting it run long enough would
>> eventually go to conclusion. Unfortunately we do our build on a
>> relatively low power machine and probably can't change that any time
>> soon (startups don't have much money).
>>
>> Does anyone know if there is a technique to tell py2exe to "put this
>> in your output, but don't bother to analyze it"? If so, I can manually
>> include this file and any relevant dependencies.
>>
>
> Here Is an email I just sent the maintainer of py2exe Jimmy Retzlaff
Have you received an answer from Jimmy? He told me (some time ago, in private
email) that he wants to step back from maintaining py2exe. So, you should at
least post your 'patch' to the py2exe-users list, and submit it to the
py2exe bug tracker. No guarantee that this will lead to inclusion of your
patch, but at least it won't get lost so easily.
>
> Adding "skip_scan" option to py2exe
> ===================================
>
> Refer To: py2exe version 0.6.9
>
> Introduction
> =============
> I'm using py2exe for exe-ing a wxPython application. Recently I had to add
> the embedded Internet Explorer module "wx.lib.iewin" to my application. This
> module uses "comtypes" which is generating *very* big module that need to be
> included in the distribution executable. Scanning this big module takes
> about 15 minutes on my computer (Duo Core, 2G RAM). The idea is to tell
> py2exe not to scan this module and add any required module manually through
> the "includes" option.
>
> Implementation
> ===============
> I added a "skip_scan" option to py2exe, which is a list of modules names
> (like: includes, excludes)
>
[...]
Have you tried to profile and speed up modulefinder, so that it gets faster?
I think that this would also be worth to try.
--
Thanks,
Thomas
Christopher Barker
Thomas Heller wrote:
> Have you tried to profile and speed up modulefinder, so that it gets faster?
> I think that this would also be worth to try.
or port py2exe to modulegraph, used by py2app, and now by bbfreeze. I
think it's faster and maybe more robust, but that may be a function of
what machines I'm running on...
It would be really nice to have a grand unification of GUI builders. It
strikes me that there are three components to any GUI builder:
1) The API presented to the user: how you specify what you want
2) The module finding code: how the system figure out what to include?
3) The actual app builder -- how everything gets packaged up.
- this is necessarily different for each platform, and appears to be
one of the key differences between different systems on the same OS.
Anyway, ti would be nice if there was more shared code between the GUI
builders -- at least (2) would help a lot. aside from coding
efficiencies, It would be nice to be more platform independent. For
instance, py2app's API is modeled after py2exe. However, they use
totally different module finding code, so I find I need to give totally
different extra directive for any but trivial packages -- that's a real
pain.
oh well, I'm not offering to write anything...
Thomas Heller
> OT, but...
>
>
> Thomas Heller wrote:
>> Have you tried to profile and speed up modulefinder, so that it gets faster?
>> I think that this would also be worth to try.
>
> or port py2exe to modulegraph, used by py2app, and now by bbfreeze. I
> think it's faster and maybe more robust, but that may be a function of
> what machines I'm running on...
>
> It would be really nice to have a grand unification of GUI builders. It
> strikes me that there are three components to any GUI builder:
GUI builders? You mean APP builders, I assume.
> 1) The API presented to the user: how you specify what you want
>
> 2) The module finding code: how the system figure out what to include?
>
> 3) The actual app builder -- how everything gets packaged up.
> - this is necessarily different for each platform, and appears to be
> one of the key differences between different systems on the same OS.
>
>
> Anyway, ti would be nice if there was more shared code between the GUI
> builders -- at least (2) would help a lot. aside from coding
> efficiencies, It would be nice to be more platform independent. For
> instance, py2app's API is modeled after py2exe. However, they use
> totally different module finding code, so I find I need to give totally
> different extra directive for any but trivial packages -- that's a real
> pain.
Sure, plus a hooks mechanism like PyInstaller has it...
> oh well, I'm not offering to write anything...
And that is the real problem. Given the popularity of py2exe and friends
I really wonder why there isn't more activity in development.
--
Thomas
j
>
> > oh well, I'm not offering to write anything...
>
> And that is the real problem. Given the popularity of py2exe and friends
> I really wonder why there isn't more activity in development.
>
- Because it is a complex task. It requires a very good knowlege
of Python *and* MiscroSoft stuff.
- It does not require only Python, also the MS tools.
- Idealy, a potential developer shoud have more than one Windows version.
(today, XP, Vista, 7).
- It's true, there are plenty of users. Most of them do not understand
how py2exe works.
- ...
- Practically , a "full time job".
You know this better than me.
Contrary to Chris' opinion, I think such a packager tool should be
platform specific. It's complicate enough [*] and the philosophy
"one for all" leads to "half backed" applications.
[*] I spent my evening yesterday toying with embedding icons in a
test exe for Vista...
py2exe is a great tool.
jmf
Erez Bibi
Thomas Heller-2 wrote:
>
> Have you received an answer from Jimmy? He told me (some time ago, in
> private
> email) that he wants to step back from maintaining py2exe. So, you should
> at
> least post your 'patch' to the py2exe-users list, and submit it to the
> py2exe bug tracker. No guarantee that this will lead to inclusion of your
> patch, but at least it won't get lost so easily.
>
I have submitted a py2exe patch and to the users list (thanks for the
advice)
Thomas Heller-2 wrote:
>
> Have you tried to profile and speed up modulefinder, so that it gets
> faster?
> I think that this would also be worth to try.
>
Like Chris wrote "oh well, I'm not offering to write anything..."
Well me too (at least for now), I just needed this feature.
--
View this message in context: http://www.nabble.com/lib.iewin-hangs-py2exe-build-tp22973879p23877937.html
Sent from the wxPython-users mailing list archive at Nabble.com.
Christopher Barker
> - Practically , a "full time job".
so true.
> Contrary to Chris' opinion, I think such a packager tool should be
> platform specific. It's complicate enough [*] and the philosophy
> "one for all" leads to "half backed" applications.
I DO think that they need to be platform specific, but the entire thing
does not -- figuring out what modules need to be included is exactly the
same on all platforms -- why have multiple *half-baked* implementations
of that?
I also don't see any reason they couldn't have the same API.
Actually bundling up the stuff certainly has to be platform dependent.
Thomas Heller
> OT, but...
>
>
> Thomas Heller wrote:
>> Have you tried to profile and speed up modulefinder, so that it gets faster?
>> I think that this would also be worth to try.
>
> or port py2exe to modulegraph, used by py2app, and now by bbfreeze. I
> think it's faster and maybe more robust, but that may be a function of
> what machines I'm running on...
Well, I finally tried out parsing the comtypes\gen\MSHTML.py module (which
pulls in this giant comtypes\gen\_3050F1C5_98B5_11CF_BB82_00AA00BDCE0B_0_4_0.py
module), and there were no diffence in the time needed to parse this code.
Both took 3.9 seconds on my machine.
--
Thanks,
Thomas
Nitro
sorry to pull out this old topic again, but I ran into exactly the same
issue today (although build time was closer to 5 minutes than 15 minutes).
The patch below basically works, but it is missing these two bits:
build_exe.py:
after line 200 ("self.dll_excludes = None") put:
self.skip_scan = None
after line 234 ("self.dll_excludes = ...") put:
self.skip_scan = fancy_split(self.skip_scan)
After these two changes I got it working. I used this as the specific
modulename for wx.lib.iewin:
'skip_scan' : [
'comtypes.gen._3050F1C5_98B5_11CF_BB82_00AA00BDCE0B_0_4_0' ]
I've also checked SVN at
http://py2exe.svn.sourceforge.net/viewvc/py2exe/trunk/py2exe/
and it doesn't seem this patch made it through so far. If I prepare a
patch is there anybody who will commit it?
-Matthias
> _______________________________________________
> wxpython-users mailing list
> wxpytho...@lists.wxwidgets.org
> http://lists.wxwidgets.org/mailman/listinfo/wxpython-users