scrapy with py2exe/pyinstaller

[Piotr]

Hello

I have a scrapy project with ~30 spiders. It is meant to be a standalone windows desktop application so I wrote a simple GUI in wxPython and I run them from Python script.

I tried to package it into an exe file using pyinstaller and py2exe and this is where I have issues.

Exporting ends with a success and no errors, but the executable fails whenever there is a pkgutil.get_data() call.

Has anyone succeeded in exporting a scrapy app?

I include tracebacks below

Piotr

Running .exe from py2exe:

Traceback (most recent call last):
  File "wx_gui.py", line 51, in <module>
  File "scrapy\responsetypes.pyc", line 114, in <module>
  File "scrapy\responsetypes.pyc", line 35, in __init__
  File "pkgutil.pyc", line 591, in get_data
IOError: [Errno 3] No such process: 'scrapy\\mime.types'

Running .exe from pyinstaller (pkgutil.get_data returns None instead of mimetypes there):

D:\dev\scrp\release\dist\scrp.exe?175104\twisted\python\modules.py:775: UserWarning: D:\dev\scrp\release\dist\scrp.exe?175104 (for module twisted.web.util) not in path importer cache (PEP 302 violation - check your local configuration).
Traceback (most recent call last):
  File "<string>", line 51, in <module>
  File "C:\dev\PyInstaller\loader\iu.py", line 386, in importHook
    mod = _self_doimport(nm, ctx, fqname)
  File "C:\dev\PyInstaller\loader\iu.py", line 480, in doimport
    exec co in mod.__dict__
  File "release\build\pyi.win32\scrp\out00-PYZ.pyz\scrapy.responsetypes", line 114, in <module>
  File "release\build\pyi.win32\scrp\out00-PYZ.pyz\scrapy.responsetypes", line 37, in __init__
TypeError: expected read buffer, NoneType found

[Pablo Hoffman]

I don't have experience with py2exe and I don't use Windows, but search on StackOverflow, there seem to be a few questions about scrapy and py2exe there.

Just a couple of those:

http://stackoverflow.com/questions/3790563/how-can-i-package-a-scrapy-project-using-cxfreeze

http://stackoverflow.com/questions/8114839/how-to-make-an-exe-file-out-of-python-scrapy-script

If you make it work, please consider writing a guide about how to do it so we can link it from Scrapy wiki and doc.

Good luck!

--
You received this message because you are subscribed to the Google Groups "scrapy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scrapy-users...@googlegroups.com.
To post to this group, send email to scrapy...@googlegroups.com.
Visit this group at http://groups.google.com/group/scrapy-users.
For more options, visit https://groups.google.com/groups/opt_out.

[Wei Oui]

Hi Piotr, 

I'm a newbie to python and I need to do a crawler with a simple GUI. I had done my spider and it work perfectly but I don't know how to incorporate with a GUI. I read a simple tutorial with wxPython but I have no idea how to implement it with scrapy. Can you please give a a help (an example)? 

Thanks in advance. 

Wei

[Karen Oganesyan]

Hi Wei, i think you'll find answer here: http://doc.scrapy.org/en/latest/topics/practices.html#run-scrapy-from-a-script. In a few words, you should use scrapi API to manage created spider from main script.

понедельник, 6 января 2014 г., 18:48:02 UTC+4 пользователь Wei Oui написал:

[cdd801]

hi,karen, do you have some demos about making scrapy project into an exe file? i am so confused by the doc,i don't know where to add the code  

在 2014年3月30日星期日UTC+8上午2时47分27秒，Karen Oganesyan写道：