Groups keyboard shortcuts have been updated
Dismiss
See shortcuts
Error running Scrapy after Installing .10
291 views
Skip to first unread message
ow3n
Sep 13, 2010, 4:54:20 AM9/13/10
to scrapy-users
Hi there,
I am running Python26 with Macports on a MBP with 10.5. I found some
errors after updating Scrapy to .10 using:
$ easy_install -U Scrapy
Read-on for solution attempts...
1. Received this error after installing Scrapy .10
$ scrapy
Traceback (most recent call last):
File "/usr/local/bin/scrapy", line 5, in <module>
pkg_resources.run_script('Scrapy==0.10', 'scrapy')
File "/System/Library/Frameworks/Python.framework/Versions/2.5/
Extras/lib/python/pkg_resources.py", line 442, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/System/Library/Frameworks/Python.framework/Versions/2.5/
Extras/lib/python/pkg_resources.py", line 1160, in run_script
execfile(script_filename, namespace, namespace)
File "/Library/Python/2.5/site-packages/Scrapy-0.10-py2.5.egg/EGG-
INFO/scripts/scrapy", line 4, in <module>
execute()
File "/Library/Python/2.5/site-packages/Scrapy-0.10-py2.5.egg/scrapy/
cmdline.py", line 110, in execute
cmds = _get_commands_dict(inproject)
File "/Library/Python/2.5/site-packages/Scrapy-0.10-py2.5.egg/scrapy/
cmdline.py", line 35, in _get_commands_dict
cmds = _get_commands_from_module('scrapy.commands', inproject)
File "/Library/Python/2.5/site-packages/Scrapy-0.10-py2.5.egg/scrapy/
cmdline.py", line 26, in _get_commands_from_module
command = getattr(__import__(modname, {}, {}, [cmdname]),
'Command', None)
File "/Library/Python/2.5/site-packages/Scrapy-0.10-py2.5.egg/scrapy/
commands/shell.py", line 8, in <module>
from scrapy.shell import Shell
File "/Library/Python/2.5/site-packages/Scrapy-0.10-py2.5.egg/scrapy/
shell.py", line 15, in <module>
from scrapy.selector import XPathSelector, XmlXPathSelector,
HtmlXPathSelector
File "/Library/Python/2.5/site-packages/Scrapy-0.10-py2.5.egg/scrapy/
selector/__init__.py", line 7, in <module>
import libxml2
ImportError: No module named libxml2
2. Realized that the installation script defaults to Python25 (maybe
because it found only the original MacOS version and not the version
referenced by /usr/local/bin/scrapy -> python26?). So edited the
scrapy-ctl.py file:
$ sudo nano /opt/local/Library/Frameworks/Python.framework/Versions/
2.6/bin/scrapy-ctl.py
...and changed...
#!/System/Library/Frameworks/Python.framework/Versions/2.5/Resources/
Python.app/Contents/MacOS/Python
# EASY-INSTALL-SCRIPT: 'Scrapy==0.10','scrapy'
__requires__ = 'Scrapy==0.10'
import pkg_resources
pkg_resources.run_script('Scrapy==0.10', 'scrapy')
...to...
#!/opt/local/Library/Frameworks/Python.framework/Versions/2.6/bin/
python2.6
# EASY-INSTALL-SCRIPT: 'Scrapy==0.10','scrapy'
__requires__ = 'Scrapy==0.10'
import pkg_resources
pkg_resources.run_script('Scrapy==0.10', 'scrapy')
4. But received this error..
$ scrapy
Traceback (most recent call last):
File "/usr/local/bin/scrapy", line 4, in <module>
import pkg_resources
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/
lib/python2.6/site-packages/pkg_resources.py", line 2675, in <module>
parse_requirements(__requires__), Environment()
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/
lib/python2.6/site-packages/pkg_resources.py", line 552, in resolve
raise DistributionNotFound(req)
pkg_resources.DistributionNotFound: Scrapy==0.10
5. At this point I was stuck and since I'm working today I decided to
roll back and delete Scrapy.10
$ sudo rm /opt/local/Library/Frameworks/Python.framework/Versions/2.6/
lib/python2.6/site-packages/scrapy
And use the instructions I got it all working with the first time...
http://stackoverflow.com/questions/2353957/do-i-need-32bit-libxml2-for-python-on-snow-leopard
$ sudo /opt/local/bin/easy_install-2.6 scrapy
So back where I started before trying to update, but hopefully these
comments will be of use.
I am running Python26 with Macports on a MBP with 10.5. I found some
errors after updating Scrapy to .10 using:
$ easy_install -U Scrapy
Read-on for solution attempts...
1. Received this error after installing Scrapy .10
$ scrapy
Traceback (most recent call last):
File "/usr/local/bin/scrapy", line 5, in <module>
pkg_resources.run_script('Scrapy==0.10', 'scrapy')
File "/System/Library/Frameworks/Python.framework/Versions/2.5/
Extras/lib/python/pkg_resources.py", line 442, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/System/Library/Frameworks/Python.framework/Versions/2.5/
Extras/lib/python/pkg_resources.py", line 1160, in run_script
execfile(script_filename, namespace, namespace)
File "/Library/Python/2.5/site-packages/Scrapy-0.10-py2.5.egg/EGG-
INFO/scripts/scrapy", line 4, in <module>
execute()
File "/Library/Python/2.5/site-packages/Scrapy-0.10-py2.5.egg/scrapy/
cmdline.py", line 110, in execute
cmds = _get_commands_dict(inproject)
File "/Library/Python/2.5/site-packages/Scrapy-0.10-py2.5.egg/scrapy/
cmdline.py", line 35, in _get_commands_dict
cmds = _get_commands_from_module('scrapy.commands', inproject)
File "/Library/Python/2.5/site-packages/Scrapy-0.10-py2.5.egg/scrapy/
cmdline.py", line 26, in _get_commands_from_module
command = getattr(__import__(modname, {}, {}, [cmdname]),
'Command', None)
File "/Library/Python/2.5/site-packages/Scrapy-0.10-py2.5.egg/scrapy/
commands/shell.py", line 8, in <module>
from scrapy.shell import Shell
File "/Library/Python/2.5/site-packages/Scrapy-0.10-py2.5.egg/scrapy/
shell.py", line 15, in <module>
from scrapy.selector import XPathSelector, XmlXPathSelector,
HtmlXPathSelector
File "/Library/Python/2.5/site-packages/Scrapy-0.10-py2.5.egg/scrapy/
selector/__init__.py", line 7, in <module>
import libxml2
ImportError: No module named libxml2
2. Realized that the installation script defaults to Python25 (maybe
because it found only the original MacOS version and not the version
referenced by /usr/local/bin/scrapy -> python26?). So edited the
scrapy-ctl.py file:
$ sudo nano /opt/local/Library/Frameworks/Python.framework/Versions/
2.6/bin/scrapy-ctl.py
...and changed...
#!/System/Library/Frameworks/Python.framework/Versions/2.5/Resources/
Python.app/Contents/MacOS/Python
# EASY-INSTALL-SCRIPT: 'Scrapy==0.10','scrapy'
__requires__ = 'Scrapy==0.10'
import pkg_resources
pkg_resources.run_script('Scrapy==0.10', 'scrapy')
...to...
#!/opt/local/Library/Frameworks/Python.framework/Versions/2.6/bin/
python2.6
# EASY-INSTALL-SCRIPT: 'Scrapy==0.10','scrapy'
__requires__ = 'Scrapy==0.10'
import pkg_resources
pkg_resources.run_script('Scrapy==0.10', 'scrapy')
4. But received this error..
$ scrapy
Traceback (most recent call last):
File "/usr/local/bin/scrapy", line 4, in <module>
import pkg_resources
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/
lib/python2.6/site-packages/pkg_resources.py", line 2675, in <module>
parse_requirements(__requires__), Environment()
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/
lib/python2.6/site-packages/pkg_resources.py", line 552, in resolve
raise DistributionNotFound(req)
pkg_resources.DistributionNotFound: Scrapy==0.10
5. At this point I was stuck and since I'm working today I decided to
roll back and delete Scrapy.10
$ sudo rm /opt/local/Library/Frameworks/Python.framework/Versions/2.6/
lib/python2.6/site-packages/scrapy
And use the instructions I got it all working with the first time...
http://stackoverflow.com/questions/2353957/do-i-need-32bit-libxml2-for-python-on-snow-leopard
$ sudo /opt/local/bin/easy_install-2.6 scrapy
So back where I started before trying to update, but hopefully these
comments will be of use.
ow3n
Sep 13, 2010, 11:38:08 AM9/13/10
to scrapy-users
To complete the rollback I also had to remove:
/usr/local/bin/scrapy-ctl.py
...which was pointing to:
/Library/Frameworks/scrapy-trunk/bin/scrapy-ctl.py
...and created a new symlink to the correct (macports) Scrapy.
sudo ln -s /opt/local/Library/Frameworks/Python.framework/Versions/2.6/
bin/scrapy-ctl.py /usr/local/bin/scrapy-ctl.py
> And use the instructions I got it all working with the first time...http://stackoverflow.com/questions/2353957/do-i-need-32bit-libxml2-fo...
/usr/local/bin/scrapy-ctl.py
...which was pointing to:
/Library/Frameworks/scrapy-trunk/bin/scrapy-ctl.py
...and created a new symlink to the correct (macports) Scrapy.
sudo ln -s /opt/local/Library/Frameworks/Python.framework/Versions/2.6/
bin/scrapy-ctl.py /usr/local/bin/scrapy-ctl.py
Pablo Hoffman
Sep 15, 2010, 4:09:23 PM9/15/10
to scrapy...@googlegroups.com
So the Mac installation instructions don't work or is this because you're using
a custom environment (Macports)?
a custom environment (Macports)?
Unfortunately, I don't have a Mac and can't test the installation steps, but it
would be great if anyone using Scrapy on Mac can shed a light on this, and what
you think would be the easier and more reliable way for installing Scrapy on
Mac.
Pablo.
> --
> You received this message because you are subscribed to the Google Groups "scrapy-users" group.
> To post to this group, send email to scrapy...@googlegroups.com.
> To unsubscribe from this group, send email to scrapy-users...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/scrapy-users?hl=en.
Kwan Lim
Sep 16, 2010, 11:14:25 AM9/16/10
to scrapy...@googlegroups.com
I had a hard time installing the previous version of scrappy as well. The problem was that the python version and the required libs that come installed on Mac OS X 10.5 (i think) had to be updated.
I ended up using Mac Ports.
I ended up using Mac Ports.
ow3n
Sep 17, 2010, 9:31:00 PM9/17/10
to scrapy-users
Correct, I was unable to install the Mac version using the
instructions on the Scrapy site.
It seems folks mainly have trouble with libxml2 for a variety of
reasons
http://www.google.com/search?q=+site:stackoverflow.com+install+scrapy+mac
http://stackoverflow.com/questions/3506327/problems-installing-libxml2-on-mac-os-x
Most of the answers on stackoverflow et al have pointed to installing/
maintaining python and all required libraries using macports.
I just compiled installation instructions for installing Scrapy using
Macports on my blog here. Feel free to republish.
http://owenmundy.com/blog/2010/09/how-to-install-scrapy-with-macports-full-version/
Owen
On Sep 15, 10:09 pm, Pablo Hoffman <pablohoff...@gmail.com> wrote:
> So the Mac installation instructions don't work or is this because you're using
> a custom environment (Macports)?
>
> Unfortunately, I don't have a Mac and can't test the installation steps, but it
> would be great if anyone using Scrapy on Mac can shed a light on this, and what
> you think would be the easier and more reliable way for installing Scrapy on
> Mac.
>
> Pablo.
>
>
>
> On Mon, Sep 13, 2010 at 08:38:08AM -0700, ow3n wrote:
> > To complete the rollback I also had to remove:
>
> > /usr/local/bin/scrapy-ctl.py
>
> > ...which was pointing to:
>
> > /Library/Frameworks/scrapy-trunk/bin/scrapy-ctl.py
>
> > ...and created a new symlink to the correct (macports) Scrapy.
>
> > sudo ln -s /opt/local/Library/Frameworks/Python.framework/Versions/2.6/
> > bin/scrapy-ctl.py /usr/local/bin/scrapy-ctl.py
>
> > > � � pkg_resources.run_script('Scrapy==0.10', 'scrapy')
> > > � File "/System/Library/Frameworks/Python.framework/Versions/2.5/
> > > � File "/System/Library/Frameworks/Python.framework/Versions/2.5/
> > > � File "/Library/Python/2.5/site-packages/Scrapy-0.10-py2.5.egg/EGG-
> > > � File "/Library/Python/2.5/site-packages/Scrapy-0.10-py2.5.egg/scrapy/
> > > � File "/Library/Python/2.5/site-packages/Scrapy-0.10-py2.5.egg/scrapy/
> > > � File "/Library/Python/2.5/site-packages/Scrapy-0.10-py2.5.egg/scrapy/
> > > 'Command', None)
> > > � File "/Library/Python/2.5/site-packages/Scrapy-0.10-py2.5.egg/scrapy/
> > > � File "/Library/Python/2.5/site-packages/Scrapy-0.10-py2.5.egg/scrapy/
> > > HtmlXPathSelector
> > > � File "/Library/Python/2.5/site-packages/Scrapy-0.10-py2.5.egg/scrapy/
> > > � � import pkg_resources
> > > � File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/
> > > � File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/
instructions on the Scrapy site.
It seems folks mainly have trouble with libxml2 for a variety of
reasons
http://www.google.com/search?q=+site:stackoverflow.com+install+scrapy+mac
http://stackoverflow.com/questions/3506327/problems-installing-libxml2-on-mac-os-x
Most of the answers on stackoverflow et al have pointed to installing/
maintaining python and all required libraries using macports.
I just compiled installation instructions for installing Scrapy using
Macports on my blog here. Feel free to republish.
http://owenmundy.com/blog/2010/09/how-to-install-scrapy-with-macports-full-version/
Owen
On Sep 15, 10:09 pm, Pablo Hoffman <pablohoff...@gmail.com> wrote:
> So the Mac installation instructions don't work or is this because you're using
> a custom environment (Macports)?
>
> Unfortunately, I don't have a Mac and can't test the installation steps, but it
> would be great if anyone using Scrapy on Mac can shed a light on this, and what
> you think would be the easier and more reliable way for installing Scrapy on
> Mac.
>
> Pablo.
>
>
>
> On Mon, Sep 13, 2010 at 08:38:08AM -0700, ow3n wrote:
> > To complete the rollback I also had to remove:
>
> > /usr/local/bin/scrapy-ctl.py
>
> > ...which was pointing to:
>
> > /Library/Frameworks/scrapy-trunk/bin/scrapy-ctl.py
>
> > ...and created a new symlink to the correct (macports) Scrapy.
>
> > sudo ln -s /opt/local/Library/Frameworks/Python.framework/Versions/2.6/
> > bin/scrapy-ctl.py /usr/local/bin/scrapy-ctl.py
>
> > On Sep 13, 10:54�am, ow3n <omu...@gmail.com> wrote:
> > > Hi there,
>
> > > I am running Python26 with Macports on a MBP with 10.5. I found some
> > > errors after updating Scrapy to .10 using:
>
> > > $ easy_install -U Scrapy
>
> > > Read-on for solution attempts...
>
> > > 1. Received this error after installing Scrapy .10
>
> > > $ scrapy
> > > Traceback (most recent call last):
> > > � File "/usr/local/bin/scrapy", line 5, in <module>
> > > Hi there,
>
> > > I am running Python26 with Macports on a MBP with 10.5. I found some
> > > errors after updating Scrapy to .10 using:
>
> > > $ easy_install -U Scrapy
>
> > > Read-on for solution attempts...
>
> > > 1. Received this error after installing Scrapy .10
>
> > > $ scrapy
> > > Traceback (most recent call last):
> > > � � pkg_resources.run_script('Scrapy==0.10', 'scrapy')
> > > � File "/System/Library/Frameworks/Python.framework/Versions/2.5/
> > > Extras/lib/python/pkg_resources.py", line 442, in run_script
> > > � � self.require(requires)[0].run_script(script_name, ns)
> > > � File "/System/Library/Frameworks/Python.framework/Versions/2.5/
> > > Extras/lib/python/pkg_resources.py", line 1160, in run_script
> > > � � execfile(script_filename, namespace, namespace)
> > > � File "/Library/Python/2.5/site-packages/Scrapy-0.10-py2.5.egg/EGG-
> > > INFO/scripts/scrapy", line 4, in <module>
> > > � � execute()
> > > � File "/Library/Python/2.5/site-packages/Scrapy-0.10-py2.5.egg/scrapy/
> > > cmdline.py", line 110, in execute
> > > � � cmds = _get_commands_dict(inproject)
> > > � File "/Library/Python/2.5/site-packages/Scrapy-0.10-py2.5.egg/scrapy/
> > > cmdline.py", line 35, in _get_commands_dict
> > > � � cmds = _get_commands_from_module('scrapy.commands', inproject)
> > > � File "/Library/Python/2.5/site-packages/Scrapy-0.10-py2.5.egg/scrapy/
> > > cmdline.py", line 26, in _get_commands_from_module
> > > � � command = getattr(__import__(modname, {}, {}, [cmdname]),
> > > 'Command', None)
> > > � File "/Library/Python/2.5/site-packages/Scrapy-0.10-py2.5.egg/scrapy/
> > > commands/shell.py", line 8, in <module>
> > > � � from scrapy.shell import Shell
> > > � File "/Library/Python/2.5/site-packages/Scrapy-0.10-py2.5.egg/scrapy/
> > > shell.py", line 15, in <module>
> > > � � from scrapy.selector import XPathSelector, XmlXPathSelector,
> > > HtmlXPathSelector
> > > � File "/Library/Python/2.5/site-packages/Scrapy-0.10-py2.5.egg/scrapy/
> > > selector/__init__.py", line 7, in <module>
> > > � � import libxml2
> > > ImportError: No module named libxml2
>
> > > 2. Realized that the installation script defaults to Python25 (maybe
> > > because it found only the original MacOS version and not the version
> > > referenced by /usr/local/bin/scrapy -> python26?). So edited the
> > > scrapy-ctl.py file:
>
> > > $ sudo nano /opt/local/Library/Frameworks/Python.framework/Versions/
> > > 2.6/bin/scrapy-ctl.py
>
> > > ...and changed...
>
> > > #!/System/Library/Frameworks/Python.framework/Versions/2.5/Resources/
> > > Python.app/Contents/MacOS/Python
> > > # EASY-INSTALL-SCRIPT: 'Scrapy==0.10','scrapy'
> > > __requires__ = 'Scrapy==0.10'
> > > import pkg_resources
> > > pkg_resources.run_script('Scrapy==0.10', 'scrapy')
>
> > > ...to...
>
> > > #!/opt/local/Library/Frameworks/Python.framework/Versions/2.6/bin/
> > > python2.6
> > > # EASY-INSTALL-SCRIPT: 'Scrapy==0.10','scrapy'
> > > __requires__ = 'Scrapy==0.10'
> > > import pkg_resources
> > > pkg_resources.run_script('Scrapy==0.10', 'scrapy')
>
> > > 4. But received this error..
>
> > > $ scrapy
> > > Traceback (most recent call last):
> > > � File "/usr/local/bin/scrapy", line 4, in <module>
>
> > > 2. Realized that the installation script defaults to Python25 (maybe
> > > because it found only the original MacOS version and not the version
> > > referenced by /usr/local/bin/scrapy -> python26?). So edited the
> > > scrapy-ctl.py file:
>
> > > $ sudo nano /opt/local/Library/Frameworks/Python.framework/Versions/
> > > 2.6/bin/scrapy-ctl.py
>
> > > ...and changed...
>
> > > #!/System/Library/Frameworks/Python.framework/Versions/2.5/Resources/
> > > Python.app/Contents/MacOS/Python
> > > # EASY-INSTALL-SCRIPT: 'Scrapy==0.10','scrapy'
> > > __requires__ = 'Scrapy==0.10'
> > > import pkg_resources
> > > pkg_resources.run_script('Scrapy==0.10', 'scrapy')
>
> > > ...to...
>
> > > #!/opt/local/Library/Frameworks/Python.framework/Versions/2.6/bin/
> > > python2.6
> > > # EASY-INSTALL-SCRIPT: 'Scrapy==0.10','scrapy'
> > > __requires__ = 'Scrapy==0.10'
> > > import pkg_resources
> > > pkg_resources.run_script('Scrapy==0.10', 'scrapy')
>
> > > 4. But received this error..
>
> > > $ scrapy
> > > Traceback (most recent call last):
> > > � � import pkg_resources
> > > � File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/
> > > lib/python2.6/site-packages/pkg_resources.py", line 2675, in <module>
> > > � � parse_requirements(__requires__), Environment()
> > > � File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/
> > > lib/python2.6/site-packages/pkg_resources.py", line 552, in resolve
> > > � � raise DistributionNotFound(req)
Pablo Hoffman
Sep 19, 2010, 9:29:36 PM9/19/10
to scrapy...@googlegroups.com
On Fri, Sep 17, 2010 at 06:31:00PM -0700, ow3n wrote:
> Most of the answers on stackoverflow et al have pointed to installing/
> maintaining python and all required libraries using macports.
>
> I just compiled installation instructions for installing Scrapy using
> Macports on my blog here. Feel free to republish.
> http://owenmundy.com/blog/2010/09/how-to-install-scrapy-with-macports-full-version/
> Most of the answers on stackoverflow et al have pointed to installing/
> maintaining python and all required libraries using macports.
>
> I just compiled installation instructions for installing Scrapy using
> Macports on my blog here. Feel free to republish.
> http://owenmundy.com/blog/2010/09/how-to-install-scrapy-with-macports-full-version/
Thanks Owen, this really helps.
A couple of suggestions about your guide:
On point 7, you mention installing simplejson port (py26-simplejson). Scrapy
doesn't need simplejson when running on Python 2.6, as it will use the standard
library "json" module instead. So I think you can remove py26-simplejson as a
minor simplification.
On point 11 and point 12, you're referring to scrapy-ctl.py which is the old
script used to control scrapy in versions prior to 0.10. Now the script is just
called "scrapy". Also, it's not clear to me why point 11 is needed. For Scrapy
0.10 and above, I think point 11 and 12 could be merged into one single point,
as follows:
* Make the scrapy command available in the shell
$ sudo ln -s /opt/local/Library/Frameworks/Python.framework/Versions/2.6/bin/scrapy-ctl.py /usr/local/bin/scrapy
The rest of the guide looks good, do you mind if I update the Scrapy
installation guide based on yours?
Btw, I managed to get a Mac OS X running on a VirtualBox, to test your
instructions, but I don't have an Apple Developer ID for getting XCode, which
is required. I think it'd be great if we could package Scrapy in a dmg/pkg Mac
installer, but I'm not sure how much work would that involve. After researching
a bit I got to this Twisted ticket that mentions using a tool called Platypus:
http://twistedmatrix.com/trac/ticket/3279
Any thoughts?
Pablo.
ow3n
Sep 21, 2010, 3:53:43 AM9/21/10
to scrapy-users
Hi Pablo,
Thanks for the feedback. I should have mentioned I created these with .
09 and then updated them when trying .10 so there still may be some
residual .09 comments.
I've incorporated your comments into my instructions, with one
question. On this note:
* Make the scrapy command available in the shell
$ sudo ln -s /opt/local/Library/Frameworks/Python.framework/Versions/
2.6/bin/scrapy-ctl. py /usr/local/bin/scrapy
should the link point to this file instead?
$ sudo ln -s /opt/local/Library/Frameworks/Python.framework/Versions/
2.6/bin/scrapy
Xcode requires a free registration
http://developer.apple.com/programs/register/
and is required to install macports. It is >900M so I don't think
packaging the whole thing is an option. Maybe a Macport of scrapy
would be useful?
http://www.macports.org/install.php
Yes, please feel free to use these instructions in the guide.
Best,
Owen
On Sep 20, 3:29 am, Pablo Hoffman <pablohoff...@gmail.com> wrote:
> On Fri, Sep 17, 2010 at 06:31:00PM -0700, ow3n wrote:
> > Most of the answers on stackoverflow et al have pointed to installing/
> > maintaining python and all required libraries using macports.
>
> > I just compiled installation instructions for installing Scrapy using
> > Macports on my blog here. Feel free to republish.
> >http://owenmundy.com/blog/2010/09/how-to-install-scrapy-with-macports...
Thanks for the feedback. I should have mentioned I created these with .
09 and then updated them when trying .10 so there still may be some
residual .09 comments.
I've incorporated your comments into my instructions, with one
question. On this note:
* Make the scrapy command available in the shell
$ sudo ln -s /opt/local/Library/Frameworks/Python.framework/Versions/
2.6/bin/scrapy-ctl. py /usr/local/bin/scrapy
$ sudo ln -s /opt/local/Library/Frameworks/Python.framework/Versions/
2.6/bin/scrapy
Xcode requires a free registration
http://developer.apple.com/programs/register/
and is required to install macports. It is >900M so I don't think
packaging the whole thing is an option. Maybe a Macport of scrapy
would be useful?
http://www.macports.org/install.php
Yes, please feel free to use these instructions in the guide.
Best,
Owen
On Sep 20, 3:29 am, Pablo Hoffman <pablohoff...@gmail.com> wrote:
> On Fri, Sep 17, 2010 at 06:31:00PM -0700, ow3n wrote:
> > Most of the answers on stackoverflow et al have pointed to installing/
> > maintaining python and all required libraries using macports.
>
> > I just compiled installation instructions for installing Scrapy using
> > Macports on my blog here. Feel free to republish.
>
> Thanks Owen, this really helps.
>
> A couple of suggestions about your guide:
>
> On point 7, you mention installing simplejson port (py26-simplejson). Scrapy
> doesn't need simplejson when running on Python 2.6, as it will use the standard
> library "json" module instead. So I think you can remove py26-simplejson as a
> minor simplification.
>
> On point 11 and point 12, you're referring to scrapy-ctl.py which is the old
> script used to control scrapy in versions prior to 0.10. Now the script is just
> called "scrapy". Also, it's not clear to me why point 11 is needed. For Scrapy
> 0.10 and above, I think point 11 and 12 could be merged into one single point,
> as follows:
>
> * Make the scrapy command available in the shell
> $ sudo ln -s /opt/local/Library/Frameworks/Python.framework/Versions/2.6/bin/scrapy-ctl. py /usr/local/bin/scrapy
> Thanks Owen, this really helps.
>
> A couple of suggestions about your guide:
>
> On point 7, you mention installing simplejson port (py26-simplejson). Scrapy
> doesn't need simplejson when running on Python 2.6, as it will use the standard
> library "json" module instead. So I think you can remove py26-simplejson as a
> minor simplification.
>
> On point 11 and point 12, you're referring to scrapy-ctl.py which is the old
> script used to control scrapy in versions prior to 0.10. Now the script is just
> called "scrapy". Also, it's not clear to me why point 11 is needed. For Scrapy
> 0.10 and above, I think point 11 and 12 could be merged into one single point,
> as follows:
>
> * Make the scrapy command available in the shell
Pablo Hoffman
Sep 21, 2010, 2:37:31 PM9/21/10
to scrapy...@googlegroups.com
On Tue, Sep 21, 2010 at 12:53:43AM -0700, ow3n wrote:
> I've incorporated your comments into my instructions, with one
> question. On this note:
>
> * Make the scrapy command available in the shell
> $ sudo ln -s /opt/local/Library/Frameworks/Python.framework/Versions/
> 2.6/bin/scrapy-ctl. py /usr/local/bin/scrapy
>
> should the link point to this file instead?
>
> $ sudo ln -s /opt/local/Library/Frameworks/Python.framework/Versions/
> 2.6/bin/scrapy
> I've incorporated your comments into my instructions, with one
> question. On this note:
>
> * Make the scrapy command available in the shell
> $ sudo ln -s /opt/local/Library/Frameworks/Python.framework/Versions/
> 2.6/bin/scrapy-ctl. py /usr/local/bin/scrapy
>
> should the link point to this file instead?
>
> $ sudo ln -s /opt/local/Library/Frameworks/Python.framework/Versions/
> 2.6/bin/scrapy
Correct.
> Xcode requires a free registration
> http://developer.apple.com/programs/register/
> and is required to install macports. It is >900M so I don't think
> packaging the whole thing is an option. Maybe a Macport of scrapy
> would be useful?
> http://www.macports.org/install.php
Thanks, I'm already downloading it.
> Yes, please feel free to use these instructions in the guide.
Thanks, I'll update the Mac OS X install instructions next weekend.
Pablo.
Reply all
Reply to author
Forward
0 new messages