Scrapy Installation Woes

[Peter]

On Ubuntu 14.04.5.  Python newbie.  Trying to install scrapy via Python 1.3.0 documentation.  Using Python 3.  Google has been guiding me, but I've hit a brick wall:

$ scrapy --version
Traceback (most recent call last):
  File "/usr/local/bin/scrapy", line 7, in <module>
    from scrapy.cmdline import execute
  File "/usr/local/lib/python3.4/dist-packages/scrapy/cmdline.py", line 9, in <module>
    from scrapy.crawler import CrawlerProcess
  File "/usr/local/lib/python3.4/dist-packages/scrapy/crawler.py", line 7, in <module>
    from twisted.internet import reactor, defer
  File "/usr/local/lib/python3.4/dist-packages/twisted/internet/reactor.py", line 38, in <module>
    from twisted.internet import default
  File "/usr/local/lib/python3.4/dist-packages/twisted/internet/default.py", line 56, in <module>
    install = _getInstallFunction(platform)
  File "/usr/local/lib/python3.4/dist-packages/twisted/internet/default.py", line 44, in _getInstallFunction
    from twisted.internet.epollreactor import install
  File "/usr/local/lib/python3.4/dist-packages/twisted/internet/epollreactor.py", line 24, in <module>
    from twisted.internet import posixbase
  File "/usr/local/lib/python3.4/dist-packages/twisted/internet/posixbase.py", line 18, in <module>
    from twisted.internet import error, udp, tcp
  File "/usr/local/lib/python3.4/dist-packages/twisted/internet/tcp.py", line 28, in <module>
    from twisted.internet._newtls import (
  File "/usr/local/lib/python3.4/dist-packages/twisted/internet/_newtls.py", line 21, in <module>
    from twisted.protocols.tls import TLSMemoryBIOFactory, TLSMemoryBIOProtocol
  File "/usr/local/lib/python3.4/dist-packages/twisted/protocols/tls.py", line 65, in <module>
    from twisted.internet._sslverify import _setAcceptableProtocols
  File "/usr/local/lib/python3.4/dist-packages/twisted/internet/_sslverify.py", line 204, in <module>
    verifyHostname, VerificationError = _selectVerifyImplementation(OpenSSL)
  File "/usr/local/lib/python3.4/dist-packages/twisted/internet/_sslverify.py", line 179, in _selectVerifyImplementation
    from service_identity import VerificationError
  File "/usr/local/lib/python3.4/dist-packages/service_identity/__init__.py", line 7, in <module>
    from . import pyopenssl
  File "/usr/local/lib/python3.4/dist-packages/service_identity/pyopenssl.py", line 14, in <module>
    from .exceptions import SubjectAltNameWarning
  File "/usr/local/lib/python3.4/dist-packages/service_identity/exceptions.py", line 21, in <module>
    @attr.s
AttributeError: 'module' object has no attribute 's'

I installed virtualenv and created a virtual environment in ~/ENV-Scrapy.  My intention was to install Scrapy and its dependencies into that environment, but I really don't know what I'm doing, and I'm wondering if that's what happened.  I'm also wondering if I'm invoking scapy correctly.

Will some kind soul help me figure out how to get a healthy Scrapy installation?  Many thanks!

[vanam raghu]

Can you try running pip install scrapy i think this should install scrapy 1.3 version 

if you want 1.2.2 version, try running this command pip install scrapy==1.2.2, let me know what you see, on ubuntu chances are that you are missing some dependencies while running the scrapy

[Peter]

Sorry -- I was expecting to be "subscribed" to this thread.  Apparently, I'm not, for some reason.

Wouldn't "pip install scrappy" install scrapy for Python 2?   I'm trying to replace Perl with Python as my "go to scripting language", and it seems like Python 3 is what I should be targeting.

Anyway, I tried your suggestion with both pip and pip3.  I ran the test-spider in the official docs, and got the same strange AttributeError.  Is there anything you can glean from the posted error message?

[Peter]

I figured it out.  The clue was the error message (as always!) showing packages being run from /usr/local.

The problem was that the Scrapy docs didn't outline how to use virtual environments well enough.  I read through the docs for virtual environments, and learned what I needed in under 10 minutes.  Installed Scrapy correctly with the virtual environment, and life is good.

[Paul Tremberth]

Hey Peter,

would you mind opening an issue on Scrapy explaining what you think is missing in Scrapy docs on virtualenvs?

We don't want to copy the docs from virtualenv but we can certainly explain a bit more.

Thanks in advance.

/Paul.

--
You received this message because you are subscribed to the Google Groups "scrapy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scrapy-users+unsubscribe@googlegroups.com.
To post to this group, send email to scrapy...@googlegroups.com.
Visit this group at https://groups.google.com/group/scrapy-users.
For more options, visit https://groups.google.com/d/optout.

[Peter]

Hi Paul, I'll try to get to it after work, but having a 2 year old makes that a bit difficult.   :)    I do have some suggestions to pass on:

1. The docs should *definitely* mention that if you see /usr/local/anything in the error messages, you're not using the virtual environment correctly.

2. In the docs, if you look at the section "Ubuntu 12.04 or above" you can see exactly why I was confused.

Inside a virtual env, you can install Scrapy with pip

after that:

pip install scrapy

I think for the bare bones beginner, like me, it would be better to say something like:

1. First install virtual env.

$ pip3 install virtualenv   (maybe make a note of pip vs pip3 for newbies?)

2. Now create a project directory.
$ mkdir quoteProject

3. Within that project directory, create your virtual environment.  It doesn't matter what you call the virtual environment.  Maybe make note of the -p switch for virtualenv to distinguish between python2 and python3?  And use a crazy name to demonstrate you can name the environment anything you want.

$ cd quoteProject
$ virtualenv -p /usr/bin/python3.4 myVE

4. Source the virtual environment.  You'll know you did it correctly if your prompt looks like "blah".  And install scrapy.

$ source myVE/bin/activate
(myVE) $ pip3 install scrapy

[ lots of stuff ]



3. Honestly, I didn't need much to understand how to use virtual environments.  The most helpful site I encountered was this:

https://realpython.com/blog/python/python-virtual-environments-a-primer/

It's very short.  Scrapy docs did a good job explaining why a virtual environment is needed, so the only thing *I* needed were these two short sections on that site:

Using virtual environments

How does a virtual environment work?

And that's it.  The other sections I read, and maybe they'll be useful in the future, but for the purposes of getting me up and running with scrapy, I just needed those two sections.   I think half a page in the scrapy docs should do it (maybe what I wrote above would be all that's needed for Linux users).


Oh, dear.  I told myself I wouldn't do this on company time.  OK, well, it's done.  If you think this will be useful, I'll try to contribute it.   How do I do that?

To unsubscribe from this group and stop receiving emails from it, send an email to scrapy-users...@googlegroups.com.