BS4-b10 install problems
skippyV
Installed lxml for both Python versions with:
easy_install --allow-hosts=lxml.de,*.python.org lxml==2.3
Installed html5lib for Python 2.x with:
python setup.py install
Couldn't install html5lib for Python 3.x because its apparently
unsupported.
Installed BS4-b10 for both Python versions with:
python setup.py install
test script from http://readthedocs.org/docs/beautiful-soup-4/en/latest/
using the "three sisters" test with following changes:
from bs4 import BeautifulSoup
#soup = BeautifulSoup(html_doc) # test 1
#soup = BeautifulSoup(html_doc, "lxml") # test 2
soup = BeautifulSoup(html_doc, "html5lib") # test 3
For all 3 tests for Python 2.x the same error:
ImportError: No module named html.parser
For Python 3.x tests 1 and 2, the output was printed but without
the newline characters.
I have no global environment variables set in regards to Python.
I set my PATH to include the Python home dir locally when I need it.
As a sanity check I tried BeautifulSoup 3.2 with Python 2.x and the
test file in the docs and it worked as expected.
Suggestions?
Leonard Richardson
You've installed the Python 3 version of Beautiful Soup in a place
where it's being picked up by Python 2.
Leonard
> --
> You received this message because you are subscribed to the Google Groups "beautifulsoup" group.
> To post to this group, send email to beauti...@googlegroups.com.
> To unsubscribe from this group, send email to beautifulsou...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/beautifulsoup?hl=en.
>
skippyV
equation.
Uninstalled everything and did the following:
1) install python 3.2.2
2) install distribute-0.6.24 (with "python setup.py install")
3) install lxml-2.3-py3.2-win32.egg (with "easy_install lxml-2.3-
py3.2-win32.egg")
4) install beautifulsoup4-4.0.0b10 (with "easy_install
beautifulsoup4")
NOT with "python setup.py install" from within
beautifulsoup4-4.0.0b10 dir
For step 4 I was using the "setup.py" method. This resulted in a few
minor errors in:
Lib\site-packages\bs4\builder\_htmlparser.py
and
Lib\site-packages\bs4\tests\test_html5lib.py
which were easy enough to fix.
But then when I ran the test I'd get
ImportError: No module named HTMLParser
Which was different from the earlier post.
But once I repeated the uninstall/install process and used
"easy_install beautifulsoup4"
to pull down and install b10 - the problem went away.
So is there a discrepancy between the code pulled down and the tarball
at
http://www.crummy.com/software/BeautifulSoup/bs4/download/4.0/
?
On Mar 6, 11:55 am, Leonard Richardson <leona...@segfault.org> wrote:
> Your problem is the same as this one:
>
> You've installed the Python 3 version of Beautiful Soup in a place
> where it's being picked up by Python 2.
>
> Leonard
>
>
>
>
>
>
>
> On Tue, Mar 6, 2012 at 11:48 AM, skippyV <skippyvondr...@gmail.com> wrote:
> > Testing on Windows with Python 2.7.2 and 3.2.2 installed.
>
> > Installed lxml for both Python versions with:
> > easy_install --allow-hosts=lxml.de,*.python.org lxml==2.3
> > Installed html5lib for Python 2.x with:
> > python setup.py install
>
> > Couldn't install html5lib for Python 3.x because its apparently
> > unsupported.
>
> > Installed BS4-b10 for both Python versions with:
> > python setup.py install
>
Leonard Richardson
> minor errors in:
> Lib\site-packages\bs4\builder\_htmlparser.py
> and
> Lib\site-packages\bs4\tests\test_html5lib.py
> which were easy enough to fix.
>
> But then when I ran the test I'd get
> ImportError: No module named HTMLParser
>
> Which was different from the earlier post.
>
> But once I repeated the uninstall/install process and used
> "easy_install beautifulsoup4"
> to pull down and install b10 - the problem went away.
>
> So is there a discrepancy between the code pulled down and the tarball
> at
> http://www.crummy.com/software/BeautifulSoup/bs4/download/4.0/
> ?
"ImportError: No module named HTMLParser" is an error you'll get when
running Python 2 code under Python 3. It's the opposite of the
"ImportError: No module named html.parser" you were getting before. In
Python 3, the HTMLParser module was renamed to html.parser.
The code pulled down and the tarball are identical. However, the
tarball contains Python 2 code. To convert it to Python 3 code, you
need to make sure that when setup.py is run, it's run with the Python
3 interpreter.
easy_install will download the Beautiful Soup tarball and run
setup.py. But if you have both Python 2 and Python 3 installed on your
system, it's a crapshoot whether the command "easy_install" will use
the Python 2 interpreter or the Python 3 interpreter.
Since you are not the first person with this sort of problem on
Windows, I think it must be easy to get into a situation where Python
2 code is installed into the Python 3 library directory, or vice
versa.
For instance, imagine that you tell easy_install to install
beautifulsoup4 into lib\python3.2\site-packages\ but easy_install uses
the Python 2 interpreter. You will download the Beautiful Soup code
(which is written for Python 2) and install it in
lib\python3.2\site-packages/ without converting it to Python 3. This
will give the ImportError you mentioned when you try to use it.
I can imagine other sources of inconsistency, such as the
"easy_install" command running Python 2 while the "python" command
runs Python 3. I don't know exactly what the problem is, but it's
probably something like this.
Here's an idea for avoiding the problem. Hopefully on Windows you have
"easy_install-2.7" and "easy_install-3.2" scripts in the same
directory as "easy_install". (Or whatever versions you're using.)
Hopefully you also have "python-2.7" and "python-3.2" interpreters in
addition to "python". If you run into more problems, I recommend
redoing the installation process, making sure that you always specify
which version of the interpreter you're running.
So instead of "python setup.py" you would run "python-3.2 setup.py".
Instead of "easy_install beautifulsoup4" you would run
"easy_install-3.2 beautifulsoup4".
With this technique you should be able to install easy_install and
beautifulsoup4 under Python 2, then install completely separate
easy_install and beautifulsoup4 under Python 3.
If you still have problems, I'd like to see:
* The commands you used to set everything up, with output if possible
* The first line of the `easy_install` script, which contains the
interpreter (although I don't know if this works on Windows)
* Any other easy_install-* scripts you have, e.g. easy_install-2.7.
* The output of python --version, python2.7 --version, and python3 --version.
* The full path of every copy of BS4's bs4/builder/_htmlparser.py on
your system, and the first 10 lines of each copy. (So that I can see
whether it imports html.parser or HTMLParser.)
Leonard
skippyV
> tarball contains Python 2 code. To convert it to Python 3 code, you
> need to make sure that when setup.py is run, it's run with the Python
> 3 interpreter.
>
> easy_install will download the Beautiful Soup tarball and run
> setup.py. But if you have both Python 2 and Python 3 installed on your
> system, it's a crapshoot whether the command "easy_install" will use
> the Python 2 interpreter or the Python 3 interpreter.
uninstalled both Python versions (using Control Panel->"Uninstall a
program") and deleted all contents of the install directories.
Then I did the following:
1) installed python 3.2.2 (with msi exe)
2) installed distribute-0.6.24
(with "easy_install lxml-2.3-py3.2-win32.egg")
4) installed beautifulsoup4-4.0.0b10
\bs4\builder\_htmlparser.py
\bs4\tests\test_html5lib.py
which appear to be Python 2 code that wasn't translated to Python3.
After manually fixing those 3 small errors and running the test
script:
> beautifulsoup4 into lib\python3.2\site-packages\ but easy_install uses
> the Python 2 interpreter. You will download the Beautiful Soup code
> (which is written for Python 2) and install it in
> lib\python3.2\site-packages/ without converting it to Python 3. This
> will give the ImportError you mentioned when you try to use it.
>
> I can imagine other sources of inconsistency, such as the
> "easy_install" command running Python 2 while the "python" command
> runs Python 3. I don't know exactly what the problem is, but it's
> probably something like this.
> Here's an idea for avoiding the problem. Hopefully on Windows you have
> "easy_install-2.7" and "easy_install-3.2" scripts in the same
> directory as "easy_install". (Or whatever versions you're using.)
> Hopefully you also have "python-2.7" and "python-3.2" interpreters in
> addition to "python". If you run into more problems, I recommend
> redoing the installation process, making sure that you always specify
> which version of the interpreter you're running.
>
> So instead of "python setup.py" you would run "python-3.2 setup.py".
> Instead of "easy_install beautifulsoup4" you would run
> "easy_install-3.2 beautifulsoup4".
the python-lib install process a little better.
But I wanted to make sure you realize that the error conditions above
resulted from a setup that had/has no Python 2.x installed. And the
only difference I did to get the beautifulsoup4 installed correctly
was
to use "easy_install". Using "setup.py" with the tarball produced
the
errors even though I never reinstalled Python 2.x after I removed it
much earlier.
In fact to verify that this error-state is repeatable - this morning
I uninstalled Python 3.x and deleted the directory contents and then
repeated the steps listed above.
I think I'll leave Python 2.x alone for a while. At least until you
(or someone) can figure out why this erroneous install happens. I
agree
this must be a Windows "phenomena". Hopefully someone else with a
Windows setup can verify this.
Skippy
Leonard Richardson
> \bs4\builder\_htmlparser.py
> \bs4\tests\test_html5lib.py
> which appear to be Python 2 code that wasn't translated to Python3.
Can you show me these errors? I think this is the point where the
problem happened.
"No module named HTMLParser" is also caused by Python 2 code that
wasn't translated to Python3. Since you don't have Python 2 installed
and you reported errors during the translation process, it seems like
the logical place to look.
It would also be helpful to see the complete output of your "python
setup.py install".
Leonard
skippyV
which may have been related to Python 2.x. Here is the grab that
I should have posted.
> Can you show me these errors? I think this is the point where the
> problem happened.
C:\PYTHON\Python_libraries\BeautifulSoup
\beautifulsoup4-4.0.0b10>python setup.py install
running install
running build
running build_py
running install_lib
creating C:\PYTHON\Python322\Lib\site-packages\bs4
creating C:\PYTHON\Python322\Lib\site-packages\bs4\builder
copying build\lib\bs4\builder\_html5lib.py -> C:\PYTHON\Python322\Lib
\site-packages\bs4\builder
copying build\lib\bs4\builder\_htmlparser.py -> C:\PYTHON\Python322\Lib
\site-packages\bs4\builder
copying build\lib\bs4\builder\_lxml.py -> C:\PYTHON\Python322\Lib\site-
packages\bs4\builder
copying build\lib\bs4\builder\__init__.py -> C:\PYTHON\Python322\Lib
\site-packages\bs4\builder
copying build\lib\bs4\dammit.py -> C:\PYTHON\Python322\Lib\site-
packages\bs4
copying build\lib\bs4\element.py -> C:\PYTHON\Python322\Lib\site-
packages\bs4
copying build\lib\bs4\testing.py -> C:\PYTHON\Python322\Lib\site-
packages\bs4
creating C:\PYTHON\Python322\Lib\site-packages\bs4\tests
copying build\lib\bs4\tests\test_builder_registry.py -> C:\PYTHON
\Python322\Lib\site-packages\bs4\te
sts
copying build\lib\bs4\tests\test_docs.py -> C:\PYTHON\Python322\Lib
\site-packages\bs4\tests
copying build\lib\bs4\tests\test_html5lib.py -> C:\PYTHON\Python322\Lib
\site-packages\bs4\tests
copying build\lib\bs4\tests\test_htmlparser.py -> C:\PYTHON
\Python322\Lib\site-packages\bs4\tests
copying build\lib\bs4\tests\test_lxml.py -> C:\PYTHON\Python322\Lib
\site-packages\bs4\tests
copying build\lib\bs4\tests\test_soup.py -> C:\PYTHON\Python322\Lib
\site-packages\bs4\tests
copying build\lib\bs4\tests\test_tree.py -> C:\PYTHON\Python322\Lib
\site-packages\bs4\tests
copying build\lib\bs4\tests\__init__.py -> C:\PYTHON\Python322\Lib
\site-packages\bs4\tests
copying build\lib\bs4\__init__.py -> C:\PYTHON\Python322\Lib\site-
packages\bs4
byte-compiling C:\PYTHON\Python322\Lib\site-packages\bs4\builder
\_html5lib.py to _html5lib.pyc
byte-compiling C:\PYTHON\Python322\Lib\site-packages\bs4\builder
\_htmlparser.py to _htmlparser.pyc
File "C:\PYTHON\Python322\Lib\site-packages\bs4\builder
\_htmlparser.py", line 62
except (ValueError, OverflowError), e:
^
SyntaxError: invalid syntax
byte-compiling C:\PYTHON\Python322\Lib\site-packages\bs4\builder
\_lxml.py to _lxml.pyc
byte-compiling C:\PYTHON\Python322\Lib\site-packages\bs4\builder
\__init__.py to __init__.pyc
byte-compiling C:\PYTHON\Python322\Lib\site-packages\bs4\dammit.py to
dammit.pyc
byte-compiling C:\PYTHON\Python322\Lib\site-packages\bs4\element.py to
element.pyc
byte-compiling C:\PYTHON\Python322\Lib\site-packages\bs4\testing.py to
testing.pyc
byte-compiling C:\PYTHON\Python322\Lib\site-packages\bs4\tests
\test_builder_registry.py to test_buil
der_registry.pyc
byte-compiling C:\PYTHON\Python322\Lib\site-packages\bs4\tests
\test_docs.py to test_docs.pyc
byte-compiling C:\PYTHON\Python322\Lib\site-packages\bs4\tests
\test_html5lib.py to test_html5lib.pyc
File "C:\PYTHON\Python322\Lib\site-packages\bs4\tests
\test_html5lib.py", line 8
except ImportError, e:
^
SyntaxError: invalid syntax
byte-compiling C:\PYTHON\Python322\Lib\site-packages\bs4\tests
\test_htmlparser.py to test_htmlparser
.pyc
byte-compiling C:\PYTHON\Python322\Lib\site-packages\bs4\tests
\test_lxml.py to test_lxml.pyc
byte-compiling C:\PYTHON\Python322\Lib\site-packages\bs4\tests
\test_soup.py to test_soup.pyc
byte-compiling C:\PYTHON\Python322\Lib\site-packages\bs4\tests
\test_tree.py to test_tree.pyc
byte-compiling C:\PYTHON\Python322\Lib\site-packages\bs4\tests
\__init__.py to __init__.pyc
byte-compiling C:\PYTHON\Python322\Lib\site-packages\bs4\__init__.py
to __init__.pyc
running install_egg_info
Writing C:\PYTHON\Python322\Lib\site-packages\beautifulsoup4-4.0.0b10-
py3.2.egg-info
================================================
> setup.py install".
output
unfortunately. Just the typical install gui where you can set the
path
for the Python install. Which I set to "C:\PYTHON\Python322\"
Hope this helps.
Skippy
Leonard Richardson
errors you see are just the tip of the iceberg: Python 2 constructs
that are syntactically invalid in Python 3. Things like "import
HTMLParser" are valid in Python 3, but the library doesn't exist
anymore.
The beautifulsoup4 setup.py starts out like this:
try:
from distutils.command.build_py import build_py_2to3 as build_py
except ImportError:
# 2.x
from distutils.command.build_py import build_py
build_py_2to3 is supposed to handle the conversion. setup.py assumes
that if build_py_2to3 is not present, then you're on Python 2 and the
code should not be converted.
That assumption fails in your case, and I think it's because you're
using Distribute instead of distutils. Distribute generally acts like
distutils, but it does not provide build_py_2to3. Instead, you need to
specify "use_2to3=True" as an argument to setup().
So, if I insert this line into setup.py, it should work with either
Distribute or distutils:
=== modified file 'setup.py'
--- setup.py 2012-03-02 13:16:53 +0000
+++ setup.py 2012-03-07 16:50:26 +0000
@@ -13,6 +13,7 @@
url="http://www.crummy.com/software/BeautifulSoup/bs4/",
download_url =
"http://www.crummy.com/software/BeautifulSoup/bs4/download/",
long_description="""Beautiful Soup sits atop an HTML or XML
parser, providing Pythonic idioms for iterating, searching, and
modifying the parse tree.""",
+ use_2to3 = True,
license="MIT",
packages=['bs4', 'bs4.builder', 'bs4.tests'],
cmdclass = {'build_py':build_py},
Would you try this (running the beautifulsoup4 setup.py manually) and
see if that works?
skippyV
\beautifulsoup4-4.0.0b10>python setup.py install
distribution option: 'use_2to3'
warnings.warn(msg)
Leonard Richardson
using distutils or setuptools, and the failure to recognize use_2to3
says you're not using Distribute. Without access to a machine that
exhibits the problem, I have no way of knowing what's going on.
I will change setup.py to work with Distribute, but I don't understand
your problem and I'm at the limit of what I can do to fix it. Sorry.
Leonard
Bruce Eckel
-- Bruce Eckel
www.Reinventing-Business.com
www.MindviewInc.com
Skippy VonDrake
> your problem and I'm at the limit of what I can do to fix it. Sorry.
As long as I use "easy_install" then it's installed correctly.
And I'm a Python newbie so... I'll forget about it too. :)
Thanks for trying. I did learn some things so its not a total loss for me.
Skippy
skippyV
> be helpful here? That way you'd have access to a system with the difficult
> configuration rather than trying to recreate it yourself.
"difficult configuration".
Installing the python libs is simple. But if you don't have access to
a Windows box (or VM), then yes. It is difficult.
Leonard Richardson
steps you reported:
1. I installed Python 3.2.2 from http://python.org/
2. I downloaded Distribute 0.6.24 from
http://pypi.python.org/pypi/distribute and installed it with
'python.exe setup.py install'
3. Download Beautiful Soup 4 beta 10 from
http://pypi.python.org/pypi/beautifulsoup4 and built it with
'c:\Python32\python.exe setup.py build'. I did *not* install the built
code, because I just wanted to see if the code was converted to Python
3.
I did see the 2to3 conversion running, and when I checked the code
located in build/, I saw that it had been converted to Python 3. From
the build/lib directory, I was able to start a Python shell, run "from
bs4 import BeautifulSoup", and parse a simple document.
4. I then removed Distribute (by deleting it from
c:\Python32\Lib\site-packages).
5. I then destroyed the Beautiful Soup build/ directory and rebuilt
it, having nothing installed but Python's built-in distutils package
("c:\Python32\python.exe setup.py build"). Again, the 2to3 conversion
ran and the code was properly converted to Python 3.
6. At this point I was pretty close to having a pristine Python
installation, since I removed Distribute and I never installed
Beautiful Soup. But just to be sure, I completely removed the Python
3.2.2 installation and destroyed the c:\Python32 directory.
------------------
Then I checked whether easy_install (as provided by Distribute)
installed the package properly.
7. I reinstalled Python 3.2.2
8. Then I installed Distribute again (with "python.exe setup.py install")
9. Then I ran "c:\Python32\Scripts\easy_install.exe beautifulsoup4"
10. Beautiful Soup was installed into c:\Python32\Lib\site-packages. I
spot-checked builder/_htmlparser.py to make sure the code was
converted to Python 3. I was able to "from bs4 import BeautifulSoup"
and parse a test file.
------------------
So, I think the "use_2to3" thing is a red herring, a feature that
Distribute supports *in addition to* the distutils-style import. I
never got into a situation where I couldn't "from distutils import
build_py_2to3", and I haven't been able to duplicate the problem on a
system that has only ever had Python 3 installed on it.
At the same time, this experience made me think it's very difficult to
forget what version of Python you're using on Windows--anything you
run has to include the directory name, which includes the version
number.
Leonard
skippyV
build and build/lib directories.
I wasn't sure where you meant because during 'easy_install' I'd
noticed that just a temp directory was made for the building.
Which is so often the case in Windows-land.
And I'd assumed (wrongly) that setup would make a tmp folder somewhere
and do the same.
But your post made me dig around and I soon realized it was in the
beautifulsoup dir from the tarball - facepalm!
So I deleted that directory and unzipped the tarball again.
After that - I can't duplicate the error.
So somehow - like you said in the original post - I must have gotten
the build dir populated with bad Python2.7 compilations.
And I didn't know it.
So every time I tried to do a "python setup.py install" - no
timestamps were changed and the same errors were thrown out.
But when I did "easy_install beautifulsoup4" - a tmp directory was
made and the code compiled with Python3 - with no errors.
Which made me think it was the install process at fault.
But all that time all I had to do was unzip the tarball (or run
"clean") and the problem would have been solved.
egads
Sorry about that.
But I'm glad you stuck it out through this "adventure"!
Skippy
Thomas Kluyver
> So, I think the "use_2to3" thing is a red herring, a feature that
> Distribute supports *in addition to* the distutils-style import. I
> never got into a situation where I couldn't "from distutils import
> build_py_2to3", and I haven't been able to duplicate the problem on a
> system that has only ever had Python 3 installed on it.
works. distribute/setuptools are extensions to distutils, not
replacements.
distribute's use_2to3 option, besides being easier to remember than
the build_py_2to3 thing, also tries to update doctests. But you do
need to have distribute installed, so I prefer the distutils way where
possible.
Best wishes,
Thomas
David
work, but it still is not working and I have 2 questions.
1. When I download the Distribute 0.6.25 as a tarball, where should I
unzip the folder to? the Python32\ directory or does it matter where
it is run? I ran the distribute_setup.py program from online that I
downloaded (instead of the tarball) and it produced some lovely errors
in the Python Shell.
2. Easy_Install doesn't seem to work. I also tried to download the
BeautifulSoup version 4 into the \site-packages directly. Am I
supposed to copy the entire zip file in there, or just the bs4
directory. Even when I copy just the bs4 directory - or all of the
zip file into \site-packages - it still gives errors when I open
setup.py in the IDLE program and run the Python Shell gives errors.
(see immediately below)
Traceback (most recent call last):
File "C:\Python32\Lib\site-packages\setup.py", line 27, in <module>
"Topic :: Software Development :: Libraries :: Python Modules",
File "C:\Python32\lib\distutils\core.py", line 136, in setup
raise SystemExit(gen_usage(dist.script_name) + "\nerror: %s" %
msg)
SystemExit: usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2
[cmd2_opts] ...]
or: setup.py --help [cmd1 cmd2 ...]
or: setup.py --help-commands
or: setup.py cmd --help
error: no commands supplied
I'm new to python but not to programming. I have a SAS macro that
pulls html tables from websites, but it isn't always stable so I
thought I would try out Python. It appears that BeautifulSoup will be
able to accomplish this task, but I can't get Python to install the
right modules. It is probably a very simple fix, but frustrating when
nothing I've read online says were to put the unzipped files so they
can be called properly...
On Mar 7, 3:11 pm, Leonard Richardson <leona...@segfault.org> wrote:
> I took my old Windows computer out of the closet and went through the
> steps you reported:
>
>
> 2. I downloaded Distribute 0.6.24 fromhttp://pypi.python.org/pypi/distributeand installed it with
> 'python.exe setup.py install'
>
> 3. Download Beautiful Soup 4 beta 10 fromhttp://pypi.python.org/pypi/beautifulsoup4and built it with
Bruce Eckel
David
first (according to the doc).
I've run the distribute_setup.py file from different places (the
download folder, python32\, and python32\Tools\Scripts\) directory and
when I open the .py file and run it, the shell gives errors of the
following (varies based on location run):
Traceback (most recent call last):
<module>
main(sys.argv[1:])
File "C:\Python32\Tools\Scripts\distribute_setup.py", line 489, in
main
tarball = download_setuptools()
File "C:\Python32\Tools\Scripts\distribute_setup.py", line 192, in
download_setuptools
log.warn("Downloading %s", url)
File "C:\Python32\lib\distutils\log.py", line 47, in warn
self._log(WARN, msg, args)
File "C:\Python32\lib\distutils\log.py", line 30, in _log
if stream.errors == 'strict':
AttributeError: errors
I would use anything that gets the BeautifulSoup module to be
recognized by Python...
It doesn't seem like this should be that hard, is it just the XP
version of Windows I'm still running!?
On Mar 13, 5:22 pm, Bruce Eckel <brucetec...@gmail.com> wrote:
> Have you tried Pip for installation? That's what I've been using and it
> works flawlessly. Pip is intended to be the replacement for easy_install.
>
> -- Bruce Eckelwww.Reinventing-Business.comwww.MindviewInc.com
>