Escape characters on Unix?

77 views

Skip to first unread message

Sunny

unread,

May 9, 2012, 1:18:20 PM5/9/12

to beautifulsoup

Hi all,

I'm relatively new to using Beautiful Soup (bs4), and I'm encountering
a problem when I try to run code on a Unix machine. I'm using the
following line to get html from a website and returns text, which I
later parse through:

page = BeautifulSoup(get_page(doc)).get_text()
[other parsing code here to be run afterwards]

It appears that BeautifulSoup is encountering some sort of end-of-file
escape sequence that is leading to only the first few hundred
characters being read. When I run the exact same script on my Windows
machine, BeautifulSoup is able to read the entire file.

Any ideas on how I can configure the BeautifulSoup call to get the
entire document, and not get hung up on some sort of escape sequence?

Many thanks. Results below.

==============================
Source HTML
==============================
<html xmlns="http://www.w3.org/1999/xhtml"><head><title>
Sentieo
</title><meta http-equiv="content-type" content="text/html;
charset=ISO-8859-1"><META NAME="ROBOTS" CONTENT="NOINDEX,
FOLLOW"><meta name="description" content=""/><meta name="keywords"
content=""/><link href="/media/css/style.css" rel="stylesheet"
type="text/css"><link rel="stylesheet" type="text/css" media="all"
href="/media/css/highslide.css" /><script type="text/javscript" >
var console = {};
console.log = function(){};
</script><![if !IE]><script src="/media/js/jquery.min.js" ></
script><script src="/media/js/json2.js" ></script><link
rel="stylesheet" type="text/css" media="all" href="/media/css/
popup.css" /><script src="/media/js/popup.js" ></script><script src="/
media/js/jquery.nicescroll.js" ></script><script src="/media/js/
jquery.scrollTo-1.4.1-min.js" ></script><script src="/media/js/
search_doc.js" ></script><script src="/media/js/doc_reader.js" ></
script><script src="/media/js/doc_load.js" ></script><![endif]></head><body class=""><div
class="full_view"><div class="highlights" style="display: none;"></
div><div class="notes" style="display: none;"></div><ul id="myMenu"
class="contextMenu" style="display:none;"><li class="copy"><a
href="#copy">Add Highlight</a></li><li class="copy"
id='add_with_high'><a href="#copy" >Save Highlight & Add Note</a></
li></ul><ul id="rightMenu"
class="contextMenu"style="display:none;"><li class="edit"><a
href="#edit" id="add_note">Add Note</a></li><li class="edit"><a
href="#edit" id="rem_all_high">Remove All Highlights</a></li></ul><ul
id="rightMenu2" class="contextMenu"style="display:none;"><li
class="edit"><a href="#edit" id="rem_high">Remove Highlight</a></
li><li class="edit"><a href="#edit" id="add_note_high">Add Note</a></
li></ul><ul id="rightMenu3"
class="contextMenu"style="display:none;"><li class="edit"><a
href="#edit" id="remove_note">Remove Note</a></li></ul><div>
[[[truncated]]]

============================================
Result returned from BeautifulSoup(get_page(doc)) on UNIX
=============================================
Sentieo

var console = {};
console.log = function(){};
if !IEendif[if IE]><script src="/media/js/ie/jquery.min.js" ></
script><script src="/media/js/ie/json2.js" ></script><script src="/
media/js/ie/ierange-m2.js" ></script><link rel="stylesheet" type="text/
css" media="all" href="/media/css/popup.css" /><script src="/media/js/
ie/popup.js" ></script><script src="/media/js/ie/jquery.nicescroll.js"
></script><script src="/media/js/ie/jquery.scrollTo-1.4.1-min.js" ></
script><script src="/media/js/ie/search_doc.js" ></script><script
src="/media/js/ie/doc_reader.js" ></script><script src="/media/js/ie/
doc_load.js" ></script><![endif]Add HighlightSave Highlight & Add Note

==============================================
Result returned from BeautifulSoup(get_page(doc)) on Windows
==============================================

Sentieo

var console = {};
console.log = function(){};
if !IEendif[if IE]><script src="/media/js/ie/jquery.min.js" ></
script><script src="/media/js/ie/json2.js" ></script><script src="/
media/js/ie/ierange-m2.js" ></script><link rel="stylesheet" type="text/
css" media="all" href="/media/css/popup.css" /><script src="/media/js/
ie/popup.js" ></script><script src="/media/js/ie/jquery.nicescroll.js"
></script><script src="/media/js/ie/jquery.scrollTo-1.4.1-min.js" ></
script><script src="/media/js/ie/search_doc.js" ></script><script
src="/media/js/ie/doc_reader.js" ></script><script src="/media/js/ie/
doc_load.js" ></script><![endif]Add HighlightSave Highlight & Add
NoteAdd NoteRemove All HighlightsRemove HighlightAdd NoteRemove
NotexAdd NoteTickersTagsNote4fa3ad52691f3b04b4000003Press
ReleaseExhibit 99.1 FOR IMMEDIATE RELEASE: May 4, 2012NW Natural
Reports Solid First Quarter 2012 Results [[Text continues]]

leonardr

unread,

May 24, 2012, 12:03:13 PM5/24/12

to beautifulsoup

It's very difficult to understand documents that contain escape
sequences without looking at the original documents. If you send me
the actual document you are trying to parse, I'll take a look at it.

It's also possible that you're using an older version of Beautiful
Soup which does not have a workaround for this bug in lxml:

https://bugs.launchpad.net/lxml/+bug/963936

Leonard

Reply all

Reply to author

Forward

0 new messages