newbie question: beautiful soup + appengine

[sofia]

Hi,

I have the following code. The idea is fetch the webpage using gae's
urlfetch and feed its contents to beautiful soup. But when I try to
get the info I want i get an index out of range error even though i
know he element is there. If I put the part of the html of that page
in a string it then works. Does anyone know what's going on?

Thanks :)

class ProcessGovBase(webapp.RequestHandler):
def get(self):

url = "http://www.base.gov.pt/_layouts/ccp/AjusteDirecto/
Detail.aspx?idAjusteDirecto=325"
result = urlfetch.fetch(url)
if result.status_code != 200:
print 'error fetching url'
else:
page = str(result.content)
soup = BeautifulSoup(page)
data_publicacao = soup('span',
{'id':'ctl00_PlaceHolderMain_lblDataRegisto'})[0].contents[0]

But this works:
#html from the webpage
page = """<div class="paddingBottom5px">
<div class="floatLeft bold paddingRight5px
width150">Data de publicação: </div>
<div class="floatLeft"><span
id="ctl00_PlaceHolderMain_lblDataRegisto" title="Data de
registo">04-09-2008</span></div>
</div>"""
soup = BeautifulSoup(page)
data_publicacao = soup('span',
{'id':'ctl00_PlaceHolderMain_lblDataRegisto'})[0].contents[0]

[sofia cardita]

Hi,

If I do print page indeed the html from the page is printed and the span I'm looking for is there as well. Any ideas?

On Tue, Dec 1, 2009 at 5:28 PM, Tal Einat <tale...@gmail.com> wrote:

Never having used GAE I can't give a knowledgable answer, but my guess is that in the first example your page variable doesn't hold precisely the text you expect it to. Try printing it or comparing it to what you expect to get.

--

You received this message because you are subscribed to the Google Groups "beautifulsoup" group.
To post to this group, send email to beauti...@googlegroups.com.
To unsubscribe from this group, send email to beautifulsou...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/beautifulsoup?hl=en.

[Aaron DeVore]

What happens when you download the file, parse it to a Beautiful Soup
tree, and run this?:

span = soup.find('span', id='ctl00_PlaceHolderMain_lblDataRegisto')
span.string

-Aaron DeVore

[Aaron DeVore]

A little change there, for more useful output.

span = soup.find('span', id='ctl00_PlaceHolderMain_lblDataRegisto')

print span.string

It *might* be useful to know the contents of Tag.string, hence the
inclusion of that line and the 'print' statement.

-Aaron

[sofia]

Hi Aaron,

(I already replied but since it's not showing up i'll repeat.)

print span.string basically outputs AttributeError: 'NoneType' object
has no attribute 'string'.

Anyway afaik string is basically a convenience for contents[0]. See
the docs "For your convenience, if a tag has only one child node, and
that child node is a string, the child node is made available as
tag.string, as well as tag.contents[0].".

The response object might be truncated by GAE since when i do print
soup.prettify() the element i want is not there but printing the page
variable itself it's there. The strange thing is that checking the
attribute by the code below

if result.content_was_truncated:
print "TRUNCATED"
else:
print "NOT TRUNCATED"

I get NOT TRUNCATED so i'm at a bit of a loss. See
http://code.google.com/appengine/docs/python/urlfetch/responseobjects.html.
Any ideas?

On Dec 1, 7:44 pm, Aaron DeVore <aaron.dev...@gmail.com> wrote:
> A little change there, for more useful output.
>
> span = soup.find('span', id='ctl00_PlaceHolderMain_lblDataRegisto')
> print span.string
>
> It *might* be useful to know the contents of Tag.string, hence the
> inclusion of that line and the 'print' statement.
>
> -Aaron
>
>
>
> On Tue, Dec 1, 2009 at 11:21 AM, Aaron DeVore <aaron.dev...@gmail.com> wrote:
> > What happens when you download the file, parse it to a Beautiful Soup
> > tree, and run this?:
>
> > span = soup.find('span', id='ctl00_PlaceHolderMain_lblDataRegisto')
> > span.string
>
> > -Aaron DeVore
>

> > On Tue, Dec 1, 2009 at 9:40 AM, sofia cardita <sofiacard...@gmail.com> wrote:
> >> Hi,
> >> If I do print page indeed the html from the page is printed and the span I'm
> >> looking for is there as well. Any ideas?
>

> >> On Tue, Dec 1, 2009 at 5:28 PM, Tal Einat <talei...@gmail.com> wrote:
>
> >>> Never having used GAE I can't give a knowledgable answer, but my guess is
> >>> that in the first example your page variable doesn't hold precisely the text
> >>> you expect it to. Try printing it or comparing it to what you expect to get.
>

[sofia]

Hi Aaron,

I get AttributeError: 'NoneType' object has no attribute 'string'. If
I'm not mistaken the string method is basically equivalent to contents
[0]

See http://www.crummy.com/software/BeautifulSoup/documentation.html#Quick%20Start

"For your convenience, if a tag has only one child node, and that
child node is a string, the child node is made available as
tag.string, as well as tag.contents[0]"

so we're getting the same. When I do print soup.prettify() i don't get
the element i'm looking for, so maybe the html was being truncated but
running
result = urlfetch.fetch(url)

if result.content_was_truncated:
print "TRUNCATED"
else:
print "NOT TRUNCATED"

tells me it wasn't and print page indeed prints the whole page so i'm
at a bit of a loss.. Ideas?

On Dec 1, 7:44 pm, Aaron DeVore <aaron.dev...@gmail.com> wrote:

> A little change there, for more useful output.
>
> span = soup.find('span', id='ctl00_PlaceHolderMain_lblDataRegisto')
> print span.string
>
> It *might* be useful to know the contents of Tag.string, hence the
> inclusion of that line and the 'print' statement.
>
> -Aaron
>
>
>
> On Tue, Dec 1, 2009 at 11:21 AM, Aaron DeVore <aaron.dev...@gmail.com> wrote:
> > What happens when you download the file, parse it to a Beautiful Soup
> > tree, and run this?:
>
> > span = soup.find('span', id='ctl00_PlaceHolderMain_lblDataRegisto')
> > span.string
>
> > -Aaron DeVore
>

> > On Tue, Dec 1, 2009 at 9:40 AM, sofia cardita <sofiacard...@gmail.com> wrote:
> >> Hi,
> >> If I do print page indeed the html from the page is printed and the span I'm
> >> looking for is there as well. Any ideas?
>

> >> On Tue, Dec 1, 2009 at 5:28 PM, Tal Einat <talei...@gmail.com> wrote:
>
> >>> Never having used GAE I can't give a knowledgable answer, but my guess is
> >>> that in the first example your page variable doesn't hold precisely the text
> >>> you expect it to. Try printing it or comparing it to what you expect to get.
>

[sofia cardita]

Hi Tal,

Well, we're getting there.. when i print the tree doing print soup.prettify() it doesn't have the element I'm looking for but when i do print page i get it.. but the strange thing is that checking the content_was_truncated attribute it's set to false. Any ideas on how to go about this?

        url = "http://www.base.gov.pt/_layouts/ccp/AjusteDirecto/Detail.aspx?idAjusteDirecto=325

        result = urlfetch.fetch(url, None, 'GET', {}, True)

        if result.status_code != 200:

          print 'error fetching url'

        else:

          page = str(result.content)

          

         if result.content_was_truncated:

            print "TRUNCATED"

         else:

            #this gets printed

            print "NOT TRUNCATED"

          

         soup = BeautifulSoup(page)

         print page

         print soup.prettify()

On Tue, Dec 1, 2009 at 6:00 PM, Tal Einat <tale...@gmail.com> wrote:

I'd look to see what the tree created by BeaitfulSoup looks like using "print soup.prettify()".

After a very brief look at the docs for urlfetch, check the content_was_truncated attribute of the Response object to make sure you have the complete response. If it was truncated this could have caused problems parsing the HTML.

If that doesn't help, check if the index error is caused by BS not finding the span at all (i.e. the first [0]) or the span not having any contents. That should give you a better idea of where the problem lies.

Hope I'm helping...
Tal

[sofia]

Finally got it.. using version 3.0.7 it's working.. I was using the
latest version.

Thanks for your help anyway :)

On Dec 1, 7:50 pm, sofia cardita <sofiacard...@gmail.com> wrote:
> Hi Tal,
>
> Well, we're getting there.. when i print the tree doing print
> soup.prettify() it doesn't have the element I'm looking for but when i do
> print page i get it.. but the strange thing is that checking the
> content_was_truncated attribute it's set to false. Any ideas on how to go
> about this?
>

>         url = "http://www.base.gov.pt/_layouts/ccp/AjusteDirecto/Detail.aspx?idAjust...

> > On Tue, Dec 1, 2009 at 7:40 PM, sofia cardita <sofiacard...@gmail.com>wrote:
>
> >> Hi,
>
> >> If I do print page indeed the html from the page is printed and the span
> >> I'm looking for is there as well. Any ideas?
>

> >> On Tue, Dec 1, 2009 at 5:28 PM, Tal Einat <talei...@gmail.com> wrote:
>
> >>> Never having used GAE I can't give a knowledgable answer, but my guess is
> >>> that in the first example your page variable doesn't hold precisely the text
> >>> you expect it to. Try printing it or comparing it to what you expect to get.
>

> >>> On Tue, Dec 1, 2009 at 7:19 PM, sofia <sofiacard...@gmail.com> wrote:
>
> >>>> Hi,
>
> >>>> I have the following code. The idea is fetch the webpage using gae's
> >>>> urlfetch  and feed its contents to beautiful soup. But when I try to
> >>>> get the info I want i get an index out of range error even though i
> >>>> know he element is there. If I put the part of the html of that page
> >>>> in a string it then works. Does anyone know what's going on?
>
> >>>> Thanks :)
>
> >>>> class ProcessGovBase(webapp.RequestHandler):
> >>>>    def get(self):
>
> >>>>        url = "http://www.base.gov.pt/_layouts/ccp/AjusteDirecto/

> >>>> Detail.aspx?idAjusteDirecto=325<http://www.base.gov.pt/_layouts/ccp/AjusteDirecto/%0ADetail.aspx?idAj...>

> >>>> beautifulsou...@googlegroups.com<beautifulsoup%2Bunsubscribe@goog legroups.com>
> >>>> .