Cleaning up BeautifulSoup objects

[Peter Saffrey]

I have a screen-scrapping application using BeautifulSoup. I have an object with two methods, each of which declares a BeautifulSoup object to do some parsing like this:

soup = BeautifulSoup(file(summarypath))

I presumed that after the method returns it the soup objects would be freed, but it seems that they are not. I've been using guppy/heapy:

http://guppy-pe.sourceforge.net/#Heapy

to inspect the heap, and BeautifulSoup data structures are still consuming memory after those methods have returned. Is there something else I need to do to get this memory released?

There is a more detailed post about this problem here:

https://groups.google.com/forum/?fromgroups#!topic/comp.lang.python/2QAuVdZYIXA

but I thought this might be soup specific and worth asking here.

Peter

[Leonard Richardson]

Peter,

Every object in a Beautiful Soup parse tree is connected to other
objects through navigational members like .parent and .next_element.
I'm guessing your methods are returning NavigableString objects from
the parse tree. Those NavigableStrings have references (direct and
indirect) to every other part of the tree, so nothing gets freed.

To make sure the parse tree is freed, you'll need to remove the
NavigableString objects from the parse tree with .extract(), or
convert them into normal Python strings with unicode().

http://www.crummy.com/software/BeautifulSoup/bs4/doc/#extract

Leonard

> --
> You received this message because you are subscribed to the Google Groups
> "beautifulsoup" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/beautifulsoup/-/PO6YRUTFTvMJ.
> To post to this group, send email to beauti...@googlegroups.com.
> To unsubscribe from this group, send email to
> beautifulsou...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/beautifulsoup?hl=en.