Ignore pages that contain specific text

[diddy]

Hi,

Is it possible to only look at web pages where certain words/phrases
exist? E.g. if I do a web search for, say "swine flu" and create a
network from this. Is there any
way I can only harvest links from pages that contains the phrase
"swine flu" somewhere in the text of the page?


Many thanks, Dave.

[Dale Hunscher]

I've been looking into this for other reasons, but I don't have working code yet. More news when that happens...

Dale A. Hunscher, MSI
Sr. Business Systems Analyst
University of Michigan Medical School
734-678-5178

[diddy]

Cool. I think this could add a real unique edge to the code.
Basically, you only want to harvest links from pages that contain (or
don't contain) a given word/phrase(s). This would keep the network
analysis "on topic".

Dave.

> > Many thanks, Dave.- Hide quoted text -
>
> - Show quoted text -

[Dale Hunscher]

Dave,

See attached ZIP.

The example program includeexclude.py should get you started. I haven't tested this a lot, but I wanted to get it to you ASAP to get your feedback on it.

Dale A. Hunscher, MSI
Sr. Business Systems Analyst
University of Michigan Medical School
734-678-5178

[diddy]

Hi - this is great news, thanks. I ran this script (below), and my
required phrase was "social media", but it still seemed to include
refs to pages that did not contain this phrase in the page text. Am I
doing something wrong?: -

# urlforest1.py
from urlnet.urltree import UrlTree
import re # for search flags


net = UrlTree(_useHostNameForDomainName = True,_maxLevel=1)

incl_patternlist = [
# new words or phrases could be added by copying and modifying
this line.
'social media',
]

net.SetProperty('include_patternlist',incl_patternlist)

urlforest = (
'list_of_urls',)



ignorableText = \
['list_of_text_to_ignore',]


net.SetIgnorableText(ignorableText)


success = net.BuildUrlForest(Urls=urlforest)
if success:
net.WritePajekFile('urlforest', 'urlforest')
net.WritePajekNetworkFile('urltree2', 'urltree2domains', urlNet =
False)
net.WritePairNetworkFile('urltree1',
'urltree1domains',
urlNet = False, # do domains instead
uniquePairs = True,
delimiter = ' ')

>  urlnetandexamples.zip
> 299KViewDownload- Hide quoted text -

[Dale Hunscher]

Is there any chance you could send me an example URL of a page you expected to be excluded? That way I can debug. If you don't want it to go to the entire list you can email it to my GMail account directly (dalehu...@gmail.com). This code is really green, so I'm not too surprised if there is a bug.

Dale A. Hunscher, MSI
Sr. Business Systems Analyst
University of Michigan Medical School
734-678-5178

[diddy]

Hi Dale - I have emailed you the file I used.

Many thanks, Dave.

> > > - Show quoted text -- Hide quoted text -