Automating search for words in a website using WSH
Hii Sing Chung
words are examples):
1. Got to http://www.phishing.org, check for the presence of word1, word2,
word3, word4, unicodeword5 for today's posting
2. Open up Outlook, send an e-mail using a predefined template, the subject
of the mail: Phishing check result [today's date]. The body text: There is
[0] increase in phishing site.
3. The team member has to enter today's date in the subject and if there is
any increase to the number of match found, change the 0 value.
4. Send the mail.
I am able to automate the part of the composing of e-mail using wsh with the
date variable in the subject. However, I am not able to automate the process
of checking at the particular website, looking for matching words, and
return the number of match found to the wsh to change the 0 value
accordingly. I found that I can do a Google search with the match word
parameter and the date parameter, like this:
http://www.google.com/search?hl=en&rlz=1T4SKPB_enSG318SG319&as_q=&as_epq=&as_oq=word1+word2+word3+%E5%AD%97&as_eq=&num=10&lr=&as_filetype=&ft=i&as_sitesearch=&as_qdr=d&as_rights=&as_occt=any&cr=&as_nlo=&as_nhi=&safe=images.
When putting this string into the url, and Enter, Google will return the
search result. I would like to be able to return this result back to wsh, as
0, 1, 2, 3, depending on the number of matches. Is there a way to accomplish
this?
Or, is there a better way?
Thanks in advance,
Sing Chung
![Pegasus [MVP]'s profile photo](https://lh3.googleusercontent.com/a/default-user=s40-c)
Pegasus [MVP]
"Hii Sing Chung" <sing...@hotmail.com> wrote in message
news:CFDF459F-A608-4A7E...@microsoft.com...
You could use the batch file below to search for your words It's not script
but it's simple and it works. Note that the text on http://www.phishing.org
changes continuously. The results of the batch file will therefore differ
from one moment to the next.
You can download wget.exe from here:
http://www.interlog.com/~tcharron/wgetwin.html.
[01] @echo off
[02] set words=mobile spyware nader
[03] SetLocal EnableDelayedExpansion
[04] set Result=
[05] set Filename=Phishing.org
[06] if exist "%FileName%" del "%FileName%"
[07] wget http://www.phishing.org/%FileName% -onul
[08] for %%a in (%words%) do (
[09] find /i "%%a" "%FileName%" > nul && set Result=!Result! %%a
[10] )
[11] echo Result=%Result%
[12] pause
Tim Harig
> These are the processes my team are doing everyday at 6am (the urls and the
> words are examples):
I think we may need a little more information. What you are asking is
still a little fuzzy.
> 1. Got to http://www.phishing.org, check for the presence of word1, word2,
> word3, word4, unicodeword5 for today's posting
If I understand you properly, this is the part where you are having your
difficulty...Right? How are is your script going to the URL -- or are you
not able to get to the URL at all right now?
Loading a URL from WSH is easy using Internet Explorer as a COM server.
You can load an Internet Explorer object as:
Wscript.CreateObject("InternetExplorer.Application")
Once you have created the IE object, you can use the "Navigate()" method to
load a website. Once your page is loaded (you will need to poll the "Busy"
property or use a reasonable delay), you can access the page using the
"Document" sub-object using all of the normal DOM methods, properties, and
sub-objects. Alternatively, if you have your own parser that you want to
use, you can grab the raw HTML from the "body.innerText" property which I
find useful for scraping using regexes or alternate parsers.
You can find all the information that you need about the IE object at:
http://msdn.microsoft.com/en-us/library/aa752043%28VS.85%29.aspx
where of the objects sub-objects properties, methods, events, etc are
listed and explained. Information on the Document (DOM) sub-object can be
found at:
http://msdn.microsoft.com/en-us/library/ms535205(VS.85).aspx
> 2. Open up Outlook, send an e-mail using a predefined template, the subject
> of the mail: Phishing check result [today's date]. The body text: There is
> [0] increase in phishing site.
> 3. The team member has to enter today's date in the subject and if there is
> any increase to the number of match found, change the 0 value.
There should be no reason for anybody having to enter the date; and, if
you can find the needed information from IE's DOM or another parser using
innerText, then you can set the count value as well.
> date variable in the subject. However, I am not able to automate the process
> of checking at the particular website, looking for matching words, and
> return the number of match found to the wsh to change the 0 value
> accordingly. I found that I can do a Google search with the match word
> parameter and the date parameter, like this:
> http://www.google.com/search?hl=en&rlz=1T4SKPB_enSG318SG319&as_q=&as_epq=&as_oq=word1+word2+word3+%E5%AD%97&as_eq=&num=10&lr=&as_filetype=&ft=i&as_sitesearch=&as_qdr=d&as_rights=&as_occt=any&cr=&as_nlo=&as_nhi=&safe=images.
> When putting this string into the url, and Enter, Google will return the
> search result. I would like to be able to return this result back to wsh, as
> 0, 1, 2, 3, depending on the number of matches. Is there a way to accomplish
> this?
You can pull the information as I have shown above. The only thing left is
to scrape the information you want from the page. You have been too vague
for me to help you in parsing out your informaton so you will have to
figure out how to do that yourself.
I have written many screen scraping scripts, including those which
automated reporting from an extremely complex and convoluted AJAX web
application. Where there is a will, there is a way. It can be done, you
just need to find an address or a pattern that you can use to find the
information that you are looking for in the page.
Tim Harig
can be an invaluable resource in helping to narrow down the path of the
information that you are looking for.
Hii Sing Chung
"Tim Harig" <use...@ilthio.net> wrote in message
news:GZrWl.20022$hc1....@flpi150.ffdc.sbc.com...
> On 2009-06-06, Hii Sing Chung <sing...@hotmail.com> wrote:
>> These are the processes my team are doing everyday at 6am (the urls and
>> the
>> words are examples):
>
> I think we may need a little more information. What you are asking is
> still a little fuzzy.
>
>> 1. Go to http://www.phishing.org (not the actual website, give an example
>> here due to confidentiality), check for the presence of word1, word2,
>> word3, word4, unicodeword5 for today's posting
>
> If I understand you properly, this is the part where you are having your
> difficulty...Right? How are is your script going to the URL -- or are you
> not able to get to the URL at all right now?
>
The URL is fixed (the one I give here is not the actual website, as I can't
give actual address due to confidentiality), the team members have to check
the website everyday. The website has a posting everyday about newly
discovered phishing using organization names. So, the members must check
manually (by human eyes) for the today's posting at this website for the
presence of predefined words of interests (word1, word2, word3, word4,
unicodeword5, are just examples I give because I cannot give the actual
words due to confidentiality). I also have to check matching for Chinese and
Japanese characters (that is why the 'unicodeword5' example is there).
For loading of Internet Explorer, I can use WSH, so that part is not the
problem.
Can I then pass this count value back to WSH?
>> I am able to automate the part of the composing of e-mail using wsh with
>> the
>> date variable in the subject. However, I am not able to automate the
>> process
>> of checking at the particular website, looking for matching words, and
>> return the number of match found to the wsh to change the 0 value
>> accordingly. I found that I can do a Google search with the match word
>> parameter and the date parameter, like this:
>> http://www.google.com/search?hl=en&rlz=1T4SKPB_enSG318SG319&as_q=&as_epq=&as_oq=word1+word2+word3+%E5%AD%97&as_eq=&num=10&lr=&as_filetype=&ft=i&as_sitesearch=&as_qdr=d&as_rights=&as_occt=any&cr=&as_nlo=&as_nhi=&safe=images.
>> When putting this string into the url, and Enter, Google will return the
>> search result. I would like to be able to return this result back to wsh,
>> as
>> 0, 1, 2, 3, depending on the number of matches. Is there a way to
>> accomplish
>> this?
>
> You can pull the information as I have shown above. The only thing left
> is
> to scrape the information you want from the page. You have been too vague
> for me to help you in parsing out your informaton so you will have to
> figure out how to do that yourself.
Vague?? Which is the part not clear? The e-mail is to be sent to Emergency
Center to notify if there is any increase in phishing messages detected that
use the names of this organization. I strongly feel that it is not wise to
manually check against website everyday using human eyes.
Tim Harig
> "Tim Harig" <use...@ilthio.net> wrote in message
> news:GZrWl.20022$hc1....@flpi150.ffdc.sbc.com...
>> On 2009-06-06, Hii Sing Chung <sing...@hotmail.com> wrote:
>> If I understand you properly, this is the part where you are having your
>> difficulty...Right? How are is your script going to the URL -- or are you
>> not able to get to the URL at all right now?
> The URL is fixed (the one I give here is not the actual website, as I can't
> give actual address due to confidentiality), the team members have to check
> unicodeword5, are just examples I give because I cannot give the actual
> words due to confidentiality). I also have to check matching for Chinese and
Thats fine; but, without specifics, I can only give you you general
answers.
>> Loading a URL from WSH is easy using Internet Explorer as a COM server.
>> You can load an Internet Explorer object as:
>> Wscript.CreateObject("InternetExplorer.Application")
> Can I then pass this count value back to WSH?
Everything is already being done inside of WSH. You are creating the COM
object from WScript (correct spelling, the 'S' should be capitalized
above). Once the COM object is created it acts just like a native object
for whatever language you are using (Javascript, VBscript, Python, etc.)
>> You can pull the information as I have shown above. The only thing left
>> is
>> to scrape the information you want from the page. You have been too vague
>> for me to help you in parsing out your informaton so you will have to
>> figure out how to do that yourself.
> Vague?? Which is the part not clear? The e-mail is to be sent to Emergency
> Center to notify if there is any increase in phishing messages detected that
> use the names of this organization. I strongly feel that it is not wise to
> manually check against website everyday using human eyes.
It is not that it isn't clear, its that I don't know what the data that you
are trying to pull looks like. I have given you enough information to
download the web page; but, without being able to see a markup example and
that data that you are trying to extract from it, I cannot give you any
ideas as to extract that data. Unless the data is preformated into some
kind of delineated data, XML, etc with stateful metadata; you are probably
going to have to parse the HTML. HTML scraping is very specific to the
page that you are trying to scrape. Without seeing a representative
page that you are trying to extract the data from, I cannot tell you
how to extract it from that page.
I gave you information on how to download that page. Without being able to
see the data, it will then be your responsibility to figure out how to
parse it.
Tim Harig
>> I am able to automate the part of the composing of e-mail using wsh with
>> the date variable in the subject. However, I am not able to automate the
>> process of checking at the particular website, looking for matching words,
>> and return the number of match found to the wsh to change the 0 value
>> accordingly. I found that I can do a Google search with the match word
> but it's simple and it works. Note that the text on http://www.phishing.org
> changes continuously. The results of the batch file will therefore differ
> from one moment to the next.
Being primarily a Unix person I often use similar similar methods with a
shell script using wget or curl when the script is simple enough. However,
since he is already doing everything else inside of WSH it would an extra
set of technologies that he would have to work with. That would complicate
things and he would still have to interface the script back to WSH.
Most scripting langauges have the ability to download files via HTTP within
their core libraries. The default WSH languages (VB Script and Javascript)
do not; but, they have access to all of the Microsoft Windows and Office
componets which is just as good. He can download the page from inside of
WSH using Internet Explorer and have all of the data in a native object
form (DOM) or just the actual markup if he wants to use a different parser.
Using Internet Explorer has another advantage over just grabbing the markup
via HTTP. When you use Internet Explorer to download a web page you also
run the Javascript on that page which may reveal content which is not
available from looking at the markup alone. This is very important for
automating AJAX style web applications. The alternative is trying to dig
through and reverse engineer all of the Javascript to try and find all of
the data sources that the Javascript draws from and trying to figure out
the protocol between the Javascript and the server side scripts which you
cannot see. In the end, it is just easier to let the browser handle the
javascript and pull the data out of the DOM after the page's javascript has
done its job.
Hii Sing Chung
I want to give you the actual URLs, but now I also can't recall the
addresses. I will be able to refer to them at the office. Now at home I have
no access to office information (also another security implementation). Let
me check if other members can recall the address now.
"Tim Harig" <use...@ilthio.net> wrote in message
news:xxvWl.17770$pr6...@flpi149.ffdc.sbc.com...
Pegasus [MVP]
I tried your scripting suggestion and it works very nicely. Its advantage
over wget.exe is, as you say, that the the web page data is nicely
formatted. The drawback is that it takes a little more time to run because
the script needs to launch IE.
Hii Sing Chung
1. http://www.rbl.jp/phishing/ At this website, we will look under today's
posting at the top portion (June 6, 2009) for the presence of the
organization's names in any forms. If we see any match, we will count the
number of occurrences.
2. http://www.fraudwatchinternational.com/phishing/search.php At this page
we will select today's date, then select the organization name from the drop
down 'target company', then click search. The result will be displayed as
"Result Found: [number of occurrence]".
The number of occurrence we will added up from these 2 sites and key in the
e-mail as number of increase in phishing detected.
Let me try to get my team member to furnish an example of the e-mail.
Thanks
Sing Chung
"Tim Harig" <use...@ilthio.net> wrote in message
news:xxvWl.17770$pr6...@flpi149.ffdc.sbc.com...
Tim Harig
> Tim,
>
> 1. http://www.rbl.jp/phishing/ At this website, we will look under today's
> posting at the top portion (June 6, 2009) for the presence of the
> organization's names in any forms. If we see any match, we will count the
> number of occurrences.
This should be extremely easy to parse. Note that all of the date
information in contained within <div class="article"></div>. So, all you
have to do is getElementByTagName("div") and run through the results until
you get the ones which have the properties nodeName=class and
nodeValue=article.
The date is then underneath with an <h3> tag. You run through the
article divs until you find the date that you are looking for; then, the
information is all under the <div class="dan"></div>. It couldn't get much
easier. You just grab the organization names and calculate them as you
please.
> 2. http://www.fraudwatchinternational.com/phishing/search.php At this page
> we will select today's date, then select the organization name from the drop
> down 'target company', then click search. The result will be displayed as
> "Result Found: [number of occurrence]".
> The number of occurrence we will added up from these 2 sites and key in the
> e-mail as number of increase in phishing detected.
> Let me try to get my team member to furnish an example of the e-mail.
This one is just as easy. All of the information is under a form with an
id so you just have to getElementByTagName("form") and check the results
until you get the one with id="alertsearch". The underlying fields can be
identified by their name and then set by changing their value attribute.
Once the form is filled in, you just grab the button and fire its onclick
event which will submit the information to th server.
All and all these are very simple pages to use because of all the context
data stored within the class attributes. I have worked with *MUCH* worse.
It shouldn't be much trouble to automate.
Hii Sing Chung
Thanks. I need to read up more to understand your explanations. Here is the
content of my WSH, I haven't put in the automation of iexplorer and the
subsequent searching action. I will need to get the value of intCount:
option explicit
on error resume next
dim objEmail, objWshShell, objOutlook, objWshNetwork
dim strTo, strCc, strText, strDate, strIncrease, strMessage, intCount,
strUsername
Const conMailItem = 0
strDate = formatdatetime (date, vblongdate)
Set objOutlook = createobject("Outlook.Application")
Set objEmail = objOutlook.createitem(conMailItem)
set objWshShell = wscript.createobject ("wscript.shell")
set objWshNetwork = CreateObject("WScript.Network")
strUsername = objWshNetwork.username
strTo = "DataCenter_Supervisor; DataCenter_Singapore_Supervisor"
strCc = "Security_Operator"
IntCount = 0
If intCount = 0 Then
strIncrease = "no"
else
strIncrease = intCount
end if
objEmail.To = strTo
objEmail.Cc = strCc
objEmail.Subject = "Report mail for the phishing site check on " & strDate
strMessage = "IT Control Unit-Security, " & vbcrlf & vbcrlf &_
"The number of phishing site is as below:" & vbcrlf & vbcrlf &_
"[" &IntCount&"]" & vbcrlf & vbcrlf & "There is " & strIncrease
& " increase." &_
vbcrlf & vbcrlf & vbcrlf & vbcrlf & "The helpful sites are
below:" &vbcrlf &_
"http://www.rbl.jp/phishing/" & vbcrlf &
"http://www.fraudwatchinternational.com/phishing/search.php" & vbcrlf &_
"Thanks and Regards," & vbcrlf & strUsername & vbcrlf & "ex.
25332" & vbcrlf & "IT Security Monitoring"
objEmail.body = strMessage
objEmail.display
objEmail.send
wscript.quit
"Tim Harig" <use...@ilthio.net> wrote in message
news:IZEWl.8892$Lr6...@flpi143.ffdc.sbc.com...
Paul Randall
I've pasted in some code to automate IE to do part of what you need to get
your information. It finds five DIV elements with <DIV class=article>,
displays them and writes them to files named 'Div n.htm', where n is one to
five. This allows you to look at them easily in IE and tell whether one or
more of them contain information you require. I realize they may not
display perfectly because they don't contain header info that
"http://www.rbl.jp/phishing/" provided to ensure correct display. You might
look at them using the free version of PSPad, and use its built-in HTML Tidy
tool to make it easier to see the details of these division elements. Then
it should be easy to figure out how to extract the items you want to count.
Option Explicit
Dim oIE 'Internet Explorer Object
' Documentation at: http://msdn.microsoft.com/en-us/library/aa752084.aspx
Set oIE = WScript.CreateObject("InternetExplorer.Application")
oIE.Navigate "http://www.rbl.jp/phishing/"
oIE.ToolBar = 0
oIE.StatusBar = 0
oIE.Width=300
oIE.Height = 150
oIE.Left = 0
oIE.Top = 0
oIE.Visible = True
Do While oIE.Busy: WScript.Sleep 20: Loop
MsgBox "Done loading web page"
Dim oIEDoc 'IE Document Object; gives access to
' Document Object Model (DOM) of the loaded web page.
'Documentation at:
' http://msdn.microsoft.com/en-us/library/ms533050(loband).aspx
Set oIEDoc = oIE.Document
MsgBox "Total number of tags = " & oIEDoc.All.length & vbCrLf & _
"Total number of 'div' tags = " & oIEDoc.All.Tags("div").length
Dim oTag, oAttribute, sMsg, iCount
For Each oTag In oIEDoc.All.Tags("div")
' look for 'article' within the <div ... > portion of the element.
If InStr(Left(oTag.OuterHtml, InStr(oTag.OuterHtml, ">")), _
"article") Then
iCount = iCount + 1
MsgBox oTag.OuterHtml
fOverWriteAU "Div" & " " & iCount & ".htm", otag.outerhtml
End If
' End If
Next 'oTag
Sub fOverWriteAU(sFilePath, sText)
On Error Resume Next
CreateObject("Scripting.FileSystemObject")._
CreateTextFile(sFilePath, True, False)._
Write(sText)
If Err Then
CreateObject("Scripting.FileSystemObject")._
CreateTextFile(sFilePath, True, True)._
Write(sText)
End if
End Sub 'fOverWriteAU(sFilePath, sText)
-Paul Randall
"Hii Sing Chung" <sing...@hotmail.com> wrote in message
news:2A45432B-BEFA-4994...@microsoft.com...
Tim Harig
> Thanks. I need to read up more to understand your explanations. Here is the
Basically once you have grabbed IE's document object you are working
directly with the DOM just as you would be programming Javascript in the
browser.
var browserObject = WScript.createObject("InternetExplorer.Application")
browserObject.Navigate("http://www.rbl.jp/phishing/")
sleep(10)
var resultDocument = browserObject.Document
Once you are working the DOM, you just need to use the context
data given in the document from the div classes to find the information
that you are looking for.
You will notice that the information for each date is contained within a
division under the class of article:
http://ilthio.net/static/depot/m.p.s.w/articleDOM.png
The screen shot shows this in the DOM inspector. I am using Firefox with
Firebug on Linux but it should look similar to the DOM inspector on IE with
the developer tools I posted above. Notice also that date header is in
a division underneath with the class of title. This can be used to find
exactly the date that you are looking for. Once you have found the proper
article, all of the information you want is within the div classed as
"dan".
http://ilthio.net/static/depot/m.p.s.w/danDOM.png
Once you have located the dan division under the article object, you just
have to grab the innerHTML and parse out the information that you want.
I have created a quick and dirty demo page to help demonstrate how to
extract the information that you are looking for:
http://ilthio.net/static/depot/m.p.s.w/phishing.htm
The page is almost identical to the origional first page that you linked
to (without the linked images); but, I have placed a new "Get Data" link
within a new div on top. If you click the link, it will grab the IP's and
hosts from the first article entry. The relevent code to extract the
information is contained in isolateContent() here:
http://ilthio.net/static/depot/m.p.s.w/demo.js
This is rough and it doesn't do everything that you want it to do; but, it
should give you enough of an idea of how to extract the data. You can add
your own date searching code etc. Once you have the data, you can
calculate it however you wish. Once you have the data, you will use
similar methods to find the relevant objects within the second page. You
can enter information into the the forms using setAttribute() to the value
attributes of the form fields. Then you just need to call the onselect()
method of the button to submit it.
Hii Sing Chung
Is there a way to bold some texts of the e-mail body through the script? I
want to bold the "IT Control Unit - Security" at the beginning of the
message body.
Thanks.
"Paul Randall" <paul...@cableone.net> wrote in message
news:etabz26...@TK2MSFTNGP06.phx.gbl...
Paul Randall
can help you with bolding certain parts of the text.
-Paul Randall
"Hii Sing Chung" <sing...@hotmail.com> wrote in message
news:C450D6BD-82D2-4A40...@microsoft.com...
Si Ballenger
wrote:
>Most scripting langauges have the ability to download files via HTTP within
>their core libraries. The default WSH languages (VB Script and Javascript)
>do not; but, they have access to all of the Microsoft Windows and Office
>componets which is just as good. He can download the page from inside of
>WSH using Internet Explorer and have all of the data in a native object
>form (DOM) or just the actual markup if he wants to use a different parser.
The below seems to be fairly native and free of IE.
test.vbs
===============================
With CreateObject("MSXML2.XMLHTTP")
.open "GET", "http://www.speakeasy.net/speedtest/", False
.send
a = .ResponseBody
End With
With CreateObject("ADODB.Stream")
.Type = 1 'adTypeBinary
.Mode = 3 'adModeReadWrite
.Open
.Write a
.SaveToFile "pagesource.txt", 2 'adSaveCreateOverwrite
.Close
End With
asdf
(jscript).
Any offense would be greatly answered by the ff community.
It it had not been for ff tabs, we would still be searching for the soul of
a browser
in the ms purchased/licensed item from Mosaic..
"Si Ballenger" <shb*NO*SPAM*@comporium.net> wrote in message
news:4ae294f0....@news.comporium.net...
asdf
It supports content-Type, that is
delivered la (times) , chinese or latvian or any other
non us army yet ocupied planet space language encoding .
In case you thought 2030 was a year and 230 years is (was) permanent.
"Si Ballenger" <shb*NO*SPAM*@comporium.net> wrote in message
news:4ae294f0....@news.comporium.net...