Parsing for urls

[cengineer]

I recently learnt about crawling web pages and parsing the html source
code using VB.NET. This has worked well so far with my code working
error free. I recently came across a website that contains links I
would like to capture but the links as they appear on the webpage are
not visible in the html source. I tried to extract them differently by
visiting articles and posts which worked but some of those links were
not included.

My question is whether this is unusual or is there something I need to
read up on. If you scroll over the link or click on it, it follows to
the link site appropriately but why is it not in the source html. Does
this have something to do with other scripts running on the page>

Any advice or suggestions appreciated.

[Processor Devil]

links can be created dynamically using javascript or maybe they are in some frameset or iframe... :D

2009/11/23 cengineer <ceng...@hushmail.com>

[Cerebrus]

Your hunch is spot on... the links are probably created dynamically
using client side script (read "Javascript").

[Adam Lee]

cengineer would you mind posting some code or sending me the basic
code to do web crawling?

[Processor Devil]

Here is simple function to extract links from html source code...
You need to use namespaces System.Text.RegularExpressions and System.Collections.Generic

        public static List<string> getLinks(string data)
        {
            Regex rx = new Regex("<a href=\"([^>\"]+)[^>]*\">");
            List<string> forReturn = new List<string>();
            foreach(Match m in rx.Matches(data))
                forReturn.Add(m.Groups[1].Value);
            return forReturn;
        }

2009/11/24 iwork iwork <iwork....@gmail.com>

Me too.