search engines
K.S. Ethan
questions about search engines
I have a few questions about how search engines find pages. From search
engines, I found useful information about cracking of two websites. I
want to learn how the search engines learned.
..................
This concerns subdirectories.
Yahoo! search engine was asked about the following text string:
"www.paladium.net" writedocsearch
Its response included the following two paragraphs:
#
www.paladium.net/writedocSearch/script/?D=A
www.paladium.net/writedocSearch/script/?D=A
#
www.paladium.net/writedocSearch/images/?D=D
www.paladium.net/writedocSearch/images/?D=D
A cracker had somehow installed two subdirectories, script and images.
I understand this. What I don't understand is the stuff that comes
after in Yahoo!:
?D=A and ?D=D
When I clicked on the second link, I got URL
http://paladium.net/writedocSearch/images/
Notice that nothing came after the final slash.
Why did the search engine have ?D=D at the end?
What does that mean?
The web host is in Canada.
That's the first question.
By the way, I have since eliminated the subdirectories although it is
possible that a cracker later re-installed them.
.......
This concerns a cascading style sheet.
Yahoo! has a link to the cascading style sheet, which sheet ends in
.css.
How could the search engine have reached that sheet? If you go into the
subdirectory (which was inserted by a cracker, by the way), you can then
click a link to go up into the parent directory. There was a time at
which, if you did that enough, you reached the main Web directory,
including the style sheet. I guess taht's how the search engine did it.
Does that seem likely?
It is good that I leaarned from Yahoo! about the cracker-installed
subdirectories which originally had enabled people to reach the entire
ftp directory of the website (by going up into parent directories).
The way I found out about this is interesting. I went to Yahoo! and
searched for links with "www.paladium.net". I went to the last page of
the multi-page printout (there wre several pages of links), then clicked
the link for the comprehensive printout which includes similar pages
which had been omitted from the original printout). That printout was
huge (about 57 pages) and had many links to many pages whcih do not have
"www.paladium.net". I started reading the links. That's where I found
a link to the cascading style sheet and to the cracker-installed
subdirectories. Remember that the rest of the website did not then have
any links to those subdirectories, so it's hard to understand how the
search engine reached them.
.....
It may be relevant that there is much cracking of that website, often
soon after files are uploaded to the ftp directory for the site.
Webfiles are modified. Entire subdirectories are inserted into
directories (for example, the images subdirectory mentioned above).
Once, soon after a cracker-installed file was deleted, the entire
website became unreachable by Web browser and so was the webhost's
website. This lasted a few minutes, maybe several minutes. The next
day, a webbpage had been maliciously modified by a cracker.
It was through Yahoo! that I learned about the cracking; for example,
the maliciously insertd subdirectories. The subdirectories wree not
listed in google or msn search engines, for example.
...
Through Google, I learned that an entire webpage ("fake.shtml") had been
maliciously installed into a different website hosted in Germany. As
far as I know, no other webpage linked to it . How did the Google
search engine reach it to discover it? It's good that I learned from
Google. Other search engines (for example, Yahoo and MSN) did not link
to that cracker-installed page.
Even if we assume that google updates more frequently so that it found
out first (which would explain why no other seach engine knew), how did
Google find fake.shtml?
Do you know?
....
In summary, I found useful information about cracking from search
engines. I want to learn how they found out.