Is there a way of adding a wildcard after a certain DROP string?
Example all searchs begin with &page=
but from there the url is dymanic and will be listed differnt depending
on the search the customer used.
example:
&page=1&from=v&srchby=title&keyword=§ion=sv&category=All
If the Crawler stopped at the &page= then it would know when it found
duplicates and I think not list duplicates.
Any ideas?