> Okay, well they still have to process the robots.txt file, but its not
> retroactive they don't go through their index and remove whats now
> blocked. What it will do is stop them from visiting the URL in the
> future but it does not stop them from keeping it in the index.
> They'll actually visit it several times before dropping it if there
> are no other links to it, if theres a link to it, they may well keep
> it in the index even without visiting it.
> Now that's blocked the URL removal request should work, but that's a
> very buggy system from what I read here every day. I'd keep
> requesting removal of it, maybe it will catch.
> On Feb 8, 11:57 am, Clint Chapman wrote:
> > I did add it since Jan27th but according to google, the robots.txt
> > file was downloaded 7 minutes ago.
> > On Feb 8, 12:38 pm, JLH wrote:
> > > When did you add the robots.txt statement? Was it since Jan 27th?
> > > On Feb 8, 11:22 am, Clint Chapman wrote:
> > > > That's because the file exists and it generates an error. Shouldn't
> > > > google respect the robots.txt file and not even fetch it?
> > > > On Feb 8, 12:15 pm, JLH wrote:
> > > > > It returns a 500 error, not a 404.
> > > > > On Feb 8, 11:03 am, Clint Chapman wrote:
> > > > > > I have the following link in google's list of indexed pages:
> > > > > >http://www.placelifthomestaging.com/WebResource.axd?d=Tr10ImzmD06rMZ6...
> > > > > > I also have the following in my robots.txt file:
> > > > > > User-agent: *
> > > > > > Disallow: *.axd
> > > > > > And according to the tool, that url should be blocked.
> > > > > > Why then am I still seeing googlebot hitting that page? And why is my
> > > > > > request to have the url removed always denied?
> > > > > > Any info would be appreciated.
> > > > > > Thanks,
> > > > > > Clint- Hide quoted text -
> > > > > - Show quoted text -- Hide quoted text -
> > > - Show quoted text -- Hide quoted text -
> - Show quoted text -
