robots.txt

I haven’t been indexed by Google since I upgraded to Drupal 4.5.0. Looking at the log files, there were requests for a robots.txt file that didn’t exist. Instead of simply giving a 404 error, Drupal was being clever and returning a blankish page. I think this was confusing the crawler and it was moving on without re-indexing the site. I’ve added a simple robots.txt file and I’m going to see if I get crawled.

Once the site has been crawled, the Google ads over on the right should contain adverts based on keywords on the main page.

garan

Leave a Reply