Reply to this post | Go Back

View Post [edit]

Poster:	leodwight	Date:	Jan 4, 2016 7:56pm
Forum:	web	Subject:	My Website Is Not Crawled Despite Removing Restrictions From Robots.txt

Hi. I modified my robots.txt file by removing crawling restrictions for all bots and even added:

User-agent: ia_archiver
Disallow:

Basically, I have just this on my robots.txt file:

User-agent: *
Disallow: /wp-admin

User-agent: ia_archiver
Disallow:

I can't understand why the old site pages can't still be crawled by Internet Archive, receiving this notice: "Page cannot be crawled or displayed due to robots.txt."

Why?