A Question on Robot.txt Fetch
Last Update: March 13, 2017
Sorry for not asking this as a question but I needed to include an image.,... so had to make it as a blog instead.
I had the following display on my GWT display.. saying that the spiders had problems crawling my site due to this error... Robots.txt Fetch
What can i do to sort this out?
Any ideas?
PS The Image does not seem to be displayed here either lol .. nightmare!!!
Join the Discussion
Write something…
Drawshot
Premium
Hi,
Examples of what your robot.txt file should say to get the results you want.
To allow full access
User-agent: *
Disallow:
To block all access
User-agent: *
Disallow: /
To block one folder
User-agent: *
Disallow: /folder/
Replace "folder" with the name of the folder. If have more folders to block, add another line - Disallow /folder/ beneath it.
Block one file
User-agent: *
Disallow: /file.html
Replace "file" with the name of the file. If have more files to block, add another line - Disallow /file/ beneath it.
Hope this helps.
Examples of what your robot.txt file should say to get the results you want.
To allow full access
User-agent: *
Disallow:
To block all access
User-agent: *
Disallow: /
To block one folder
User-agent: *
Disallow: /folder/
Replace "folder" with the name of the folder. If have more folders to block, add another line - Disallow /folder/ beneath it.
Block one file
User-agent: *
Disallow: /file.html
Replace "file" with the name of the file. If have more files to block, add another line - Disallow /file/ beneath it.
Hope this helps.