
Short Guide To Robots.txt For SEO
What is it?
robots.txt is a regular text file that sit's in the public_html folder of your website.
What does it do?
Basically it tells the bots information, in this case it tells search engines which of them are allowed to crawl your site and which are not, it also states which files to index and which to leave alone. Robots.txt does hold more information but for now i will just use the above examples.
Why would I be intrested?
Well if you're trying to climb the search engines this helps and you can also use robots.txt to save your bandwidth by stopping search engines going to pages that you consider unecessary.
Is this compulsory for SEO?
Truthfully, I don't think it is compulsory for SEO as you can gain a SE presence without the use of a robots.txt file but it does no harm having it and it helps the SE's out with their indexing etc'
Right, how do I do this?
Google the words "robots.txt" and you should find a website that creates the file for you, it takes no time at all. once you have the file upload it to your public_html folder. I got mine from
www.mcanerin.com/EN/search-engine/robots-txt.asp
I don't have a website!
Build one
Ok, I read the information above but i'm stuck!
Leave a comment below or PM me and i'll help you out. I really is as easy as 1,2,3.
What does this look like on my site?
example:
www.greatlumley.com/robots.txt 99% of people will never see it, as stated above it's for SE's.
Note: robots.txt will give instructions to search engines. Spambots ignore robots.txt or choose not to follow instructions so this will not stop spambots crawling your website.
