What is it?
robots.txt is a regular text file that sit's in the public_html folder of your website.

What does it do?
Basically it tells the bots information, in this case it tells search engines which of them are allowed to crawl your site and which are not, it also states which files to index and which to leave alone. Robots.txt does hold more information but for now i will just use the above examples.

Why would I be intrested?
Well if you're trying to climb the search engines this helps and you can also use robots.txt to save your bandwidth by stopping search engines going to pages that you consider unecessary.

Is this compulsory for SEO?
Truthfully, I don't think it is compulsory for SEO as you can gain a SE presence without the use of a robots.txt file but it does no harm having it and it helps the SE's out with their indexing etc'

Right, how do I do this?
Google the words "robots.txt" and you should find a website that creates the file for you, it takes no time at all. once you have the file upload it to your public_html folder. I got mine from www.mcanerin.com/EN/search-engine/robots-txt.asp

I don't have a website!
Build one

Ok, I read the information above but i'm stuck!
Leave a comment below or PM me and i'll help you out. I really is as easy as 1,2,3.

What does this look like on my site?
example: www.greatlumley.com/robots.txt 99% of people will never see it, as stated above it's for SE's.

Note: robots.txt will give instructions to search engines. Spambots ignore robots.txt or choose not to follow instructions so this will not stop spambots crawling your website.

Hi Billy,

Thanks for sharing that info with us, i've just made it up for my site, its really easy

http://www.bideasynow.com/robots.txt

Not sure about the permissions though at the mo its 0644 is that ok or should it be 0755 ???

Thanks

Gerry

I'm not 100% but I think it will be fine as I think 0755 is to edit the file and search engines only need read the file not edit it, I may be wrong though, someone please correct me if I am.

It is worth bearing in mind that you don't need to do anything if you want the search engine spiders to crawl your entire website - they will do this irrespective of whether there is a robots.txt file attached.

Their main value comes from preventing them from indexing and following links on pages, i.e. when you see noindex,nofollow in the page source

Anthony