Robots.txt

Robots.txt is a text (not html) file which we put on our site to tell search engines robots which pages you would like them to visit or not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. Robots.txt is not a way from preventing search engines from crawling our site and the fact that we put a robots.txt file is something like putting a note.
The location of robots.txt is very important. It must be kept in the main directory because if it is not kept then search engines will not be able to find it – they do not search the whole site for a file named robots.txt. In fact, they look first in the main directory and if they don't find it there, they simply assume that site does not have a robots.txt file and therefore they index everything they find along the way. So, if our website doesn't have a robots.txt file then search engines will automatically index everything they can find on your site.

Creating the robots.txt file

To all robots out
User-agent: *
Disallow: /

To prevent pages from all crawlers
User-agent: *
Disallow: /page name/

To prevent pages from specific crawler
User-agent: GoogleBot
Disallow: /page name/

To prevent images from specific crawler
User-agent: Googlebot-Image
Disallow: /

To allows all robots
User-agent: *
Disallow:

To disallow all crawlers from your site EXCEPT Google
User-agent: *
Disallow: /
User-agent: Googlebot
Allow: /