Creating and Using a Robots.txt File

If
you have made the wise decision to have the professionals at AuroIN
work on your SEO campaign you will notice that we are very detail
oriented. An example of this is the way we deal with the robots.txt
file. Quite frankly this is not the most important feature of your
website although it does have a significant role to play.

A
robots.txt is a simple text file whose function is to tell the
various search engine spiders not to crawl or index certain sections
or pages of your site. It can be used to completely prevent your
website from being indexed, or just certain areas, or it can issue
indexing instructions to specific search engines.

Essentially
all search engines will look for a robots.txt file as soon their
spiders access your site. So, even if you don’t need to exclude the
spiders from any part of your site, having a robots.txt file is still
useful as it can act as an introduction to your website.

Reasons
for excluding spiders from all or portions of your website could be
something as simple as having an unfinished website, and not wishing
to have it appear in search results, or perhaps there are portions of
your website that are of no interest to anyone conducting a search.
If you use similar doorway pages, each optimized for a specific
search engine, you probably will not want every robot to have access
to all of them. This will avoid penalties for spamming search engines
with several similar pages. You may wish to exclude some spiders
altogether for some reason.

There
is nothing difficult about creating a basic robots.txt file. It can
be created using notepad or whatever is your favorite text editor.
Each entry has just two lines:

User-Agent:
[Spider or Bot name]
Disallow: [Directory or File Name]

For more information about New
York SEO call 888 272 8734 or use this link:
http://www.auroin.com/seo-topics/search/node/robot.txt

For pricing information, or to sign up to have AuroIN begin SEO work
on your website, click here: http://www.auroin.com/seo-pricing/