Robots.txt is a text file that contains instructions for web robots, programs that operate on the internet automatically collecting information for purposes including search engine indexing, and how these robots view your site. If you have a shop with EKM you can see your robots file by adding ‘robots.txt’ to your domain name. For example, the robots file for http://www.danstestplatform.co.uk would be http://www.danstestplatform.co.uk/robots.txt .
Most eCommerce sites are going to look to be as visible as possible in search engine results so aren’t going to need to use robots.txt to restrict access to their shops. EKM do use some ‘DISALLOW’ values as standard, but these are for pages that would be customer-specific; for example, the contents of the cart page for one user is going to be different for another user, so it wouldn’t make sense for this to be indexed in a search engine.
What if Robots.txt is blocking access to my products or categories?
While Robots.txt does restrict access to dynamic areas such as the cart, checkout and search pages on your shop, it does not restrict access to your content. There is only actually one robots.txt file for each EKM server and none of the content is shop-specific.
Some tools will report that Robots.txt is preventing access to certain pages on your shop, but while this may be accurate on some platforms it’s not the case on EKM. The most common cause for this kind of error being reported is that your products are hidden or have been deleted and no longer exist. You can resolve this by adding in 301 redirects to a relevant address that still works.
What if I want to add my shop to the Robots.txt file?
As we discussed above, the robots file is the same for the entire server, so EKM Shop owners are unable to change it on an individual basis. There are other ways of avoiding your shop being crawled by robots, however. Adding content to meta tags in your design can have the same effect if your shop shouldn’t be visible to search engines or to other robots.
[more]Something related you may be interested in - 301 Redirects.[/more]
[more]Something related you may be interested in - Block search indexing with meta tags[/more]