Bots and internet crawlers can be going through all the links they can find on your site at any given time. In some cases, this can be a pain for your business. Bots may click on your “add to cart” buttons. This may not seem like a big deal, but if you’re running a large-scale eCommerce store, this could be costing you quite a bit in server and CPU processing.
There are many backend WooCommerce functions and calculations that happen every time a product is added to the cart. Even bots can be seen and handled as a “guest” customer by WooCommerce.
On a low level, all you need to do is tell bots not to visit any add to cart links. Simple enough?
In the website’s root directory look for the file called robots.txt. If it’s not there, you can just create it.
Most kind-hearted robots/crawlers like Googlebot and MsnBot should respect any rules in your robots.txt file. Have a look at the below example. We are telling bots not to crawl any links that contain add-to-cart= in the URL.
The star * represents ‘anything’. We use the star symbol because each URL for adding a product to the cart is unique, and contains the product id. E.g add-to-cart=21
We’ll also prevent bots from going to pages where the isn’t any SEO juice.
Create a new file called robots.txt and copy the below contents into it. You want to save the file in the root of your webspace. You should be able to access the file by going to yourdomain.com/robots.txt
User-agent: *
Disallow: /*add-to-cart=*
Disallow: /cart/
Disallow: /checkout/
Disallow: /my-account/Code language: HTTP (http)What is the performance impact?
Have a look at this article by Closte (awesome hosting provider). Closte put together some graphs where you can see just how much load robots added to the server. It’s quite a lot, especially if you are running WooCommerce stores on a large scale.
Reserved Stock Pro for WooCommerce & Robots
Reserved Stock Pro is our plugin for allowing customers to hold products in their carts for a limited amount of time. We want to make sure that only real customers get to reserve stock in their cart, not robots. Therefore we do have specific code to make sure that an add-to-cart request isn’t sent by a common robot. However, updating your robots.txt file to disallow robots in the first place will greatly lessen the chance of any robots reserving stock.