Bots that copy your prices and catalogue: how to recognise and stop them

The shop is slow at certain times, the statistics show visits that never buy anything, and one day you find your photos and descriptions on another website, with your prices undercut slightly. It is no coincidence: part of the traffic of any online shop is bots that crawl the catalogue and take whatever they find.

What kinds of bots visit a shop

Not all of them are bad, and blocking them all is a costly mistake. It is worth telling them apart:

• Search engines: Google, Bing and others. You want them, because without them you do not rank.
• Services you use: comparison sites where you list products, SEO tools, uptime monitors, your marketing platform's crawler.
• Artificial intelligence crawlers, which collect content to train models or to answer questions. Everyone can decide for themselves, but the decision should be yours.
• Price-monitoring bots, hired by competitors or by services that sell that information. They visit your product pages daily to find out what you charge and whether you have stock.
• Catalogue copiers, which download product pages, descriptions and images to set up another shop or feed a marketplace.
• Abuse bots: they test cards at checkout, create fake accounts, try passwords on the customer login or look for vulnerable modules.

What damage they do

Server load

A copying bot does not browse like a person: it requests hundreds of pages in a row, combines every filter in the faceted navigation and requests pages nobody visits that are not cached. Each one forces PrestaShop to calculate prices, stock and combinations. On tight hosting, that shows in the speed your real customers experience.

An advantage for competitors

If a competitor knows your prices and stock in real time, they can adjust theirs to always stay slightly below. Public prices are public and that cannot be avoided entirely, but you can make it harder for them.

Duplicate content

If your descriptions appear on other websites, search engines have to decide which is the original. They usually get it right, but not always.

Fake statistics

Bot visits inflate traffic and sink the apparent conversion rate, leading you to make decisions with dirty data.

How to spot them in the logs

The main tool is the server access logs (the hosting's, not the shop's statistics, which do not see most bots because they do not run JavaScript). What gives a bot away:

1. Volume and rhythm. Many requests per minute from the same IP or the same range, at a steady pace, at any hour.
2. The path it takes. Only product pages, in order of ID, or every filter combination in a category. Never the cart, never the home page.
3. The user agent. Legitimate bots identify themselves. Many copiers pose as a normal browser, but sometimes an old one, or the same one for thousands of requests.
4. The origin. Traffic from data centres and cloud providers instead of home or mobile connections. A person rarely browses from a server.
5. No resources. They request the HTML but not the stylesheets, small images or scripts.

Which measures work

On the server

• Rate limiting per IP: no address should be able to request more than a certain number of pages per minute. A person is not affected; a copier is, a lot.
• A web application firewall with bot rules, at the hosting or in a service in front of the shop. Many can tell verified search engines from the rest.
• Block data-centre ranges that bring you nothing, with care: your own services may also come from there.

In the shop

• Full-page cache wherever possible: if pages are served from cache, the bot puts far less load on the server.
• Do not index useless filter combinations: controlling which faceted navigation URLs can be crawled shrinks the ground where bots get lost, the good ones too.
• robots.txt for bots that respect it: search engines and most well-known AI crawlers. Copiers ignore it, so it is not protection but a way of organising legitimate traffic.
• Protect the login and registration forms against mass attempts.

In the content

• Images with your watermark if having your photos copied harms you.
• Your own descriptions, not the supplier's: what is yours and adds something is easier to defend.

What not to do

• Block everything that is not a known browser without verifying: you will end up blocking a search engine or your own comparison site.
• Put a captcha in front of every visitor: you punish your customers because of the bots.
• Hide prices unless your model is B2B and justifies it.
• React to a single IP: serious copiers rotate addresses. Look for patterns, not addresses.

Bots looking for vulnerabilities

Among all that traffic are bots that are not after prices but modules with known flaws, forgotten installation folders or administration panels with the default name. Against those, the main measure is not blocking them: it is having nothing for them to find. PrestaShop, PHP and modules up to date, no orphaned modules left on disk and the back-office folder renamed.

How to solve it in PrestaShop

RKR Security Alert is not a firewall and does not block bots: it detects and alerts. What it does do is cover what those bots are looking for. Its audit flags an install/ folder that is still present, a back-office folder with the default name, a robots.txt that reveals the back-office path, backups accessible in the root and upload folders where PHP could be executed, and it alerts you via Telegram about changes to employees, modules and configuration and about back-office logins from new IP addresses. For the bot traffic itself, with a Rekire support hours pack we analyse your server logs, identify which bots are loading your server and set up the limits and rules that make sense on your hosting.