Accepting projects ·CZ Get in touch →

When bots scrape your website: how caching protects your cloud bill

On our website with public data, database reads jumped 140-fold. What failed in the caching and what to check on your own website.

You pay for a website in the cloud by how much work it does. When a bot starts scraping it, the bill grows even though not a single extra person visited. We show this on our own website, where database reads jumped 140-fold.

What happened

We run a website with public data whose pages are built from the database only when someone requests them. In front of the pages sits a content delivery network (CDN), which is supposed to keep finished pages in its cache so that the database does as little work as possible.

On the evening of 7 May, a bot started scraping the website. It added random parameters to the URLs, so to the cache every URL was new. Every request therefore went all the way to the database. Reads rose from the usual thousands per hour to about a million, 140 times the normal level. Then they settled at around a hundred times normal and stayed there until the fix on the evening of 9 May.

We didn't identify the bot. The CDN access logs were switched off to save costs. So we built the fix to work against all bots.

We caught it thanks to a budget alarm that reported a higher estimate of the monthly bill before the invoice arrived. After the fix, database reads dropped from 423,000 units per hour to one or two.

Lukáš Hrubý Managing director, HRUBY Software

Why the cache didn't help

Three things failed at once. Each on its own would only have limited the damage. Together they let the bot through to the bottom.

  • The CDN ignored the website's caching instructions. The distribution used an older type of settings that doesn't take the website's Cache-Control header into account and keeps pages for only an hour.
  • URL parameters were part of the cache key. A URL with ?a=1 and one with ?a=2 were two different pages to the CDN.
  • The application didn't remember anything. Even the same page requested twice in a row was read from the database again.

Financially it turned out well. The bill estimate went up by 57 dollars a month. The same mistake on a bigger website or with a more expensive database would be counted in different orders of magnitude.

How we fixed it

  1. New cache settings in the CDN. They take the website's headers into account and ignore URL parameters, so random parameters no longer bypass the cache.
  2. Longer page lifetime. The data changes rarely, so detail pages may stay in the cache for 90 days. Listings and the sitemap for a shorter time.
  3. A cache inside the application. When the CDN still passes a request on, the application answers from memory and doesn't call the database.
  4. A stricter robots.txt. Search engines and AI search engines get longer intervals between requests. Data scraping tools that bring no search benefit are denied access.

The last step is more of a tidy-up. Well-behaved bots respect robots.txt, the others don't. The real protection is the cache.

What to check on your own website

  • Do you have a budget alarm set up? On the bill estimate, not just on actual spend. The estimate warns you days earlier.
  • Does a request with a nonsense parameter go all the way to the database? Try opening a page with ?test=123 and check in the response headers whether it came from the cache.
  • Does the website send a sensible Cache-Control header? A page that changes once a month doesn't need a lifetime of five minutes.
  • Do you keep access logs? They cost little, and in an incident they show who was behind it.

When to bring in an expert

Do you pay for the cloud by usage, and the bill grows faster than your traffic? Or can't you say what a month would cost in which an aggressive bot discovers you? We'll go through the CDN, cache and alarm settings and tell you where the ceiling is. Usually it takes a few configuration changes, not a rebuild of the application.

Written by Ing. Lukáš Hrubý, HRUBY Software · ← All articles

Dealing with the same thing?

We'll look at it with you. The first consultation is free. More about the service: Custom Software.

Book a consultation →
// Contact

Get in touch.

Tell us what you need, or just call. We reply within one business day.

// Encrypted in transit · privacy policy