RSS Subscriber analytics

19 Feb, 2025

With the dramatic rise of LLM scraper bots and other data collection services, the RSS and Atom feeds of blogs on all platforms, including Bear, are being mercilessly scraped.

These bots do not identify themselves as scraper bots, but as browsers and RSS readers. Unfortunately there is no way (that I'm currently aware of¹) to determine whether this is a legitimate RSS reader or a bot since the IP addresses change with each request. This suggests that the bots are part of a broader bot network (or many bot networks) and completely ignore robots.txt.

Since RSS readers cannot execute Javascript or CSS, I can't present a challenge or captcha on the feeds since this will block both legitimate and bot traffic.

This has lead to the RSS subscriber count in analytics being completely incorrect, since each bot IP address is earmarked as a unique RSS subscriber during a 24 hour period.

This is compounded by the fact that large RSS reader platforms like Feedly do one request for all of their users subscribed to a specific feed, essentially logging many subscribers as 1.

Due to these reasons I've opted to remove the RSS subscriber count from the analytics dashboard.

Herman