I think it is best to block AI bots. I use Cloudflare as the CDN for the whole website.
It is quite easy to make it happen.
Cloudflare AI Bots setting
You can choose to block AI bots by activating Block AI bots. Activating this setting will block verified bots that are classified as AI crawlers, as well as a number of unverified bots that behave similarly.
To block AI bots:
- Log in to the Cloudflare dashboard ↗, and select your account and domain.
- Go to Security > Settings.
- Filter by Bot traffic.
- Go to Block AI bots.
- Under Configurations, select the edit icon. Choose from:
- Only block on hostnames with ads: Use this option if you wish to block AI bots only on portions of your site that show ads. Cloudflare automatically detects whether ads are present on a subdomain, and only block on hostnames that contain those ad units.
- Block on all pages: Use this option if you wish to block AI bots on all your pages.
- Do not block (off): Use this option if you wish to allow AI bots on all your pages.
- Select Save to save your configuration.
The AI bot is just like Pixiu, which takes in but never gives back.
When Search Engine bots scrape the content of the website, the search results page will give the link to the original page. It is kind of giving credit to the original author. AI bot ate your content, used it, and you get nothing. Not even a name.
So, from 2025, I will put this security rule on all my websites.
