Additionally.... if you felt the website response time was slower than usual in the past few days, it was.
We had 2 players frantically harvesting (90% of the traffic was only for these 2) every pages on this website at high speed, bringing the database and web-server to its knees at times.
Who are they?
- OpenAI (yeah, the ChatGPT guys)
- ByteDance (as in TikTok)
OpenAI => IP blocked (
20.15.240.102 from Microsoft) + robots.txt updated to "disallow /" for
GPTBot/1.0; +https://openai.com/gptbotByteDance being a Chinese company was more interesting as I block ALL official IPs coming from China (they are 99.99% hack attempts/spammers).
But they were accessing via AWS Singapore => Whole /14 subnet blocked + robots.txt updated to "disallow /" for
Bytespider; spider-feedback@bytedance.comJulien