|
It is currently 11 May 2024, 14:30
|
View unsolved topics | View unanswered posts
|
|
|
|
Author |
Message |
admin
|
Post subject: Sessions table crashed, forum was offline (Botnet attack) Posted: 27 Nov 2023, 03:35 |
Site Admin |
|
|
Joined: 07 Aug 2002, 23:37 Posts: 4569 Location: Tokyo Has thanked: 299 times Been thanked: 1166 times
|
Hello! Not sure what happened but an automated script got stuck at 100% and the scripts started after pushed the CPU to 2,600% while hogging the database server to the point of damaging the session table. Process killed, remaining script slowly terminating and session table repaired -- you may need to reconnect via https://www.lddb.com/forum.php to trigger a new forum session. MySQL server restarted as well for safety. Normal #of URl hits in a month is around 6M. We are at 34M now. Someone/something decided to bring the website down by constantly hitting from various IPs or creating a complete backup off all URLs (which is stupid). Julien
_________________ HARDWARE DATABASE HLD-X0/9 LD-S9 OPPO 105/205 SL-1200G LDD-1 MSC-4000 R2144 PONTUS II C45 MC257
|
|
|
|
|
admin
|
Post subject: Re: Session table had crashed, forum was offline Posted: 27 Nov 2023, 04:14 |
Site Admin |
|
|
Joined: 07 Aug 2002, 23:37 Posts: 4569 Location: Tokyo Has thanked: 299 times Been thanked: 1166 times
|
admin wrote: Someone/something decided to bring the website down by constantly hitting from various IPs or creating a complete backup off all URLs (which is stupid).
That would be Alibaba Cloud (Singapore) Private Limited... Blocking the whole /12. Last time I had to block ByteDance and AWS Singapore... Julien
_________________ HARDWARE DATABASE HLD-X0/9 LD-S9 OPPO 105/205 SL-1200G LDD-1 MSC-4000 R2144 PONTUS II C45 MC257
|
|
|
|
|
admin
|
Post subject: Re: Sessions table had crashed, forum was offline Posted: 05 Dec 2023, 03:45 |
Site Admin |
|
|
Joined: 07 Aug 2002, 23:37 Posts: 4569 Location: Tokyo Has thanked: 299 times Been thanked: 1166 times
|
Finally managed to put a dynamic filter/ban ("apache-badbots") on the server: Only by catching these weird user-agents (Samsung Galaxy 5, Google Pixel 2): Linux; Android 5.0; SM-G900P Linux; Android 8.0; Pixel 2 I'm catching TONS of hits: 2023-12-05 03:43:51,773 fail2ban.filter [4181]: INFO [apache-badbots] Found 90.198.50.137 - 2023-12-05 03:43:51 2023-12-05 03:43:51,914 fail2ban.filter [4181]: INFO [apache-badbots] Found 5.80.133.50 - 2023-12-05 03:43:51 2023-12-05 03:43:52,016 fail2ban.filter [4181]: INFO [apache-badbots] Found 31.117.231.108 - 2023-12-05 03:43:52 2023-12-05 03:43:52,033 fail2ban.filter [4181]: INFO [apache-badbots] Found 131.217.255.240 - 2023-12-05 03:43:52 2023-12-05 03:43:52,086 fail2ban.filter [4181]: INFO [apache-badbots] Found 90.198.50.137 - 2023-12-05 03:43:52 2023-12-05 03:43:52,158 fail2ban.actions [4181]: NOTICE [apache-badbots] Ban 81.38.68.112 2023-12-05 03:43:52,249 fail2ban.filter [4181]: INFO [apache-badbots] Found 5.80.133.50 - 2023-12-05 03:43:52 2023-12-05 03:43:52,249 fail2ban.filter [4181]: INFO [apache-badbots] Found 220.245.130.162 - 2023-12-05 03:43:52 2023-12-05 03:43:52,359 fail2ban.filter [4181]: INFO [apache-badbots] Found 86.140.36.49 - 2023-12-05 03:43:52 2023-12-05 03:43:52,489 fail2ban.filter [4181]: INFO [apache-badbots] Found 90.198.50.137 - 2023-12-05 03:43:52 2023-12-05 03:43:52,601 fail2ban.actions [4181]: NOTICE [apache-badbots] Ban 94.0.61.232 2023-12-05 03:43:52,623 fail2ban.filter [4181]: INFO [apache-badbots] Found 131.217.255.240 - 2023-12-05 03:43:52 2023-12-05 03:43:52,892 fail2ban.filter [4181]: INFO [apache-badbots] Found 220.245.130.162 - 2023-12-05 03:43:52 2023-12-05 03:43:53,017 fail2ban.actions [4181]: NOTICE [apache-badbots] Ban 86.162.149.181 2023-12-05 03:43:53,238 fail2ban.filter [4181]: INFO [apache-badbots] Found 131.217.255.240 - 2023-12-05 03:43:53 2023-12-05 03:43:53,452 fail2ban.actions [4181]: NOTICE [apache-badbots] Ban 90.207.160.147 2023-12-05 03:43:53,553 fail2ban.filter [4181]: INFO [apache-badbots] Found 220.245.130.162 - 2023-12-05 03:43:53 2023-12-05 03:43:53,830 fail2ban.filter [4181]: INFO [apache-badbots] Found 131.217.255.240 - 2023-12-05 03:43:53 2023-12-05 03:43:53,872 fail2ban.actions [4181]: NOTICE [apache-badbots] Ban 115.188.111.129 2023-12-05 03:43:54,207 fail2ban.filter [4181]: INFO [apache-badbots] Found 220.245.130.162 - 2023-12-05 03:43:54 2023-12-05 03:43:54,298 fail2ban.actions [4181]: NOTICE [apache-badbots] Ban 120.148.60.134 2023-12-05 03:43:54,427 fail2ban.filter [4181]: INFO [apache-badbots] Found 131.217.255.240 - 2023-12-05 03:43:54 2023-12-05 03:43:54,707 fail2ban.actions [4181]: NOTICE [apache-badbots] Ban 161.29.139.181 2023-12-05 03:43:54,724 fail2ban.filter [4181]: INFO [apache-badbots] Found 89.19.88.35 - 2023-12-05 03:43:54 2023-12-05 03:43:54,908 fail2ban.filter [4181]: INFO [apache-badbots] Found 220.245.130.162 - 2023-12-05 03:43:54 2023-12-05 03:43:55,011 fail2ban.filter [4181]: INFO [apache-badbots] Found 131.217.255.240 - 2023-12-05 03:43:55 2023-12-05 03:43:55,117 fail2ban.actions [4181]: NOTICE [apache-badbots] Ban 161.29.255.250 2023-12-05 03:43:55,306 fail2ban.filter [4181]: INFO [apache-badbots] Found 89.19.88.35 - 2023-12-05 03:43:55 2023-12-05 03:43:55,531 fail2ban.actions [4181]: NOTICE [apache-badbots] Ban 139.218.139.47
1,635 IPs banned for 30 days after only a few minutes and it's not slowing down. New ones are popping up whenever old ones are blocked. Curious too see how high the count will go, but I expect more than 10K IPs. Julien
_________________ HARDWARE DATABASE HLD-X0/9 LD-S9 OPPO 105/205 SL-1200G LDD-1 MSC-4000 R2144 PONTUS II C45 MC257
|
|
|
|
|
admin
|
Post subject: Re: Sessions table had crashed, forum was offline Posted: 06 Dec 2023, 03:49 |
Site Admin |
|
|
Joined: 07 Aug 2002, 23:37 Posts: 4569 Location: Tokyo Has thanked: 299 times Been thanked: 1166 times
|
I had to restart the blocking using ipset for iptables because iptables alone was too slow. Passed 25K entries, it was taking 3 seconds to add another one. With IPSET, I already maxed out the 65,535 limit and had to split into 3 additional ipsets. Currently blocking about 69,631 IPs, this is definitely a big botnet. Julien
_________________ HARDWARE DATABASE HLD-X0/9 LD-S9 OPPO 105/205 SL-1200G LDD-1 MSC-4000 R2144 PONTUS II C45 MC257
|
|
|
|
|
admin
|
Post subject: Re: Sessions table had crashed, forum was offline Posted: 06 Dec 2023, 04:07 |
Site Admin |
|
|
Joined: 07 Aug 2002, 23:37 Posts: 4569 Location: Tokyo Has thanked: 299 times Been thanked: 1166 times
|
admin wrote: Having thousands of IP addresses compromised to piggyback is entirely different! Given the locations of the IPs + origin subnet, I would say that a popular smarthphone App (in English) has been comprised, maybe via a poisoned library, and it is now operating as a botnet, hitting site/IP/port on request. Why is LDDb.com a target? I have no idea. And the flooding level is not enough to bring the server down either. Usually these attacks are not free, someone pays to initiate. So... why? Julien
_________________ HARDWARE DATABASE HLD-X0/9 LD-S9 OPPO 105/205 SL-1200G LDD-1 MSC-4000 R2144 PONTUS II C45 MC257
|
|
|
|
|
admin
|
Post subject: Re: Sessions table had crashed, forum was offline Posted: 07 Dec 2023, 03:48 |
Site Admin |
|
|
Joined: 07 Aug 2002, 23:37 Posts: 4569 Location: Tokyo Has thanked: 299 times Been thanked: 1166 times
|
Found the 4 user-agents used by almost all hits: Mozilla/5.0 (Linux; Android 5.0; SM-G900P Build/LRX21T) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2223.1058 Mobile Safari/537.36 Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.5064.1455 Mobile Safari/537.36 Mozilla/5.0 (Linux; Android 8.0; Pixel 2 Build/OPD3.170816.012) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2733.1676 Mobile Safari/537.36 Mozilla/5.0 (iPhone; CPU iPhone OS 11_0 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.4917.1427 Mobile Safari/537.36
Right now, blocking 216,194 IPs, and more keep coming every second. Julien
_________________ HARDWARE DATABASE HLD-X0/9 LD-S9 OPPO 105/205 SL-1200G LDD-1 MSC-4000 R2144 PONTUS II C45 MC257
|
|
|
|
|
admin
|
Post subject: Re: Sessions table had crashed, forum was offline Posted: 08 Dec 2023, 03:30 |
Site Admin |
|
|
Joined: 07 Aug 2002, 23:37 Posts: 4569 Location: Tokyo Has thanked: 299 times Been thanked: 1166 times
|
Normal hits rate is 6~7M, we reached 36M before the log parser gave up on Nov 27th. Attachment:
Botnet.png [ 6.03 KiB | Viewed 798 times ]
It was taking more than 10 minutes to process the logs each 10 minutes... leading to a racing processes depleting the memory (half the SWAP had to be used), crashing the forum dabatase, and making each web request painfully slow. It seems that something weird started in October as well (bandwidth consumed jumped from ~100GB to ~1TB). The stats finally resumed from today (Dec 8th) as I can't back-process the GB of logs generated by the flooding from Nov 27. We should be OK now, only saw 8 hits in the past 15 minutes! Latest count was 379,927 IPs blocked. Julien
_________________ HARDWARE DATABASE HLD-X0/9 LD-S9 OPPO 105/205 SL-1200G LDD-1 MSC-4000 R2144 PONTUS II C45 MC257
|
|
|
|
|
admin
|
Post subject: Re: Sessions table crashed, forum was offline (Botnet attack Posted: 08 Dec 2023, 07:56 |
Site Admin |
|
|
Joined: 07 Aug 2002, 23:37 Posts: 4569 Location: Tokyo Has thanked: 299 times Been thanked: 1166 times
|
substance wrote: I heard Ricardo went back to college for a computer science degree after giving up on Laserdiscs So much passive-agressivity and no showing off ... I'd say it sounds more like forper! Julien
_________________ HARDWARE DATABASE HLD-X0/9 LD-S9 OPPO 105/205 SL-1200G LDD-1 MSC-4000 R2144 PONTUS II C45 MC257
|
|
|
|
|
substance
|
Post subject: Re: Sessions table crashed, forum was offline (Botnet attack Posted: 08 Dec 2023, 08:31 |
Confirmed Padawan |
|
|
Joined: 16 May 2009, 18:05 Posts: 3592 Location: California, USA Has thanked: 28 times Been thanked: 328 times
|
admin wrote: substance wrote: I heard Ricardo went back to college for a computer science degree after giving up on Laserdiscs So much passive-agressivity and no showing off ... I'd say it sounds more like forper! Julien Forper? Maybe Fortran or Basic or whatever for analog computing (Abacus?)
_________________ Coming Soon Derman Labs Anything Of Substance
|
|
|
|
|
admin
|
Post subject: Re: Sessions table had crashed, forum was offline Posted: 08 Dec 2023, 09:51 |
Site Admin |
|
|
Joined: 07 Aug 2002, 23:37 Posts: 4569 Location: Tokyo Has thanked: 299 times Been thanked: 1166 times
|
Nov 23 - Dec 8Mozilla/5.0 (Linux; Android 5.0; SM-G900P Build/LRX21T) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2223.1058 Mobile Safari/537.36
=> 7,891,465 hits Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.5064.1455 Mobile Safari/537.36
=> 7,892,724 hits Mozilla/5.0 (Linux; Android 8.0; Pixel 2 Build/OPD3.170816.012) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2733.1676 Mobile Safari/537.36
=> 7,900,601 hits Mozilla/5.0 (iPhone; CPU iPhone OS 11_0 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.4917.1427 Mobile Safari/537.36
=> 7,970,588 hits Well balanced, clearly not random. Julien
_________________ HARDWARE DATABASE HLD-X0/9 LD-S9 OPPO 105/205 SL-1200G LDD-1 MSC-4000 R2144 PONTUS II C45 MC257
|
|
|
|
|
admin
|
Post subject: Re: Session table had crashed, forum was offline Posted: 08 Dec 2023, 10:46 |
Site Admin |
|
|
Joined: 07 Aug 2002, 23:37 Posts: 4569 Location: Tokyo Has thanked: 299 times Been thanked: 1166 times
|
admin wrote: That would be Alibaba Cloud (Singapore) Private Limited... And permanently blocked Hong Kong Aberdeen Alibaba Cloud Llc who was also quickly harvesting all the forum. Julien
_________________ HARDWARE DATABASE HLD-X0/9 LD-S9 OPPO 105/205 SL-1200G LDD-1 MSC-4000 R2144 PONTUS II C45 MC257
|
|
|
|
|
admin
|
Post subject: Re: Sessions table crashed, forum was offline (Botnet attack Posted: 06 Jan 2024, 17:54 |
Site Admin |
|
|
Joined: 07 Aug 2002, 23:37 Posts: 4569 Location: Tokyo Has thanked: 299 times Been thanked: 1166 times
|
admin wrote: It's been a month and the 320K+ blocked IPs are slowly getting removed from the ban lists. Done. All botnet IPs have been unblocked now. I still see a few hits but few come back for another attempt, and they get blocked right away if they do. Julien
_________________ HARDWARE DATABASE HLD-X0/9 LD-S9 OPPO 105/205 SL-1200G LDD-1 MSC-4000 R2144 PONTUS II C45 MC257
|
|
|
|
|
|
|
|
|
|
|
You cannot post new topics in this forum You cannot reply to topics in this forum You cannot edit your posts in this forum You cannot delete your posts in this forum You cannot post attachments in this forum
|
|