|
It is currently 11 Dec 2024, 16:03
|
View unsolved topics | View unanswered posts
|
|
|
|
admin
|
Post subject: Re: Googlebot says I'm blocking them... Posted: 12 Jun 2024, 19:40 |
Site Admin |
|
|
Joined: 07 Aug 2002, 23:37 Posts: 4756 Location: Tokyo Has thanked: 323 times Been thanked: 1285 times
|
mth1986 wrote: Craw-delay: 10
I guess due to the missing "l" it is interpreted as "disallow" or ignored altogether. Well spotted! Trying to re-validate with Google to see if they like it better. Julien
_________________ HARDWARE DATABASE HLD-X0/9 LD-S9 OPPO 105/205 SL-1200G LDD-1 MSC-4000 R2144 PONTUS II C45 MC257
|
|
|
|
|
admin
|
Post subject: Re: Googlebot says I'm blocking them... Posted: 22 Jun 2024, 11:47 |
Site Admin |
|
|
Joined: 07 Aug 2002, 23:37 Posts: 4756 Location: Tokyo Has thanked: 323 times Been thanked: 1285 times
|
Google robots-test page seems to consider ANY directive to be aimed at them, regardless of the user-agent defined above. I'll rewrite the robots.txt from scratch but I think Google changed something ~1 month ago. Anyway, back on track: For LDDb.com Quote: Google has started validating your fix of Page indexing issues on your site. Specifically, we are checking for ‘Blocked by robots.txt’, which currently affects 602292 pages. Quote: Google has started validating your fix of Page indexing issues on your site. Specifically, we are checking for ‘Crawled - currently not indexed’, which currently affects 2290577 pages. For forum.LDDb.com (not yet started) I found that, since 2021, Google is publishing the list of their crawlers' IP addresses on https://developers.google.com/search/apis/ipranges/googlebot.json. I'll pre-allow them for safety. Julien
_________________ HARDWARE DATABASE HLD-X0/9 LD-S9 OPPO 105/205 SL-1200G LDD-1 MSC-4000 R2144 PONTUS II C45 MC257
|
|
|
|
|
admin
|
Post subject: Re: Googlebot says I'm blocking them... Posted: 04 Jul 2024, 23:45 |
Site Admin |
|
|
Joined: 07 Aug 2002, 23:37 Posts: 4756 Location: Tokyo Has thanked: 323 times Been thanked: 1285 times
|
mth1986 wrote: Just curious: Is the problem solved? Only by completely removing the robots.txt file, which is really weird. It used to work before for many, many years! Something happened at Google that we haven't heard of yet. Julien
_________________ HARDWARE DATABASE HLD-X0/9 LD-S9 OPPO 105/205 SL-1200G LDD-1 MSC-4000 R2144 PONTUS II C45 MC257
|
|
|
|
|
admin
|
Post subject: Re: Googlebot says I'm blocking them... Posted: 17 Jul 2024, 03:14 |
Site Admin |
|
|
Joined: 07 Aug 2002, 23:37 Posts: 4756 Location: Tokyo Has thanked: 323 times Been thanked: 1285 times
|
I was not paranoid...! https://tech.slashdot.org/story/24/07/16/1843240/google-now-defaults-to-not-indexing-your-contentQuote: Google is no longer trying to index the entire web. In fact, it's become extremely selective, refusing to index most content. This isn't about content creators failing to meet some arbitrary standard of quality. Rather, it's a fundamental change in how Google approaches its role as a search engine.
From my experience, Google now seems to operate on a "default to not index" basis. It only includes content in its index when it perceives a genuine need. I can tell you it started around May 13th and increased until I removed the robots.txt altogether and asked for a re-indexing around June 22nd. Attachment:
block_by_robots.txt.png [ 13.1 KiB | Viewed 859 times ]
Attachment:
LDDb-indexed.png [ 19.14 KiB | Viewed 859 times ]
For some reason, it didn't work on this Forum, we are slowly disappearing from Google searches... Attachment:
block_by_robots.txt-Forum.png [ 10.53 KiB | Viewed 859 times ]
Attachment:
Forum-indexed.png [ 18.17 KiB | Viewed 859 times ]
Julien
_________________ HARDWARE DATABASE HLD-X0/9 LD-S9 OPPO 105/205 SL-1200G LDD-1 MSC-4000 R2144 PONTUS II C45 MC257
|
|
|
|
|
admin
|
Post subject: Re: Googlebot says I'm blocking them... Posted: 28 Jul 2024, 09:09 |
Site Admin |
|
|
Joined: 07 Aug 2002, 23:37 Posts: 4756 Location: Tokyo Has thanked: 323 times Been thanked: 1285 times
|
Solution was to get rid of robots.txt and block AI crawlers / pre-allow Google and few others to crawl. It seemed to improve Google indexing on the main LDDb.com: Attachment:
LDDb.png [ 17.65 KiB | Viewed 796 times ]
But not for the Forum: Attachment:
Forum.png [ 17.54 KiB | Viewed 796 times ]
Here is the bandwidth (== Out) when the AI crawlers started acting crazy leading to 2 rounds of 30-day block per IP/User-Agent: Attachment:
Clipboard_07-28-2024_01.png [ 43.34 KiB | Viewed 796 times ]
Julien
_________________ HARDWARE DATABASE HLD-X0/9 LD-S9 OPPO 105/205 SL-1200G LDD-1 MSC-4000 R2144 PONTUS II C45 MC257
|
|
|
|
|
admin
|
Post subject: Re: Googlebot says I'm blocking them... Posted: 31 Jul 2024, 02:20 |
Site Admin |
|
|
Joined: 07 Aug 2002, 23:37 Posts: 4756 Location: Tokyo Has thanked: 323 times Been thanked: 1285 times
|
Oh... Google does not honor crawl-rate anymore :-/ https://developers.google.com/search/blog/2023/11/sc-crawl-limiter-byebyehttps://developers.google.com/search/docs/crawling-indexing/large-site-managing-crawl-budget#emergenciesQuote: The non-standard "crawl-delay" robots.txt rule is not processed by Googlebot. That sure doesn't help. Here is the top 10 crawlers this month: Hosts : 30,192 Known, 23,483 Unknown (unresolved ip) 37,617 Unique visitors Pages Hits Bandwidth crawl-66-249-79-75.googlebot.com 267,211 267,211 2.53 GB crawl-66-249-76-69.googlebot.com 217,311 217,311 2.19 GB crawl-66-249-79-76.googlebot.com 214,158 214,158 2.04 GB crawl-66-249-76-70.googlebot.com 166,465 166,465 1.69 GB crawl-66-249-79-64.googlebot.com 164,493 164,493 1.58 GB crawl-66-249-76-71.googlebot.com 120,471 120,471 1.23 GB crawl-66-249-79-65.googlebot.com 111,521 111,521 1.10 GB crawl-66-249-76-72.googlebot.com 78,509 78,509 828.70 MB crawl-66-249-79-66.googlebot.com 71,149 71,149 728.98 MB crawl-66-249-66-165.googlebot.com 56,581 56,581 537.56 MB
If you feel the website is slower than usual, thank Google for that. Attachment:
CPU.png [ 53.56 KiB | Viewed 781 times ]
Julien
_________________ HARDWARE DATABASE HLD-X0/9 LD-S9 OPPO 105/205 SL-1200G LDD-1 MSC-4000 R2144 PONTUS II C45 MC257
|
|
|
|
|
|
|
|
|
|
|
You cannot post new topics in this forum You cannot reply to topics in this forum You cannot edit your posts in this forum You cannot delete your posts in this forum You cannot post attachments in this forum
|
|