It is currently 17 Jul 2024, 10:18

All times are UTC [ DST ]




 Page 1 of 1 [ 13 posts ] 
Author Message
 Post subject: Googlebot says I'm blocking them...
PostPosted: 12 Jun 2024, 08:52 
Site Admin
Site Admin
User avatar

Joined: 07 Aug 2002, 23:37
Posts: 4651
Location: Tokyo
Has thanked: 312 times
Been thanked: 1214 times
... and so the number of results on Google for LDDb or forum-related topics is slowly disappearing.

Problem is: I do not block Google

robots.txt is here => (file removed)

Does any of you know what filter breaks Google?

Julien
_________________
HARDWARE DATABASE
HLD-X0/9 LD-S9 OPPO 105/205 SL-1200G
LDD-1 MSC-4000 R2144 PONTUS II C45 MC257
Offline
 Profile  
 
 Post subject: Re: Googlebot ays I'm blocking them...
PostPosted: 12 Jun 2024, 09:00 
Absolute fan
Absolute fan
User avatar

Joined: 11 Jun 2008, 06:10
Posts: 1626
Location: Milky Way-Sol System-Terra-USA-North Carlolina.
Has thanked: 588 times
Been thanked: 246 times
What does that mean exactly ?
_________________
Acta Non Verba .....
Si Vis Pacem Para Bellum ....
Si Gorgiamus Allos Subjectatos Nunc ......
Offline
 Profile  
 
 Post subject: Re: Googlebot says I'm blocking them...
PostPosted: 12 Jun 2024, 09:17 
Serious fan
Serious fan
User avatar

Joined: 22 Oct 2023, 20:20
Posts: 153
Location: Germany
Has thanked: 55 times
Been thanked: 82 times
I am not an expert on the topic.
That said:
I think the problem is a simple typo in the robots.txt.

"User-agent: Googlebot
Craw-delay: 10"

should be:
"User-agent: Googlebot
Crawl-delay: 10"

I guess due to the missing "l" it is interpreted as "disallow" or ignored alltogether.

Cheers,
Markus
Offline
 Profile  
 
 Post subject: Re: Googlebot says I'm blocking them...
PostPosted: 12 Jun 2024, 19:40 
Site Admin
Site Admin
User avatar

Joined: 07 Aug 2002, 23:37
Posts: 4651
Location: Tokyo
Has thanked: 312 times
Been thanked: 1214 times
mth1986 wrote:
Craw-delay: 10

I guess due to the missing "l" it is interpreted as "disallow" or ignored altogether.


Well spotted!

Trying to re-validate with Google to see if they like it better.

Julien
_________________
HARDWARE DATABASE
HLD-X0/9 LD-S9 OPPO 105/205 SL-1200G
LDD-1 MSC-4000 R2144 PONTUS II C45 MC257
Offline
 Profile  
 
 Post subject: Re: Googlebot says I'm blocking them...
PostPosted: 18 Jun 2024, 09:41 
Site Admin
Site Admin
User avatar

Joined: 07 Aug 2002, 23:37
Posts: 4651
Location: Tokyo
Has thanked: 312 times
Been thanked: 1214 times
Didn't work, Googlebot ignores the crawl-delay anyway.

But I see that the sitemaps failed to be retrieved by Google and they never mentioned it before...

Re-activating the Sitemaps.

Julien
_________________
HARDWARE DATABASE
HLD-X0/9 LD-S9 OPPO 105/205 SL-1200G
LDD-1 MSC-4000 R2144 PONTUS II C45 MC257
Offline
 Profile  
 
 Post subject: Re: Googlebot says I'm blocking them...
PostPosted: 18 Jun 2024, 11:33 
Serious fan
Serious fan
User avatar

Joined: 22 Oct 2023, 20:20
Posts: 153
Location: Germany
Has thanked: 55 times
Been thanked: 82 times
Let's see if that did the trick. :problem:
Offline
 Profile  
 
 Post subject: Re: Googlebot says I'm blocking them...
PostPosted: 21 Jun 2024, 11:04 
Site Admin
Site Admin
User avatar

Joined: 07 Aug 2002, 23:37
Posts: 4651
Location: Tokyo
Has thanked: 312 times
Been thanked: 1214 times
It was still "blocked by robots.txt" when nothing was related to disallowing Google user-agents.

Removed the robots.txt and it worked again. No idea why.

Julien
_________________
HARDWARE DATABASE
HLD-X0/9 LD-S9 OPPO 105/205 SL-1200G
LDD-1 MSC-4000 R2144 PONTUS II C45 MC257
Offline
 Profile  
 
 Post subject: Re: Googlebot says I'm blocking them...
PostPosted: 21 Jun 2024, 16:36 
Serious fan
Serious fan
User avatar

Joined: 26 Mar 2024, 10:01
Posts: 208
Location: Australia
Has thanked: 113 times
Been thanked: 56 times
KILL Google
_________________
PM me if you want to swap s*bti*le files. Have a lot to share and currently need Mospeada TV.
Offline
 Profile  
 
 Post subject: Re: Googlebot says I'm blocking them...
PostPosted: 21 Jun 2024, 21:59 
Serious fan
Serious fan
User avatar

Joined: 22 Oct 2023, 20:20
Posts: 153
Location: Germany
Has thanked: 55 times
Been thanked: 82 times
Have you tried tools like:

https://technicalseo.com/tools/robots-txt/

https://sanofeld.de/robots-tester/

Never had to do so, but maybe it is worth a try.
For sanofeld you need to implement the robots.txt again so it can be checked.
Offline
 Profile  
 
 Post subject: Re: Googlebot says I'm blocking them...
PostPosted: 22 Jun 2024, 11:47 
Site Admin
Site Admin
User avatar

Joined: 07 Aug 2002, 23:37
Posts: 4651
Location: Tokyo
Has thanked: 312 times
Been thanked: 1214 times
Google robots-test page seems to consider ANY directive to be aimed at them, regardless of the user-agent defined above.
I'll rewrite the robots.txt from scratch but I think Google changed something ~1 month ago.

Anyway, back on track:

For LDDb.com
Quote:
Google has started validating your fix of Page indexing issues on your site. Specifically, we are checking for ‘Blocked by robots.txt’, which currently affects 602292 pages.

Quote:
Google has started validating your fix of Page indexing issues on your site. Specifically, we are checking for ‘Crawled - currently not indexed’, which currently affects 2290577 pages.


For forum.LDDb.com
(not yet started)

I found that, since 2021, Google is publishing the list of their crawlers' IP addresses on https://developers.google.com/search/apis/ipranges/googlebot.json.

I'll pre-allow them for safety.

Julien
_________________
HARDWARE DATABASE
HLD-X0/9 LD-S9 OPPO 105/205 SL-1200G
LDD-1 MSC-4000 R2144 PONTUS II C45 MC257
Offline
 Profile  
 
 Post subject: Re: Googlebot says I'm blocking them...
PostPosted: 04 Jul 2024, 19:23 
Serious fan
Serious fan
User avatar

Joined: 22 Oct 2023, 20:20
Posts: 153
Location: Germany
Has thanked: 55 times
Been thanked: 82 times
Just curious:
Is the problem solved?
Offline
 Profile  
 
 Post subject: Re: Googlebot says I'm blocking them...
PostPosted: 04 Jul 2024, 23:45 
Site Admin
Site Admin
User avatar

Joined: 07 Aug 2002, 23:37
Posts: 4651
Location: Tokyo
Has thanked: 312 times
Been thanked: 1214 times
mth1986 wrote:
Just curious:
Is the problem solved?


Only by completely removing the robots.txt file, which is really weird.
It used to work before for many, many years!

Something happened at Google that we haven't heard of yet.

Julien
_________________
HARDWARE DATABASE
HLD-X0/9 LD-S9 OPPO 105/205 SL-1200G
LDD-1 MSC-4000 R2144 PONTUS II C45 MC257
Offline
 Profile  
 
 Post subject: Re: Googlebot says I'm blocking them...
PostPosted: 17 Jul 2024, 03:14 
Site Admin
Site Admin
User avatar

Joined: 07 Aug 2002, 23:37
Posts: 4651
Location: Tokyo
Has thanked: 312 times
Been thanked: 1214 times
I was not paranoid...!

https://tech.slashdot.org/story/24/07/16/1843240/google-now-defaults-to-not-indexing-your-content

Quote:
Google is no longer trying to index the entire web. In fact, it's become extremely selective, refusing to index most content. This isn't about content creators failing to meet some arbitrary standard of quality. Rather, it's a fundamental change in how Google approaches its role as a search engine.

From my experience, Google now seems to operate on a "default to not index" basis. It only includes content in its index when it perceives a genuine need.


I can tell you it started around May 13th and increased until I removed the robots.txt altogether and asked for a re-indexing around June 22nd.

Attachment:
block_by_robots.txt.png
block_by_robots.txt.png [ 13.1 KiB | Viewed 6 times ]


Attachment:
LDDb-indexed.png
LDDb-indexed.png [ 19.14 KiB | Viewed 6 times ]


For some reason, it didn't work on this Forum, we are slowly disappearing from Google searches...

Attachment:
block_by_robots.txt-Forum.png
block_by_robots.txt-Forum.png [ 10.53 KiB | Viewed 6 times ]


Attachment:
Forum-indexed.png
Forum-indexed.png [ 18.17 KiB | Viewed 6 times ]


Julien
_________________
HARDWARE DATABASE
HLD-X0/9 LD-S9 OPPO 105/205 SL-1200G
LDD-1 MSC-4000 R2144 PONTUS II C45 MC257
Offline
 Profile  
 
Display posts from previous:  Sort by  
 Page 1 of 1 [ 13 posts ] 

All times are UTC [ DST ]


Who is online

Users browsing this forum: No registered users and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to: