Google crawler bot question...

Rogue 9 · Post by **Rogue 9** » 2005-03-11 05:57pm

Okay, I've got three IPs tracing to Google's little catalog bot thing on my forums right now. This wasn't a problem for me until I looked at the admin panel and noted that they were ALL looking at the moderator forum. How?!? Until about half an hour ago, I was having problems getting some of my mods to see it, and now guests are looking at it?!?

Edit: Make that nine iterations of the Google bot and counting, three of them in Moderators. What the hell is going on?

General Zod · Post by **General Zod** » 2005-03-11 06:44pm

does your board have some sort of archive feature? if so, it might have archived them before you were able to hide them and google's bots is reading the archived hidden boards.

Rogue 9 · Post by **Rogue 9** » 2005-03-11 08:45pm

No, it doesn't. The only thing that changed was that the mod forum had just been posted in for the first time, and they all scrambled to look at it. How, I have no idea; there's no way a guest should be able to see the forum, for obvious reasons.

Terr Fangbite · Post by **Terr Fangbite** » 2005-03-12 05:14pm

It may be that they tried to read them but got a page saying "guests cannot view this information" page. I heard that was a reason to a similar problem elsewhere.

Rogue 9 · Post by **Rogue 9** » 2005-03-12 08:51pm

A guest shouldn't even be able to see the forum link or even the category. I have it set for admins and the moderator and supermod groups only.

XaLEv · Post by **XaLEv** » 2005-03-13 07:59am

http://www.phpbb.com/phpBB/viewtopic.ph ... le+crawler

The thing is probably just checking different forum numbers. You might try creating a few forums that only you can see, with random numbers if you can and see what happens.

Vertigo1 · Post by **Vertigo1** » 2005-03-16 12:28am

I only see two choices for you. Ban the bot via robots.txt (which most search bots ignore anyhow) or ban the offending IP addresses via .htaccess. I bet when you ban it, your guest viewer count drops dramatically.

Post by **Darth Wong** » 2005-03-16 01:03am

It can try to access the mod forum but it won't be able to view it if you have your permissions set properly, so who cares?