Page 1 of 1

Google crawler bot question...

Posted: 2005-03-11 05:57pm
by Rogue 9
Okay, I've got three IPs tracing to Google's little catalog bot thing on my forums right now. This wasn't a problem for me until I looked at the admin panel and noted that they were ALL looking at the moderator forum. How?!? Until about half an hour ago, I was having problems getting some of my mods to see it, and now guests are looking at it?!?

Edit: Make that nine iterations of the Google bot and counting, three of them in Moderators. What the hell is going on?

Posted: 2005-03-11 06:44pm
by General Zod
does your board have some sort of archive feature? if so, it might have archived them before you were able to hide them and google's bots is reading the archived hidden boards.

Posted: 2005-03-11 08:45pm
by Rogue 9
No, it doesn't. The only thing that changed was that the mod forum had just been posted in for the first time, and they all scrambled to look at it. How, I have no idea; there's no way a guest should be able to see the forum, for obvious reasons.

Posted: 2005-03-12 05:14pm
by Terr Fangbite
It may be that they tried to read them but got a page saying "guests cannot view this information" page. I heard that was a reason to a similar problem elsewhere.

Posted: 2005-03-12 08:51pm
by Rogue 9
A guest shouldn't even be able to see the forum link or even the category. I have it set for admins and the moderator and supermod groups only.

Posted: 2005-03-13 07:59am
by XaLEv
http://www.phpbb.com/phpBB/viewtopic.ph ... le+crawler

The thing is probably just checking different forum numbers. You might try creating a few forums that only you can see, with random numbers if you can and see what happens.

Posted: 2005-03-16 12:28am
by Vertigo1
I only see two choices for you. Ban the bot via robots.txt (which most search bots ignore anyhow) or ban the offending IP addresses via .htaccess. I bet when you ban it, your guest viewer count drops dramatically.

Posted: 2005-03-16 01:03am
by Darth Wong
It can try to access the mod forum but it won't be able to view it if you have your permissions set properly, so who cares?