Support Forums

This is why you only give bots the same permissions as guests, otherwise it will index private links such as moderator sections and the administrator panel.

(10-15-2010, 05:31 PM)Jay Wrote: [ -> ]This is why you only give bots the same permissions as guests, otherwise it will index private links such as moderator sections and the administrator panel.

In theory you could block the admin links etc in robots.txt and still have a bot as registered or god forbid higher!
However i think its considered bad by google if it can see content others can not.
For example if you made google a member and your forum was only visible to logged in users.
This could get your forum blacklisted even i suspect.

So yes i agree. Bots have to see exactly what users not logged in can see.

yea googlebot is allways online going through forums, googlebot is everywhere at the same time. god od the interwebz

Hahaha, @DAMINK...

Lol "Google and all spiders scrape up all links on your forum. (unless in robots.txt file as deny)"

maybe its stalking people :O

(10-16-2010, 12:01 AM)L3g1tWa5te Wrote: [ -> ]Hahaha, @DAMINK...

Lol "Google and all spiders scrape up all links on your forum. (unless in robots.txt file as deny)"

Yep exactly what i said in my 2 posts in this thread.
However some bots dont obey this rule. Many actually.
Some actually look for the deny and scrape those.
However that really is getting a little off topic i guess.

My googlebot is always viewing the no permission page, i have no freakin idea what its trying to do.

(10-17-2010, 06:28 PM)Kaleb Wrote: [ -> ]My googlebot is always viewing the no permission page, i have no freakin idea what its trying to do.

It is haxxoring your forum right now, go and ban it!

Noticed this today.
Damn google bot!

(08-21-2010, 01:32 PM)Omniscient Wrote: [ -> ]As the biggest search engine online it does the most indexing. It's normal to see Googlebot on your site often.

You would be the best person to talk to about that. Wouldn't you. Everytime i go on HF, i look at the Home page and see a new member then refresh and you have a new sign up.