+- Support Forums (https://www.supportforums.net)
+-- Forum: Categories (https://www.supportforums.net/forumdisplay.php?fid=87)
+--- Forum: Life Support (https://www.supportforums.net/forumdisplay.php?fid=12)
+---- Forum: Omni's Secret Hidden Inside Tricks (https://www.supportforums.net/forumdisplay.php?fid=13)
+---- Thread: GoogleBot Loves My Forum So Much... (/showthread.php?tid=11439)
Pages:
1
2
RE: GoogleBot Loves My Forum So Much... - Solidus - 10-15-2010
This is why you only give bots the same permissions as guests, otherwise it will index private links such as moderator sections and the administrator panel.
RE: GoogleBot Loves My Forum So Much... - DAMINK™ - 10-15-2010
(10-15-2010, 05:31 PM)Jay Wrote: This is why you only give bots the same permissions as guests, otherwise it will index private links such as moderator sections and the administrator panel.
In theory you could block the admin links etc in robots.txt and still have a bot as registered or god forbid higher!
However i think its considered bad by google if it can see content others can not.
For example if you made google a member and your forum was only visible to logged in users.
This could get your forum blacklisted even i suspect.
So yes i agree. Bots have to see exactly what users not logged in can see.
RE: GoogleBot Loves My Forum So Much... - deantehnoob - 10-15-2010
yea googlebot is allways online going through forums, googlebot is everywhere at the same time. god od the interwebz
RE: GoogleBot Loves My Forum So Much... - Resistance - 10-16-2010
Hahaha, @DAMINK...
Lol "Google and all spiders scrape up all links on your forum. (unless in robots.txt file as deny)"
RE: GoogleBot Loves My Forum So Much... - Fremo - 10-17-2010
maybe its stalking people :O
RE: GoogleBot Loves My Forum So Much... - DAMINK™ - 10-17-2010
(10-16-2010, 12:01 AM)L3g1tWa5te Wrote: Hahaha, @DAMINK...
Lol "Google and all spiders scrape up all links on your forum. (unless in robots.txt file as deny)"
Yep exactly what i said in my 2 posts in this thread.
However some bots dont obey this rule. Many actually.
Some actually look for the deny and scrape those.
However that really is getting a little off topic i guess.
RE: GoogleBot Loves My Forum So Much... - Kaleb - 10-17-2010
My googlebot is always viewing the no permission page, i have no freakin idea what its trying to do.
RE: GoogleBot Loves My Forum So Much... - NekoChan - 10-18-2010
(10-17-2010, 06:28 PM)Kaleb Wrote: My googlebot is always viewing the no permission page, i have no freakin idea what its trying to do.
It is haxxoring your forum right now, go and ban it!
RE: GoogleBot Loves My Forum So Much... - DAMINK™ - 10-23-2010
![[Image: 12878930704218_download.jpg]](http://www.ftw.net.au/fileuploads2/12878930704218_download.jpg)
Noticed this today.
Damn google bot!
RE: GoogleBot Loves My Forum So Much... - Positive - 10-23-2010
(08-21-2010, 01:32 PM)Omniscient Wrote: As the biggest search engine online it does the most indexing. It's normal to see Googlebot on your site often.
You would be the best person to talk to about that. Wouldn't you. Everytime i go on HF, i look at the Home page and see a new member then refresh and you have a new sign up.
