Search Engines Optimize

[sensovision]

I would need to select which boards could or couldn't be spidered by using robots.txt
On mine, some boards are 100% in need of a good spidering, but the rest is just plain chit chat and I don't fancy having the spiders crawl through 1500 pointless threads.

Any thought on how I could get around that problem ?

Fizzy, you still can use Unknown's mod even if you don't wish you boards to be spidered, just put in robots.txt

User-agent: *
Disallow: /

this code will force all legitimate search engines to avoid spidering of your pages.
check out this link for more info on how to write robots.txt
http://www.robotstxt.org/wc/exclusion.html

Folks just want to say that today I have new spider which wasn't moving anywhere from index page... this one was from LookSmart:

Mozilla/4.0 compatible ZyBorg/1.0 - [email protected]  (46)

[Fizzy]

Denis,

That's not what I said at all.

I want some boards to be spidered, but some others not to be spidered.
If 's mod only goes by message index number and not board number how can you tell robots.txt not to spider certain threads/boards?

How can I exclude some boards if the numbers are intermixed ?

[sensovision]

Denis,

That's not what I said at all.

I want some boards to be spidered, but some others not to be spidered.
If 's mod only goes by message index number and not board number how can you tell robots.txt not to spider certain threads/boards?

How can I exclude some boards if the numbers are intermixed ?

ah sorry:( English isn't my native language so I misunderstood you for the first time. as for excluding bots from certain pages I have an idea but not sure if it possible to do so... if you need it here it's so you can ask someone in mod ideas for help. You just need to make different boards to show different meta tags, e.g. on some board you will be able to place noindex and nofollow directive. so when particular thread will be genereted it will contain or not contain such directives..
But I see the easier solution IMHO... using something like boards for particular group, or something like Allow mod (I think it's doing this but I didn't check it for a while) which will make it possible to make some boards visible only after you'll log in.
Hope it would be some of help.

[Fizzy]

Hey Denis,

No need to apologise, I bet you speak English better than I do

I appreciate your help with answering the question.

Fortunately with the supermod you get the option of individual templates for the boards, I had not thought of using noindex metas in them Great suggestion.

Also I have added robots.txt denys for the board numbers and I'll see how I get on with that.

Code Select Expand


User-agent: *
Disallow: /cgi-bin/
Disallow: /yabbse/
Disallow: /yabbse/index.php?board=13
Disallow: /yabbse/index.php?board=14
Disallow: /yabbse/index.php?board=15
Disallow: /yabbse/index.php?board=10
Disallow: /yabbse/index.php?board=11
Disallow: /yabbse/index.php?board=17
Disallow: /yabbse/index.php?board=12


I will report back to let everyone know if I get problems.

[Surfy]

Is there any solution for IIS Webserver under W2K or Server 2003?

httacces solutions doestnt work on the IIS -


Surfy

[[Unknown]]

Is there any solution for IIS Webserver under W2K or Server 2003?

httacces solutions doestnt work on the IIS -


Surfy

The thread= thing works.

-[Unknown]

[Fizzy]

Since making the changes I have noticed that the "who's online" mod doesn't report threads as links any more

Nick (80.192.xx.xxx) Today at 10:28:34pm Viewing Who's Online.
Guest (64.68.82.165) Today at 10:26:31pm thread=452
Guest (64.68.xx.xxx) Today at 10:26:29pm thread=2592
Guest (64.68.82.37) Today at 10:26:16pm thread=3234
Guest (64.68.82.204) Today at 10:26:09pm thread=3011
Guest (64.68.82.34) Today at 10:26:08pm thread=2077
Guest (64.68.82.77) Today at 10:26:02pm thread=3090
Guest (64.68.82.176) Today at 10:26:00pm thread=3258
Guest (64.68.82.78) Today at 10:25:57pm thread=3167
Guest (64.68.82.189) Today at 10:25:55pm thread=3199
Guest (64.68.82.143) Today at 10:25:49pm thread=2901
Gobe (81.77.xx.xx) Today at 10:25:26pm Viewing the main index of SkinCell Forum.
Guest (80.194.62.52) Today at 10:06:07pm Viewing the main index of SkinCell Forum.
peterb (217.134.xx.xxx) Today at 10:05:58pm Viewing the main index of SkinCell Forum.
Cybe (81.77.xx.xxx) Today at 10:01:00pm thread=3003/msg36541
Monika (143.88.xx.xxx) Today at 09:54:53pm Viewing the main index of SkinCell Forum.

Any thoughts or suggestiosn how I could correct that please ?

[Fizzy]

Also the "report to moderator" function has gone south.
It redirects back to index.php

There's more.
The split, highlight etc buttons don't work either.

[[Unknown]]

No, and yes I know split does not work.

-[Unknown]

[Fizzy]

I'm mpst worried about the Report to moderator function not working.

Has it got anything to do with the fact that $cgi= has been added to index.php?
It's different to the one in subs.php

[[Unknown]]

I'm mpst worried about the Report to moderator function not working.

Has it got anything to do with the fact that $cgi= has been added to index.php?
It's different to the one in subs.php

Trust me, that's fine... it's the fact that it grabs the thread= before doing the split..

-[Unknown]

[Fizzy]

Any ideas on how to fix this please ?

[[Unknown]]

Any ideas on how to fix this please ?

Use SMF.

-[Unknown]

[Fizzy]

With an attitude like that Unknown?

You must be joking  

You wrote the code, it doesn't work and yet now you decide not to even try to help those that have used it?

If support on SMF is going to based on this then you can forget it.
SMF isn't even working yet and you have slammed the door on the users of YaBBSE. What does this say about the attitude towards the very same people who have made YaBBSE so much of a success.

At least many of the other coders give genuine support to their work.

[Ben_S]

Unknown is busy working on SMF, probably far too busy to bother looking at this mod, if its causing problems you could always take a long hard look at it yourself or take the mod out.

I know what I'd rather Unknown spent his time doing...