[beta 1.5.4] Spider URLs 1.2

[Anguz]

This mod removes the symbols that search-engine spiders don't like in urls so that your forum can be indexed. You should also use Relative URLs mod to fix relative urls so that they work correctly.

Spider_URLs_v1.2.zip

Here's a before/after example:
http://example.com/yabbse/index.php?board=1
http://example.com/yabbse/index.php/board-1

You can see it working in my forum:
http://laakademia.com/foros

It's been tested quite a lot already, but please let me know if you run into any problems.

History:
03 SEP 16 v1.2 released
03 SEP 15 v1.1 released
03 SEP 14 v1 released

[dschwab9]

It doesn't work for me (even using index.php rather than "content")

The OB Anguz URL mod is what's causing my problem.  It gives "page not found" and throws apache into a loop that uses 100% cpu and take the server down until the script times out.  

[Shadow's Pawn]

The OB Anguz script seems to be causing my Apache to crash as well..

[Anguz]

that's weird, cause it's basically the same code I used in Friendly URLs, the only difference being the last line was replaced with the callback to the functions

it's not causing me that problem, that's why I don't understand... I'll make sure the copy in the zip is ok

do you know what that loop could be?

[dschwab9]

do you know what that loop could be?

Is there any way I can tell?  It crashes apache before any output is generated, so I just get "page cannot be displayed.

[Anguz]

here's the code I have in my forum, I left the relative urls function out cause I don't know if you want to use it

Code Select Expand

ob_start('ob_anguz_url');
function ob_anguz_url($buffer) {
   global $scripturl;
   function spider_url($match){
      if($match['2'] == '' || strstr($match['2'], 'search') || strstr($match['2'], 'gallery') || strstr($match['2'], 'profile') || strstr($match['2'], 'imsend') || strstr($match['2'], 'chat') || strstr($match['2'], '_') || strstr($match['2'], '-'))
         return $match['0'];
      else
         return $match['1'] . str_replace(array("=", ";", "&", "?"), array("-", "_", "_", "/"), $match['2']);
   }
   $buffer = (function_exists('smart_url') ? preg_replace_callback('//i', 'smart_url', $buffer) : $buffer);
   $buffer = (function_exists('spider_url') ? preg_replace_callback('/(' . preg_quote($scripturl, '/') . ')([\?\w;=~&+%]+)/i', 'spider_url', $buffer) : $buffer);
   $buffer = (function_exists('relative_url') ? preg_replace_callback('/(href|src)="([^"]+)"/i', "relative_url", $buffer) : $buffer);
   return $buffer;
}

[dschwab9]

Is that different from the mod you posted here?  I'll try it and see what happens.  I'd like to use the relative urls also if I can get them to work.

[Craig]

nice work but i have to refresh the login and home button everytime it is pressed after i installed it so have took it out.

but just wondered this, My forum has a google page rank 5 and all post have a page rank 3 so dosen't it already spider this forum or is this mod for other search engines?

also your pagerank on your board is high but you dont have any on your posts. Is this because your board hasnt been spidered since you installed this mod?

I'm by no means an expert on this but will be looking at you PR next week to see if you have one and if this mod makes it higher.

keep up the good work.

[Anguz]

dschwab9 - mod updated, update the ob_anguz_url mod to 1.1 too

Craig - thank you try the new versions of the mods, you shouldn't have that problem with your session, if you still have it, let me know

about indexing, it's supposed to index much more if the url doesn't have certain symbols present... I've only made my change recently and I don't know if google's been there after that or not... I'm not that learned about pagerank actually, how do you measure that?

[Craig]

Thanks for the reply  

I'm not that learned about pagerank actually, how do you measure that?

use the free google toolbar

http://toolbar.google.com/

its good to see what pages have been spidered and  how important google classes each page of your site

[Anguz]

looks like google's been to my forum yesterday or today, cause a couple of days ago this search returned around 30 results and there's 2,680 right now

I'll try to improve it more so that it actually returns all the thousands it should, but as it is, the spider certainly likes it better  

[Craig]

nice work

im still having the refresh problems  

ive added both got the refresh problem then i added just the spider mod and got the refresh problem  

so ive took it out again. maybe its my server  

[dschwab9]

Haven't tried the new code yet, gonna work with that some tonight.

Just looked at my logs, though.  Inktomi has been crawling my forums and has downloaded over 600 pages the last day or so.  Google came back and got some more too.

I'm not seeing in more results in the google search, but I know they don't update their database immediatly, so it may just not be there yet.

[Anguz]

im still having the refresh problems

in mod settings, in the admin area, is 'Enable local storage of cookies' checked or unchecked? try unchecking it, that's how I have it and it works fine for me, that may do the trick for you too (I'm no cookies genius, so I can't give much more advice than this )

[Anguz]

just updated the mod to v1.2

fixed a bug with removethread links, it didn't delete the thread, it does now