search engine spidering

David
Destroyer Dave
Global Moderator
YaBB God

Posts: 5761

I'm not a llama!

Re:search engine spidering

« Reply #45 on: April 24, 2003, 12:29:43 AM »

Quote from: [Unknown] on April 23, 2003, 09:59:46 PM
http://www.yabbse.org/community/index.php?board=158;action=display;threadid=21933

-[Unknown]

Still has the ? in the url though.


	Logged

Forum-Forum.com, A Forum For Forum Owners and Enthusiasts!

sensovision
Full Member

Posts: 100

Re:search engine spidering

« Reply #46 on: April 24, 2003, 12:45:16 AM »

Quote from: David on April 24, 2003, 12:29:43 AM
Quote from: [Unknown] on April 23, 2003, 09:59:46 PM
http://www.yabbse.org/community/index.php?board=158;action=display;threadid=21933

-[Unknown]
Still has the ? in the url though.

hi I've check this mod and it's still not search engine friendly links

BTW I've read in other thread that Google have not much trobules with indexing site and even dynamically site could be crawled... I check my site and only occasional threads are crawled in Google... (I've getting freshbot 1-2 times a week) see what Google saying about this:
"Your pages are dynamically generated. We are able to index dynamically generated pages. However, because our web crawler can easily overwhelm and crash sites serving dynamic content, we limit the amount of dynamic pages we index. "
http://www.google.com/webmasters/2.html
so maybe it would be possible and more easier to make mod_rewrite only for certain bots...? or have static links for guests only and after log-in they would convert to dynamic ones? or it's silly and technically not possible?


	Logged

Denis

Are you good with the graphic? check out our design logo contest!

groundup
Disciple of Joe, Head Cleric
Mod Team
YaBB God

Posts: 2983

Error 309: Please notify the administrator of this

Re:search engine spidering

« Reply #47 on: April 24, 2003, 12:48:56 AM »

It wouldn't really work if you checked for googlebot, that would make all links bad. The best way to do it is allow for both types of links. mod_rewrite is the best way to acheive this, though php can do it too.


	Logged

http://www.fastfinancialfreedom.org - financial freedom is a myth
http://www.fastfinancialfreedom.org - financial freedom is a myth
http://www.fastfinancialfreedom.org - financial freedom is a myth
fastfinancialfreedom.org

groundup
Disciple of Joe, Head Cleric
Mod Team
YaBB God

Posts: 2983

Error 309: Please notify the administrator of this

Re:search engine spidering

« Reply #48 on: April 24, 2003, 12:54:15 AM »

replace ? with / and ; with /


	Logged

[Unknown]
Global Moderator
YaBB God

Posts: 7830

Re:search engine spidering

« Reply #49 on: April 24, 2003, 12:59:38 AM »

You misunerstand.... I was refering to cyc's post here:
http://www.yabbse.org/community/index.php?board=141;action=display;threadid=14858;start=30#msg147917

He said that he used vBulletin, and it was indexed while YaBB SE was not.

The only difference is that YaBB SE uses full URLs everywhere... while vB, phpBB, etc. do not... as seen in this phpBB link:
<a href="index.php?sid=a06ae669b58b34c78273b1433722747a"><img src="/images/title_comm.gif" width="195" height="39" border="0"></a>

Not using the full URL only helps - it does not solve the ?/;/& issue.

-[Unknown]


	Logged

[Unknown]
Global Moderator
YaBB God

Posts: 7830

Re:search engine spidering

« Reply #50 on: April 24, 2003, 01:00:26 AM »

Quote from: groundup on April 24, 2003, 12:54:15 AM
replace ? with / and ; with /

Do note that this cannot be done on IIS. (not to say that IIS is the best server, but it's not going to solve the issue for everyone.)

-[Unknown]


	Logged

groundup
Disciple of Joe, Head Cleric
Mod Team
YaBB God

Posts: 2983

Error 309: Please notify the administrator of this

Re:search engine spidering

« Reply #51 on: April 24, 2003, 02:38:20 AM »

* groundup makes note of that.

Good to know.


	Logged

islanding
Noobie

Posts: 2

I am a llama!

Re:search engine spidering

« Reply #52 on: May 26, 2003, 10:21:39 PM »

Hi, I have not gone through all of the posts but thought this may help you.

You need to rewrite the way your script outputs the HTML. This is how you remove the query strings from the URL not Mod_Rewrite.
Getting rid of the query string is as easy as putting the following few lines at the start of your PHP scripts.

Code Select

ob_start('post_process');
# function post_process($buffer) {
return preg_replace("'script.php\?([^\"\']+)'e", "'script-'.implode('-', preg_split('/&|=/', '\\1')).'.html'",
$test);
}

Example

When your PHP script would have output the following HTML

Code Select

<html>
<p> This is a standard compliant HTML document containing two links:
<a href="script.php?name=aaron&age=15">Aaron</a>
<a href="script.php?name=aschenbach&age=dead">Gustav</a>
</p>
</html>

it will now output

Code Select

<html>
<p> This is a standard compliant HTML document containing two links:
<a href="script-name-aaron-age-15.html">Aaron</a>
<a href="script-name-aschenbach-age-dead.html">Gustav</a>
</p>
</html>

Then you can use Mod-Rewrite get Rewrite the query string. Since there are no such files on your server you will need a way to undo these changes when the UA requests them.

Code Select

RewriteRule (script.*)-([^-]+)-([^-]+)\.html$ $1?$2=$3 [N,QSA]
RewriteRule script\.html script.php

Those two rules will turn the fake script name back into the real thing with query strings, just like your scripts will expect them.

Code Select


	Logged

[Unknown]
Global Moderator
YaBB God

Posts: 7830

Re:search engine spidering

« Reply #53 on: May 26, 2003, 11:37:16 PM »

Again, this cannot be done on IIS... but it would work.

You can also replace them with slashes, which is popular... like so:

function url_process($buffer)
{
global $scripturl;

return preg_replace('~' . preg_quote($scripturl, '~') . '\?([^"]+)~ie', "$scripturl . '/' . implode('/', preg_split('~[&=;]~', '\\1')) . '.html'",
$buffer);
}

That should work fine. (this is written for YaBB SE...)

RewriteRule (index.php)/([^/]+)/([^/]+)\.html$ $1?$2=$3 [N,QSA]
RewriteRule index\.html index.php

That's just guess work... but I think it'd work. No idea what N,QSA means or anything.... (what? I don't use apache!)

-[Unknown]


« Last Edit: May 26, 2003, 11:39:33 PM by [Unknown] »	Logged

islanding
Noobie

Posts: 2

I am a llama!

Re:search engine spidering

« Reply #54 on: May 27, 2003, 10:43:58 AM »

you are right Mod_Rewrite is a Apache function not usable with IIS.


	Logged

Pages: 1 2 3 [4]

YaBB SE Community \| Development \| Mod Ideas and Creation \| search engine spidering	« previous - next »

	Welcome, Guest. Please Login or Register. November 19, 2024, 05:37:20 PM

News: SMF is the next generation in forum software, almost completely re-written from the ground up, make sure you don't fall for cheap imitations that suffer from feature bloat!