Jens Gutzeit | 23 Aug 10:23 2002

Re: Indexer doesn't follow links with an "&" in it

Am Fri, 23 Aug 2002 03:07:25 +0200
Daniel Naber <daniel.naber <at> t-online.de> schrieb:

Hi

Thanks for your help, but it didn't the trick.

> Did you check the things mentioned here?:
> 
> http://perlfect.com/freescripts/search/faq.shtml#T10

Yes, I've checked that. $HTTP_DEBUG is set to 1, I've disabbled the
append_sid function in my php scripts (which put the &amp;sid=... at the
end of all urls), but this wasn't the problem, so the problem is not the
&amp;. The indexers says, it index http://lfsforum.org/,
http://lfsforum.org/howtos/ (and some other files as well), both links
results in the index.php file of this directory. On both pages are links
to howtos (that should be indexed) the link is now like this:

<li><span class="gen"><a href="/howtos/read.php?howto=9">Optimierung
...</a></span></li><li><span class="gen"><a
href="/howtos/read.php?howto=8">PAM, Cracklib u...</a></span></li>

But it didn't index these files
I hope you can help me.

> regards
>  Daniel

best regards
(Continue reading)

Daniel Naber | 23 Aug 13:51 2002
Picon

Re: Indexer doesn't follow links with an "&" in it

On Friday 23 August 2002 10:23, Jens Gutzeit wrote:

> I've disabbled the
> append_sid function

You probably shouldn't do that. It tried to index a part of your server from here with 
Perlfect Search 3.30 and it works. For the howto=9 link it says:

Fetched  'http://lfsforum.org/howtos/read.php?howto=9&sid=30c3d336d3e48d4cca0fa95170ad53b0',
29315 bytes
	 5: http://lfsforum.org/howtos/read.php?howto=9&sid=30c3d336d3e48d4cca0fa95170ad53b0

I used these values in conf.pl:

$HTTP_START_URL = 'http://lfsforum.org/';
 <at> HTTP_LIMIT_URLS = ("http://lfsforum.org/howtos");

Regards
 Daniel

--

-- 
http://www.danielnaber.de
_______________________________________________
perlfect-search mailing list
perlfect-search <at> perlfect.com
To unsubscribe, set other personal options or view the list archives please visit:
http://perlfect.com/mailman/listinfo/perlfect-search


(Continue reading)

Jens Gutzeit | 23 Aug 15:12 2002

Re: Indexer doesn't follow links with an "&" in it

Am Fri, 23 Aug 2002 13:51:04 +0200
Daniel Naber <daniel.naber <at> t-online.de> schrieb:

Hi

> You probably shouldn't do that. 

Yes, I know, it was for testing only.

> It tried to index a part of your server from here with 
> Perlfect Search 3.30 and it works. For the howto=9 link it says:
> 
> Fetched 
> 'http://lfsforum.org/howtos/read.php?howto=9&sid=30c3d336d3e48d4cca0f
> a95170ad53b0', 29315 bytes
> 	 5:
> 	 http://lfsforum.org/howtos/read.php?howto=9&sid=30c3d336d3e48d4cca0fa95170ad53b0
> 
> I used these values in conf.pl:
> 
> $HTTP_START_URL = 'http://lfsforum.org/';
>  <at> HTTP_LIMIT_URLS = ("http://lfsforum.org/howtos");

Thanks,  <at> HTTP_LIMITS_URLS was the problem, it was set to
http://lfsforum.org
I've set it to http://lfsforum.org/howtos/ then it index my howtos, but
only the howtos, and nothing else. Then I have tried this:

 <at> HTTP_LIMIT_URLS = ('http://lfsforum.org/howtos/',
'http://lfsforum.org/');
(Continue reading)

Daniel Naber | 23 Aug 17:06 2002
Picon

Re: Indexer doesn't follow links with an "&" in it

On Friday 23 August 2002 15:12, Jens Gutzeit wrote:

> But then it index everything else but not the howtos. Do I do something
> wrong?

It tries to index an unlimited number of URL like these: 
http://lfsforum.org/ghostbb/login.php?goto=...

So you have to limit (conf/no_index.txt) the URLs so that these pages are 
ignored. Then (if $HTTP_MAX_PAGES is big enough) the howto pages will be 
found.

Regards
 Daniel

--

-- 
http://www.danielnaber.de
_______________________________________________
perlfect-search mailing list
perlfect-search <at> perlfect.com
To unsubscribe, set other personal options or view the list archives please visit:
http://perlfect.com/mailman/listinfo/perlfect-search


Jens Gutzeit | 23 Aug 17:25 2002

Re: Indexer doesn't follow links with an "&" in it

Am Fri, 23 Aug 2002 17:06:34 +0200
Daniel Naber <daniel.naber <at> t-online.de> schrieb:

> It tries to index an unlimited number of URL like these: 
> http://lfsforum.org/ghostbb/login.php?goto=...
> 
> So you have to limit (conf/no_index.txt) the URLs so that these pages
> are ignored. Then (if $HTTP_MAX_PAGES is big enough) the howto pages
> will be found.

Hi

That's not the problem, the no_index.txt is configured ok, it indexes
only that files that it should index, but it didn't index the howtos.
Only if I set  <at> HTTP_LIMIT_URLS it indexes my howtos, but then it didn't
index anything else.  <at> HTTP_FILES_URLS is an arry, so I have tried
 <at> HTTP_FILES_URLS = ('http://lfsforum.org/',
'http://lfsforum.org/howtos/'), I think then it should index the howtos
and the other pages (that should be indexed), but its always indexing
the other pages only and not the howtos. I hope you understand what I
say, my english is not so good. :-(

> Regards
>  Daniel

best regards
Jens

--

-- 
Hilfe zu LFS Problemen: http://www.lfsforum.de
(Continue reading)

Jens Gutzeit | 23 Aug 17:31 2002
Daniel Naber | 23 Aug 17:56 2002
Picon

Re: Indexer doesn't follow links with an "&" in it

On Friday 23 August 2002 17:31, Jens Gutzeit wrote:

> $HTTP_MAX_PAGES = 100;

That's too low (no matter if only 30 files are indexed), at least I get 
errors with that setting after some files have been indexed.

Regards
 Daniel

--

-- 
http://www.danielnaber.de
_______________________________________________
perlfect-search mailing list
perlfect-search <at> perlfect.com
To unsubscribe, set other personal options or view the list archives please visit:
http://perlfect.com/mailman/listinfo/perlfect-search


Jens Gutzeit | 23 Aug 18:23 2002

Re: Indexer doesn't follow links with an "&" in it

Am Fri, 23 Aug 2002 17:56:11 +0200
Daniel Naber <daniel.naber <at> t-online.de> schrieb:

> That's too low (no matter if only 30 files are indexed), at least I
> get errors with that setting after some files have been indexed.

Ah, thanks, now it works. I haven't known that my site has so many pages
*g*

> Regards
>  Daniel

best regards
Jens

--

-- 
Hilfe zu LFS Problemen: http://www.lfsforum.de
Public Key: http://lfsforum.org/jens-gutzeit.asc

Gmane