Re: interwiki.py
Merlijn van Deen <valhallasw <at> arctus.nl>
2012-01-15 00:31:10 GMT
On 15 January 2012 00:32, Nickanc Wikipedia
<nickanc.wiki <at> gmail.com> wrote:
In my opinion, the problem is not how many bots are working, but that
interwiki.py seems to use to overuse memory space. For example, my
interwiki.py was running, it was spending 982 megabyte and it was
killed. You may find a verbose log of its work at
http://toolserver.org/~nickanc/interwiki.log .
No, that is not the problem. Multichill was referring to CPU usage, not memory usage. And although interwiki.py in general is using a large amount of memory, your specific case has a different origin (being the use of the ReferringPageGenerator, which results in a memory leak).
You may be able to partially mitigate your problem by using
interwiki_contents_on_disk = True,
but this will not solve the actual memory leak - it will only release memory used by the page contents.
Merlijn
<div><div class="gmail_quote">On 15 January 2012 00:32, Nickanc Wikipedia <span dir="ltr"><<a href="mailto:nickanc.wiki <at> gmail.com">nickanc.wiki <at> gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote">
In my opinion, the problem is not how many bots are working, but that<br>
interwiki.py seems to use to overuse memory space. For example, my<br>
interwiki.py was running, it was spending 982 megabyte and it was<br>
killed. You may find a verbose log of its work at<br><a href="http://toolserver.org/~nickanc/interwiki.log" target="_blank">http://toolserver.org/~nickanc/interwiki.log</a> .<br><br>
</blockquote>
<div>No, that is not the problem. Multichill was referring to CPU usage, not memory usage. And although interwiki.py in general is using a large amount of memory, your specific case has a different origin (being the use of the ReferringPageGenerator, which results in a memory leak).</div>
<div><br></div>
<div>You may be able to partially mitigate your problem by using</div>
<div>interwiki_contents_on_disk = True,</div>
<div>but this will not solve the actual memory leak - it will only release memory used by the page contents.</div>
<div><br></div>
<div>If you'd like to discuss details on your problem, please mail to the pywikipedia mailing list <<a href="mailto:pywikipedia-l <at> lists.wikimedia.org">pywikipedia-l <at> lists.wikimedia.org</a>> or visit on IRC.</div>
<div><br></div>
<div>Merlijn</div>
</div></div>
RSS Feed