headers
dieter | 28 Aug 07:26

Process "Killed"

Hi,

Overview
=======

I'm doing some simple file manipulation work and the process gets
"Killed" everytime I run it. No traceback, no segfault... just the
word "Killed" in the bash shell and the process ends. The first few
batch runs would only succeed with one or two files being processed
(out of 60) before the process was "Killed". Now it makes no
successful progress at all. Just a little processing then "Killed".

Question
=======

Any Ideas? Is there a buffer limitation? Do you think it could be the
filesystem?
Any suggestions appreciated.... Thanks.

The code I'm running:
==================

from glob import glob

def manipFiles():
    filePathList = glob('/data/ascii/*.dat')
    for filePath in filePathList:
        f = open(filePath, 'r')
        lines = f.readlines()[2:]
        f.close()
(Continue reading)

Permalink | Reply |
headers
Matt Nordhoff | 28 Aug 16:10

Re: Process "Killed"

dieter wrote:
> Hi,
> 
> Overview
> =======
> 
> I'm doing some simple file manipulation work and the process gets
> "Killed" everytime I run it. No traceback, no segfault... just the
> word "Killed" in the bash shell and the process ends. The first few
> batch runs would only succeed with one or two files being processed
> (out of 60) before the process was "Killed". Now it makes no
> successful progress at all. Just a little processing then "Killed".

That isn't a Python thing. Run "sleep 60" in one shell, then "kill -9"
the process in another shell, and you'll get the same message.

I know my shared web host has a daemon that does that to processes that
consume too many resources.

Wait a minute. If you ran this multiple times, won't it have removed the
first two lines from the first files multiple times, deleting some data
you actually care about? I hope you have backups...

> Question
> =======
> 
> Any Ideas? Is there a buffer limitation? Do you think it could be the
> filesystem?
> Any suggestions appreciated.... Thanks.
>
(Continue reading)

Permalink | Reply |
headers
Glenn Hutchings | 29 Aug 21:12

Re: Process "Killed"

dieter <vel.accel <at> gmail.com> writes:

> I'm doing some simple file manipulation work and the process gets
> "Killed" everytime I run it. No traceback, no segfault... just the
> word "Killed" in the bash shell and the process ends. The first few
> batch runs would only succeed with one or two files being processed
> (out of 60) before the process was "Killed". Now it makes no
> successful progress at all. Just a little processing then "Killed".
>
> Any Ideas? Is there a buffer limitation? Do you think it could be the
> filesystem?
> Any suggestions appreciated.... Thanks.
>
> The code I'm running:
> ==================
>
> from glob import glob
>
> def manipFiles():
>     filePathList = glob('/data/ascii/*.dat')
>     for filePath in filePathList:
>         f = open(filePath, 'r')
>         lines = f.readlines()[2:]
>         f.close()
>         f = open(filePath, 'w')
>         f.writelines(lines)
>         f.close()
>         print file

Have you checked memory usage while your program is running?  Your
(Continue reading)

Permalink | Reply |
headers
Fredrik Lundh | 29 Aug 21:37

Re: Process "Killed"

Glenn Hutchings wrote:

> Have you checked memory usage while your program is running?  Your
> 
>     lines = f.readlines()[2:]
> 
> statement will need almost twice the memory of your largest file.

footnote: list objects contain references to string objects, not the 
strings themselves.  the above temporarily creates two list objects, but 
the actual file content is only stored once.

</F>

--
http://mail.python.org/mailman/listinfo/python-list
Permalink | Reply |
headers
Paul Boddie | 29 Aug 21:25

Re: Process "Killed"

On 28 Aug, 07:30, dieter <vel.ac...@gmail.com> wrote:
>
> I'm doing some simple file manipulation work and the process gets
> "Killed" everytime I run it. No traceback, no segfault... just the
> word "Killed" in the bash shell and the process ends. The first few
> batch runs would only succeed with one or two files being processed
> (out of 60) before the process was "Killed". Now it makes no
> successful progress at all. Just a little processing then "Killed".

It might be interesting to check the various limits in your shell. Try
this command:

  ulimit -a

Documentation can found in the bash manual page. The limits include
memory size, CPU time, open file descriptors, and a few other things.

Paul
--
http://mail.python.org/mailman/listinfo/python-list
Permalink | Reply |
headers
Fredrik Lundh | 29 Aug 21:34

Re: Process "Killed"

dieter wrote:

> Any Ideas? Is there a buffer limitation? Do you think it could be the
> filesystem?

what does "ulimit -a" say?

</F>

--
http://mail.python.org/mailman/listinfo/python-list
Permalink | Reply |
headers
Eric Wertman | 30 Aug 17:07

Re: Process "Killed"

> I'm doing some simple file manipulation work and the process gets
> "Killed" everytime I run it. No traceback, no segfault... just the
> word "Killed" in the bash shell and the process ends. The first few
> batch runs would only succeed with one or two files being processed
> (out of 60) before the process was "Killed". Now it makes no
> successful progress at all. Just a little processing then "Killed".

This is the behavior you'll see when your os has run out of some
memory resource.  The kernel sends a 9 signal.  I'm pretty sure that
if you exceed a soft limit your program will abort with out of memory
error.

Eric
--
http://mail.python.org/mailman/listinfo/python-list
Permalink | Reply |
headers
dieter h | 2 Sep 06:56

Re: Process "Killed"

On Sat, Aug 30, 2008 at 11:07 AM, Eric Wertman <ewertman <at> gmail.com> wrote:
>> I'm doing some simple file manipulation work and the process gets
>> "Killed" everytime I run it. No traceback, no segfault... just the
>> word "Killed" in the bash shell and the process ends. The first few
>> batch runs would only succeed with one or two files being processed
>> (out of 60) before the process was "Killed". Now it makes no
>> successful progress at all. Just a little processing then "Killed".
>
> This is the behavior you'll see when your os has run out of some
> memory resource.  The kernel sends a 9 signal.  I'm pretty sure that
> if you exceed a soft limit your program will abort with out of memory
> error.
>
> Eric
>

Eric, thank you very much for your response.
--
http://mail.python.org/mailman/listinfo/python-list
Permalink | Reply |

Gmane