gmane.comp.python.general

Descriptor leak in python 2.4 subprocess module

Hi,

I hit an issue with the following python code:

    try:
        get_orient = subprocess.Popen (['jpegexiforient', '-n', pathfull],
                                       stdin = subprocess.PIPE,
                                       stdout = subprocess.PIPE)
        orient = get_orient.communicate ()[0]
    except:
        orient = None

The intent of this was to read the exif orientation of a picture, or just
use None if jpegexiforient can not run.

The application worked fine on my devel machine but I noticed that on a
different host, it crashed due to running out of file descriptors.
After investigation I found out that the above code was responsible,
leaking two file descriptors per invocation if jpegexiforient is not
installed on the host.

I don't see any way to fix it in my code either, since get_orient is not
defined in the exception path, there is no way I can close its file
descriptors myself. I believe this is a bug in the subprocess module,
it should make sure to clean up after itself when getting out on the
exception path.

This is with python 2.4.4 on linux (debian etch distribution).

Hope this helps. I would prefer to be copied in any replies as I'm not

(Continue reading)

headers

Tim Golden | 28 Aug 11:37

From: Tim Golden <mail <at> timgolden.me.uk>

Subject: Re: Descriptor leak in python 2.4 subprocess module

Re: Descriptor leak in python 2.4 subprocess module

Michel Lespinasse wrote:
> Hi,
> 
> I hit an issue with the following python code:
> 
>     try:
>         get_orient = subprocess.Popen (['jpegexiforient', '-n', pathfull],
>                                        stdin = subprocess.PIPE,
>                                        stdout = subprocess.PIPE)
>         orient = get_orient.communicate ()[0]
>     except:
>         orient = None
> 
> The intent of this was to read the exif orientation of a picture, or just
> use None if jpegexiforient can not run.
> 
> The application worked fine on my devel machine but I noticed that on a
> different host, it crashed due to running out of file descriptors.
> After investigation I found out that the above code was responsible,
> leaking two file descriptors per invocation if jpegexiforient is not
> installed on the host.
> 
> I don't see any way to fix it in my code either, since get_orient is not
> defined in the exception path, there is no way I can close its file
> descriptors myself. I believe this is a bug in the subprocess module,
> it should make sure to clean up after itself when getting out on the
> exception path.

This looks like a duplicate of http://bugs.python.org/issue3210.
Can you confirm if this seems likely (and, if so, perhaps add

(Continue reading)

headers

Michel Lespinasse | 28 Aug 22:31

Re: Descriptor leak in python 2.4 subprocess module

On Thu, Aug 28, 2008 at 10:37:48AM +0100, Tim Golden wrote:
> Michel Lespinasse wrote:
> >I hit an issue with the following python code:
> >    try:
> >        get_orient = subprocess.Popen (['jpegexiforient', '-n', pathfull],
> >                                       stdin = subprocess.PIPE,
> >                                       stdout = subprocess.PIPE)
> >        orient = get_orient.communicate ()[0]
> >    except:
> >        orient = None
> >
> >The application worked fine on my devel machine but I noticed that on a
> >different host, it crashed due to running out of file descriptors.
> >After investigation I found out that the above code was responsible,
> >leaking two file descriptors per invocation if jpegexiforient is not
> >installed on the host.
> 
> This looks like a duplicate of http://bugs.python.org/issue3210.
> Can you confirm if this seems likely (and, if so, perhaps add
> a note to the bug indicating that the problem is on Linux as well)?

The root cause is different (subprocess has separate implementations
for windows and posix), but the issues are similar. In the posix case,
I only observed the issue when you using subprocess.PIPE

The attached patch against subprocess.py (as shipped on debian etch,
based on python version 2.4.4) seems to work for me:

--- /usr/lib/python2.4/subprocess.py	2008-04-16 10:59:00.000000000 -0700
+++ subprocess.py	2008-08-28 13:09:50.000000000 -0700

(Continue reading)