NodeJazz | 6 Feb 16:25 2012
Picon

node clustered http server workers die with misleading error

Hi,

I am running node 0.6.8 on Windows Server 2008 R2. I have a basic 6
worker clustered http server script serving a static file using
stream.pipe. I have 13000 http connections attempting to retrieve the
static file 60 times over a period of 2 minutes. As the number of
connections increase some of the workers die due to an unknown error.
The messages from node as follows:

C:\files>"C:\Program Files (x86)\nodejs\node.exe" "C:\Program Files
(x86)\nodejs\hs.js"

stream.js:105
      throw er; // Unhandled stream error in pipe.
            ^
Error: OK, success 'C:\files\test.dat'
worker 17104 died

stream.js:105
      throw er; // Unhandled stream error in pipe.
            ^
Error: OK, success 'C:\files\test.dat'
worker 18884 died

stream.js:105
      throw er; // Unhandled stream error in pipe.
            ^
Error: OK, success 'C:\files\test.dat'
worker 14100 died

(Continue reading)

Ben Noordhuis | 6 Feb 16:57 2012
Picon

Re: node clustered http server workers die with misleading error

On Mon, Feb 6, 2012 at 16:25, NodeJazz <infinicosm@...> wrote:
> Hi,
>
> I am running node 0.6.8 on Windows Server 2008 R2. I have a basic 6
> worker clustered http server script serving a static file using
> stream.pipe. I have 13000 http connections attempting to retrieve the
> static file 60 times over a period of 2 minutes. As the number of
> connections increase some of the workers die due to an unknown error.
> The messages from node as follows:
>
>
> C:\files>"C:\Program Files (x86)\nodejs\node.exe" "C:\Program Files
> (x86)\nodejs\hs.js"
>
> stream.js:105
>      throw er; // Unhandled stream error in pipe.
>            ^
> Error: OK, success 'C:\files\test.dat'
> worker 17104 died
>
> stream.js:105
>      throw er; // Unhandled stream error in pipe.
>            ^
> Error: OK, success 'C:\files\test.dat'
> worker 18884 died
>
> stream.js:105
>      throw er; // Unhandled stream error in pipe.
>            ^
> Error: OK, success 'C:\files\test.dat'
(Continue reading)

NodeJazz | 6 Feb 19:09 2012
Picon

Re: node clustered http server workers die with misleading error

>
> You need to attach an error event listener to the stream. Connections
> may terminate unexpectedly for a number of reasons. If you don't
> handle the error, you'll get the behaviour you're seeing now.

I was not able to find a way to specify a error handler when using
fs.createReadStream(path) for the input (static file). Is this
possible?

So I tried a 2 step approach using
fs.open(path, flags, mode, callback)
and then calling
fs.createReadStream(null, options)
in the open callback with options.fd set approriately

With this approach (err, fd) received by the callback is good at the
beginning. But subsequently fd becomes 'undefined' and err has
'Error: OK, success 'C:\files\test.dat' with err.errno being 0. Why is
fd undefined when there is no apparent error? Is there a way to view
underlying errors if any?

Thanks

Ben Noordhuis | 6 Feb 20:08 2012
Picon

Re: Re: node clustered http server workers die with misleading error

On Mon, Feb 6, 2012 at 19:09, NodeJazz <infinicosm@...> wrote:
> I was not able to find a way to specify a error handler when using
> fs.createReadStream(path) for the input (static file). Is this
> possible?

Yes.

  var stream = fs.createReadStream(path, options);
  stream.on('error', function() { /* ... */ });

> So I tried a 2 step approach using
> fs.open(path, flags, mode, callback)
> and then calling
> fs.createReadStream(null, options)
> in the open callback with options.fd set approriately
>
> With this approach (err, fd) received by the callback is good at the
> beginning. But subsequently fd becomes 'undefined' and err has
> 'Error: OK, success 'C:\files\test.dat' with err.errno being 0. Why is
> fd undefined when there is no apparent error? Is there a way to view
> underlying errors if any?

What do you mean with 'fd becomes undefined'? Where/when does that
happen? A more complete code example would help.

Cosmere Infahm | 6 Feb 21:13 2012
Picon

Re: Re: node clustered http server workers die with misleading error

>
>   var stream = fs.createReadStream(path, options);
>   stream.on('error', function() { /* ... */ });
>

Wouldn't this mean that the error handler is unlikely to be invoked in case of an error during creation/opening of the stream?

>
> What do you mean with 'fd becomes undefined'? Where/when does that
> happen? A more complete code example would help.

I meant that the values of (err, fd) received by the open callback change over time. fd is >= 0 and err is null in response to the initial http requests. After a few seconds the callback parameter values are ('Error: OK, success 'C:,\files\test.dat', undefined).

var fileName = 'C:\\files\\test.dat';

var errHandler = function (exception) {
    console.log(exception);
};

var reqListener = function (req, res) {
    req.error = errHandler;
    res.error = errHandler;
    var openHandler = function (err, fd) {
        console.log(fd+ ' ' +err);
        if (fd >= 0) {
            options = {};
            options.fd = fd;
            rdstream = fs.createReadStream(null, options);
            rdstream.error = errHandler;
   
            res.writeHead(200, {
                'Content-Type': 'application/octet-stream'
            });
            rdstream.pipe(res);
        }
    };
    openHandler.res = res;
    fs.open(fileName, 'r', 0666, openHandler);
};

//var hs = http.createServer(reqListener);


The output looks like:

C:\files>"C:\Program Files (x86)\nodejs\node.exe" "C:\Program Files (x86)\nodejs\hs.js"
3 null
3 null
4 null
4 null
5 null
5 null
6 null
6 null
7 null
7 null
8 null
8 null
9 null
9 null
10 null
10 null
...
...
...
2007 null
1831 null
1872 null
2045 null
1907 null
2008 null
2012 null
1873 null
1832 null
2046 null
1908 null
2009 null
2013 null
1833 null
1874 null
2047 null
1909 null
2010 null
2014 null
1875 null
1834 null
2011 null
1835 null
2015 null
1910 null
2012 null
undefined Error: OK, success 'C:\files\test.dat'
1876 null
2016 null
1911 null
undefined Error: OK, success 'C:\files\test.dat'
1877 null
1912 null
2017 null
1878 null
undefined Error: OK, success 'C:\files\test.dat'
1913 null
1879 null
undefined Error: OK, success 'C:\files\test.dat'
1914 null
1880 null
.... errors increase in frequency

--

--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nodejs-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
To unsubscribe from this group, send email to
nodejs+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en
Ben Noordhuis | 6 Feb 21:28 2012
Picon

Re: Re: node clustered http server workers die with misleading error

On Mon, Feb 6, 2012 at 21:13, Cosmere Infahm <infinicosm@...> wrote:
>>   var stream = fs.createReadStream(path, options);
>>   stream.on('error', function() { /* ... */ });
>
> Wouldn't this mean that the error handler is unlikely to be invoked in case
> of an error during creation/opening of the stream?

No, because the actual work - open file, read file, etc. - doesn't
start until the next iteration of the event loop (colloquially: the
next tick).

>> What do you mean with 'fd becomes undefined'? Where/when does that
>> happen? A more complete code example would help.
>
> I meant that the values of (err, fd) received by the open callback change
> over time. fd is >= 0 and err is null in response to the initial http
> requests. After a few seconds the callback parameter values are ('Error: OK,
> success 'C:,\files\test.dat', undefined).

Right, I see what you mean. The system-to-node error mappings are not
perfect yet, particularly on Windows with its myriad error codes.

--

-- 
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nodejs@...
To unsubscribe from this group, send email to
nodejs+unsubscribe@...
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en

NodeJazz | 6 Feb 21:50 2012
Picon

Re: node clustered http server workers die with misleading error

> No, because the actual work - open file, read file, etc. - doesn't
> start until the next iteration of the event loop (colloquially: the
> next tick).
>
Is there a rule of thumb or API documentation that gives details about
the sync & async processing associated with a call?

> Right, I see what you mean. The system-to-node error mappings are not
> perfect yet, particularly on Windows with its myriad error codes.

Is this just an insufficiency? I get the feeling it is some kind of
bug since the problem is seen only with a high number of connections
and it gets worse as the eventing increases.

Would there be a way to work around the issue?

.--

Ben Noordhuis | 6 Feb 22:46 2012
Picon

Re: Re: node clustered http server workers die with misleading error

On Mon, Feb 6, 2012 at 21:50, NodeJazz <infinicosm@...> wrote:
>> No, because the actual work - open file, read file, etc. - doesn't
>> start until the next iteration of the event loop (colloquially: the
>> next tick).
>>
> Is there a rule of thumb or API documentation that gives details about
> the sync & async processing associated with a call?

The rule of thumb is that all I/O related operations are asynchronous
unless explicitly marked otherwise, e.g. fs.open() vs. fs.openSync().

>> Right, I see what you mean. The system-to-node error mappings are not
>> perfect yet, particularly on Windows with its myriad error codes.
>
> Is this just an insufficiency? I get the feeling it is some kind of
> bug since the problem is seen only with a high number of connections
> and it gets worse as the eventing increases.
>
> Would there be a way to work around the issue?

Probably an insufficiency unless demonstrated otherwise. Network
errors happen, even on loopback devices.

Cosmere Infahm | 7 Feb 11:56 2012
Picon

Re: Re: node clustered http server workers die with misleading error

> Probably an insufficiency unless demonstrated otherwise. Network
> errors happen, even on loopback devices.

The return value is not being checked after the line
   result = _open_osfhandle((intptr_t)file, flags);
in fs.c

In my case the return value is -1 and errno is set to EMFILE ie I encounter the Windows open file limit at the stream i/o level

http://msdn.microsoft.com/en-us/library/6e3b887c%28v=vs.100%29.aspx

C run-time I/O now supports many more open files on Win32 platforms than in previous versions. Up to 2,048 files can be open simultaneously at the lowio level (that is, opened and accessed by means of the _open, _read, _write, and so forth family of I/O functions). Up to 512 files can be open simultaneously at the stdio level (that is, opened and accessed by means of the fopen, fgetc, fputc, and so forth family of functions). The limit of 512 open files at the stdio level can be increased to a maximum of 2,048 by means of the _setmaxstdio function.

Because stdio-level functions, such as fopen, are built on top of the lowio functions, the maximum of 2,048 is a hard upper limit for the number of simultaneously open files accessed through the C run-time library.

How can apps avoid bumping into these limits? Can node automatically increase this limit to the max allowed by the lowio level? Can node switch to using Windows API alone?

--

--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nodejs-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
To unsubscribe from this group, send email to
nodejs+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en
Bert Belder | 7 Feb 14:05 2012
Picon

Re: node clustered http server workers die with misleading error

On Feb 7, 11:56 am, Cosmere Infahm <infinic...@...> wrote:
> > Probably an insufficiency unless demonstrated otherwise. Network
> > errors happen, even on loopback devices.
>
> The return value is not being checked after the line
>    result = _open_osfhandle((intptr_t)file, flags);
> in fs.c
>
> In my case the return value is -1 and errno is set to EMFILE ie I encounter
> the Windows open file limit at the stream i/o level
>
> http://msdn.microsoft.com/en-us/library/6e3b887c%28v=vs.100%29.aspx
>
> > C run-time I/O now supports many more open files on Win32 platforms than
> > in previous versions. Up to 2,048 files can be open simultaneously at the lowio
> > level <http://msdn.microsoft.com/en-us/library/40bbyw78.aspx> (that is,
> > opened and accessed by means of the _open, _read, _write, and so forth
> > family of I/O functions). Up to 512 files can be open simultaneously at the stdio
> > level <http://msdn.microsoft.com/en-us/library/c565h7xx.aspx> (that is,
> > opened and accessed by means of the fopen, fgetc, fputc, and so forth
> > family of functions). The limit of 512 open files at the stdio level can
> > be increased to a maximum of 2,048 by means of the _setmaxstdio function.
>
> > Because stdio-level functions, such as fopen, are built on top of the
> > lowio functions, the maximum of 2,048 is a hard upper limit for the
> > number of simultaneously open files accessed through the C run-time library.
>
> How can apps avoid bumping into these limits? Can node automatically
> increase this limit to the max allowed by the lowio level? Can node switch
> to using Windows API alone?

Node will switch to using the win32 api alone. That was always the
plan, but we didn't get there yet.

--

-- 
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nodejs@...
To unsubscribe from this group, send email to
nodejs+unsubscribe@...
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en

NodeJazz | 7 Feb 15:38 2012
Picon

Re: node clustered http server workers die with misleading error

> Node will switch to using the win32 api alone. That was always the
> plan, but we didn't get there yet.

In the meanwhile could you please patch node with a check on the
return value of _open_osfhandle() and bump up the file handle limit to
2048 using _setmaxstdio()?

--

Stefan Rakanov | 14 Feb 09:17 2012
Picon

Re: node clustered http server workers die with misleading error

Cosmere Infahm <infinicosm <at> ...> writes:

> 
> 
> > Probably an insufficiency unless demonstrated otherwise. Network> errors 
happen, even on loopback devices.The return value is not being checked after the 
line   result = _open_osfhandle((intptr_t)file, flags);
> in fs.cIn my case the return value is -1 and errno is set to EMFILE ie I 
encounter the Windows open file limit at the stream i/o level
> 
> http://msdn.microsoft.com/en-us/library/6e3b887c%28v=vs.100%29.aspxC run-time 
I/O now supports many
>  more open files on Win32 platforms than in previous versions. Up to 
> 2,048 files can be open simultaneously at the lowio level (that is, opened and 
accessed by means of the _open, _read, _write, and so forth family of I/O 
functions). Up to 512 files can be open simultaneously at the stdio level (that 
is, opened and accessed by means of the fopen, fgetc, fputc, and so forth family 
of functions). The limit of 512 open files at the stdio level can be increased 
to a maximum of 2,048 by means of the _setmaxstdio function.
> Because stdio-level functions, such as fopen, are built on top of the lowio
>  functions, the maximum of 2,048 is a hard upper limit for the number of
>  simultaneously open files accessed through the C run-time library.
> How can apps avoid bumping into these limits? Can node automatically increase 
this limit to the max allowed by the lowio level? Can node switch to using 
Windows API alone?--
> 
You discuss windows build here, but I'm sure there are limits in linux too. 
Got same issue on Ubuntu 11.10 by running "ab -n20000 -c1000 <server>"

stream.js:105
      throw er; // Unhandled stream error in pipe.
            ^
Error: EMFILE, too many open files 

It is fully reproducable in v0.6.10 

--

-- 
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nodejs@...
To unsubscribe from this group, send email to
nodejs+unsubscribe@...
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en

mscdex | 14 Feb 12:07 2012
Picon

Re: node clustered http server workers die with misleading error

On Feb 14, 3:17 am, Stefan Rakanov <sraka...@...> wrote:
> You discuss windows build here, but I'm sure there are limits in linux too.
> Got same issue on Ubuntu 11.10 by running "ab -n20000 -c1000 <server>"
>
> stream.js:105
>       throw er; // Unhandled stream error in pipe.
>             ^
> Error: EMFILE, too many open files
>
> It is fully reproducable in v0.6.10

Check your value of `ulimit -n`

--

-- 
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nodejs@...
To unsubscribe from this group, send email to
nodejs+unsubscribe@...
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en


Gmane