headers
Ryan Gunn | 20 Aug 2012 15:36
Picon
Favicon

RabbitMQ 2.8.2 - Service Crash on Windows 2008 R2

Good Day,

This morning my RabbitMQ service crashed for some reason. I'm still trying to understand these log files
and hoping someone can assist. I'm using RabbitMQ 2.8.2, Windows Server 2008 R2 and Erlang R15B. There
seemed to be a lot of connections being made to the rabbit service when this crashed happened. I don't know
if that would maybe be the cause of the service crash. Please find attached log file for more information.
I've also looked through the Windows Event Log and cannot find anything.

Thank You

Ryan Gunn

=INFO REPORT==== 20-Aug-2012::11:01:50 ===
accepting AMQP connection <0.27926.618> (10.0.5.254:55682 -> 10.0.5.36:5207)

=INFO REPORT==== 20-Aug-2012::11:01:50 ===
accepting AMQP connection <0.27930.618> (10.0.5.254:55683 -> 10.0.5.36:5207)

=INFO REPORT==== 20-Aug-2012::11:01:51 ===
accepting AMQP connection <0.27948.618> (10.0.5.254:55687 -> 10.0.5.36:5207)

=INFO REPORT==== 20-Aug-2012::11:01:51 ===
accepting AMQP connection <0.27952.618> (10.0.5.254:55688 -> 10.0.5.36:5207)

=INFO REPORT==== 20-Aug-2012::11:01:51 ===
accepting AMQP connection <0.27970.618> (10.0.5.254:55689 -> 10.0.5.36:5207)

=INFO REPORT==== 20-Aug-2012::11:01:51 ===
accepting AMQP connection <0.27986.618> (10.0.5.254:55690 -> 10.0.5.36:5207)
(Continue reading)

Permalink | Reply |
headers
Emile Joubert | 20 Aug 2012 19:17
Favicon

Re: RabbitMQ 2.8.2 - Service Crash on Windows 2008 R2

Hi Ryan,

On 20/08/12 14:36, Ryan Gunn wrote:
> ** Reason for termination == 
> ** {accept_failed,enfile}

This is a POSIX error code meaning you have run out of file descriptors.
The errors that follow are a consequence of this. You should be able to
raise the limit by setting the ERL_MAX_PORTS variable. Make sure you
really need to open that many connections and that your application is
not leaking connections. I see that connections are accepted at an
alarming rate just before the error.

-Emile
Permalink | Reply |
headers
Matthias Radestock | 20 Aug 2012 20:27
Favicon

Re: RabbitMQ 2.8.2 - Service Crash on Windows 2008 R2

On 20/08/12 18:17, Emile Joubert wrote:
> On 20/08/12 14:36, Ryan Gunn wrote:
>> ** Reason for termination ==
>> ** {accept_failed,enfile}
>
> This is a POSIX error code meaning you have run out of file descriptors.
> The errors that follow are a consequence of this. You should be able to
> raise the limit by setting the ERL_MAX_PORTS variable. Make sure you
> really need to open that many connections and that your application is
> not leaking connections. I see that connections are accepted at an
> alarming rate just before the error.

Alas rabbit generally refuses to accept more connection when file 
descriptors are scarce. I wonder why this isn't happening here.

What limits are reported by 'rabbitmqctl status'?

Matthias.
Permalink | Reply |
headers
Icidis | 21 Aug 2012 07:26
Picon
Favicon

Re: RabbitMQ 2.8.2 - Service Crash on Windows 2008 R2


Matthias Radestock-3 wrote
> 
> On 20/08/12 18:17, Emile Joubert wrote:
>> On 20/08/12 14:36, Ryan Gunn wrote:
>>> ** Reason for termination ==
>>> ** {accept_failed,enfile}
>>
>> This is a POSIX error code meaning you have run out of file descriptors.
>> The errors that follow are a consequence of this. You should be able to
>> raise the limit by setting the ERL_MAX_PORTS variable. Make sure you
>> really need to open that many connections and that your application is
>> not leaking connections. I see that connections are accepted at an
>> alarming rate just before the error.
> 
> Alas rabbit generally refuses to accept more connection when file 
> descriptors are scarce. I wonder why this isn't happening here.
> 
> What limits are reported by 'rabbitmqctl status'?
> 
> Matthias.
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss <at> .rabbitmq
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
> 

Good Day Emile & Matthias,

I've attached the rabbitmqctl status report, I will go through all my
(Continue reading)

Permalink | Reply |
headers
Matthias Radestock | 21 Aug 2012 16:15
Favicon

Re: RabbitMQ 2.8.2 - Service Crash on Windows 2008 R2

Ryan,

On 21/08/12 06:26, Icidis wrote:
> I've attached the rabbitmqctl status report

Those limits look sane. However, I do notice that you are running an old 
version of Erlang, and in particular a 32-bit version. I suggest you 
upgrade to the 64-bit Windows version of R15B01. Might as well update 
rabbit at the same time.

I doubt the upgrades will fix the problem, but they are worth a try and 
make it easier for us to investigate further.

Also, since you are running the management plug-in, I suggest you keep 
and eye on file descriptor and socket descriptor counts reported there. 
In particular it would be interesting to see whether when the 'used' 
counts get near the limit there is any discrepancy between those counts 
and those reported by 'rabbitmqctl status'.

Regards,

Matthias.
Permalink | Reply |

Gmane