Breno Silva | 7 Sep 18:49 2011
Picon

Re: Failed to access DBM file (data/ip): Resource deadlock avoided

Hi Kevin,

We are planning to implement  a DDoS learning algorithm for ModSecurity. I know using data collection in heavy systems can cause performance issues. Most of Apache modules which works for  bandwidth limitation also stores data into temporary files, and it will also cause performance problems. So we need some code to store network information only in memory, persist only the learned data for recovery. It will also requires a time-to-time cleanup of internal modsecurity data structs. So i think in the future we will have a good solution.

Until we do not have a better solution, maybe you could try limit per ip connections in a another network device (firewall, sw) ?

Thanks

Breno

On Wed, Sep 7, 2011 at 10:42 AM, Kevin Jackson <kevin <at> linuxservices.co.uk> wrote:
Dear all,
It's been a while since I posted this issue and I want to share with
you some of my experiences as I'm seeing issues when the IP collection
grows in size.

My setup is Apache 2.2.20, ModSecurity 2.6.1 and CRS 2.2.1.
After a lot of performance testing and configuring ModSecurity to do 1
job (protect against a DoS against the web servers [which in turn
would protect against the real assets that ModSecurity setup is
proxying for]) on a "2CPU", 7Gb box (in EC2, m1.large) I get somewhere
between 200 and 300 hits per second against a static resource served
from Apache.  Not great, but acceptable.  What isn't acceptable is the
wildly fluctuating figures I get and load hitting 200+ when the IP
data collection grows in size.

My performance tests are replaying existing logs with real IP and URIs
in, but I use the IP to set an X-Forwarded-For header (so ModSec sees
it as a new IP), it uses the hash of the UA to give a level of
uniqueness to the IP as per the CRS configs and I rewrite all URLs to
a static file (to ensure I'm not running stuff against live in any
shape or form).

All is good at the beginning of the run and my IP collection database
is at 0Mb.  I get around 300 hits per second at a load avg of around
0.4.  Over a short period of time (at the rate mentioned above), this
IP collection builds up and when it hits a certain size, its game
over.  High load starts around 5-10 minutes.  Leaving this running for
20 mins sees load shoot up to 200+.  This isn't a gradual increase, it
hits a point and fails.  This seems to be file size as interrogating
the SDBM file lists for the IP collections it is a small figure.  For
example, I have a 700Mb SDBM file and inside are only 28 IP
collections.

I've played with TIMEOUT values to see if keeping less is helpful and
won't affect things too much, but ideally for DoS protection I'd hope
that keeping the IP in the database for as long as possible will be
more beneficial.

Any help is appreciated - as a potential mitigation step against a
Layer 7 DDoS (network and other good stuff in place to protect against
this aside) - what can I do from here?

Cheers,

Kev

On 19 August 2011 13:37, Ryan Barnett <RBarnett <at> trustwave.com> wrote:
>
> On 8/19/11 6:55 AM, "Kevin Jackson" <kevin <at> linuxservices.co.uk> wrote:
>
>>Dear all,
>>After a number of weeks fine tuning the CRS rules appropriate for a
>>busy website, we did a full live test pushing traffic through Apache
>>with ModSecurity 2.6.1 proxying requests to evaluate the performance
>>of ModSecurity through the backend service.
>>
>>Traffic is around 1k/second and we had Apache running on 18 servers
>>sitting behind a load balancer and Apache coping without issue.
>>
>>ModSecurity on the other hand didn't fare so well - it struggled to
>>keep pace causing resource deadlock issues on the ip database.  The
>>upshot was incredibly high load averages for obvious reasons and poor
>>user experience.
>>
>>--ba66410d-H--
>>Message: Failed to access DBM file "/usr/local/apache/data/ip":
>>Resource deadlock avoided
>>Apache-Handler: proxy-server
>>Stopwatch: 1313660329409820 283469 (- - -)
>>Stopwatch2: 1313660329409820 283469; combined=99526, p1=3446,
>>p2=12838, p3=3, p4=0, p5=41666, sr=3272, sw=41573, l=0, gc=0
>>Producer: ModSecurity for Apache/2.6.1 (http://www.modsecurity.org/);
>>core ruleset/2.2.1.
>>Server: Apache
>>
>>--ba66410d-Z--
>>
>>My question is what to suggest here? People must be running
>>ModSecurity on high volume websites? Have you encountered this issue?
>>This traffic is a fraction of our peak traffic at a quiet time during
>>the day and the environment only coped for about 10 minutes before
>>keeling over.
>
> Hey Kevin,
> Question for you - are you using the IP persistent collection to track
> data about clients across requests?  If not, then you may want to disable
> the initcol rules at the end of the modsecurity_crs_10_config.conf file.
>
> See if that helps.
>
> -Ryan
>
>>
>>Cheers,
>>
>>Kev
>>
>>--------------------------------------------------------------------------
>>----
>>Get a FREE DOWNLOAD! and learn more about uberSVN rich system,
>>user administration capabilities and model configuration. Take
>>the hassle out of deploying and managing Subversion and the
>>tools developers use with it. http://p.sf.net/sfu/wandisco-d2d-2
>>_______________________________________________
>>mod-security-users mailing list
>>mod-security-users <at> lists.sourceforge.net
>>https://lists.sourceforge.net/lists/listinfo/mod-security-users
>>ModSecurity Services from Trustwave's SpiderLabs:
>>https://www.trustwave.com/application-security.php
>>
>
>
> This transmission may contain information that is privileged, confidential, and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or use of the information contained herein (including any reliance thereon) is STRICTLY PROHIBITED. If you received this transmission in error, please immediately contact the sender and destroy the material in its entirety, whether in electronic or hard copy format.
>
>

------------------------------------------------------------------------------
Using storage to extend the benefits of virtualization and iSCSI
Virtualization increases hardware utilization and delivers a new level of
agility. Learn what those decisions are and how to modernize your storage
and backup environments for virtualization.
http://www.accelacomm.com/jaw/sfnl/114/51434361/
_______________________________________________
mod-security-users mailing list
mod-security-users <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/mod-security-users
ModSecurity Services from Trustwave's SpiderLabs:
https://www.trustwave.com/application-security.php

------------------------------------------------------------------------------
Using storage to extend the benefits of virtualization and iSCSI
Virtualization increases hardware utilization and delivers a new level of
agility. Learn what those decisions are and how to modernize your storage 
and backup environments for virtualization.
http://www.accelacomm.com/jaw/sfnl/114/51434361/
_______________________________________________
mod-security-users mailing list
mod-security-users <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/mod-security-users
ModSecurity Services from Trustwave's SpiderLabs:
https://www.trustwave.com/application-security.php
Michael Haas | 8 Sep 00:22 2011
Picon

Re: Failed to access DBM file (data/ip): Resource deadlock avoided

Hello,

the size is also a problem for the session collection, i need for the
persistence of 15000 sessions around 1GB Storage (around 64k per
session).
Is it possible to reduce this or is this really needed?
I use a modified version of modsecurity_crs_16_session_hijacking.conf
from crs 2.2.1 where i also save the data to a backend and there the
size is not that big.

Thanks in advance
Michael

2011/9/7 Breno Silva <breno.silva <at> gmail.com>:
> Hi Kevin,
>
> We are planning to implement  a DDoS learning algorithm for ModSecurity. I
> know using data collection in heavy systems can cause performance issues.
> Most of Apache modules which works for  bandwidth limitation also stores
> data into temporary files, and it will also cause performance problems. So
> we need some code to store network information only in memory, persist only
> the learned data for recovery. It will also requires a time-to-time cleanup
> of internal modsecurity data structs. So i think in the future we will have
> a good solution.
>
> Until we do not have a better solution, maybe you could try limit per ip
> connections in a another network device (firewall, sw) ?
>
> Thanks
>
> Breno
>
> On Wed, Sep 7, 2011 at 10:42 AM, Kevin Jackson <kevin <at> linuxservices.co.uk>
> wrote:
>>
>> Dear all,
>> It's been a while since I posted this issue and I want to share with
>> you some of my experiences as I'm seeing issues when the IP collection
>> grows in size.
>>
>> My setup is Apache 2.2.20, ModSecurity 2.6.1 and CRS 2.2.1.
>> After a lot of performance testing and configuring ModSecurity to do 1
>> job (protect against a DoS against the web servers [which in turn
>> would protect against the real assets that ModSecurity setup is
>> proxying for]) on a "2CPU", 7Gb box (in EC2, m1.large) I get somewhere
>> between 200 and 300 hits per second against a static resource served
>> from Apache.  Not great, but acceptable.  What isn't acceptable is the
>> wildly fluctuating figures I get and load hitting 200+ when the IP
>> data collection grows in size.
>>
>> My performance tests are replaying existing logs with real IP and URIs
>> in, but I use the IP to set an X-Forwarded-For header (so ModSec sees
>> it as a new IP), it uses the hash of the UA to give a level of
>> uniqueness to the IP as per the CRS configs and I rewrite all URLs to
>> a static file (to ensure I'm not running stuff against live in any
>> shape or form).
>>
>> All is good at the beginning of the run and my IP collection database
>> is at 0Mb.  I get around 300 hits per second at a load avg of around
>> 0.4.  Over a short period of time (at the rate mentioned above), this
>> IP collection builds up and when it hits a certain size, its game
>> over.  High load starts around 5-10 minutes.  Leaving this running for
>> 20 mins sees load shoot up to 200+.  This isn't a gradual increase, it
>> hits a point and fails.  This seems to be file size as interrogating
>> the SDBM file lists for the IP collections it is a small figure.  For
>> example, I have a 700Mb SDBM file and inside are only 28 IP
>> collections.
>>
>> I've played with TIMEOUT values to see if keeping less is helpful and
>> won't affect things too much, but ideally for DoS protection I'd hope
>> that keeping the IP in the database for as long as possible will be
>> more beneficial.
>>
>> Any help is appreciated - as a potential mitigation step against a
>> Layer 7 DDoS (network and other good stuff in place to protect against
>> this aside) - what can I do from here?
>>
>> Cheers,
>>
>> Kev
>>
>> On 19 August 2011 13:37, Ryan Barnett <RBarnett <at> trustwave.com> wrote:
>> >
>> > On 8/19/11 6:55 AM, "Kevin Jackson" <kevin <at> linuxservices.co.uk> wrote:
>> >
>> >>Dear all,
>> >>After a number of weeks fine tuning the CRS rules appropriate for a
>> >>busy website, we did a full live test pushing traffic through Apache
>> >>with ModSecurity 2.6.1 proxying requests to evaluate the performance
>> >>of ModSecurity through the backend service.
>> >>
>> >>Traffic is around 1k/second and we had Apache running on 18 servers
>> >>sitting behind a load balancer and Apache coping without issue.
>> >>
>> >>ModSecurity on the other hand didn't fare so well - it struggled to
>> >>keep pace causing resource deadlock issues on the ip database.  The
>> >>upshot was incredibly high load averages for obvious reasons and poor
>> >>user experience.
>> >>
>> >>--ba66410d-H--
>> >>Message: Failed to access DBM file "/usr/local/apache/data/ip":
>> >>Resource deadlock avoided
>> >>Apache-Handler: proxy-server
>> >>Stopwatch: 1313660329409820 283469 (- - -)
>> >>Stopwatch2: 1313660329409820 283469; combined=99526, p1=3446,
>> >>p2=12838, p3=3, p4=0, p5=41666, sr=3272, sw=41573, l=0, gc=0
>> >>Producer: ModSecurity for Apache/2.6.1 (http://www.modsecurity.org/);
>> >>core ruleset/2.2.1.
>> >>Server: Apache
>> >>
>> >>--ba66410d-Z--
>> >>
>> >>My question is what to suggest here? People must be running
>> >>ModSecurity on high volume websites? Have you encountered this issue?
>> >>This traffic is a fraction of our peak traffic at a quiet time during
>> >>the day and the environment only coped for about 10 minutes before
>> >>keeling over.
>> >
>> > Hey Kevin,
>> > Question for you - are you using the IP persistent collection to track
>> > data about clients across requests?  If not, then you may want to
>> > disable
>> > the initcol rules at the end of the modsecurity_crs_10_config.conf file.
>> >
>> > See if that helps.
>> >
>> > -Ryan
>> >
>> >>
>> >>Cheers,
>> >>
>> >>Kev
>> >>
>>
>> >> >>--------------------------------------------------------------------------
>> >>----
>> >>Get a FREE DOWNLOAD! and learn more about uberSVN rich system,
>> >>user administration capabilities and model configuration. Take
>> >>the hassle out of deploying and managing Subversion and the
>> >>tools developers use with it. http://p.sf.net/sfu/wandisco-d2d-2
>> >>_______________________________________________
>> >>mod-security-users mailing list
>> >>mod-security-users <at> lists.sourceforge.net
>> >>https://lists.sourceforge.net/lists/listinfo/mod-security-users
>> >>ModSecurity Services from Trustwave's SpiderLabs:
>> >>https://www.trustwave.com/application-security.php
>> >>
>> >
>> >
>> > This transmission may contain information that is privileged,
>> > confidential, and/or exempt from disclosure under applicable law. If you are
>> > not the intended recipient, you are hereby notified that any disclosure,
>> > copying, distribution, or use of the information contained herein (including
>> > any reliance thereon) is STRICTLY PROHIBITED. If you received this
>> > transmission in error, please immediately contact the sender and destroy the
>> > material in its entirety, whether in electronic or hard copy format.
>> >
>> >
>>
>>
>> ------------------------------------------------------------------------------
>> Using storage to extend the benefits of virtualization and iSCSI
>> Virtualization increases hardware utilization and delivers a new level of
>> agility. Learn what those decisions are and how to modernize your storage
>> and backup environments for virtualization.
>> http://www.accelacomm.com/jaw/sfnl/114/51434361/
>> _______________________________________________
>> mod-security-users mailing list
>> mod-security-users <at> lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/mod-security-users
>> ModSecurity Services from Trustwave's SpiderLabs:
>> https://www.trustwave.com/application-security.php
>
>
> ------------------------------------------------------------------------------
> Using storage to extend the benefits of virtualization and iSCSI
> Virtualization increases hardware utilization and delivers a new level of
> agility. Learn what those decisions are and how to modernize your storage
> and backup environments for virtualization.
> http://www.accelacomm.com/jaw/sfnl/114/51434361/
> _______________________________________________
> mod-security-users mailing list
> mod-security-users <at> lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/mod-security-users
> ModSecurity Services from Trustwave's SpiderLabs:
> https://www.trustwave.com/application-security.php
>
>

------------------------------------------------------------------------------
Using storage to extend the benefits of virtualization and iSCSI
Virtualization increases hardware utilization and delivers a new level of
agility. Learn what those decisions are and how to modernize your storage 
and backup environments for virtualization.
http://www.accelacomm.com/jaw/sfnl/114/51434361/
_______________________________________________
mod-security-users mailing list
mod-security-users <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/mod-security-users
ModSecurity Services from Trustwave's SpiderLabs:
https://www.trustwave.com/application-security.php

Michael Haas | 8 Sep 16:40 2011
Picon

Re: Failed to access DBM file (data/ip): Resource deadlock avoided

Found also this bug
https://www.modsecurity.org/tracker/browse/MODSEC-160 seems that i
need to disable the session rules because the file size is only
increasing.
Is there a schedule for the availability of 2.7.0.

Thanks
Michael

2011/9/8 Michael Haas <michael.haas10 <at> gmail.com>:
> Hello,
>
> the size is also a problem for the session collection, i need for the
> persistence of 15000 sessions around 1GB Storage (around 64k per
> session).
> Is it possible to reduce this or is this really needed?
> I use a modified version of modsecurity_crs_16_session_hijacking.conf
> from crs 2.2.1 where i also save the data to a backend and there the
> size is not that big.
>
> Thanks in advance
> Michael
>
>
> 2011/9/7 Breno Silva <breno.silva <at> gmail.com>:
>> Hi Kevin,
>>
>> We are planning to implement  a DDoS learning algorithm for ModSecurity. I
>> know using data collection in heavy systems can cause performance issues.
>> Most of Apache modules which works for  bandwidth limitation also stores
>> data into temporary files, and it will also cause performance problems. So
>> we need some code to store network information only in memory, persist only
>> the learned data for recovery. It will also requires a time-to-time cleanup
>> of internal modsecurity data structs. So i think in the future we will have
>> a good solution.
>>
>> Until we do not have a better solution, maybe you could try limit per ip
>> connections in a another network device (firewall, sw) ?
>>
>> Thanks
>>
>> Breno
>>
>> On Wed, Sep 7, 2011 at 10:42 AM, Kevin Jackson <kevin <at> linuxservices.co.uk>
>> wrote:
>>>
>>> Dear all,
>>> It's been a while since I posted this issue and I want to share with
>>> you some of my experiences as I'm seeing issues when the IP collection
>>> grows in size.
>>>
>>> My setup is Apache 2.2.20, ModSecurity 2.6.1 and CRS 2.2.1.
>>> After a lot of performance testing and configuring ModSecurity to do 1
>>> job (protect against a DoS against the web servers [which in turn
>>> would protect against the real assets that ModSecurity setup is
>>> proxying for]) on a "2CPU", 7Gb box (in EC2, m1.large) I get somewhere
>>> between 200 and 300 hits per second against a static resource served
>>> from Apache.  Not great, but acceptable.  What isn't acceptable is the
>>> wildly fluctuating figures I get and load hitting 200+ when the IP
>>> data collection grows in size.
>>>
>>> My performance tests are replaying existing logs with real IP and URIs
>>> in, but I use the IP to set an X-Forwarded-For header (so ModSec sees
>>> it as a new IP), it uses the hash of the UA to give a level of
>>> uniqueness to the IP as per the CRS configs and I rewrite all URLs to
>>> a static file (to ensure I'm not running stuff against live in any
>>> shape or form).
>>>
>>> All is good at the beginning of the run and my IP collection database
>>> is at 0Mb.  I get around 300 hits per second at a load avg of around
>>> 0.4.  Over a short period of time (at the rate mentioned above), this
>>> IP collection builds up and when it hits a certain size, its game
>>> over.  High load starts around 5-10 minutes.  Leaving this running for
>>> 20 mins sees load shoot up to 200+.  This isn't a gradual increase, it
>>> hits a point and fails.  This seems to be file size as interrogating
>>> the SDBM file lists for the IP collections it is a small figure.  For
>>> example, I have a 700Mb SDBM file and inside are only 28 IP
>>> collections.
>>>
>>> I've played with TIMEOUT values to see if keeping less is helpful and
>>> won't affect things too much, but ideally for DoS protection I'd hope
>>> that keeping the IP in the database for as long as possible will be
>>> more beneficial.
>>>
>>> Any help is appreciated - as a potential mitigation step against a
>>> Layer 7 DDoS (network and other good stuff in place to protect against
>>> this aside) - what can I do from here?
>>>
>>> Cheers,
>>>
>>> Kev
>>>
>>> On 19 August 2011 13:37, Ryan Barnett <RBarnett <at> trustwave.com> wrote:
>>> >
>>> > On 8/19/11 6:55 AM, "Kevin Jackson" <kevin <at> linuxservices.co.uk> wrote:
>>> >
>>> >>Dear all,
>>> >>After a number of weeks fine tuning the CRS rules appropriate for a
>>> >>busy website, we did a full live test pushing traffic through Apache
>>> >>with ModSecurity 2.6.1 proxying requests to evaluate the performance
>>> >>of ModSecurity through the backend service.
>>> >>
>>> >>Traffic is around 1k/second and we had Apache running on 18 servers
>>> >>sitting behind a load balancer and Apache coping without issue.
>>> >>
>>> >>ModSecurity on the other hand didn't fare so well - it struggled to
>>> >>keep pace causing resource deadlock issues on the ip database.  The
>>> >>upshot was incredibly high load averages for obvious reasons and poor
>>> >>user experience.
>>> >>
>>> >>--ba66410d-H--
>>> >>Message: Failed to access DBM file "/usr/local/apache/data/ip":
>>> >>Resource deadlock avoided
>>> >>Apache-Handler: proxy-server
>>> >>Stopwatch: 1313660329409820 283469 (- - -)
>>> >>Stopwatch2: 1313660329409820 283469; combined=99526, p1=3446,
>>> >>p2=12838, p3=3, p4=0, p5=41666, sr=3272, sw=41573, l=0, gc=0
>>> >>Producer: ModSecurity for Apache/2.6.1 (http://www.modsecurity.org/);
>>> >>core ruleset/2.2.1.
>>> >>Server: Apache
>>> >>
>>> >>--ba66410d-Z--
>>> >>
>>> >>My question is what to suggest here? People must be running
>>> >>ModSecurity on high volume websites? Have you encountered this issue?
>>> >>This traffic is a fraction of our peak traffic at a quiet time during
>>> >>the day and the environment only coped for about 10 minutes before
>>> >>keeling over.
>>> >
>>> > Hey Kevin,
>>> > Question for you - are you using the IP persistent collection to track
>>> > data about clients across requests?  If not, then you may want to
>>> > disable
>>> > the initcol rules at the end of the modsecurity_crs_10_config.conf file.
>>> >
>>> > See if that helps.
>>> >
>>> > -Ryan
>>> >
>>> >>
>>> >>Cheers,
>>> >>
>>> >>Kev
>>> >>
>>>
>>> >> >>--------------------------------------------------------------------------
>>> >>----
>>> >>Get a FREE DOWNLOAD! and learn more about uberSVN rich system,
>>> >>user administration capabilities and model configuration. Take
>>> >>the hassle out of deploying and managing Subversion and the
>>> >>tools developers use with it. http://p.sf.net/sfu/wandisco-d2d-2
>>> >>_______________________________________________
>>> >>mod-security-users mailing list
>>> >>mod-security-users <at> lists.sourceforge.net
>>> >>https://lists.sourceforge.net/lists/listinfo/mod-security-users
>>> >>ModSecurity Services from Trustwave's SpiderLabs:
>>> >>https://www.trustwave.com/application-security.php
>>> >>
>>> >
>>> >
>>> > This transmission may contain information that is privileged,
>>> > confidential, and/or exempt from disclosure under applicable law. If you are
>>> > not the intended recipient, you are hereby notified that any disclosure,
>>> > copying, distribution, or use of the information contained herein (including
>>> > any reliance thereon) is STRICTLY PROHIBITED. If you received this
>>> > transmission in error, please immediately contact the sender and destroy the
>>> > material in its entirety, whether in electronic or hard copy format.
>>> >
>>> >
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> Using storage to extend the benefits of virtualization and iSCSI
>>> Virtualization increases hardware utilization and delivers a new level of
>>> agility. Learn what those decisions are and how to modernize your storage
>>> and backup environments for virtualization.
>>> http://www.accelacomm.com/jaw/sfnl/114/51434361/
>>> _______________________________________________
>>> mod-security-users mailing list
>>> mod-security-users <at> lists.sourceforge.net
>>> https://lists.sourceforge.net/lists/listinfo/mod-security-users
>>> ModSecurity Services from Trustwave's SpiderLabs:
>>> https://www.trustwave.com/application-security.php
>>
>>
>> ------------------------------------------------------------------------------
>> Using storage to extend the benefits of virtualization and iSCSI
>> Virtualization increases hardware utilization and delivers a new level of
>> agility. Learn what those decisions are and how to modernize your storage
>> and backup environments for virtualization.
>> http://www.accelacomm.com/jaw/sfnl/114/51434361/
>> _______________________________________________
>> mod-security-users mailing list
>> mod-security-users <at> lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/mod-security-users
>> ModSecurity Services from Trustwave's SpiderLabs:
>> https://www.trustwave.com/application-security.php
>>
>>
>

------------------------------------------------------------------------------
Doing More with Less: The Next Generation Virtual Desktop 
What are the key obstacles that have prevented many mid-market businesses
from deploying virtual desktops?   How do next-generation virtual desktops
provide companies an easier-to-deploy, easier-to-manage and more affordable
virtual desktop model.http://www.accelacomm.com/jaw/sfnl/114/51426474/
_______________________________________________
mod-security-users mailing list
mod-security-users <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/mod-security-users
ModSecurity Services from Trustwave's SpiderLabs:
https://www.trustwave.com/application-security.php

Breno Silva | 8 Sep 16:54 2011
Picon

Re: Failed to access DBM file (data/ip): Resource deadlock avoided

Hey Michel,

I'm going to change it for 2.6.3 version. So i can investigate it sooner.

Thanks

Breno

On Thu, Sep 8, 2011 at 9:40 AM, Michael Haas <michael.haas10 <at> gmail.com> wrote:
Found also this bug
https://www.modsecurity.org/tracker/browse/MODSEC-160 seems that i
need to disable the session rules because the file size is only
increasing.
Is there a schedule for the availability of 2.7.0.

Thanks
Michael

2011/9/8 Michael Haas <michael.haas10 <at> gmail.com>:
> Hello,
>
> the size is also a problem for the session collection, i need for the
> persistence of 15000 sessions around 1GB Storage (around 64k per
> session).
> Is it possible to reduce this or is this really needed?
> I use a modified version of modsecurity_crs_16_session_hijacking.conf
> from crs 2.2.1 where i also save the data to a backend and there the
> size is not that big.
>
> Thanks in advance
> Michael
>
>
> 2011/9/7 Breno Silva <breno.silva <at> gmail.com>:
>> Hi Kevin,
>>
>> We are planning to implement  a DDoS learning algorithm for ModSecurity. I
>> know using data collection in heavy systems can cause performance issues.
>> Most of Apache modules which works for  bandwidth limitation also stores
>> data into temporary files, and it will also cause performance problems. So
>> we need some code to store network information only in memory, persist only
>> the learned data for recovery. It will also requires a time-to-time cleanup
>> of internal modsecurity data structs. So i think in the future we will have
>> a good solution.
>>
>> Until we do not have a better solution, maybe you could try limit per ip
>> connections in a another network device (firewall, sw) ?
>>
>> Thanks
>>
>> Breno
>>
>> On Wed, Sep 7, 2011 at 10:42 AM, Kevin Jackson <kevin <at> linuxservices.co.uk>
>> wrote:
>>>
>>> Dear all,
>>> It's been a while since I posted this issue and I want to share with
>>> you some of my experiences as I'm seeing issues when the IP collection
>>> grows in size.
>>>
>>> My setup is Apache 2.2.20, ModSecurity 2.6.1 and CRS 2.2.1.
>>> After a lot of performance testing and configuring ModSecurity to do 1
>>> job (protect against a DoS against the web servers [which in turn
>>> would protect against the real assets that ModSecurity setup is
>>> proxying for]) on a "2CPU", 7Gb box (in EC2, m1.large) I get somewhere
>>> between 200 and 300 hits per second against a static resource served
>>> from Apache.  Not great, but acceptable.  What isn't acceptable is the
>>> wildly fluctuating figures I get and load hitting 200+ when the IP
>>> data collection grows in size.
>>>
>>> My performance tests are replaying existing logs with real IP and URIs
>>> in, but I use the IP to set an X-Forwarded-For header (so ModSec sees
>>> it as a new IP), it uses the hash of the UA to give a level of
>>> uniqueness to the IP as per the CRS configs and I rewrite all URLs to
>>> a static file (to ensure I'm not running stuff against live in any
>>> shape or form).
>>>
>>> All is good at the beginning of the run and my IP collection database
>>> is at 0Mb.  I get around 300 hits per second at a load avg of around
>>> 0.4.  Over a short period of time (at the rate mentioned above), this
>>> IP collection builds up and when it hits a certain size, its game
>>> over.  High load starts around 5-10 minutes.  Leaving this running for
>>> 20 mins sees load shoot up to 200+.  This isn't a gradual increase, it
>>> hits a point and fails.  This seems to be file size as interrogating
>>> the SDBM file lists for the IP collections it is a small figure.  For
>>> example, I have a 700Mb SDBM file and inside are only 28 IP
>>> collections.
>>>
>>> I've played with TIMEOUT values to see if keeping less is helpful and
>>> won't affect things too much, but ideally for DoS protection I'd hope
>>> that keeping the IP in the database for as long as possible will be
>>> more beneficial.
>>>
>>> Any help is appreciated - as a potential mitigation step against a
>>> Layer 7 DDoS (network and other good stuff in place to protect against
>>> this aside) - what can I do from here?
>>>
>>> Cheers,
>>>
>>> Kev
>>>
>>> On 19 August 2011 13:37, Ryan Barnett <RBarnett <at> trustwave.com> wrote:
>>> >
>>> > On 8/19/11 6:55 AM, "Kevin Jackson" <kevin <at> linuxservices.co.uk> wrote:
>>> >
>>> >>Dear all,
>>> >>After a number of weeks fine tuning the CRS rules appropriate for a
>>> >>busy website, we did a full live test pushing traffic through Apache
>>> >>with ModSecurity 2.6.1 proxying requests to evaluate the performance
>>> >>of ModSecurity through the backend service.
>>> >>
>>> >>Traffic is around 1k/second and we had Apache running on 18 servers
>>> >>sitting behind a load balancer and Apache coping without issue.
>>> >>
>>> >>ModSecurity on the other hand didn't fare so well - it struggled to
>>> >>keep pace causing resource deadlock issues on the ip database.  The
>>> >>upshot was incredibly high load averages for obvious reasons and poor
>>> >>user experience.
>>> >>
>>> >>--ba66410d-H--
>>> >>Message: Failed to access DBM file "/usr/local/apache/data/ip":
>>> >>Resource deadlock avoided
>>> >>Apache-Handler: proxy-server
>>> >>Stopwatch: 1313660329409820 283469 (- - -)
>>> >>Stopwatch2: 1313660329409820 283469; combined=99526, p1=3446,
>>> >>p2=12838, p3=3, p4=0, p5=41666, sr=3272, sw=41573, l=0, gc=0
>>> >>Producer: ModSecurity for Apache/2.6.1 (http://www.modsecurity.org/);
>>> >>core ruleset/2.2.1.
>>> >>Server: Apache
>>> >>
>>> >>--ba66410d-Z--
>>> >>
>>> >>My question is what to suggest here? People must be running
>>> >>ModSecurity on high volume websites? Have you encountered this issue?
>>> >>This traffic is a fraction of our peak traffic at a quiet time during
>>> >>the day and the environment only coped for about 10 minutes before
>>> >>keeling over.
>>> >
>>> > Hey Kevin,
>>> > Question for you - are you using the IP persistent collection to track
>>> > data about clients across requests?  If not, then you may want to
>>> > disable
>>> > the initcol rules at the end of the modsecurity_crs_10_config.conf file.
>>> >
>>> > See if that helps.
>>> >
>>> > -Ryan
>>> >
>>> >>
>>> >>Cheers,
>>> >>
>>> >>Kev
>>> >>
>>>
>>> >> >>--------------------------------------------------------------------------
>>> >>----
>>> >>Get a FREE DOWNLOAD! and learn more about uberSVN rich system,
>>> >>user administration capabilities and model configuration. Take
>>> >>the hassle out of deploying and managing Subversion and the
>>> >>tools developers use with it. http://p.sf.net/sfu/wandisco-d2d-2
>>> >>_______________________________________________
>>> >>mod-security-users mailing list
>>> >>mod-security-users <at> lists.sourceforge.net
>>> >>https://lists.sourceforge.net/lists/listinfo/mod-security-users
>>> >>ModSecurity Services from Trustwave's SpiderLabs:
>>> >>https://www.trustwave.com/application-security.php
>>> >>
>>> >
>>> >
>>> > This transmission may contain information that is privileged,
>>> > confidential, and/or exempt from disclosure under applicable law. If you are
>>> > not the intended recipient, you are hereby notified that any disclosure,
>>> > copying, distribution, or use of the information contained herein (including
>>> > any reliance thereon) is STRICTLY PROHIBITED. If you received this
>>> > transmission in error, please immediately contact the sender and destroy the
>>> > material in its entirety, whether in electronic or hard copy format.
>>> >
>>> >
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> Using storage to extend the benefits of virtualization and iSCSI
>>> Virtualization increases hardware utilization and delivers a new level of
>>> agility. Learn what those decisions are and how to modernize your storage
>>> and backup environments for virtualization.
>>> http://www.accelacomm.com/jaw/sfnl/114/51434361/
>>> _______________________________________________
>>> mod-security-users mailing list
>>> mod-security-users <at> lists.sourceforge.net
>>> https://lists.sourceforge.net/lists/listinfo/mod-security-users
>>> ModSecurity Services from Trustwave's SpiderLabs:
>>> https://www.trustwave.com/application-security.php
>>
>>
>> ------------------------------------------------------------------------------
>> Using storage to extend the benefits of virtualization and iSCSI
>> Virtualization increases hardware utilization and delivers a new level of
>> agility. Learn what those decisions are and how to modernize your storage
>> and backup environments for virtualization.
>> http://www.accelacomm.com/jaw/sfnl/114/51434361/
>> _______________________________________________
>> mod-security-users mailing list
>> mod-security-users <at> lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/mod-security-users
>> ModSecurity Services from Trustwave's SpiderLabs:
>> https://www.trustwave.com/application-security.php
>>
>>
>

------------------------------------------------------------------------------
Doing More with Less: The Next Generation Virtual Desktop 
What are the key obstacles that have prevented many mid-market businesses
from deploying virtual desktops?   How do next-generation virtual desktops
provide companies an easier-to-deploy, easier-to-manage and more affordable
virtual desktop model.http://www.accelacomm.com/jaw/sfnl/114/51426474/
_______________________________________________
mod-security-users mailing list
mod-security-users <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/mod-security-users
ModSecurity Services from Trustwave's SpiderLabs:
https://www.trustwave.com/application-security.php
Michael Haas | 8 Sep 16:57 2011
Picon

Re: Failed to access DBM file (data/ip): Resource deadlock avoided

Hi,

that's great, if you need assistance in testing some new versions or
the old, i can help with this.

Thanks
Michael

2011/9/8 Breno Silva <breno.silva <at> gmail.com>:
> Hey Michel,
>
> I'm going to change it for 2.6.3 version. So i can investigate it sooner.
>
> Thanks
>
> Breno
>
> On Thu, Sep 8, 2011 at 9:40 AM, Michael Haas <michael.haas10 <at> gmail.com>
> wrote:
>>
>> Found also this bug
>> https://www.modsecurity.org/tracker/browse/MODSEC-160 seems that i
>> need to disable the session rules because the file size is only
>> increasing.
>> Is there a schedule for the availability of 2.7.0.
>>
>> Thanks
>> Michael
>>
>> 2011/9/8 Michael Haas <michael.haas10 <at> gmail.com>:
>> > Hello,
>> >
>> > the size is also a problem for the session collection, i need for the
>> > persistence of 15000 sessions around 1GB Storage (around 64k per
>> > session).
>> > Is it possible to reduce this or is this really needed?
>> > I use a modified version of modsecurity_crs_16_session_hijacking.conf
>> > from crs 2.2.1 where i also save the data to a backend and there the
>> > size is not that big.
>> >
>> > Thanks in advance
>> > Michael
>> >
>> >
>> > 2011/9/7 Breno Silva <breno.silva <at> gmail.com>:
>> >> Hi Kevin,
>> >>
>> >> We are planning to implement  a DDoS learning algorithm for
>> >> ModSecurity. I
>> >> know using data collection in heavy systems can cause performance
>> >> issues.
>> >> Most of Apache modules which works for  bandwidth limitation also
>> >> stores
>> >> data into temporary files, and it will also cause performance problems.
>> >> So
>> >> we need some code to store network information only in memory, persist
>> >> only
>> >> the learned data for recovery. It will also requires a time-to-time
>> >> cleanup
>> >> of internal modsecurity data structs. So i think in the future we will
>> >> have
>> >> a good solution.
>> >>
>> >> Until we do not have a better solution, maybe you could try limit per
>> >> ip
>> >> connections in a another network device (firewall, sw) ?
>> >>
>> >> Thanks
>> >>
>> >> Breno
>> >>
>> >> On Wed, Sep 7, 2011 at 10:42 AM, Kevin Jackson
>> >> <kevin <at> linuxservices.co.uk>
>> >> wrote:
>> >>>
>> >>> Dear all,
>> >>> It's been a while since I posted this issue and I want to share with
>> >>> you some of my experiences as I'm seeing issues when the IP collection
>> >>> grows in size.
>> >>>
>> >>> My setup is Apache 2.2.20, ModSecurity 2.6.1 and CRS 2.2.1.
>> >>> After a lot of performance testing and configuring ModSecurity to do 1
>> >>> job (protect against a DoS against the web servers [which in turn
>> >>> would protect against the real assets that ModSecurity setup is
>> >>> proxying for]) on a "2CPU", 7Gb box (in EC2, m1.large) I get somewhere
>> >>> between 200 and 300 hits per second against a static resource served
>> >>> from Apache.  Not great, but acceptable.  What isn't acceptable is the
>> >>> wildly fluctuating figures I get and load hitting 200+ when the IP
>> >>> data collection grows in size.
>> >>>
>> >>> My performance tests are replaying existing logs with real IP and URIs
>> >>> in, but I use the IP to set an X-Forwarded-For header (so ModSec sees
>> >>> it as a new IP), it uses the hash of the UA to give a level of
>> >>> uniqueness to the IP as per the CRS configs and I rewrite all URLs to
>> >>> a static file (to ensure I'm not running stuff against live in any
>> >>> shape or form).
>> >>>
>> >>> All is good at the beginning of the run and my IP collection database
>> >>> is at 0Mb.  I get around 300 hits per second at a load avg of around
>> >>> 0.4.  Over a short period of time (at the rate mentioned above), this
>> >>> IP collection builds up and when it hits a certain size, its game
>> >>> over.  High load starts around 5-10 minutes.  Leaving this running for
>> >>> 20 mins sees load shoot up to 200+.  This isn't a gradual increase, it
>> >>> hits a point and fails.  This seems to be file size as interrogating
>> >>> the SDBM file lists for the IP collections it is a small figure.  For
>> >>> example, I have a 700Mb SDBM file and inside are only 28 IP
>> >>> collections.
>> >>>
>> >>> I've played with TIMEOUT values to see if keeping less is helpful and
>> >>> won't affect things too much, but ideally for DoS protection I'd hope
>> >>> that keeping the IP in the database for as long as possible will be
>> >>> more beneficial.
>> >>>
>> >>> Any help is appreciated - as a potential mitigation step against a
>> >>> Layer 7 DDoS (network and other good stuff in place to protect against
>> >>> this aside) - what can I do from here?
>> >>>
>> >>> Cheers,
>> >>>
>> >>> Kev
>> >>>
>> >>> On 19 August 2011 13:37, Ryan Barnett <RBarnett <at> trustwave.com> wrote:
>> >>> >
>> >>> > On 8/19/11 6:55 AM, "Kevin Jackson" <kevin <at> linuxservices.co.uk>
>> >>> > wrote:
>> >>> >
>> >>> >>Dear all,
>> >>> >>After a number of weeks fine tuning the CRS rules appropriate for a
>> >>> >>busy website, we did a full live test pushing traffic through Apache
>> >>> >>with ModSecurity 2.6.1 proxying requests to evaluate the performance
>> >>> >>of ModSecurity through the backend service.
>> >>> >>
>> >>> >>Traffic is around 1k/second and we had Apache running on 18 servers
>> >>> >>sitting behind a load balancer and Apache coping without issue.
>> >>> >>
>> >>> >>ModSecurity on the other hand didn't fare so well - it struggled to
>> >>> >>keep pace causing resource deadlock issues on the ip database.  The
>> >>> >>upshot was incredibly high load averages for obvious reasons and
>> >>> >> poor
>> >>> >>user experience.
>> >>> >>
>> >>> >>--ba66410d-H--
>> >>> >>Message: Failed to access DBM file "/usr/local/apache/data/ip":
>> >>> >>Resource deadlock avoided
>> >>> >>Apache-Handler: proxy-server
>> >>> >>Stopwatch: 1313660329409820 283469 (- - -)
>> >>> >>Stopwatch2: 1313660329409820 283469; combined=99526, p1=3446,
>> >>> >>p2=12838, p3=3, p4=0, p5=41666, sr=3272, sw=41573, l=0, gc=0
>> >>> >>Producer: ModSecurity for Apache/2.6.1
>> >>> >> (http://www.modsecurity.org/);
>> >>> >>core ruleset/2.2.1.
>> >>> >>Server: Apache
>> >>> >>
>> >>> >>--ba66410d-Z--
>> >>> >>
>> >>> >>My question is what to suggest here? People must be running
>> >>> >>ModSecurity on high volume websites? Have you encountered this
>> >>> >> issue?
>> >>> >>This traffic is a fraction of our peak traffic at a quiet time
>> >>> >> during
>> >>> >>the day and the environment only coped for about 10 minutes before
>> >>> >>keeling over.
>> >>> >
>> >>> > Hey Kevin,
>> >>> > Question for you - are you using the IP persistent collection to
>> >>> > track
>> >>> > data about clients across requests?  If not, then you may want to
>> >>> > disable
>> >>> > the initcol rules at the end of the modsecurity_crs_10_config.conf
>> >>> > file.
>> >>> >
>> >>> > See if that helps.
>> >>> >
>> >>> > -Ryan
>> >>> >
>> >>> >>
>> >>> >>Cheers,
>> >>> >>
>> >>> >>Kev
>> >>> >>
>> >>>
>> >>> >>
>> >>> >> >> >>--------------------------------------------------------------------------
>> >>> >>----
>> >>> >>Get a FREE DOWNLOAD! and learn more about uberSVN rich system,
>> >>> >>user administration capabilities and model configuration. Take
>> >>> >>the hassle out of deploying and managing Subversion and the
>> >>> >>tools developers use with it. http://p.sf.net/sfu/wandisco-d2d-2
>> >>> >>_______________________________________________
>> >>> >>mod-security-users mailing list
>> >>> >>mod-security-users <at> lists.sourceforge.net
>> >>> >>https://lists.sourceforge.net/lists/listinfo/mod-security-users
>> >>> >>ModSecurity Services from Trustwave's SpiderLabs:
>> >>> >>https://www.trustwave.com/application-security.php
>> >>> >>
>> >>> >
>> >>> >
>> >>> > This transmission may contain information that is privileged,
>> >>> > confidential, and/or exempt from disclosure under applicable law. If
>> >>> > you are
>> >>> > not the intended recipient, you are hereby notified that any
>> >>> > disclosure,
>> >>> > copying, distribution, or use of the information contained herein
>> >>> > (including
>> >>> > any reliance thereon) is STRICTLY PROHIBITED. If you received this
>> >>> > transmission in error, please immediately contact the sender and
>> >>> > destroy the
>> >>> > material in its entirety, whether in electronic or hard copy format.
>> >>> >
>> >>> >
>> >>>
>> >>>
>> >>>
>> >>> ------------------------------------------------------------------------------
>> >>> Using storage to extend the benefits of virtualization and iSCSI
>> >>> Virtualization increases hardware utilization and delivers a new level
>> >>> of
>> >>> agility. Learn what those decisions are and how to modernize your
>> >>> storage
>> >>> and backup environments for virtualization.
>> >>> http://www.accelacomm.com/jaw/sfnl/114/51434361/
>> >>> _______________________________________________
>> >>> mod-security-users mailing list
>> >>> mod-security-users <at> lists.sourceforge.net
>> >>> https://lists.sourceforge.net/lists/listinfo/mod-security-users
>> >>> ModSecurity Services from Trustwave's SpiderLabs:
>> >>> https://www.trustwave.com/application-security.php
>> >>
>> >>
>> >>
>> >> ------------------------------------------------------------------------------
>> >> Using storage to extend the benefits of virtualization and iSCSI
>> >> Virtualization increases hardware utilization and delivers a new level
>> >> of
>> >> agility. Learn what those decisions are and how to modernize your
>> >> storage
>> >> and backup environments for virtualization.
>> >> http://www.accelacomm.com/jaw/sfnl/114/51434361/
>> >> _______________________________________________
>> >> mod-security-users mailing list
>> >> mod-security-users <at> lists.sourceforge.net
>> >> https://lists.sourceforge.net/lists/listinfo/mod-security-users
>> >> ModSecurity Services from Trustwave's SpiderLabs:
>> >> https://www.trustwave.com/application-security.php
>> >>
>> >>
>> >
>
>

------------------------------------------------------------------------------
Doing More with Less: The Next Generation Virtual Desktop 
What are the key obstacles that have prevented many mid-market businesses
from deploying virtual desktops?   How do next-generation virtual desktops
provide companies an easier-to-deploy, easier-to-manage and more affordable
virtual desktop model.http://www.accelacomm.com/jaw/sfnl/114/51426474/
_______________________________________________
mod-security-users mailing list
mod-security-users <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/mod-security-users
ModSecurity Services from Trustwave's SpiderLabs:
https://www.trustwave.com/application-security.php

Breno Silva | 8 Sep 16:58 2011
Picon

Re: Failed to access DBM file (data/ip): Resource deadlock avoided

Yes.. i will need! I will contact you.
Thanks

Breno

On Thu, Sep 8, 2011 at 9:57 AM, Michael Haas <michael.haas10 <at> gmail.com> wrote:
Hi,

that's great, if you need assistance in testing some new versions or
the old, i can help with this.

Thanks
Michael

2011/9/8 Breno Silva <breno.silva <at> gmail.com>:
> Hey Michel,
>
> I'm going to change it for 2.6.3 version. So i can investigate it sooner.
>
> Thanks
>
> Breno
>
> On Thu, Sep 8, 2011 at 9:40 AM, Michael Haas <michael.haas10 <at> gmail.com>
> wrote:
>>
>> Found also this bug
>> https://www.modsecurity.org/tracker/browse/MODSEC-160 seems that i
>> need to disable the session rules because the file size is only
>> increasing.
>> Is there a schedule for the availability of 2.7.0.
>>
>> Thanks
>> Michael
>>
>> 2011/9/8 Michael Haas <michael.haas10 <at> gmail.com>:
>> > Hello,
>> >
>> > the size is also a problem for the session collection, i need for the
>> > persistence of 15000 sessions around 1GB Storage (around 64k per
>> > session).
>> > Is it possible to reduce this or is this really needed?
>> > I use a modified version of modsecurity_crs_16_session_hijacking.conf
>> > from crs 2.2.1 where i also save the data to a backend and there the
>> > size is not that big.
>> >
>> > Thanks in advance
>> > Michael
>> >
>> >
>> > 2011/9/7 Breno Silva <breno.silva <at> gmail.com>:
>> >> Hi Kevin,
>> >>
>> >> We are planning to implement  a DDoS learning algorithm for
>> >> ModSecurity. I
>> >> know using data collection in heavy systems can cause performance
>> >> issues.
>> >> Most of Apache modules which works for  bandwidth limitation also
>> >> stores
>> >> data into temporary files, and it will also cause performance problems.
>> >> So
>> >> we need some code to store network information only in memory, persist
>> >> only
>> >> the learned data for recovery. It will also requires a time-to-time
>> >> cleanup
>> >> of internal modsecurity data structs. So i think in the future we will
>> >> have
>> >> a good solution.
>> >>
>> >> Until we do not have a better solution, maybe you could try limit per
>> >> ip
>> >> connections in a another network device (firewall, sw) ?
>> >>
>> >> Thanks
>> >>
>> >> Breno
>> >>
>> >> On Wed, Sep 7, 2011 at 10:42 AM, Kevin Jackson
>> >> <kevin <at> linuxservices.co.uk>
>> >> wrote:
>> >>>
>> >>> Dear all,
>> >>> It's been a while since I posted this issue and I want to share with
>> >>> you some of my experiences as I'm seeing issues when the IP collection
>> >>> grows in size.
>> >>>
>> >>> My setup is Apache 2.2.20, ModSecurity 2.6.1 and CRS 2.2.1.
>> >>> After a lot of performance testing and configuring ModSecurity to do 1
>> >>> job (protect against a DoS against the web servers [which in turn
>> >>> would protect against the real assets that ModSecurity setup is
>> >>> proxying for]) on a "2CPU", 7Gb box (in EC2, m1.large) I get somewhere
>> >>> between 200 and 300 hits per second against a static resource served
>> >>> from Apache.  Not great, but acceptable.  What isn't acceptable is the
>> >>> wildly fluctuating figures I get and load hitting 200+ when the IP
>> >>> data collection grows in size.
>> >>>
>> >>> My performance tests are replaying existing logs with real IP and URIs
>> >>> in, but I use the IP to set an X-Forwarded-For header (so ModSec sees
>> >>> it as a new IP), it uses the hash of the UA to give a level of
>> >>> uniqueness to the IP as per the CRS configs and I rewrite all URLs to
>> >>> a static file (to ensure I'm not running stuff against live in any
>> >>> shape or form).
>> >>>
>> >>> All is good at the beginning of the run and my IP collection database
>> >>> is at 0Mb.  I get around 300 hits per second at a load avg of around
>> >>> 0.4.  Over a short period of time (at the rate mentioned above), this
>> >>> IP collection builds up and when it hits a certain size, its game
>> >>> over.  High load starts around 5-10 minutes.  Leaving this running for
>> >>> 20 mins sees load shoot up to 200+.  This isn't a gradual increase, it
>> >>> hits a point and fails.  This seems to be file size as interrogating
>> >>> the SDBM file lists for the IP collections it is a small figure.  For
>> >>> example, I have a 700Mb SDBM file and inside are only 28 IP
>> >>> collections.
>> >>>
>> >>> I've played with TIMEOUT values to see if keeping less is helpful and
>> >>> won't affect things too much, but ideally for DoS protection I'd hope
>> >>> that keeping the IP in the database for as long as possible will be
>> >>> more beneficial.
>> >>>
>> >>> Any help is appreciated - as a potential mitigation step against a
>> >>> Layer 7 DDoS (network and other good stuff in place to protect against
>> >>> this aside) - what can I do from here?
>> >>>
>> >>> Cheers,
>> >>>
>> >>> Kev
>> >>>
>> >>> On 19 August 2011 13:37, Ryan Barnett <RBarnett <at> trustwave.com> wrote:
>> >>> >
>> >>> > On 8/19/11 6:55 AM, "Kevin Jackson" <kevin <at> linuxservices.co.uk>
>> >>> > wrote:
>> >>> >
>> >>> >>Dear all,
>> >>> >>After a number of weeks fine tuning the CRS rules appropriate for a
>> >>> >>busy website, we did a full live test pushing traffic through Apache
>> >>> >>with ModSecurity 2.6.1 proxying requests to evaluate the performance
>> >>> >>of ModSecurity through the backend service.
>> >>> >>
>> >>> >>Traffic is around 1k/second and we had Apache running on 18 servers
>> >>> >>sitting behind a load balancer and Apache coping without issue.
>> >>> >>
>> >>> >>ModSecurity on the other hand didn't fare so well - it struggled to
>> >>> >>keep pace causing resource deadlock issues on the ip database.  The
>> >>> >>upshot was incredibly high load averages for obvious reasons and
>> >>> >> poor
>> >>> >>user experience.
>> >>> >>
>> >>> >>--ba66410d-H--
>> >>> >>Message: Failed to access DBM file "/usr/local/apache/data/ip":
>> >>> >>Resource deadlock avoided
>> >>> >>Apache-Handler: proxy-server
>> >>> >>Stopwatch: 1313660329409820 283469 (- - -)
>> >>> >>Stopwatch2: 1313660329409820 283469; combined=99526, p1=3446,
>> >>> >>p2=12838, p3=3, p4=0, p5=41666, sr=3272, sw=41573, l=0, gc=0
>> >>> >>Producer: ModSecurity for Apache/2.6.1
>> >>> >> (http://www.modsecurity.org/);
>> >>> >>core ruleset/2.2.1.
>> >>> >>Server: Apache
>> >>> >>
>> >>> >>--ba66410d-Z--
>> >>> >>
>> >>> >>My question is what to suggest here? People must be running
>> >>> >>ModSecurity on high volume websites? Have you encountered this
>> >>> >> issue?
>> >>> >>This traffic is a fraction of our peak traffic at a quiet time
>> >>> >> during
>> >>> >>the day and the environment only coped for about 10 minutes before
>> >>> >>keeling over.
>> >>> >
>> >>> > Hey Kevin,
>> >>> > Question for you - are you using the IP persistent collection to
>> >>> > track
>> >>> > data about clients across requests?  If not, then you may want to
>> >>> > disable
>> >>> > the initcol rules at the end of the modsecurity_crs_10_config.conf
>> >>> > file.
>> >>> >
>> >>> > See if that helps.
>> >>> >
>> >>> > -Ryan
>> >>> >
>> >>> >>
>> >>> >>Cheers,
>> >>> >>
>> >>> >>Kev
>> >>> >>
>> >>>
>> >>> >>
>> >>> >> >> >>--------------------------------------------------------------------------
>> >>> >>----
>> >>> >>Get a FREE DOWNLOAD! and learn more about uberSVN rich system,
>> >>> >>user administration capabilities and model configuration. Take
>> >>> >>the hassle out of deploying and managing Subversion and the
>> >>> >>tools developers use with it. http://p.sf.net/sfu/wandisco-d2d-2
>> >>> >>_______________________________________________
>> >>> >>mod-security-users mailing list
>> >>> >>mod-security-users <at> lists.sourceforge.net
>> >>> >>https://lists.sourceforge.net/lists/listinfo/mod-security-users
>> >>> >>ModSecurity Services from Trustwave's SpiderLabs:
>> >>> >>https://www.trustwave.com/application-security.php
>> >>> >>
>> >>> >
>> >>> >
>> >>> > This transmission may contain information that is privileged,
>> >>> > confidential, and/or exempt from disclosure under applicable law. If
>> >>> > you are
>> >>> > not the intended recipient, you are hereby notified that any
>> >>> > disclosure,
>> >>> > copying, distribution, or use of the information contained herein
>> >>> > (including
>> >>> > any reliance thereon) is STRICTLY PROHIBITED. If you received this
>> >>> > transmission in error, please immediately contact the sender and
>> >>> > destroy the
>> >>> > material in its entirety, whether in electronic or hard copy format.
>> >>> >
>> >>> >
>> >>>
>> >>>
>> >>>
>> >>> ------------------------------------------------------------------------------
>> >>> Using storage to extend the benefits of virtualization and iSCSI
>> >>> Virtualization increases hardware utilization and delivers a new level
>> >>> of
>> >>> agility. Learn what those decisions are and how to modernize your
>> >>> storage
>> >>> and backup environments for virtualization.
>> >>> http://www.accelacomm.com/jaw/sfnl/114/51434361/
>> >>> _______________________________________________
>> >>> mod-security-users mailing list
>> >>> mod-security-users <at> lists.sourceforge.net
>> >>> https://lists.sourceforge.net/lists/listinfo/mod-security-users
>> >>> ModSecurity Services from Trustwave's SpiderLabs:
>> >>> https://www.trustwave.com/application-security.php
>> >>
>> >>
>> >>
>> >> ------------------------------------------------------------------------------
>> >> Using storage to extend the benefits of virtualization and iSCSI
>> >> Virtualization increases hardware utilization and delivers a new level
>> >> of
>> >> agility. Learn what those decisions are and how to modernize your
>> >> storage
>> >> and backup environments for virtualization.
>> >> http://www.accelacomm.com/jaw/sfnl/114/51434361/
>> >> _______________________________________________
>> >> mod-security-users mailing list
>> >> mod-security-users <at> lists.sourceforge.net
>> >> https://lists.sourceforge.net/lists/listinfo/mod-security-users
>> >> ModSecurity Services from Trustwave's SpiderLabs:
>> >> https://www.trustwave.com/application-security.php
>> >>
>> >>
>> >
>
>

------------------------------------------------------------------------------
Doing More with Less: The Next Generation Virtual Desktop 
What are the key obstacles that have prevented many mid-market businesses
from deploying virtual desktops?   How do next-generation virtual desktops
provide companies an easier-to-deploy, easier-to-manage and more affordable
virtual desktop model.http://www.accelacomm.com/jaw/sfnl/114/51426474/
_______________________________________________
mod-security-users mailing list
mod-security-users <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/mod-security-users
ModSecurity Services from Trustwave's SpiderLabs:
https://www.trustwave.com/application-security.php
Kevin Jackson | 9 Sep 10:21 2011
Picon

Re: Failed to access DBM file (data/ip): Resource deadlock avoided

Hi Breno,
I'll help with this too.  At the moment I see this as a blocker to put
this into production for us.

Regards,

Kev

On 8 September 2011 15:58, Breno Silva <breno.silva <at> gmail.com> wrote:
> Yes.. i will need! I will contact you.
> Thanks
>
> Breno
>
> On Thu, Sep 8, 2011 at 9:57 AM, Michael Haas <michael.haas10 <at> gmail.com>
> wrote:
>>
>> Hi,
>>
>> that's great, if you need assistance in testing some new versions or
>> the old, i can help with this.
>>
>> Thanks
>> Michael
>>
>> 2011/9/8 Breno Silva <breno.silva <at> gmail.com>:
>> > Hey Michel,
>> >
>> > I'm going to change it for 2.6.3 version. So i can investigate it
>> > sooner.
>> >
>> > Thanks
>> >
>> > Breno
>> >
>> > On Thu, Sep 8, 2011 at 9:40 AM, Michael Haas <michael.haas10 <at> gmail.com>
>> > wrote:
>> >>
>> >> Found also this bug
>> >> https://www.modsecurity.org/tracker/browse/MODSEC-160 seems that i
>> >> need to disable the session rules because the file size is only
>> >> increasing.
>> >> Is there a schedule for the availability of 2.7.0.
>> >>
>> >> Thanks
>> >> Michael
>> >>
>> >> 2011/9/8 Michael Haas <michael.haas10 <at> gmail.com>:
>> >> > Hello,
>> >> >
>> >> > the size is also a problem for the session collection, i need for the
>> >> > persistence of 15000 sessions around 1GB Storage (around 64k per
>> >> > session).
>> >> > Is it possible to reduce this or is this really needed?
>> >> > I use a modified version of modsecurity_crs_16_session_hijacking.conf
>> >> > from crs 2.2.1 where i also save the data to a backend and there the
>> >> > size is not that big.
>> >> >
>> >> > Thanks in advance
>> >> > Michael
>> >> >
>> >> >
>> >> > 2011/9/7 Breno Silva <breno.silva <at> gmail.com>:
>> >> >> Hi Kevin,
>> >> >>
>> >> >> We are planning to implement  a DDoS learning algorithm for
>> >> >> ModSecurity. I
>> >> >> know using data collection in heavy systems can cause performance
>> >> >> issues.
>> >> >> Most of Apache modules which works for  bandwidth limitation also
>> >> >> stores
>> >> >> data into temporary files, and it will also cause performance
>> >> >> problems.
>> >> >> So
>> >> >> we need some code to store network information only in memory,
>> >> >> persist
>> >> >> only
>> >> >> the learned data for recovery. It will also requires a time-to-time
>> >> >> cleanup
>> >> >> of internal modsecurity data structs. So i think in the future we
>> >> >> will
>> >> >> have
>> >> >> a good solution.
>> >> >>
>> >> >> Until we do not have a better solution, maybe you could try limit
>> >> >> per
>> >> >> ip
>> >> >> connections in a another network device (firewall, sw) ?
>> >> >>
>> >> >> Thanks
>> >> >>
>> >> >> Breno
>> >> >>
>> >> >> On Wed, Sep 7, 2011 at 10:42 AM, Kevin Jackson
>> >> >> <kevin <at> linuxservices.co.uk>
>> >> >> wrote:
>> >> >>>
>> >> >>> Dear all,
>> >> >>> It's been a while since I posted this issue and I want to share
>> >> >>> with
>> >> >>> you some of my experiences as I'm seeing issues when the IP
>> >> >>> collection
>> >> >>> grows in size.
>> >> >>>
>> >> >>> My setup is Apache 2.2.20, ModSecurity 2.6.1 and CRS 2.2.1.
>> >> >>> After a lot of performance testing and configuring ModSecurity to
>> >> >>> do 1
>> >> >>> job (protect against a DoS against the web servers [which in turn
>> >> >>> would protect against the real assets that ModSecurity setup is
>> >> >>> proxying for]) on a "2CPU", 7Gb box (in EC2, m1.large) I get
>> >> >>> somewhere
>> >> >>> between 200 and 300 hits per second against a static resource
>> >> >>> served
>> >> >>> from Apache.  Not great, but acceptable.  What isn't acceptable is
>> >> >>> the
>> >> >>> wildly fluctuating figures I get and load hitting 200+ when the IP
>> >> >>> data collection grows in size.
>> >> >>>
>> >> >>> My performance tests are replaying existing logs with real IP and
>> >> >>> URIs
>> >> >>> in, but I use the IP to set an X-Forwarded-For header (so ModSec
>> >> >>> sees
>> >> >>> it as a new IP), it uses the hash of the UA to give a level of
>> >> >>> uniqueness to the IP as per the CRS configs and I rewrite all URLs
>> >> >>> to
>> >> >>> a static file (to ensure I'm not running stuff against live in any
>> >> >>> shape or form).
>> >> >>>
>> >> >>> All is good at the beginning of the run and my IP collection
>> >> >>> database
>> >> >>> is at 0Mb.  I get around 300 hits per second at a load avg of
>> >> >>> around
>> >> >>> 0.4.  Over a short period of time (at the rate mentioned above),
>> >> >>> this
>> >> >>> IP collection builds up and when it hits a certain size, its game
>> >> >>> over.  High load starts around 5-10 minutes.  Leaving this running
>> >> >>> for
>> >> >>> 20 mins sees load shoot up to 200+.  This isn't a gradual increase,
>> >> >>> it
>> >> >>> hits a point and fails.  This seems to be file size as
>> >> >>> interrogating
>> >> >>> the SDBM file lists for the IP collections it is a small figure.
>> >> >>>  For
>> >> >>> example, I have a 700Mb SDBM file and inside are only 28 IP
>> >> >>> collections.
>> >> >>>
>> >> >>> I've played with TIMEOUT values to see if keeping less is helpful
>> >> >>> and
>> >> >>> won't affect things too much, but ideally for DoS protection I'd
>> >> >>> hope
>> >> >>> that keeping the IP in the database for as long as possible will be
>> >> >>> more beneficial.
>> >> >>>
>> >> >>> Any help is appreciated - as a potential mitigation step against a
>> >> >>> Layer 7 DDoS (network and other good stuff in place to protect
>> >> >>> against
>> >> >>> this aside) - what can I do from here?
>> >> >>>
>> >> >>> Cheers,
>> >> >>>
>> >> >>> Kev
>> >> >>>
>> >> >>> On 19 August 2011 13:37, Ryan Barnett <RBarnett <at> trustwave.com>
>> >> >>> wrote:
>> >> >>> >
>> >> >>> > On 8/19/11 6:55 AM, "Kevin Jackson" <kevin <at> linuxservices.co.uk>
>> >> >>> > wrote:
>> >> >>> >
>> >> >>> >>Dear all,
>> >> >>> >>After a number of weeks fine tuning the CRS rules appropriate for
>> >> >>> >> a
>> >> >>> >>busy website, we did a full live test pushing traffic through
>> >> >>> >> Apache
>> >> >>> >>with ModSecurity 2.6.1 proxying requests to evaluate the
>> >> >>> >> performance
>> >> >>> >>of ModSecurity through the backend service.
>> >> >>> >>
>> >> >>> >>Traffic is around 1k/second and we had Apache running on 18
>> >> >>> >> servers
>> >> >>> >>sitting behind a load balancer and Apache coping without issue.
>> >> >>> >>
>> >> >>> >>ModSecurity on the other hand didn't fare so well - it struggled
>> >> >>> >> to
>> >> >>> >>keep pace causing resource deadlock issues on the ip database.
>> >> >>> >>  The
>> >> >>> >>upshot was incredibly high load averages for obvious reasons and
>> >> >>> >> poor
>> >> >>> >>user experience.
>> >> >>> >>
>> >> >>> >>--ba66410d-H--
>> >> >>> >>Message: Failed to access DBM file "/usr/local/apache/data/ip":
>> >> >>> >>Resource deadlock avoided
>> >> >>> >>Apache-Handler: proxy-server
>> >> >>> >>Stopwatch: 1313660329409820 283469 (- - -)
>> >> >>> >>Stopwatch2: 1313660329409820 283469; combined=99526, p1=3446,
>> >> >>> >>p2=12838, p3=3, p4=0, p5=41666, sr=3272, sw=41573, l=0, gc=0
>> >> >>> >>Producer: ModSecurity for Apache/2.6.1
>> >> >>> >> (http://www.modsecurity.org/);
>> >> >>> >>core ruleset/2.2.1.
>> >> >>> >>Server: Apache
>> >> >>> >>
>> >> >>> >>--ba66410d-Z--
>> >> >>> >>
>> >> >>> >>My question is what to suggest here? People must be running
>> >> >>> >>ModSecurity on high volume websites? Have you encountered this
>> >> >>> >> issue?
>> >> >>> >>This traffic is a fraction of our peak traffic at a quiet time
>> >> >>> >> during
>> >> >>> >>the day and the environment only coped for about 10 minutes
>> >> >>> >> before
>> >> >>> >>keeling over.
>> >> >>> >
>> >> >>> > Hey Kevin,
>> >> >>> > Question for you - are you using the IP persistent collection to
>> >> >>> > track
>> >> >>> > data about clients across requests?  If not, then you may want to
>> >> >>> > disable
>> >> >>> > the initcol rules at the end of the
>> >> >>> > modsecurity_crs_10_config.conf
>> >> >>> > file.
>> >> >>> >
>> >> >>> > See if that helps.
>> >> >>> >
>> >> >>> > -Ryan
>> >> >>> >
>> >> >>> >>
>> >> >>> >>Cheers,
>> >> >>> >>
>> >> >>> >>Kev
>> >> >>> >>
>> >> >>>
>> >> >>> >>
>> >> >>> >> >>
>> >> >>> >> >> >> >>--------------------------------------------------------------------------
>> >> >>> >>----
>> >> >>> >>Get a FREE DOWNLOAD! and learn more about uberSVN rich system,
>> >> >>> >>user administration capabilities and model configuration. Take
>> >> >>> >>the hassle out of deploying and managing Subversion and the
>> >> >>> >>tools developers use with it. http://p.sf.net/sfu/wandisco-d2d-2
>> >> >>> >>_______________________________________________
>> >> >>> >>mod-security-users mailing list
>> >> >>> >>mod-security-users <at> lists.sourceforge.net
>> >> >>> >>https://lists.sourceforge.net/lists/listinfo/mod-security-users
>> >> >>> >>ModSecurity Services from Trustwave's SpiderLabs:
>> >> >>> >>https://www.trustwave.com/application-security.php
>> >> >>> >>
>> >> >>> >
>> >> >>> >
>> >> >>> > This transmission may contain information that is privileged,
>> >> >>> > confidential, and/or exempt from disclosure under applicable law.
>> >> >>> > If
>> >> >>> > you are
>> >> >>> > not the intended recipient, you are hereby notified that any
>> >> >>> > disclosure,
>> >> >>> > copying, distribution, or use of the information contained herein
>> >> >>> > (including
>> >> >>> > any reliance thereon) is STRICTLY PROHIBITED. If you received
>> >> >>> > this
>> >> >>> > transmission in error, please immediately contact the sender and
>> >> >>> > destroy the
>> >> >>> > material in its entirety, whether in electronic or hard copy
>> >> >>> > format.
>> >> >>> >
>> >> >>> >
>> >> >>>
>> >> >>>
>> >> >>>
>> >> >>>
>> >> >>> ------------------------------------------------------------------------------
>> >> >>> Using storage to extend the benefits of virtualization and iSCSI
>> >> >>> Virtualization increases hardware utilization and delivers a new
>> >> >>> level
>> >> >>> of
>> >> >>> agility. Learn what those decisions are and how to modernize your
>> >> >>> storage
>> >> >>> and backup environments for virtualization.
>> >> >>> http://www.accelacomm.com/jaw/sfnl/114/51434361/
>> >> >>> _______________________________________________
>> >> >>> mod-security-users mailing list
>> >> >>> mod-security-users <at> lists.sourceforge.net
>> >> >>> https://lists.sourceforge.net/lists/listinfo/mod-security-users
>> >> >>> ModSecurity Services from Trustwave's SpiderLabs:
>> >> >>> https://www.trustwave.com/application-security.php
>> >> >>
>> >> >>
>> >> >>
>> >> >>
>> >> >> ------------------------------------------------------------------------------
>> >> >> Using storage to extend the benefits of virtualization and iSCSI
>> >> >> Virtualization increases hardware utilization and delivers a new
>> >> >> level
>> >> >> of
>> >> >> agility. Learn what those decisions are and how to modernize your
>> >> >> storage
>> >> >> and backup environments for virtualization.
>> >> >> http://www.accelacomm.com/jaw/sfnl/114/51434361/
>> >> >> _______________________________________________
>> >> >> mod-security-users mailing list
>> >> >> mod-security-users <at> lists.sourceforge.net
>> >> >> https://lists.sourceforge.net/lists/listinfo/mod-security-users
>> >> >> ModSecurity Services from Trustwave's SpiderLabs:
>> >> >> https://www.trustwave.com/application-security.php
>> >> >>
>> >> >>
>> >> >
>> >
>> >
>
>

------------------------------------------------------------------------------
Why Cloud-Based Security and Archiving Make Sense
Osterman Research conducted this study that outlines how and why cloud
computing security and archiving is rapidly being adopted across the IT 
space for its ease of implementation, lower cost, and increased 
reliability. Learn more. http://www.accelacomm.com/jaw/sfnl/114/51425301/
_______________________________________________
mod-security-users mailing list
mod-security-users <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/mod-security-users
ModSecurity Services from Trustwave's SpiderLabs:
https://www.trustwave.com/application-security.php


Gmane