Ivan Ristic | 17 Aug 12:57 2006
Picon

Performance tip

I thought you might find this interesting:
http://www.modsecurity.org/blog/archives/2006/08/modsecurity_per.html

--

-- 
Ivan Ristic, Technical Director
Thinking Stone, http://www.thinkingstone.com
ModSecurity: Open source Web Application Firewall

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
Ryan Barnett | 17 Aug 13:26 2006
Picon

Re: Performance tip

Interesting.  I have been wondering the same thing with regards to overall performance (clean regular expressions + number of total signatures).  A few road blocks that I see preventing more people from trying to consolidate their sigs like this are -
 
1) Readibility - sometimes is becomes hard to read the regexp string (and it associated meaning) when you combine many different rules into on.  For instance, those users who use the snort2modsec.pl script usually like to see the Snort message info for the vuln to clearly understand what this sigs is looking for.
 
2) Signature IDs - this may cause problems with tracking signatures that trigger.  Example - I have associated unique sig IDs with every filter rule.  This way, when I read the mod_security-message info the gets sent to me in email when a rule triggers, I can quick do a search for the sig ID in my rule conf file for the rule that triggered.
 
Then again, I am sure that there is some middle ground here where easy/simple rules could be combined together to help reduce the overhead.
 
--
Ryan C. Barnett
Web Application Security Consortium (WASC) Member
CIS Apache Benchmark Project Lead
SANS Instructor, GCIA, GCFA, GCIH, GSNA, GCUX, GSEC
Author: Preventing Web Attacks with Apache

 
On 8/17/06, Ivan Ristic <ivan.ristic <at> gmail.com> wrote:
I thought you might find this interesting:
http://www.modsecurity.org/blog/archives/2006/08/modsecurity_per.html

--
Ivan Ristic, Technical Director
Thinking Stone, http://www.thinkingstone.com
ModSecurity: Open source Web Application Firewall

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
mod-security-users mailing list
mod-security-users <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/mod-security-users
-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
mod-security-users mailing list
mod-security-users <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/mod-security-users
Alex V. | 17 Aug 13:54 2006
Picon

Re: Performance tip

I'm absolutely not a regexp expert, but I'm wondering if it could not be
possible to write the regexp like this (or something similar):

SecFilterSelective VAR (
      KEYWORD1|     # Comment explaining this match
      KEYWORD2|     # Comment explaining this match
      KEYWORD3|     # Comment explaining this match
      KEYWORD4|     # Comment explaining this match
      KEYWORD5|     # Comment explaining this match
      KEYWORD6     # Comment explaining this match
)

This way, it could possibly lead to something readable, comprehensive and
optimized.

To sum up, the problem is : Is it possible to put into the regexp some
comments ? And if not, maybe someone could try (I'll try to do it if I've
some times) to develop a script to convert a file with such rules well
commented to a file for use in modsec... Then, people just have to read
and edit this commented file and then launch the script => No more problem
with readability.

Cheers,

Alex

On Jeu 17 août 2006 13:26, Ryan Barnett a écrit :
> Interesting.  I have been wondering the same thing with regards to overall
> performance (clean regular expressions + number of total signatures).  A
> few
> road blocks that I see preventing more people from trying to consolidate
> their sigs like this are -
>
> 1) Readibility - sometimes is becomes hard to read the regexp string (and
> it
> associated meaning) when you combine many different rules into on.  For
> instance, those users who use the snort2modsec.pl script usually like to
> see
> the Snort message info for the vuln to clearly understand what this sigs
> is
> looking for.
>
> 2) Signature IDs - this may cause problems with tracking signatures that
> trigger.  Example - I have associated unique sig IDs with every filter
> rule.  This way, when I read the mod_security-message info the gets sent
> to
> me in email when a rule triggers, I can quick do a search for the sig ID
> in
> my rule conf file for the rule that triggered.
>
> Then again, I am sure that there is some middle ground here where
> easy/simple rules could be combined together to help reduce the overhead.
>
> --
> Ryan C. Barnett
> Web Application Security Consortium (WASC) Member
> CIS Apache Benchmark Project Lead
> SANS Instructor, GCIA, GCFA, GCIH, GSNA, GCUX, GSEC
> Author: Preventing Web Attacks with Apache
>
>
> On 8/17/06, Ivan Ristic <ivan.ristic <at> gmail.com> wrote:
>>
>> I thought you might find this interesting:
>> http://www.modsecurity.org/blog/archives/2006/08/modsecurity_per.html
>>
>> --
>> Ivan Ristic, Technical Director
>> Thinking Stone, http://www.thinkingstone.com
>> ModSecurity: Open source Web Application Firewall
>>
>> -------------------------------------------------------------------------
>> Using Tomcat but need to do more? Need to support web services,
>> security?
>> Get stuff done quickly with pre-integrated technology to make your job
>> easier
>> Download IBM WebSphere Application Server v.1.0.1 based on Apache
>> Geronimo
>> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
>> _______________________________________________
>> mod-security-users mailing list
>> mod-security-users <at> lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/mod-security-users
>>
> -------------------------------------------------------------------------
> Using Tomcat but need to do more? Need to support web services, security?
> Get stuff done quickly with pre-integrated technology to make your job
> easier
> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642_______________________________________________
> mod-security-users mailing list
> mod-security-users <at> lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/mod-security-users
>

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
Ivan Ristic | 17 Aug 14:34 2006
Picon

Re: Performance tip

On 8/17/06, Alex V. <alex-security <at> ssji.net> wrote:
> I'm absolutely not a regexp expert, but I'm wondering if it could not be
> possible to write the regexp like this (or something similar):
>
> SecFilterSelective VAR (
>       KEYWORD1|     # Comment explaining this match
>       KEYWORD2|     # Comment explaining this match
>       KEYWORD3|     # Comment explaining this match
>       KEYWORD4|     # Comment explaining this match
>       KEYWORD5|     # Comment explaining this match
>       KEYWORD6     # Comment explaining this match
> )

It is not possible to use comments, but it is possible to break the
regular expressions into multiple lines. like this.

SecFilterSelective VAR "(\
KEYWORD1|\
KEYWORD2|\
KEYWORD3)"

It is important to not have any whitespace at the beginning of the
line or before the continuation character "\".

--

-- 
Ivan Ristic, Technical Director
Thinking Stone, http://www.thinkingstone.com
ModSecurity: Open source Web Application Firewall

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
Brian Rectanus | 17 Aug 14:59 2006
Picon

Re: Performance tip

On 8/17/06, Ivan Ristic <ivan.ristic <at> gmail.com> wrote:
> On 8/17/06, Alex V. <alex-security <at> ssji.net> wrote:
> > I'm absolutely not a regexp expert, but I'm wondering if it could not be
> > possible to write the regexp like this (or something similar):
> >
> > SecFilterSelective VAR (
> >       KEYWORD1|     # Comment explaining this match
> >       KEYWORD2|     # Comment explaining this match
> >       KEYWORD3|     # Comment explaining this match
> >       KEYWORD4|     # Comment explaining this match
> >       KEYWORD5|     # Comment explaining this match
> >       KEYWORD6     # Comment explaining this match
> > )
>
> It is not possible to use comments, but it is possible to break the
> regular expressions into multiple lines. like this.
>
> SecFilterSelective VAR "(\
> KEYWORD1|\
> KEYWORD2|\
> KEYWORD3)"
>
> It is important to not have any whitespace at the beginning of the
> line or before the continuation character "\".

Does Apache2 support the 'x' option to ignore whitespace and support
comments?  If so, you should be able to do this (but I have no time
right now to try it):

SecFilterSelective VAR "(?x:
  KEYWORD1|   #comment
  KEYWORD2|   #comment
  KEYWORD3    #comment
)"

-B

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
Ivan Ristic | 17 Aug 15:21 2006
Picon

Re: Performance tip

> Does Apache2 support the 'x' option to ignore whitespace and support
> comments?  If so, you should be able to do this (but I have no time
> right now to try it):
>
> SecFilterSelective VAR "(?x:
>   KEYWORD1|   #comment
>   KEYWORD2|   #comment
>   KEYWORD3    #comment
> )"

It does (if you write "(?x)" to enable extended regular expressions),
but there is a problem with that. Apache configuration directives can
only consume a single line. When you use the continuation character to
spread them across several lines Apache simply concatenates the lines
together. The first time you use a comment like that you will end the
regular expression. In the case above, the regex will be invalid
because of the missing closing parenthesis.

--

-- 
Ivan Ristic, Technical Director
Thinking Stone, http://www.thinkingstone.com
ModSecurity: Open source Web Application Firewall

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
Ryan Barnett | 17 Aug 15:04 2006
Picon

Re: Performance tip

Doing some RegEx searching about comments...  Doesn't this syntax work for adding comments -  (?#comment)
 
So the updated RegEx code would look like this -
 
SecFilterSelective VAR "(\
KEYWORD1(?#signature comment1)|\
KEYWORD2(?#signature comment2)|\
KEYWORD3(?#signature comment3))"
 
Here is an example entry -
 
SecFilterSelective THE_REQUEST "(\
ps\x20(?#WEB-ATTACKS /bin/ps command attempt)|\
wget\x20(?#WEB-ATTACKS wget command attempt)|\
uname\x20-a(?#WEB-ATTACKS uname -a command attempt))"
 
I just ran a test and it worked.  Here is the audit_log entry -

========================================
Request: 192.168.1.102 - - [[17/Aug/2006:09:02:21 --0400]] "GET /wget%20http://www.test.com/test.zip HTTP/1.1" 404 0
Handler: cgi-script
----------------------------------------
GET /wget%20http://www.test.com/test.zip HTTP/1.1
Accept: */*
Accept-Language: en-us
Accept-Encoding: gzip, deflate
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0 ; Windows NT 5.1; SV1; .NET CLR 1.1.4322)
Host: 192.168.1.103
Connection: Keep-Alive
mod_security-message: Access denied with code 403. Pattern match "(ps\x20(?#WEB-ATTACKS /bin/ps command attempt)|wget\x20(?#WEB-ATTACKS wget command attempt)|uname\x20-a(?#WEB-ATTACKS uname -a command attempt))" at THE_REQUEST.
mod_security-action: 403

HTTP/1.1 (null)

--
Ryan C. Barnett
Web Application Security Consortium (WASC) Member
CIS Apache Benchmark Project Lead
SANS Instructor, GCIA, GCFA, GCIH, GSNA, GCUX, GSEC
Author: Preventing Web Attacks with Apache 

 
On 8/17/06, Ivan Ristic <ivan.ristic <at> gmail.com > wrote:
On 8/17/06, Alex V. < alex-security <at> ssji.net> wrote:
> I'm absolutely not a regexp expert, but I'm wondering if it could not be
> possible to write the regexp like this (or something similar):
>
> SecFilterSelective VAR (
>       KEYWORD1|     # Comment explaining this match
>       KEYWORD2|     # Comment explaining this match
>       KEYWORD3|     # Comment explaining this match
>       KEYWORD4|     # Comment explaining this match
>       KEYWORD5|     # Comment explaining this match
>       KEYWORD6     # Comment explaining this match
> )

It is not possible to use comments, but it is possible to break the
regular expressions into multiple lines. like this.

SecFilterSelective VAR "(\
KEYWORD1|\
KEYWORD2|\
KEYWORD3)"

It is important to not have any whitespace at the beginning of the
line or before the continuation character "\".

--
Ivan Ristic, Technical Director
Thinking Stone, http://www.thinkingstone.com
ModSecurity: Open source Web Application Firewall



-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
mod-security-users mailing list
mod-security-users <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/mod-security-users
Ivan Ristic | 17 Aug 15:10 2006
Picon

Re: Performance tip

On 8/17/06, Ryan Barnett <rcbarnett <at> gmail.com> wrote:
>
> Doing some RegEx searching about comments...  Doesn't this syntax work for
> adding comments -  (?#comment)
>
> ...
>
> I just ran a test and it worked.  Here is the audit_log entry -

It does, it's the PCRE comment syntax (and thus works when ModSecurity
is used with Apache 2.x, but not when it is used with Apache 1.x.)

--

-- 
Ivan Ristic, Technical Director
Thinking Stone, http://www.thinkingstone.com
ModSecurity: Open source Web Application Firewall

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
Ryan Barnett | 17 Aug 15:42 2006
Picon

Re: Performance tip

This is yet another reason to switch to the Apache 2.X version.  The most compelling reason is the OUTPUT filtering capability, but this is also a nice advantage.
 
--
Ryan C. Barnett
Web Application Security Consortium (WASC) Member
CIS Apache Benchmark Project Lead
SANS Instructor, GCIA, GCFA, GCIH, GSNA, GCUX, GSEC
Author: Preventing Web Attacks with Apache

 
On 8/17/06, Ivan Ristic <ivan.ristic <at> gmail.com > wrote:
On 8/17/06, Ryan Barnett < rcbarnett <at> gmail.com> wrote:
>
> Doing some RegEx searching about comments...  Doesn't this syntax work for
> adding comments -  (?#comment)
>
> ...
>
> I just ran a test and it worked.  Here is the audit_log entry -

It does, it's the PCRE comment syntax (and thus works when ModSecurity
is used with Apache 2.x, but not when it is used with Apache 1.x.)

--
Ivan Ristic, Technical Director
Thinking Stone, http://www.thinkingstone.com
ModSecurity: Open source Web Application Firewall



-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
mod-security-users mailing list
mod-security-users <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/mod-security-users
Ivan Ristic | 17 Aug 14:30 2006
Picon

Re: Performance tip

On 8/17/06, Ryan Barnett <rcbarnett <at> gmail.com> wrote:
>
> Interesting.  I have been wondering the same thing with regards to overall
> performance (clean regular expressions + number of total signatures).  A few
> road blocks that I see preventing more people from trying to consolidate
> their sigs like this are -
>
> 1) Readibility ...
> 2) Signature IDs ...

Those are valid points, thank you for bringing them up. I agree
completely. I should perhaps clarify the consolidation is a valid
approach for those that use ModSecurity for spam detection, where so
many keywords need to be used and IDs are not important.

--

-- 
Ivan Ristic, Technical Director
Thinking Stone, http://www.thinkingstone.com
ModSecurity: Open source Web Application Firewall

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
Tom Anderson | 17 Aug 16:22 2006

Re: Performance tip

Ivan Ristic wrote:
> I thought you might find this interesting:
> http://www.modsecurity.org/blog/archives/2006/08/modsecurity_per.html
> 
 > So instead of:
 >
 > SecFilterSelective VAR KEYWORD1
 > SecFilterSelective VAR KEYWORD2
 > SecFilterSelective VAR KEYWORD3
 >
 > from now on you want to write:
 >
 > SecFilterSelective VAR (KEYWORD1|KEYWORD2|KEYWORD3)

It is the regex engine which makes this more efficient.  It would be 
even faster if you wrote:

SecFilterSelective VAR KEYWORD(1|2|3)

And even faster still like this:

SecFilterSelective VAR KEYWORD(?:1|2|3)

It has to do with redundant code and memory allocation in the first 
instance, reduction of backtracking in the second, and saving memory in 
the third.

Regarding readability, as far as I'm concerned, the shorter the better, 
which is why condensed regexes are great.  If you cannot read regexes, 
that's what comments are for.  But let's get serious -- any sysadmin 
worth his salt needs to know how to read and write regular expressions. 
  To me, a regular expression is far more readable than a long English 
comment or a multi-line block of code.  If you have to translate it, do 
so in a comment, but you shouldn't really need subtitles to use a common 
system tool like regexes unless some particular pattern is really 
convoluted or tricky.

Tom

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
Matt Wrycraft | 17 Aug 23:34 2006
Picon

Mod-Security and php forums

Hi all, this is my first question, so please let me know if I've missed 
anything.

After a hacking attempt at another part of my site the sysadmins 
installed Mod Security and rules from 
http://www.gotroot.com/tiki-index.php?page=mod_security+rules
I now get a number of false positives when posting on the forums (which 
are very active and people are getting annoyed at the number of 403s). 
I've run google search and checked out the gotroot forum to no avail. 
Looking at the audit.log most of the denied posts relate to:

mod_security-message: Access denied with code 403. Pattern match 
"(insert[[:space:]]+into.+values|select.+from|bulk[[:space:]]+insert|union.+select)" 
at POST_PAYLOAD

I've tried loads of variations of posts on the forum and having multiple 
spaces or using "into" "select" and "from" in the right order is 
obviously what is causing me these problems. I tried commenting out the 
rule but Apache failed to start, a quick look suggested that it was part 
of a chain of rules and I didn't want to dig too deeply in case I broke 
it further, so I uncommented the rule and successfully restarted the 
webserver.
  I don't really know regex so am not in a position to completely 
re-write the rules, as I'm just as likely to make things worse, so I've 
reached the conclusion that I would just like to disable mod security 
for forum posts.

I've checked /etc/modsecurity/exclude.conf and there already seems to be 
relevant rules for other php-based forums ie

<LocationMatch "/index.php?name=PNphpBB2&file=posting&mode=reply.*">
SecFilterRemove 300013
</LocationMatch>

Now, I'm unsure what rule 300013 does. I've grepped for it and it seems 
to be commented out in the rules.conf..

Anyway, my forum uses the following POST URL, which i assume is the 
location match:
  /modules.php?name=Forums&file=posting&mode=newtopic&f=13
(with different information after 'posting')

So I tried putting that into exclude.conf  - adding a number of rules 
that I thought could be problems (and taking them from other rules 
excluded in the exclude.conf) as follows:

<Location "/modules.php?name=Forums&file=posting.*">
SecFilterRemove 300013
SecFilterRemove 300014
SecFilterRemove 300016
SecFilterRemove 380000
SecFilterRemove 360001
</Location>

This made no difference and I still suffer the false positives.

All I want to do is exclude mod security from checking forum posts (yes 
I know the risks of this, but I'm using recent code, with additional 
handling, and feel that mod security is causing more harm than good 
right now)

Can I for instance use a variation of:
SecFilterSelective REQUEST_URI "/.*/Merchant2/merchant\.mv.*" allow,nolog

??

Any help would be graciously accepted!

Many thanks in advance.

Matt

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
Tom Anderson | 17 Aug 23:53 2006

Re: Mod-Security and php forums

If you're not running an SQL database, then you can comment out the 
chain of rules dealing with SQL injection attacks.  Otherwise, only 
enable them on programs that deal directly with a database.  If your 
forums use a database, the best thing is to ensure there is sanitation 
of input within your forum software.  If so, you can disable the SQL 
injection filters in mod_security.  Otherwise, it might be prudent to 
keep it or to change forum software.

Tom

Matt Wrycraft wrote:
> Hi all, this is my first question, so please let me know if I've missed 
> anything.
> 
> After a hacking attempt at another part of my site the sysadmins 
> installed Mod Security and rules from 
> http://www.gotroot.com/tiki-index.php?page=mod_security+rules
> I now get a number of false positives when posting on the forums (which 
> are very active and people are getting annoyed at the number of 403s). 
> I've run google search and checked out the gotroot forum to no avail. 
> Looking at the audit.log most of the denied posts relate to:
> 
> mod_security-message: Access denied with code 403. Pattern match 
> "(insert[[:space:]]+into.+values|select.+from|bulk[[:space:]]+insert|union.+select)" 
> at POST_PAYLOAD
> 
> I've tried loads of variations of posts on the forum and having multiple 
> spaces or using "into" "select" and "from" in the right order is 
> obviously what is causing me these problems. I tried commenting out the 
> rule but Apache failed to start, a quick look suggested that it was part 
> of a chain of rules and I didn't want to dig too deeply in case I broke 
> it further, so I uncommented the rule and successfully restarted the 
> webserver.
>   I don't really know regex so am not in a position to completely 
> re-write the rules, as I'm just as likely to make things worse, so I've 
> reached the conclusion that I would just like to disable mod security 
> for forum posts.
> 
> I've checked /etc/modsecurity/exclude.conf and there already seems to be 
> relevant rules for other php-based forums ie
> 
> <LocationMatch "/index.php?name=PNphpBB2&file=posting&mode=reply.*">
> SecFilterRemove 300013
> </LocationMatch>
> 
> Now, I'm unsure what rule 300013 does. I've grepped for it and it seems 
> to be commented out in the rules.conf..
> 
> Anyway, my forum uses the following POST URL, which i assume is the 
> location match:
>   /modules.php?name=Forums&file=posting&mode=newtopic&f=13
> (with different information after 'posting')
> 
> So I tried putting that into exclude.conf  - adding a number of rules 
> that I thought could be problems (and taking them from other rules 
> excluded in the exclude.conf) as follows:
> 
> <Location "/modules.php?name=Forums&file=posting.*">
> SecFilterRemove 300013
> SecFilterRemove 300014
> SecFilterRemove 300016
> SecFilterRemove 380000
> SecFilterRemove 360001
> </Location>
> 
> This made no difference and I still suffer the false positives.
> 
> All I want to do is exclude mod security from checking forum posts (yes 
> I know the risks of this, but I'm using recent code, with additional 
> handling, and feel that mod security is causing more harm than good 
> right now)
> 
> Can I for instance use a variation of:
> SecFilterSelective REQUEST_URI "/.*/Merchant2/merchant\.mv.*" allow,nolog
> 
> ??
> 
> Any help would be graciously accepted!
> 
> Many thanks in advance.
> 
> Matt
> 
> -------------------------------------------------------------------------
> Using Tomcat but need to do more? Need to support web services, security?
> Get stuff done quickly with pre-integrated technology to make your job easier
> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
> _______________________________________________
> mod-security-users mailing list
> mod-security-users <at> lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/mod-security-users
> 
> 

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
Matt Wrycraft | 18 Aug 00:02 2006
Picon

Re: Mod-Security and php forums

Thanks Tom!

I am running SQL and can certainly understand the need to keep the rules 
dealing with injection attacks. SQL is used throughout my site and I 
would like to keep the rules generally, just exclude them from forum 
posts. I do have sanitation of forum posts anyway, which is why I'm 
happy to avoid using modsec there.

Unfortunately, changing the forum software isn't really an option for me 
at the moment, and I really don't want to anyway as in all other 
respects its working fine. It's just the modsec false positives that are 
giving me sleepless nights.

I just need the code to exclude it I guess.

Matt

Tom Anderson wrote:
> If you're not running an SQL database, then you can comment out the 
> chain of rules dealing with SQL injection attacks.  Otherwise, only 
> enable them on programs that deal directly with a database.  If your 
> forums use a database, the best thing is to ensure there is sanitation 
> of input within your forum software.  If so, you can disable the SQL 
> injection filters in mod_security.  Otherwise, it might be prudent to 
> keep it or to change forum software.
> 
> Tom
> 
> Matt Wrycraft wrote:
>> Hi all, this is my first question, so please let me know if I've missed 
>> anything.
>>
>> After a hacking attempt at another part of my site the sysadmins 
>> installed Mod Security and rules from 
>> http://www.gotroot.com/tiki-index.php?page=mod_security+rules
>> I now get a number of false positives when posting on the forums (which 
>> are very active and people are getting annoyed at the number of 403s). 
>> I've run google search and checked out the gotroot forum to no avail. 
>> Looking at the audit.log most of the denied posts relate to:
>>
>> mod_security-message: Access denied with code 403. Pattern match 
>> "(insert[[:space:]]+into.+values|select.+from|bulk[[:space:]]+insert|union.+select)" 
>> at POST_PAYLOAD
>>
>> I've tried loads of variations of posts on the forum and having multiple 
>> spaces or using "into" "select" and "from" in the right order is 
>> obviously what is causing me these problems. I tried commenting out the 
>> rule but Apache failed to start, a quick look suggested that it was part 
>> of a chain of rules and I didn't want to dig too deeply in case I broke 
>> it further, so I uncommented the rule and successfully restarted the 
>> webserver.
>>   I don't really know regex so am not in a position to completely 
>> re-write the rules, as I'm just as likely to make things worse, so I've 
>> reached the conclusion that I would just like to disable mod security 
>> for forum posts.
>>
>> I've checked /etc/modsecurity/exclude.conf and there already seems to be 
>> relevant rules for other php-based forums ie
>>
>> <LocationMatch "/index.php?name=PNphpBB2&file=posting&mode=reply.*">
>> SecFilterRemove 300013
>> </LocationMatch>
>>
>> Now, I'm unsure what rule 300013 does. I've grepped for it and it seems 
>> to be commented out in the rules.conf..
>>
>> Anyway, my forum uses the following POST URL, which i assume is the 
>> location match:
>>   /modules.php?name=Forums&file=posting&mode=newtopic&f=13
>> (with different information after 'posting')
>>
>> So I tried putting that into exclude.conf  - adding a number of rules 
>> that I thought could be problems (and taking them from other rules 
>> excluded in the exclude.conf) as follows:
>>
>> <Location "/modules.php?name=Forums&file=posting.*">
>> SecFilterRemove 300013
>> SecFilterRemove 300014
>> SecFilterRemove 300016
>> SecFilterRemove 380000
>> SecFilterRemove 360001
>> </Location>
>>
>> This made no difference and I still suffer the false positives.
>>
>> All I want to do is exclude mod security from checking forum posts (yes 
>> I know the risks of this, but I'm using recent code, with additional 
>> handling, and feel that mod security is causing more harm than good 
>> right now)
>>
>> Can I for instance use a variation of:
>> SecFilterSelective REQUEST_URI "/.*/Merchant2/merchant\.mv.*" allow,nolog
>>
>> ??
>>
>> Any help would be graciously accepted!
>>
>> Many thanks in advance.
>>
>> Matt
>>
>> -------------------------------------------------------------------------
>> Using Tomcat but need to do more? Need to support web services, security?
>> Get stuff done quickly with pre-integrated technology to make your job easier
>> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
>> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
>> _______________________________________________
>> mod-security-users mailing list
>> mod-security-users <at> lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/mod-security-users
>>
>>
> 
> 
> -------------------------------------------------------------------------
> Using Tomcat but need to do more? Need to support web services, security?
> Get stuff done quickly with pre-integrated technology to make your job easier
> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
> _______________________________________________
> mod-security-users mailing list
> mod-security-users <at> lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/mod-security-users
> 
> 

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
Michael Shinn | 18 Aug 18:30 2006

Re: Mod-Security and php forums

Matt, 

Send me your audit log entries (sanitize if you like, just leave me the
context of the way phpbb is being used), and I'll tweak the rules to
prevent these false positives. 

On Thu, 2006-08-17 at 23:02 +0100, Matt Wrycraft wrote:
> Thanks Tom!
> 
> I am running SQL and can certainly understand the need to keep the rules 
> dealing with injection attacks. SQL is used throughout my site and I 
> would like to keep the rules generally, just exclude them from forum 
> posts. I do have sanitation of forum posts anyway, which is why I'm 
> happy to avoid using modsec there.
> 
> Unfortunately, changing the forum software isn't really an option for me 
> at the moment, and I really don't want to anyway as in all other 
> respects its working fine. It's just the modsec false positives that are 
> giving me sleepless nights.
> 
> I just need the code to exclude it I guess.
> 
> Matt
> 
> Tom Anderson wrote:
> > If you're not running an SQL database, then you can comment out the 
> > chain of rules dealing with SQL injection attacks.  Otherwise, only 
> > enable them on programs that deal directly with a database.  If your 
> > forums use a database, the best thing is to ensure there is sanitation 
> > of input within your forum software.  If so, you can disable the SQL 
> > injection filters in mod_security.  Otherwise, it might be prudent to 
> > keep it or to change forum software.
> > 
> > Tom
> > 
> > Matt Wrycraft wrote:
> >> Hi all, this is my first question, so please let me know if I've missed 
> >> anything.
> >>
> >> After a hacking attempt at another part of my site the sysadmins 
> >> installed Mod Security and rules from 
> >> http://www.gotroot.com/tiki-index.php?page=mod_security+rules
> >> I now get a number of false positives when posting on the forums (which 
> >> are very active and people are getting annoyed at the number of 403s). 
> >> I've run google search and checked out the gotroot forum to no avail. 
> >> Looking at the audit.log most of the denied posts relate to:
> >>
> >> mod_security-message: Access denied with code 403. Pattern match 
> >> "(insert[[:space:]]+into.+values|select.+from|bulk[[:space:]]+insert|union.+select)" 
> >> at POST_PAYLOAD
> >>
> >> I've tried loads of variations of posts on the forum and having multiple 
> >> spaces or using "into" "select" and "from" in the right order is 
> >> obviously what is causing me these problems. I tried commenting out the 
> >> rule but Apache failed to start, a quick look suggested that it was part 
> >> of a chain of rules and I didn't want to dig too deeply in case I broke 
> >> it further, so I uncommented the rule and successfully restarted the 
> >> webserver.
> >>   I don't really know regex so am not in a position to completely 
> >> re-write the rules, as I'm just as likely to make things worse, so I've 
> >> reached the conclusion that I would just like to disable mod security 
> >> for forum posts.
> >>
> >> I've checked /etc/modsecurity/exclude.conf and there already seems to be 
> >> relevant rules for other php-based forums ie
> >>
> >> <LocationMatch "/index.php?name=PNphpBB2&file=posting&mode=reply.*">
> >> SecFilterRemove 300013
> >> </LocationMatch>
> >>
> >> Now, I'm unsure what rule 300013 does. I've grepped for it and it seems 
> >> to be commented out in the rules.conf..
> >>
> >> Anyway, my forum uses the following POST URL, which i assume is the 
> >> location match:
> >>   /modules.php?name=Forums&file=posting&mode=newtopic&f=13
> >> (with different information after 'posting')
> >>
> >> So I tried putting that into exclude.conf  - adding a number of rules 
> >> that I thought could be problems (and taking them from other rules 
> >> excluded in the exclude.conf) as follows:
> >>
> >> <Location "/modules.php?name=Forums&file=posting.*">
> >> SecFilterRemove 300013
> >> SecFilterRemove 300014
> >> SecFilterRemove 300016
> >> SecFilterRemove 380000
> >> SecFilterRemove 360001
> >> </Location>
> >>
> >> This made no difference and I still suffer the false positives.
> >>
> >> All I want to do is exclude mod security from checking forum posts (yes 
> >> I know the risks of this, but I'm using recent code, with additional 
> >> handling, and feel that mod security is causing more harm than good 
> >> right now)
> >>
> >> Can I for instance use a variation of:
> >> SecFilterSelective REQUEST_URI "/.*/Merchant2/merchant\.mv.*" allow,nolog
> >>
> >> ??
> >>
> >> Any help would be graciously accepted!
> >>
> >> Many thanks in advance.
> >>
> >> Matt
> >>
> >> -------------------------------------------------------------------------
> >> Using Tomcat but need to do more? Need to support web services, security?
> >> Get stuff done quickly with pre-integrated technology to make your job easier
> >> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
> >> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
> >> _______________________________________________
> >> mod-security-users mailing list
> >> mod-security-users <at> lists.sourceforge.net
> >> https://lists.sourceforge.net/lists/listinfo/mod-security-users
> >>
> >>
> > 
> > 
> > -------------------------------------------------------------------------
> > Using Tomcat but need to do more? Need to support web services, security?
> > Get stuff done quickly with pre-integrated technology to make your job easier
> > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
> > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
> > _______________________________________________
> > mod-security-users mailing list
> > mod-security-users <at> lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/mod-security-users
> > 
> > 
> 
> -------------------------------------------------------------------------
> Using Tomcat but need to do more? Need to support web services, security?
> Get stuff done quickly with pre-integrated technology to make your job easier
> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
> _______________________________________________
> mod-security-users mailing list
> mod-security-users <at> lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/mod-security-users
--

-- 
Michael T. Shinn                                    KeyID:0xDAE2EC86
Key Fingerprint:  1884 E657 A6DF DF1B BFB9 E2C5 DCC6 5297 DAE2 EC86
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xDAE2EC86

Got Root?  http://www.gotroot.com
modsecurity rules: http://www.modsecurityrules.com
Troubleshooting Firewalls:  http://troubleshootingfirewalls.com

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
Matt Wrycraft | 18 Aug 21:14 2006
Picon

Re: Mod-Security and php forums

Thanks Mike! I'll do that.

You're a star.

Matt

Michael Shinn wrote:
> Matt, 
> 
> Send me your audit log entries (sanitize if you like, just leave me the
> context of the way phpbb is being used), and I'll tweak the rules to
> prevent these false positives. 
> 
> On Thu, 2006-08-17 at 23:02 +0100, Matt Wrycraft wrote:
>> Thanks Tom!
>>
>> I am running SQL and can certainly understand the need to keep the rules 
>> dealing with injection attacks. SQL is used throughout my site and I 
>> would like to keep the rules generally, just exclude them from forum 
>> posts. I do have sanitation of forum posts anyway, which is why I'm 
>> happy to avoid using modsec there.
>>
>> Unfortunately, changing the forum software isn't really an option for me 
>> at the moment, and I really don't want to anyway as in all other 
>> respects its working fine. It's just the modsec false positives that are 
>> giving me sleepless nights.
>>
>> I just need the code to exclude it I guess.
>>
>> Matt
>>
>> Tom Anderson wrote:
>>> If you're not running an SQL database, then you can comment out the 
>>> chain of rules dealing with SQL injection attacks.  Otherwise, only 
>>> enable them on programs that deal directly with a database.  If your 
>>> forums use a database, the best thing is to ensure there is sanitation 
>>> of input within your forum software.  If so, you can disable the SQL 
>>> injection filters in mod_security.  Otherwise, it might be prudent to 
>>> keep it or to change forum software.
>>>
>>> Tom
>>>
>>> Matt Wrycraft wrote:
>>>> Hi all, this is my first question, so please let me know if I've missed 
>>>> anything.
>>>>
>>>> After a hacking attempt at another part of my site the sysadmins 
>>>> installed Mod Security and rules from 
>>>> http://www.gotroot.com/tiki-index.php?page=mod_security+rules
>>>> I now get a number of false positives when posting on the forums (which 
>>>> are very active and people are getting annoyed at the number of 403s). 
>>>> I've run google search and checked out the gotroot forum to no avail. 
>>>> Looking at the audit.log most of the denied posts relate to:
>>>>
>>>> mod_security-message: Access denied with code 403. Pattern match 
>>>> "(insert[[:space:]]+into.+values|select.+from|bulk[[:space:]]+insert|union.+select)" 
>>>> at POST_PAYLOAD
>>>>
>>>> I've tried loads of variations of posts on the forum and having multiple 
>>>> spaces or using "into" "select" and "from" in the right order is 
>>>> obviously what is causing me these problems. I tried commenting out the 
>>>> rule but Apache failed to start, a quick look suggested that it was part 
>>>> of a chain of rules and I didn't want to dig too deeply in case I broke 
>>>> it further, so I uncommented the rule and successfully restarted the 
>>>> webserver.
>>>>   I don't really know regex so am not in a position to completely 
>>>> re-write the rules, as I'm just as likely to make things worse, so I've 
>>>> reached the conclusion that I would just like to disable mod security 
>>>> for forum posts.
>>>>
>>>> I've checked /etc/modsecurity/exclude.conf and there already seems to be 
>>>> relevant rules for other php-based forums ie
>>>>
>>>> <LocationMatch "/index.php?name=PNphpBB2&file=posting&mode=reply.*">
>>>> SecFilterRemove 300013
>>>> </LocationMatch>
>>>>
>>>> Now, I'm unsure what rule 300013 does. I've grepped for it and it seems 
>>>> to be commented out in the rules.conf..
>>>>
>>>> Anyway, my forum uses the following POST URL, which i assume is the 
>>>> location match:
>>>>   /modules.php?name=Forums&file=posting&mode=newtopic&f=13
>>>> (with different information after 'posting')
>>>>
>>>> So I tried putting that into exclude.conf  - adding a number of rules 
>>>> that I thought could be problems (and taking them from other rules 
>>>> excluded in the exclude.conf) as follows:
>>>>
>>>> <Location "/modules.php?name=Forums&file=posting.*">
>>>> SecFilterRemove 300013
>>>> SecFilterRemove 300014
>>>> SecFilterRemove 300016
>>>> SecFilterRemove 380000
>>>> SecFilterRemove 360001
>>>> </Location>
>>>>
>>>> This made no difference and I still suffer the false positives.
>>>>
>>>> All I want to do is exclude mod security from checking forum posts (yes 
>>>> I know the risks of this, but I'm using recent code, with additional 
>>>> handling, and feel that mod security is causing more harm than good 
>>>> right now)
>>>>
>>>> Can I for instance use a variation of:
>>>> SecFilterSelective REQUEST_URI "/.*/Merchant2/merchant\.mv.*" allow,nolog
>>>>
>>>> ??
>>>>
>>>> Any help would be graciously accepted!
>>>>
>>>> Many thanks in advance.
>>>>
>>>> Matt
>>>>
>>>> -------------------------------------------------------------------------
>>>> Using Tomcat but need to do more? Need to support web services, security?
>>>> Get stuff done quickly with pre-integrated technology to make your job easier
>>>> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
>>>> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
>>>> _______________________________________________
>>>> mod-security-users mailing list
>>>> mod-security-users <at> lists.sourceforge.net
>>>> https://lists.sourceforge.net/lists/listinfo/mod-security-users
>>>>
>>>>
>>>
>>> -------------------------------------------------------------------------
>>> Using Tomcat but need to do more? Need to support web services, security?
>>> Get stuff done quickly with pre-integrated technology to make your job easier
>>> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
>>> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
>>> _______________________________________________
>>> mod-security-users mailing list
>>> mod-security-users <at> lists.sourceforge.net
>>> https://lists.sourceforge.net/lists/listinfo/mod-security-users
>>>
>>>
>> -------------------------------------------------------------------------
>> Using Tomcat but need to do more? Need to support web services, security?
>> Get stuff done quickly with pre-integrated technology to make your job easier
>> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
>> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
>> _______________________________________________
>> mod-security-users mailing list
>> mod-security-users <at> lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/mod-security-users

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
Tom Anderson | 18 Aug 20:05 2006

Re: Mod-Security and php forums

Matt Wrycraft wrote:
> I am running SQL and can certainly understand the need to keep the rules 
> dealing with injection attacks. SQL is used throughout my site and I 
> would like to keep the rules generally, just exclude them from forum 
> posts. I do have sanitation of forum posts anyway, which is why I'm 
> happy to avoid using modsec there.

In my experience, it's mostly the "select from" part of the rule which 
trips up normal speech in discussion forums.  Most other SQL commands 
are not a part of normal speech.  Therefore, allowing "select from" (by 
removing that part of the rule) may be worth your while if seperate 
sanitation is done within the software.  If your discussion forums 
actually have SQL as a topic, then that theory goes out the window 
though and sanitation is absolutely required so that all text goes 
through unfiltered (and unexecuted), and the entire SQL-injection rule 
should be removed.

In the latter case, your location-based rule removal should work, 
assuming you've specified your SQL-injection rule correctly. 
Alternatively, you can start fresh with new rules for the forums:

<Location /forums/≥
	SecFilterInheritance Off
	SecFilterImport ...
	SecFilter ...
</Location>

Tom

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
Matt Wrycraft | 18 Aug 21:15 2006
Picon

Re: Mod-Security and php forums

I certainly think it's something I need to do.

Many thanks for your help! :)

Matt

Tom Anderson wrote:
> Matt Wrycraft wrote:
>> I am running SQL and can certainly understand the need to keep the rules 
>> dealing with injection attacks. SQL is used throughout my site and I 
>> would like to keep the rules generally, just exclude them from forum 
>> posts. I do have sanitation of forum posts anyway, which is why I'm 
>> happy to avoid using modsec there.
> 
> In my experience, it's mostly the "select from" part of the rule which 
> trips up normal speech in discussion forums.  Most other SQL commands 
> are not a part of normal speech.  Therefore, allowing "select from" (by 
> removing that part of the rule) may be worth your while if seperate 
> sanitation is done within the software.  If your discussion forums 
> actually have SQL as a topic, then that theory goes out the window 
> though and sanitation is absolutely required so that all text goes 
> through unfiltered (and unexecuted), and the entire SQL-injection rule 
> should be removed.
> 
> In the latter case, your location-based rule removal should work, 
> assuming you've specified your SQL-injection rule correctly. 
> Alternatively, you can start fresh with new rules for the forums:
> 
> <Location /forums/≥
> 	SecFilterInheritance Off
> 	SecFilterImport ...
> 	SecFilter ...
> </Location>
> 
> Tom
> 
> 
> -------------------------------------------------------------------------
> Using Tomcat but need to do more? Need to support web services, security?
> Get stuff done quickly with pre-integrated technology to make your job easier
> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
> _______________________________________________
> mod-security-users mailing list
> mod-security-users <at> lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/mod-security-users
> 
> 

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
Brian Rectanus | 18 Aug 15:08 2006
Picon

Re: Performance tip

On 8/17/06, Tom Anderson <tanderso <at> oac-design.com> wrote:
> Regarding readability, as far as I'm concerned, the shorter the better,
> which is why condensed regexes are great.  If you cannot read regexes,
> that's what comments are for.  But let's get serious -- any sysadmin
> worth his salt needs to know how to read and write regular expressions.
>   To me, a regular expression is far more readable than a long English
> comment or a multi-line block of code.  If you have to translate it, do
> so in a comment, but you shouldn't really need subtitles to use a common
> system tool like regexes unless some particular pattern is really
> convoluted or tricky.

I don't think the comments should be for translating the RE, but
instead what the REs purpose is or the logical steps it is is
following in a chained rule.  So, not 'matches foo followed by any
chars up to bar', but 'detect foobar attack by looking for attack
signature from CVE-blah'.  Other comments could be an example of an
attack request, etc.  I think these comments are far more useful then
trying to explain the RE syntax.  For multiple attack signatures
combined into a single rule via '|', this becomes hard to comment with
just Apache comments.

Although Ivan's note was for ORing simple rules -- which I think is
good -- I am still not convinced that this gives that much performance
benefit from anything but ORing simple matches like keywords.  In
other words, I don't think all cases will benefit here.  I still have
yet to see hard numbers that show that combining 100 complex rules
down to one has performance benefits worth the extra complexity and
error-prone nature of more complex rules.

Anyone have hard stats with numbers they would like to share?

-B

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
Tom Anderson | 18 Aug 19:52 2006

Re: Performance tip

Brian Rectanus wrote:
> I don't think the comments should be for translating the RE, but
> instead what the REs purpose is or the logical steps it is is
> following in a chained rule.  So, not 'matches foo followed by any
> chars up to bar', but 'detect foobar attack by looking for attack
> signature from CVE-blah'.  Other comments could be an example of an
> attack request, etc.  I think these comments are far more useful then
> trying to explain the RE syntax.  For multiple attack signatures
> combined into a single rule via '|', this becomes hard to comment with
> just Apache comments.

Fair enough, I agree that rules should be commented to document their 
general effect or purpose.  Completely unrelated rules should be kept 
seperate for logical distinction without too much impact on performance.

> Although Ivan's note was for ORing simple rules -- which I think is
> good -- I am still not convinced that this gives that much performance
> benefit from anything but ORing simple matches like keywords.  In
> other words, I don't think all cases will benefit here.  I still have
> yet to see hard numbers that show that combining 100 complex rules
> down to one has performance benefits worth the extra complexity and
> error-prone nature of more complex rules.
> 
> Anyone have hard stats with numbers they would like to share?

I threw together a quick script to benchmark the difference between 
various optimizations.  You can view/download the script here: 
http://orderamidchaos.com/modsec/regex-benchmark

You can either enter your own request input as a parameter or choose 1, 
2, or 3 to test my built-in samples (actually taken from my audit log).

I tested four different cases using the same 29 distinct rules.  In the 
first case, they are atomized into one line per rule.  Next, they are 
combined into two distinct rules.  Third, they are condensed to reduce 
backtracking.  And finally, they are made non-capturing.  Here are the 
results:

./regex-benchmark 1

                  Rate    atomized     combined    condensed noncapturing
atomized       8029/s          --         -93%         -93%         -93%
combined     113298/s       1311%           --          -0%          -1%
condensed    113752/s       1317%           0%           --          -0%
noncapturing 114323/s       1324%           1%           1%           --

./regex-benchmark 2

                  Rate    atomized     combined    condensed noncapturing
atomized       6894/s          --         -93%         -93%         -93%
combined      99964/s       1350%           --          -0%          -0%
condensed    100161/s       1353%           0%           --          -0%
noncapturing 100238/s       1354%           0%           0%           --

./regex-benchmark 3

                  Rate    atomized     combined    condensed noncapturing
atomized       7854/s          --         -93%         -93%         -93%
combined     109253/s       1291%           --          -1%          -1%
condensed    110287/s       1304%           1%           --          -0%
noncapturing 110636/s       1309%           1%           0%           --

So you can see that there is a fairly significant performance 
difference, with the non-capturing condensed rules performing over 1300% 
better than the distinct rules.  There isn't a huge difference between 
the other optimizations, but simply combining distinct rules into fewer, 
more complex rules provides a major improvement.

Granted, this is done natively in Perl, not in Apache.  But since 
ModSecurity is using the Perl regex engine, the comparision should be 
close.  If anything, the differences should be more drastic in Apache 
due to additional overhead between rules such as ModSecurity's 
processing and logging.

Furthermore, this is only a tiny subset of the rules contained in most 
ModSecurity configurations.  Combining dozens or hundreds of rules 
should provide even more benefit.

It would also follow that eliminating rules altogether should provide a 
nice performance boost, so weeding out those which are extraneous may 
well be worth your time.

Tom

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642

Gmane