gmane.comp.apache.mod-security.user

Re: problem with my regex and single line HTMLcomment in RESPONSE_BODY

Hi Achim,

Pardon me if my rant is off-topic, but this seems to be a good place
for it for me now.

In the 60+ hours in the last 5 days that I have spent writing
ModSecurity rules for WebGoat vulnerabilities, more than half of that
time has been spent on getting the regex's working. I am so tired of
reading the debug file to see how my regex is being interpreted.

I feel like I am a slave to the PCRE engine instead of the opposite.

It's not rocket science:
1. I want an account number that has digits, characters and a hyphen,
but no spaces or special characters.
2. I want a password that has alphanumeric and special chars, but has
no spaces or '>' and '<'.
3. I want a user name with chars, ', -, and spaces but nothing else.

I could do this much easier and faster writing Java, C#, or C (which
is why ModSecurity is written in C; check the source for
urlDecodeUni).

I'm at the point where I think it's easier to write my own routines in
Lua and build my own library for reuse; disclaimer: I don't need
speed.

(/end of rant)

Stephen

(Continue reading)

headers

Ivan Ristic | 3 Jul 16:22

Re: problem with my regex and single line HTMLcomment in RESPONSE_BODY

I think you want a ModSecurity IDE... I've dreamed of one myself. It's
feasible, from a technical point of view, but requires a commitment of
resources to implement.

Questions below.

On Thu, Jul 3, 2008 at 3:14 PM, Stephen Craig Evans
<stephencraig.evans <at> gmail.com> wrote:
> Hi Achim,
>
> Pardon me if my rant is off-topic, but this seems to be a good place
> for it for me now.
>
> In the 60+ hours in the last 5 days that I have spent writing
> ModSecurity rules for WebGoat vulnerabilities, more than half of that
> time has been spent on getting the regex's working. I am so tired of
> reading the debug file to see how my regex is being interpreted.
>
> I feel like I am a slave to the PCRE engine instead of the opposite.

We are all slaves, Stephen, just slaves :)

Can we channel your rant into something useful: how about a suggestion?

>
> It's not rocket science:
> 1. I want an account number that has digits, characters and a hyphen,
> but no spaces or special characters.
> 2. I want a password that has alphanumeric and special chars, but has
> no spaces or '>' and '<'.

(Continue reading)

headers

Stephen Craig Evans | 3 Jul 16:45

Re: problem with my regex and single line HTMLcomment in RESPONSE_BODY

Hi Ivan,

This is nothing directed at you guys.

For my use, bypassing regex altogether; as in C, using a running
buffer pointer like I see so much in your code   Lua might have the
same functionality.

I just feel like I am using a hammer with regex's while everything is
not a nail.

Stephen

On Thu, Jul 3, 2008 at 10:22 PM, Ivan Ristic <ivan.ristic <at> gmail.com> wrote:
> I think you want a ModSecurity IDE... I've dreamed of one myself. It's
> feasible, from a technical point of view, but requires a commitment of
> resources to implement.
>
> Questions below.
>
> On Thu, Jul 3, 2008 at 3:14 PM, Stephen Craig Evans
> <stephencraig.evans <at> gmail.com> wrote:
>> Hi Achim,
>>
>> Pardon me if my rant is off-topic, but this seems to be a good place
>> for it for me now.
>>
>> In the 60+ hours in the last 5 days that I have spent writing
>> ModSecurity rules for WebGoat vulnerabilities, more than half of that
>> time has been spent on getting the regex's working. I am so tired of

(Continue reading)

headers

Ivan Ristic | 3 Jul 16:50

Re: problem with my regex and single line HTMLcomment in RESPONSE_BODY

On Thu, Jul 3, 2008 at 3:45 PM, Stephen Craig Evans
<stephencraig.evans <at> gmail.com> wrote:
> Hi Ivan,
>
> This is nothing directed at you guys.

No worries, I didn't think it was.

> For my use, bypassing regex altogether; as in C, using a running
> buffer pointer like I see so much in your code   Lua might have the
> same functionality.
>
> I just feel like I am using a hammer with regex's while everything is
> not a nail.

That could be true. Perhaps it would help if we knew what you were
doing. Since I am supposed to be helping with your work anyway, I will
spend some time on that next week.

> Stephen
>
> On Thu, Jul 3, 2008 at 10:22 PM, Ivan Ristic <ivan.ristic <at> gmail.com> wrote:
>> I think you want a ModSecurity IDE... I've dreamed of one myself. It's
>> feasible, from a technical point of view, but requires a commitment of
>> resources to implement.
>>
>> Questions below.
>>
>> On Thu, Jul 3, 2008 at 3:14 PM, Stephen Craig Evans
>> <stephencraig.evans <at> gmail.com> wrote:

(Continue reading)

headers

Ryan Barnett | 3 Jul 16:33

Re: problem with my regex and single lineHTMLcomment in RESPONSE_BODY

Perhaps I missed it but have you tried testing your regexs with tools
like Expresso before trying them in Mod?  They have a good description
view that states what the regex components actually mean.

http://blog.modsecurity.org/2007/03/regular-express.html

The debug log doesn't really show you "how" the regex is being
interpreted.

-Ryan

> -----Original Message-----
> From: mod-security-users-bounces <at> lists.sourceforge.net [mailto:mod-
> security-users-bounces <at> lists.sourceforge.net] On Behalf Of Stephen
Craig
> Evans
> Sent: Thursday, July 03, 2008 10:14 AM
> To: Achim Hoffmann
> Cc: mod-security-users <at> lists.sourceforge.net
> Subject: Re: [mod-security-users] problem with my regex and single
> lineHTMLcomment in RESPONSE_BODY
> 
> Hi Achim,
> 
> Pardon me if my rant is off-topic, but this seems to be a good place
> for it for me now.
> 
> In the 60+ hours in the last 5 days that I have spent writing
> ModSecurity rules for WebGoat vulnerabilities, more than half of that
> time has been spent on getting the regex's working. I am so tired of

(Continue reading)

headers

Stephen Craig Evans | 3 Jul 16:50

Re: problem with my regex and single lineHTMLcomment in RESPONSE_BODY

Hi Ryan,

Yes, I use Expresso 3.0 and The Regex Coach and as you know the
results often don't jive with the results in ModSecurity.

> The debug log doesn't really show you "how" the regex is being
> interpreted.
Yeah, it's not a good thing when I have to use the debug log to
interpret how a regex is being processed.

Again, please don't take this as anything against any of you guys at
Breach or how ModSecurity is implemented. I'm just venting my current
frustration at the regex stuff.

Cheers,
Stephen

On Thu, Jul 3, 2008 at 10:33 PM, Ryan Barnett <Ryan.Barnett <at> breach.com> wrote:
> Perhaps I missed it but have you tried testing your regexs with tools
> like Expresso before trying them in Mod?  They have a good description
> view that states what the regex components actually mean.
>
> http://blog.modsecurity.org/2007/03/regular-express.html
>
> The debug log doesn't really show you "how" the regex is being
> interpreted.
>
> -Ryan
>
>> -----Original Message-----

(Continue reading)

headers

Ivan Ristic | 3 Jul 16:57

Re: problem with my regex and single lineHTMLcomment in RESPONSE_BODY

On Thu, Jul 3, 2008 at 3:50 PM, Stephen Craig Evans
<stephencraig.evans <at> gmail.com> wrote:
> Hi Ryan,
>
> Yes, I use Expresso 3.0 and The Regex Coach and as you know the
> results often don't jive with the results in ModSecurity.
>
>> The debug log doesn't really show you "how" the regex is being
>> interpreted.
> Yeah, it's not a good thing when I have to use the debug log to
> interpret how a regex is being processed.

But the debug log is not telling you how regular expressions are
processed. It is only telling you which regex ModSecurity is running,
and against what text. Are you frustrated with not being to able to
write regular expressions effortlessly, or by not being able to know
exactly how ModSecurity is executing the rules (and in which order,
etc).

> Again, please don't take this as anything against any of you guys at
> Breach or how ModSecurity is implemented. I'm just venting my current
> frustration at the regex stuff.
>
> Cheers,
> Stephen
>
>
> On Thu, Jul 3, 2008 at 10:33 PM, Ryan Barnett <Ryan.Barnett <at> breach.com> wrote:
>> Perhaps I missed it but have you tried testing your regexs with tools
>> like Expresso before trying them in Mod?  They have a good description

(Continue reading)

headers

Stephen Craig Evans | 3 Jul 17:14

Re: problem with my regex and single lineHTMLcomment in RESPONSE_BODY

Ivan,

Of course I am "frustrated with not being to able to write regular
expressions effortlessly" 

I think when Ryan cleans the gunk from under his fingernails, there's
more regex knowledge dispelled than what I have now.

Hope you all can make it for the OWASP conference in Portugal in November.

Stephen

On Thu, Jul 3, 2008 at 10:57 PM, Ivan Ristic <ivan.ristic <at> gmail.com> wrote:
> On Thu, Jul 3, 2008 at 3:50 PM, Stephen Craig Evans
> <stephencraig.evans <at> gmail.com> wrote:
>> Hi Ryan,
>>
>> Yes, I use Expresso 3.0 and The Regex Coach and as you know the
>> results often don't jive with the results in ModSecurity.
>>
>>> The debug log doesn't really show you "how" the regex is being
>>> interpreted.
>> Yeah, it's not a good thing when I have to use the debug log to
>> interpret how a regex is being processed.
>
> But the debug log is not telling you how regular expressions are
> processed. It is only telling you which regex ModSecurity is running,
> and against what text. Are you frustrated with not being to able to
> write regular expressions effortlessly, or by not being able to know
> exactly how ModSecurity is executing the rules (and in which order,

(Continue reading)

headers

Tom Anderson | 3 Jul 18:19

Re: problem with my regex and single lineHTMLcomment in RESPONSE_BODY

For what it's worth, I think that regexes are the best feature of 
ModSecurity and the most appropriate way to analyze a stream of text.  I 
couldn't imagine the headache of having to write rules in C.

Tom

Ryan Barnett wrote:
> Perhaps I missed it but have you tried testing your regexs with tools
> like Expresso before trying them in Mod?  They have a good description
> view that states what the regex components actually mean.
> 
> http://blog.modsecurity.org/2007/03/regular-express.html
> 
> The debug log doesn't really show you "how" the regex is being
> interpreted.
> 
> -Ryan
> 
>> -----Original Message-----
>> From: mod-security-users-bounces <at> lists.sourceforge.net [mailto:mod-
>> security-users-bounces <at> lists.sourceforge.net] On Behalf Of Stephen
> Craig
>> Evans
>> Sent: Thursday, July 03, 2008 10:14 AM
>> To: Achim Hoffmann
>> Cc: mod-security-users <at> lists.sourceforge.net
>> Subject: Re: [mod-security-users] problem with my regex and single
>> lineHTMLcomment in RESPONSE_BODY
>>
>> Hi Achim,

(Continue reading)

headers

Christian Folini | 3 Jul 18:23

Re: problem with my regex and single

On Thu, July 3, 2008 5:14 pm, Stephen Craig Evans wrote:

> Hope you all can make it for the OWASP conference in Portugal in November.

We'd all be happy to do that and drink on the success of
your project! By then amazon will have updated their
online shop to new system running modsecurity enhanced webgoat ...

cheers,

Christian

-------------------------------------------------------------------------
Sponsored by: SourceForge.net Community Choice Awards: VOTE NOW!
Studies have shown that voting for your favorite open source project,
along with a healthy diet, reduces your potential for chronic lameness
and boredom. Vote Now at http://www.sourceforge.net/community/cca08