Re: mod-security-users Digest, Vol 68, Issue 16

Chris -
For the ctl action, use this instead – ctl:ruleRemoveById:960015

While you do have the one use-case of wanting to remove that one rule when GoogleBot is indexing your site,
you can certainly also use these rules to alert if a client is masquerading as GoogleBot and flag them as a
malicious client.

I will work on your rules a bit and get back to you.

-Ryan

From: chris derham <chris <at> derham.me.uk<mailto:chris <at> derham.me.uk>>
Reply-To: "chris <at> derham.me.uk<mailto:chris <at> derham.me.uk>" <chris <at> derham.me.uk<mailto:chris <at> derham.me.uk>>
Date: Mon, 30 Jan 2012 11:50:01 -0600
To:
"mod-security-users <at> lists.sourceforge.net<mailto:mod-security-users <at> lists.sourceforge.net>" <mod-security-users <at> lists.sourceforge.net<mailto:mod-security-users <at> lists.sourceforge.net>>
Subject: Re: [mod-security-users] mod-security-users Digest, Vol 68, Issue 16

I think Josh recommended a good approach. Check the User-Agent for google-bot and the execute a Lua script
to do the double lookups and alert on mismatches.

This selective lookup check vs just enabling double hostname lookups in Apache us a better choice for performance.

Thinking on this more though, we should probably add a new operator for this - @dnsLookup.

Ryan

Replied to Ryan only - sorry

All,

So I have played about, and have something. I think it is about 90% there - I can't seem to work out how to access
variables declared in mod_security and loop over them in lua. Also I said before, I don't know lua and might
be doing something stupid. It will need more robot user-agent and domains added for other search engine
robots, but presumably this can be done once they trigger false positives on live sites

SecAction "pass,setvar:'tx.robot_user_
agents=google-bot'"
SecAction "pass,setvar:'tx.robot_domains=googlebot.com<http://googlebot.com/>'"
SecRule &REQUEST_HEADERS:User-Agent "!@within %{tx.robot_user_agents}" \
            "skipAfter:END_SEARCH_ENGINE_BOT_CHECK,phase:2,msg:'searchEngineBotReverseDnsLookup skipped
as not search engine bot activity',allow"
SecRuleScript modsecurity2/searchEngineBotReverseDnsLookup.lua "phase:1,msg:'search engine bot allowed',allow,ctl:ruleEngine=Off"
SecMarker END_SEARCH_ENGINE_BOT_CHECK

I wanted to use ctl:removeRuleById:960015 - this would then just turn off the single rule that is causing
issues, but leave all the other rules in place - I know their motto is do no evil, but you can't trust everyone
right? However this isn't working for me - perhaps I have a version of mod_security that doesn't support
this? We will update if this is the case. The error message I get is

Error parsing actions: Missing ctl value for name: removeRuleById:960015

The searchEngineBotReverseDnsLookup.lua file contains the following

require("io");

function trim(s)
  return (s:gsub("^%s*(.-)%s*$", "%1"))
end

function dnsLookup(source, search, hitCountRequired)
    local n = os.tmpname()

    os.execute ("nslookup " .. source .. " > " .. n)
    local hitCount = 0

    for line in io.lines (n) do
      if string.match(line, search) then
        hitCount = hitCount + 1
        if (hitCount == hitCountRequired) then
            result = string.sub(line, string.len(search) + 1)
            break
        end
      end
    end
    os.remove(n)
    return trim(result)
end

function string.ends(String,End)
   return End=='' or string.sub(String,-string.len(End))==End
end

function isKnownRobotDomain(domain)
    local result = true
    local d = m.getvars("tx.robot_domains")
    for i = 1, #d do
        if (string.ends(domain, d[i].value)) then
            return true;
        end
    end
    return false;
end

function main()
    local clientIp = m.getvar("REMOTE_ADDR");
    local reverseDnsName = dnsLookup(clientIp, "Name:", 1)
    local dnsClientIp = dnsLookup(reverseDnsName, "Address:", 2)
    local match = clientIp == dnsClientIp
    local matchDescription = ""
    local robotDomainDescription = "n/a"
    local result = nil;
    if (match) then
        matchDescription = "matches"
        if (isKnownRobotDomain(reverseDnsName)) then
            robotDomainDescription = "match"
            result = "match"
        else
            robotDomainDescription = "missmatch"
        end
    else
        matchDescription = "fails"
    end
    m.log(4, "searchBotReverseDnsLookup: clientIp " .. clientIp .. " " .. matchDescription .." reverse dns
lookup: " .. reverseDnsName .. " != " .. dnsClientIp .. " robotDomain: " .. robotDomainDescription);
    return result;
end

This bit that doesn't work is the " local d = m.getvars("tx.robot_domains") for i = 1, #d do" - I can't seem to
get the for loop to start. Any thoughts on what is wrong? Other than that I think that the approach will work
ok During testing as expected the first hit takes a while, but then the dns cache kicks in so subsequent
accesses are very quick

Thanks

Chris

________________________________
This transmission may contain information that is privileged, confidential, and/or exempt from
disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any
disclosure, copying, distribution, or use of the information contained herein (including any reliance
thereon) is STRICTLY PROHIBITED. If you received this transmission in error, please immediately
contact the sender and destroy the material in its entirety, whether in electronic or hard copy format.

------------------------------------------------------------------------------
Try before you buy = See our experts in action!
The most comprehensive online learning library for Microsoft developers
is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3,
Metro Style Apps, more. Free future releases when you subscribe now!
http://p.sf.net/sfu/learndevnow-dev2
_______________________________________________
mod-security-users mailing list
mod-security-users <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/mod-security-users
Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
http://www.modsecurity.org/projects/commercial/rules/
http://www.modsecurity.org/projects/commercial/support/