The Kemp WAF is based on the OWASP ModSecurity Core Rule Set (CRS), it is a set of firewall rules that are loaded that provide generic signatures for common attack types (e.g. SQL Injection, XSS and so on), it uses string matching, regex techniques to look for and block attacks in a general way. What this means in practice is rather than having a specific signature for an attack which looks like this, it has a general set of rules that can be used to identify traffic that looks like an attack.
There are advantages and disadvantages to this, an advantage is in being general it can pick up a wider range of attacks including those that it doesn’t have a specific signature for (something that the non-heuristic signature based techniques can’t do) ergo zero-day attacks, the downside however is in being general it may get more false positives because it is not specific like a signature based solution.
If I Needed to Read One Thing What Would it Be?
If you are looking for a really good overview of OWASP, ModSecurity etc. then this gives you the generic information.
If you are looking for how to tune false positives in one fell swoop, read this:
False Positives and Tuning
The CRS can be quite strict in detecting certain attacks. This means that sometimes a normal, bona fide, request may be blocked as a false alert or false positive, for instance, if it contains suspicious character sequences. We aim to minimize false positives as much as possible, but in some situations it may be necessary for you to write an exclusion rule which selectively disables some CRS checks.
!! CRS is updated from time to time in order to address false positives and add new protections, which requires you to replace the files with a new release. Furthermore, the CRS adds some, usually minor, processing overhead to any request.
All the hits appearing in the Anomaly Histogram are just that, requests. Unless they are over your Anomaly Level Threshold they are not being blocked, but they might be considered noise that you’d rather not see, you can tune these away if you wish. If for example your are seeing particular requests being blocked, these are known as “False Positives” and you’ll need to identify which rule is being triggered, why and then develop a way of tuning this false positive away, and ideally without opening up a larger than it needs to be security hole!
The anomaly scoring is how the rules work. Instead of a rule matching and then blocking based on just one rule, the system runs through all the rules and if multiple triggers are hit or one serious trigger is hit depending on the anomaly score it will block the transaction. This is better for detecting unknown exploit attempts and requires less rules to detect and block more intrusion attempts.
Showing the False Positive Analysis
The easiest way to determine what is going on with the WAF and what is being blocked is to examine the False Positive Analysis once the WAF has been enabled.
Expand “Web Application Firewall” click on “False Positive Analysis”, you’ll then be able to select any VS or subVS where the WAF has been enabled, then see the logs of any impacted issues in flight.
Any issues that the WAF has identified are logged here (and via Splunk) for further analysis.
For example the Rule: 92040 seems to be triggered a lot, now in this case this appears to be completely normal traffic, these particular application request types are normal for this particular application, now if we were blocking to a level that these were over threshold they would be blocked, in this case we’re not blocking but we’d like to clear them away because these are not relevant and are allowed traffic.
Request content type is not allowed by policy
21279|application/mapi-http|
3479|application/vnd.ms-sync.wbxml|
2|application/vnd.ms-sync|Tuning Away (i.e. Overriding) a Triggering Rule
After turning on the WAF to protect the application(s) you’ll be able to use the Web Application Firewall → False Postive Analysis to help you identify any problematic requests, any requests that are dangerous and also assist you in finding, diagnosing and removing False Positives.
The Kemp WAF cannot tell you if a request that is triggering a rule is a false positive or not, you must understand what the “normal” behaviour is of our application, then “tune out” any anomalies where “normal” behaviour is triggering a rule, the assumption then being that anything remaining will then be a true positive and therefore should be considered to be something that should be blocked (if sufficiently severe and/or if it exceeds the Anomaly Count threshold).
The process of “tuning away” or overriding triggering rules that are either generating false positives or are just creating loads of noise that is not relevant when certain requests come through is a very big topic, you can find loads of interesting information on this within the following links, but this section will cover a few examples to illustrate the process of: 1. Finding False Postitives, 2: Diagnosing them, 3: Creating an exclusion rule, 4: Uploading that custom rule to the Kemp WAF and finally 5: activating the custom rule and examining its effects.
- https://support.kemptechnologies.com/hc/en-us/articles/208109226-Whitelist-an-IP-using-WAF-ModSecurity-Whitelisting-IP-s
- https://www.netnea.com/cms/nginx-tutorial-7_including-owasp-modsecurity-core-rule-set/
- https://kemptechnologies.com/blog/false-positive-handling-loadbalancer
- https://support.kemptechnologies.com/hc/en-us/articles/210399183-WAF-Rule-Writing-Guide
In this example we are using the Microsoft Exchange Web Services, there are a number of services that utilise the HTTPS protocol in weird and wonderful ways to provide a simple method to allows clients to access email services (and mailboxes) via the web from a myriad different clients.
If you are looking for how to tune false positives out, in a single very detailed guide, read this link, my documentation gives a high-level step by step, but not the same level of detail as this:
Step 1 – Finding False Positives
Logs. That’s where to look.
Kemp WAF False Postitive Analysis Console
First port of call is: Web Application Firewall → False Postive Analysis , this can provide a quick overview of what is going on, it provides a Rule Counts and an Anomaly Histogram so you can see what is being blocked and why.
Clean Requests are requests that make their way through the WAF without triggering any rules hence have a Anomaly Level of 0.
Splunk SYSLOGs
You also notice at the bottom of the False Postitive Screen the event log, this is basically what gets sent out to Splunk, you’ll find it much easier to find stuff in Splunk than here. An example query to run in Splunk is as follows:
index=<yourindex> host="kemp-01" wafd |
You can then tweak the log to hone in on what you want to pick out, e.g. a particular rule ID or IP address etc.
Additional Clarity
Here is some more examples. Clean Requests are requests that make their way through the WAF without triggering any rules hence have a Anomaly Level of 0.
Let’s look at the third line down, which has requests with a Anomaly Level of 5, there have been 58383 request that are showing a Anomaly Level of 5. The “Rules” section with the number in brackets are which of the rules are being triggered and how many of this total 58383 have triggered this particular rule. For example the 920420 rule has been triggered by 58294 of the 58383 requests, while the 932140 rule has been triggered by only 5.
Step 2 – Diagnosing a False Positive
A topic that could run to PhD length, but a brief example here. Normally when a false positive appears you’ll just see something within the application failing to work, then you’ll need to consult the logs to find what rule is being triggered that is blocking it, why and then how you can tune it out (remediate) it without opening a hole in the security but also resolving the problem and eliminating the False Positive. Perhaps in an application you click a button on a web page, and it doesn’t do anything because that request is being blocked by the WAF, or perhaps it partially completes but then gives an error.
In this example we’re using the normal Microsoft Outlook, Microsoft Outlook for Mac, Microsoft Outlook (Andriod and iOS) mobile phone apps to access mailboxes via Microsoft Web Services, now when we checked the logs we saw these items appearing regularly. We know these are OK because they are coming from my home IP address, I know I’m not generating dodgy requests, but I do know I just have my Outlook client up and running so if I’m seeing requests triggering a rule I know that is normal behaviour being incorrectly marked as (potentially) nefarious. The way the OWASP CRS rules work is heuristically rather than signature based, which is why it is both very good at matching dodgy requests but at the risk of picking up requests that are “normal” but are utilising non-standard and very application specific request characteristics.
Let’s look at the logs, here you’ll see three different requests which were appearing with much regularity.
2022-12-07T13:29:57+00:00 kemp-01a wafd: [client 1.2.3.4] ModSecurity: Warning. Match of "within %{tx.allowed_request_content_type}" against "TX:content_type" required. [file "/tmp/waf/73/REQUEST-920-PROTOCOL-ENFORCEMENT.conf"] [line "956"] [id "920420"] [msg "Request content type is not allowed by policy"] [data "|application/mapi-http|"] [severity "CRITICAL"] [ver "OWASP_CRS/3.3.2"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-protocol"] [tag "paranoia-level/1"] [tag "OWASP_CRS"] [tag "capec/1000/255/153"] [tag "PCI/12.1"] [hostname "100.100.100.100 (TEST-EXCHANGE)"] [uri "/mapi/emsmdb/"] [unique_id "ec660956-c900-4112-99c0-8b8b70332b0e"]
2022-12-07T13:39:47+00:00 kemp-01a wafd: [client 5.6.7.8] ModSecurity: Warning. Match of "pm AppleWebKit Android" against "REQUEST_HEADERS:User-Agent" required. [file "/tmp/waf/73/REQUEST-920-PROTOCOL-ENFORCEMENT.conf"] [line "1228"] [id "920300"] [msg "Request Missing an Accept Header"] [severity "NOTICE"] [ver "OWASP_CRS/3.3.2"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-protocol"] [tag "OWASP_CRS"] [tag "capec/1000/210/272"] [tag "PCI/6.5.10"] [tag "paranoia-level/2"] [hostname "100.100.100.100 (TEST-EXCHANGE)"] [uri "/ews/exchange.asmx"] [unique_id "cbbf7470-8f82-4d45-a2fa-77cab65e4e83"]
2022-12-07T13:41:25+00:00 kemp-01a wafd: [client 5.6.7.8] ModSecurity: Warning. Match of "pm AppleWebKit Android" against "REQUEST_HEADERS:User-Agent" required. [file "/tmp/waf/73/REQUEST-920-PROTOCOL-ENFORCEMENT.conf"] [line "1228"] [id "920300"] [msg "Request Missing an Accept Header"] [severity "NOTICE"] [ver "OWASP_CRS/3.3.2"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-protocol"] [tag "OWASP_CRS"] [tag "capec/1000/210/272"] [tag "PCI/6.5.10"] [tag "paranoia-level/2"] [hostname "100.100.100.100 (TEST-EXCHANGE)"] [uri "/autodiscover/autodiscover.xml"] [unique_id "bcd901f8-5e86-4cfd-be99-7cac8f10688a"]Taking these apart we want to find things within the request we can use to allow us to create some sort of exception, some things that stand out are:
[id "920420"]
[data "|application/mapi-http|"]
[uri "/mapi/emsmdb/"]
[id "920300"]
[uri "/ews/exchange.asmx"]
[id "920300"]
[uri "/autodiscover/autodiscover.xml"]There are lots of ways to skin a cat, and in this case lots of ways to identify what the traffic is to exempt it, you could use the IP address, but this isn’t ideal unless in specific circumstances (LINK), you should also read: https://kemptechnologies.com/blog/false-positive-handling-loadbalancer for some more techniques.
3: Creating an Exclusion Rule
Okay so looking at the above, we can see some URIs, these are what the Outlook application(s) are using to query, clearly the OWASP CRS rules think that in the case of 920300 the “request missing an accept header” is a problem, now for the Microsoft implementation of the application this is probably entirely normal, so we can safely exclude this type of request and one way to do this would be to use the URI as a way to exclude these requests, obviously we want to be as explicit as possible but
When creating a rule start with next available number, check the Kemp before you create it, for this example the rule ID: 10100 is available, so you’d create a file called 10100.conf to contain the rules. Now you may well want to add multiple sub-rules under this, so to ensure you don’t have to check for any IDs else where you’re best of creating these as 10101, then 10102, 10103 and so on.
For example: when you come to create a new custom rule to do something else you’d create it as rule ID: 102, that would give you up to 99 sub-rules 10201, 10202, 10203 etc you can include within.
https://coreruleset.org/docs/rules/ruleid/#id-reservations
Use the ID range 1-99,999 for our own internal Sanger CRS rules.
# == ModSec Rule ID Namespace Definition
# Service-specific before Core-Rules: 10000 - 49999
# Service-specific after Core-Rules: 50000 - 79999
# Locally shared rules: 80000 - 99999
# Recommended ModSec Rules (few): 200000 - 200010
# OWASP Core-Rules: 900000 - 999999Right so let’s create a file then called 10100.conf and stick in our modsec rules. You notice the ruleRemoveById directive at the end, this specifies that if the REQUEST_URI matches say “/mapi” then to deactivate Rule ID: 920420. Same thing for the rest of the rules.
Use a hash/octothope # character to comment out a line.
Pay attention to where i.e. which phase the rule is triggering, in these examples Phase 1 (Request Headers), this is mentioned earlier in this document, and in: https://kemptechnologies.com/blog/false-positive-handling-loadbalancer
10100.conf
# ModSec Rule Exclusion 920420 & 920300 : Request content type is not allowed by policy.# This overrides for the Outlook Exchange Web Services
# SecRule REQUEST_URI "@beginsWith /mapi" "phase:1,nolog,msg:'MAPI Excemption',pass,id:10101,severity:'notice',ctl:ruleRemoveById=920420"
# SecRule REQUEST_URI "@beginsWith /ews" "phase:1,nolog,msg:'MAPI Excemption',pass,id:10102,severity:'notice',ctl:ruleRemoveById=920300"
# SecRule REQUEST_URI "@beginsWith /autodiscover" "phase:1,nolog,msg:'MAPI Excemption',pass,id:10103,severity:'notice',ctl:ruleRemoveById=920300"
SecRule REQUEST_URI "@beginsWith /mapi" "phase:1,log,msg:'MAPI Excemption',pass,id:10101,severity:'notice',ctl:ruleRemoveById=920420"
SecRule REQUEST_URI "@beginsWith /ews" "phase:1,log,msg:'EWS Excemption',pass,id:10102,severity:'notice',ctl:ruleRemoveById=920300"
SecRule REQUEST_URI "@beginsWith /autodiscover" "phase:1,log,msg:'Autodiscover Excemption',pass,id:10103,severity:'notice',ctl:ruleRemoveById=920300- log = Shows the match of this (exclusion rule) in the log.
- nolog = Doesn’t show the match of this (exclusion rule) in the log, you might well want this, if you’re happily matching and allowing this traffic you don’t want to see it cluttering up the logs, then you can set this.
See the https://kemptechnologies.com/blog/false-positive-handling-loadbalancer for more examples.
4: Upload the Custom Rule to Kemp Load Master WAF
Take your saved 10100.conf file and then upload it by going to Web Application Firewall → Custom Rules, then click on “Ruleset File”, pick the file and click “Add Ruleset”.
The rule is now ready for use.
5: Activating the Custom Rule to Exclude and Examining the Effect
Now you have your rule ready and uploaded you can put it into place.
Activate the Custom Rule
To use your custom rule click on Virtual Services→ View/Modify Services then click on the particular Virtual Service your WAF is activated on and click “Modify”
Now expand the WAF section.
You can see your newly added rule: 10100, click on it to tick the box and then click “Apply” to activate it.
You can also click on the name itself (i.e. the number in this case) and this will expand it to show all the subrules beneath, so you can turn bits of this custom rule on and off as you wish to test or activate accordingly.
Viewing Its Effect
Now your rule has been activated you need to go back to the Web Application Firewall → False Positive Analysis, then select the relevant VS which has the WAF and this custom rule applied.
Of course, you’ll have all the old data there, so click “Reset FPA Counters” to clear everything back.
The requests should now come through but instead of triggering a rule, your exclusion rule(s) will ensure that the rule is not triggered for that particular traffic, ergo that traffic gets dropped into the “Clean Requests”.
Example Rule Trigger
Let’s try an SQL injection attack, so from a machine with CURL run the following:
curl -X POST https://server.domain.com/accounts/login/ -d "username=davemchackface&password=' UNION TABLE spaceships;"Additional Information
These documents should be taken as giving a in-depth tech deep dive into all the mechanisms.