must_not is not working for some of the alerts

[Mridul0902]

I have various elastalert rules that use must_not filter, but they are working fine for some of cases & for others this is not working.

Sample file:

  High-Search-API-Failures: |-
    ---
    name: "Integration-[High-Search-API-Failures-in-last-30-minutes][ElastAlert]"
    type: frequency
    index: "airarabia-*"
    is_enabled: true
    use_count_query: true
    scan_entire_timeframe: true
    num_events: 401
    realert:
      minutes: 30  
    timeframe:
      minutes: 30
    filter:
    - term:
        apiMethod.keyword: "search"    
    - term:
        level.levelStr.keyword: "ERROR"           
    - bool:
        must_not:
          - term:
              responseCode: "TFI2_0000"
    alert:
    - "email"
    email:
    - "[email protected]"
    smtp_host: test.mail.server
    smtp_port: 25
    smtp_ssl: false
    from_addr: [email protected]
    email_format: html
    alert_text_type: alert_text_jinja
    alert_text: |
      <p>Hello Oncall,</p><div><p>A spike in the number of <b style=\"color:red;\">Search API errors in last 30 minutes</b> have been detected.<br />The normal threshold for last half an hour is : <b style=\"color:green;\">400</b>Cases.<br />It has breached the threshold at : <b style=\"color:red;\">{{num_hits}}</b> Cases currently.</p></div>
  

For this alert, the count that I am getting in mail is not correct. The count matches if we disable the must_not filter.
Similar filter is working for other rules. There are 5-10 alerts out of 100 alerts that have this issue.

Expected Behaviour:
If the filter would we working fine, we have 0 hits that match this filter for 30 minutes, hence we should not be receiving mails.

Current Behaviour:
I got the mail with hit count value that matched with below filter

    filter:
    - term:
        apiMethod.keyword: "search"    
    - term:
        level.levelStr.keyword: "ERROR" 

Any idea what is wrong with the filter that it is not working for some cases?

[jertel]

Everything in the filter block is sent directly to Elasticsearch as-is. Therefore any filter issues or questions are better directed at the Elasticsearch community as they'll be better experienced in identifying mistakes or reasons for your observed behavior.

From an ElastAlert 2 perspective I suggest enabling debug logging so you can capture the exact query being sent to Elasticsearch. Then take that same query and compare it to a similar, working search in Elasticsearch Discover's Inspect panel.