Realert does not seems to be taking effect?

[mparmar-pure]

I am using the flatline rule type to monitor the ELK indexes for any data drops. I am trying out the flatline with threshold = 1, timeframe = 10 minutes, ie. ElastAlert2 will alert me once there is zero log entry within last 10 minutes. Elastalert will scrap off data every , with run_every = 1 minute in the config.yaml. However, it turns out ElastAlert2 keeps sending me an alert every minute for same match every minute?

`name: "Nodata flatline - Infoblox"
type: flatline
is_enabled: false
index: "infoblox*"
realert:
days: 3
query_key: message
threshold: 1
timeframe:
minutes: 10
filter:

• query:
  match_all: {}
  doc_type: _doc
  alert:
• "jira"
  jira_server: "https://XX.atlassian.net/"
  jira_project: "XX"
  jira_issuetype: "Incident"
  jira_account_file: "/jira-serviceccnt.txt"
  jira_assignee: "XX"
  jira_priority: 0
  jira_description: "Index infoblox* has no data flowing in Prod ELK cluster. \n"
  `
  Thanks!

[jertel]

Perhaps the query_key: message is causing each unique message to have its own realert timer. Have you tried with that property commented out?

[mparmar-pure]

Actually added the query_key: message after the realert was not working, so yes i have tried out without this property.

[jertel]

You'll need to enable debug logging and watch the request/response from ES to find out what's going on.

[mparmar-pure]

tried turning on the debug logging for the container, and see the following (disregard the Jira error message as the error seems to be for assignee user) -

`INFO:elastalert:Queried rule Nodata flatline Switches from 2023-11-13 22:47 UTC to 2023-11-13 22:52 UTC: 0 / 0 hits
INFO:elastalert:Ran Nodata flatline Switches from 2023-11-13 22:47 UTC to 2023-11-13 22:52 UTC: 0 query hits (0 already seen), 0 matches, 0 alerts sent
INFO:elastalert:Nodata flatline Switches range 284
INFO:elastalert:Disabled rules are: ['Nodata flatline - Infoblox']
INFO:elastalert:Sleeping for 59.999846 seconds
WARNING:apscheduler.scheduler:Execution of job "Rule: Nodata flatline - Infoblox (trigger: interval[0:01:00], next run at: 2023-11-13 22:52:52 UTC)" skipped: maximum number of running instances reached (1)
INFO:elastalert:Background configuration change check run at 2023-11-13 22:53 UTC
ERROR:elastalert:Error while running alert jira: Error creating Jira ticket using jira_args ({'project': {'key': 'XX'}, 'issuetype': {'name': 'Incident'}, 'priority': {'id': '1'}, 'summary': 'ElastAlert: Nodata flatline Switches - 2023-11-07 23:17 UTC', 'description': 'Index switches* has no data coming in Prod ELK cluster. \n\nNodata flatline Switches\n\nAn abnormally low number of events occurred around 2023-11-07 23:17 UTC.\nBetween 2023-11-07 23:07 UTC and 2023-11-07 23:17 UTC, there were less than 1 events.\n\n{code}{\n    "@timestamp": "2023-11-07T23:17:06.830483Z",\n    "count": 0,\n    "key": "all",\n    "num_hits": 0,\n    "num_matches": 575,\n    "threshold": 1\n}{code}'}): JiraError HTTP None
        text: JiraError HTTP None
        text: No matching user found for: 'XX'



INFO:elastalert:Background alerts thread 0 pending alerts sent at 2023-11-13 22:53 UTC
INFO:elastalert:Queried rule Nodata flatline Switches from 2023-11-13 22:47 UTC to 2023-11-13 22:53 UTC: 0 / 0 hits
INFO:elastalert:Ran Nodata flatline Switches from 2023-11-13 22:47 UTC to 2023-11-13 22:53 UTC: 0 query hits (0 already seen), 0 matches, 0 alerts sent
INFO:elastalert:Nodata flatline Switches range 348
INFO:elastalert:Disabled rules are: ['Nodata flatline - Infoblox']
INFO:elastalert:Sleeping for 59.999794 seconds
WARNING:apscheduler.scheduler:Execution of job "Rule: Nodata flatline - Infoblox (trigger: interval[0:01:00], next run at: 2023-11-13 22:53:57 UTC)" skipped: maximum number of running instances reached (1)
INFO:elastalert:Background configuration change check run at 2023-11-13 22:54 UTC
ERROR:elastalert:Error while running alert jira: Error creating Jira ticket using jira_args ({'project': {'key': 'XX'}, 'issuetype': {'name': 'Incident'}, 'priority': {'id': '1'}, 'summary': 'ElastAlert: Nodata flatline Switches - 2023-11-07 23:17 UTC', 'description': 'Index switches* has no data coming in Prod ELK cluster. \n\nNodata flatline Switches\n\nAn abnormally low number of events occurred around 2023-11-07 23:17 UTC.\nBetween 2023-11-07 23:07 UTC and 2023-11-07 23:17 UTC, there were less than 1 events.\n\n{code}{\n    "@timestamp": "2023-11-07T23:17:06.830483Z",\n    "count": 0,\n    "key": "all",\n    "num_hits": 0,\n    "num_matches": 575,\n    "threshold": 1\n}{code}'}): JiraError HTTP None
        text: JiraError HTTP None
        text: No matching user found for: 'XX'


INFO:elastalert:Background alerts thread 0 pending alerts sent at 2023-11-13 22:54 UTC
INFO:elastalert:Queried rule Nodata flatline Switches from 2023-11-13 22:47 UTC to 2023-11-13 22:54 UTC: 0 / 0 hits
INFO:elastalert:Ran Nodata flatline Switches from 2023-11-13 22:47 UTC to 2023-11-13 22:54 UTC: 0 query hits (0 already seen), 0 matches, 0 alerts sent
INFO:elastalert:Nodata flatline Switches range 411
INFO:elastalert:Disabled rules are: ['Nodata flatline - Infoblox']
INFO:elastalert:Sleeping for 59.999863 seconds
WARNING:apscheduler.scheduler:Execution of job "Rule: Nodata flatline - Infoblox (trigger: interval[0:01:00], next run at: 2023-11-13 22:55:01 UTC)" skipped: maximum number of running instances reached (1)
INFO:elastalert:Background configuration change check run at 2023-11-13 22:55 UTC
ERROR:elastalert:Error while running alert jira: Error creating Jira ticket using jira_args ({'project': {'key': 'XX'}, 'issuetype': {'name': 'Incident'}, 'priority': {'id': '1'}, 'summary': 'ElastAlert: Nodata flatline Switches - 2023-11-07 23:17 UTC', 'description': 'Index switches* has no data coming in Prod ELK cluster. \n\nNodata flatline Switches\n\nAn abnormally low number of events occurred around 2023-11-07 23:17 UTC.\nBetween 2023-11-07 23:07 UTC and 2023-11-07 23:17 UTC, there were less than 1 events.\n\n{code}{\n    "@timestamp": "2023-11-07T23:17:06.830483Z",\n    "count": 0,\n    "key": "all",\n    "num_hits": 0,\n    "num_matches": 575,\n    "threshold": 1\n}{code}'}): JiraError HTTP None
        text: JiraError HTTP None
        text: No matching user found for: 'XX'


INFO:elastalert:Background alerts thread 0 pending alerts sent at 2023-11-13 22:55 UTC
INFO:elastalert:Queried rule Nodata flatline Switches from 2023-11-13 22:47 UTC to 2023-11-13 22:55 UTC: 0 / 0 hits
INFO:elastalert:Ran Nodata flatline Switches from 2023-11-13 22:47 UTC to 2023-11-13 22:55 UTC: 0 query hits (0 already seen), 0 matches, 0 alerts sent
INFO:elastalert:Nodata flatline Switches range 473
INFO:elastalert:Disabled rules are: ['Nodata flatline - Infoblox']
INFO:elastalert:Sleeping for 59.999776 seconds
`

Created same rule for another index switches* which is not active at the moment for testing and i see duplicate alert being created on Jira with exactly matching message. Please see below -

[jertel]

If a rule encounters an error while sending an alert, it will consider the alert to not have been sent. Since the error is from JIRA, and the rule name matches the picture you included, that tells me that the error is causing the repeated alerts. I know you stated to disregard the JIRA error but I can't disregard it since it's tied to the same rule for which you're asking for help.