Cannot get flatline rule to work, please help #452

mkostechuk · 2021-09-06T08:03:59Z

mkostechuk
Sep 6, 2021

Hello all, I am having an issue with a flatline rule.

Testing the rule works fine, but then when I run elastalert with this rule - it doesn't trigger any alerts.

Please see below the configuration files and the run outputs:

$ cat rules/flatline_test.yaml 

name: Flatline_TEST
type: flatline
index: default_test

threshold: 1

timeframe:
    minutes: 60

filter:
- query_string:
    query: "robotName: robot"

use_count_query: true

doc_type: _doc

alert:
- "elastalert_modules.dtalert.DingTalkAlerter"
dingtalk_webhook: "https://oapi.dingtalk.com/robot/send?access_token=..."
dingtalk_msgtype: text

$ cat config.yaml 

# This is the folder that contains the rule yaml files
# This can also be a list of directories
# Any .yaml file will be loaded as a rule
rules_folder: /etc/elastalert/rules

# How often ElastAlert will query Elasticsearch
# The unit can be anything from weeks to seconds
run_every:
  seconds: 30

# ElastAlert will buffer results from the most recent
# period of time, in case some log sources are not in real time
buffer_time:
  minutes: 15

# The Elasticsearch hostname for metadata writeback
# Note that every rule can have its own Elasticsearch host
es_host: localhost

# The Elasticsearch port
es_port: 9200

# The index on es_host which is used for metadata storage
# This can be a unmapped index, but it is recommended that you run
# elastalert-create-index to set a mapping
writeback_index: elastalert_status

# If an alert fails for some reason, ElastAlert will retry
# sending the alert until this time period has elapsed
alert_time_limit:
  days: 1

$ elastalert-test-rule rules/flatline_test.yaml 

INFO:elastalert:Note: In debug mode, alerts will be logged to console but NOT actually sent.
            To send them but remain verbose, use --verbose instead.
Didn't get any results.
INFO:elastalert:Note: In debug mode, alerts will be logged to console but NOT actually sent.
                To send them but remain verbose, use --verbose instead.
1 rules loaded
INFO:apscheduler.scheduler:Adding job tentatively -- it will be properly scheduled when the scheduler starts
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 14:42 +08 to 2021-09-06 14:43 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 14:43 +08 to 2021-09-06 14:43 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 14:43 +08 to 2021-09-06 14:44 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 14:44 +08 to 2021-09-06 14:44 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 14:44 +08 to 2021-09-06 14:45 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 14:45 +08 to 2021-09-06 14:45 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 14:45 +08 to 2021-09-06 14:46 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 14:46 +08 to 2021-09-06 14:46 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 14:46 +08 to 2021-09-06 14:47 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 14:47 +08 to 2021-09-06 14:47 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 14:47 +08 to 2021-09-06 14:48 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 14:48 +08 to 2021-09-06 14:48 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 14:48 +08 to 2021-09-06 14:49 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 14:49 +08 to 2021-09-06 14:49 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 14:49 +08 to 2021-09-06 14:50 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 14:50 +08 to 2021-09-06 14:50 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 14:50 +08 to 2021-09-06 14:51 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 14:51 +08 to 2021-09-06 14:51 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 14:51 +08 to 2021-09-06 14:52 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 14:52 +08 to 2021-09-06 14:52 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 14:52 +08 to 2021-09-06 14:53 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 14:53 +08 to 2021-09-06 14:53 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 14:53 +08 to 2021-09-06 14:54 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 14:54 +08 to 2021-09-06 14:54 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 14:54 +08 to 2021-09-06 14:55 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 14:55 +08 to 2021-09-06 14:55 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 14:55 +08 to 2021-09-06 14:56 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 14:56 +08 to 2021-09-06 14:56 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 14:56 +08 to 2021-09-06 14:57 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 14:57 +08 to 2021-09-06 14:57 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 14:57 +08 to 2021-09-06 14:58 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 14:58 +08 to 2021-09-06 14:58 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 14:58 +08 to 2021-09-06 14:59 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 14:59 +08 to 2021-09-06 14:59 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 14:59 +08 to 2021-09-06 15:00 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 15:00 +08 to 2021-09-06 15:00 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 15:00 +08 to 2021-09-06 15:01 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 15:01 +08 to 2021-09-06 15:01 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 15:01 +08 to 2021-09-06 15:02 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 15:02 +08 to 2021-09-06 15:02 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 15:02 +08 to 2021-09-06 15:03 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 15:03 +08 to 2021-09-06 15:03 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 15:03 +08 to 2021-09-06 15:04 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 15:04 +08 to 2021-09-06 15:04 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 15:04 +08 to 2021-09-06 15:05 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 15:05 +08 to 2021-09-06 15:05 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 15:05 +08 to 2021-09-06 15:06 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 15:06 +08 to 2021-09-06 15:06 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 15:06 +08 to 2021-09-06 15:07 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 15:07 +08 to 2021-09-06 15:07 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 15:07 +08 to 2021-09-06 15:08 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 15:08 +08 to 2021-09-06 15:08 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 15:08 +08 to 2021-09-06 15:09 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 15:09 +08 to 2021-09-06 15:09 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 15:09 +08 to 2021-09-06 15:10 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 15:10 +08 to 2021-09-06 15:10 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 15:10 +08 to 2021-09-06 15:11 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 15:11 +08 to 2021-09-06 15:11 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 15:11 +08 to 2021-09-06 15:12 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 15:12 +08 to 2021-09-06 15:12 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 15:12 +08 to 2021-09-06 15:13 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 15:13 +08 to 2021-09-06 15:13 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 15:13 +08 to 2021-09-06 15:14 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 15:14 +08 to 2021-09-06 15:14 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 15:14 +08 to 2021-09-06 15:15 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 15:15 +08 to 2021-09-06 15:15 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 15:15 +08 to 2021-09-06 15:16 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 15:16 +08 to 2021-09-06 15:16 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 15:16 +08 to 2021-09-06 15:17 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 15:17 +08 to 2021-09-06 15:17 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 15:17 +08 to 2021-09-06 15:18 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 15:18 +08 to 2021-09-06 15:18 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 15:18 +08 to 2021-09-06 15:19 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 15:19 +08 to 2021-09-06 15:19 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 15:19 +08 to 2021-09-06 15:20 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 15:20 +08 to 2021-09-06 15:20 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 15:20 +08 to 2021-09-06 15:21 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 15:21 +08 to 2021-09-06 15:21 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 15:21 +08 to 2021-09-06 15:22 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 15:22 +08 to 2021-09-06 15:22 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 15:22 +08 to 2021-09-06 15:23 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 15:23 +08 to 2021-09-06 15:23 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 15:23 +08 to 2021-09-06 15:24 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 15:24 +08 to 2021-09-06 15:24 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 15:24 +08 to 2021-09-06 15:25 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 15:25 +08 to 2021-09-06 15:25 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 15:25 +08 to 2021-09-06 15:26 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 15:26 +08 to 2021-09-06 15:26 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 15:26 +08 to 2021-09-06 15:27 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 15:27 +08 to 2021-09-06 15:27 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 15:27 +08 to 2021-09-06 15:28 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 15:28 +08 to 2021-09-06 15:28 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 15:28 +08 to 2021-09-06 15:29 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 15:29 +08 to 2021-09-06 15:29 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 15:29 +08 to 2021-09-06 15:30 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 15:30 +08 to 2021-09-06 15:30 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 15:30 +08 to 2021-09-06 15:31 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 15:31 +08 to 2021-09-06 15:31 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 15:31 +08 to 2021-09-06 15:32 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 15:32 +08 to 2021-09-06 15:32 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 15:32 +08 to 2021-09-06 15:33 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 15:33 +08 to 2021-09-06 15:33 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 15:33 +08 to 2021-09-06 15:34 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 15:34 +08 to 2021-09-06 15:34 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 15:34 +08 to 2021-09-06 15:35 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 15:35 +08 to 2021-09-06 15:35 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 15:35 +08 to 2021-09-06 15:36 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 15:36 +08 to 2021-09-06 15:36 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 15:36 +08 to 2021-09-06 15:37 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 15:37 +08 to 2021-09-06 15:37 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 15:37 +08 to 2021-09-06 15:38 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 15:38 +08 to 2021-09-06 15:38 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 15:38 +08 to 2021-09-06 15:39 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 15:39 +08 to 2021-09-06 15:39 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 15:39 +08 to 2021-09-06 15:40 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 15:40 +08 to 2021-09-06 15:40 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 15:40 +08 to 2021-09-06 15:41 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 15:41 +08 to 2021-09-06 15:41 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 15:41 +08 to 2021-09-06 15:42 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 15:42 +08 to 2021-09-06 15:42 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 15:42 +08 to 2021-09-06 15:43 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 15:43 +08 to 2021-09-06 15:43 +08: 0 hits
INFO:elastalert:Alert for Flatline_TEST at 2021-09-06T07:43:24.216998Z:
INFO:elastalert:Flatline_TEST

An abnormally low number of events occurred around 2021-09-06 15:43 +08.
Between 2021-09-06 14:43 +08 and 2021-09-06 15:43 +08, there were less than 1 events.

@timestamp: 2021-09-06T07:43:24.216998Z
count: 0
key: all
num_hits: 0
num_matches: 4

INFO:elastalert:Ignoring match for silenced rule Flatline_TEST.all
INFO:elastalert:Ignoring match for silenced rule Flatline_TEST.all
INFO:elastalert:Ignoring match for silenced rule Flatline_TEST.all

Would have written the following documents to writeback index (default is elastalert_status):

silence - {'exponent': 0, 'rule_name': 'Flatline_TEST.all', '@timestamp': datetime.datetime(2021, 9, 6, 7, 43, 30, 987135, tzinfo=tzutc()), 'until': datetime.datetime(2021, 9, 6, 7, 44, 30, 987121, tzinfo=tzutc())}

elastalert_status - {'rule_name': 'Flatline_TEST', 'endtime': datetime.datetime(2021, 9, 6, 7, 43, 30, 216998, tzinfo=tzutc()), 'starttime': datetime.datetime(2021, 9, 6, 6, 42, 54, 216998, tzinfo=tzutc()), 'matches': 4, 'hits': 0, '@timestamp': datetime.datetime(2021, 9, 6, 7, 43, 30, 989180, tzinfo=tzutc()), 'time_taken': 0.5851669311523438}

$ elastalert --verbose --rule rules/flatline_test.yaml 

1 rules loaded
INFO:elastalert:Starting up
INFO:elastalert:Disabled rules are: []
INFO:elastalert:Sleeping for 29.999881 seconds
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 15:41 +08 to 2021-09-06 15:41 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 15:41 +08 to 2021-09-06 15:42 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 15:42 +08 to 2021-09-06 15:42 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 15:42 +08 to 2021-09-06 15:43 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 15:43 +08 to 2021-09-06 15:43 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 15:43 +08 to 2021-09-06 15:43 +08: 0 hits
INFO:elastalert:Ran Flatline_TEST from 2021-09-06 15:41 +08 to 2021-09-06 15:43 +08: 0 query hits (0 already seen), 0 matches, 0 alerts sent
INFO:elastalert:Flatline_TEST range 162
INFO:elastalert:Background configuration change check run at 2021-09-06 15:44 +08
INFO:elastalert:Background alerts thread 0 pending alerts sent at 2021-09-06 15:44 +08
INFO:elastalert:Disabled rules are: []
INFO:elastalert:Sleeping for 29.999777 seconds
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 15:43 +08 to 2021-09-06 15:44 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 15:44 +08 to 2021-09-06 15:44 +08: 0 hits
INFO:elastalert:Ran Flatline_TEST from 2021-09-06 15:43 +08 to 2021-09-06 15:44 +08: 0 query hits (0 already seen), 0 matches, 0 alerts sent
INFO:elastalert:Flatline_TEST range 32
INFO:elastalert:Background configuration change check run at 2021-09-06 15:44 +08
INFO:elastalert:Background alerts thread 0 pending alerts sent at 2021-09-06 15:44 +08
INFO:elastalert:Disabled rules are: []
INFO:elastalert:Sleeping for 29.999384 seconds
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 15:44 +08 to 2021-09-06 15:44 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-06 15:44 +08 to 2021-09-06 15:44 +08: 0 hits
INFO:elastalert:Ran Flatline_TEST from 2021-09-06 15:44 +08 to 2021-09-06 15:44 +08: 0 query hits (0 already seen), 0 matches, 0 alerts sent
INFO:elastalert:Flatline_TEST range 34
INFO:elastalert:Background configuration change check run at 2021-09-06 15:45 +08
INFO:elastalert:Background alerts thread 0 pending alerts sent at 2021-09-06 15:45 +08
INFO:elastalert:Disabled rules are: []
INFO:elastalert:Sleeping for 29.999737 seconds
...

Answered by jertel

Sep 6, 2021

Your logs from the elastalert command only show 4 minutes of log data, but you are using a frequency of 50 or 60 minutes. So I have to assume you are not getting alerts even after 50 or 60 minutes either. Is that true? You have to let the app run for the full frequency duration before you will see alerts.

Without knowing what documents exist and at what times it's difficult to troubleshoot. When you search the default_test for robotName: robot in Kibana, are there any hits? If so, how often are they?

I suggest trimming back your frequency to 30 seconds to make it easier to troubleshoot, and then enable debug logging, and possibly es trace logging. The documentation for elastalert2 explain…

View full answer

mkostechuk · 2021-09-06T08:06:20Z

mkostechuk
Sep 6, 2021
Author

also, for some reason, elastalert-test-rule actually finds 4 matches, not one.. hmm

0 replies

mkostechuk · 2021-09-06T09:49:54Z

mkostechuk
Sep 6, 2021
Author

Putting timeframe lower than 50 minutes makes elastalert-test-rule not to see any matches too. I must be missing something...

0 replies

jertel · 2021-09-06T12:05:10Z

jertel
Sep 6, 2021
Maintainer

Your logs from the elastalert command only show 4 minutes of log data, but you are using a frequency of 50 or 60 minutes. So I have to assume you are not getting alerts even after 50 or 60 minutes either. Is that true? You have to let the app run for the full frequency duration before you will see alerts.

Without knowing what documents exist and at what times it's difficult to troubleshoot. When you search the default_test for robotName: robot in Kibana, are there any hits? If so, how often are they?

I suggest trimming back your frequency to 30 seconds to make it easier to troubleshoot, and then enable debug logging, and possibly es trace logging. The documentation for elastalert2 explains how to do this.

2 replies

mkostechuk Sep 7, 2021
Author

Hello Jason, thank you for the answer!

Actually, I didn't realize that I have to let it run for the full duration, I thought it would just query "current time minus timeframe" and start giving me alerts right away. After running for the whole hour it correctly starts sending alerts.

There are no hits for this filter, I put a nonexistent robot as the robotName just to trigger this rule.

A couple more questions related to the same topic:
1/ Why does elastalert find 4 matches for this rule? (as seen in elastalert-test-rule output above)
2/ If I lower the timeframe to 15 minutes, elastalert-test-rule shows no alerts, however, the rule works just fine and elastalert correctly sends alerts. Could you please help me understand why? (see outputs below)

$ cat rules/flatline_test.yaml

name: Flatline_TEST
type: flatline
index: default_test

threshold: 1

timeframe:
    minutes: 15

realert:
    minutes: 30

filter:
- query_string:
    query: "robotName: robot"

use_count_query: true

doc_type: _doc

alert:
- "elastalert_modules.dtalert.DingTalkAlerter"
dingtalk_webhook: "https://oapi.dingtalk.com/robot/send?access_token=..."
dingtalk_msgtype: text

$ elastalert-test-rule rules/flatline_test.yaml

INFO:elastalert:Note: In debug mode, alerts will be logged to console but NOT actually sent.
            To send them but remain verbose, use --verbose instead.
Didn't get any results.
INFO:elastalert:Note: In debug mode, alerts will be logged to console but NOT actually sent.
                To send them but remain verbose, use --verbose instead.
1 rules loaded
INFO:apscheduler.scheduler:Adding job tentatively -- it will be properly scheduled when the scheduler starts
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-07 10:44 +08 to 2021-09-07 10:45 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-07 10:45 +08 to 2021-09-07 10:45 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-07 10:45 +08 to 2021-09-07 10:46 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-07 10:46 +08 to 2021-09-07 10:46 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-07 10:46 +08 to 2021-09-07 10:47 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-07 10:47 +08 to 2021-09-07 10:47 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-07 10:47 +08 to 2021-09-07 10:48 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-07 10:48 +08 to 2021-09-07 10:48 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-07 10:48 +08 to 2021-09-07 10:49 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-07 10:49 +08 to 2021-09-07 10:49 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-07 10:49 +08 to 2021-09-07 10:50 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-07 10:50 +08 to 2021-09-07 10:50 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-07 10:50 +08 to 2021-09-07 10:51 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-07 10:51 +08 to 2021-09-07 10:51 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-07 10:51 +08 to 2021-09-07 10:52 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-07 10:52 +08 to 2021-09-07 10:52 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-07 10:52 +08 to 2021-09-07 10:53 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-07 10:53 +08 to 2021-09-07 10:53 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-07 10:53 +08 to 2021-09-07 10:54 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-07 10:54 +08 to 2021-09-07 10:54 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-07 10:54 +08 to 2021-09-07 10:55 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-07 10:55 +08 to 2021-09-07 10:55 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-07 10:55 +08 to 2021-09-07 10:56 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-07 10:56 +08 to 2021-09-07 10:56 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-07 10:56 +08 to 2021-09-07 10:57 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-07 10:57 +08 to 2021-09-07 10:57 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-07 10:57 +08 to 2021-09-07 10:58 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-07 10:58 +08 to 2021-09-07 10:58 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-07 10:58 +08 to 2021-09-07 10:59 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-07 10:59 +08 to 2021-09-07 10:59 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-07 10:59 +08 to 2021-09-07 11:00 +08: 0 hits

Would have written the following documents to writeback index (default is elastalert_status):

elastalert_status - {'rule_name': 'Flatline_TEST', 'endtime': datetime.datetime(2021, 9, 7, 3, 0, 0, 677179, tzinfo=tzutc()), 'starttime': datetime.datetime(2021, 9, 7, 2, 44, 51, 677179, tzinfo=tzutc()), 'matches': 0, 'hits': 0, '@timestamp': datetime.datetime(2021, 9, 7, 3, 0, 1, 52823, tzinfo=tzutc()), 'time_taken': 0.15590977668762207}

$ elastalert --verbose --rule rules/flatline_test.yaml

1 rules loaded
INFO:elastalert:Starting up
INFO:elastalert:Disabled rules are: []
INFO:elastalert:Sleeping for 29.99989 seconds
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-07 10:36 +08 to 2021-09-07 10:36 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-07 10:36 +08 to 2021-09-07 10:37 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-07 10:37 +08 to 2021-09-07 10:37 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-07 10:37 +08 to 2021-09-07 10:38 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-07 10:38 +08 to 2021-09-07 10:38 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-07 10:38 +08 to 2021-09-07 10:39 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-07 10:39 +08 to 2021-09-07 10:39 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-07 10:39 +08 to 2021-09-07 10:40 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-07 10:40 +08 to 2021-09-07 10:40 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-07 10:40 +08 to 2021-09-07 10:41 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-07 10:41 +08 to 2021-09-07 10:41 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-07 10:41 +08 to 2021-09-07 10:42 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-07 10:42 +08 to 2021-09-07 10:42 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-07 10:42 +08 to 2021-09-07 10:43 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-07 10:43 +08 to 2021-09-07 10:43 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-07 10:43 +08 to 2021-09-07 10:44 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-07 10:44 +08 to 2021-09-07 10:44 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-07 10:44 +08 to 2021-09-07 10:45 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-07 10:45 +08 to 2021-09-07 10:45 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-07 10:45 +08 to 2021-09-07 10:46 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-07 10:46 +08 to 2021-09-07 10:46 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-07 10:46 +08 to 2021-09-07 10:47 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-07 10:47 +08 to 2021-09-07 10:47 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-07 10:47 +08 to 2021-09-07 10:48 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-07 10:48 +08 to 2021-09-07 10:48 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-07 10:48 +08 to 2021-09-07 10:49 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-07 10:49 +08 to 2021-09-07 10:49 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-07 10:49 +08 to 2021-09-07 10:50 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-07 10:50 +08 to 2021-09-07 10:50 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-07 10:50 +08 to 2021-09-07 10:51 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-07 10:51 +08 to 2021-09-07 10:51 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-07 10:51 +08 to 2021-09-07 10:52 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-07 10:52 +08 to 2021-09-07 10:52 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-07 10:52 +08 to 2021-09-07 10:53 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-07 10:53 +08 to 2021-09-07 10:53 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-07 10:53 +08 to 2021-09-07 10:54 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-07 10:54 +08 to 2021-09-07 10:54 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-07 10:54 +08 to 2021-09-07 10:55 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-07 10:55 +08 to 2021-09-07 10:55 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-07 10:55 +08 to 2021-09-07 10:56 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-07 10:56 +08 to 2021-09-07 10:56 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-07 10:56 +08 to 2021-09-07 10:57 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-07 10:57 +08 to 2021-09-07 10:57 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-07 10:57 +08 to 2021-09-07 10:58 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-07 10:58 +08 to 2021-09-07 10:58 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-07 10:58 +08 to 2021-09-07 10:59 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-07 10:59 +08 to 2021-09-07 10:59 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-07 10:59 +08 to 2021-09-07 11:00 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-07 11:00 +08 to 2021-09-07 11:00 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-07 11:00 +08 to 2021-09-07 11:01 +08: 0 hits
INFO:elastalert:Ignoring match for silenced rule Flatline_TEST.all
INFO:elastalert:Ignoring match for silenced rule Flatline_TEST.all
INFO:elastalert:Ignoring match for silenced rule Flatline_TEST.all
INFO:elastalert:Ignoring match for silenced rule Flatline_TEST.all
INFO:elastalert:Ignoring match for silenced rule Flatline_TEST.all
INFO:elastalert:Ignoring match for silenced rule Flatline_TEST.all
INFO:elastalert:Ignoring match for silenced rule Flatline_TEST.all
INFO:elastalert:Ignoring match for silenced rule Flatline_TEST.all
INFO:elastalert:Ignoring match for silenced rule Flatline_TEST.all
INFO:elastalert:Ignoring match for silenced rule Flatline_TEST.all
INFO:elastalert:Ignoring match for silenced rule Flatline_TEST.all
INFO:elastalert:Ignoring match for silenced rule Flatline_TEST.all
INFO:elastalert:Ignoring match for silenced rule Flatline_TEST.all
INFO:elastalert:Ignoring match for silenced rule Flatline_TEST.all
INFO:elastalert:Ignoring match for silenced rule Flatline_TEST.all
INFO:elastalert:Ignoring match for silenced rule Flatline_TEST.all
INFO:elastalert:Ignoring match for silenced rule Flatline_TEST.all
INFO:elastalert:Ignoring match for silenced rule Flatline_TEST.all
INFO:elastalert:Ignoring match for silenced rule Flatline_TEST.all
INFO:elastalert:Ignoring match for silenced rule Flatline_TEST.all
INFO:elastalert:Ignoring match for silenced rule Flatline_TEST.all
INFO:elastalert:Ignoring match for silenced rule Flatline_TEST.all
INFO:elastalert:Ignoring match for silenced rule Flatline_TEST.all
INFO:elastalert:Ignoring match for silenced rule Flatline_TEST.all
INFO:elastalert:Ignoring match for silenced rule Flatline_TEST.all
INFO:elastalert:Ignoring match for silenced rule Flatline_TEST.all
INFO:elastalert:Ignoring match for silenced rule Flatline_TEST.all
INFO:elastalert:Ignoring match for silenced rule Flatline_TEST.all
INFO:elastalert:Ignoring match for silenced rule Flatline_TEST.all
INFO:elastalert:Ignoring match for silenced rule Flatline_TEST.all
INFO:elastalert:Ignoring match for silenced rule Flatline_TEST.all
INFO:elastalert:Ignoring match for silenced rule Flatline_TEST.all
INFO:elastalert:Ignoring match for silenced rule Flatline_TEST.all
INFO:elastalert:Ignoring match for silenced rule Flatline_TEST.all
INFO:elastalert:Ignoring match for silenced rule Flatline_TEST.all
INFO:elastalert:Ignoring match for silenced rule Flatline_TEST.all
INFO:elastalert:Ignoring match for silenced rule Flatline_TEST.all
INFO:elastalert:Ignoring match for silenced rule Flatline_TEST.all
INFO:elastalert:Ignoring match for silenced rule Flatline_TEST.all
INFO:elastalert:Ran Flatline_TEST from 2021-09-07 10:36 +08 to 2021-09-07 11:01 +08: 0 query hits (0 already seen), 40 matches, 1 alerts sent
INFO:elastalert:Flatline_TEST range 1480
INFO:elastalert:Background configuration change check run at 2021-09-07 11:01 +08
INFO:elastalert:Background alerts thread 0 pending alerts sent at 2021-09-07 11:01 +08
INFO:elastalert:Disabled rules are: []
INFO:elastalert:Sleeping for 29.999905 seconds
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-07 11:01 +08 to 2021-09-07 11:01 +08: 0 hits
INFO:elastalert:Queried rule Flatline_TEST from 2021-09-07 11:01 +08 to 2021-09-07 11:01 +08: 0 hits
INFO:elastalert:Ignoring match for silenced rule Flatline_TEST.all
INFO:elastalert:Ignoring match for silenced rule Flatline_TEST.all
INFO:elastalert:Ignoring match for silenced rule Flatline_TEST.all
INFO:elastalert:Ignoring match for silenced rule Flatline_TEST.all
INFO:elastalert:Ran Flatline_TEST from 2021-09-07 11:01 +08 to 2021-09-07 11:01 +08: 0 query hits (0 already seen), 4 matches, 0 alerts sent
INFO:elastalert:Flatline_TEST range 31

jertel Sep 7, 2021
Maintainer

Depending on your timeframe, buffer times, re-alert times, and other factors it's possible to get multiple matches on an alert. Typically ElastAlert2 will auto-silence duplicates until the next valid re-alert time. If you want to dig deeper you should enable DEBUG/trace logging and walk through the results.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cannot get flatline rule to work, please help #452

{{title}}

Replies: 3 comments 2 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Cannot get flatline rule to work, please help #452

mkostechuk Sep 6, 2021

Replies: 3 comments · 2 replies

mkostechuk Sep 6, 2021 Author

mkostechuk Sep 6, 2021 Author

jertel Sep 6, 2021 Maintainer

mkostechuk Sep 7, 2021 Author

jertel Sep 7, 2021 Maintainer

mkostechuk
Sep 6, 2021

Replies: 3 comments 2 replies

mkostechuk
Sep 6, 2021
Author

mkostechuk
Sep 6, 2021
Author

jertel
Sep 6, 2021
Maintainer

mkostechuk Sep 7, 2021
Author

jertel Sep 7, 2021
Maintainer