No matches in Spike Aggregation

[shaigbdb]

I'm trying to create a spike aggregation down rule for the last 30 minutes period compared to the respective period on the previous day.

I wrote a filter to filter out everything except those two 30 minutes periods, which if I understand correctly, should allow me to compare them when the current and reference windows are 1 day long. The query works, and I do see hits in the log, but no matches. I’m also getting results for the buckets when running in Elasticsearch, and I can see the difference when comparing the time windows in Elasticsearch.

I’ve tried both up and down spikes, none of them match (even after decreasing the spike height to a very small size)

I've turned on the debug log - the two queries that appear for each run of the rule look identical, including the range filter.

Any help would be appreciated.

name: Verify Down Spike
type: spike_aggregation
index: indices-*
doc_type: _doc
timestamp_type: unix_ms
filter:
  - query:
      query_string:
        query: "(@timestamp:[now-1d-30m TO now]) AND NOT (@timestamp:[now-1d TO now-30m])"
spike_height: 1.001
spike_type: down
metric_agg_type: value_count
metric_agg_key: _id
query_key: ["field1.keyword", "field2.keyword"]
timeframe:
  days: 1
buffer_time:
  days: 1
alert:
- "slack"
alert_on_new_data: true

possibly Relevant config.yaml settings:

run_every:
  minutes: 1
old_query_limit:
  minutes: 0
buffer_time:
  days: 1
alert_time_limit:
  days: 2
scan_entire_timeframe: true

query as it appears in the log:

{
  "aggs": {
    "bucket_aggs": {
      "aggs": {
        "bucket_aggs": {
          "aggs": {
            "metric__id_value_count": {
              "value_count": {
                "field": "_id"
              }
            }
          },
          "terms": {
            "field": "field1.keyword",
            "min_doc_count": 1,
            "size": 50
          }
        }
      },
      "terms": {
        "field": "field2.keyword",
        "min_doc_count": 1,
        "size": 50
      }
    }
  },
  "query": {
    "bool": {
      "filter": {
        "bool": {
          "must": [
            {
              "range": {
                "@timestamp": {
                  "gt": 1699813095000,
                  "lte": 1699899495000
                }
              }
            },
            {
              "query_string": {
                "query": "(@timestamp:[now-1d-30m TO now]) AND NOT (@timestamp:[now-1d TO now-30m])"
              }
            }
          ]
        }
      }
    }
  }
}

[jertel]

What does the response look like for this type of query? I've not tried to use both a range filter and then also query filter that also does a time based filter, so I'm curious how your ES server is handling this. I tried it on one of my servers and got shard errors.

[shaigbdb]

Sample output at the bottom of the comment.
I debugged Elastalert 2 to try and find the cause.
This seems to be it:

# Don't alert if ref window has not yet been filled for this key AND
if lookup_es_key(event, self.ts_field) - self.first_event[qk][self.ts_field] < self.rules['timeframe'] * 2:
    # ElastAlert has not been running long enough for any alerts OR

Looks like I actually needed to wait until the reference window is filled - meaning that if timeframe (or buffer_time in the case of spike_aggregation?) is 1 day, then alerts would only start after one day.
What I thought happens was that Elastalert actually queries both the current and the reference period and compares them each time - that doesn't seem to be the case?
If so, I guess the query filter can be simplified a bit, since it only needs to capture the last 30 minutes at the time of the query run, rather than encompass the respective 30 minutes in the current and reference windows.

{
  "took" : 78,
  "timed_out" : false,
  "_shards" : {
    "total" : 295,
    "successful" : 295,
    "skipped" : 294,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 2452,
      "relation" : "eq"
    },
    "max_score" : null,
    "hits" : [ ]
  },
  "aggregations" : {
    "bucket_aggs" : {
      "doc_count_error_upper_bound" : 0,
      "sum_other_doc_count" : 0,
      "buckets" : [
        {
          "key" : "1",
          "doc_count" : 1018,
          "bucket_aggs" : {
            "doc_count_error_upper_bound" : 0,
            "sum_other_doc_count" : 0,
            "buckets" : [
              {
                "key" : "X",
                "doc_count" : 1018,
                "metric__id_value_count" : {
                  "value" : 1018
                }
              }
            ]
          }
        },
        {
          "key" : "2",
          "doc_count" : 332,
          "bucket_aggs" : {
            "doc_count_error_upper_bound" : 0,
            "sum_other_doc_count" : 0,
            "buckets" : [
              {
                "key" : "Z",
                "doc_count" : 332,
                "metric__id_value_count" : {
                  "value" : 332
                }
              }
            ]
          }
        },
        {
          "key" : "3",
          "doc_count" : 104,
          "bucket_aggs" : {
            "doc_count_error_upper_bound" : 0,
            "sum_other_doc_count" : 0,
            "buckets" : [
              {
                "key" : "Y",
                "doc_count" : 104,
                "metric__id_value_count" : {
                  "value" : 104
                }
              }
            ]
          }
        },
        {
          "key" : "4",
          "doc_count" : 1,
          "bucket_aggs" : {
            "doc_count_error_upper_bound" : 0,
            "sum_other_doc_count" : 0,
            "buckets" : [
              {
                "key" : "W",
                "doc_count" : 1,
                "metric__id_value_count" : {
                  "value" : 1
                }
              }
            ]
          }
        }
      ]
    }
  }
}

[jertel]

From the docs:

Note that it mentions the ability to override that requirement, which may be useful to you.

[shaigbdb]

I'm not sure I understand how that should happen:
It appears that self.ref_window_filled_once only becomes true after twice the timeframe has passed - alert_on_new_data isn't being considered.

        if lookup_es_key(event, self.ts_field) - self.first_event[qk][self.ts_field] < self.rules['timeframe'] * 2:
            # ElastAlert has not been running long enough for any alerts OR
            if not self.ref_window_filled_once:
                return
            # This rule is not using alert_on_new_data (with query_key) OR
            if not (self.rules.get('query_key') and self.rules.get('alert_on_new_data')):
                return
            # An alert for this qk has recently fired
            if qk in self.skip_checks and lookup_es_key(event, self.ts_field) < self.skip_checks[qk]:
                return
        else:
            self.ref_window_filled_once = True

[jertel]

I see that. Perhaps the code was changed later and no one updated the docs.

[shaigbdb]

Would every change to the rule require the wait period before it's active?

[nsano-rururu]

Yelp/elastalert#40
Yelp/elastalert#41

[jertel]

It appears from the init and init_rule functions that reloaded/changed rules would start fresh. There are a handful of properties that init_rule copies from the old rule into the new rule, but the sliding ref/cur windows isn't in that list.

[shaigbdb]

This indeed ended up being a matter of waiting for the reference window to be filled.
However, there's another issue:
It looks like a value from beyond the reference window is being used, or something even weirder. In this example I got an alert for a spike from 6 to 1 for a buffer_time of 1 - but the current window is 1 day, and the reference window is another day, and the last hits before the "1" are more than two days before it. The correct reference count should've been zero.
I've noticed min_doc_count was not set, which means it was 1 by default, but if I understand correctly it should only apply to the current window, so I'm not sure changing it to zero achieves the correct effect.

Any ideas?

[shaigbdb]

This happens for regular Spike as well, not just spike aggregation.

[jertel]

If you could provide a reproducible test case to show it failing that would help move this along toward getting someone to work on a fix. Ex: a set of command lines to ingest the sample data, and then a rule yaml that consistently reproduces it.