Skipped: maximum number of running instances reached (1)

[Harris-Chan]

hi there,

I try to do a port scan alert with type: cardinality. However, I find that it might happen a case that the rules will be skipped.

Here is my yaml file of rule set. And I have around 12 rules run parallels with simple query.

type: cardinality

timeframe:
  minutes: 1

query_key : [source, dst]
cardinality_field: "dst_Port"
max_cardinality: 5

filter:
- query:
    query_string:
      query: "eventName:Session Allowed"

Below is the warning message received. Any possible method to solve this problem. Or are there any method to handle large amount of data. I have tried to adjust misfire_grace_time and max_threads but seems the problem is still not solved.
WARNING:apscheduler.scheduler:Execution of job "Rule: (rule_name) (trigger: interval[0:05:00], next run at: 2021-09-08 08:02:24 UTC)" skipped: maximum number of running instances reached (1)

Besides, I am not so sure does the scrolling will affect it. Or the scrolling takes too long time so the rules are overtime.
Does scrolling will miss out any alerts too?
INFO:elastalert:Queried rule (rule_name) from 2021-09-08 07:57 UTC to 2021-09-08 08:02 UTC: 40000 / 10000 hits (scrolling..)

Any ideas what might is causing that?

Thanks
Harris

[jertel]

A particular rule can only be executed sequentially, so if the rule query (ES time + data transfer time + ElastAlert2 parse time) exceeds the run interval then your only solution is to improve your query performance. This could require boosting the hardware components of the ES cluster, network, tuning the rule query, or tuning the ES indexing and document ingest strategy.

It's also possible that your queries are simply trying to fetch too much data. Below are two configuration options that might help, but I'm not sure if they will be compatible with a cardinality rule type.

Alternatively, you can also adjust buffer_time to reduce the query window.

[Harris-Chan]

if using max_query_size and buffer, we want to lower the maximum number of query. but if I shorten the buffer time which shorter than run_every and limiting the return query , we will lose some data right?

Apart from this, I am trying to use metric aggregation to solve it. However, I found that the numbers of return is limit to 10000 query. Are there any method to increase the size of the numbers of hits. I have tried max_query_size but seems no different.

[Harris-Chan]

I found that there is a way to change track_total_hits value in elasticsearch. How can I change this value through alert.py or init.py?

[Harris-Chan]

Hi jertel,

Thanks for your help. I might have a try on metric aggregation.
Any ideas to edit the parameter of track_total_hits in the rules.yaml or in the init.py.

Thanks

[nsano-rururu]

@Harris-Chan

As I posted in the discussion below, the current elastalert2 does not implement a setting to change track_total_hits.
#461

[jertel]

If you're interested in submitting a PR to support this option it would be welcomed. This would require the following:

• adjusting a few calls to deprecated_search() in elastalert.py to pass in the track_total_hits arg
• define a default value for this new parameter in the config and also allow rules to override it.
• define this new value in the schema.yaml file
• update the docs to show that this new arg is supported

[nsano-rururu]

As a side note, since it is a change from Elasticsearch 7, it is necessary to add track_total_hits to the argument when Elasticsearch version >= 7.

[Harris-Chan]

Thanks, I think I might try on the cardinality aggregation to check whether this have a better performance.