Elastalert2 with .monitoring indexes

[justkind]

Hi everyone,

Sorry to bother y'all. I had a quick question regarding Elastalert and the .monitoring indexes.
Through my testing, Elastalert doesn't seem to be working with these indexes:

test_monitoring_rule.yaml:

type: frequency
index: .monitoring-es-7-%Y.%m.%d
use_strftime_index: true
num_events: 1
use_count_query: true
timeframe:
  minutes: 15
run_every:
  minutes : 15
## testing with a filter that should alert ##
filter:
- query:
    query_string:
      query: "cluster_state.status:'green'"

Elastalert logs:

INFO elastalert: elastalert: Loaded new rule rules/test_monitoring_rule.yaml
INFO elastalert: elastalert: Queried rule rules/test_monitoring_rule from 2023-10-11 20:51 UTC to 2023-10-11 20:51 UTC: 0 hits
... (same as above, 0 hits for for each query in the 15 minutes)
INFO elastalert: elastalert: Queried rule rules/test_monitoring_rule from 2023-10-11 21:06 UTC to 2023-10-11 21:06 UTC: 0 hits
INFO elastalert: elastalert: rules/test_monitoring_rule range 903
INFO elastalert: elastalert: Ran rules/test_monitoring_rule from 2023-10-11 20:51 UTC to 2023-10-11 21:06 UTC: 0 query hits (0 already seen), 0 matches, 0 alerts sent

I'm assuming it has to do with the . at the front of the index, as I've tried the same filters on other indexes with no issues.
I've tried a few variations of wildcards, escapes, and quotes, but can't seem to get it working.
Would anyone be able to confirm if this is possible? Couldn't find any documentation or discussions on the hidden indexes.
Just trying to setup some simple alerts based on metrics found in the basic monitoring indexes. Any information would be greatly appreciated.

Thanks for your time

Edit:
Apologies, I missed this previous discussion that discusses the same use case: #773
I tried following the same format for the index and filters, but I still can't seem to get a successful alert on this index.

[jertel]

There is nothing in ElastAlert 2 that prevents access to hidden indexes (indexes that begin with a period). Below is my debug log output from querying the hidden .kibana* indices:

2023-10-11 21:28:49,484     INFO        elasticsearch POST http://192.168.41.12:9200/.kibana*/_search?_source_includes=%2A%2Ccreated_at&ignore_unavailable=true&scroll=30s&size=10000 [status:200 request:0.024s]
2023-10-11 21:28:49,484    DEBUG        elasticsearch > {"query":{"bool":{"filter":{"bool":{"must":[{"range":{"created_at":{"gt":"2023-10-11T21:08:49.451792Z","lte":"2023-10-11T21:28:49.451792Z"}}},{"term":{"type":"ui-metric"}}]}}}},"sort":[{"created_at":{"order":"asc"}}]}
2023-10-11 21:28:49,484    DEBUG        elasticsearch < {"_scroll_id":"FGluY2x1ZGVfY29udGV4dF91dWlkDnF1ZXJ5VGhlbkZldGNoAhZBVktCTjNwWlJ0eUIyZzhoYmFERHBBAAAAAAAABYsWcXZKZ25GS1ZTRWF5TzN2QUl2LU43ZxZBVktCTjNwWlJ0eUIyZzhoYmFERHBBAAAAAAAABYoWcXZKZ25GS1ZTRWF5TzN2QUl2LU43Zw==","took":20,"timed_out":false,"_shards":{"total":3,"successful":3,"skipped":1,"failed":0},"hits":{"total":{"value":1,"relation":"eq"},"max_score":null,"hits":[{"_index":".kibana_8.7.1_001","_id":"ui-metric:kibana-user_agent:Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/118.0","_score":null,"_source":{"ui-metric":{"count":1},"type":"ui-metric","references":[],"coreMigrationVersion":"8.7.1","updated_at":"2023-10-11T21:13:49.767Z","created_at":"2023-10-11T21:13:49.767Z"},"sort":[1697058829767]}]}}
2023-10-11 21:28:49,484    DEBUG           elastalert {'_scroll_id': 'FGluY2x1ZGVfY29udGV4dF91dWlkDnF1ZXJ5VGhlbkZldGNoAhZBVktCTjNwWlJ0eUIyZzhoYmFERHBBAAAAAAAABYsWcXZKZ25GS1ZTRWF5TzN2QUl2LU43ZxZBVktCTjNwWlJ0eUIyZzhoYmFERHBBAAAAAAAABYoWcXZKZ25GS1ZTRWF5TzN2QUl2LU43Zw==', 'took': 20, 'timed_out': False, '_shards': {'total': 3, 'successful': 3, 'skipped': 1, 'failed': 0}, 'hits': {'total': {'value': 1, 'relation': 'eq'}, 'max_score': None, 'hits': [{'_index': '.kibana_8.7.1_001', '_id': 'ui-metric:kibana-user_agent:Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/118.0', '_score': None, '_source': {'ui-metric': {'count': 1}, 'type': 'ui-metric', 'references': [], 'coreMigrationVersion': '8.7.1', 'updated_at': '2023-10-11T21:13:49.767Z', 'created_at': '2023-10-11T21:13:49.767Z'}, 'sort': [1697058829767]}]}}
2023-10-11 21:28:49,485     INFO           elastalert Queried rule hidden5 from 2023-10-11 21:08 UTC to 2023-10-11 21:28 UTC: 1 / 1 hits
2023-10-11 21:28:49,490    DEBUG urllib3.connectionpool http://192.168.41.12:9200 "DELETE /_search/scroll HTTP/1.1" 200 32
2023-10-11 21:28:49,491     INFO        elasticsearch DELETE http://192.168.41.12:9200/_search/scroll [status:200 request:0.005s]
2023-10-11 21:28:49,491    DEBUG        elasticsearch > {"scroll_id":["FGluY2x1ZGVfY29udGV4dF91dWlkDnF1ZXJ5VGhlbkZldGNoAhZBVktCTjNwWlJ0eUIyZzhoYmFERHBBAAAAAAAABYsWcXZKZ25GS1ZTRWF5TzN2QUl2LU43ZxZBVktCTjNwWlJ0eUIyZzhoYmFERHBBAAAAAAAABYoWcXZKZ25GS1ZTRWF5TzN2QUl2LU43Zw=="]}
2023-10-11 21:28:49,491    DEBUG        elasticsearch < {"succeeded":true,"num_freed":2}
2023-10-11 21:28:49,498    DEBUG urllib3.connectionpool http://192.168.41.12:9200 "POST /elastalert_silence/_search?_source_includes=until%2Cexponent&size=1 HTTP/1.1" 200 160
2023-10-11 21:28:49,498     INFO        elasticsearch POST http://192.168.41.12:9200/elastalert_silence/_search?_source_includes=until%2Cexponent&size=1 [status:200 request:0.006s]
2023-10-11 21:28:49,498    DEBUG        elasticsearch > {"query":{"term":{"rule_name":"hidden5._silence"}},"sort":{"until":{"order":"desc"}}}
2023-10-11 21:28:49,498    DEBUG        elasticsearch < {"took":1,"timed_out":false,"_shards":{"total":1,"successful":1,"skipped":0,"failed":0},"hits":{"total":{"value":0,"relation":"eq"},"max_score":null,"hits":[]}}
2023-10-11 21:28:49,502    DEBUG urllib3.connectionpool http://192.168.41.12:9200 "POST /elastalert_silence/_search?_source_includes=until%2Cexponent&size=1 HTTP/1.1" 200 160
2023-10-11 21:28:49,502     INFO        elasticsearch POST http://192.168.41.12:9200/elastalert_silence/_search?_source_includes=until%2Cexponent&size=1 [status:200 request:0.004s]
2023-10-11 21:28:49,502    DEBUG        elasticsearch > {"query":{"term":{"rule_name":"hidden5"}},"sort":{"until":{"order":"desc"}}}
2023-10-11 21:28:49,502    DEBUG        elasticsearch < {"took":0,"timed_out":false,"_shards":{"total":1,"successful":1,"skipped":0,"failed":0},"hits":{"total":{"value":0,"relation":"eq"},"max_score":null,"hits":[]}}
2023-10-11 21:28:49,510    DEBUG urllib3.connectionpool http://192.168.41.12:9200 "POST /elastalert_silence/_doc HTTP/1.1" 201 170
2023-10-11 21:28:49,510     INFO        elasticsearch POST http://192.168.41.12:9200/elastalert_silence/_doc [status:201 request:0.008s]
2023-10-11 21:28:49,510    DEBUG        elasticsearch > {"exponent":0,"rule_name":"hidden5","@timestamp":"2023-10-11T21:28:49.503057Z","until":"2023-10-11T21:29:49.503035Z"}
2023-10-11 21:28:49,510    DEBUG        elasticsearch < {"_index":"elastalert_silence","_id":"wnmkIIsBDKtVT4GxcJ9g","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":1,"_primary_term":1}
2023-10-11 21:28:49,511     INFO           elastalert Alert for hidden5 at 2023-10-11T21:13:49.767Z:
2023-10-11 21:28:49,511     INFO           elastalert hidden5

_id: ui-metric:kibana-user_agent:Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/118.0
_index: .kibana_8.7.1_001
coreMigrationVersion: 8.7.1
created_at: 2023-10-11T21:13:49.767Z
num_hits: 1
num_matches: 1
references: []
type: ui-metric
ui-metric: {
    "count": 1
}
updated_at: 2023-10-11T21:13:49.767Z

And the rule file I used:

name: hidden5
type: any

filter:
- term:
    type: ui-metric

timestamp_field: created_at

index: ".kibana*"
alert: debug

and the relevant portion of my config:

run_every:
  seconds: 10

realert:
  minutes: 1
  
buffer_time:
  minutes: 20

And the alerted record as seen in Kibana:

I suggest broadening the index value in the rule yaml to index: .monitoring* first. Do you get hits? If not, try removing the filter completely. If still no hits, enable debug logging in ElastAlert 2 so you have full visibility into the query request and response. Compare that query to the query that Kibana is using internally (use the Inspect popup to view it).

Also, note the following warning logged out by Elasticsearch's driver:

ElasticsearchDeprecationWarning: this request accesses system indices: [.kibana_8.7.1_001, .kibana_task_manager_8.7.1_001], but in a future major version, direct access to system indices will be prevented by default

Perhaps you're using a newer Elastic version that has disabled API access to the hidden indices. My test above is using 8.7.1.

[justkind]

Thanks so much for confirming and for your time in looking and testing this out @jertel!
I will take all of the suggestions you mentioned and keep testing things out.

Appreciate the quick and detailed reply! Hope you have a great day!