Alerting issue with url-based blacklist

[gatrene]

When using a blacklist rule, based on url, detection is correclty performed, but alerting doesn't work. Message TypeError: unhashable type: 'list' occurs.

The entry in my blacklist file is a typical URL like https://bidder.criteo.com/cdb
Here below the full trace

INFO:elastalert:Queried rule url-full from 2023-09-25 11:31 UTC to 2023-10-25 11:31 UTC: 2393 / 2393 hits
Traceback (most recent call last):
  File "/usr/local/bin/elastalert-test-rule", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/usr/local/lib/python3.12/site-packages/elastalert/test_rule.py", line 511, in main
    test_instance.run_rule_test()
  File "/usr/local/lib/python3.12/site-packages/elastalert/test_rule.py", line 501, in run_rule_test
    self.run_elastalert(rule_yaml, conf)
  File "/usr/local/lib/python3.12/site-packages/elastalert/test_rule.py", line 420, in run_elastalert
    client.run_rule(rule, self.endtime, self.starttime)
  File "/usr/local/lib/python3.12/site-packages/elastalert/elastalert.py", line 874, in run_rule
    if not self.run_query(rule, rule['starttime'], tmp_endtime):
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/elastalert/elastalert.py", line 639, in run_query
    rule_inst.add_data(data)
  File "/usr/local/lib/python3.12/site-packages/elastalert/ruletypes.py", line 114, in add_data
    if self.compare(event):
       ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/elastalert/ruletypes.py", line 128, in compare
    if term in self.rules['blacklist']:
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: unhashable type: 'list'

Any insight?

[jertel]

What do you mean when you say "detection is correctly performed"? From the stack trace above it looks like it failed to get into the detection logic.

[gatrene]

I mean that the initial step detection logic works well : the part based on elasticsearch response, with non-null hits returned value.
Then the next phase detection logic failed.
To me, it looks like the logic fails to parse URLs because of the prefixes 'http://' or 'https://'.
When working with domain like 'bidder.criteo.com', blacklist works perfectly fine.

[jertel]

The problem is most likely due to your data record in Elasticsearch containing a list value inside the field set as your compare_key. ElastAlert 2 is expecting a single value, not a list, to be in those records.