--filter argument for trace CLI commands to filter on trace log mes…

…sage content. (#1101)

CHANGELOG.md

@@ -3,6 +3,7 @@
 ## Unreleased
 
 - `task_with()` function for creating task variants.
+- Added `--filter` argument to trace CLI commands for filtering on trace log message content.
 - Print model conversations to terminal with `--display=conversation` (was formerly `--trace`, which is now deprecated).
 
 ## v0.3.57 (09 January 2025)

docs/tracing.qmd

@@ -31,6 +31,12 @@ Trace logs are written using [JSON Lines](https://jsonlines.org/) format and are
 inspect trace dump trace-86396.log.gz
 ```
 
+You can also apply a filter to the trace file using the `--filter` argument (which will match log message text case insensitively). For example:
+
+``` bash
+inspect trace dump trace-86396.log.gz --filter model
+```
+
 ## Anomalies {#anomalies}
 
 If an evaluation is running and is not terminating, you can execute the following command to list instances of actions (e.g. model API generates, docker compose commands, tool calls, etc.) that are still running:
@@ -57,6 +63,12 @@ inspect trace anomalies --all
 
 Note that errors and timeouts are not by themselves evidence of problems, since both occur in the normal course of running evaluations (e.g. model generate calls can return errors that are retried and Docker or S3 can also return retryable errors or timeout when they are under heavy load).
 
+As with the `inspect trace dump` command, you can apply a filter when listing anomolies. For example:
+
+```bash
+inspect trace anomalies --filter model
+```
+
 ## Tracing API {#tracing-api}
 
 In addition to the standard set of actions which are trace logged, you can do your own custom trace logging using the `trace_action()` and `trace_message()` APIs. Trace logging is a great way to make sure that logging context is *always captured* (since the last 10 trace logs are always available) without cluttering up the console or eval transcripts.

src/inspect_ai/_cli/trace.py

@@ -62,29 +62,48 @@ def list_command(json: bool) -> None:
 
 @trace_command.command("dump")
 @click.argument("trace-file", type=str, required=False)
-def dump_command(trace_file: str | None) -> None:
+@click.option(
+    "--filter",
+    type=str,
+    help="Filter (applied to trace message field).",
+)
+def dump_command(trace_file: str | None, filter: str | None) -> None:
     """Dump a trace file to stdout (as a JSON array of log records)."""
     trace_file_path = _resolve_trace_file_path(trace_file)
 
     traces = read_trace_file(trace_file_path)
+
+    if filter:
+        filter = filter.lower()
+        traces = [trace for trace in traces if filter in trace.message.lower()]
+
     print(
         to_json(traces, indent=2, exclude_none=True, fallback=lambda _: None).decode()
     )
 
 
 @trace_command.command("anomalies")
 @click.argument("trace-file", type=str, required=False)
+@click.option(
+    "--filter",
+    type=str,
+    help="Filter (applied to trace message field).",
+)
 @click.option(
     "--all",
     is_flag=True,
     default=False,
     help="Show all anomolies including errors and timeouts (by default only still running and cancelled actions are shown).",
 )
-def anomolies_command(trace_file: str | None, all: bool) -> None:
+def anomolies_command(trace_file: str | None, filter: str | None, all: bool) -> None:
     """Look for anomalies in a trace file (never completed or cancelled actions)."""
     trace_file_path = _resolve_trace_file_path(trace_file)
     traces = read_trace_file(trace_file_path)
 
+    if filter:
+        filter = filter.lower()
+        traces = [trace for trace in traces if filter in trace.message.lower()]
+
     # Track started actions
     running_actions: dict[str, ActionTraceRecord] = {}
     canceled_actions: dict[str, ActionTraceRecord] = {}