ApiException pod does not exist

[craigwalton-dsit]

Migrated from internal repo.
Complete stack trace and logs (sensitive) https://github.com/AI-Safety-Institute/aisi-inspect-tools/issues/149
Original date: 25 Oct 2024

During an eval, the an ApiException was raised which caused a task to fail.

    "traceback": "Traceback (most recent call last):
  File \"redacted/.venv/lib/python3.12/site-packages/kubernetes/stream/ws_client.py\", line 528, in websocket_call
    client = WSClient(configuration, url, headers, capture_all, binary=binary)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File \"redacted/.venv/lib/python3.12/site-packages/kubernetes/stream/ws_client.py\", line 68, in __init__
    self.sock = create_websocket(configuration, url, headers)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File \"redacted/.venv/lib/python3.12/site-packages/kubernetes/stream/ws_client.py\", line 494, in create_websocket
    websocket.connect(url, **connect_opt)
  File \"redacted/.venv/lib/python3.12/site-packages/websocket/_core.py\", line 261, in connect
    self.handshake_response = handshake(self.sock, url, *addrs, **options)
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File \"redacted/.venv/lib/python3.12/site-packages/websocket/_handshake.py\", line 65, in handshake
    status, resp = _get_resp_headers(sock)
                   ^^^^^^^^^^^^^^^^^^^^^^^
  File \"redacted/.venv/lib/python3.12/site-packages/websocket/_handshake.py\", line 150, in _get_resp_headers
    raise WebSocketBadStatusException(
websocket._exceptions.WebSocketBadStatusException: Handshake status 404 Not Found -+-+- {'content-length': '18', 'content-type': 'text/plain; charset=utf-8', 'date': 'Fri, 25 Oct 2024 12:35:35 GMT'} -+-+- b'pod does not exist'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File \"redacted/.venv/lib/python3.12/site-packages/inspect_ai/_eval/task/run.py\", line 266, in task_run
    sample_results = await asyncio.gather(*sample_coroutines)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File \"redacted/.venv/lib/python3.12/site-packages/inspect_ai/_eval/task/run.py\", line 431, in task_run_sample
    error = sample_error(ex)
            ^^^^^^^^^^^^^^^^
  File \"redacted/.venv/lib/python3.12/site-packages/inspect_ai/_eval/task/error.py\", line 22, in __call__
    raise ex
  File \"redacted/.venv/lib/python3.12/site-packages/inspect_ai/_eval/task/run.py\", line 423, in task_run_sample
    state = await plan(state, generate)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File \"redacted/.venv/lib/python3.12/site-packages/inspect_ai/solver/_plan.py\", line 106, in __call__
    state = await solver(state, generate)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File \"redacted/.venv/lib/python3.12/site-packages/inspect_ai/solver/_basic_agent.py\", line 184, in solve
    tool_results = await call_tools(state.output.message, state.tools)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File \"redacted/.venv/lib/python3.12/site-packages/inspect_ai/model/_call_tools.py\", line 154, in call_tools
    results = await asyncio.gather(*tasks)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File \"redacted/.venv/lib/python3.12/site-packages/inspect_ai/model/_call_tools.py\", line 74, in call_tool_task
    result = await call_tool(tdefs, message.text, call)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File \"redacted/.venv/lib/python3.12/site-packages/inspect_ai/model/_call_tools.py\", line 196, in call_tool
    result = await tool_def.tool(**arguments)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File \"redacted/.venv/lib/python3.12/site-packages/...bash.py\", line 48, in bash
    result, new_cwd = await run_bash_command(
                      ^^^^^^^^^^^^^^^^^^^^^^^
  File \"redacted/.venv/lib/python3.12/site-packages/...bash.py\", line 98, in run_bash_command
    result = await bash_sandbox.exec([\"bash\", \"-c\", code], timeout=timeout_seconds)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File \"redacted/.venv/lib/python3.12/site-packages/aisitools/k8s_sandbox/sandbox_environment.py\", line 102, in exec
    return await self._pod.exec(cmd, input, cwd, timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File \"redacted/.venv/lib/python3.12/site-packages/aisitools/k8s_sandbox/pod.py\", line 91, in exec
    result = await self._run_asynchronously(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File \"redacted/.venv/lib/python3.12/site-packages/aisitools/k8s_sandbox/pod.py\", line 149, in _run_asynchronously
    return await loop.run_in_executor(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File \"/usr/lib/python3.12/concurrent/futures/thread.py\", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File \"redacted/.venv/lib/python3.12/site-packages/aisitools/k8s_sandbox/pod.py\", line 92, in <lambda>
    lambda: executor.exec(cmd, stdin, cwd, timeout)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File \"redacted/.venv/lib/python3.12/site-packages/aisitools/k8s_sandbox/pod.py\", line 246, in exec
    with self._interactive_shell(timeout) as ws_client:
  File \"/usr/lib/python3.12/contextlib.py\", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File \"redacted/.venv/lib/python3.12/site-packages/aisitools/k8s_sandbox/pod.py\", line 266, in _interactive_shell
    ws_client: WSClient = stream(
                          ^^^^^^^
  File \"redacted/.venv/lib/python3.12/site-packages/kubernetes/stream/stream.py\", line 36, in _websocket_request
    out = api_method(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File \"redacted/.venv/lib/python3.12/site-packages/kubernetes/client/api/core_v1_api.py\", line 994, in connect_get_namespaced_pod_exec
    return self.connect_get_namespaced_pod_exec_with_http_info(name, namespace, **kwargs)  # noqa: E501
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File \"redacted/.venv/lib/python3.12/site-packages/kubernetes/client/api/core_v1_api.py\", line 1101, in connect_get_namespaced_pod_exec_with_http_info
    return self.api_client.call_api(
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File \"redacted/.venv/lib/python3.12/site-packages/kubernetes/client/api_client.py\", line 348, in call_api
    return self.__call_api(resource_path, method,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File \"redacted/.venv/lib/python3.12/site-packages/kubernetes/client/api_client.py\", line 180, in __call_api
    response_data = self.request(
                    ^^^^^^^^^^^^^
  File \"redacted/.venv/lib/python3.12/site-packages/kubernetes/stream/ws_client.py\", line 538, in websocket_call
    raise ApiException(status=0, reason=str(e))
kubernetes.client.exceptions.ApiException: (0)
Reason: Handshake status 404 Not Found -+-+- {'content-length': '18', 'content-type': 'text/plain; charset=utf-8', 'date': 'Fri, 25 Oct 2024 12:35:35 GMT'} -+-+- b'pod does not exist'
",

The key info is "pod does not exist". Looking at kubectl get events agent-env-w3qxf7hy-default-0` for we can see

2024-10-25T12:20:11Z   Normal    agent-env-w3qxf7hy-default-0                  Scheduled               Successfully assigned agent/agent-env-w3qxf7hy-default-0 to ip-192-168-118-142.eu-west-2.compute.internal
2024-10-25T12:20:13Z   Normal    agent-env-w3qxf7hy-default-0                  Created                 Created container resolve-coredns-ip
2024-10-25T12:20:13Z   Normal    agent-env-w3qxf7hy-default-0                  Pulled                  Container image "toolbelt/dig:2024-09-23" already present on machine
2024-10-25T12:20:13Z   Normal    agent-env-w3qxf7hy-default-0                  Started                 Started container resolve-coredns-ip
2024-10-25T12:20:14Z   Normal    agent-env-w3qxf7hy-default-0                  Pulled                  Container image "redacted" already present on machine
2024-10-25T12:20:15Z   Normal    agent-env-w3qxf7hy-default-0                  Created                 Created container default
2024-10-25T12:20:15Z   Normal    agent-env-w3qxf7hy-default-0                  Started                 Started container default
2024-10-25T12:22:27Z   Warning   agent-env-w3qxf7hy-default-0                  NodeNotReady            Node is not ready
2024-10-25T12:27:32Z   Normal    agent-env-w3qxf7hy-default-0                  TaintManagerEviction    Marking for deletion Pod agent/agent-env-w3qxf7hy-default-0
2024-10-25T12:27:32Z   Normal    agent-env-w3qxf7hy-default-0                  Scheduled               Successfully assigned agent/agent-env-w3qxf7hy-default-0 to ip-192-168-103-228.eu-west-2.compute.internal
2024-10-25T12:27:32Z   Normal    agent-env-w3qxf7hy-default                    SuccessfulCreate        create Pod agent-env-w3qxf7hy-default-0 in StatefulSet agent-env-w3qxf7hy-default successful
2024-10-25T12:27:33Z   Normal    agent-env-w3qxf7hy-default-0                  Pulled                  Container image "toolbelt/dig:2024-09-23" already present on machine
2024-10-25T12:27:33Z   Normal    agent-env-w3qxf7hy-default-0                  Created                 Created container resolve-coredns-ip
2024-10-25T12:27:34Z   Normal    agent-env-w3qxf7hy-default-0                  Started                 Started container resolve-coredns-ip
2024-10-25T12:27:34Z   Normal    agent-env-w3qxf7hy-default-0                  Pulled                  Container image "redacted" already present on machine
2024-10-25T12:27:34Z   Normal    agent-env-w3qxf7hy-default-0                  Created                 Created container default
2024-10-25T12:27:35Z   Normal    agent-env-w3qxf7hy-default-0                  Started                 Started container default
2024-10-25T12:35:35Z   Normal    agent-env-w3qxf7hy-default-0                  Killing                 Stopping container default
2024-10-25T12:35:36Z   Normal    agent-env-w3qxf7hy-default-0                  Killing                 Stopping container default

Note the node not ready, after which it looks like the pod was rescheduled.

I don't know the reason the node went not ready.

Lots of evals on old versions of challenges where resource requests were not specified were also running.