Wrong API Key Spawns Phantom Inference Pipeline Process

[yeldarby]

Search before asking

• I have searched the Inference issues and found no similar bug report.

Bug

If you pass the wrong API key to /inference_pipelines/initialise it will create a zombie process that spits out all sorts of errors indefinitely.

Logs:

[12/26/24 14:52:42] ERROR    Could not handle Command. request_id=02820ce1-9dcb-4dd3-b233-f75c3a74f4a1, error=Could not find requested Roboflow resource. Check that the provided  inference_pipeline_manager.py:517
                             dataset and version are correct, and check that the provided Roboflow API key has the correct permissions., error_type=ErrorType.NOT_FOUND,                                            
                             public_error_message=Requested Roboflow resources (models / workflows etc.) not available or wrong API key used.                                                                       
                             Traceback (most recent call last):                                                                                                                                                     
                               File "/Volumes/Code/inference/inference/core/roboflow_api.py", line 87, in wrapper                                                                                                   
                                 return function(*args, **kwargs)                                                                                                                                                   
                                        ^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                   
                               File "/Volumes/Code/inference/inference/core/roboflow_api.py", line 499, in get_workflow_specification                                                                               
                                 response = _get_from_url(url=api_url)                                                                                                                                              
                                            ^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                              
                               File "/Volumes/Code/inference/inference/core/roboflow_api.py", line 599, in _get_from_url                                                                                            
                                 api_key_safe_raise_for_status(response=response)                                                                                                                                   
                               File "/Volumes/Code/inference/inference/core/utils/requests.py", line 15, in api_key_safe_raise_for_status                                                                           
                                 response.raise_for_status()                                                                                                                                                        
                               File "/Users/yeldarb/venvs/inference-devel/lib/python3.11/site-packages/requests/models.py", line 1024, in raise_for_status                                                          
                                 raise HTTPError(http_error_msg, response=self)                                                                                                                                     
                             requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://api.roboflow.com/roboflow-docs/workflows/clip-frames?api_key=*******                                       
                                                                                                                                                                                                                    
                             The above exception was the direct cause of the following exception:                                                                                                                   
                                                                                                                                                                                                                    
                             Traceback (most recent call last):                                                                                                                                                     
                               File "/Volumes/Code/inference/inference/core/interfaces/stream_manager/manager_app/inference_pipeline_manager.py", line 175, in                                                      
                             _initialise_pipeline                                                                                                                                                                   
                                 self._inference_pipeline = InferencePipeline.init_with_workflow(                                                                                                                   
                                                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                   
                               File "/Volumes/Code/inference/inference/core/utils/function.py", line 35, in wrapper                                                                                                 
                                 return func(*args, **kwargs)                                                                                                                                                       
                                        ^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                       
                               File "/Volumes/Code/inference/inference/core/interfaces/stream/inference_pipeline.py", line 596, in init_with_workflow                                                               
                                 workflow_specification = get_workflow_specification(                                                                                                                               
                                                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                               
                               File "/Volumes/Code/inference/inference/core/roboflow_api.py", line 100, in wrapper                                                                                                  
                                 error_handler(error)                                                                                                                                                               
                               File "/Volumes/Code/inference/inference/core/roboflow_api.py", line 73, in <lambda>                                                                                                  
                                 404: lambda e: raise_from_lambda(                                                                                                                                                  
                                                ^^^^^^^^^^^^^^^^^^                                                                                                                                                  
                               File "/Volumes/Code/inference/inference/core/roboflow_api.py", line 63, in raise_from_lambda                                                                                         
                                 raise exception_type(message) from inner_error                                                                                                                                     
                             inference.core.exceptions.RoboflowAPINotNotFoundError: Could not find requested Roboflow resource. Check that the provided dataset and version are                                     
                             correct, and check that the provided Roboflow API key has the correct permissions.                                                                                                     
                    ERROR    Could not handle Command. request_id=d0ab647dc91a42ef92faa560c88a5051, error=None, error_type=ErrorType.OPERATION_ERROR, public_error_message=Cannot  inference_pipeline_manager.py:517
                             retrieve InferencePipeline status. Try again later - Inference Pipeline not initialised.                                                                                               
                             NoneType: None                                                                                                                                                                         
[12/26/24 14:52:42] ERROR    Error with command handling raised by InferencePipeline Manager. error_type=not_found error_class=RoboflowAPINotNotFoundError error_message=Could not find stream_manager_client.py:342
                             requested Roboflow resource. Check that the provided dataset and version are correct, and check that the provided Roboflow API key has the correct                                     
                             permissions.                                                                                                                                                                           
Traceback (most recent call last):
  File "/Volumes/Code/inference/inference/core/interfaces/http/http_api.py", line 265, in wrapped_route
    return await route(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Volumes/Code/inference/inference/core/interfaces/http/http_api.py", line 1387, in initialise
    return await self.stream_manager_client.initialise_pipeline(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Volumes/Code/inference/inference/core/interfaces/stream_manager/api/stream_manager_client.py", line 132, in initialise_pipeline
    response = await self._handle_command(command=command)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Volumes/Code/inference/inference/core/interfaces/stream_manager/api/stream_manager_client.py", line 213, in _handle_command
    dispatch_error(error_response=response)
  File "/Volumes/Code/inference/inference/core/interfaces/stream_manager/api/stream_manager_client.py", line 348, in dispatch_error
    raise ERRORS_MAPPING[error_type](
inference.core.interfaces.stream_manager.api.errors.ProcessesManagerNotFoundError: Error with command handling raised by InferencePipeline Manager. Error type: not_found. Details: Could not find requested Roboflow resource. Check that the provided dataset and version are correct, and check that the provided Roboflow API key has the correct permissions.
INFO:     127.0.0.1:59145 - "POST /inference_pipelines/initialise HTTP/1.1" 404 Not Found
[12/26/24 14:52:43] ERROR    Could not handle Command. request_id=7ffd4a3afd21479098ee9a5fc5f5e882, error=None, error_type=ErrorType.OPERATION_ERROR, public_error_message=Cannot  inference_pipeline_manager.py:517
                             retrieve InferencePipeline status. Try again later - Inference Pipeline not initialised.                                                                                               
                             NoneType: None                                                                                                                                                                         
[12/26/24 14:52:44] ERROR    Could not handle Command. request_id=0f268db1ab6845d99055293492e6fb00, error=None, error_type=ErrorType.OPERATION_ERROR, public_error_message=Cannot  inference_pipeline_manager.py:517
                             retrieve InferencePipeline status. Try again later - Inference Pipeline not initialised.                                                                                               
                             NoneType: None                                                                                                                                                                         
[12/26/24 14:52:45] ERROR    Could not handle Command. request_id=f4788cc9e67a431292d3a056ae1d30ca, error=None, error_type=ErrorType.OPERATION_ERROR, public_error_message=Cannot  inference_pipeline_manager.py:517
                             retrieve InferencePipeline status. Try again later - Inference Pipeline not initialised.                                                                                               
                             NoneType: None                                                                                                                                                                         
[12/26/24 14:52:46] ERROR    Could not handle Command. request_id=6bb10427411245e5bd6f3c85bbdba14b, error=None, error_type=ErrorType.OPERATION_ERROR, public_error_message=Cannot  inference_pipeline_manager.py:517
                             retrieve InferencePipeline status. Try again later - Inference Pipeline not initialised.                                                                                               
                             NoneType: None                                                                                                                                                                         
[12/26/24 14:52:47] ERROR    Could not handle Command. request_id=3fa4944af1214028b175977df5a8e871, error=None, error_type=ErrorType.OPERATION_ERROR, public_error_message=Cannot  inference_pipeline_manager.py:517
                             retrieve InferencePipeline status. Try again later - Inference Pipeline not initialised.                                                                                               
                             NoneType: None  

Environment

• Inference installed from main branch
• MacOS

Minimal Reproducible Example

Using a public Workflow that doesn't require an API Key but passing an invalid one:

from inference_sdk import InferenceHTTPClient

client = InferenceHTTPClient(
    api_url="http://localhost:9001", # use local inference server
    api_key="invalid" # optional to access your private data and models
)

result = client.start_inference_pipeline_with_workflow(
    video_reference=[0],
    workspace_name="roboflow-docs",
    workflow_id="clip-frames",
    max_fps=max_fps,
    workflows_parameters={
        "prompt": "blurry",
        "threshold": 0.16
    }
)

Additional

No response

Are you willing to submit a PR?

• Yes I'd like to help by submitting a PR!

[farhan0167]

Hi, is anyone working on this? I'd love to make a contribution if this is still unassigned

[grzegorz-roboflow]

Hi @farhan0167 , I have not started working on this issue yet so please feel free to give it a try! Thanks!

[farhan0167]

hi @yeldarby , so I tried running the minimum reproducible as is:

from inference_sdk import InferenceHTTPClient

client = InferenceHTTPClient(
    api_url="http://localhost:9001", # use local inference server
    api_key="invalid" # optional to access your private data and models
)

result = client.start_inference_pipeline_with_workflow(
    video_reference=[0],
    workspace_name="roboflow-docs",
    workflow_id="clip-frames",
    max_fps=max_fps,
    workflows_parameters={
        "prompt": "blurry",
        "threshold": 0.16
    }
)

and I'm getting the following error which I believe is what you are expecting:

InferenceSDKDeprecationWarning: decorate is experimental: Video processing in inference server is under development. Breaking changes are possible.
Traceback (most recent call last):
  File "/Users/farhanishraq/Downloads/No_Code_AI/inference/inference_sdk/http/client.py", line 94, in decorate
    return function(*args, **kwargs)
  File "/Users/farhanishraq/Downloads/No_Code_AI/inference/inference_sdk/http/client.py", line 1850, in start_inference_pipeline_with_workflow
    response.raise_for_status()
  File "/Users/farhanishraq/Downloads/No_Code_AI/inference/venv/lib/python3.10/site-packages/requests/models.py", line 1024, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: http://localhost:9001/inference_pipelines/initialise

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/farhanishraq/Downloads/No_Code_AI/inference/a.py", line 24, in <module>
    result = client.start_inference_pipeline_with_workflow(
  File "/Users/farhanishraq/Downloads/No_Code_AI/inference/inference_sdk/utils/decorators.py", line 56, in wrapper
    return func(*args, **kwargs)
  File "/Users/farhanishraq/Downloads/No_Code_AI/inference/inference_sdk/http/client.py", line 104, in decorate
    raise HTTPCallErrorError(
inference_sdk.http.errors.HTTPCallErrorError: HTTPCallErrorError(description='401 Client Error: Unauthorized for url: http://localhost:9001/inference_pipelines/initialise', api_message='Unauthorized access to roboflow API - check API key and make sure the key is valid for workspace you use. Visit https://docs.roboflow.com/api-reference/authentication#retrieve-an-api-key to learn how to retrieve one.',status_code=401)

Environment

• Same as the one's you've listed
• Python 3.10

Is there anything that I'm missing that I could try out? Is there another public workflow that I could try out that gives you the error?

[grzegorz-roboflow]

Hi @farhan0167 , do you get more output?

@yeldarby suggests we should see infinite chain of lines like the one below:

[12/26/24 14:52:47] ERROR    Could not handle Command. request_id=3fa4944af1214028b175977df5a8e871, error=None, error_type=ErrorType.OPERATION_ERROR, public_error_message=Cannot  inference_pipeline_manager.py:517
                             retrieve InferencePipeline status. Try again later - Inference Pipeline not initialised.                                                                                               
                             NoneType: None  

[farhan0167]

@grzegorz-roboflow yeah I actually tried some different examples, but I wasn't able to reproduce the error with an infinite chain of logs. If i don't provide an api key for the following for example:

from inference_sdk import InferenceHTTPClient

client = InferenceHTTPClient(
    api_url="http://localhost:9001", # use local inference server
    #api_key="<YOUR API KEY>" # optional to access your private data and models
)

result = client.run_workflow(
    workspace_name="roboflow-docs",
    workflow_id="model-comparison",
    images={
        "image": "https://media.roboflow.com/workflows/examples/bleachers.jpg"
    },
    parameters={
        "model1": "yolov8n-640",
        "model2": "yolov11n-640"
    }
)

Then i get a normal response back but when i uncomment the line, and re-run it, I get only:

Traceback (most recent call last):
  File "/Users/farhanishraq/Downloads/No_Code_AI/inference/venv/lib/python3.11/site-packages/inference_sdk/http/client.py", line 94, in decorate
    return function(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/farhanishraq/Downloads/No_Code_AI/inference/venv/lib/python3.11/site-packages/inference_sdk/http/client.py", line 1558, in run_workflow
    return self._run_workflow(
           ^^^^^^^^^^^^^^^^^^^
  File "/Users/farhanishraq/Downloads/No_Code_AI/inference/venv/lib/python3.11/site-packages/inference_sdk/http/client.py", line 1630, in _run_workflow
    api_key_safe_raise_for_status(response=response)
  File "/Users/farhanishraq/Downloads/No_Code_AI/inference/venv/lib/python3.11/site-packages/inference_sdk/http/utils/requests.py", line 21, in api_key_safe_raise_for_status
    response.raise_for_status()
  File "/Users/farhanishraq/Downloads/No_Code_AI/inference/venv/lib/python3.11/site-packages/requests/models.py", line 1024, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: http://localhost:9001/roboflow-docs/workflows/model-comparison

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/farhanishraq/Downloads/No_Code_AI/test/a.py", line 42, in <module>
    result = client.run_workflow(
             ^^^^^^^^^^^^^^^^^^^^
  File "/Users/farhanishraq/Downloads/No_Code_AI/inference/venv/lib/python3.11/site-packages/inference_sdk/http/client.py", line 104, in decorate
    raise HTTPCallErrorError(
inference_sdk.http.errors.HTTPCallErrorError: HTTPCallErrorError(description='401 Client Error: Unauthorized for url: http://localhost:9001/roboflow-docs/workflows/model-comparison', api_message='Unauthorized access to roboflow API - check API key and make sure the key is valid for workspace you use. Visit https://docs.roboflow.com/api-reference/authentication#retrieve-an-api-key to learn how to retrieve one.',status_code=401)