Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: calculate Groq audio transcription cost #7601

Open
liuhu opened this issue Jan 7, 2025 · 0 comments · May be fixed by #7610
Open

[Bug]: calculate Groq audio transcription cost #7601

liuhu opened this issue Jan 7, 2025 · 0 comments · May be fixed by #7610
Labels
bug Something isn't working mlops user request

Comments

@liuhu
Copy link

liuhu commented Jan 7, 2025 •
edited
Loading

What happened?

S3 config

litellm_settings:
  drop_params: True
  num_retries: 2
  request_timeout: 600
  telemetry: False
  failure_callback: ["s3"]
  success_callback: ["s3"]
  s3_callback_params:
    s3_bucket_name: xxx
    s3_region_name: xxx
    s3_aws_access_key_id: xxx
    s3_aws_secret_access_key: xxx
    s3_path: litellm-test
    s3_endpoint_url: https://s3.us-west-2.amazonaws.com

1. custom_pricing not working for Groq whisper-large-v3 (stt model)

model_list:
  - model_name: groq/whisper
    litellm_params:
      model: groq/whisper-large-v3
      api_key: os.environ/GROQ_API_KEY
      input_cost_per_second: 0.00185
      output_cost_per_second: 0
    model_info:
      mode: audio_transcription

reasons:

  1. SPECIAL_MODEL_INFO_PARAMS not contains input_cost_per_second , and we also need to add output_cost_per_second for tts model

litellm/litellm/litellm_core_utils/litellm_logging.py

Lines 2560 to 2575 in 55139b8

def use_custom_pricing_for_model(litellm_params: Optional[dict]) -> bool:
if litellm_params is None:
return False
for k, v in litellm_params.items():
if k in SPECIAL_MODEL_INFO_PARAMS and v is not None:
return True
metadata: Optional[dict] = litellm_params.get("metadata", {})
if metadata is None:
return False
model_info: Optional[dict] = metadata.get("model_info", {})
if model_info is not None:
for k, v in model_info.items():
if k in SPECIAL_MODEL_INFO_PARAMS:
return True
return False

litellm/litellm/types/router.py

Lines 373 to 378 in 55139b8

SPECIAL_MODEL_INFO_PARAMS = [
"input_cost_per_token",
"output_cost_per_token",
"input_cost_per_character",
"output_cost_per_character",
]

2. model_prices_and_context_window_backup.json not contains some groq stt models

whisper-large-v3, whisper-large-v3-turbo, distil-whisper-large-v3-en

3. when i fix 1 and 2 on above issues, appear response_cost Exception

curl:

curl --location 'http://litellm.local/v1/audio/transcriptions' \
--header 'Authorization: Bearer sk-xxx' \
--form 'file=@"/test.ogg"' \
--form 'model="groq/whisper"'

some error log for response_cost :
Exception: This model isn't mapped yet. model=whisper-1, custom_llm_provider=groq. Add it here - https://github.com/BerriAI/litellm/blob/main/model_prices_and_context_window.json.\n

reasons:
I use groq whisper-large-v3, but the response_cost log still is model=whisper-1, custom_llm_provider=groq,
becasue groq use OpenAIAudioTranscription and it is hard code to set model name and provider

litellm/litellm/llms/openai/transcriptions/handler.py

Line 145 in 55139b8

hidden_params = {"model": "whisper-1", "custom_llm_provider": "openai"}

4. groq stt miss some unit test

May i create a PR to fix this bug?

Relevant log output

In S3 file:
{
  "id": "51b6e5bd-021e-40b9-9ea5-030a7e887883",
  "trace_id": "4abd3078-da10-40e9-b107-af12f659f481",
  "call_type": "atranscription",
  "cache_hit": null,
  "stream": null,
  "status": "success",
  "custom_llm_provider": "groq",
  "saved_cache_cost": 0,
  "startTime": 1735818392.247911,
  "endTime": 1735818393.023638,
  "completionStartTime": 1735818393.023638,
  "response_time": 0.7757270336151123,
  "model": "whisper-large-v3",
  "metadata": {
    "user_api_key_hash": "acd92d7246d8cbaa71465db0eceea907e8a56d13dd5458e15e73353fae067b4b",
    "user_api_key_alias": "stt-service",
    "user_api_key_team_id": "d70b5ad8-6e95-4352-ae9c-881291ead2e0",
    "user_api_key_org_id": null,
    "user_api_key_user_id": null,
    "user_api_key_team_alias": "flow",
    "spend_logs_metadata": null,
    "requester_ip_address": "",
    "requester_metadata": {},
    "user_api_key_end_user_id": null
  },
  "cache_key": null,
  "response_cost": 0,
  "total_tokens": 0,
  "prompt_tokens": 0,
  "completion_tokens": 0,
  "request_tags": [],
  "end_user": "",
  "api_base": "https://api.groq.com/openai/v1",
  "model_group": "groq/whisper",
  "model_id": "0309cbac7cf03230d0a72980613e1e26d6df45d4aad37c585560f0d41c7687a5",
  "requester_ip_address": "",
  "messages": [
    {
      "role": "user",
      "content": "02151bfb-3382-4c1e-b77c-3aab54e753c4 (1).ogg"
    }
  ],
  "response": {
    "text": " tagliando auto",
    "language": "Italian",
    "task": "transcribe",
    "duration": 3.2,
    "words": null,
    "segments": [
      {
        "id": 0,
        "avg_logprob": -0.18575256,
        "compression_ratio": 0.8181818,
        "end": 3.2,
        "no_speech_prob": 0.07431701,
        "seek": 0,
        "start": 0,
        "temperature": 0,
        "text": " tagliando auto",
        "tokens": [
          50365,
          6162,
          2081,
          1806,
          8399,
          50525
        ]
      }
    ]
  },
  "model_parameters": {},
  "hidden_params": {
    "model_id": "0309cbac7cf03230d0a72980613e1e26d6df45d4aad37c585560f0d41c7687a5",
    "cache_key": null,
    "api_base": "https://api.groq.com/openai/v1",
    "response_cost": null,
    "additional_headers": {}
  },
  "model_map_information": {
    "model_map_key": "whisper-1",
    "model_map_value": null
  },
  "error_str": null,
  "error_information": {
    "error_code": "",
    "error_class": "",
    "llm_provider": ""
  },
  "response_cost_failure_debug_info": {
    "error_str": "This model isn't mapped yet. model=whisper-1, custom_llm_provider=groq. Add it here - https://github.com/BerriAI/litellm/blob/main/model_prices_and_context_window.json.",
    "traceback_str": "Traceback (most recent call last):\n  File \"/usr/local/lib/python3.11/site-packages/litellm/utils.py\", line 4258, in _get_model_info_helper\n    raise ValueError(\nValueError: This model isn't mapped yet. Add it here - https://github.com/BerriAI/litellm/blob/main/model_prices_and_context_window.json\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"/usr/local/lib/python3.11/site-packages/litellm/litellm_core_utils/litellm_logging.py\", line 821, in _response_cost_calculator\n    response_cost = litellm.response_cost_calculator(\n                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.11/site-packages/litellm/cost_calculator.py\", line 807, in response_cost_calculator\n    raise e\n  File \"/usr/local/lib/python3.11/site-packages/litellm/cost_calculator.py\", line 795, in response_cost_calculator\n    response_cost = completion_cost(\n                    ^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.11/site-packages/litellm/cost_calculator.py\", line 745, in completion_cost\n    raise e\n  File \"/usr/local/lib/python3.11/site-packages/litellm/cost_calculator.py\", line 724, in completion_cost\n    ) = cost_per_token(\n        ^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.11/site-packages/litellm/cost_calculator.py\", line 237, in cost_per_token\n    return openai_cost_per_second(\n           ^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.11/site-packages/litellm/llms/openai/cost_calculation.py\", line 94, in cost_per_second\n    model_info = get_model_info(\n                 ^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.11/site-packages/litellm/utils.py\", line 4442, in get_model_info\n    _model_info = _get_model_info_helper(\n                  ^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.11/site-packages/litellm/utils.py\", line 4364, in _get_model_info_helper\n    raise Exception(\nException: This model isn't mapped yet. model=whisper-1, custom_llm_provider=groq. Add it here - https://github.com/BerriAI/litellm/blob/main/model_prices_and_context_window.json.\n",
    "model": "whisper-large-v3",
    "cache_hit": null,
    "custom_llm_provider": "groq",
    "base_model": null,
    "call_type": "atranscription",
    "custom_pricing": false
  },
  "guardrail_information": null
}


Run /health check:
{
  "healthy_endpoints": [],
  "unhealthy_endpoints": [
    {
      "model": "openai/whisper-1",
      "error": "[Errno 2] No such file or directory: '/usr/local/lib/python3.11/site-packages/litellm/llms/OpenAI/../../../tests/gettysburg.wav'\nHave you set 'mode' - https://docs.litellm.ai/docs/proxy/health#embedding-models\nstack trace: Traceback (most recent call last):\n  File \"/usr/local/lib/python3.11/site-packages/litellm/main.py\", line 5222, in ahealth_check\n    response = await openai_chat_completions.ahealth_check(\n               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.11/site-packages/litellm/llms/OpenAI/openai.py\", line 1552, in ahealth_check\n    audio_file = open(file_path, \"rb\")\n                 ^^^^^^^^^^^^^^^^^^^^^\nFileNotFoundError: [Errno 2] No such file or directory: '/usr/local/lib/python3.11/site-packages/litellm/llms/OpenAI/../../../tests/gettysburg.wav'\n"
    },
    {
      "input_cost_per_second": 0,
      "output_cost_per_second": 0.0001,
      "model": "groq/whisper-large-v3",
      "cache": {
        "no-cache": true
      },
      "error": "litellm.BadRequestError: GroqException - {\"error\":{\"message\":\"The model `whisper-large-v3` does not support chat completions\",\"type\":\"invalid_request_error\"}}\n\nHave you set 'mode' - https://docs.litellm.ai/docs/proxy/health#embedding-models\nstack trace: Traceback (most recent call last):\n  File \"/usr/local/lib/python3.11/site-packages/litellm/llms/openai_like/chat/handler.py\", line 203, in acompletion_function\n    response = await client.post(\n               ^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.11/site-packages/litellm/llms/custom_httpx/http_handler.py\", line 224, in post\n    raise e\n  File \"/usr/local/lib/python3.11/site-packages/litellm/llms/custom_httpx/http_handler.py\", line 181, in post\n    response.raise_for_status()\n  File \"/usr/local/lib/python3.11/site-packages/httpx/_models.py\", line 761, in raise_for_status\n    raise HTTPStatusError(message, request=request, response=self)\nhttpx.HTTPStatusError: Client error '400 Bad Request' for url 'https://api.groq.com/openai/v1/chat/completions'\nFor more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/400\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"/usr/local/lib/python3.11/site"
    }
  ],
  "healthy_count": 0,
  "unhealthy_count": 2
}

Are you a ML Ops Team?

Yes

What LiteLLM version are you on ?

v1.56.9

Twitter / LinkedIn details

@Hugo_Liu_X
@liuhu liuhu added the bug Something isn't working label Jan 7, 2025
@liuhu liuhu changed the title [Bug]: calculate groq audio transcription cost [Bug]: calculate Groq audio transcription cost Jan 7, 2025
liuhu pushed a commit to liuhu/litellm that referenced this issue Jan 7, 2025
fix(asr-groq): calculate Groq audio transcription cost (BerriAI#7601)
60b90f7
This was referenced Jan 7, 2025
Closed
Closed
liuhu pushed a commit to liuhu/litellm that referenced this issue Jan 7, 2025
fix(asr-groq): calculate groq audio transcription cost (BerriAI#7601)
52df28b
@liuhu liuhu linked a pull request Jan 7, 2025 that will close this issue
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working mlops user request
Projects
None yet
Milestone
No milestone
1 participant
@liuhu