Bug: Error handler fails to parse pydantic models

[dacianf]

Expected Behaviour

When I return a Response with a pydantic model in an exception handler function I expect it to be able to serialize it successfully

Current Behaviour

It throws an error: Object of type ErrorModel is not JSON serializable

Code snippet

from typing import List

import requests
from pydantic import BaseModel, Field

from aws_lambda_powertools import Logger, Tracer
from aws_lambda_powertools.event_handler import APIGatewayRestResolver, Response, content_types
from aws_lambda_powertools.utilities.typing import LambdaContext

tracer = Tracer()
logger = Logger()
app = APIGatewayRestResolver(enable_validation=True, debug=True)

class ErrorModel(BaseModel):
    status_code:int
    message: str

@app.exception_handler(ValueError)
def handle_invalid_limit_qs(ex: ValueError) -> Response[ErrorModel]:  # receives exception raised
    metadata = {"path": app.current_event.path, "query_strings": app.current_event.query_string_parameters}
    logger.error(f"Malformed request: {ex}", extra=metadata)

    return Response(
        status_code=400,
        content_type=content_types.APPLICATION_JSON,
        body=ErrorModel(
            status_code=400,
            message="Invalid data"
        ),
    )


class Todo(BaseModel):
    userId: int
    id_: int = Field(alias="id")
    title: str
    completed: bool


@app.get("/todos")
def get_todos() -> Response[List[Todo]]:
    todo = requests.get("https://jsonplaceholder.typicode.com/todos")
    todo.raise_for_status()
    raise ValueError("TestError")
    return Response(
        status_code=200,
        content_type=content_types.APPLICATION_JSON,
        body=todo,
    )


def lambda_handler(event: dict, context: LambdaContext) -> dict:
    return app.resolve(event, context)

Possible Solution

No response

Steps to Reproduce

1. Create a pydantic model for error ErrorModel
2. Create an exception_handler that returns Response[ErrorModel], e.g:

@app.exception_handler(ValueError)
def handle_invalid_limit_qs(ex: ValueError) -> Response[ErrorModel]:  # receives exception raised
    metadata = {"path": app.current_event.path, "query_strings": app.current_event.query_string_parameters}
    logger.error(f"Malformed request: {ex}", extra=metadata)

    return Response(
        status_code=400,
        content_type=content_types.APPLICATION_JSON,
        body=ErrorModel(
            status_code=400,
            message="Invalid data"
        ),
    )

3. Have the function raise a ValueError

Powertools for AWS Lambda (Python) version

latest

AWS Lambda function runtime

3.11

Packaging format used

Lambda Layers

Debugging logs

Response
{
  "errorMessage": "Object of type ErrorModel is not JSON serializable",
  "errorType": "TypeError",
  "requestId": "ddb831d2-d0db-424b-ace3-a4dd295bce90",
  "stackTrace": [
    "  File \"/var/task/lambda_function.py\", line 53, in lambda_handler\n    return app.resolve(event, context)\n",
    "  File \"/opt/python/aws_lambda_powertools/event_handler/api_gateway.py\", line 1737, in resolve\n    response = self._resolve().build(self.current_event, self._cors)\n",
    "  File \"/opt/python/aws_lambda_powertools/event_handler/api_gateway.py\", line 796, in build\n    self.response.body = self.serializer(self.response.body)\n",
    "  File \"/var/lang/lib/python3.11/json/__init__.py\", line 238, in dumps\n    **kw).encode(obj)\n",
    "  File \"/var/lang/lib/python3.11/json/encoder.py\", line 200, in encode\n    chunks = self.iterencode(o, _one_shot=True)\n",
    "  File \"/var/lang/lib/python3.11/json/encoder.py\", line 258, in iterencode\n    return _iterencode(o, 0)\n",
    "  File \"/opt/python/aws_lambda_powertools/shared/json_encoder.py\", line 16, in default\n    return super().default(obj)\n",
    "  File \"/var/lang/lib/python3.11/json/encoder.py\", line 180, in default\n    raise TypeError(f'Object of type {o.__class__.__name__} '\n"
  ]
}

Function Logs
l":"HTTP/1.1","requestId":"id=","requestTime":"04/Mar/2020:19:15:17 +0000","requestTimeEpoch":1583349317135,"resourceId":null,"resourcePath":"/todos/1","stage":"$default"},"pathParameters":null,"stageVariables":null,"body":"","isBase64Encoded":false}
2023-12-05 05:55:54,349 aws_lambda_powertools.event_handler.api_gateway [DEBUG] Converting event to API Gateway REST API contract
[DEBUG]	2023-12-05T05:55:54.349Z	ddb831d2-d0db-424b-ace3-a4dd295bce90	Converting event to API Gateway REST API contract
2023-12-05 05:55:54,349 aws_lambda_powertools.event_handler.api_gateway [DEBUG] Found a registered route. Calling function
[DEBUG]	2023-12-05T05:55:54.349Z	ddb831d2-d0db-424b-ace3-a4dd295bce90	Found a registered route. Calling function
2023-12-05 05:55:54,349 aws_lambda_powertools.event_handler.api_gateway [DEBUG] Building middleware stack: [<aws_lambda_powertools.event_handler.middlewares.openapi_validation.OpenAPIValidationMiddleware object at 0x7f8fce8c10>]
[DEBUG]	2023-12-05T05:55:54.349Z	ddb831d2-d0db-424b-ace3-a4dd295bce90	Building middleware stack: [<aws_lambda_powertools.event_handler.middlewares.openapi_validation.OpenAPIValidationMiddleware object at 0x7f8fce8c10>]
Processing Route:::get_todos (/todos)
Middleware Stack:
=================
OpenAPIValidationMiddleware
_registered_api_adapter
=================
2023-12-05 05:55:54,349 aws_lambda_powertools.event_handler.api_gateway [DEBUG] MiddlewareFrame: [OpenAPIValidationMiddleware] next call chain is OpenAPIValidationMiddleware -> _registered_api_adapter
[DEBUG]	2023-12-05T05:55:54.349Z	ddb831d2-d0db-424b-ace3-a4dd295bce90	MiddlewareFrame: [OpenAPIValidationMiddleware] next call chain is OpenAPIValidationMiddleware -> _registered_api_adapter
2023-12-05 05:55:54,350 aws_lambda_powertools.event_handler.middlewares.openapi_validation [DEBUG] OpenAPIValidationMiddleware handler
[DEBUG]	2023-12-05T05:55:54.350Z	ddb831d2-d0db-424b-ace3-a4dd295bce90	OpenAPIValidationMiddleware handler
2023-12-05 05:55:54,353 aws_lambda_powertools.event_handler.api_gateway [DEBUG] MiddlewareFrame: [_registered_api_adapter] next call chain is _registered_api_adapter -> get_todos
[DEBUG]	2023-12-05T05:55:54.353Z	ddb831d2-d0db-424b-ace3-a4dd295bce90	MiddlewareFrame: [_registered_api_adapter] next call chain is _registered_api_adapter -> get_todos
2023-12-05 05:55:54,353 aws_lambda_powertools.event_handler.api_gateway [DEBUG] Calling API Route Handler: {}
[DEBUG]	2023-12-05T05:55:54.353Z	ddb831d2-d0db-424b-ace3-a4dd295bce90	Calling API Route Handler: {}
{"level":"ERROR","location":"handle_invalid_limit_qs:21","message":"Malformed request: TestError","timestamp":"2023-12-05 05:55:54,657+0000","service":"service_undefined","path":"/todos","query_strings":{},"xray_trace_id":"1-656ebb69-5644e31746b3689523135c90"}
[ERROR] TypeError: Object of type ErrorModel is not JSON serializable
Traceback (most recent call last):
  File "/var/task/lambda_function.py", line 53, in lambda_handler
    return app.resolve(event, context)
  File "/opt/python/aws_lambda_powertools/event_handler/api_gateway.py", line 1737, in resolve
    response = self._resolve().build(self.current_event, self._cors)
  File "/opt/python/aws_lambda_powertools/event_handler/api_gateway.py", line 796, in build
    self.response.body = self.serializer(self.response.body)
  File "/var/lang/lib/python3.11/json/__init__.py", line 238, in dumps
    **kw).encode(obj)
  File "/var/lang/lib/python3.11/json/encoder.py", line 200, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/var/lang/lib/python3.11/json/encoder.py", line 258, in iterencode
    return _iterencode(o, 0)
  File "/opt/python/aws_lambda_powertools/shared/json_encoder.py", line 16, in default
    return super().default(obj)
  File "/var/lang/lib/python3.11/json/encoder.py", line 180, in default
    raise TypeError(f'Object of type {o.__class__.__name__} 'END RequestId: ddb831d2-d0db-424b-ace3-a4dd295bce90
REPORT RequestId: ddb831d2-d0db-424b-ace3-a4dd295bce90	Duration: 389.92 ms	Billed Duration: 390 ms	Memory Size: 128 MB	Max Memory Used: 74 MB	Init Duration: 749.22 ms

[heitorlessa]

hey @dacianf thanks a lot for opening this. remapped as a feature since that was intentional - I'm gonna work on this. Meanwhile, you can workaround by dumping the model as JSON in the body.

Background. When writing the docs before the launch, we noticed there wasn't a mechanism to override the validation error to allow standardization by customers. While we did (RequestValidationError) allow it, the exception handler happens after OpenAPI validation, and therefore it tries to serialize the JSON response as JSON... except it's a Pydantic model here and json.dumps() doesn't understand it; erroring out.

For the time being, I'll treat as a feature, and handle both Pydantic and Dataclasses in the response builder.

[heitorlessa]

Agreed with @leandrodamascena that we should treat as a bug due to hyrum law (someone will depend on this to work implicitly).

PR is up, we just need to test Pydantic v1/v2 compatibility and we should be good to release this so you don't depend on workarounds :-).

[heitorlessa]

@dacianf done! Leandro is out for the day, unless that's a major blocker for you, I'd like to get a second pair of eyes before we merge and make a hotfix release to get this over to you tomorrow.

Let me know otherwise :)

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

This issue is now closed. Please be mindful that future comments are hard for our team to see.

If you need more assistance, please either tag a team member or open a new issue that references this one.

If you wish to keep having a conversation with other community members under this issue feel free to do so.

[heitorlessa]

releasing now - 2.29.0.

[github-actions]

This is now released under 2.29.0 version!