Skip to content

Commit

Permalink
Add env vars for configuring constants.
Browse files Browse the repository at this point in the history
  • Loading branch information
aaugustin committed Aug 8, 2024
1 parent e35c15a commit bbb3161
Show file tree
Hide file tree
Showing 17 changed files with 149 additions and 69 deletions.
6 changes: 6 additions & 0 deletions docs/project/changelog.rst
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,12 @@ New features

* Validated compatibility with Python 3.12.

* Added :doc:`environment variables <../reference/variables>` to configure debug
logs, the ``Server`` and ``User-Agent`` headers, as well as security limits.

If you were monkey-patching constants, be aware that they were renamed, which
will break your configuration. You must switch to the environment variables.

12.0
----

Expand Down
1 change: 1 addition & 0 deletions docs/reference/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -85,6 +85,7 @@ These low-level APIs are shared by all implementations.
datastructures
exceptions
types
variables

API stability
-------------
Expand Down
48 changes: 48 additions & 0 deletions docs/reference/variables.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
Environment variables
=====================

Logging
-------

.. envvar:: WEBSOCKETS_MAX_LOG_SIZE

How much of each frame to show in debug logs.

The default value is ``75``.

See the :doc:`logging guide <../topics/logging>` for details.

Security
........

.. envvar:: WEBSOCKETS_SERVER

Server header sent by websockets.

The default value uses the format ``"Python/x.y.z websockets/X.Y"``.

.. envvar:: WEBSOCKETS_USER_AGENT

User-Agent header sent by websockets.

The default value uses the format ``"Python/x.y.z websockets/X.Y"``.

.. envvar:: WEBSOCKETS_MAX_LINE_LENGTH

Maximum length of the request or status line in the opening handshake.

The default value is ``8192``.

.. envvar:: WEBSOCKETS_MAX_NUM_HEADERS

Maximum number of HTTP headers in the opening handshake.

The default value is ``128``.

.. envvar:: WEBSOCKETS_MAX_BODY_SIZE

Maximum size of the body of an HTTP response in the opening handshake.

The default value is ``1_048_576`` (1 MiB).

See the :doc:`security guide <../topics/security>` for details.
4 changes: 4 additions & 0 deletions docs/topics/logging.rst
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,10 @@ Here's how to enable debug logs for development::
level=logging.DEBUG,
)

By default, websockets elides the content of messages to improve readability.
If you want to see more, you can increase the :envvar:`WEBSOCKETS_MAX_LOG_SIZE`
environment variable. The default value is 75.

Furthermore, websockets adds a ``websocket`` attribute to log records, so you
can include additional information about the current connection in logs.

Expand Down
38 changes: 29 additions & 9 deletions docs/topics/security.rst
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
Security
========

.. currentmodule:: websockets

Encryption
----------

Expand All @@ -27,15 +29,33 @@ an amplification factor of 1000 between network traffic and memory usage.
Configuring a server to :doc:`optimize memory usage <memory>` will improve
security in addition to improving performance.

Other limits
------------
HTTP limits
-----------

In the opening handshake, websockets applies limits to the amount of data that
it accepts in order to minimize exposure to denial of service attacks.

The request or status line is limited to 8192 bytes. Each header line, including
the name and value, is limited to 8192 bytes too. No more than 128 HTTP headers
are allowed. When the HTTP response includes a body, it is limited to 1 MiB.

You may change these limits by setting the :envvar:`WEBSOCKETS_MAX_LINE_LENGTH`,
:envvar:`WEBSOCKETS_MAX_NUM_HEADERS`, and :envvar:`WEBSOCKETS_MAX_BODY_SIZE`
environment variables respectively.

Identification
--------------

By default, websockets identifies itself with a ``Server`` or ``User-Agent``
header in the format ``"Python/x.y.z websockets/X.Y"``.

websockets implements additional limits on the amount of data it accepts in
order to minimize exposure to security vulnerabilities.
You can set the ``server_header`` argument of :func:`~server.serve` or the
``user_agent_header`` argument of :func:`~client.connect` to configure another
value. Setting them to :obj:`None` removes the header.

In the opening handshake, websockets limits the number of HTTP headers to 256
and the size of an individual header to 4096 bytes. These limits are 10 to 20
times larger than what's expected in standard use cases. They're hard-coded.
Alternatively, you can set the :envvar:`WEBSOCKETS_SERVER` and
:envvar:`WEBSOCKETS_USER_AGENT` environment variables respectively. Setting them
to an empty string removes the header.

If you need to change these limits, you can monkey-patch the constants in
``websockets.http11``.
If both the argument and the environment variable are set, the argument takes
precedence.
5 changes: 2 additions & 3 deletions src/websockets/asyncio/client.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,7 @@
from ..extensions.base import ClientExtensionFactory
from ..extensions.permessage_deflate import enable_client_permessage_deflate
from ..headers import validate_subprotocols
from ..http import USER_AGENT
from ..http11 import Response
from ..http11 import USER_AGENT, Response
from ..protocol import CONNECTING, Event
from ..typing import LoggerLike, Origin, Subprotocol
from ..uri import parse_uri
Expand Down Expand Up @@ -71,7 +70,7 @@ async def handshake(
self.request = self.protocol.connect()
if additional_headers is not None:
self.request.headers.update(additional_headers)
if user_agent_header is not None:
if user_agent_header:
self.request.headers["User-Agent"] = user_agent_header
self.protocol.send_request(self.request)

Expand Down
11 changes: 5 additions & 6 deletions src/websockets/asyncio/server.py
Original file line number Diff line number Diff line change
Expand Up @@ -20,8 +20,7 @@
from ..extensions.base import ServerExtensionFactory
from ..extensions.permessage_deflate import enable_server_permessage_deflate
from ..headers import validate_subprotocols
from ..http import USER_AGENT
from ..http11 import Request, Response
from ..http11 import SERVER, Request, Response
from ..protocol import CONNECTING, Event
from ..server import ServerProtocol
from ..typing import LoggerLike, Origin, Subprotocol
Expand Down Expand Up @@ -88,7 +87,7 @@ async def handshake(
]
| None
) = None,
server_header: str | None = USER_AGENT,
server_header: str | None = SERVER,
) -> None:
"""
Perform the opening handshake.
Expand Down Expand Up @@ -131,7 +130,7 @@ async def handshake(
assert isinstance(response, Response) # help mypy
self.response = response

if server_header is not None:
if server_header:
self.response.headers["Server"] = server_header

response = None
Expand Down Expand Up @@ -243,7 +242,7 @@ def __init__(
]
| None
) = None,
server_header: str | None = USER_AGENT,
server_header: str | None = SERVER,
open_timeout: float | None = 10,
logger: LoggerLike | None = None,
) -> None:
Expand Down Expand Up @@ -631,7 +630,7 @@ def __init__(
]
| None
) = None,
server_header: str | None = USER_AGENT,
server_header: str | None = SERVER,
compression: str | None = "deflate",
# Timeouts
open_timeout: float | None = 10,
Expand Down
17 changes: 9 additions & 8 deletions src/websockets/frames.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
import dataclasses
import enum
import io
import os
import secrets
import struct
from typing import Callable, Generator, Sequence
Expand Down Expand Up @@ -146,8 +147,8 @@ class Frame:
rsv2: bool = False
rsv3: bool = False

# Monkey-patch if you want to see more in logs. Should be a multiple of 3.
MAX_LOG = 75
# Configure if you want to see more in logs. Should be a multiple of 3.
MAX_LOG_SIZE = int(os.environ.get("WEBSOCKETS_MAX_LOG_SIZE", "75"))

def __str__(self) -> str:
"""
Expand All @@ -166,8 +167,8 @@ def __str__(self) -> str:
# We'll show at most the first 16 bytes and the last 8 bytes.
# Encode just what we need, plus two dummy bytes to elide later.
binary = self.data
if len(binary) > self.MAX_LOG // 3:
cut = (self.MAX_LOG // 3 - 1) // 3 # by default cut = 8
if len(binary) > self.MAX_LOG_SIZE // 3:
cut = (self.MAX_LOG_SIZE // 3 - 1) // 3 # by default cut = 8
binary = b"".join([binary[: 2 * cut], b"\x00\x00", binary[-cut:]])
data = " ".join(f"{byte:02x}" for byte in binary)
elif self.opcode is OP_CLOSE:
Expand All @@ -183,16 +184,16 @@ def __str__(self) -> str:
coding = "text"
except (UnicodeDecodeError, AttributeError):
binary = self.data
if len(binary) > self.MAX_LOG // 3:
cut = (self.MAX_LOG // 3 - 1) // 3 # by default cut = 8
if len(binary) > self.MAX_LOG_SIZE // 3:
cut = (self.MAX_LOG_SIZE // 3 - 1) // 3 # by default cut = 8
binary = b"".join([binary[: 2 * cut], b"\x00\x00", binary[-cut:]])
data = " ".join(f"{byte:02x}" for byte in binary)
coding = "binary"
else:
data = "''"

if len(data) > self.MAX_LOG:
cut = self.MAX_LOG // 3 - 1 # by default cut = 24
if len(data) > self.MAX_LOG_SIZE:
cut = self.MAX_LOG_SIZE // 3 - 1 # by default cut = 24
data = data[: 2 * cut] + "..." + data[-cut:]

metadata = ", ".join(filter(None, [coding, length, non_final]))
Expand Down
9 changes: 0 additions & 9 deletions src/websockets/http.py
Original file line number Diff line number Diff line change
@@ -1,10 +1,8 @@
from __future__ import annotations

import sys
import typing

from .imports import lazy_import
from .version import version as websockets_version


# For backwards compatibility:
Expand All @@ -26,10 +24,3 @@
"read_response": ".legacy.http",
},
)


__all__ = ["USER_AGENT"]


PYTHON_VERSION = "{}.{}".format(*sys.version_info)
USER_AGENT = f"Python/{PYTHON_VERSION} websockets/{websockets_version}"
36 changes: 28 additions & 8 deletions src/websockets/http11.py
Original file line number Diff line number Diff line change
@@ -1,23 +1,43 @@
from __future__ import annotations

import dataclasses
import os
import re
import sys
import warnings
from typing import Callable, Generator

from . import datastructures, exceptions
from .version import version as websockets_version


__all__ = ["SERVER", "USER_AGENT", "Request", "Response"]


PYTHON_VERSION = "{}.{}".format(*sys.version_info)

# User-Agent header for HTTP requests.
USER_AGENT = os.environ.get(
"WEBSOCKETS_USER_AGENT",
f"Python/{PYTHON_VERSION} websockets/{websockets_version}",
)

# Server header for HTTP responses.
SERVER = os.environ.get(
"WEBSOCKETS_SERVER",
f"Python/{PYTHON_VERSION} websockets/{websockets_version}",
)

# Maximum total size of headers is around 128 * 8 KiB = 1 MiB.
MAX_HEADERS = 128
MAX_NUM_HEADERS = int(os.environ.get("WEBSOCKETS_MAX_NUM_HEADERS", "128"))

# Limit request line and header lines. 8KiB is the most common default
# configuration of popular HTTP servers.
MAX_LINE = 8192
MAX_LINE_LENGTH = int(os.environ.get("WEBSOCKETS_MAX_LINE_LENGTH", "8192"))

# Support for HTTP response bodies is intended to read an error message
# returned by a server. It isn't designed to perform large file transfers.
MAX_BODY = 2**20 # 1 MiB
MAX_BODY_SIZE = int(os.environ.get("WEBSOCKETS_MAX_BODY_SIZE", "1_048_576")) # 1 MiB


def d(value: bytes) -> str:
Expand Down Expand Up @@ -258,12 +278,12 @@ def parse(

if content_length is None:
try:
body = yield from read_to_eof(MAX_BODY)
body = yield from read_to_eof(MAX_BODY_SIZE)
except RuntimeError:
raise exceptions.SecurityError(
f"body too large: over {MAX_BODY} bytes"
f"body too large: over {MAX_BODY_SIZE} bytes"
)
elif content_length > MAX_BODY:
elif content_length > MAX_BODY_SIZE:
raise exceptions.SecurityError(
f"body too large: {content_length} bytes"
)
Expand Down Expand Up @@ -309,7 +329,7 @@ def parse_headers(
# We don't attempt to support obsolete line folding.

headers = datastructures.Headers()
for _ in range(MAX_HEADERS + 1):
for _ in range(MAX_NUM_HEADERS + 1):
try:
line = yield from parse_line(read_line)
except EOFError as exc:
Expand Down Expand Up @@ -355,7 +375,7 @@ def parse_line(
"""
try:
line = yield from read_line(MAX_LINE)
line = yield from read_line(MAX_LINE_LENGTH)
except RuntimeError:
raise exceptions.SecurityError("line too long")
# Not mandatory but safe - https://www.rfc-editor.org/rfc/rfc7230.html#section-3.5
Expand Down
4 changes: 2 additions & 2 deletions src/websockets/legacy/client.py
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@
parse_subprotocol,
validate_subprotocols,
)
from ..http import USER_AGENT
from ..http11 import USER_AGENT
from ..typing import ExtensionHeader, LoggerLike, Origin, Subprotocol
from ..uri import WebSocketURI, parse_uri
from .handshake import build_request, check_response
Expand Down Expand Up @@ -307,7 +307,7 @@ async def handshake(
if self.extra_headers is not None:
request_headers.update(self.extra_headers)

if self.user_agent_header is not None:
if self.user_agent_header:
request_headers.setdefault("User-Agent", self.user_agent_header)

self.write_http_request(wsuri.resource_name, request_headers)
Expand Down
Loading

0 comments on commit bbb3161

Please sign in to comment.