How to make pyright work with complex TypedDicts? i.e. return correct type of inner typed dict instead of 'object'

[owocado]

Hello,

I have a bit unusual use-case of TypedDict like so:

import typing
from collections import deque


SIZE = typing.Literal[128, 256, 512, 1024, 2048, 4096]

class SizeMap(typing.TypedDict):
    cl: int
    lm: float


class Explode(typing.TypedDict):
    gif: typing.Dict[SIZE, SizeMap]
    png: typing.Dict[SIZE, SizeMap]
    webp: typing.Dict[SIZE, SizeMap]


aaaa: typing.Dict[str, Explode] = {
	'avatar': {
		'gif': {},
		'png': {
			128: {'cl': 15711, 'lm': 1414141414.321},
			256: {'cl': 15711, 'lm': 1414141414.321},
			512: {'cl': 15711, 'lm': 1414141414.321},
			1024: {'cl': 15711, 'lm': 1414141414.321},
			2048: {'cl': 15711, 'lm': 1414141414.321},
			4096: {'cl': 15711, 'lm': 1414141414.321},
		},
		'webp': {
			128: {'cl': 15711, 'lm': 1414141414.321},
			256: {'cl': 15711, 'lm': 1414141414.321},
			512: {'cl': 15711, 'lm': 1414141414.321},
			1024: {'cl': 15711, 'lm': 1414141414.321},
			2048: {'cl': 15711, 'lm': 1414141414.321},
			4096: {'cl': 15711, 'lm': 1414141414.321},
		}
	}
}

for arg, inner in aaaa.items():

    a1: deque[int] = deque()
    b1: deque[float] = deque()

    for key, val in inner.items():
		# pyright here says 'val' is of type 'object' 😭
		# shouldn't it return the type of 'val' here as SizeMap ?
        a1.append(val)
	# rest of my logic

Screenshot (please ignore the red error):

My question is, why is pyright here returning the type of 'val' in inner for-loop as 'object'? It should be of typing.Dict[SIZE, SizeMap] type instead. What am I doing wrong here? or is pyright in wrong here?

[erictraut]

Pyright's behavior here is correct. A TypedDict is a structural type that defines certain items within a dictionary. Dictionaries that are compatible with a given TypedDict are allowed to contain additional items beyond what is specified in a TypedDict. When you call the items method, a type checker must assume that additional items are presents, and their values have a type of object.

There is a new proposed feature in draft PEP 728 that introduces the concept of "closed" TypedDicts. Pyright has provisional (experimental) support for this feature if you want to play with it, but I wouldn't recommend using this feature in production code until the PEP has been finalized and accepted by the steering council.

[owocado]

I'm sorry but what you said in first paragraph went over my head, it barely makes sense. Let me break it down for me by each sentence:

A TypedDict is a structural type that defines certain items within a dictionary.

Ok I agree here, this makes sense. This is why I use TypedDicts in general.

Dictionaries that are compatible with a given TypedDict are allowed to contain additional items beyond what is specified in a TypedDict.

But my dictionary does not contain any additional items beyond what I have defined in my two TypedDict classes, namely Explode and SizeMap. Can you please elaborate what are you saying here specifically?

When you call the items method, a type checker must assume that additional items are presents, and their values have a type of object.

Again, I'm not sure what do you mean by additional items? If you have seen the sample of data I posted in OP for variable aaaa, there is no additional key-value pairs in it besides what I have defined in Explode TypedDict class. That's all there is in main dictionary payload and I have verified it on my end multiple times.

Let me write a small example with comments to better explain my case:

# 'aaaa' here is same dict as I've posted in OP
for arg, inner in aaaa.items(): # OK GOOD: pyright shows type here as ` -> dict_items[str, Explode]`
	# if I call it like this, pyright shows all types correctly with autocomplete
	# OK: satisfied
	inner['png'][512]['cl']

	# But if I try to iterate over the items of `inner` dict here, then
	# pyright shows the type of 'val' as 'object' instead of 'typing.Dict[SIZE, SizeMap]'
	# as I have defined in Explode TypedDict class above
	for key, val in inner.items():
		# Type of 'val' here is of type 'object' which doesnt make sense to me.
		# Are you saying 'val' here is addiitonal items beyond what I have specified?
		...

[erictraut]

To understand the difference between a nominal and a structural type, refer to this documentation.

You may also want to read the motivation section of PEP 728. In particular, note the definition of a "closed" TypedDict. Currently, all TypedDict types are assumed to be "open" — and can therefore contain an arbitrary number of keys beyond what is defined in the TypedDict.

Keep in mind that type checkers don't run your code. They perform type analysis based on the type information you've provided. A type checker needs to consider all possible values that satisfy that type.

[owocado]

Alright, I have checked the linked documentation and read it and it helps me understand that my case is a case of structural types and the set of possible values it can/might contain.

Currently, all TypedDict types are assumed to be "open" — and can therefore contain an arbitrary number of keys beyond what is defined in the TypedDict.

I understand the current open nature of TypedDicts and it makes sense theory wise but I still don't understand how this applies with respect to my case and the specific example/code snippet I have shared for context above. Which additional items/arbitrary number of keys are you referring to that are specific to my example/code?
Specifically, why pyright is able to show types correctly as I call it in x[y][z]['z'] fashion but when I iterate over it's items, pyright somehow is unable to tell the correct type? Respectfully, this seems like odd behaviour from pyright.
And what can I do to make pyright show correct type? besides using typing.cast or waiting for the PEP 728 to be merged or using the experimental pyright feature as workaround.

If you can help me understand this in layman's terms, it would be much appreciated. Thanks.

[erictraut]

Respectfully, this seems like odd behaviour from pyright

Pyright is a standards-compliant type checker. This behavior is mandated by the Python typing spec. You'll see the same behavior in other compliant type checkers like mypy.

And what can I do to make pyright show correct type?

Pyright is showing the correct type based on the type information you've provided. If you want pyright to show a different type, then you need to change the way you're defining your types. For example, rather than defining Explode as a TypedDict, you could define it as a type alias to a regular dict:

Explode: TypeAlias = dict[str, dict[SIZE, SizeMap]]

While I don't mind answering questions about the Python type system, you may find that the Python typing forum is a better place to get help with your general typing questions. The questions you're asking here are not specific to pyright.