Generic Type Narrowing Consistency/Scope Question

[GBeauregard]

When helping debug some attempts to use generics by a friend (they really wanted sum types), i came across some behaviors I can't quite explain, but I'm also not really sure are bugs and I've mostly exhausted trying to connect the dots with documentation and examples. Mypy behavior here isn't any less confusing. Hopefully you can help elucidate what's intended behavior (and why) or just a bug.

In the following code sample, why aren't we be allowed to assign A() to x in func? Why does it work for x=y in func but not x=y in func3 where I'm using typeguards on a generic class? Should the type checker be able to use consistency of T to to figure out the type of y in func2?

I'll admit the code is a bit weird, but I can imagine some contrived case where you want the type consistency between x and y (justifying generic).

from typing import Any, Generic, TypeGuard, TypeVar

class A:
    def bar(self): ...
class B: 
    def foo(self): ...
T = TypeVar('T')
class M(Generic[T]): ...

def is_A_M(p: M[Any]) -> TypeGuard[M[A]]: ...

# error (??)
def func(x: T, y: T) -> None:
    if isinstance(x, A) and isinstance(y, A):
        reveal_type(x)  # A
        reveal_type(y)  # A
        x.bar()  # no error
        x = A()  # error
        x = y  # no error

# error
def func2(x: T, y: T) -> None:
    if isinstance(x, A):
        reveal_type(x)  # A
        reveal_type(y)  # T@func3
        y.bar()  # error
        y = x
        y.bar()

# error (??)
def func3(x: M[T], y: M[T]) -> None:
    if is_A_M(x) and is_A_M(y):
        reveal_type(x)  # M[A]
        reveal_type(y)  # M[A]
        x = y  # error

[erictraut]

When you're using a TypeVar, the type must be consistent throughout the scope. The isinstance check in func is checking to see if x and y are subclasses of A, but that doesn't mean they are type A. They could be a class that is derived from A. For that reason x = A() is not allowed.

In func2, the isinstance check is applying type narrowing to the expression x. A type guard narrows only the type of the expression found within the conditional statement. In this case, that expression is x, so only the type of x is narrowed. If you want to narrow the type of y, you would need to include it in the type guard conditional expression. That's just how type guards work.

In func3, the declared type of x is M[T]. You cannot assign a value of type M[A] to it. You may have temporarily narrowed the type of x to M[A] based on the type guard, but its declared type cannot be violated by an assignment.

Incidentally, mypy emits the same errors as pyright.

[GBeauregard]

Mostly less specifically! More about the foundations more broadly than Python that would be needed to make informed opinions on where python or related languages are going or that would be needed to understand subtleties on implementing more exotic features like HKTs. Or maybe this is all something most people learned 'on the job' playing with implementations, as it were.

Also thanks for mentioning PEP 483! It has some information that I always missed when trying to find things in 484..(apparently because I skipped the abstract..😅)

[saaketp]

The behavior of the type checkers for func3 is confusing to me too.
I get @erictraut 's point about the isinstance check for func1 and func2, but I still don't get why assignment in func3 should be a problem.

Here's a simpler version of func3 that also fails type check because of assignment:

from typing import TypeGuard, TypeVar


T = TypeVar("T", str, int)


def is_list_str(p: list[T]) -> TypeGuard[list[str]]:
    return all(isinstance(x, str) for x in p)


def func(x: list[T], y: list[T]) -> None:
    if is_list_str(x) and is_list_str(y):
        y.append("hi")
        x.append("test")
        y[:] = x[:]
        y = x

Here, the appends are allowed and replacing the contents of y with contents of x is also allowed.
I don't see how those can be allowed but the last assignment cannot be allowed.

[erictraut]

The last assignment isn't allowed because the expression x at that point has been narrowed to list[str], and you're attempting to assign it to the variable y which is declared as type list[T]. A value of type list[str] is not assignable to list[T].

Some built-in type guards offer a more sophisticated behavior. For example, the isinstance type guard...

def func(x: T, y: list[T]) -> None:
    if isinstance(x, str):
        reveal_type(x) # str*
        x.lower() # It can act like a 'str'
        y = [x] # And it can act like a 'T'

In this case the revealed type is str*. The asterisk means that this is a "conditional type" — that is, it's a str that is conditioned on T. In effect, it is both type T and type str at the same time. (For more details about conditional types, refer to this documentation.)

Conditional types cannot be used for user-defined type guards. PEP 647 does not place any constraints on the "narrowed" type returned by a user-defined type guard function (in particular, it doesn't require that the "narrowed" type be a subtype of the input type). That means a type checker cannot make assumptions about how the input type maps to the "narrowed" type, which means that conditional types cannot be used. In your sample above, you'll see that the narrowed type of x is list[str], not list[str*]. The relationship between T and str has been lost. That explains why the assignment y = x fails in the above example.

Interestingly, mypy also emits an error for this sample, but there is an apparent bug in the error message:
test.py:16: error: Incompatible types in assignment (expression has type "List[str]", variable has type "List[int]")
I'm not sure where List[int] is coming from!

[GBeauregard]

Interestingly, mypy also emits an error for this sample, but there is an apparent bug in the error message:
test.py:16: error: Incompatible types in assignment (expression has type "List[str]", variable has type "List[int]")
I'm not sure where List[int] is coming from!

This one got me earlier. I think the origin of mypy's error is a pathology of mypy's algorithm. If you reveal the type in the following sample you'll note it gets revealed twice; once with each type, yet it has the exact same error:

from typing import TypeGuard, TypeVar


T = TypeVar("T", str, int)


def is_list_str(p: list[T]) -> TypeGuard[list[str]]:
    return all(isinstance(x, str) for x in p)


def func(x: list[T], y: list[T]) -> None:
    if is_list_str(x):
        reveal_type(y)
        y = x

main.py:13: note: Revealed type is "builtins.list[builtins.str*]"
main.py:13: note: Revealed type is "builtins.list[builtins.int*]"
main.py:14: error: Incompatible types in assignment (expression has type "List[str]", variable has type "List[int]")

So I assume (?) what's happening when you're seeing that error message (even in the other sample) is you're only seeing the int side of the doubled up type in the error (and in actuality it's working similar to pyright)

(In any case it's a horribly confusing error message and could be improved)

[saaketp]

I get that list[str] cannot be assigned to list[T], and I have no problems with x being treated like list[str] instead of list[str*].
But there is extra information in that code, that this assignment will only happen if is_list_str(y) is True. y is treated like list[str] everywhere in that if block, except on the LHS of the assignment, which still seems wrong to me.

I'm thinking that this will not actually cause problems at runtime for any possible values of x or y, but just a limitation of the type checkers. Because if y was not list[str] that line of code will be unreachable, so shouldn't give any errors, and if it is list[str] then that assignment is no problem.
Is this line of thinking wrong? Am I missing of any other possible cases that can cause problems?

On the mypy error, I think that List[int] is because it does double pass of the code with T = str one time and T = int the second time. T = str case doesn't cause errors but T = int case does (though it shouldn't error there too as I said before, because then that code should be unreachable).