Variable redefinition revisited

[ronf]

In #2374, I asked about a pyright equivalent for mypy's "allow_redefinition". While there was another way to deal with that issue, there's another pattern I sometimes use that doesn't seem as amenable to this. I basically want to convert a variable from one type to another part with through a code block, with the old type applying for the first part of the block and the new type applying for the remainder of it. As an example, consider the case of loading configuration data from a file:

from pathlib import Path
from typing import Dict

def load_config(config: str) -> Dict[str, object]:
     # In the real code, this would load config from the location passed in
    return {'example': 123}

def test(config: str):
    config = str(Path(config).expanduser())

    config = load_config(config)

    print(config.get('example'))

test('abc')

So, the example function is passed the path of where to load the config and it might do some manipulation of that path, but at a certain point in the function it will call load_config(), after which the type of "config" needs to be a Dict rather than a str. This works fine in mypy (with or without 'allow_redefinition'), but pyright gives errors:

/tmp/c.py
  /tmp/c.py:11:14 - error: Expression of type "Dict[str, object]" cannot be assigned to declared type "str"
    "Dict[str, object]" is incompatible with "str" (reportGeneralTypeIssues)
  /tmp/c.py:13:18 - error: Cannot access member "get" for type "str"
    Member "get" is unknown (reportGeneralTypeIssues)

I've tried various options using cast() or just declaring the types again, but this doesn't seem to work. Wherever I set the type, it seems to apply to all references to that variable in that scope, both before and after the place where I put the declaration. I'd basically have to break the function in two to make pyright happy.

The simplest answer is "don't reuse the variable", but I have some public APIs that currently work this way, and while I could rename the variable inside each of the API functions that do this, that's a pretty painful change just to make a type checker happy. Is there any way to tell pyright what's happening here?

[erictraut]

When a variable or parameter has a declared (annotated) type, it's the job of a type checker to validate that all values assigned to that variable are compatible with the declared type. This is how type checking works in all languages that I'm familiar with.

The recommended approach is to use a different variable name if you need a different type. You can choose to declare the type of this other variable or let pyright infer its type.

Your example above demonstrates nicely why it's a bad idea to use the same variable name for different uses. The input parameter to test is apparently a path name to a configuration file, so it should be named config_path or simply path, not config.

This version of the function not only type checks, but it's easier to understand because the variable names reflect their real purpose.

def test(config_path: str):
    config_path = str(Path(config_path).expanduser())

    config = load_config(config_path)

    print(config.get("example"))

[ronf]

Thanks for the response, Eric.

If I were starting fresh, that's the approach I would take. However, as I said, the argument name in this case is part of a public API, so changing that from "config" to "config_path" (or actually "config_paths", since the current code accepts either a single path or a sequence of paths) would break that API. I won't have an option to rename this until a roll a new major (incompatible) version.

Renaming the variable inside the functions that accept this argument to something like "loaded_config" would solve the problem, but it would involve a VERY large number of changes, as I would have to touch every single line of code that reads data from the config file, and honestly I think that makes the code less readable.

At this point, my question is just whether there's any way to make the existing code work with pyright without major structural changes. If there isn't, I may simply not be able to support pyright with my current code base.

[erictraut]

No, I don't see a way to make this work without changes in your code.

I would argue that mypy's current behavior is a bug and should be fixed. Even with allow_redefinition, it should not allow value assignments that don't conform to the declared type of a symbol. My understanding is that allow_redefinition was added to work around another (IMO, unfortunate) default behavior in mypy where it treats the inferred type of the first assignment as the "declared" type of a variable even though no such declaration was made by the author. If this bug is ever fixed in mypy, you'll need to fix your code accordingly.

[ronf]

Actually, the example given for "--allow-redefinition" is pretty much my use case:

By default, mypy won’t allow a variable to be redefined with an unrelated type. This flag enables redefinition of a variable with an arbitrary type in some contexts: only redefinitions within the same block and nesting depth as the original definition are allowed. Example where this can be useful:

def process(items: List[str]) -> None:
    # 'items' has type List[str]
    items = [item.split() for item in items]
    # 'items' now has type List[List[str]]
    ...

The variable starts off as one type, and then the data is processed in some way that changes its type for the remainder of the function.

I'll keep looking for a less invasive way to migrate my existing code, but I'm starting to run out of energy to spend on this.