Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

VSCode unable to resolve imports from secondary airflow namespace package in monorepo with duplicate package names #6649

Open
JDarDagran opened this issue Nov 6, 2024 · 16 comments
Assignees
Labels
needs repro Issue has not been reproduced yet waiting for user response Requires more information from user

Comments

@JDarDagran
Copy link

Type: Bug

Behaviour

VSCode successfully recognizes and imports modules from the main airflow package but doesn’t detect imports from the secondary airflow namespace package.

Steps to reproduce:

  1. Clone the Apache Airflow repository: https://github.com/apache/airflow
  2. Install packages in editable mode (I've done it in multiple ways including:
  • uv sync
  • uv pip install -e . -e ./providers -e ./task_sdk
  • pip install -e . -e ./providers -e ./task_sdk --config-setting editable_mode=compat
  1. Open any file that tries to import from airflow.sdk or airflow.providers.

Diagnostic data

Extension version: 2024.18.0
VS Code version: Code 1.95.1 (Universal) (65edc4939843c90c34d61f4ce11704f09d3e5cb6, 2024-10-31T05:14:54.222Z)
OS version: Darwin arm64 23.4.0
Modes:

  • Python version (& distribution if applicable, e.g. Anaconda): 3.9.17
  • Type of virtual environment used (e.g. conda, venv, virtualenv, etc.): Venv
  • Value of the python.languageServer setting: Default
User Settings


languageServer: "Pylance"

 "python.analysis.extraPaths": [
            "${workspaceFolder}/airflow",
            "${workspaceFolder}/task_sdk/src"
            "${workspaceFolder}/providers/src"
        ]

System Info
Item Value
CPUs Apple M1 Pro (10 x 2400)
GPU Status 2d_canvas: enabled
canvas_oop_rasterization: enabled_on
direct_rendering_display_compositor: disabled_off_ok
gpu_compositing: enabled
multiple_raster_threads: enabled_on
opengl: enabled_on
rasterization: enabled
raw_draw: disabled_off_ok
skia_graphite: disabled_off
video_decode: enabled
video_encode: enabled
webgl: enabled
webgl2: enabled
webgpu: enabled
webnn: disabled_off
Load (avg) 6, 6, 6
Memory (System) 16.00GB (0.03GB free)
Process Argv --crash-reporter-id 88217b29-6a1c-4f04-987d-351f65f1d656
Screen Reader no
VM 0%
@karthiknadig karthiknadig transferred this issue from microsoft/vscode-python Nov 6, 2024
@github-actions github-actions bot added the needs repro Issue has not been reproduced yet label Nov 6, 2024
@batkins61
Copy link

batkins61 commented Jan 8, 2025

We have this same problem. We have multiple packages at the root of the workspace the provide modules into the same top level modules, for example:

workspace
  +- pkg1
    +- src
      +- moduleA
        +- <files and modules>
  +- pkg2
    +- src
      +- moduleA
        +- <different files and modules>

Some submodule names under moduleA may exist in both packages, but the py files are different.

We have this in settings.json

"python.analysis.extraPaths": [
  "pkg1/src",
  "pkg2/src"
],

Imports for moduleA and submodules are resolved under pkg1, but not pkg2.

@heejaechang
Copy link
Contributor

Imports for moduleA and submodules are resolved under pkg1, but not pkg2.

not sure I understand you correctly. you mean when there are the 2 same names it always go to the first? or when there is only 1, it doesn't find it?

can you give us full example, rather than just describing it?

...

if you are talking about same name existing in 2 places, what gets resolved is based on order of search path in extra path. since pkg1/src is 1st one, moduleA will be resolved to the one in pkg1. if you change the order to pkg2, then it will be resolved to that one.

and the behavior should be same for runtime. it will get resolved to the one show up first in sys.path.

@heejaechang
Copy link
Contributor

heejaechang commented Jan 8, 2025

@JDarDagran can you show give us log as described in here - https://github.com/microsoft/pylance-release/wiki/Collecting-data-for-an-investigation.#collecting-trace-logs - this will give us some insight on how pylance is configured in your dev env.

...

that said, I think the issue is your mono repo is not recognized correctly to us, especially if you are using multi root workspace with editable install between roots. the log above can help us understand your env.

meanwhile, these documents could help you understand some of how pylance set up workspace and how to manually tweak it for your env.

https://github.com/microsoft/pylance-release/wiki/Opening-Large-Workspaces-in-VS-Code#manually-configure-your-workspace
https://github.com/microsoft/pylance-release/blob/main/docs/settings/python_analysis_extraPaths.md
https://github.com/microsoft/pylance-release/blob/e003ecb6f41e15ff25406a7916e20a5666fab0c3/TROUBLESHOOTING.md#editable-install-modules-not-found

@heejaechang heejaechang added the waiting for user response Requires more information from user label Jan 8, 2025
@batkins61
Copy link

The order of paths in extraPaths does not matter, it always resolves the same modules, in this example the modules under pkg1. If I remove pkg1 from extraPaths, then the module imports for modules under pkg2 resolve. If I add pkg1 back in, module imports for it resolve but not for pkg2. Again, it does not matter what the order in extraPaths, in my case only pkg1 module imports resolve.

Believe me, I have scoured those articles are more regarding this issue.

I will work on getting you a log.

@heejaechang
Copy link
Contributor

our code says the order matter so not sure how it doesn't matter in your case. but here is what I see

Recording.2025-01-08.155701.mp4

and this is the behavior of python interpreter as well. when there are multiple modules with same name in multiple import roots (ex, each folder in sys.path), the first that match will win.

@batkins61
Copy link

batkins61 commented Jan 9, 2025

At the start of the collected log below, pkg1 modules were resolved and its path was the second item in the extraPaths setting. When I removed the pkg1 path from extraPaths, pkg2 modules became resolved and pkg1 modules became unresolved.
Here is the log:

2025-01-08 19:09:57.131 [info] (Client) Pylance async client (2024.12.1) started with python extension (2024.22.2)
2025-01-08 19:11:30.953 [info] [Info  - 7:11:30 PM] (1853817) Setting environmentName for service "sga": "3.11.2 (usr venv)"
2025-01-08 19:11:30.954 [info] [Info  - 7:11:30 PM] (1853817) Setting pythonPath for service "sga": "/bin/python"
2025-01-08 19:11:30.954 [info] [Info  - 7:11:30 PM] (1853817) No include entries specified; assuming /home/abrian/git/sga
2025-01-08 19:11:30.976 [info] [Info  - 7:11:30 PM] (1853817) Assuming Python version 3.11.2.final.0
2025-01-08 19:11:31.080 [info] [Info  - 7:11:31 PM] (1853817) Found 111 source files
2025-01-08 19:11:31.086 [info] [Info  - 7:11:31 PM] (1853817) Setting environmentName for service "storagegrid": "3.11.2 (usr venv)"
2025-01-08 19:11:31.086 [info] [Info  - 7:11:31 PM] (1853817) Setting pythonPath for service "storagegrid": "/bin/python"
2025-01-08 19:11:31.086 [info] [Info  - 7:11:31 PM] (1853817) No include entries specified; assuming /home/abrian/git/storagegrid
2025-01-08 19:11:31.086 [info] [Warn  - 7:11:31 PM] (1853817) stubPath file:///home/abrian/git/storagegrid/.pylance/typings is not a valid directory.
2025-01-08 19:11:31.107 [info] [Info  - 7:11:31 PM] (1853817) Assuming Python version 3.11.2.final.0
2025-01-08 19:11:31.363 [info] [Info  - 7:11:31 PM] (1853817) Found 329 source files
2025-01-08 19:11:31.738 [info] [Info  - 7:11:31 PM] (1853817) Indexer background runner(4) root directory: file:///home/abrian/.vscode-server/extensions/ms-python.vscode-pylance-2024.12.1/dist (index)
2025-01-08 19:11:31.738 [info] [Info  - 7:11:31 PM] (1853817) Indexing(4) started
2025-01-08 19:11:31.989 [info] [Info  - 7:11:31 PM] (1853817) scanned(4) 123 files over 1 exec env
2025-01-08 19:11:32.158 [info] [Info  - 7:11:32 PM] (1853817) indexed(4) 123 files over 1 exec env
2025-01-08 19:11:32.189 [info] [Info  - 7:11:32 PM] (1853817) Indexing finished(4).

@heejaechang
Copy link
Contributor

this is info log, it doesn't have much info. we need trace info as described here - https://github.com/microsoft/pylance-release/wiki/Collecting-data-for-an-investigation.#collecting-trace-logs

@heejaechang
Copy link
Contributor

heejaechang commented Jan 9, 2025

by the way,

2025-01-08 19:11:30.953 [info] [Info  - 7:11:30 PM] (1853817) Setting environmentName for service "sga": "3.11.2 (usr venv)"
2025-01-08 19:11:30.954 [info] [Info  - 7:11:30 PM] (1853817) Setting pythonPath for service "sga": "/bin/python"
2025-01-08 19:11:30.954 [info] [Info  - 7:11:30 PM] (1853817) No include entries specified; assuming /home/abrian/git/sga
2025-01-08 19:11:30.976 [info] [Info  - 7:11:30 PM] (1853817) Assuming Python version 3.11.2.final.0
2025-01-08 19:11:31.080 [info] [Info  - 7:11:31 PM] (1853817) Found 111 source files
2025-01-08 19:11:31.086 [info] [Info  - 7:11:31 PM] (1853817) Setting environmentName for service "storagegrid": "3.11.2 (usr venv)"
2025-01-08 19:11:31.086 [info] [Info  - 7:11:31 PM] (1853817) Setting pythonPath for service "storagegrid": "/bin/python"
2025-01-08 19:11:31.086 [info] [Info  - 7:11:31 PM] (1853817) No include entries specified; assuming /home/abrian/git/storagegrid
2025-01-08 19:11:31.086 [info] [Warn  - 7:11:31 PM] (1853817) stubPath file:///home/abrian/git/storagegrid/.pylance/typings is not a valid directory.
2025-01-08 19:11:31.107 [info] [Info  - 7:11:31 PM] (1853817) Assuming Python version 3.11.2.final.0
2025-01-08 19:11:31.363 [info] [Info  - 7:11:31 PM] (1853817) Found 329 source files

this seems different than your pkg1/2 examples. are you saying pkg1/2 examples works but not your actual one? or did you share wrong log? if it repro with simple example, that would be nice for us to figure out what's going on. if it only repro with your real repo, then log will provide us more insight.

@batkins61
Copy link

batkins61 commented Jan 9, 2025

Yes, pkg1 and pkg2 are examples.

Here's an example with a single path in extraPaths, which resolves some of the imports (e.g. netapp.helpers.client). It also suggests which path to add (Note, the behavior is the same with or without leading ./ in the paths).
Image

When I add the suggested path to resolve the other imports, it becomes the second item in the list. The other imports are now resolved, and the previously resolved imports are not. For example, netapp.helpers.client is no longer resolved, but netapp.networking.firewall is. It even suggests adding the first path in the list to extraPaths, which is, of course, already there.
Image

@batkins61
Copy link

This is what I have set in settings.json, but I still only have info messages in the output.

    "python.analysis.logLevel": "Trace",
    "python.analysis.logTypeEvaluationTime": true,
    "python.analysis.typeEvaluationTimeThreshold": 500,

@batkins61
Copy link

I restarted vscode and got more detail. The restart happened with the following extraPaths:

    "python.analysis.extraPaths": [
        "./container-utils/src/container-common",
        "./docker-utils/src/platform-common"
    ],

Here are attempts to resolve two unresolved imports. Interestingly, it does resolve them, but appears to continue on.

2025-01-09 11:37:23.857 [info] (1861931) [FG]   parsing: file:///home/abrian/git/storagegrid/keepalived-bycastify/src/usr/local/sbin/update_keepalived.py (77ms)
2025-01-09 11:37:23.857 [info] [Info  - 11:37:23 AM] (1861931) Could not import 'netapp.helpers.client' in file '/home/abrian/git/storagegrid/keepalived-bycastify/src/usr/local/sbin/update_keepalived.py'
2025-01-09 11:37:23.857 [info] [Info  - 11:37:23 AM] (1861931)   Looking in stubPath 'file:///home/abrian/git/storagegrid/.pylance/typings'
2025-01-09 11:37:23.857 [info] [Info  - 11:37:23 AM] (1861931)   Attempting to resolve stub package using root path 'file:///home/abrian/git/storagegrid/.pylance/typings'
2025-01-09 11:37:23.857 [info] [Info  - 11:37:23 AM] (1861931)   Attempting to resolve using root path 'file:///home/abrian/git/storagegrid/.pylance/typings'
2025-01-09 11:37:23.857 [info] [Info  - 11:37:23 AM] (1861931)   Looking in root directory of execution environment 'file:///home/abrian/git/storagegrid'
2025-01-09 11:37:23.857 [info] [Info  - 11:37:23 AM] (1861931)   Attempting to resolve stub package using root path 'file:///home/abrian/git/storagegrid'
2025-01-09 11:37:23.857 [info] [Info  - 11:37:23 AM] (1861931)   Attempting to resolve using root path 'file:///home/abrian/git/storagegrid'
2025-01-09 11:37:23.857 [info] [Info  - 11:37:23 AM] (1861931)   Looking in extraPath 'file:///home/abrian/git/storagegrid/container-utils/src/container-common'
2025-01-09 11:37:23.858 [info] [Info  - 11:37:23 AM] (1861931)   Attempting to resolve stub package using root path 'file:///home/abrian/git/storagegrid/container-utils/src/container-common'
2025-01-09 11:37:23.858 [info] [Info  - 11:37:23 AM] (1861931)   Attempting to resolve using root path 'file:///home/abrian/git/storagegrid/container-utils/src/container-common'
2025-01-09 11:37:23.858 [info] [Info  - 11:37:23 AM] (1861931)   Resolved import with file 'file:///home/abrian/git/storagegrid/container-utils/src/container-common/netapp/helpers/__init__.py'
2025-01-09 11:37:23.858 [info] [Info  - 11:37:23 AM] (1861931)   Resolved import with file 'file:///home/abrian/git/storagegrid/container-utils/src/container-common/netapp/helpers/client.py'
2025-01-09 11:37:23.858 [info] [Info  - 11:37:23 AM] (1861931)   Looking in extraPath 'file:///home/abrian/git/storagegrid/docker-utils/src/platform-common'
2025-01-09 11:37:23.858 [info] [Info  - 11:37:23 AM] (1861931)   Attempting to resolve stub package using root path 'file:///home/abrian/git/storagegrid/docker-utils/src/platform-common'
2025-01-09 11:37:23.858 [info] [Info  - 11:37:23 AM] (1861931)   Attempting to resolve using root path 'file:///home/abrian/git/storagegrid/docker-utils/src/platform-common'
2025-01-09 11:37:23.858 [info] [Info  - 11:37:23 AM] (1861931)   Resolved import with file 'file:///home/abrian/git/storagegrid/docker-utils/src/platform-common/netapp/__init__.py'
2025-01-09 11:37:23.858 [info] [Info  - 11:37:23 AM] (1861931)   Looking for typeshed stdlib path
2025-01-09 11:37:23.858 [info] [Info  - 11:37:23 AM] (1861931)   Looking for typeshed stdlib path
2025-01-09 11:37:23.858 [info] [Info  - 11:37:23 AM] (1861931)   Attempting to resolve using root path 'file:///home/abrian/.vscode-server/extensions/ms-python.vscode-pylance-2024.12.1/dist/typeshed-fallback/stdlib'
2025-01-09 11:37:23.858 [info] [Info  - 11:37:23 AM] (1861931)   Typeshed path not found
2025-01-09 11:37:23.858 [info] [Info  - 11:37:23 AM] (1861931)   Finding python search paths
2025-01-09 11:37:23.858 [info] [Info  - 11:37:23 AM] (1861931)   Executing interpreter: '/bin/python'
2025-01-09 11:37:23.858 [info] [Info  - 11:37:23 AM] (1861931)   Skipping '/usr/lib/python311.zip' because it is not a valid directory
2025-01-09 11:37:23.858 [info] [Info  - 11:37:23 AM] (1861931)   Received 4 paths from interpreter
2025-01-09 11:37:23.858 [info] [Info  - 11:37:23 AM] (1861931)     file:///usr/lib/python3.11
2025-01-09 11:37:23.858 [info] [Info  - 11:37:23 AM] (1861931)     file:///usr/lib/python3.11/lib-dynload
2025-01-09 11:37:23.858 [info] [Info  - 11:37:23 AM] (1861931)     file:///usr/local/lib/python3.11/dist-packages
2025-01-09 11:37:23.858 [info] [Info  - 11:37:23 AM] (1861931)     file:///usr/lib/python3/dist-packages
2025-01-09 11:37:23.858 [info] [Info  - 11:37:23 AM] (1861931)   Looking in python search path 'file:///usr/lib/python3.11'
2025-01-09 11:37:23.858 [info] [Info  - 11:37:23 AM] (1861931)   Attempting to resolve stub package using root path 'file:///usr/lib/python3.11'
2025-01-09 11:37:23.858 [info] [Info  - 11:37:23 AM] (1861931)   Attempting to resolve using root path 'file:///usr/lib/python3.11'
2025-01-09 11:37:23.858 [info] [Info  - 11:37:23 AM] (1861931)   Looking in python search path 'file:///usr/lib/python3.11/lib-dynload'
2025-01-09 11:37:23.858 [info] [Info  - 11:37:23 AM] (1861931)   Attempting to resolve stub package using root path 'file:///usr/lib/python3.11/lib-dynload'
2025-01-09 11:37:23.858 [info] [Info  - 11:37:23 AM] (1861931)   Attempting to resolve using root path 'file:///usr/lib/python3.11/lib-dynload'
2025-01-09 11:37:23.858 [info] [Info  - 11:37:23 AM] (1861931)   Looking in python search path 'file:///usr/local/lib/python3.11/dist-packages'
2025-01-09 11:37:23.858 [info] [Info  - 11:37:23 AM] (1861931)   Attempting to resolve stub package using root path 'file:///usr/local/lib/python3.11/dist-packages'
2025-01-09 11:37:23.858 [info] [Info  - 11:37:23 AM] (1861931)   Attempting to resolve using root path 'file:///usr/local/lib/python3.11/dist-packages'
2025-01-09 11:37:23.858 [info] [Info  - 11:37:23 AM] (1861931)   Looking in python search path 'file:///usr/lib/python3/dist-packages'
2025-01-09 11:37:23.858 [info] [Info  - 11:37:23 AM] (1861931)   Attempting to resolve stub package using root path 'file:///usr/lib/python3/dist-packages'
2025-01-09 11:37:23.858 [info] [Info  - 11:37:23 AM] (1861931)   Attempting to resolve using root path 'file:///usr/lib/python3/dist-packages'
2025-01-09 11:37:23.858 [info] [Info  - 11:37:23 AM] (1861931)   Looking in bundled stubs path 'file:///home/abrian/.vscode-server/extensions/ms-python.vscode-pylance-2024.12.1/dist/bundled/stubs'
2025-01-09 11:37:23.858 [info] [Info  - 11:37:23 AM] (1861931)   Attempting to resolve stub package using root path 'file:///home/abrian/.vscode-server/extensions/ms-python.vscode-pylance-2024.12.1/dist/bundled/stubs'
2025-01-09 11:37:23.858 [info] [Info  - 11:37:23 AM] (1861931)   Attempting to resolve using root path 'file:///home/abrian/.vscode-server/extensions/ms-python.vscode-pylance-2024.12.1/dist/bundled/stubs'
2025-01-09 11:37:23.858 [info] [Info  - 11:37:23 AM] (1861931)   Looking for typeshed third-party path
2025-01-09 11:37:23.858 [info] [Info  - 11:37:23 AM] (1861931)   Looking for typeshed stubs path
2025-01-09 11:37:23.858 [info] [Info  - 11:37:23 AM] (1861931)   Typeshed path not found
2025-01-09 11:37:23.858 [info] [Info  - 11:37:23 AM] (1861931) Could not import 'netapp.networking.link' in file '/home/abrian/git/storagegrid/keepalived-bycastify/src/usr/local/sbin/update_keepalived.py'
2025-01-09 11:37:23.858 [info] [Info  - 11:37:23 AM] (1861931)   Looking in stubPath 'file:///home/abrian/git/storagegrid/.pylance/typings'
2025-01-09 11:37:23.858 [info] [Info  - 11:37:23 AM] (1861931)   Attempting to resolve stub package using root path 'file:///home/abrian/git/storagegrid/.pylance/typings'
2025-01-09 11:37:23.859 [info] [Info  - 11:37:23 AM] (1861931)   Attempting to resolve using root path 'file:///home/abrian/git/storagegrid/.pylance/typings'
2025-01-09 11:37:23.859 [info] [Info  - 11:37:23 AM] (1861931)   Looking in root directory of execution environment 'file:///home/abrian/git/storagegrid'
2025-01-09 11:37:23.859 [info] [Info  - 11:37:23 AM] (1861931)   Attempting to resolve stub package using root path 'file:///home/abrian/git/storagegrid'
2025-01-09 11:37:23.859 [info] [Info  - 11:37:23 AM] (1861931)   Attempting to resolve using root path 'file:///home/abrian/git/storagegrid'
2025-01-09 11:37:23.859 [info] [Info  - 11:37:23 AM] (1861931)   Looking in extraPath 'file:///home/abrian/git/storagegrid/container-utils/src/container-common'
2025-01-09 11:37:23.859 [info] [Info  - 11:37:23 AM] (1861931)   Attempting to resolve stub package using root path 'file:///home/abrian/git/storagegrid/container-utils/src/container-common'
2025-01-09 11:37:23.859 [info] [Info  - 11:37:23 AM] (1861931)   Attempting to resolve using root path 'file:///home/abrian/git/storagegrid/container-utils/src/container-common'
2025-01-09 11:37:23.859 [info] [Info  - 11:37:23 AM] (1861931)   Resolved import with file 'file:///home/abrian/git/storagegrid/container-utils/src/container-common/netapp/networking/link.py'
2025-01-09 11:37:23.859 [info] [Info  - 11:37:23 AM] (1861931)   Looking in extraPath 'file:///home/abrian/git/storagegrid/docker-utils/src/platform-common'
2025-01-09 11:37:23.859 [info] [Info  - 11:37:23 AM] (1861931)   Attempting to resolve stub package using root path 'file:///home/abrian/git/storagegrid/docker-utils/src/platform-common'
2025-01-09 11:37:23.859 [info] [Info  - 11:37:23 AM] (1861931)   Attempting to resolve using root path 'file:///home/abrian/git/storagegrid/docker-utils/src/platform-common'
2025-01-09 11:37:23.859 [info] [Info  - 11:37:23 AM] (1861931)   Resolved import with file 'file:///home/abrian/git/storagegrid/docker-utils/src/platform-common/netapp/__init__.py'
2025-01-09 11:37:23.859 [info] [Info  - 11:37:23 AM] (1861931)   Looking for typeshed stdlib path
2025-01-09 11:37:23.859 [info] [Info  - 11:37:23 AM] (1861931)   Looking for typeshed stdlib path
2025-01-09 11:37:23.859 [info] [Info  - 11:37:23 AM] (1861931)   Attempting to resolve using root path 'file:///home/abrian/.vscode-server/extensions/ms-python.vscode-pylance-2024.12.1/dist/typeshed-fallback/stdlib'
2025-01-09 11:37:23.859 [info] [Info  - 11:37:23 AM] (1861931)   Typeshed path not found
2025-01-09 11:37:23.859 [info] [Info  - 11:37:23 AM] (1861931)   Finding python search paths
2025-01-09 11:37:23.859 [info] [Info  - 11:37:23 AM] (1861931)   Executing interpreter: '/bin/python'
2025-01-09 11:37:23.859 [info] [Info  - 11:37:23 AM] (1861931)   Skipping '/usr/lib/python311.zip' because it is not a valid directory
2025-01-09 11:37:23.859 [info] [Info  - 11:37:23 AM] (1861931)   Received 4 paths from interpreter
2025-01-09 11:37:23.859 [info] [Info  - 11:37:23 AM] (1861931)     file:///usr/lib/python3.11
2025-01-09 11:37:23.859 [info] [Info  - 11:37:23 AM] (1861931)     file:///usr/lib/python3.11/lib-dynload
2025-01-09 11:37:23.859 [info] [Info  - 11:37:23 AM] (1861931)     file:///usr/local/lib/python3.11/dist-packages
2025-01-09 11:37:23.859 [info] [Info  - 11:37:23 AM] (1861931)     file:///usr/lib/python3/dist-packages
2025-01-09 11:37:23.859 [info] [Info  - 11:37:23 AM] (1861931)   Looking in python search path 'file:///usr/lib/python3.11'
2025-01-09 11:37:23.859 [info] [Info  - 11:37:23 AM] (1861931)   Attempting to resolve stub package using root path 'file:///usr/lib/python3.11'
2025-01-09 11:37:23.859 [info] [Info  - 11:37:23 AM] (1861931)   Attempting to resolve using root path 'file:///usr/lib/python3.11'
2025-01-09 11:37:23.859 [info] [Info  - 11:37:23 AM] (1861931)   Looking in python search path 'file:///usr/lib/python3.11/lib-dynload'
2025-01-09 11:37:23.859 [info] [Info  - 11:37:23 AM] (1861931)   Attempting to resolve stub package using root path 'file:///usr/lib/python3.11/lib-dynload'
2025-01-09 11:37:23.859 [info] [Info  - 11:37:23 AM] (1861931)   Attempting to resolve using root path 'file:///usr/lib/python3.11/lib-dynload'
2025-01-09 11:37:23.860 [info] [Info  - 11:37:23 AM] (1861931)   Looking in python search path 'file:///usr/local/lib/python3.11/dist-packages'
2025-01-09 11:37:23.860 [info] [Info  - 11:37:23 AM] (1861931)   Attempting to resolve stub package using root path 'file:///usr/local/lib/python3.11/dist-packages'
2025-01-09 11:37:23.860 [info] [Info  - 11:37:23 AM] (1861931)   Attempting to resolve using root path 'file:///usr/local/lib/python3.11/dist-packages'
2025-01-09 11:37:23.860 [info] [Info  - 11:37:23 AM] (1861931)   Looking in python search path 'file:///usr/lib/python3/dist-packages'
2025-01-09 11:37:23.860 [info] [Info  - 11:37:23 AM] (1861931)   Attempting to resolve stub package using root path 'file:///usr/lib/python3/dist-packages'
2025-01-09 11:37:23.860 [info] [Info  - 11:37:23 AM] (1861931)   Attempting to resolve using root path 'file:///usr/lib/python3/dist-packages'
2025-01-09 11:37:23.860 [info] [Info  - 11:37:23 AM] (1861931)   Looking in bundled stubs path 'file:///home/abrian/.vscode-server/extensions/ms-python.vscode-pylance-2024.12.1/dist/bundled/stubs'
2025-01-09 11:37:23.860 [info] [Info  - 11:37:23 AM] (1861931)   Attempting to resolve stub package using root path 'file:///home/abrian/.vscode-server/extensions/ms-python.vscode-pylance-2024.12.1/dist/bundled/stubs'
2025-01-09 11:37:23.860 [info] [Info  - 11:37:23 AM] (1861931)   Attempting to resolve using root path 'file:///home/abrian/.vscode-server/extensions/ms-python.vscode-pylance-2024.12.1/dist/bundled/stubs'
2025-01-09 11:37:23.860 [info] [Info  - 11:37:23 AM] (1861931)   Looking for typeshed third-party path
2025-01-09 11:37:23.860 [info] [Info  - 11:37:23 AM] (1861931)   Looking for typeshed stubs path
2025-01-09 11:37:23.860 [info] [Info  - 11:37:23 AM] (1861931)   Typeshed path not found

@heejaechang
Copy link
Contributor

please, provide full log as the doc I linked above said, everything from the start of service. otherwise, it would hard for us to figure out.

@batkins61
Copy link

The log is over 15k lines long, and I'm not willing to post it here. Is there another way I can provide the log to you?

@heejaechang
Copy link
Contributor

you can send it to heejae.chang at microsoft dot com

@heejaechang
Copy link
Contributor

heejaechang commented Jan 9, 2025

so I looked the log, the issue is missing __init__.py file in container-utils/src/container-common/netapp. once you put __init__.py file there, it should win the import resolution.

Could not import 'netapp.helpers.client' in file '/home/abrian/git/storagegrid/keepalived-bycastify/src/usr/local/sbin/update_keepalived.py'
2025-01-09 11:37:23.857 [info] [Info  - 11:37:23 AM] (1861931)   Looking in extraPath 'file:///home/abrian/git/storagegrid/container-utils/src/container-common'
2025-01-09 11:37:23.858 [info] [Info  - 11:37:23 AM] (1861931)   Attempting to resolve stub package using root path 'file:///home/abrian/git/storagegrid/container-utils/src/container-common'
2025-01-09 11:37:23.858 [info] [Info  - 11:37:23 AM] (1861931)   Attempting to resolve using root path 'file:///home/abrian/git/storagegrid/container-utils/src/container-common'
2025-01-09 11:37:23.858 [info] [Info  - 11:37:23 AM] (1861931)   Resolved import with file 'file:///home/abrian/git/storagegrid/container-utils/src/container-common/netapp/helpers/__init__.py'
2025-01-09 11:37:23.858 [info] [Info  - 11:37:23 AM] (1861931)   Resolved import with file 'file:///home/abrian/git/storagegrid/container-utils/src/container-common/netapp/helpers/client.py'
2025-01-09 11:37:23.858 [info] [Info  - 11:37:23 AM] (1861931)   Looking in extraPath 'file:///home/abrian/git/storagegrid/docker-utils/src/platform-common'
2025-01-09 11:37:23.858 [info] [Info  - 11:37:23 AM] (1861931)   Attempting to resolve stub package using root path 'file:///home/abrian/git/storagegrid/docker-utils/src/platform-common'
2025-01-09 11:37:23.858 [info] [Info  - 11:37:23 AM] (1861931)   Attempting to resolve using root path 'file:///home/abrian/git/storagegrid/docker-utils/src/platform-common'
2025-01-09 11:37:23.858 [info] [Info  - 11:37:23 AM] (1861931)   Resolved import with file 'file:///home/abrian/git/storagegrid/docker-utils/src/platform-common/netapp/__init__.py'

what's happening is, our import resolver is going through import roots and resolved netapp.helpers.client in container-common/netapp/helpers/client.py after that, it went to next, and found netapp in platform-common/netapp/__init__.py but unlike container-common/netapp, netapp in platform-common has __init__.py making regular python package that wins over netapp in container-common/netapp that is considered as namespace package.

that's why netapp is always resolved to platform-common

and I just confirmed that's runtime behavior as well.

I didn't look other cases, but try fix this issue (not just this one but other module/packages having the same issue) and see whether you still having import resolve issue.

by the way, if you still have issue, we would appreciate if you create a new issue rather than keep re-using other people's issue.

@batkins61
Copy link

batkins61 commented Jan 9, 2025

Excellent, thank you! I thought this was the same issue, but I will submit new issues in the future.
Adding the __init__.py files solved the problem.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
needs repro Issue has not been reproduced yet waiting for user response Requires more information from user
Projects
None yet
Development

No branches or pull requests

3 participants