Prefect stops using cloud after updating to latest version

[chiang-yuan]

Bug summary

I have been trying to update to the latest version as the old version starts to fail the connection to prefect cloud as well.

After fighting with the same error, I decided to erase all the data under .prefect. It seems solving the authentication problem but now prefect never uses the cloud server anymore, even after I have set PREFECT_SERVER_ALLOW_EPHEMERAL_MODE='False' in prefect.toml

Below is the snippet to use dask Slumcluster to run prefect flow. Any help is very appreciated!

from dask.distributed import Client
from dask_jobqueue import SLURMCluster
from prefect_dask import DaskTaskRunner

nodes_per_alloc = 1
gpus_per_alloc = 4
ntasks = 1

cluster_kwargs = dict(
    cores=1,
    memory="64 GB",
    shebang="#!/bin/bash",
    account="xxxx",
    walltime="00:30:00",
    job_mem="0",
    job_script_prologue=[
        "source ~/.bashrc",
        "module load python",
        "source activate xxxx",
    ],
    job_directives_skip=["-n", "--cpus-per-task", "-J"],
    job_extra_directives=[
        "-J vm",
        "-q debug",
        f"-N {nodes_per_alloc}",
        "-C gpu",
        f"-G {gpus_per_alloc}",
    ],
)

cluster = SLURMCluster(**cluster_kwargs)
print(cluster.job_script())
cluster.adapt(minimum_jobs=2, maximum_jobs=2)
client = Client(cluster)

results = someflow.with_options(
    task_runner=DaskTaskRunner(address=client.scheduler.address),
)()

Version info

Version:             3.1.11
API version:         0.8.4
Python version:      3.11.8
Git commit:          e448bd34
Built:               Thu, Jan 2, 2025 1:11 PM
OS/Arch:             linux/x86_64
Profile:             default
Server type:         ephemeral
Pydantic version:    2.9.2
Server:
  Database:          sqlite
  SQLite version:    3.45.2
Integrations:
  prefect-dask:      0.3.2

Additional context

PREFECT_PROFILE='default'
PREFECT_API_KEY='********' (from profile)
PREFECT_API_URL='https://api.prefect.cloud/api/accounts/f7d40474-9362-4bfa-8950-ee6a43ec00f3/workspaces/d4bb0913-5f5e-49f7-bfc5-06509088baeb' (from profile)
PREFECT_HOME='/pscratch/sd/c/cyrusyc/.prefect' (from env)
PREFECT_SERVER_ALLOW_EPHEMERAL_MODE='False' (from profile)