Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OverwriteNewerLocalError raised despite force_overwrite_from_cloud=True in parallel read only scenario #492

Closed
YishGene opened this issue Dec 10, 2024 · 1 comment

Comments

@YishGene
Copy link

YishGene commented Dec 10, 2024

Hi,

In v0.20.0, I'm running into an issue where, when trying to read an S3 file from multiple readers in a parallel scenario (multiple readers, no write access), I am passing force_overwrite_from_cloud as instructed in the error report, but still getting an OverwriteNewerLocalError.

I will try to set different caches for each process, but this could be wasteful as I am only reading and will have to copy already downloaded files.

Related to isses: #283 and #128 , but not exactly.

The error reads like this:

...
fp = file_path.open('rb', force_overwrite_from_cloud=True)
File "/opt/conda/lib/python3.10/site-packages/cloudpathlib/cloudpath.py", line 686, in open
self._refresh_cache(force_overwrite_from_cloud=force_overwrite_from_cloud)
File "/opt/conda/lib/python3.10/site-packages/cloudpathlib/cloudpath.py", line 1303, in _refresh_cache
raise OverwriteNewerLocalError(
cloudpathlib.exceptions.OverwriteNewerLocalError: Local file (/tmp/tmp9h3c_4qk/s3://|bucket|/|ObjectPath|/GT_32.h5) for cloud path (s3://|bucket|/|ObjectPath|/GT_32.h5) is newer on disk, but is being requested for download from cloud. Either (1) push your changes to the cloud, (2) r:emove the local file, or (3) pass force_overwrite_from_cloud=True to overwrite; or set env var CLOUDPATHLIB_FORCE_OVERWRITE_FROM_CLOUD=1.

@pjbull
Copy link
Member

pjbull commented Dec 11, 2024

Thanks for this report. I'm going to close this issue since it is almost certainly the same root cause as #283.

Let's continue the conversation there. I will add some thoughts.

@pjbull pjbull closed this as not planned Won't fix, can't repro, duplicate, stale Dec 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants