Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add provider for downloading datasets from scicat #18

Closed
wants to merge 12 commits into from
Closed

feat: add provider for downloading datasets from scicat #18

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
6778d7c
feat: add provider for downloading datasets from scicat
jokasimr Apr 8, 2024
0c1fb42
fix
jokasimr Apr 8, 2024
4fb9418
remove unnecessary whitespace
jokasimr Apr 8, 2024
ffa86ed
use scitacean testing fixtures, separate client provider
jokasimr Apr 8, 2024
cb9036c
refactor tests to use fake file transfer
jokasimr Apr 9, 2024
bb795bd
better name
jokasimr Apr 9, 2024
fc2ca3a
simplify
jokasimr Apr 9, 2024
405fbfc
remove extras, add conftest
jokasimr Apr 9, 2024
1e7a61c
fix: lock for threading
jokasimr Apr 9, 2024
09b361c
fix: allow parallel download of different data
jokasimr Apr 9, 2024
29aa5f1
fix: add test returning local path but doesnt download file
jokasimr Apr 9, 2024
5f23eac
add related datasets getter
jokasimr Apr 11, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions conda/meta.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,8 @@ requirements:
- python>=3.10
- scipp>=24.02.0
- scippnexus>=24.03.0
- scitacean
- paramiko

test:
imports:
Expand Down
1 change: 1 addition & 0 deletions pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@ requires-python = ">=3.10"
dependencies = [
"scipp >= 24.02.0",
"scippnexus >= 24.03.0",
"scitacean[sftp, test]",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we avoid using extras then the new way of defining meta.yaml automatically (see copier_template) will work. Also, why do we depend on the test extra, that seems odd?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes I'll remove the test extra. How can we avoid depending on the sftp extra? By manually adding the dependencies?

]

dynamic = ["version"]
Expand Down
1 change: 1 addition & 0 deletions requirements/base.in
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,3 +4,4 @@
# The following was generated by 'tox -e deps', DO NOT EDIT MANUALLY!
scipp >= 24.02.0
scippnexus >= 24.03.0
scitacean[sftp, test]
62 changes: 59 additions & 3 deletions requirements/base.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,26 +1,82 @@
# SHA1:b5fdb6600edc83ab95fb0e848607edef52cdd293
# SHA1:9f94c3c17a1fa254f95cd3b142a3a3a996202efd
#
# This file is autogenerated by pip-compile-multi
# To update, run:
#
# pip-compile-multi
#
annotated-types==0.6.0
# via pydantic
attrs==23.2.0
# via hypothesis
bcrypt==4.1.2
# via paramiko
certifi==2024.2.2
# via requests
cffi==1.16.0
# via
# cryptography
# pynacl
charset-normalizer==3.3.2
# via requests
cryptography==42.0.5
# via paramiko
dnspython==2.6.1
# via email-validator
email-validator==2.1.1
# via scitacean
exceptiongroup==1.2.0
# via hypothesis
filelock==3.13.3
# via scitacean
h5py==3.10.0
# via scippnexus
hypothesis==6.100.0
# via scitacean
idna==3.6
# via
# email-validator
# requests
numpy==1.26.4
# via
# h5py
# scipp
# scipy
paramiko==3.4.0
# via scitacean
pycparser==2.22
# via cffi
pydantic==2.6.4
# via scitacean
pydantic-core==2.16.3
# via pydantic
pynacl==1.5.0
# via paramiko
python-dateutil==2.9.0.post0
# via scippnexus
# via
# scippnexus
# scitacean
pyyaml==6.0.1
# via scitacean
requests==2.31.0
# via scitacean
scipp==24.2.0
# via
# -r base.in
# scippnexus
scippnexus==24.3.1
# via -r base.in
scipy==1.12.0
scipy==1.13.0
# via scippnexus
scitacean[sftp,test]==23.10.0
# via -r base.in
six==1.16.0
# via python-dateutil
sortedcontainers==2.4.0
# via hypothesis
typing-extensions==4.11.0
# via
# pydantic
# pydantic-core
urllib3==2.2.1
# via requests
6 changes: 3 additions & 3 deletions requirements/ci.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,13 +17,13 @@ colorama==0.4.6
# via tox
distlib==0.3.8
# via virtualenv
filelock==3.13.1
filelock==3.13.3
# via
# tox
# virtualenv
gitdb==4.0.11
# via gitpython
gitpython==3.1.42
gitpython==3.1.43
# via -r ci.in
idna==3.6
# via requests
Expand All @@ -48,7 +48,7 @@ tomli==2.0.1
# via
# pyproject-api
# tox
tox==4.14.1
tox==4.14.2
# via -r ci.in
urllib3==2.2.1
# via requests
Expand Down
34 changes: 11 additions & 23 deletions requirements/dev.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,6 @@
-r static.txt
-r test.txt
-r wheels.txt
annotated-types==0.6.0
# via pydantic
anyio==4.3.0
# via
# httpx
Expand All @@ -26,13 +24,11 @@ arrow==1.3.0
# via isoduration
async-lru==2.0.4
# via jupyterlab
cffi==1.16.0
# via argon2-cffi-bindings
click==8.1.7
# via
# pip-compile-multi
# pip-tools
copier==9.1.1
copier==9.2.0
# via -r dev.in
dunamai==1.19.2
# via copier
Expand All @@ -42,15 +38,15 @@ funcy==2.0
# via copier
h11==0.14.0
# via httpcore
httpcore==1.0.4
httpcore==1.0.5
# via httpx
httpx==0.27.0
# via jupyterlab
isoduration==20.11.0
# via jsonschema
jinja2-ansible-filters==1.3.2
# via copier
json5==0.9.22
json5==0.9.24
# via jupyterlab-server
jsonpointer==2.4
# via jsonschema
Expand All @@ -59,7 +55,7 @@ jsonschema[format-nongpl]==4.21.1
# jupyter-events
# jupyterlab-server
# nbformat
jupyter-events==0.9.0
jupyter-events==0.10.0
# via jupyter-server
jupyter-lsp==2.2.4
# via jupyterlab
Expand All @@ -69,11 +65,11 @@ jupyter-server==2.13.0
# jupyterlab
# jupyterlab-server
# notebook-shim
jupyter-server-terminals==0.5.2
jupyter-server-terminals==0.5.3
# via jupyter-server
jupyterlab==4.1.4
jupyterlab==4.1.6
# via -r dev.in
jupyterlab-server==2.25.3
jupyterlab-server==2.26.0
# via jupyterlab
notebook-shim==0.2.4
# via jupyterlab
Expand All @@ -89,16 +85,8 @@ plumbum==1.8.2
# via copier
prometheus-client==0.20.0
# via jupyter-server
pycparser==2.21
# via cffi
pydantic==2.6.3
# via copier
pydantic-core==2.16.3
# via pydantic
python-json-logger==2.0.7
# via jupyter-events
pyyaml-include==1.3.2
# via copier
questionary==1.10.0
# via copier
rfc3339-validator==0.1.4
Expand All @@ -109,27 +97,27 @@ rfc3986-validator==0.1.1
# via
# jsonschema
# jupyter-events
send2trash==1.8.2
send2trash==1.8.3
# via jupyter-server
sniffio==1.3.1
# via
# anyio
# httpx
terminado==0.18.0
terminado==0.18.1
# via
# jupyter-server
# jupyter-server-terminals
toposort==1.10
# via pip-compile-multi
types-python-dateutil==2.8.19.20240311
types-python-dateutil==2.9.0.20240316
# via arrow
uri-template==1.3.0
# via jsonschema
webcolors==1.13
# via jsonschema
websocket-client==1.7.0
# via jupyter-server
wheel==0.42.0
wheel==0.43.0
# via pip-tools

# The following packages are considered to be unsafe in a requirements file:
Expand Down
42 changes: 11 additions & 31 deletions requirements/docs.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,10 +12,6 @@ alabaster==0.7.16
# via sphinx
asttokens==2.4.1
# via stack-data
attrs==23.2.0
# via
# jsonschema
# referencing
babel==2.14.0
# via
# pydata-sphinx-theme
Expand All @@ -26,11 +22,7 @@ beautifulsoup4==4.12.3
# pydata-sphinx-theme
bleach==6.1.0
# via nbconvert
certifi==2024.2.2
# via requests
charset-normalizer==3.3.2
# via requests
comm==0.2.1
comm==0.2.2
# via ipykernel
debugpy==1.8.1
# via ipykernel
Expand All @@ -44,19 +36,15 @@ docutils==0.20.1
# nbsphinx
# pydata-sphinx-theme
# sphinx
exceptiongroup==1.2.0
# via ipython
executing==2.0.1
# via stack-data
fastjsonschema==2.19.1
# via nbformat
idna==3.6
# via requests
imagesize==1.4.1
# via sphinx
ipykernel==6.29.3
ipykernel==6.29.4
# via -r docs.in
ipython==8.22.2
ipython==8.23.0
# via
# -r docs.in
# ipykernel
Expand All @@ -72,11 +60,11 @@ jsonschema==4.21.1
# via nbformat
jsonschema-specifications==2023.12.1
# via jsonschema
jupyter-client==8.6.0
jupyter-client==8.6.1
# via
# ipykernel
# nbclient
jupyter-core==5.7.1
jupyter-core==5.7.2
# via
# ipykernel
# jupyter-client
Expand Down Expand Up @@ -105,11 +93,11 @@ mistune==3.0.2
# via nbconvert
myst-parser==2.0.0
# via -r docs.in
nbclient==0.9.0
nbclient==0.10.0
# via nbconvert
nbconvert==7.16.2
nbconvert==7.16.3
# via nbsphinx
nbformat==5.9.2
nbformat==5.10.4
# via
# nbclient
# nbconvert
Expand All @@ -126,7 +114,7 @@ packaging==24.0
# sphinx
pandocfilters==1.5.1
# via nbconvert
parso==0.8.3
parso==0.8.4
# via jedi
pexpect==4.9.0
# via ipython
Expand All @@ -149,18 +137,14 @@ pygments==2.17.2
# nbconvert
# pydata-sphinx-theme
# sphinx
pyyaml==6.0.1
# via myst-parser
pyzmq==25.1.2
# via
# ipykernel
# jupyter-client
referencing==0.33.0
referencing==0.34.0
# via
# jsonschema
# jsonschema-specifications
requests==2.31.0
# via sphinx
rpds-py==0.18.0
# via
# jsonschema
Expand Down Expand Up @@ -204,7 +188,7 @@ tornado==6.4
# via
# ipykernel
# jupyter-client
traitlets==5.14.1
traitlets==5.14.2
# via
# comm
# ipykernel
Expand All @@ -216,10 +200,6 @@ traitlets==5.14.1
# nbconvert
# nbformat
# nbsphinx
typing-extensions==4.10.0
# via pydata-sphinx-theme
urllib3==2.2.1
# via requests
wcwidth==0.2.13
# via prompt-toolkit
webencodings==0.5.1
Expand Down
2 changes: 0 additions & 2 deletions requirements/mypy.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,5 +10,3 @@ mypy==1.9.0
# via -r mypy.in
mypy-extensions==1.0.0
# via mypy
typing-extensions==4.10.0
# via mypy
1 change: 1 addition & 0 deletions requirements/nightly.in
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,3 +3,4 @@
# The following was generated by 'tox -e deps', DO NOT EDIT MANUALLY!
scipp >= 24.02.0
scippnexus >= 24.03.0
scitacean[sftp, test]
Loading
Loading