Transition to versioned bids functions

Versioned functions are created on the fly upon import. Correspondance between the bids_functions and the available specs is kept by the update_bids.py script and corresponding poe task. The `latest` spec stays automatically updated to the latest of the specs.
khanlab · Feb 1, 2024 · 9c68e2e · 9c68e2e
1 parent e11b828
commit 9c68e2e
Show file tree

Hide file tree

Showing 29 changed files with 1,177 additions and 624 deletions.
diff --git a/docs/_static/css/typealiases.css b/docs/_static/css/typealiases.css
@@ -1,3 +1,4 @@
+#snakebids\.paths\.specs\.BidsPathSpec,
 #snakebids\.types\.InputsConfig,
 #snakebids\.types\.ZipList,
 #snakebids\.types\.ZipListLike {

diff --git a/docs/api/main.rst b/docs/api/main.rst
@@ -1,63 +1,65 @@
 .. _main api:
 
+
 ================
 API
 ================
 
+.. py:currentmodule:: snakebids
 
-snakebids
----------
-
-.. autoclass:: snakebids.BidsComponent
-    :members:
-    :exclude-members: input_wildcards, input_lists, input_name, input_path, input_zip_lists
-    :inherited-members:
-
-.. dropdown:: Legacy ``BidsComponents`` properties
-    :icon: info
-    :class-title: sd-outline-info
-
-    The following properties are historical aliases of ``BidsComponents`` properties. There are no current plans to deprecate them, but new code should avoid them.
+.. toctree::
+    :hidden:
 
-    .. autoproperty:: snakebids.BidsComponent.input_zip_lists
+    paths
+    creation
+    manipulation
+    structures
 
-    .. autoproperty:: snakebids.BidsComponent.input_wildcards
 
-    .. autoproperty:: snakebids.BidsComponent.input_name
+Path Creation
+-------------
 
-    .. autoproperty:: snakebids.BidsComponent.input_path
+.. Need to manually create this table because bids does not have a proper docstring
 
-    .. autoproperty:: snakebids.BidsComponent.input_lists
+.. ===================================  ================================
+.. :func:`bids <bids>`                  Generate bids or bids-like paths
+.. :func:`bids_factory <bids_factory>`  Create new :func:`bids` functions according to a spec
+.. ===================================  ================================
 
+.. autosummary::
+    :nosignatures:
 
-.. autoclass:: snakebids.BidsPartialComponent
+    bids
+    bids_factory
 
-.. autoclass:: snakebids.BidsComponentRow
-    :members:
-    :exclude-members: zip_lists
+Dataset Creation
+----------------
 
-.. autoclass:: snakebids.BidsDataset
-    :members:
-    :exclude-members: input_wildcards, input_lists, input_path, input_zip_lists
+.. autosummary::
 
-.. dropdown:: Legacy ``BidsDataset`` properties
-    :icon: info
-    :class-title: sd-outline-info
+    generate_inputs
 
-    The following properties are historical aliases of :class:`~snakebids.BidsDataset` properties. There are no current plans to deprecate them, but new code should avoid them.
+Dataset Manipulation
+--------------------
 
-    .. autoproperty:: snakebids.BidsDataset.input_zip_lists
+.. autosummary::
+    :nosignatures:
 
-    .. autoproperty:: snakebids.BidsDataset.input_wildcards
+    filter_list
+    get_filtered_ziplist_index
 
-    .. autoproperty:: snakebids.BidsDataset.input_path
+Data Structures
+---------------
 
-    .. autoproperty:: snakebids.BidsDataset.input_lists
+.. autosummary::
+    :nosignatures:
 
+    BidsComponent
+    BidsPartialComponent
+    BidsComponentRow
+    BidsDataset
+    BidsDatasetDict
 
-.. automodule:: snakebids
-    :exclude-members: from_bids_lists, BidsComponent, BidsPartialComponent, BidsComponentRow, BidsDataset
-    :members:
 
 
 app

diff --git a/docs/bids_app/overview.md b/docs/bids_app/overview.md
@@ -1,4 +1,12 @@
-# Overview
+# Bids Apps
+
+```{toctree}
+:hidden:
+
+config
+workflow
+plugins
+```
 
 Snakebids apps rely on a configuration file (`snakebids.yml`). This file specifies which files from a BIDS dataset should be used as input. The apps also utilize workflow definitions, which are written in one or more Snakefile(s) and determine how the input files are processed.
 

diff --git a/docs/bids_function/overview.rst b/docs/bids_function/overview.rst
@@ -1,5 +1,5 @@
-Overview
-========
+Bids Function
+=============
 
 The ``bids`` function generates a BIDS-like filepath corresponding to its keyword arguments. The generated filepath has the form::
 

diff --git a/docs/conf.py b/docs/conf.py
@@ -36,6 +36,7 @@
     "sphinxarg.ext",
     "sphinx.ext.intersphinx",
     "sphinx.ext.napoleon",
+    "sphinx.ext.autosummary",
     "sphinx.ext.autodoc",
     "sphinxcontrib.asciinema",
     "myst_parser",
@@ -69,7 +70,8 @@
     "InputsConfig": "snakebids.types.InputsConfig",
 }
 autodoc_typehints_format = "short"
-autosummary_imported_members = True
+# autosummary_imported_members = True
+autosummary_generate = True
 
 
 # Add any paths that contain templates here, relative to this directory.

diff --git a/docs/index.md b/docs/index.md
@@ -7,67 +7,30 @@ Snakebids is migrating to a more robust, extensible API! If you're coming from p
 
 
 ```{toctree}
-:caption: General
-:name: general
+:caption: User Guide
+:name: guide
 :hidden:
-:maxdepth: 2
+:maxdepth: 1
 
 general/why_snakebids
-```
-
-```{toctree}
-:caption: Tutorial
-:name: tutorial
-:hidden:
-:maxdepth: 2
-
 tutorial/tutorial
-```
-
-```{toctree}
-:caption: Using the bids function
-:name: bids_function
-:hidden:
-:maxdepth: 2
-
 bids_function/overview
-```
-
-```{toctree}
-:caption: Creating a bids app
-:name: bids_app
-:hidden:
-:maxdepth: 2
 
 bids_app/overview
-bids_app/config
-bids_app/workflow
-bids_app/plugins
-```
-
-```{toctree}
-:caption: Running a snakebids app
-:name: running_snakebids
-:hidden:
-:maxdepth: 2
 
 running_snakebids/overview
+
+migration/index
 ```
 
-```{toctree}
-:caption: Migration
-:name: migration
-:hidden:
-:maxdepth: 2
 
-migration/0.5_to_0.8
-migration/0.7_to_0.8
-```
+
+
 
 ```{toctree}
 :caption: Reference
 :hidden:
-:maxdepth: 1
+:maxdepth: 2
 
 api/main
 api/internals

diff --git a/docs/running_snakebids/overview.md b/docs/running_snakebids/overview.md
@@ -1,5 +1,4 @@
-Overview
-========
+# Running Snakebids
 
 Once you've specified a snakebids app with a config file and one or more workflow files, you're ready to invoke your snakebids app with the standard BIDS app CLI.
 
@@ -12,15 +11,14 @@ Note that if any rules in the Snakebids workflow use Singularity containers, spe
 1. Inputs are copied into a working subdirectory of the output directory before any processing that requires a Singularity container is performed, or:
 2. The `SINGULARITY_BINDPATH` environment variable binds the location of the input dataset.
 
-Indexing of large datasets can be a time-consuming process. Leveraging the functionality of `PyBIDS`, Snakebids offers a convenient solution by allowing you to create or utilize an existing database. With this approach, the indexing of datasets is only performed when explictly requested, typically when there are changes to the dataset. To create or use an existing database, you can invoke the following CLI arguments: 
+Indexing of large datasets can be a time-consuming process. Leveraging the functionality of `PyBIDS`, Snakebids offers a convenient solution by allowing you to create or utilize an existing database. With this approach, the indexing of datasets is only performed when explictly requested, typically when there are changes to the dataset. To create or use an existing database, you can invoke the following CLI arguments:
 
 1. `--pybidsdb-dir {dir}`: specify the path to the database directory
 1. `--pybidsdb-reset`: indicate that an existing database should be updated
 
 The boilerplate app starts with the validator plugin enabled - without it, validation is not performed. By default, this feature uses the command-line (node.js) version of the [validator](https://www.npmjs.com/package/bids-validator). If this is not found to be installed on the system, the `pybids` version of validation will be performed instead. To opt-out of validation, invoke the `--skip-bids-validation` flag. Details related to using and creating plugins can be found on the [plugins](/bids_app/plugins) page.
 
-Workflow mode
-=============
+## Workflow mode
 
 Snakebids apps use a BIDS app CLI, giving great flexibility when switching datasets. However, when developing a Snakebids app or when running the app repeatedly on the same dataset, it can be more convenient to directly call the Snakemake CLI. Snakebids facilitates this using workflow mode.
 

diff --git a/docs/tutorial/tutorial.md b/docs/tutorial/tutorial.md
@@ -1,8 +1,10 @@
 # Tutorial
 
+
 % links
 [expand_func]: inv:snakemake:std:label#snakefiles_expand
 
+(tutorial_getting_started)=
 ## Getting started
 
 In this example we will make a workflow to smooth ``bold`` scans from a bids dataset.
@@ -62,11 +64,11 @@ $ cp -r snakebids/docs/tutorial/bids ./data
 
 It's also perfectly possible (and probably better!) to try the tutorial on your own dataset. Just adjust any paths below so that they match your data!
 
-# Part I: Snakemake
+(snakemake_tutorial)=
+## Part I: Snakemake
 
 (step_0)=
-
-## Step 0: a basic non-generic workflow
+### Step 0: a basic non-generic workflow
 
 In this rule, we start by creating a rule that is effectively hard-coding the paths for input and output to re-create the command as above.
 
@@ -89,8 +91,7 @@ When we invoke ``snakemake``, it uses the first rule in the snakefile as the ``t
 So far, we just have a fancy way of specifying the exact same command we started with, so there is no added benefit (yet). But we will soon add to this rule to make it more generalizable.
 
 (step_1)=
-
-## Step 1: adding wildcards
+### Step 1: adding wildcards
 
 First step to make the workflow generalizeable is to replace the hard-coded identifiers (e.g. the subject, task and run) with wildcards.
 
@@ -127,8 +128,7 @@ Now, try changing the output smoothing value, e.g. ``fwhm-10mm``, and see what h
 As expected the command still uses a smoothing value of 2.12, since that has been hard-coded, but we will see how to rectify this in the next step.
 
 (step_2)=
-
-## Step 2: adding a params function
+### Step 2: adding a params function
 
 As we noted, the sigma parameter needs to be computed from the FWHM. We can use a function to do this. Functions can be used for any ``input`` or ``params``, and must take ``wildcards`` as an input argument, which provides a mechanism to pass the wildcards (determined from the output file) to the function.
 
@@ -169,8 +169,7 @@ Now try running the workflow again, with `fwhm-5` as well as `fwhm-10`.
 ```
 
 (step_3)=
-
-## Step 3: adding a target rule
+### Step 3: adding a target rule
 
 Now we have a generic rule, but it is pretty tedious to have to type out the filename of each target from the command-line in order to use it.
 
@@ -206,8 +205,7 @@ The entire Snakefile for reference is:
 ```
 
 (step_4)=
-
-## Step 4: adding a config file
+### Step 4: adding a config file
 
 We have a functional workflow, but suppose you need to configure or run it on another bids dataset with different subjects, tasks, runs, or you want to run it for different smoothing values. You have to actually modify your workflow in order to do this.
 
@@ -247,13 +245,14 @@ After these changes, the workflow should still run just like the last step, but
 
 ```
 
-# Part II: Snakebids
+
+(snakebids_tutorial)=
+## Part II: Snakebids
 
 Now that we have a fully functioning and generic Snakemake workflow, let's see what Snakebids can add.
 
 (step_5)=
-
-## Step 5: the bids() function
+### Step 5: the bids() function
 
 The first thing we can make use of is the {func}`~snakebids.bids` function. This provides an easy way to generate bids filenames. This is especially useful when defining output files in your workflow and you have many bids entities.
 
@@ -295,7 +294,8 @@ The Snakefile with the output filename replaced (in both rules) is below:
   :caption: Snakefile
 ```
 
-## Step 6: parsing the BIDS dataset
+
+### Step 6: parsing the BIDS dataset
 
 So far, we have had to manually enter the path to input bold file in the config file, and also specify what subjects, tasks, and runs we want processed. Can't we use the fact that we have a BIDS dataset to automate this a bit more?
 
@@ -381,7 +381,8 @@ Notice that `inputs['bold'].path`{l=python} is the same as the path we wrote und
 :emphasize-lines: 3
 ```
 
-## Step 7: using input wildcards
+
+### Step 7: using input wildcards
 
 {attr}`BidsComponent.path <snakebids.BidsComponent.path>` already grants us a lot of flexibility, but we can still do more! In addition to the three main attributes of {class}`BidsComponents <snakebids.BidsComponent>` already described, the class offers a number of special properties we can use in our workflows. First, we'll look at {attr}`BidsComponent.wildcards <snakebids.BidsComponent.wildcards>`. This is a dict that maps each entity to the brace-wrapped `{wildcards}` we specified in `pybids_config`. If you printed this value in our test workflow, it would look like this:
 
@@ -449,7 +450,7 @@ For reference, here is the updated config file and Snakefile after these changes
   :caption: Snakefile
 ```
 
-## Step 8: creating a command-line executable
+### Step 8: creating a command-line executable
 
 Now that we have pybids parsing to dynamically configure our workflow inputs based on our BIDS dataset, we are ready to turn our workflow into a [BIDS App](http://bids-apps.neuroimaging.io/). BIDS Apps are command-line apps with a standardized interface (e.g. three required positional arguments: ``bids_directory``, ``output_directory``, and ``analysis_level``).
 

diff --git a/poetry.lock b/poetry.lock