Merge branch 'main' of github.com:snel-repo/falcon-challenge

README.md

@@ -1,6 +1,6 @@
 # FALCON Benchmark and Challenge
 
-This package contains core code for submitting decoders to the FALCON challenge. Full github contains additional examples and documentation.
+This package contains core code for submitting decoders to the FALCON challenge. For a more general overview of FALCON, please see the [main website](https://snel-repo.github.io/falcon/).
 
 ## Installation
 Install `falcon_challenge` with:
@@ -15,24 +15,45 @@ See, e.g. https://docs.docker.com/desktop/install/linux-install/.
 ## Getting started
 
 ### Data downloading
-The FALCON datasets are available on DANDI (or through private correspondence, if beta-testing). 
+The FALCON datasets are available on DANDI ([H1](https://dandiarchive.org/dandiset/000954?search=falcon&pos=3), [H2](https://dandiarchive.org/dandiset/000950?search=falcon&pos=4), [M1](https://dandiarchive.org/dandiset/000941?search=falcon&pos=1), [M2](https://dandiarchive.org/dandiset/000953?search=falcon&pos=2)). H1 and H2 are human intractorical brain-computer interface (iBCI) datasets, M1 and M2 are monkey iBCI datasets, and B1 is a songbird iBCI dataset.
 
-NOTE FOR BETA TESTERS:
-- Some of the sample code expects your data directory to be set up in `./data`. Specifically, the following hierarchy is expected:
+Data from each dataset is broken down as follows:
+
+- Held-in 
+    - Data from the first several recording sessions. 
+    - All non-evaluation data is released and split into calibration (large portion) and minival (small portion) sets. 
+    - Held-in calibration data is intended to train decoders from scratch.
+    - Minival data enables validation of held-in decoder generalization.
+- Held-out: 
+    - Data from the latter several recording sessions. 
+    - A small portion of non-evaluation data is released for calibration. 
+    - Held-out calibration data is intentionally small to discourage training decoders from scratch on this data and provides an opportunity for few-shot recalibration.
+
+Some of the sample code expects your data directory to be set up in `./data`. Specifically, the following hierarchy is expected:
 
 `data`
 - `h1`
     - `held_in_calib`
     - `held_out_calib`
-    - `minival`
-    - `eval` (Note this is private data)
+    - `minival` (Copy dandiset minival folder into this folder)
+- `h2`
+    - `held_in_calib`
+    - `held_out_calib`
+    - `minival` (Copy dandiset minival folder into this folder)
 - `m1`
     - `sub-MonkeyL-held-in-calib`
     - `sub-MonkeyL-held-out-calib`
     - `minival` (Copy dandiset minival folder into this folder)
-    - `eval` (Copy the ground truth held in and held out data into this folder)
+- `m2`
+    - `held_in_calib`
+    - `held_out_calib`
+    - `minival` (Copy dandiset minival folder into this folder)
+<!-- - `b1`
+    - `held_in_calib`
+    - `held_out_calib`
+    - `minival` (Copy dandiset minival folder into this folder) -->
 
-H1 should unfold correctly just from unzipping the provided directory. M1 should work by renaming the provided dandiset to `m1` and `minival` folder inside, and then copying the provided eval data into this folder. Each of the lowest level dirs holds the NWB files.
+Each of the lowest level dirs holds the data files (in Neurodata Without Borders (NWB) format). Data from some sessions is distributed across multiple NWB files. Some data from each file is allocated to calibration, minival, and evaluation splits as appropriate. 
 
 ### Code
 This codebase contains starter code for implementing your own method for the FALCON challenge. 
@@ -46,6 +67,8 @@ python decoder_demos/sklearn_decoder.py --training_dir data/h1/held_in_calib/ --
 python decoder_demos/sklearn_sample.py --evaluation local --phase minival --split h1
 ```
 
+Note: During evaluation, data file names are hashed into unique tags. Submitted solutions receive data to decode along with tags indicating the file from which the data originates in the call to their `reset` function. These tags are the keys of the the `DATASET_HELDINOUT_MAP` dictionary in `falcon_challenge/evaluator.py`. Submissions that intend to condition decoding on the data file from which the data comes should make use of these tags. For an example, see `fit_many_decoders` and `reset` in `decoder_demos/sklearn_decoder.py`.
+
 ### Docker Submission
 To interface with our challenge, your code will need to be packaged in a Docker container that is submitted to EvalAI. Try this process by building and running the provided `sklearn_sample.Dockerfile`, to confirm your setup works. Do this with the following commands (once Docker is installed)
 ```bash
@@ -54,6 +77,8 @@ docker build -t sk_smoke -f ./decoder_demos/sklearn_sample.Dockerfile .
 bash test_docker_local.sh --docker-name sk_smoke
 ```
 
+For an example Dockerfile with annotations regarding the necessity and function of each line, see `decoder_demos/template.Dockerfile`.
+
 ## EvalAI Submission
 Please ensure that your submission runs locally before running remote evaluation. You can run the previously listed commands with your own Dockerfile (in place of sk_smoke). This should produce a log of nontrivial metrics (evaluation is run on locally available minival).
 

data_demos/h1.ipynb

@@ -66,11 +66,11 @@
     "import seaborn as sns\n",
     "import numpy as np\n",
     "from pynwb import NWBHDF5IO\n",
-    "from data_demos.styleguide import set_style\n",
+    "from styleguide import set_style\n",
     "set_style()\n",
     "from visualization import plot_split_bars, plot_timeline, rasterplot, plot_firing_rate_distributions\n",
     "\n",
-    "data_dir = Path(\"data/h1/\")\n",
+    "data_dir = Path(\"../data/h1/\")\n",
     "\n",
     "train_query = 'held_in_calib'\n",
     "test_query = 'held_out_calib'\n",

data_demos/h2.ipynb

@@ -76,8 +76,8 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "train_path = Path('/snel/share/share/data/bg2/t5_handwriting/CORP_data/stability_benchmark/nwb/held_in_calib')\n",
-    "test_path = Path('/snel/share/share/data/bg2/t5_handwriting/CORP_data/stability_benchmark/nwb/held_out_calib')\n",
+    "train_path = Path('../data/h2/held_in_calib')\n",
+    "test_path = Path('../data/h2/held_out_calib')\n",
     "\n",
     "train_files = sorted(train_path.glob('*.nwb'))\n",
     "test_files = sorted(test_path.glob('*.nwb'))\n",

data_demos/m2.ipynb

@@ -34,7 +34,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 1,
+   "execution_count": null,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -44,7 +44,7 @@
     "import seaborn as sns\n",
     "import numpy as np\n",
     "from pynwb import NWBHDF5IO\n",
-    "from data_demos.styleguide import set_style\n",
+    "from styleguide import set_style\n",
     "set_style()"
    ]
   },
@@ -54,7 +54,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "data_dir = Path('data/m2')\n",
+    "data_dir = Path('../data/m2')\n",
     "train_files = sorted((data_dir / 'held_in_calib').glob('*calib.nwb'))\n",
     "test_files = sorted((data_dir / 'held_out_calib').glob('*calib.nwb'))\n",
     "\n",
@@ -686,7 +686,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.11.6"
+   "version": "3.11.9"
   }
  },
  "nbformat": 4,

decoder_demos/template.Dockerfile

@@ -0,0 +1,75 @@
+# This is a stripped-down version of the sklearn_sample.Dockerfile in the same directory. 
+# This file is intended to be a template for users to create their own Dockerfiles for EvalAI submissions.
+# Comments are provided to indicate which lines are mandatory for a valid submission and which are submission-specific.
+
+# NOTE: The environment variables PREDICTION_PATH, PREDICTION_PATH_LOCAL, GT_PATH, , and EVAL_DATA_PATH are used by the evaluator and should not be changed.
+# The other environment variables could be hardcoded in the runfile, but it is recommended to set them in the Dockerfile for clarity.
+
+# Dockerfile flow for EvalAI for a specific track
+
+# The user prepares a Docker image with this file and decoder defined locally
+# The user then directly pushes this image to EvalAI with the eval-ai cli.
+# see e.g. https://eval.ai/web/challenges/challenge-page/1615/submission (You may have to register for the challenge)
+
+# RECOMMENDED: Use a base Docker image that has the necessary dependencies for your decoder. The one provided is a good choice for PyTorch-based decoders.
+# Note: If you're using TF/Jax, you may want to use a different base image.
+FROM pytorch/pytorch:2.1.2-cuda11.8-cudnn8-devel
+
+# MANDATORY: Leave these lines as they are. This will install an up-to-date copy of the falcon-challenge package into the Docker image.
+RUN apt-get update && apt-get install -y git
+RUN git clone https://github.com/snel-repo/falcon-challenge.git
+RUN pip install -e falcon-challenge
+
+
+# MANDATORY: Leave these environment variables as they are. This is how EvalAI will find the predictions and answers during evaluation.
+ENV PREDICTION_PATH "/submission/submission.csv"
+ENV PREDICTION_PATH_LOCAL "/tmp/submission.pkl"
+ENV GT_PATH "/tmp/ground_truth.pkl"
+
+# OPTIONAL: Users should install additional decoder-specific dependencies here (e.g. RUN pip install package_name)
+
+# MANDATORY: Set EVALUATION_LOC. Use "remote" for submissions to EvalAI and "local" for local testing. This is passed to the evaluator in the runfile.
+ENV EVALUATION_LOC remote
+# ENV EVALUATION_LOC local
+
+# Add files from local context into Docker image
+# Note local context reference is the working dir by default, see https://docs.docker.com/engine/reference/commandline/build/
+
+# MANDATORY: Add the decoder file to the Docker image. This should be the path to the decoder file in your local directory.
+# Note that Docker cannot easily import across symlinks; make sure data is not symlinked
+ADD ./local_data/sklearn_FalconTask.m2.oracle.pkl data/decoder.pkl
+
+# OPTIONAL: Add any additional files that your decoder needs to the Docker image. The lines below are relevant to the sklearn_sample decoder.
+ADD ./decoder_demos/ decoder_demos/
+ADD ./data_demos/ data_demos/
+ADD ./decoder_demos/filtering.py filtering.py
+
+# MANDATORY:  Add the runfile to the Docker image. This should be the path to the runfile in your local directory. This line is relevant to the sklearn_sample decoder.
+ADD ./decoder_demos/sklearn_sample.py decode.py
+
+
+# MANDATORY: Set SPLIT. This specifies the dataset that your decoder is being evaluated on. This is passed to the evaluator in the runfile.
+# ENV SPLIT "m1"
+# ENV SPLIT "m2"
+ENV SPLIT "h1"
+# ENV SPLIT "h2"
+# ENV SPLIT "b1"
+
+# MANDATORY: Set PHASE. "minival" should be used for local testing and submission to the minival phase on EvalAI. "test" should be used for submissions to the test phase.
+# The minival set is a small subset of the held-in sessions, and is used for local testing and debugging. The test set is the full evaluation set for all sessions.
+# Note that the test set is only available remotely on EvalAI. The minival set is available locally and remotely on EvalAI.
+# This is passed to the evaluator in the runfile.
+# ENV PHASE "minival"
+ENV PHASE "test"
+
+# OPTIONAL: Set BATCH_SIZE for batched evaluation. Batching accelerates evaluation by processing multiple samples at once.
+# Note: the batch size is an attribute of the decoder. For the example sklearn_sample decoder, the batch size is passed to the decoder's constructor in the runfile.
+ENV BATCH_SIZE 1
+
+# MANDATORY: Leave this line as it is. This is a path to the evaluation data in the remote EvalAI environment.
+ENV EVAL_DATA_PATH "/dataset/evaluation_data"
+
+# MANDATORY: Specify a command for the container to run when it is launched. This command should run the decoder on the evaluation data.
+# It can be overridden with any cmd e.g. sudo docker run -it my_image /bin/bash
+CMD ["/bin/bash", "-c", \
+    "python decode.py --evaluation $EVALUATION_LOC --model-path data/decoder.pkl --split $SPLIT --phase $PHASE --batch-size $BATCH_SIZE"]