python/how-tos/How_to_find_and_access_EMIT_data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# How to: Find and Access EMIT Data\n",
    "\n",
    "**Summary**  \n",
    "\n",
    "There are currently 4 ways to find EMIT data:\n",
    "\n",
    "1. [EarthData Search](https://search.earthdata.nasa.gov/search)\n",
    "2. [NASA's CMR API](https://www.earthdata.nasa.gov/eosdis/science-system-description/eosdis-components/cmr) (`earthaccess` uses this)\n",
    "3. [NASA's CMR-STAC API](https://cmr.earthdata.nasa.gov/search/site/docs/search/stac)\n",
    "3. [Visions Open Access Data Portal](https://earth.jpl.nasa.gov/emit/data/data-portal/coverage-and-forecasts/)\n",
    "\n",
    "This notebook will explain how to access Earth Surface Mineral Dust Source Investigation (EMIT) data programmaticly using the [earthaccess python library](https://github.com/nsidc/earthaccess). `earthaccess` is an easy to use library that reduces finding and downloading or streaming data over https or s3 to only a few lines of code. `earthaccess` searches NASA's Common Metadata Repository (CMR), a metadata system that catalogs Earth Science data and associated metadata records, then can be used to download granules or generate lists granule search result URLs.\n",
    "\n",
    "**Requirements:**\n",
    "- A NASA [Earthdata Login](https://urs.earthdata.nasa.gov/) account is required to download EMIT data   \n",
    "- *No Python setup requirements if connected to the workshop cloud instance!*\n",
    "- **Local Only** Set up Python Environment - See **setup_instructions.md** in the `/setup/` folder to set up a local compatible Python environment\n",
    "\n",
    "**Learning Objectives**  \n",
    "- How to get information about data collections using `earthaccess`\n",
    "- How to search and access EMIT data using `earthaccess`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup\n",
    "Import the required packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import earthaccess\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import geopandas as gp\n",
    "from shapely.geometry.polygon import orient\n",
    "import xarray as xr\n",
    "import sys\n",
    "sys.path.append('../modules/')\n",
    "from emit_tools import emit_xarray\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Authentication\n",
    "\n",
    "`earthaccess` creates and leverages Earthdata Login tokens to authenticate with NASA systems. Earthdata Login tokens expire after a month. To retrieve a token from Earthdata Login, you can either enter your username and password each time you use `earthaccess`, or use a `.netrc` file. A `.netrc` file is a configuration file that is commonly used to store login credentials for remote systems. If you don't have a `.netrc` or don't know if you have one or not, you can use the `persist` argument with the `login` function below to create or update an existing one, then use it for authentication.\n",
    "\n",
    "If you do not have an Earthdata Account, you can create one [here](https://urs.earthdata.nasa.gov/home). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "auth = earthaccess.login(persist=True)\n",
    "print(auth.authenticated)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you receive a message that your token has expired, use `refresh_tokens()` like below to generate a new one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# auth.refresh_tokens"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Searching for Collections\n",
    "\n",
    "The EMIT mission produces several collections or datasets available via the LP DAAC cloud archive.\n",
    "\n",
    "To view what's available, we can use the `search_datasets` function and with the `keyword` and and `provider` arguments. The `provider` is the data location, in this case `LPCLOUD`. Specifying the provider isn't necessary, but the \"emit\" keyword can be found in metadata for some other datasets, and additional collections may be returned.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Retrieve Collections\n",
    "collections = earthaccess.search_datasets(provider='LPCLOUD', keyword='emit')\n",
    "# Print Quantity of Results\n",
    "print(f'Collections found: {len(collections)}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you print the `collections` object you can explore all of the json metadata."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# # Print collections\n",
    "# collections"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also create a list of the `short-name`, `concept-id`, and `version` of each result collection using list comprehension. These fields are important for specifying and searching for data within collections. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "collections_info = [\n",
    "    {\n",
    "        'short_name': c.summary()['short-name'],\n",
    "        'collection_concept_id': c.summary()['concept-id'],\n",
    "        'version': c.summary()['version'],\n",
    "        'entry_title': c['umm']['EntryTitle']\n",
    "    }\n",
    "    for c in collections\n",
    "]\n",
    "pd.set_option('display.max_colwidth', 150)\n",
    "collections_info = pd.DataFrame(collections_info)\n",
    "collections_info"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The collection `concept-id` is the best way to search for data within a collection, as this is unique to each collection. The `short-name` can be used as well, however the `version` should be passed as well as there can be multiple versions available with the same short name. After finding the collection you want to search, you can use the `concept-id` to search for granules within that collection."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Searching for Granules\n",
    "\n",
    "A `granule` can be thought of as a unique spatiotemporal grouping within a collection. To search for `granules`, we can use the `search_data` function from `earthaccess` and provide the arguments for our search. Its possible to specify search products using several criteria shown in the table below:\n",
    "\n",
    "|dataset origin and location|spatio temporal parameters|dataset metadata parameters|\n",
    "|:---|:---|:---|\n",
    "|archive_center|bounding_box|concept_id\n",
    "|data_center|temporal|entry_title\n",
    "|daac|point|keyword\n",
    "|provider|polygon|version\n",
    "|cloud_hosted|line|short_name\n",
    "\n",
    "### Point Search\n",
    "\n",
    "In this case, we specify the `shortname`, `point` coordinates, `temporal` range, and min and max `cloud_cover` percentages, as well as `count`, which limits the maximum number of results returned. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Search example using a Point\n",
    "results = earthaccess.search_data(\n",
    "    short_name='EMITL2ARFL',\n",
    "    point=(-62.1123,-39.89402),\n",
    "    temporal=('2022-09-03','2022-09-04'),\n",
    "    cloud_cover=(0,90),\n",
    "    count=100\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Bounding Box Search\n",
    "\n",
    "You can also use a bounding box to search. To do this we will first open a geojson file containing our region of interest (ROI) then simplify it to a bounding box by getting the bounds and putting them into a Python object called a tuple. We will use the `total_bounds` property to get the bounding box of our ROI, and add that to a Python tuple, which is the expected data type for the bounding_box parameter `earthaccess` `search_data`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "geojson = gp.read_file('../../data/isla_gaviota.geojson')\n",
    "geojson.geometry"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bbox = tuple(list(geojson.total_bounds))\n",
    "bbox"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can search for granules using the a bounding box."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Search example using bounding box\n",
    "results = earthaccess.search_data(\n",
    "    short_name='EMITL2ARFL',\n",
    "    bounding_box=bbox,\n",
    "    temporal=('2022-09-03','2022-09-04'),\n",
    "    cloud_cover=(0,90),\n",
    "    count=100\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "### Polygon Search\n",
    "\n",
    "A polygon can also be used to search. For a simple polygon without holes we can take the geojson we opened and grab the coordinates of the exterior ring vertices and place them in a list. Note that this list of vertices must be in **counter-clockwise order** to be accepted by the `search_data` function. If necessary, the external ring vertices of your polygon can be reordered using the `orient` function from the shapely library."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Orient External Ring Vertices\n",
    "oriented = orient(geojson.geometry[0], sign=1.0)\n",
    "# Create List of External Ring vertices coordinates\n",
    "polygon = list(oriented.exterior.coords)\n",
    "polygon"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With this list of coordinate pairs we can use the `polygon` parameter for our search. \n",
    "> Note that we overwrote the `results` object, because for all 3 types spatial search, the `results` are the same for this example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Search Example using a Polygon\n",
    "results = earthaccess.search_data(\n",
    "    short_name='EMITL2ARFL',\n",
    "    polygon=polygon,\n",
    "    temporal=('2022-09-03','2022-09-04'),\n",
    "    cloud_cover=(0,90),\n",
    "    count=100\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Working with Search Results\n",
    "\n",
    "All three of these examples will have the same result, since the spatiotemporal parameters fall within the same single granule. Results is a `list`, so we can use an index to view a single result."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "result = results[0]\n",
    "result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also retrieve specific metadata for a result using `.keys()` since this object also acts as a dictionary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "result.keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Look at each of the keys to see what is available."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "result['meta']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "result['size']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `umm` metadata contains a lot of fields, so instead of printing the entire object, we can just look at the keys. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "result['umm'].keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One important piece of info here is the Look at the cloud cover percentage."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "result['umm']['CloudCover']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another of note is the `AdditionalAttributes` key, which contains other useful information about the EMIT granule, like solar zenith and azimuth."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "result['umm']['AdditionalAttributes']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From here, we can do other things, such as convert the results to a `pandas` dataframe, or filter down your results further using string matching and list comprehension."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.json_normalize(results)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Downloading or Streaming Data\n",
    "\n",
    "After we have our results, there are 2 ways we an work with the data:\n",
    "\n",
    "1. Download All Assets\n",
    "2. Selectively Download Assets\n",
    "3. Access in place / Stream the data. \n",
    "\n",
    "To download the data we can simply use the download function. This will retrieve all assets associated with a granule, and is nice if you plan to work with the data in this way and need all of the assets included with the product. For the EMIT L2A Reflectance, this includes the Uncertainty and Masks files."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "# earthaccess.download(results, '../../data/')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we want to stream the data or further filter the assets for download we want to first create a list of URLs nested by granule using list comprehesion."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "emit_results_urls = [granule.data_links() for granule in results]\n",
    "emit_results_urls"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can also split these into results for specific assets or filter out an asset using the following. In this example, we only want to access or download reflectance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "filtered_asset_links = []\n",
    "# Pick Desired Assets - Use underscores to aid in stringmatching of the filenames (_RFL_, _RFLUNCERT_, _MASK_)\n",
    "desired_assets = ['_RFL_']\n",
    "# Step through each sublist (granule) and filter based on desired assets.\n",
    "for n, granule in enumerate(emit_results_urls):\n",
    "    for url in granule: \n",
    "        asset_name = url.split('/')[-1]\n",
    "        if any(asset in asset_name for asset in desired_assets):\n",
    "            filtered_asset_links.append(url)\n",
    "filtered_asset_links"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After we have our filtered list, we can stream the reflectance asset or download it. Start an https session then open it to stream the data, or download to save the file.\n",
    "\n",
    "#### Stream Data\n",
    "\n",
    "This may take a while to load the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get Https Session using Earthdata Login Info\n",
    "fs = earthaccess.get_fsspec_https_session()\n",
    "# Retrieve granule asset ID from URL (to maintain existing naming convention)\n",
    "url = filtered_asset_links[0]\n",
    "granule_asset_id = url.split('/')[-1]\n",
    "# Define Local Filepath\n",
    "fp = fs.open(url)\n",
    "# Open with `emit_xarray` function\n",
    "ds = emit_xarray(fp)\n",
    "ds"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Download Filtered "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get requests https Session using Earthdata Login Info\n",
    "fs = earthaccess.get_requests_https_session()\n",
    "# Retrieve granule asset ID from URL (to maintain existing naming convention)\n",
    "for url in filtered_asset_links:\n",
    "    granule_asset_id = url.split('/')[-1]\n",
    "    # Define Local Filepath\n",
    "    fp = f'../../data/{granule_asset_id}'\n",
    "    # Download the Granule Asset if it doesn't exist\n",
    "    if not os.path.isfile(fp):\n",
    "        with fs.get(url,stream=True) as src:\n",
    "            with open(fp,'wb') as dst:\n",
    "                for chunk in src.iter_content(chunk_size=64*1024*1024):\n",
    "                    dst.write(chunk)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Contact Info:  \n",
    "\n",
    "Email: LPDAAC@usgs.gov  \n",
    "Voice: +1-866-573-3222  \n",
    "Organization: Land Processes Distributed Active Archive Center (LP DAAC)¹  \n",
    "Website: <https://lpdaac.usgs.gov/>  \n",
    "Date last modified: 11-06-2024  \n",
    "\n",
    "¹Work performed under USGS contract G15PD00467 for NASA contract NNG14HH33I. "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "lpdaac",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}