Skip to content

Backend

Jump to bottom
svchot edited this page May 23, 2024 · 27 revisions

Backend Deployment for Development

The recommended way to run FMTM is with Docker.

You can also develop on your local machine outside of Docker, see below.

NOTE: If you haven't yet downloaded the Repository and setup your environment variables, please check the Getting Started wiki page.

Now let's get started 👍

1. Start the API with Docker

The easiest way to get up and running is by using the FMTM Docker deployment. Docker creates a virtual environment, isolated from your computer's environment, installs all necessary dependencies, and creates a container for each the database, the api, and the frontend. These containers talk to each other via the URLs defined in the docker-compose file and your env file.

  1. You will need to Install Docker and ensure that it is running on your local machine.
  2. From the command line: navigate to the top level directory of the FMTM project.
  3. From the command line run: docker-compose pull. This will pull the latest container builds from main branch.
  4. Make sure you have a .env file with all required variables, see Getting Started.
  5. Once everything is pulled, from the command line run: docker compose up -d api
  6. If everything goes well you should now be able to navigate to the project in your browser: http://api.fmtm.localhost:7050/docs

Note: If that link doesn't work, check the logs with docker compose logs api. Note: the database host fmtm-db is automatically resolved by docker compose to the database container IP.

Bundled ODK Central

  • FMTM uses ODK Central to store ODK data.
  • To facilitate faster development, the Docker setup includes a Central server.
  • The credentials are provided via the .env file, and the default URL to access Central from within containers is: https://proxy.

Alternatively, you may provide credentials to an external Central server in the .env.

To run the local development setup without ODK Central (use external server):

dc --profile no-odk up -d

2. Start the API without Docker

  • To run FMTM without Docker, you will need to start the database, then the API.
  • First start a Postgres database running on a port on your machine.
    • The database must have the Postgis extension installed.
  • After starting the database, from the command line:
  1. Navigate to the top level directory of the FMTM project.
  2. Install PDM with: pip install pdm
  3. Install backend dependencies with PDM: pdm install
  4. Run the Fast API backend with: pdm run uvicorn app.main:api --host 0.0.0.0 --port 8000

The API should now be accessible at: http://api.fmtm.localhost:7050/docs

Backend Tips

Database Migration

Creating Migration Files

  • Migrations can be written to src/backend/migrations.
  • Each file must be an SQL script that is:
    • Idempotent: can be run multiple times without consequence.
    • Atomic: Run within a BEGIN/COMMIT transaction.
  • Migrations must also include an equivalent revert migration under: src/backend/migrations/revert

Applying Migrations

  • Should occur automatically as part of the docker compose stack (migration service).
  • To run manually:
docker compose up -d migrations

Type Checking

  • It is a good idea to have your code 'type checked' to avoid potential future bugs.
  • To do this, install pyright (VSCode has an extension).
  • You may need to add the backend dependencies to extraPaths. In VSCode your settings.json would include:
{
  "python.analysis.extraPaths": ["src/backend/__pypackages__/3.10/lib/"]
}

Interactive Debugging

  • The docker-compose.yml builds FMTM using the debug target in the Dockerfile.
  • The debug image contains debugpy to assist debugging in the container.

To use it:

  1. Re-build the docker image docker compose build api

  2. Uncomment the debug port in docker-compose.yml:

    services:
  ...
  api:
    ...
    ports:
      - "7052:8000"
    #   - "5678:5678" # Debugger port

  3. Start the docker container docker compose up -d api

  4. Connect to the debugger on port 5678.

You can configure your IDE to do this with the build in debugger.

Example launch.json config for vscode:

{
  "configurations": [
    {
      "name": "Remote - Server Debug",
      "type": "python",
      "request": "attach",
      "host": "localhost",
      "port": 5678,
      "pathMappings": [
        {
          "localRoot": "${workspaceFolder}/src/backend/app",
          "remoteRoot": "/opt/app"
        }
      ],
      "justMyCode": false
    }
  ]
}

Note: either port 5678 needs to be bound to your localhost (default), or the host parameter can be set to the container IP address.

Running Tests

To run the backend tests locally, run:

docker compose run --rm api pytest

To assess coverage of tests, run:

docker compose run --rm --entrypoint='sh -c' api \
  'coverage run -m pytest && coverage report -m'

To assess performance of endpoints:

  • We can use the pyinstrument profiler.
  • While in debug mode (DEBUG=True), access any endpoint.
  • Add the ?profile=true arg to the URL to view the execution time.

Debugging osm-fieldwork

osm-fieldwork is an integral package for much of the functionality in FMTM.

Creating a new release during development may not always be feasible.

  • A development version of osm-fieldwork can be mounted into the FMTM container via bind mount.
  • Clone the osm-fieldwork repo to the same root directory as FMTM.
  • Uncomment the line in docker-compose.yml
- ../osm-fieldwork/osm_fieldwork:/home/appuser/.local/lib/python3.10/site-packages/osm_fieldwork
  • Run the docker container with your local version of osm-fieldwork.
  • Code changes to osm-fieldwork should be reflected immediately. If they are not, run: docker compose restart api.

Note: this is useful for debugging features during active development.

Accessing S3 Files use s3fs

The s3fs tool allows you to mount an S3 bucket on your filesystem, to browse like any other directory.

Install:

sudo apt update
sudo apt install s3fs

Create a credentials file:

# Replace ACCESS_KEY_ID and SECRET_ACCESS_KEY
echo ACCESS_KEY_ID:SECRET_ACCESS_KEY > ${HOME}/.passwd-s3fs
chmod 600 ${HOME}/.passwd-s3fs

Mount your bucket:

If you wish for this to be permanent, see below.

sudo mkdir /mnt/fmtm/local
sudo chown $(whoami):$(whoami) /mnt/fmtm/local
s3fs fmtm-data /mnt/fmtm/local \
  -o passwd_file=/home/$(whoami)/s3-creds/fmtm-local \
  -o url=http://s3.fmtm.localhost:7050 \
  -o use_path_request_style

Access the files like a directory under: /mnt/fmtm/local.

To mount permanently, add the following to /etc/fstab:

fmtm-data /mnt/fmtm/local fuse.s3fs _netdev,allow_other,\ use_path_request_style,passwd_file=/home/USERNAME/s3-creds/fmtm-local,\ url=http://s3.fmtm.localhost:7050 0 0

Note: you should replace USERNAME with your linux username.

Running JOSM in the dev stack

  • Run JOSM with FMTM:
docker compose \
  -f docker-compose.yml \
  -f contrib/josm/docker-compose.yml \
  up -d

This adds JOSM to the docker compose stack for local development. Access the JOSM Remote API: http://localhost:8111 Access the JOSM GUI in browser: http://localhost:8112

You can now call the JOSM API from FMTM and changes will be reflected in the GUI.

Debugging local FMTM on mobile

  • It's difficult to debug services running on localhost from your mobile phone.

  • An easy way to do this is by tunneling: Cloudflare provides a great free solution for this (an alternative is Ngrok).

  • To run the tunnel to the FMTM API:

    docker compose \
  -f docker-compose.yml \
  -f contrib/tunnel/fmtm/docker-compose.yml \
  up -d

  • View the website to access FMTM remotely (e.g. via mobile):

    docker compose \
  -f docker-compose.yml \
  -f contrib/tunnel/fmtm/docker-compose.yml \
  logs api-tunnel

  • Now the final step is to add the provided tunnel URL to the allowed CORS origins.

  • Add to your .env file:

    EXTRA_CORS_ORIGINS=https://the-url-you-were-given.trycloudflare.com

  • Then restart your API service:

    docker compose restart api

Using local ODK Central on mobile

  • Sometimes you wish to use a project in your local ODK Central, via ODK Collect on your mobile.

  • To run the tunnel to the ODK Central API:

    docker compose \
  -f docker-compose.yml \
  -f contrib/tunnel/odk/docker-compose.yml \
  up -d

  • View the website to access ODK Central remotely (e.g. via mobile):

    docker compose \
  -f docker-compose.yml \
  -f contrib/tunnel/odk/docker-compose.yml \
  logs central-tunnel

  1. Requirement: Restart ODK Central using the domain override (required for form download URLs):

    CENTRAL_DOMAIN_OVERRIDE=tramadol-handbags-protecting-date.trycloudflare.com \
  docker compose up -d central

  2. Requirement: During project creation, set the ODK Central server URL to the provided tunnel URL for the ODK Central API.

    The credentials for the local ODK Central instance are: Username: [email protected] Password: Password1234

  • Now when you access the project via a QRCode on mobile, the connection to ODK Central should work.
Clone this wiki locally