Project Structure

Project Structure foundation or template

This repository provides a minimal, yet opinionated baseline for Python [1] software projects. It includes:

• Basic Python "distribution"/project [2] metadata
• Command-line console script with subcommand boilerplate
• A Docker [3] container image for end users
• A Docker image for development, which runs tests
• A Makefile [4] for local development, build, test, and maintenance
• A target that formats all source, including Black [5] for Python code style
• A kitchen sink linter configuration [6] for Prospector [7] that runs all available Python code checks
• A tox.ini [8] configuration for Tox [9] to run all tests and linters across supported Python versions, including some checks not provided by Prospector.
• Version Control System (VCS) hooks [10] to enforce conventional commits [11], successful build and test on commit and push, and Towncrier [12] end-user oriented release notes on push
• Targets that automate releases
• Targets that automate dependency upgrades
• Targets used for both local development and on CI/CD platforms
• A GitLab CI/CD [13] pipeline integrating those CI/CD recipes and targets
• A GitHub Actions [14] workflow integrating those CI/CD recipes and targets
• Organize source by feature [15], for example \./src/foo/(template|view|model)\..*, rather than by source type, for example \./src/(templates|views|models)/foo\..*.

Add a VCS remote for this repository to a real project. When the template adds structure specific to certain types of projects, for example command-line tools, web services, UI apps, the it adds branches for each variant. When the template makes changes to structure common to different variants it can merge those changes into those variants. Real projects can also merge those changes.

Table of Contents

• Template usage
• Installation
  • Local Installation
  • Docker Container Image
• Usage
• Contributing
• Motivation
• References

Template usage

This is a rough guide for how to use this template in your project. This isn't widely tested. Such tests are meta and wasteful to create and support. Report any issues you have or better yet submit a PR with corrections.

1. Pick the branch to use:

   Pick the branch for your project type:

   • py:

     Basic Python distribution metadata and packaging.

   • py-cli:

     The preceding plus boilerplate for a command-line console script with subcommand.

   • docker:

     Docker images for development and end-users or deployments.

   • py-docker:

     The preceding Docker images with the Python package installed.

   • ci:

     The preceding plus GitLab CI/CD pipelines that run tests and linters as CI and publish releases from develop and main as CD.

   • py-ci:

     The py-docker and ci branches merged together.

   • py-ci-cli:

     All the preceding combined together.

   It's important to use one of the preceding branches to merge into your project and not the develop or main branches from the template. Template developers use those branches to test the release process. They contain bumped versions, release notes, and other release artifacts that real projects shouldn't merge into their code or history.

2. Merge into your project:

   If starting a fresh project:

   $ git clone --origin "template" --branch "${TEMPLATE_BRANCH:?}" \
   "https://gitlab.com/rpatterson/project-structure.git" "./foo-project"
   $ cd "./foo-project"
   $ git config remote.template.tagOpt --no-tags
   $ git remote add "origin" "[email protected]:foo-username/foo-project.git"
   $ git switch -C "main" --track "origin/main"
   

   If merging into an existing project:

   $ git remote add "template" \
   "https://gitlab.com/rpatterson/project-structure.git"
   $ git config remote.template.tagOpt --no-tags
   $ git merge --allow-unrelated-histories "template/${TEMPLATE_BRANCH:?}"
   

3. Rename files and directories derived from the project name:

   $ git ls-files | grep -iE 'project.?structure'
   

4. Rename project name and template creator identity strings:

   $ git grep -iE 'project.?structure|ross|Patterson'
   

5. Make changes described in # TEMPLATE: comments:

   These bits need the developer's attention and reasoning. Read the comments and follow them with care:

   $ git grep "TEMPLATE"
   

Lastly, remove this Template usage section and update the rest of this ./README.rst for your project. When the template adds fixes and features, merge them into your project and repeat steps 3--5.

This template publishes pre-releases on push to the develop branch and final releases on push to the main branch. Project owners can decide the types of changes that require a pre-release before final release and the types of changes that go straight to final release. For example they can decide that:

• Merge public contributions into develop. See the contributing documentation [16] for an example public contributions policy and workflow.
• Optionally commit fixes for bugs in final releases to a branch off main. After passing all tests and checks, merge back into main to publish final releases directly.
• Optionally also merge version upgrades for security updates directly to main.

Installation

Install locally or use the Docker container image:

Local Installation

Install by using any tool for installing standard Python 3 distributions such as pip [17]:

$ pip3 install --user project-structure

Optional shell prompt tab completion is available by using argcomplete [18].

Docker Container Image

The recommended way to use the container image is by using Docker Compose [19]. See the example ./docker-compose.yml file [20]. Write your configuration and run the container:

$ docker compose up

You can also use the image directly. Pull the Docker image [21]. Use it to create and run a container:

$ docker pull "registry.gitlab.com/rpatterson/project-structure"
$ docker run --rm -it "registry.gitlab.com/rpatterson/project-structure" ...

Use image variant tags to control when the image updates. Releases publish tags for the branch and for major and minor versions. For example, to keep up to date with a specific branch, use a tag such as registry.gitlab.com/rpatterson/project-structure:main. Releases from develop publish pre-releases. Releases from main publish final releases. Releases from main also publish tags without a branch, for example registry.gitlab.com/rpatterson/project-structure. Releases from main also publish tags for the major and minor version, for example registry.gitlab.com/rpatterson/project-structure:v0.8.

Releases publish multi-platform images for the following platforms:

• linux/amd64
• linux/arm64
• linux/arm/v7

Usage

See the command-line help for details on options and arguments:

$ project-structure --help
usage: project-structure [-h]

project structure foundation or template, top-level package.

Pptional arguments:
  -h, --help  show this help message and exit

The Docker container image can run the command-line script as well:

$ docker compose run "project-structure" project-structure --help
usage: project-structure [-h]

Project structure foundation or template, top-level package.

optional arguments:
  -h, --help  show this help message and exit

Contributing

GitLab hosts this project [22] and mirrors it to GitHub [23] but use GitLab for reporting issues, submitting pull or merge requests and any other development or maintenance activity. See the contributing documentation [16] for more details on how to get started with development.

Motivation

Plenty other project templates exists. Why make another? I've been a full-stack web developer from 1998 on. I've had plenty of time to develop plenty of opinions of my own. From a template I want a full tool set (for example test coverage, linting, formatting, CI). Conversely, I want minimal dependencies, structure, and opinion beyond a full tool set (for example some build or task system, structure for unused frameworks or libraries). I couldn't find a template that manages that balance and I created this one.

I also find it hard to discern from other templates why they made what choices the did. As such, I also use this template to try out different options and learn for myself. You can learn about my findings and the reasons the choices I've made in the commit history.

Most importantly I've never found a satisfactory approach to keeping project structure up to date over time. As such, the primary motivation is providing a template upstream remote, merging structure updates into real projects over their lifetime.

References

[1]	https://docs.python.org/3/library/logging.html

[2]	https://docs.python.org/3/distributing/index.html

[3]	https://docs.docker.com/

[4]	https://gitlab.com/rpatterson/project-structure/-/blob/main/Makefile

[5]	https://github.com/psf/black

[6]	https://gitlab.com/rpatterson/project-structure/-/blob/main/.prospector.yaml

[7]	https://prospector.landscape.io/en/master/

[8]	https://gitlab.com/rpatterson/project-structure/-/blob/main/tox.ini

[9]	https://tox.wiki/en/stable/

[10]	https://gitlab.com/rpatterson/project-structure/-/blob/main/.pre-commit-config.yaml

[11]	https://www.conventionalcommits.org

[12]	https://towncrier.readthedocs.io/en/stable/

[13]	https://docs.gitlab.com/ee/ci/

[14]	https://docs.github.com/en/actions

[15]	https://www.seancdavis.com/posts/organize-components-by-keeping-related-files-close/

[16]	(1, 2) https://gitlab.com/rpatterson/project-structure/-/blob/main/docs/contributing.rst

[17]	https://pip.pypa.io/en/stable/installation/

[18]	https://kislyuk.github.io/argcomplete/#installation

[19]	https://docs.docker.com/compose/

[20]	https://gitlab.com/rpatterson/project-structure/-/blob/main/docker-compose.yml

[21]	https://hub.docker.com/r/merpatterson/project-structure

[22]	https://gitlab.com/rpatterson/project-structure

[23]	https://github.com/rpatterson/project-structure