Publish image to container registries

Add Github Action for Container build and push

Limit number of container layers and reduce size

Update COPY chmod to use octal

Stop copying DOCS directory into container

Reduce number of production container layers

Remove linux/arm64 container platform

Convert django container to multi-stage build

Reduction of image from ~1GB to <550MB

Conslidate production/local docker-compose/Dockerfile

Move compose/production/* into parent compose/ folder

Remove defunct compose/local folder

Remove sharing lock from cache mount on RUN

The cache was empty when using a `locked` sharing type.
Reverting to the default sharing of `shared` as the build should
not be affected with the current workflow.

https://github.com/moby/buildkit/blob/f2a6e83adcb0295099870489b76d3ce74d6f7f42/frontend/dockerfile/docs/syntax.md#run---mounttypecache

> This mount type allows the build container to cache directories for
compilers and package managers.

|Option               |Description|
|---------------------|-----------|
|`sharing`            | One of `shared`, `private`, or `locked`.
Defaults to `shared`. A `shared` cache mount can be used concurrently by
multiple writers. `private` creates a new mount if there are multiple
writers. `locked` pauses the second writer until the first one releases
the mount.|

Update workflow to run on 'master' branch

Limit GITHUB_TOKEN permissions for job

Only push the container on push and schedule events

ghaction-docker-meta action moved from crazy-max to docker org

https://github.com/docker/metadata-action/releases/tag/v3.0.0

Use bind mount rather than cache for wheels

cache is not guaranteed.  build the wheels and allow pip to cache during the process.
bind mount the wheel-dir when installing in django stage

Consolidate start commands

Remove extraneous instructions from Dockerfile

- no need to install the requirements in the build stage - only build the
wheels

- gecos is for storing metadata about a user (full name, phone number...)

- copy of requirements from build is now handled transparently through a
bind mount without requiring the additional layer

- du of /tmp/wheels was only for debugging the cache mount which is now
a bind mount

- /tmp/requirements is a bind mount so it does not need to be removed
from the stage

Cache first stage of multi-stage build

By default the mode is set to `min`, which only exports layers to the
cache in the final build stage.  We want to cache the first stage in
order to not always build wheels.

Specify ghostwriter:2.2 image in production.yml

Add missing EOF newline to docker components

Update to default compose in run-unit-tests github workflow job

local.yml was replaced with docker-compose.override.yml which is automatically
applied when no other compose files are specified

.dockerignore

@@ -2,3 +2,4 @@
 !.coveragerc
 !.env
 !.pylintrc
+DOCS/**

.gitattributes

@@ -1 +1,3 @@
-* text=auto
+* text=auto eol=lf
+*.ico binary
+ghostwriter/static/images/** binary

.github/workflows/push-container.yml

@@ -0,0 +1,187 @@
+name: push-container
+
+on:
+  push:
+    branches:
+      - 'master'
+      - 'feature/push-to-container-registry'
+      - 'feature/push-to-container-registry-v3.0.0'
+    tags:
+      - 'v*'
+  schedule:
+    # * is a special character in YAML so you have to quote this string
+    # at 03:00 on the 1st and 15th of the month
+    - cron:  '0 3 1,15 * *'
+
+env:
+  PLATFORMS: linux/amd64 # multiple platforms can be specified as: linux/amd64,linux/arm64
+
+jobs:
+  build_and_push_container_image:
+    runs-on: ubuntu-latest
+
+    permissions:
+      # when permissions are defined only those that are explicitly set will be enabled
+      # this workflow job currently only requires reading contents and writing packages.
+      # https://docs.github.com/en/actions/reference/authentication-in-a-workflow#modifying-the-permissions-for-the-github_token
+      contents: read
+      packages: write
+
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v2
+
+      - name: Validate secret defined
+        id: from_secrets
+        run: |
+          github_container_push="true";
+          dockerhub_token_exists="false";
+          dockerhub_username_exists="false";
+          dockerhub_namespace_exists="false";
+
+          [[ -n "${{ secrets.DOCKERHUB_TOKEN }}" ]] && dockerhub_token_exists="true";
+          [[ -n "${{ secrets.DOCKERHUB_USERNAME }}" ]] && dockerhub_username_exists="true";
+          [[ -n "${{ secrets.DOCKERHUB_NAMESPACE }}" ]] && dockerhub_namespace_exists="true";
+          [[ "true" = "${{ secrets.GITHUB_CONTAINER_PUSH_DISABLED }}" ]] && github_container_push="false";
+
+          echo "::set-output name=dockerhub_token_exists::${dockerhub_token_exists}";
+          echo "::set-output name=dockerhub_username_exists::${dockerhub_username_exists}";
+          echo "::set-output name=dockerhub_namespace_exists::${dockerhub_namespace_exists}";
+          echo "::set-output name=github_container_push::${github_container_push}";
+
+      - name: Generate container image names
+        id: generate_image_names
+        run: |
+          repository_name="$(basename "${GITHUB_REPOSITORY}")";
+          images=();
+
+          if [[ "${{ steps.from_secrets.outputs.github_container_push }}" = "true" ]];
+          then
+            # set GITHUB_CONTAINER_PUSH_DISABLED to a value of true to disable pushing to github container registry
+            images+=("ghcr.io/${GITHUB_REPOSITORY}");
+          fi
+
+          if [[ -n "${{ secrets.DOCKERHUB_TOKEN }}" ]] && [[ -n "${{ secrets.DOCKERHUB_USERNAME }}" ]] && [[ -n "${{ secrets.DOCKERHUB_NAMESPACE }}" ]];
+          then
+            # dockerhub repository should be the same as the github repository name, within the dockerhub namespace (organization or personal)
+            images+=("${{ secrets.DOCKERHUB_NAMESPACE }}/${repository_name}");
+          fi
+
+          # join the array for Docker meta job to produce image tags
+          # https://github.com/crazy-max/ghaction-docker-meta#inputs
+          echo "::set-output name=images::$(IFS=,; echo "${images[*]}")";
+
+      - name: Docker ghostwriter meta
+        id: meta
+        uses: docker/metadata-action@v3
+        with:
+          images: ${{ steps.generate_image_names.outputs.images }}
+          tags: |
+            type=schedule,pattern={{date 'YYYYMMDD'}}
+            type=edge,branch=master
+            type=ref,event=branch
+            type=ref,event=pr
+            type=ref,event=tag
+            type=semver,pattern={{version}}
+            type=semver,pattern={{major}}.{{minor}}
+            type=semver,pattern={{major}}
+
+      - name: Docker ghostwriter:postgres meta
+        id: meta-postgres
+        uses: docker/metadata-action@v3
+        with:
+          images: ${{ steps.generate_image_names.outputs.images }}
+          flavor: |
+            prefix=postgres-
+          tags: |
+            type=schedule,pattern={{date 'YYYYMMDD'}}
+            type=edge,branch=master
+            type=ref,event=branch
+            type=ref,event=pr
+            type=ref,event=tag
+            type=semver,pattern={{version}}
+            type=semver,pattern={{major}}.{{minor}}
+            type=semver,pattern={{major}}
+
+      - name: Set up QEMU
+        uses: docker/setup-qemu-action@v1
+
+      - name: Set up Docker Buildx
+        id: buildx
+        uses: docker/setup-buildx-action@v1
+        with:
+          version: latest
+
+      - name: Login to DockerHub
+        uses: docker/login-action@v1
+        # conditions do not have direct access to github secrets so we check the output of the step from_secrets
+        if: ${{ steps.from_secrets.outputs.dockerhub_namespace_exists == 'true' && steps.from_secrets.outputs.dockerhub_token_exists == 'true' && steps.from_secrets.outputs.dockerhub_username_exists == 'true' }}
+        with:
+          username: ${{ secrets.DOCKERHUB_USERNAME }}
+          password: ${{ secrets.DOCKERHUB_TOKEN }}
+
+      - name: Login to GitHub Container Registry
+        uses: docker/login-action@v1
+        if: ${{ steps.from_secrets.outputs.github_container_push == 'true' }}
+        with:
+          registry: ghcr.io
+          username: ${{ github.repository_owner }}
+          password: ${{ secrets.GITHUB_TOKEN }}
+
+      - name: Cache Docker layers
+        uses: actions/cache@v2
+        with:
+          path: /tmp/.buildx-cache
+          # Caches are scoped to the current branch and parent branch.
+          # Cache miss can happen on first run of a new branch
+          # If there is a matching cache key in the default branch then that should be used.
+          # https://docs.github.com/en/actions/guides/caching-dependencies-to-speed-up-workflows#matching-a-cache-key
+          # cache key is a hash of the base and production requirements.  Changes to these files will cause a full rebuild.
+          key: ${{ runner.os }}-buildx-${{ hashFiles('requirements/base.txt', 'requirements/production.txt') }}
+          restore-keys: |
+            ${{ runner.os }}-buildx-
+
+      - name: Build and push - ghostwriter
+        uses: docker/build-push-action@v2
+        with:
+          builder: ${{ steps.buildx.outputs.name }}
+          context: .
+          file: ./compose/django/Dockerfile
+          platforms: ${{ env.PLATFORMS }}
+          push: ${{ contains(fromJson('["push", "schedule"]'), github.event_name) }}
+          labels: ${{ steps.meta.outputs.labels }}
+          tags: ${{ steps.meta.outputs.tags }}
+          cache-from: type=local,src=/tmp/.buildx-cache
+          cache-to: type=local,dest=/tmp/.buildx-cache-new,mode=max
+          # type=gha will replace type=local when a buildx release containing
+          # https://github.com/docker/buildx/commit/5ca0cbff8ed63450a6d4a3b32659e9521d329a43 is published
+          # https://github.com/docker/buildx/pull/535
+          # cache-from: type=gha
+          # cache-to: type=gha
+
+      - name: Build and push - ghostwriter:postgres
+        uses: docker/build-push-action@v2
+        with:
+          builder: ${{ steps.buildx.outputs.name }}
+          context: .
+          file: ./compose/postgres/Dockerfile
+          platforms: ${{ env.PLATFORMS }}
+          push: ${{ contains(fromJson('["push", "schedule"]'), github.event_name) }}
+          labels: ${{ steps.meta-postgres.outputs.labels }}
+          tags: ${{ steps.meta-postgres.outputs.tags }}
+          cache-from: type=local,src=/tmp/.buildx-cache-new
+          cache-to: type=local,dest=/tmp/.buildx-cache-new,mode=max
+          # type=gha will replace type=local when a buildx release containing
+          # https://github.com/docker/buildx/commit/5ca0cbff8ed63450a6d4a3b32659e9521d329a43 is published
+          # https://github.com/docker/buildx/pull/535
+          # cache-from: type=gha
+          # cache-to: type=gha
+
+      - name: Move cache
+        # This step can be removed when cache-from/cache-to have been updated to use type=gha
+        # https://github.com/docker/build-push-action/issues/252
+        # https://github.com/moby/buildkit/issues/1896
+        if: always()
+        run: |
+          rm -rf /tmp/.buildx-cache
+          mv /tmp/.buildx-cache-new /tmp/.buildx-cache

compose/django/Dockerfile

@@ -0,0 +1,162 @@
+#syntax=docker/dockerfile:1
+ARG STAGE=production
+
+# ---------------------------------------------
+# BEGIN build image stage
+# ---------------------------------------------
+FROM python:3.8-alpine as build
+ARG STAGE=production
+
+# only update build build when requirements have changed
+COPY ./requirements /requirements
+# install build dependencies
+RUN --mount=type=cache,mode=0755,target=/root/.cache/pip \
+  apk update \
+  && apk add --no-cache build-base \
+  # psycopg2 dependencies
+  && apk add --no-cache --virtual build-deps gcc python3-dev musl-dev \
+  && apk add --no-cache postgresql-dev \
+  # Pillow dependencies
+  && apk add --no-cache jpeg-dev zlib-dev freetype-dev lcms2-dev openjpeg-dev tiff-dev tk-dev tcl-dev \
+  # CFFI dependencies
+  && apk add --no-cache libffi-dev py-cffi \
+  # XLSX dependencies
+  && apk add --no-cache libxml2-dev libxslt-dev \
+  # Rust and Cargo required by the ``cryptography`` Python package - only required during build
+  && apk add --no-cache rust \
+  && apk add --no-cache cargo \
+  # build wheels
+  && pip install wheel && pip wheel --wheel-dir=/tmp/wheels -r /requirements/${STAGE}.txt \
+  # remove the virtual package group 'build-deps'
+  && apk del build-deps
+# ---------------------------------------------
+# END build image stage
+# ---------------------------------------------
+
+# ---------------------------------------------
+# BEGIN django image stage
+# ---------------------------------------------
+FROM python:3.8-alpine as django
+ARG STAGE=production
+
+# stream python output for django logs
+ENV PYTHONUNBUFFERED 1
+
+ENV PYTHONPATH="$PYTHONPATH:/app/config"
+
+ARG USER_UID=1000
+ARG USER_GID=$USER_UID
+RUN if [ -n "$(getent group ${USER_GID})" ]; \
+  then \
+  apk --no-cache add shadow; \
+  groupmod -n "django" "${USER_GID}"; \
+  else \
+  addgroup --gid "${USER_GID}" "django"; \
+  fi && \
+  if [ -n "$(getent passwd ${USER_UID})" ]; \
+  then \
+  apk --no-cache add shadow; \
+  usermod -l "django" -g "${USER_GID}" -d "/app"; \
+  else \
+  adduser \
+  --home "/app" \
+  --shell /bin/ash \
+  --ingroup "django" \
+  --system \
+  --disabled-password \
+  --no-create-home \
+  --uid "${USER_UID}" \
+  "django"; \
+  fi
+
+# install runtime dependencies.  `add --no-cache` performs an apk update, adds packages and excludes caching
+# in order to not require deletion of apk cache.
+RUN apk add --no-cache postgresql-dev \
+  # Pillow dependencies
+  jpeg-dev zlib-dev freetype-dev lcms2-dev openjpeg-dev tiff-dev tk-dev tcl-dev \
+  # CFFI dependencies
+  libffi-dev py-cffi \
+  # XLSX dependencies
+  libxml2-dev libxslt-dev
+
+# combine build and ${STAGE}.txt - remove --no-binary to install our own wheels
+RUN --mount=type=bind,target=/tmp/wheels,source=/tmp/wheels,from=build \
+  --mount=type=bind,target=/requirements,source=/requirements,from=build,readwrite \
+  --mount=type=cache,mode=0755,target=/root/.cache/pip \
+  ( cat /requirements/base.txt; sed -e 's/--no-binary.*//' -e 's/^-r .*//' /requirements/${STAGE}.txt ) | tee /tmp/requirements.txt >/dev/null \
+  && pip install --find-links=/tmp/wheels -r /tmp/requirements.txt \
+  && rm -rf /tmp/requirements.txt
+# ---------------------------------------------
+# END django image stage
+# ---------------------------------------------
+
+# ---------------------------------------------
+# BEGIN production stage
+# ---------------------------------------------
+FROM django as django-production
+
+# add our application
+COPY --chown=django . /app
+
+# copy the entrypoint and run scripts
+RUN for target in /app/compose/django/*; \
+  do ln "$target" /"$(basename "$target")" \
+  && chmod -v 0755 /"$(basename "$target")" \
+  # remove all carriage returns in the case that a user checks out the files on a windows system
+  # and has their git core.eol set to native or crlf
+  && sed -i 's/\r$//g' /"$(basename "$target")"; \
+  done \
+  # due to volumes mounted to these locations we must created and set the ownership of the underlying directory
+  # so that it is correctly propagated to the named volume
+  && mkdir -p "/app/ghostwriter/media" "/app/staticfiles" \
+  && chown -R "django": "/app/ghostwriter/media" "/app/staticfiles"
+# ---------------------------------------------
+# END production stage
+# ---------------------------------------------
+
+# ---------------------------------------------
+# BEGIN local stage
+# ---------------------------------------------
+FROM django as django-local
+
+# add our application CMD scripts
+COPY --chown=django ./compose/django/ /
+
+# copy the entrypoint and run scripts
+RUN find / -maxdepth 1 -type f -exec chmod -v 0755 {} \; \
+  # remove all carriage returns in the case that a user checks out the files on a windows system
+  # and has their git core.eol set to native or crlf
+  && find / -maxdepth 1 -type f -exec sed -i 's/\r$//g' {} \; \
+  # due to volumes mounted to these locations we must created and set the ownership of the underlying directory
+  # so that it is correctly propagated to the named volume
+  && mkdir -p "/app/ghostwriter/media" "/app/staticfiles" \
+  && chown -R "django": "/app/ghostwriter/media" "/app/staticfiles"
+# ---------------------------------------------
+# END local stage
+# ---------------------------------------------
+
+# ---------------------------------------------
+# BEGIN conditional stage
+# with buildkit/bake only referenced stages will be built starting from this stage
+# ---------------------------------------------
+FROM django-${STAGE} as conditional
+
+USER "django"
+
+WORKDIR /app
+# ---------------------------------------------
+# END conditional stage
+# ---------------------------------------------
+
+# ---------------------------------------------
+# BEGIN live stage
+# ---------------------------------------------
+FROM conditional as live
+
+VOLUME ["/app/ghostwriter/media", "/app/staticfiles"]
+
+CMD ["/start"]
+ENTRYPOINT ["/entrypoint"]
+# ---------------------------------------------
+# END live stage
+# ---------------------------------------------