Netbox Restarts in Nomad

[k1rb]

Current Behavior

In nomad, using the docker driver, the "netbox" application exits (0) and is started again, constantly. I think the process is being killed due to resource (over) utilization but I can't seem to give it enough resources to keep it from restarting. The task or allocation associated with the netbox application isn't being killed or HUP'd by nomad, the process within the allocation/container is exiting and starting again.

The result is a momentary pause when working with netbox or worse, a 3 second http 503 from the reverse proxy, if it happens to coincide with a healthcheck.

The netbox-worker and netbox-housekeeping containers from the docker-compose haven't been implemented in my jobspec yet so this might be part of the problem.

Expected Behavior

The process starts once and doesn't exit.

Docker Compose Version

NA

Docker Version

Client: Docker Engine - Community
 Version:           24.0.7
 API version:       1.43
 Go version:        go1.20.10
 Git commit:        afdd53b
 Built:             Thu Oct 26 09:09:18 2023
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          24.0.7
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.10
  Git commit:       311b9ff
  Built:            Thu Oct 26 09:08:20 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.24
  GitCommit:        61f9fd88f79f081d64d6fa3bb1a0dc71ec870523
 runc:
  Version:          1.1.9
  GitCommit:        v1.1.9-0-gccaecfc
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

The git Revision

NA

The git Status

NA

Startup Command

nomad job run

NetBox Logs

2024/03/27 21:45:22 [info] 672#672 "netbox" application started
# normal http access logs...
🧬 loaded config '/etc/netbox/config/configuration.py'
🧬 loaded config '/etc/netbox/config/extra.py'
🧬 loaded config '/etc/netbox/config/logging.py'
🧬 loaded config '/etc/netbox/config/plugins.py'
2024/03/27 21:45:23 [notice] 19#19 app process 669 exited with code 0
# normal http access logs...

Content of docker-compose.override.yml

job "netbox" {

  datacenters = ["dc1"]
  type = "service"

  group "app" {

    count = 1

    network {
      port "http"        { to = 8080 }
      port "psql"        { to = 5432 }
      port "redis"       { to = 6379 }
      port "redis-cache" { to = 6379 }

    }

    # service provider consul with health checks

    task "netbox-app" {
      driver = "docker"
      config {
        image = "docker.io/netboxcommunity/netbox:v3.6-2.7.0"
        ports = ["http"]
        volumes = [
          "/srv/containers/netbox/configuration:/etc/netbox/config",
          "/srv/containers/netbox/data/media:/opt/netbox/netbox/media",
          "/srv/containers/netbox/data/reports:/opt/netbox/netbox/reports",
          "/srv/containers/netbox/data/scripts:/opt/netbox/netbox/scripts"
        ]
      }
      # template for env values from consul kv & vault
      resources {
        cpu        = 6000 # 6Ghz
        memory     = 6144 # 6G
      }
    }

#   task "netbox-worker" {} from docker-compose, not implemented 

#   task "netbox-housekeeping" {} from docker-compose, not implemented 

    task "netbox-postgres" {
      driver = "docker"
      config {
        image = "docker.io/postgres:14-alpine"
        ports = ["psql"]
        volumes = [
          "/srv/containers/netbox/data/postgres/data:/var/lib/postgresql/data"
        ]
      }
      # template for env values from consul kv & vault
      resources {
        cpu        = 1000
        memory     = 1024
      }
    }

    task "netbox-redis" {
      driver = "docker"
      config {
        image = "docker.io/redis:6-alpine"
        command = "redis-server"
        args = ["--appendonly","yes","--requirepass","${REDIS_PASSWORD}",]
        ports = ["redis"]
        mount {
          type = "bind"
          source = "/srv/containers/netbox/data/redis/data"
          target = "/data"
          readonly = false
        }
      }
      # template for env values from consul kv & vault
    }

    task "netbox-redis-cache" {
      driver = "docker"
      config {
        image = "docker.io/redis:6-alpine"
        ports = ["redis-cache"]
        command = "redis-server"
        args = ["--requirepass","${REDIS_PASSWORD}"]
        volumes = [
          "/srv/containers/netbox/data/redis-cache/data:/data"
        ]
      }
      # template for env values from consul kv & vault
    }
  }
}

[k1rb]

I've actually found that this also happens in a completely separate environment, being run with podman-compose version 1.0.6. 🤔

netbox version v3.6.4 with podman-compose
netbox version v3.6.9 with nomad

[k1rb]

The behavior that I'm seeing seems to be normal (I see it in every deployment) so I've decreased my task resources to something reasonable and then increased the service check timeout from 3s to 10s and that seems to have taken care of the issue. There's still a delay when the process restarts but it's not bad enough to cause the service to fail the check. If I continue to have problems working with netbox I'll increase this timeout up to 30s before trying something else.

[kkthxbye-code]

This is intended and caused by nginx unit.

See:

https://unit.nginx.org/configuration/#application-processes

https://unit.nginx.org/configuration/#request-limits

I recommend setting max and spare processes to the same number and optionally setting a request limit to avoid any potential memory leaks. Having the processes exit makes the first request very slow if it has to spawn a new one, so I always disable the behavior.