Add Functional Horizontal scaling for Slack

[pablonyx]

Description

The gist of it

• Tenant Acquisition Loop

  • Pods run acquire_tenants_loop at intervals (TENANT_ACQUISITION_INTERVAL).
    • Fetch all tenant IDs from the database and attempt to acquire a Redi lock which expires after TENANT_LOCK_EXPIRATION. If acquired, begin handling

• Heartbeat Mechanism

  • Pods run heartbeat_loop at intervals (TENANT_HEARTBEAT_INTERVAL).
    • For each tenant the pod handles:
      • Update a heartbeat key in Redis (DanswerRedisLocks.SLACK_BOT_HEARTBEAT_PREFIX:{pod_id}).
        • Indicates the pod is alive.
        • Heartbeat expires after TENANT_HEARTBEAT_EXPIRATION.
  • Ensuring One Pod per Tenant:
    • If a pod fails or stops sending heartbeats:
      • Its locks expire after TENANT_LOCK_EXPIRATION.
      • Other pods can acquire the lock and take over the tenant.
    • Prevents a single pod from monopolizing a tenant if it becomes unresponsive.

• Scaling with HPA

  • Pods expose active_tenants_gauge metric to Prometheus.
    • Reflects the number of tenants each pod is handling.
  • Horizontal Pod Autoscaler (HPA) uses this metric to adjust pod replicas.
    • Scales Up:
      • When active tenants per pod exceed a threshold.
    • Scales Down:
      • When the number of active tenants decreases.
  • Ensures sufficient pods are available to handle all tenants efficiently.

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
internal-search	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Nov 6, 2024 9:53pm