Extend Kyma Provisioning Parameters to allow multiple Worker Node Pools definitions

[PK85]

See:

kyma-project/kyma#18709 (comment)

Plan:

1. We keep our main worker pool nodes as they are. It will be for Kyma workloads
2. We need to support additionalWorkerNodes for customers (we decided that they are by default min 3 and three zones)

AC

• 0) extend schema with _load_current_config parameter
  Enable current configuration load on Preview #1654
• 1) design KEB schema API, validations, defaults. Present to the team.
• 2) use our preview plan where new schema is visible, and we update RuntimeCR
  Additional worker node pools schema on Preview #1659, Introduce HA zones #1668,
  Disable changing HA zones setting #1671
• 3) Check how frogs wants to implement that in KIM, see: Multiple worker groups [KIM/feature] infrastructure-manager#46 (comment)
• 4) new e2e SKR tests
• 5) docs + RN Additional worker node pools docs #1617, Update additional worker node pools docs #1678
• 6) deploy
  • 6.1) dev
  • 6.2) stage
  • 6.3) prod

[KsaweryZietara]

23.12.2024

1. We've decided to extend machine type list with new types (for example GPUs) in next task.
2. We've decided to extend the schema with _load_current_config: true parameter, which loads current instance configuration into the update instance schema. In the next step we will test additionalWorkerNodePools on preview plan. If everything works, we will release additionalWorkerNodePools on every plan.

[KsaweryZietara]

Additional Worker Node Pools will be available on:

• azure, aws, gcp, sap-converged-cloud, preview:
  • autoScalerMin: 3 (default - 3)
  • autoScalerMax: 300 (default - 20)
• azure_lite:
  • autoScalerMin: 2 (default - 2)
  • autoScalerMax: 40 (default - 10)