Support for Multiple Kyma Runtime Clusters in a single VPC

[pbochynski]

Currently, each Kyma Runtime cluster is provisioned in its own private VPC, which isolates resources but limits inter-cluster connectivity and complicates peering with shared backing services (e.g., databases, message queues). This Epic introduces the ability to create multiple Kyma Runtime clusters within a single VPC.

By enabling inter-cluster connectivity while maintaining private network access (no internet exposure) and simplifying resource peering (e.g., one-time VPC peering for shared services), this feature addresses critical customer requirements for scalability and flexibility.

Goals

1. Allow multiple Kyma Runtime clusters to be created within a single VPC.
2. Enable inter-cluster connectivity while preserving isolation at the subnet level.
3. Facilitate easier peering with shared backing services by requiring only one VPC connection.
4. Ensure backward compatibility with the existing setup where each cluster has its own VPC.

Requirements

• Each cluster must have its own subnet within the shared VPC.
• Subnet configuration must prevent IP address conflicts across clusters.
• Utilize Gardener’s support for creating clusters within a specified VPC.
• Introduce a new service instance parameter (VPC Name) in the Kyma Runtime provisioning flow:
  • If provided, the cluster joins the existing VPC.
  • If not provided, a new VPC is created for the cluster.
  • Consider other parameters that will be required to create VPC
• Make sure that orphan resources (VPCs) are deleted when not used by any of the cluster

Acceptance Criteria

• A Kyma Runtime cluster can be successfully created in an existing VPC.
• A new VPC is created when the VPC Name parameter is omitted.
• Clusters within the same VPC can communicate with each other.
• Multiple clusters in the same VPC can access shared backing services using a single VPC peering connection.
• Existing clusters in their own VPCs remain unaffected by this feature.

Out of Scope

• Modifying existing clusters to transition into a shared VPC setup.
• Implementing advanced inter-cluster service discovery mechanisms.

[marsteg]

Could we add to the requirements to choose the node network subnets to use? or even the option to provide a broad range and kyma slicing the big range into small ranges for each of the clusters?

[maxmoehl]

Besides the already mentioned points we would like to be able to configure the size of the VPC as well as the clusters. The initial idea was to use /16 for the VPC, /22 for the clusters (allowing for 64 clusters) and each cluster is split into two /23, one for hyperscaler stuff like NAT-GW, LBs, ... and the other for the nodes. This should allow for up to 512 nodes per cluster which leaves some head-room to the current recommendation from gardener to not go beyond 300 nodes per cluster.

Not sure how fine-granular you want to expose these things as configuration options but as these are mostly based on assumptions the exact numbers might change.

[maxmoehl]

An additional requirement is that we can peer two (or more) of these VPCs together to extend a single installation beyond 64 clusters.

[frnksgr]

Kyma clusters don't benefit from this feature. This feature is required to support Gardener in the context of Korifi and depends on certain constraints that cannot be made in Kyma but in Korifi.

The main usage sharing backing services between Korifi and CF Classic by providing private L3 connectivity to our existing multi tenant backing service networks via peering. Since the backing service networks do not implement any tenant isolation, tenant isolation is expected to be done by packet egress filtering in the client, i.e. the k8s clusters. This requires a separation of admin and dev roles. A developer must not be allowed to overwrite network policies that implement egress filtering. Kyma violates this constraint. Although there might be a more complex way to implement ingress filtering on the backing services side this is difficult as long as PODs/Containers (of different tenants, like in Korifi) share IP addresses because of SNAT.

I don't see a need for private cluster communication in our current context. The reason we want to have multiple clusters within the same VPC is to simplify peering and IPAM management. The clusters are still set up without external routable POD addresses (although Gardener does support this, but it would only make sense when using IPv6). Communication between clusters can be done privately within that VPC but the clusters still require edge gateways for egress and ingress.

The main role for Kyma in that scenario might be the reuse of existing Kyma functionality. The feature needs:

1. a system that creates the VPCs and other network entities across 4+1
   1.1. the VPC must be compliant with the Gardener constraints on the specific hyper-scaler to let Gardener manage shoots within that VPC
   1.2. the VPCs must be compliant with the IP Layout of the existing backing service networks to allow peering
2. a machinery that can modify the shoots of clusters and does IPAM on the shoot node network config.

I would expect that Kyma already has a machinery that allows for shoot manipulation that can be used or extended for 2.
AFAIK there is not yet a machinery in Kyma that is does create the VPCs (according to 1.1), but I might be wrong.

Therefore goals should not focus on Kyma as the supported target but Gardener.

1. Allow multiple Gardener clusters to be created within a single VPC.
2. Enable inter-cluster connectivity while preserving isolation at the subnet level.
3. The VPC network need to be compliant with the IP layout of the backing services network.
4. Ensure backward compatibility with the existing setup where each cluster has its own VPC.
5. ...

Unfortunately we cannot use that feature for Kyma. We expect existing Kyma functionality to help implementing that feature.
We require that feature to support sharing of backing services between clusters and CF Classic runtime in the context of Korifi.

NOTE: We would need bigger investments into BTP networking across different groups to help sharing (non public) backing services across all run-times, i.e. Kyma and CF (Classic and Korifi). A project like Super-6 could solve that issue.