How to deal with operator that creates CRD / dependency / order issue

[davisford]

We are using the nats-operator in our cluster.

The example specs to set it up are here. We are actually deploying it per namespace, not as a cluster wide operator. Each namespace gets its own operator and cluster.

The nats operator is responsible for creating a CRD that defines the NatsCluster resource. The problem I have is that when flux kustomize controller runs, it fails on flux-system False validation failed: error: unable to recognize "1852fed5-e87e-44e6-a432-7608857aa525.yaml": no matches for kind "NatsCluster" in version "nats.io/v1alpha2" False

...because the operator is part of the same kustomize.yaml file and hasn't had a chance to spawn and create the CRD yet.

The directory structure I have for nats looks like this:

gitops/infrastructure/nats on 🌱 main on 🅰 va
➜ tree .
.
├── README.md
├── bootstrap
│   ├── cluster-role.yaml
│   └── kustomization.yaml
└── sources
    ├── cluster-role-binding.yaml
    ├── deployment-nats-operator.yaml
    ├── kustomization.yaml
    ├── nats-cluster.yaml
    └── service-account.yaml

The bootstrap folder contains stuff that should not be deployed per namespace as a ClusterRole is not namespaced, so we add a flux Kustomization to run that only once like so:

apiVersion: kustomize.toolkit.fluxcd.io/v1beta1
kind: Kustomization
metadata:
  name: nats
  namespace: flux-system
spec:
  interval: 10m0s
  path: ./gitops/infrastructure/nats/bootstrap
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system
  validation: client

But the stuff in the sources/ dir is meant to be overridden per namespace. The kustomization.yaml file there looks like this:

➜ cat sources/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - cluster-role-binding.yaml
  - deployment-nats-operator.yaml
  - service-account.yaml
  - nats-cluster.yaml

... and this is pulled in as a base in a namespaced overlay like so:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: foobar
resources:
  - namespace.yaml
  - pvc.yaml
bases:
  - ../../../../../infrastructure/nats/sources/

The problem here is we include the nats-cluster.yaml as part of the resources, but we need to wait for the operator to spin up first to generate that CRD or update it. So, the whole flux kustomization fails. The only way I can get past it now is the hack to comment out the nats-cluster.yaml -- let the operator come up, then comment it back in.

I know the flux Kustomization resource has dependsOn, but this only works for those resource types. This is straight up kustomize.config.k8s.io/v1beta1 Kustomization type, and I see no way to provide any kind of alternative dependency or ordering there in the main Kustomize docs.

Is there a way to re-structure this to make it work how I want?

[kingdonb]

I think I understood your issue as, there is a Custom Resource defined, and the Custom Resource Definition needs to be applied to the cluster, but the Custom Resource is invalid before the CRD is deployed, so you can't keep these resources together and apply them at the same time.

This is correct, and you're on the right track looking at DependsOn – it is a way to solve this issue.

You can use an additional Kustomization eg kustomizations.kustomize.toolkit.fluxcd.io Flux Kustomization resource to solve this (actually, two.) Since CRDs need to be applied before the resources that they define, create a Flux kustomization for the NATS CRDs. They will have to be in a separate directory so that you can use Kustomization.spec.path to select them.

You need this Kustomization to be applied first, so the second Kustomization points to the remaining resources that you mentioned, nats-cluster.yaml for example. This Kustomization should name the CRDs kustomization in its spec.dependsOn.

Since this can quickly explode into a large number of reconcilers, many more than you probably want, this usually shakes out a different way for clusters with many CRDs from different sources. The pattern I've heard most commonly: one team (the cluster admin) manages the CRDs in a single crds Kustomization for everyone on the cluster. Then a second Kustomization is for infrastructure like Flux, anything else ... istio, kyverno, anything else that is managed centrally for the whole cluster.

Finally another Flux Kustomization for all apps, (or one for each app, perhaps just whenever needed for apps that have inter-dependencies between one another.) Instead of one for each app, you might prefer to use one Kustomization for each tenant. The important thing is that the apps Kustomization should probably dependOn both crds and infrastructure.

If that sounds too heavyweight for your situation, you can also just make sure "infrastructure" depends on "crds" and pile everything into infrastructure. Usually you want at least three Kustomizations crds, infrastructure, and apps so that you can be sure any policy enforcement engines such as Kyverno with admission webhooks are in place before tenant apps that should be subject to their enforcement can be allowed to begin to synchronize.

If that also sounds like too much formalism which may be the case depending on your stage of implementation, the quick and dirty way to solve this so it just applies and be eventually consistent without stopping the presses based on the validation failure that you identified, is to set on your Kustomization spec.validation to none. Then it will apply successfully. This is not the best way to solve it, but in some strategies like when you are installing CRDs from a trusted source, it can make sense to just disable validation, rather than requiring another separate Kustomization be used.

Be careful when disabling validation though, because Flux's atomicity guarantees are only able to be achieved through this validating before applying. If you have disabled validation and some resources in the repo are invalid, then you will get partial applications and no more atomic apply guarantees. (This is of course why it works.)

[davisford]

Thanks @kingdonb -- I did hack around it before I saw your response. Essentially what I did is just grab the yaml for the crds that the nats-operator generates, and put them in as part of the spec, so flux creates them. The operator is smart of enough to see they are there and can update them, if necessary.

I did work another solution similar to your first solution above, but ran into a chicken/egg issue...whereby, my-api depends on nats, and the nats crd kustomization I made depends on flux-system kustomization which includes the app overlays, including my-api. Hard to explain. without showing, but I think it may make more sense to break out the app/overlays into an altogether separate kustomization.

Thank you for the quick reply!

[maximveksler]

I believe https://github.com/fluxcd/flux2-kustomize-helm-example is a great example for the described design pattern.

[asfernandes]

Is there any demo repository solving this problem?

[spolab]

Hi, I know that belated is a bit of an understatement, but I tried to build something here.

It kinda works. I managed to get Crossplane´s AWS Provider to be loaded correctly after Crossplane. At some point though, as I start adding resources, it goes in timeout. Not sure if this has to do with me not understanding Flux or Flux itself.

EDIT: the timeout problem is related to the nginx-ingresshelm chart. I can therefore say that it seems to work.

[sumukhballal]

We are facing an extremely similar issue but it does not get solved even after using dependsOn. We have a crds Kustomization, where we install a helm chart and this helm chart has a crds folder, which contains about 5-6 crds which helm installs into the cluster. This is marked as a success and the tests show that the crds are installed.

Another Kustomization applications, which depends on crds is then installed, but still throws up an error which says unable to find Kind of the custom crd we are deploying. I suspect it is because of some time it takes to register the crds with the kube-api or the flux operator is using an older state of the cluster without the new/updated CRD's.

The only way I presume this can be solved is using a delay in dependsOn. Can a delay be added in the spec for dependsOn incase we have a large number of crds and the next kustomization has a dependency that all the crds are present on the cluster.

[spolab]

Try to give a look at https://github.com/spolab/spolab-flux

[malarinv]

This video, while it critiques argocd and provides a solution also applies to fluxcd. Btw, he issue title is worth changing to "Eventual consistency using fluxcd" 😅 .

Argo CD Synchronization is BROKEN! It Should Switch to Eventual Consistency!