diff --git a/.chloggen/move-operator-crd-install-method.yaml b/.chloggen/move-operator-crd-install-method.yaml index 5a69759ed5..26d18e666e 100644 --- a/.chloggen/move-operator-crd-install-method.yaml +++ b/.chloggen/move-operator-crd-install-method.yaml @@ -1,12 +1,16 @@ # One of 'breaking', 'deprecation', 'new_component', 'enhancement', 'bug_fix' -change_type: bug_fix +change_type: breaking # The name of the component, or a single word describing the area of concern, (e.g. agent, clusterReceiver, gateway, operator, chart, other) component: operator # A brief description of the change. Surround your text with quotes ("") if it needs to start with a backtick (`). note: Move operator CRD installation to the crds/ folder via a subchart to resolve Helm install ordering issues # One or more tracking issues related to the change -issues: [1561] +issues: [1561,1619] # (Optional) One or more lines of additional information to render under the primary note. # These lines will be padded with 2 spaces and then inserted directly into the document. # Use pipe (|) for multiline entries. -subtext: +subtext: | + - Users enabling the operator (`.Values.operator.enabled=true`) must now set `operatorcrds.install=true` in Helm values or [manually manage CRD installation](https://github.com/signalfx/splunk-otel-collector-chart/blob/main/docs/auto-instrumentation-install.md#crd-management). + - Previously, CRDs were installed using templates (`operator.crds.create=true`), which could cause race conditions and installation failures. + - CRD installation is now handled via Helm's native `crds/` directory for better stability, using a [localized subchart](https://github.com/signalfx/splunk-otel-collector-chart/tree/main/helm-charts/splunk-otel-collector/charts/opentelemetry-operator-crds). + - If you use `operator.enabled=true` you may have to follow some migration steps, please see the [Upgrade guidelines](https://github.com/signalfx/splunk-otel-collector-chart/blob/main/UPGRADING.md#0113-to-0116). diff --git a/UPGRADING.md b/UPGRADING.md index 41f83040dd..248da0c3f6 100644 --- a/UPGRADING.md +++ b/UPGRADING.md @@ -1,5 +1,86 @@ # Upgrade guidelines +## 0.113.0 to 0.116.0 + +This guide provides steps for new users, transitioning users, and those maintaining previous operator CRD configurations: +- New users: No migration for CRDs is required. +- Previous users: Migration may be needed if using `operator.enabled=true`. + +CRD deployment has evolved over chart versions: +- Before 0.110.0: CRDs were deployed via a crds/ directory (upstream default). +- 0.110.0 to 1.113.0: CRDs were deployed using Helm templates (upstream default), which had reported issues. +- 0.116.0 and later: Users must now explicitly configure their preferred CRD deployment method or deploy the + [CRDs manually](https://github.com/signalfx/splunk-otel-collector-chart/blob/main/docs/auto-instrumentation-install.md#crd-management) + to avoid potential issues. Users can deploy CRDs via a crds/ directory again by enabling a newly added value. + +### **New Users** + +New users are advised to deploy CRDs via the `crds/` directory. For a fresh installation, use the following Helm values: + +```yaml +operatorcrds: + install: true +operator: + enabled: true +``` + +To install the chart: + +```bash +helm install splunk-otel-collector-chart/splunk-otel-collector --set operatorcrds.install=true,operator.enabled=true +``` + +### **Current Users (Recommended Migration to `crds/` Directory)** + +If you're using chart versions 0.110.0 to 1.113.0, CRDs are likely deployed via Helm templates. To migrate to the recommended `crds/` directory deployment: + +#### Step 1: Delete the Existing Chart + +Remove the chart to prepare for a fresh installation: + +```bash +helm delete +``` + +#### Step 2: Verify or Remove Existing CRDs + +Check if the following CRDs are present and delete them if necessary: + +```bash +kubectl get crds | grep opentelemetry +``` + +```bash +kubectl delete crd opentelemetrycollectors.opentelemetry.io +kubectl delete crd opampbridges.opentelemetry.io +kubectl delete crd instrumentations.opentelemetry.io +``` + +#### Step 3: Reinstall with Recommended Values + +Reinstall the chart with the updated configuration: +```bash +helm install splunk-otel-collector --set operatorcrds.install=true,operator.enabled=true +``` + +### **Previous Users (Maintaining Legacy Helm Templates)** + +If you're using chart versions 0.110.0 to 1.113.0 and prefer to continue deploying CRDs via Helm templates (not recommended), you can do so with the following values: + +```yaml +operator: + enabled: true +operator: + crds: + create: true +``` + +**Warning**: This method may cause race conditions during installation or upgrades, leading to errors like: +```plaintext +ERROR: INSTALLATION FAILED: failed post-install: warning: Hook post-install splunk-otel-collector/templates/operator/instrumentation.yaml failed: 1 error occurred: +* Internal error occurred: failed calling webhook "minstrumentation.kb.io": failed to call webhook: Post "https://splunk-otel-collector-operator-webhook.default.svc:443/mutate-opentelemetry-io-v1alpha1-instrumentation?timeout=10s": dial tcp X.X.X.X:443: connect: connection refused +``` + ## 0.105.5 to 0.108.0 We've simplified the Helm chart configuration for `operator` auto-instrumentation. diff --git a/docs/auto-instrumentation-install.md b/docs/auto-instrumentation-install.md index 7a07132e4e..5248eda8e9 100644 --- a/docs/auto-instrumentation-install.md +++ b/docs/auto-instrumentation-install.md @@ -26,6 +26,8 @@ these frameworks often have pre-built instrumentation capabilities already avail ### 1. Deploy the Helm Chart with the Operator enabled - **Operator Deployment (Required)** + - `operatorcrds.install`: Set to `true` to install the CRDs required by the operator. + - **Required**: Must be set unless CRDs are pre-installed manually. - `operator.enabled`: Set to `true` to enable deploying the operator. - **Required**: This configuration is necessary for the operator's deployment within your cluster. @@ -70,7 +72,7 @@ these frameworks often have pre-built instrumentation capabilities already avail kubectl get pods -l app=cert-manager --all-namespaces # If cert-manager is not deployed, make sure to add certmanager.enabled=true to the list of values to set -helm install splunk-otel-collector -f ./my_values.yaml --set operator.enabled=true,environment=dev splunk-otel-collector-chart/splunk-otel-collector +helm install splunk-otel-collector -f ./my_values.yaml --set operatorcrds.install=true,operator.enabled=true,environment=dev splunk-otel-collector-chart/splunk-otel-collector ``` ### 2. Verify all the OpenTelemetry resources (collector, operator, webhook, instrumentation) are deployed successfully @@ -284,6 +286,11 @@ in a Kubernetes environment. An operator is a method of packaging, deploying, an In the context of setting up observability in a Kubernetes environment, an operator simplifies the management of application auto-instrumentation, making it easier to gain valuable insights into application performance. +The OpenTelemetry operator relies on +[Custom Resource Definitions (CRDs)](https://github.com/signalfx/splunk-otel-collector-chart/tree/main/helm-charts/splunk-otel-collector/charts/opentelemetry-operator-crds) +to manage auto-instrumentation configurations in Kubernetes. +Ensure the required CRDs are deployed before the operator by configuring `operatorcrds.install=true`. + With this Splunk OTel Collector chart, the [OpenTelemetry Operator](https://github.com/open-telemetry/opentelemetry-operator#opentelemetry-auto-instrumentation-injection) can be deployed (by configuring `operator.enabled=true`) to your cluster and start auto-instrumenting your applications. @@ -403,6 +410,58 @@ provides best effort support with issues related to native OpenTelemetry instrum | apache-httpd | OpenTelemetry | Available | Needs Validation | | [Link](https://github.com/open-telemetry/opentelemetry-apache-httpd-instrumentation) | ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-apache-httpd | | nginx | OpenTelemetry | Available | Needs Validation | | [Link](https://github.com/open-telemetry/opentelemetry-apache-httpd-instrumentation) | ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-apache-httpd | +### CRD Management + +When deploying the operator, the required Custom Resource Definitions (CRDs) must be deployed beforehand. + +#### Recommended Approach: Automated CRD Deployment + +Set the Helm chart value `operatorcrds.install=true` to allow the chart to handle CRD installation automatically. +_This option deploys the CRDs using a local subchart, available at [opentelemetry-operator-crds](https://github.com/signalfx/splunk-otel-collector-chart/tree/main/helm-charts/splunk-otel-collector/charts/opentelemetry-operator-crds)._ +_Please note, helm will not update or delete these CRDs after initial install as noted in their [documentation](https://helm.sh/docs/chart_best_practices/custom_resource_definitions/#some-caveats-and-explanations)._ + +#### Alternative Approach: Manual CRD Deployment + +If you prefer to manage CRD deployment manually, apply the CRDs using the commands below before installing the Helm chart: + +```bash +curl -sL https://raw.githubusercontent.com/splunk-otel-collector/charts/opentelemetry-operator-crds/crds/opentelemetry.io_opentelemetrycollectors.yaml | kubectl apply -f - +curl -sL https://raw.githubusercontent.com/splunk-otel-collector/charts/opentelemetry-operator-crds/crds/opentelemetry.io_opampbridges.yaml | kubectl apply -f - +curl -sL https://raw.githubusercontent.com/splunk-otel-collector/charts/opentelemetry-operator-crds/crds/opentelemetry.io_instrumentations.yaml | kubectl apply -f - +``` + +You can also use below helm template command to get the CRD yamls from the helm chart. This method can be helpful in keeping CRDs in-sync with the version bundled with our helm chart. + +```bash +helm template splunk-otel-collector-chart/splunk-otel-collector --include-crds \ +--set="splunkObservability.realm=us0,splunkObservability.accessToken=xxxxxx,clusterName=my-cluster,operatorcrds.install=true" \ +| yq e '. | select(.kind == "CustomResourceDefinition")' \ +| kubectl apply -f - +``` + +#### CRD Updates + +With Helm v3.0 and later, CRDs created by this chart are not updated automatically. To update CRDs, you must apply the updated CRD definitions manually. +Refer to the [Helm Documentation on CRDs](https://helm.sh/docs/chart_best_practices/custom_resource_definitions/) for more details. + +#### CRD Cleanup + +When uninstalling this chart, the OpenTelemetry CRDs are not removed automatically. To delete them manually, use the following commands: + +```bash +kubectl delete crd opentelemetrycollectors.opentelemetry.io +kubectl delete crd opampbridges.opentelemetry.io +kubectl delete crd instrumentations.opentelemetry.io +``` +You can use below combination of helm and kubectl command to delete CRDs. + +```bash +helm template splunk-otel-collector-chart/splunk-otel-collector --include-crds \ +--set="splunkObservability.realm=us0,splunkObservability.accessToken=xxxxxx,clusterName=my-cluster,operatorcrds.install=true" \ +| yq e '. | select(.kind == "CustomResourceDefinition")' \ +| kubectl delete --dry-run=client -f - +``` + ### Documentation Resources - https://developers.redhat.com/devnation/tech-talks/using-opentelemetry-on-kubernetes diff --git a/examples/enable-operator-and-auto-instrumentation/enable-operator-and-auto-instrumentation-values.yaml b/examples/enable-operator-and-auto-instrumentation/enable-operator-and-auto-instrumentation-values.yaml index bad14c1eb6..def8c7d41c 100644 --- a/examples/enable-operator-and-auto-instrumentation/enable-operator-and-auto-instrumentation-values.yaml +++ b/examples/enable-operator-and-auto-instrumentation/enable-operator-and-auto-instrumentation-values.yaml @@ -7,6 +7,8 @@ splunkObservability: clusterName: CHANGEME environment: CHANGEME +operatorcrds: + install: true operator: enabled: true certmanager: diff --git a/examples/enable-operator-and-auto-instrumentation/otel-demo-dotnet.md b/examples/enable-operator-and-auto-instrumentation/otel-demo-dotnet.md index 3eb6a695b9..d28b6b2a41 100644 --- a/examples/enable-operator-and-auto-instrumentation/otel-demo-dotnet.md +++ b/examples/enable-operator-and-auto-instrumentation/otel-demo-dotnet.md @@ -24,7 +24,7 @@ curl https://raw.githubusercontent.com/signalfx/splunk-otel-collector-chart/main #### 2.1 Deploy the Helm Chart with the Operator enabled To install the chart with operator in an existing cluster, make sure you have cert-manager installed and available. -Both the cert-manager and operator are subcharts of this chart and can be enabled with `--set certmanager.enabled=true,operator.enabled=true`. +Both the cert-manager and operator are subcharts of this chart and can be enabled with `--set operatorcrds.install=true,operator.enabled=true,certmanager.enabled=true`. These helm install commands will deploy the chart to the current namespace for this example. ```bash @@ -32,7 +32,7 @@ These helm install commands will deploy the chart to the current namespace for t kubectl get pods -l app=cert-manager --all-namespaces # If cert-manager is deployed, make sure to remove certmanager.enabled=true to the list of values to set -helm install splunk-otel-collector -f ./my_values.yaml --set operator.enabled=true,certmanager.enabled=true,environment=dev splunk-otel-collector-chart/splunk-otel-collector +helm install splunk-otel-collector -f ./my_values.yaml --set operatorcrds.install=true,operator.enabled=true,certmanager.enabled=true,environment=dev splunk-otel-collector-chart/splunk-otel-collector ``` #### 2.2 Verify all the OpenTelemetry resources (collector, operator, webhook, instrumentation) are deployed successfully diff --git a/examples/enable-operator-and-auto-instrumentation/otel-demo-nodejs.md b/examples/enable-operator-and-auto-instrumentation/otel-demo-nodejs.md index 3185e554c2..dfd9a08fa4 100644 --- a/examples/enable-operator-and-auto-instrumentation/otel-demo-nodejs.md +++ b/examples/enable-operator-and-auto-instrumentation/otel-demo-nodejs.md @@ -20,7 +20,7 @@ curl https://raw.githubusercontent.com/signalfx/splunk-otel-collector-chart/main #### 2.1 Deploy the Helm Chart with the Operator enabled To install the chart with operator in an existing cluster, make sure you have cert-manager installed and available. -Both the cert-manager and operator are subcharts of this chart and can be enabled with `--set certmanager.enabled=true,operator.enabled=true`. +Both the cert-manager and operator are subcharts of this chart and can be enabled with `--set operatorcrds.install=true,operator.enabled=true,certmanager.enabled=true`. These helm install commands will deploy the chart to the current namespace for this example. ```bash @@ -28,7 +28,7 @@ These helm install commands will deploy the chart to the current namespace for t kubectl get pods -l app=cert-manager --all-namespaces # If cert-manager is deployed, make sure to remove certmanager.enabled=true to the list of values to set -helm install splunk-otel-collector -f ./my_values.yaml --set operator.enabled=true,certmanager.enabled=true,environment=dev splunk-otel-collector-chart/splunk-otel-collector +helm install splunk-otel-collector -f ./my_values.yaml --set operatorcrds.install=true,operator.enabled=true,certmanager.enabled=true,environment=dev splunk-otel-collector-chart/splunk-otel-collector ``` #### 2.2 Verify all the OpenTelemetry resources (collector, operator, webhook, instrumentation) are deployed successfully diff --git a/examples/enable-operator-and-auto-instrumentation/spring-petclinic-java.md b/examples/enable-operator-and-auto-instrumentation/spring-petclinic-java.md index d918e6be6b..fb5d93b640 100644 --- a/examples/enable-operator-and-auto-instrumentation/spring-petclinic-java.md +++ b/examples/enable-operator-and-auto-instrumentation/spring-petclinic-java.md @@ -20,7 +20,7 @@ curl https://raw.githubusercontent.com/signalfx/splunk-otel-collector-chart/main #### 2.1 Deploy the Helm Chart with the Operator enabled To install the chart with operator in an existing cluster, make sure you have cert-manager installed and available. -Both the cert-manager and operator are subcharts of this chart and can be enabled with `--set certmanager.enabled=true,operator.enabled=true`. +Both the cert-manager and operator are subcharts of this chart and can be enabled with `--set operatorcrds.install=true,operator.enabled=true,certmanager.enabled=true`. These helm install commands will deploy the chart to the current namespace for this example. ```bash @@ -28,7 +28,7 @@ These helm install commands will deploy the chart to the current namespace for t kubectl get pods -l app=cert-manager --all-namespaces # If cert-manager is deployed, make sure to remove certmanager.enabled=true to the list of values to set -helm install splunk-otel-collector -f ./my_values.yaml --set operator.enabled=true,certmanager.enabled=true,environment=dev splunk-otel-collector-chart/splunk-otel-collector +helm install splunk-otel-collector -f ./my_values.yaml --set operatorcrds.install=true,operator.enabled=true,certmanager.enabled=true,environment=dev splunk-otel-collector-chart/splunk-otel-collector ``` #### 2.2 Verify all the OpenTelemetry resources (collector, operator, webhook, instrumentation) are deployed successfully diff --git a/helm-charts/splunk-otel-collector/Chart.yaml b/helm-charts/splunk-otel-collector/Chart.yaml index 9a9bd97dd3..93a9d37107 100644 --- a/helm-charts/splunk-otel-collector/Chart.yaml +++ b/helm-charts/splunk-otel-collector/Chart.yaml @@ -28,7 +28,7 @@ dependencies: repository: https://charts.jetstack.io condition: certmanager.enabled - name: opentelemetry-operator-crds - version: "0.0.1" + version: 0.0.1 alias: operatorcrds condition: operatorcrds.install - name: opentelemetry-operator diff --git a/helm-charts/splunk-otel-collector/values.schema.json b/helm-charts/splunk-otel-collector/values.schema.json index 9d4c549b86..399fbe6660 100644 --- a/helm-charts/splunk-otel-collector/values.schema.json +++ b/helm-charts/splunk-otel-collector/values.schema.json @@ -1424,6 +1424,24 @@ "type": "boolean", "deprecated": true }, + "operatorcrds": { + "description": "Deploys Operator related CRDs from https://github.com/open-telemetry/opentelemetry-operator/tree/main/config/crd via a Helm subchart and using the chart crds/ directory.", + "type": "object", + "properties": { + "install": { + "type": "boolean" + } + } + }, + "operator-crds": { + "description": "Configuration for operator-crds (legacy). Prefer using 'operatorcrds' for new installations. Deploys Operator related CRDs from https://github.com/open-telemetry/opentelemetry-operator/tree/main/config/crd via a Helm subchart and using the chart crds/ directory.", + "type": "object", + "properties": { + "install": { + "type": "boolean" + } + } + }, "operator": { "description": "OpenTelemetry Operator configuration. A subchart that is used to install the operator, see https://github.com/open-telemetry/opentelemetry-helm-charts/blob/main/charts/opentelemetry-operator/values.schema.json for more info.", "type": "object", @@ -1684,24 +1702,6 @@ } } }, - "operatorcrds": { - "description": "Deploys Operator related CRDs from https://github.com/open-telemetry/opentelemetry-operator/tree/main/config/crd via a Helm subchart and using the chart crds/ directory.", - "type": "object", - "properties": { - "install": { - "type": "boolean" - } - } - }, - "operator-crds": { - "description": "Configuration for operator-crds (legacy). Prefer using 'operatorcrds' for new installations. Deploys Operator related CRDs from https://github.com/open-telemetry/opentelemetry-operator/tree/main/config/crd via a Helm subchart and using the chart crds/ directory.", - "type": "object", - "properties": { - "install": { - "type": "boolean" - } - } - }, "certmanager": { "description": "cert-manager adds certificates and certificate issuers as resource types in Kubernetes clusters, and simplifies the process of obtaining, renewing and using those certificates.", "type": "object", diff --git a/helm-charts/splunk-otel-collector/values.yaml b/helm-charts/splunk-otel-collector/values.yaml index 97d3d98584..35e0060f10 100644 --- a/helm-charts/splunk-otel-collector/values.yaml +++ b/helm-charts/splunk-otel-collector/values.yaml @@ -1174,9 +1174,10 @@ service: # Full list of Helm value configurations: https://artifacthub.io/packages/helm/opentelemetry-helm/opentelemetry-operator?modal=values ################################################################################ -# Should the CRDs be installed by a chart's crd/ directory or by using templates. +# Specify whether the chart should install CRDs automatically. +# Related Documentation: https://github.com/signalfx/splunk-otel-collector-chart/blob/main/docs/auto-instrumentation-install.md#crd-management operatorcrds: - # Should the CRDs be installed + # Set to true to install CRDs automatically, or false to manage them manually. install: false operator: