For enhanced control of the Config Connector version, replace the add-on installation with a manual one 

With Config Connector, you can manage GCP resources using Kubernetes manifests. It is a  great tool created by Google and available as an add-on installation on GKE clusters (that fit the requirements). These are just some of its advantages and use cases:

• Creating underlying infrastructure for a service deployed to GKE using the same pipelines and paradigms (it's just extra YAML manifests)
• Using a central "control" cluster to spin up other clusters, effectively achieving Infrastructure as Code using Kubernetes
• Allowing for for spinning up infrastructure (think dev environments or even a new production cluster) with a single kubectl apply

Config Connector works by deploying an Operator StatefulSet (configconnector-operator-system/configconnector-operator), which in turn creates all CustomResourceDefinition resources (CRDs) and all supporting workloads in the cnrm-system namespace.

As Config Connector is constantly evolving, new features are often added as new versions are released (a new resource type to be managed by Config Connector or even a field added to a CRD). Unfortunately, the version of Config Connector installed using the GKE add-on is tied to the GKE master version. This means that, in order to update the Config Connector version installed by the add-on, you'll need to update your GKE version (probably to the next minor version, at least), which is not always desirable. Furthermore, the bundled version is usually a few versions behind the latest available Config Connector version, which means that even then you might not get a new feature or a fix added in the latest version.

From testing, I can tell that the following versions of Config Connector are installed with the GKE add-on:

• v1.69.0 for GKE 1.21.x
• v1.71.0 for GKE 1.22.x

The latest Config Connector version at the time of writing this is v1.83.0.

Manual installation

To improve control of the Config Connector version on a GKE cluster, you can replace the add-on installation with a manual installation [1]. Note that doing so transfers responsibility for updating Config Connector to you, as it will not be updated automatically with your GKE cluster anymore.

Replacing the add-on installation with a manual one

1. disable the add-on installation on the cluster:

gcloud container clusters update <CLUSTER_NAME> --update-addons ConfigConnector=DISABLED

2. delete the Config Connector operator StatefulSet:

kubectl -nconfigconnector-operator-system delete statefulset configconnector-operator

1. 3. install the latest (or any desired version) of the Config Connector operator as per the documentation [2]

With this process, all the CRDs should remain intact, as well as the ConfigConnector object [3], so all resources created with Config Connector should remain unaffected as well.

Make sure that the new operator pod is running using the desired version:

$ kubectl -nconfigconnector-operator-system get statefulsets configconnector-operator -oyaml |grep operator-version | head -n1

    cnrm.cloud.google.com/operator-version: 1.83.0

and that it recreated all the workloads running in the cnrm-system namespace (this might take a couple of minutes to reconcile):

$ kubectl -ncnrm-system get po -oyaml |grep /version

      cnrm.cloud.google.com/version: 1.83.0
      cnrm.cloud.google.com/version: 1.83.0
      cnrm.cloud.google.com/version: 1.83.0
      cnrm.cloud.google.com/version: 1.83.0

From now on, if you want to upgrade Config Connector, you have to do this manually as described in the documentation [4].

Reverting to the add-on installation

If you upgrade your GKE cluster to a version that contains the desired Config Connector version in the add-on, you can revert to the add-on installation and offload management back to GKE.

The process basically reverses the previous instructions:

1. 1. 1. enable the Config Connector add-on on your cluster:

gcloud container clusters update <CLUSTER_NAME> --update-addons ConfigConnector=ENABLED

1. 1. 1. delete the Config Connector operator StatefulSet:

kubectl -nconfigconnector-operator-system delete statefulset configconnector-operator

If you watch pods in the configconnector-operator-system namespace, you'll eventually see that the operator pod starts up again (since the StatefulSet is recreated by the add-on) and is running the version managed by the add-on. The workloads in the cnrm-system namespace will also be replaced during the operator's first reconciliation loop.

[1] https://cloud.google.com/config-connector/docs/how-to/advanced-install#manual

[2] https://cloud.google.com/config-connector/docs/how-to/advanced-install#installing_the_operator

[3] https://cloud.google.com/config-connector/docs/how-to/install-upgrade-uninstall#addon-configuring

[4] https://cloud.google.com/config-connector/docs/how-to/advanced-install#upgrading