On 17 march 2025, all our ArgoCD application that make use of a Azure Key Vault Secret, were suddenly Out of Sync.

When trying to sync e.g. keycloak-development, we got:

Failed sync attempt to 1def86278e2f7171c77625bbfc2964fac809aeb6: one or more objects failed to apply, reason: azurekeyvaultsecrets.spv.no "keycloak-admin-password" is invalid: 
metadata.resourceVersion: Invalid value: 0x0: must be specified for an update,azurekeyvaultsecrets.spv.no "keycloak-credentials" is invalid: 
metadata.resourceVersion: Invalid value: 0x0: must be specified for an update

For all the services in the picture, we get the same sync error message. As example, for wiki-development:

Failed sync attempt to 1def86278e2f7171c77625bbfc2964fac809aeb6: 
one or more objects failed to apply, reason: 
azurekeyvaultsecrets.spv.no "wiki-credentials" is invalid: metadata.resourceVersion: 
Invalid value: 0x0: must be specified for an update (retried 5 times).

Problem Analysis:

In the error message by ArgoCD, we see similar results. It focuses on:

→ resourceVersion data being invalid → that the resourceVersion must be specified for an update.

Now, let’s check for Keycloak-development application, the difference in state that ArgoCD within in the Kubernetes cluster and in our code repository:

The left side shows what is currently deployed in Kubernetes.

The right side is the desired state, so what ArgoCD sees in the code.

It can be seen, that in the live manifest there are additional fields assigned like resourceVersion, uid, lastAzureUpdate, secretHash, secretName. These are typically added by Kubernetes as metadata - so, that shouldn’t be an issue.

A solution provided by ChatGPT for our problem, is the following:

<aside> 💡

1. Additional fields (resourceVersion, uid, etc.) are dynamically added by Kubernetes and the Azure Key Vault Operator after deployment.

2. These fields are NOT in the desired state (right side) because they are runtime metadata, not part of the GitOps-managed configuration.

→ Fixing Drift in ArgoCD:

If ArgoCD keeps trying to revert these fields, add this rule to ignore them in ArgoCD:

spec:
  ignoreDifferences:
    - group: "spv.no"
      kind: "AzureKeyVaultSecret"
      jsonPointers:
        - "/metadata/resourceVersion"
        - "/metadata/uid"
        - "/status/lastAzureUpdate"
        - "/status/secretHash"
        - "/status/secretName"

This allows ArgoCD to focus only on the actual secret data, not runtime metadata that changes dynamically.

</aside>

However, we never faced this issue beforehand with ArgoCD, so … what is causing it? Is there maybe something else happening underneath that we are not able to see?

“WHY DO WE LIVE?”

Another possibility, that we must check, before we proceed with the possible solution provided to us by ChatGPT - is to check the akv2k8s controller pod.

I asked ChatGPT for a quick update about the function of akv2k8s:

<aside> 💡

🔹 What is AKV2K8S?

AKV2K8S (Azure Key Vault to Kubernetes) is a Kubernetes operator that automatically syncs secrets from Azure Key Vault into Kubernetes as native secrets.

🔹 How It Works

1. You define an AzureKeyVaultSecret resource in Kubernetes.
2. AKV2K8S pulls the secret from Azure Key Vault.
3. It injects the secret into a Kubernetes Secret or Environment Variable.

🔹 Why Use It?

✅ Automatic secret syncing (no manual updates needed).

✅ Secure – No secrets stored in Git.

✅ Integrates with Azure Managed Identity (no extra credentials).

🔹 AKV2K8S via Helm Chart

Since you use the Helm chart, it deploys:

• CRDs (AzureKeyVaultSecret) to define Key Vault secrets.
• A controller that syncs secrets periodically.

🚀 AKV2K8S ensures your Kubernetes apps always get the latest Azure Key Vault secrets securely! 🔥

</aside>

<aside> 💡

📌 Example CRD (AzureKeyVaultSecret)

---
apiVersion: spv.no/v2beta1
kind: AzureKeyVaultSecret
metadata:
  name: keycloak-admin-password
spec:
  vault:
    name: sensey-dev-kv
    object:
      contentType: application/x-json
      name: keycloak-admin-password
      type: multi-key-value-secret
  output:
    secret:
      name: keycloak-admin-password

🔍 Explanation of This Example

• ✅ apiVersion: spv.no/v2beta1 → Uses the newer AKV2K8S API version.
• ✅ vault.name: sensey-dev-kv → The Azure Key Vault that contains the secret.
• ✅ object.name: keycloak-admin-password → The actual secret name inside Azure Key Vault.
• ✅ output.secret.name: keycloak-admin-password → The Kubernetes secret where it will be stored. </aside>