This blog post is part of a series about building a production ready Continuous Integration setup for Sitecore using K8s containers on AKS. For more information and other articles on this subject check out the series index.

Secrets management

In general it is a bad idea to store credentials or secrets in your code repository. But when adopting the Infrastructure-as-Code principle, your infrastructure definition files do reside in your repository. When deploying to PaaS Web Apps, having ARM as our infra definition, it was quite easy to use Azure Key Vault references in your ARM templates. When deploying from the K8s specification files, things are a little bit different.

As you may have already seen, Sitecore Kubernetes deployments use Secrets to store sensitive information:

This of course is great to keep (database) user names, passwords and certificates out of your container images, but still, when provisioning an AKS cluster out of your code repository following the IaC principle, all secrets would be in your repository in plain text.

Hint: when putting your own custom values into these secrets files, make sure you do not insert a newline character at the end! This will potentially break your deployment (at least the database login will not work when having a username ending with a newline character).

Switching to Token Replacement

So how do I provide these secrets using Azure Key Vault? Actually there’s a neat little trick for this. You can replace the secrets with a variable, a reference token, and replace them with the actual value at release time. Like so:

There’s an Azure DevOps Task you can use in your Release pipeline called “Replace Tokens”. So now, we’re one step closer, because you can now use Azure DevOps Pipeline variables or variables out of a Variable group in your Pipeline Library. I am not satisfied yet though, because I want to store my secrets and certificates into an Azure Key Vault (AKV) instance to profit from its proper key management and audit trail logging. Well, that turns out to be very easy to accomplish: just add an extra Task in your Release pipeline called “Azure Key Vault”. This will retrieve all (or a filtered list of) secrets from your Key Vault instance and make them available for the tasks that follow:

After which you can perform the actual token replacement. Azure DevOps will now be able to read the secrets from the AKV and inject them into the secret files at release time, before deploying the secrets to your AKS cluster:

The YAML of this specific job would look somewhat like this:

Azure Key Vault provisioning

So the only thing left to do is to edit all the secrets text files, place a reference token in it, and generate the accompanying Key Vault secrets. Doing this manually for over 40 secrets is quite a tedious job though, and it conflicts with our IaC principle. So..

Let’s script all the things!

I developed a handy PowerShell script that does all the hard work for you. Because I want this script to be flexible and forward compatible, I didn’t want to hard-code all of the secrets. And I didn’t have to, because the secrets directory in your K8s specification files contains a kustomization.yaml file that holds an index of all the secrets. My script uses this file as the input for editing all files and generate the AKV secrets.

Basically, there are two options for the state of a secrets file: a secrets file is still empty, or a (custom) value has already been pre-filled. My script handles both situations in different ways: if a value is already pre-filled, it lifts this value and moves it over to an AKV secret, replacing it by the corresponding token; if the file is still empty, it will generate a random secret value of which the default length is configurable. It also respects the recommended increased length of a few keys as indicated by the Sitecore Installation Guide:

If you run the certificate scripts for the Identity Server token signing certificate script and the TLS certificate generation script for the Ingress controller first, it will also neatly lift those over to your AKV instance. The license file is skipped by default due to its size and the fact that it is usually tolerated being stored in the repository, as is the license.xml file itself.

If you want to use this handy script yourself, head over to my GitHub repository where I’m building my own boilerplate repository for Sitecore AKS deployments:

https://github.com/robhabraken/paas-to-aks

Other PowerShell scripts

Sitecore provides some example scripts in the Installation Guide to compress your license file and to generate self-signed certificates for the Identity Server and the Ingress controller. I extended them so they automatically write the output to the correct location in your repository, and added some parameters for ease of use. You can now specify for which topology you would like to run the script so it outputs to the corresponding directory within your repository:

These scripts are located in the ./scripts folder. You can find the Populate KeyVault script in the ./azure/scripts folder, along with the AKS deployment and NGINX Ingress controller deployment scripts.