← Back to overview

OIDC Authentication in RKE2 Kubernetes with Keycloak

In order to avoid certificate- and token-based authentication in your Kubernetes cluster, you can use an OpenID Connect (OIDC) identity provider to authenticate users. When running kubectl, we want to automatically open a browser window to sign in and then retrieve a short-lived token to authenticate our further requests against the Kubernetes API Server.

This guide will show you how to use the open-source OIDC-compatible identity provider Keycloak to achieve exactly that. I run all Kubernetes clusters with RKE2, so this guide will focus on RKE2, however the majority of steps are absolutely identical for other ways of deploying Kubernetes.

Keycloak

I assume that you already have a Keycloak instance up and running somewhere. This instance needs to be reachable from your Kubernetes cluster as well as from the clients you want to authenticate. Usually you can operate Keycloak publicly, as long as you adhere to some basic production security best practices such as running it behind a reverse proxy and restricting access to the admin console (which you need to enter for the next step).

Client

First of all, you need to configure a so-called Client in Keycloak. This might sound like creating an end-user, but a Client in Keycloak is simply some kind of application that wants to authenticate users against Keycloak. In our case, this is a Kubernetes cluster.

  1. Open your Keycloak admin console, select the Clients tab on the left and click on Create.
  2. Choose OpenID Connect as client type.
  3. Enter a Client ID of your choice. This is the identifier of the Kubernetes cluster in Keycloak. If you’re authenticating multiple clusters against the same Keycloak instance, you should prefix the client IDs and use a unique name for each cluster (e.g. rke2-cluster-1, rke2-cluster-2 etc.).
  4. On the second page (Capability config), enable Client authentication and make sure that the Standard flow as well as Direct access grants are activated (which should be the case by default).
  5. On the third page (Login settings), we need to configure the Valid redirect URIs. This is the address where the end-user is allowed to be redirected to after a successful authentication, depending on what was requested during the authentication process. You can enter http://* as wildcard to allow all addresses, as the authentication is happening in the end-user’s browser locally anyway.
  6. No other settings need to be changed, so you can complete the form and click on Save.

Groups

Later, when authenticated users are making requests to the Kubernetes API Server, you somehow assign certain RBAC permissions to them. I’d suggest using groups to manage these permissions, instead of assigning permissions to users directly. You can create as many groups as you want and need. Simply give them a meaningful name.

  1. Open your Keycloak admin console, select the Groups tab on the left and click on Create.
  2. Enter a Name for the group and click on Create.

Client Scope

We now need to create a so-called Client Scope for the group membership information. This is important so that Keycloak adds a list of groups, the authenticated user is a member of, to the token payload it issues. This is then used by the Kubernetes API Server to grant the user the corresponding RBAC permissions. Without that client scope, Kubernetes knows who the user is, but not which groups they belong to and subsequently not which group permissions apply. If you decided to assign permissions to users instead of groups, you can skip this step.

  1. Open your Keycloak admin console, select the Client scopes tab on the left and click on Create client scope.
  2. Use groups as Name and hit Save.
  3. After the client scope has been created, two more tabs will appear: Mappers and Scope.
  4. Open the Mappers tab and click on Add mapper (by configuration).
  5. Search for Group Membership and select it.
  6. Again, set groups as Name and Token Claim Name.
  7. Make sure that you disable the Full group path option and click Save.

In order to glue the client scope to our client, we need to assign it.

  1. Go back to the Clients tab on the left and select your Kubernetes cluster client.
  2. Open the Client scopes tab and click Add client scope.
  3. Choose your freshly created groups client-scope and click Add (Default).
  4. Save the changes to the client.

Now, whenever the end-user authenticates against the Kubernetes Keycloak client, the group membership(s) will be added to the issued token, so Kubernetes can evaluate it and grant the user the corresponding RBAC permissions.

User

Finally, you can create a user that you can authenticate with. You can do so in the Users tab on the left of your Keycloak admin console. Click on Create and enter the user’s credentials. Don’t forget to join the user into the group(s) you created before.

Kubernetes (RKE2)

Now that we have the Keycloak instance set up and configured for Kubernetes to use, we need to configure RKE2 to use it. To do so, you need to edit your RKE2 configuration file on all your RKE2 server nodes:

/etc/rancher/rke2/config.yaml
token: b444fa51-2e1b-4c44-9f28-deff326a2cbd
tls-san:
  - kubernetes.example.com
node-taint:
  - "CriticalAddonsOnly=true:NoExecute"
kube-apiserver-arg:
  - "--oidc-issuer-url=https://keycloak.example.com/realms/master"
  - "--oidc-client-id=<your-client-id>"
  - "--oidc-username-claim=preferred_username"
  - "--oidc-groups-claim=groups"
  - "--oidc-ca-file=/etc/rancher/rke2/oidc-ca.crt"
  - "--oidc-username-prefix=oidc:"
  - "--oidc-groups-prefix=oidc:"

Issuer URL

Make sure to replace the keycloak.example.com placeholder with your actual Keycloak domain. The /realms/master path is the default path for the master realm in Keycloak. If you named your realm differently, or created a new one specifically for your Kubernetes cluster, make sure to adjust the path accordingly. You can verify that your OIDC issuer URL is correct by navigating to the URL in your browser and appending /.well-known/openid-configuration to it:

Terminal window
curl https://keycloak.example.com/realms/master/.well-known/openid-configuration  | jq .

This will give you the OpenID Connect configuration file, your Kubernetes API server will attempt to fetch upon startup. If you see the configuration in your terminal, you know that your issuer URL has been set correctly.

Client ID

Also, make sure you replace <your-client-id> with the actual client ID you chose in Keycloak when setting up the client.

CA File

Last but not least, you can see a --oidc-ca-file parameter in the configuration. This is the file containing the CA that issued the TLS certificate for your Keycloak instance. Kubernetes uses this to verify that it’s talking to the correct Keycloak instance.

If you’re unsure what the CA is (e.g. because you’re using Let’s Encrypt and didn’t think about that, yet), you can retrieve it with a bit of openssl magic in your terminal:

Terminal window
openssl s_client -showcerts -partial_chain -connect keycloak.example.com:443 < /dev/null

Your CA certificate should be the last in the chain. You can copy it and store it in a file, e.g. under /etc/rancher/rke2/oidc-ca.crt. You don’t need to bother about adding the file or the containing directory as extra volume mount to the Kubernetes API server. RKE2 automatically detects the file and adds the corresponding mount. Nice!

Username Claim

When the JWT token is issued by Keycloak, it will contain a field called preferred_username which contains your Keycloak username.

Alternatively, you can use the sub field as the username. This is the subject identifier and is unique for each Keycloak user. However, in case of Keycloak, this will be a UUID. Not so handy for humans, if you want to directly assign permissions to a user or want to see at a glance which user did what in the audit log.

However, don’t use the username field, because it will contain the full name of the user, including spaces and other special characters.

This username will then be prefixed with the --oidc-username-prefix which I recommend setting to e.g. oidc: (as in the example configuration above). That will cause the Kubernetes API server to prepend oidc: to the username in Kubernetes which allows you to easily distinguish between OIDC-authenticated users and “regular” users. This is especially useful for RBAC configuration and for audit logging.

Groups Claim

Same as with the username claim, the groups field will contain an array of group names the user is a member of in Keycloak. Thanks to the --oidc-groups-prefix parameter, the oidc: prefix will be prepended to each group name as well. As with the usernames, this is neat for distinguishing OIDC-authenticated groups from other groups, such as Kubernetes-native system groups.

If not prefixed, the groups are directly recognized by Kubernetes. This can be useful as well, but for an extra layer of security and transparency, I recommend prefixing it and then mapping it back in Kubernetes using ClusterRoleBindings or RoleBindings.

Restart RKE2 Server

After you’ve updated the configuration, you need to restart all affected RKE2 server nodes. On each node run the following command:

Terminal window
systemctl restart rke2-server.service

Please make extra-sure that you wait between restarting each node until the node gets back into a Ready state, otherwise you’ll risk losing your quorum. That would be a really bad day for you, trust me.

You can check the node status by running kubectl get nodes.

Also, don’t worry about your existing authentication. You’re probably using a token or a certificate to authenticate and this won’t change. It will still work exactly as before in addition to the OIDC authentication.

kubectl

Now that you have your Kubernetes API server configured to use Keycloak for OIDC authentication, you’re ready to try it out for the first time. In order to get a smooth authentication experience, you should install the great kubelogin plugin. This plugin will automatically open your browser and point it to the Keycloak instance to sign in and take care of token retrieval and renewal.

Once you have installed the plugin, it’s time to adjust your kubectl client configuration to use it. Open your kubeconfig file (usually located at ~/.kube/config) and amend your user configuration (or create a new one) with the following parameters:

~/.kube/config
- name: john.doe-oidc
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1beta1
      args:
      - oidc-login
      - get-token
      - --oidc-issuer-url=https://keycloak.example.com/realms/master
      - --oidc-client-id=<client-id>
      - --oidc-client-secret=<client-secret>
      command: kubectl
      env: null
      interactiveMode: IfAvailable
      provideClusterInfo: false

As with the RKE2 configuration, you need to replace the Issuer URL, Client ID and this time the Client Secret as well. Latter of which can be found in your Keycloak admin console under the Credentials tab of your client. Also, don’t forget to assign the user to a context, so your new configuration is actually used.

Then you’re ready to go and authenticate with kubectl for the first time. Try it out by running

Terminal window
kubectl get nodes

Your browser will be opened, you’ll see the Keycloak login screen and after a successful login you should see … an error. I know, I know, but it’s worth it in the end, no worries. Your user is now authenticated, but that doesn’t mean that you have any permissions to do anything whatsoever on your Kubernetes cluster. RBAC is saving the day again!

RBAC

The final step is now to assign certain Kubernetes permissions to your user - or better: to the group(s) you created in Keycloak. To test things out, you can simply create a new ClusterRole that allows your group to list nodes and then bind it to your group using a ClusterRoleBinding.

ClusterRole

"clusterrole.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: oidc-test
rules:
- apiGroups:
  - ""
  resources:
  - nodes
  verbs:
  - list

Save this to a file, e.g. clusterrole.yaml and apply it using kubectl apply -f clusterrole.yaml. If you’re unfamiliar with RBAC in Kubernetes, this will create a role without any namespace scope, granting the permission to list all nodes across the whole cluster.

However, the cluster still doesn’t know who the role we just created is assigned to.

ClusterRoleBinding

This is the final piece of glue that ties everything together. The ClusterRoleBinding will bind the oidc-test cluster role to the group(s) you created in Keycloak.

clusterrolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: oidc-test
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: oidc-test
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: oidc:my-group-name

Take a look at the highlighted line where we specified the group name. We’re using the oidc: prefix here, as we also did in the RKE2 configuration. This is mandatory, otherwise Kubernetes will not recognize the group binding.

Save this to a file, e.g. clusterrolebinding.yaml and apply it using kubectl apply -f clusterrolebinding.yaml.

Final Test

Now that everything is set up, you can test everything again.

Terminal window
kubectl get nodes

You should now see the nodes in the output. If you don’t, check the logs of the Kubernetes API server for any errors and double-check your configuration.

Terminal window
$ kubectl get nodes
NAME            STATUS   ROLES                       AGE   VERSION
k8s-master-1    Ready    control-plane,etcd,master   30h   v1.30.5+rke2r1
k8s-master-2    Ready    control-plane,etcd,master   30h   v1.30.5+rke2r1
k8s-master-3    Ready    control-plane,etcd,master   30h   v1.30.5+rke2r1
k8s-worker-1    Ready    <none>                      30h   v1.30.5+rke2r1
k8s-worker-10   Ready    <none>                      30h   v1.30.5+rke2r1
k8s-worker-11   Ready    <none>                      30h   v1.30.5+rke2r1
k8s-worker-12   Ready    <none>                      30h   v1.30.5+rke2r1
k8s-worker-2    Ready    <none>                      30h   v1.30.5+rke2r1
k8s-worker-3    Ready    <none>                      30h   v1.30.5+rke2r1
k8s-worker-4    Ready    <none>                      30h   v1.30.5+rke2r1
k8s-worker-5    Ready    <none>                      30h   v1.30.5+rke2r1
k8s-worker-6    Ready    <none>                      30h   v1.30.5+rke2r1
k8s-worker-7    Ready    <none>                      30h   v1.30.5+rke2r1
k8s-worker-8    Ready    <none>                      30h   v1.30.5+rke2r1
k8s-worker-9    Ready    <none>                      30h   v1.30.5+rke2r1

🎉 It works!

Final Words

For my security-focused audience, I’d like to point out a neat feature of this authentication mechanism. The JWT token that is issued by Keycloak is only valid for 60 seconds. It probably took you a bit longer to create the RBAC-related manifests in the last step. However, kubectl didn’t prompt you to authenticate again, even though the token had already expired. This is because kubelogin automatically retrieved a new token for you behind the scenes.

This token is statically signed and the Kubernetes API server trusts the token issuer (Keycloak). It does not revalidate the token by contacting Keycloak each time it receives a request. This makes the authentication mechanism lightning fast. However, what if we now suspend a Keycloak user? Doesn’t that mean the suspension is useless?

Nope. As the token is only valid for 60 seconds before it needs to be renewed, the worst case is that the user still has access to the Kubernetes API for another 60 seconds. After that, the token will expire. Then, during renewal, Keycloak will notice the suspension and the token won’t be issued.

If you’re interested in more details or need help setting up OIDC authentication in your Kubernetes cluster, feel free to reach out to me for consulting services and assistance.