Understanding kubeadm Bootstrap Tokens (through Node Bootstrapping)

personal

(Notes as I read and try to understand the paper, Usable Access Control in Cloud Management Systems, by Lucas Käldström.)

Today, I wanted to learn how a Kubernetes node joins a cluster (with Kubeadm join). And something caught my attention during the process.

Kubernetes is very deliberate about identity. Certificates everywhere. Mutual TLS (mTLS). Explicit authorization layers. And yet, right in the middle of the kubeadm join process, there’s a simple bearer token, passed on the command line.

That didn’t feel wrong, exactly. But it made me curious.

What I learnt is - the API server is configured to trust a list of bootstrap tokens.
The joining node (actually, the kubelet running on the joining node) sends an Authorization: Bearer <token> header in the request(s).
If the token matches a Secret in the kube-system namespace, the request is authenticated.

That’s it.

No identity provider.
No JWT verification.

So, I found myself wondering - if I manually create such a token Secret and pass it to the kubeadm join command, will it actually work?

So, I decided to try it and follow it all the way through.

Setup a control plane and a blank node

I started with a regular kind cluster:

kind create cluster

This created a control-plane container with the IP address:

172.18.0.2

To simulate a new node, I didn’t use kind’s built-in multi-node support. Instead, I created a plain docker container on the same network (the docker bridge network with the name, kind):

docker run --rm -it --privileged \
  --network kind \
  kindest/node:latest \
  /bin/bash

This new container had no certificates, no kubeconfig, no identity. Just kubeadm, kubelet, kubectl and a clean filesystem (coming from the kindest/node image).

And this is exactly what I needed.

manually create a bootstrap token

Rather than letting kubeadm generate a token for me this time, I created one manually.

A kubeadm bootstrap token has two parts:

<token-id>.<token-secret>

where:

• token-id: 6 lowercase hex chars
• token-secret: 16 lowercase hex chars

Following is the Secret object that I’m trying to make the API server trust:

apiVersion: v1
kind: Secret
metadata:
  name: bootstrap-token-pqrstu
  namespace: kube-system
type: bootstrap.kubernetes.io/token
stringData:
  token-id: pqrstu
  token-secret: abcdef1234567890
  usage-bootstrap-authentication: "true"
  usage-bootstrap-signing: "true"
  expiration: "2030-01-01T00:00:00Z"

So, if a client presents the bearer token:

pqrstu.abcdef1234567890

the API server will authenticate that client as:

system:bootstrap:pqrstu

No certificates involved yet. Just a shared secret.

First attempts: learning kubeadm’s expectations

Inside the new node docker container, I started by running kubeadm join with no arguments:

kubeadm join --v=9

That failed immediately:

I1215 14:45:12.391755     164 join.go:423] [preflight] found NodeName empty; using OS hostname as NodeName
I1215 14:45:12.392000     164 initconfiguration.go:122] detected and using CRI socket: unix:///var/run/containerd/containerd.sock

error: discovery: Invalid value: "": bootstrapToken or file must be set
no stack trace

So I tried passing just the token:

kubeadm join pqrstu.abcdef1234567890 --v=9

That produced a more informative error:

error: [discovery.bootstrapToken.caCertHashes: Invalid value: "": using token-based discovery without caCertHashes can be unsafe.
Set unsafeSkipCAVerification as true in your kubeadm config file or pass --discovery-token-unsafe-skip-ca-verification flag to continue,

discovery.bootstrapToken.apiServerEndpoint: Invalid value: "pqrstu.abcdef1234567890": address pqrstu.abcdef1234567890: missing port in address,
discovery.tlsBootstrapToken: Invalid value: "": the bootstrap token is invalid]

Right - Kubeadm expects an API server endpoint, not just credentials.

I tried again, also skipping CA verification (as suggested in the above logs) to keep things simple:

kubeadm join pqrstu.abcdef1234567890 \
  --discovery-token-unsafe-skip-ca-verification --v=9

Still not enough. Same errors as above!

first meaningful failure(s)

Once I provided everything kubeadm actually wanted, things changed:

kubeadm join 172.18.0.2:6443 \
  --token="pqrstu.abcdef1234567890" \
  --discovery-token-unsafe-skip-ca-verification --v=9

This time, authentication succeeded.

And then Kubernetes refused to continue:

219 round_trippers.go:527] "Request" curlCommand=<
	curl -v -XGET  -H "Accept: application/vnd.kubernetes.protobuf,application/json" -H "User-Agent: kubeadm/v1.35.0 (linux/amd64) kubernetes/f35f950" -H "Authorization: Bearer <masked>" 'https://kind-control-plane:6443/api/v1/namespaces/kube-system/configmaps/kubeadm-config?timeout=10s'
 >

219 round_trippers.go:632] "Response" verb="GET" url="https://kind-control-plane:6443/api/v1/namespaces/kube-system/configmaps/kubeadm-config?timeout=10s" status="403 Forbidden" headers=<...>

unable to fetch the kubeadm-config ConfigMap:
configmaps "kubeadm-config" is forbidden:
User "system:bootstrap:pqrstu" cannot get resource "configmaps"
in namespace "kube-system"

This error was the first clear signal that something important was happening.

Authentication is not authorization

At this point, the bootstrap token had done its job.

The API server knew who I was. It accepted the bearer token. The request was authenticated.

And yet the request was still denied.

This is intentional.

Bootstrap identities are deliberately weak. They exist only to start the process, not to inspect cluster state, not to read cluster configuration, and not to act as real nodes.

So to understand what happens next, I intentionally provided the required permissions.

grant extra permissions (on purpose)

I created a ClusterRoleBinding object that allowed my bootstrap identity (system:bootstrap:pqrstu) to read cluster configuration:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: allow-bootstrap-read-kubeadm-config
subjects:
- kind: User
  name: system:bootstrap:pqrstu
roleRef:
  kind: ClusterRole
  name: cluster-admin
  apiGroup: rbac.authorization.k8s.io

This is not something I would do in a real cluster. But it made the system’s layers visible.

The join completes

With that binding in place, I ran the same command again:

kubeadm join 172.18.0.2:6443 \
  --token="pqrstu.abcdef1234567890" \
  --discovery-token-unsafe-skip-ca-verification --v=9

This time, kubeadm progressed all the way through: