Priyanka Saggu

Is Kubernetes ServiceAccount a JWT token? And how to verify it?

Fri, 19 Dec 2025 00:00:00 +0000

(More of my “notes for self”, as I continue reading the thesis paper, Usable Access Control in Cloud Management Systems, written by Lucas Käldström.)

I ran another small experiment today.

Today, I learnt that the Kubernetes Service Account tokens that I use very often to authenticate with the API server (using Authorization: Bearer <token> header with the HTTP request) are JWT (JSON Web Token) tokens.

I learnt this as a verbal fact first, so, I wanted to verify it in my mighty Kind cluster.

So, let’s first create a Kind cluster.

❯ kind create cluster
Creating cluster "kind" ...
 ✓ Ensuring node image (kindest/node:v1.34.0) 🖼
 ✓ Preparing nodes 📦  
 ✓ Writing configuration 📜 
 ✓ Starting control-plane 🕹️ 
 ✓ Installing CNI 🔌 
 ✓ Installing StorageClass 💾 
Set kubectl context to "kind-kind"
You can now use your cluster with:

kubectl cluster-info --context kind-kind

Thanks for using kind! 😊

Create a test pod and exec inside the container.

❯ kubectl run jwt-test \
   --image=busybox \
   --restart=Never \
   -- sleep 3600
pod/jwt-test created

❯ kubectl exec -it jwt-test -- sh
/ # 

Now, I’m inside our container, let’s run some tests.

First, print the contents of the /var/run/secrets/kubernetes.io/serviceaccount/token file.

/ # cat /var/run/secrets/kubernetes.io/serviceaccount/token

eyJhbGciOiJSUzI1NiIsImtpZCI6IncwY3FpcXhvZGt1SFlGelNQa1FwenFMcmpoeEFkVi1McjFYcTZVTEh3X1kifQ.eyJhdWQiOlsiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwiXSwiZXhwIjoxNzk3Njg0MTc3LCJpYXQiOjE3NjYxNDgxNzcsImlzcyI6Imh0dHBzOi8va3ViZXJuZXRlcy5kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2FsIiwianRpIjoiMGQyMTFiNGEtNjVjMi00ODEyLWIwYjEtNGUzY2I2NzI5ZGMzIiwia3ViZXJuZXRlcy5pbyI6eyJuYW1lc3BhY2UiOiJkZWZhdWx0Iiwibm9kZSI6eyJuYW1lIjoia2luZC1jb250cm9sLXBsYW5lIiwidWlkIjoiNWU0MmM4M2YtMmI2NC00ZjU3LWEyZGMtMjI3M2ZmZjk3ZTBlIn0sInBvZCI6eyJuYW1lIjoiand0LXRlc3QiLCJ1aWQiOiJlNDdmMDVlZi00MWMzLTRmNDctYTdmNC01MDc1ZmIzZGQ2ZDMifSwic2VydmljZWFjY291bnQiOnsibmFtZSI6ImRlZmF1bHQiLCJ1aWQiOiI4Y2ZhNWYxNS0wOWJhLTRmM2QtODE2Ny02OGFhNjE5ZjRmN2YifSwid2FybmFmdGVyIjoxNzY2MTUxNzg0fSwibmJmIjoxNzY2MTQ4MTc3LCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6ZGVmYXVsdDpkZWZhdWx0In0.PA010UKl5ldQCOAk5s-iNRHEsbxkyIscTUsNn1c3hE9TL-uTCTl7_7QnI8-NOmx5Qjj7GPvF2QaHCeynOXlLq-Nt5mcvnOb6IipTfcH0Mfa7OCBufgPo82ggUA7T09kwcs7pmxZoL_lHxBBElFOMl9cMyhYO7I46JZ_AmvmzO4ctD3_ojQ6cyciXx4YZt78IwbM9QdM24e64BjyI_rdCGk3Y8990zodydn447VP9V6UAVQJJV49eleUnWMnQHTc3Z8UGjmawLeSaDQTqxXQ_fr9YTpHwbA_MqmXggFAmVIVQo0hTjfZxtcxuJe-8mM69Lm9krNJ7PsEuQeUB_9WyxA

Now base64 decode the above token I got.

❯ echo "eyJhbGciOiJSUzI1NiIsImtpZCI6IncwY3FpcXhvZGt1SFlGelNQa1FwenFMcmpoeEFkVi1McjFYcTZVTEh3X1kifQ.eyJhdWQiOlsiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwiXSwiZXhwIjoxNzk3Njg0MTc3LCJpYXQiOjE3NjYxNDgxNzcsImlzcyI6Imh0dHBzOi8va3ViZXJuZXRlcy5kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2FsIiwianRpIjoiMGQyMTFiNGEtNjVjMi00ODEyLWIwYjEtNGUzY2I2NzI5ZGMzIiwia3ViZXJuZXRlcy5pbyI6eyJuYW1lc3BhY2UiOiJkZWZhdWx0Iiwibm9kZSI6eyJuYW1lIjoia2luZC1jb250cm9sLXBsYW5lIiwidWlkIjoiNWU0MmM4M2YtMmI2NC00ZjU3LWEyZGMtMjI3M2ZmZjk3ZTBlIn0sInBvZCI6eyJuYW1lIjoiand0LXRlc3QiLCJ1aWQiOiJlNDdmMDVlZi00MWMzLTRmNDctYTdmNC01MDc1ZmIzZGQ2ZDMifSwic2VydmljZWFjY291bnQiOnsibmFtZSI6ImRlZmF1bHQiLCJ1aWQiOiI4Y2ZhNWYxNS0wOWJhLTRmM2QtODE2Ny02OGFhNjE5ZjRmN2YifSwid2FybmFmdGVyIjoxNzY2MTUxNzg0fSwibmJmIjoxNzY2MTQ4MTc3LCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6ZGVmYXVsdDpkZWZhdWx0In0.PA010UKl5ldQCOAk5s-iNRHEsbxkyIscTUsNn1c3hE9TL-uTCTl7_7QnI8-NOmx5Qjj7GPvF2QaHCeynOXlLq-Nt5mcvnOb6IipTfcH0Mfa7OCBufgPo82ggUA7T09kwcs7pmxZoL_lHxBBElFOMl9cMyhYO7I46JZ_AmvmzO4ctD3_ojQ6cyciXx4YZt78IwbM9QdM24e64BjyI_rdCGk3Y8990zodydn447VP9V6UAVQJJV49eleUnWMnQHTc3Z8UGjmawLeSaDQTqxXQ_fr9YTpHwbA_MqmXggFAmVIVQo0hTjfZxtcxuJe-8mM69Lm9krNJ7PsEuQeUB_9WyxA" | base64 -d

{"alg":"RS256","kid":"w0cqiqxodkuHYFzSPkQpzqLrjhxAdV-Lr1Xq6ULHw_Y"}
base64: invalid input

Ah, I got an invalid input.

But what I learnt is that a JWT token is a three part thing.
Each part is a Base64 encoded blob, separated (or joined by a dot).
So, a full token will look something like:

<base64url(header)>.<base64url(payload)>.<base64url(signature)>

What happened in our first attempt at decoding the ServiceAccount token is -
it just decoded the first blob and then reached a dot, and failed there.

So, if I divide the above token into 3 parts now, it will be:

Part 1:

eyJhbGciOiJSUzI1NiIsImtpZCI6IncwY3FpcXhvZGt1SFlGelNQa1FwenFMcmpoeEFkVi1McjFYcTZVTEh3X1kifQ

Part 2:

eyJhdWQiOlsiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwiXSwiZXhwIjoxNzk3Njg0MTc3LCJpYXQiOjE3NjYxNDgxNzcsImlzcyI6Imh0dHBzOi8va3ViZXJuZXRlcy5kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2FsIiwianRpIjoiMGQyMTFiNGEtNjVjMi00ODEyLWIwYjEtNGUzY2I2NzI5ZGMzIiwia3ViZXJuZXRlcy5pbyI6eyJuYW1lc3BhY2UiOiJkZWZhdWx0Iiwibm9kZSI6eyJuYW1lIjoia2luZC1jb250cm9sLXBsYW5lIiwidWlkIjoiNWU0MmM4M2YtMmI2NC00ZjU3LWEyZGMtMjI3M2ZmZjk3ZTBlIn0sInBvZCI6eyJuYW1lIjoiand0LXRlc3QiLCJ1aWQiOiJlNDdmMDVlZi00MWMzLTRmNDctYTdmNC01MDc1ZmIzZGQ2ZDMifSwic2VydmljZWFjY291bnQiOnsibmFtZSI6ImRlZmF1bHQiLCJ1aWQiOiI4Y2ZhNWYxNS0wOWJhLTRmM2QtODE2Ny02OGFhNjE5ZjRmN2YifSwid2FybmFmdGVyIjoxNzY2MTUxNzg0fSwibmJmIjoxNzY2MTQ4MTc3LCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6ZGVmYXVsdDpkZWZhdWx0In0

Part 3:

PA010UKl5ldQCOAk5s-iNRHEsbxkyIscTUsNn1c3hE9TL-uTCTl7_7QnI8-NOmx5Qjj7GPvF2QaHCeynOXlLq-Nt5mcvnOb6IipTfcH0Mfa7OCBufgPo82ggUA7T09kwcs7pmxZoL_lHxBBElFOMl9cMyhYO7I46JZ_AmvmzO4ctD3_ojQ6cyciXx4YZt78IwbM9QdM24e64BjyI_rdCGk3Y8990zodydn447VP9V6UAVQJJV49eleUnWMnQHTc3Z8UGjmawLeSaDQTqxXQ_fr9YTpHwbA_MqmXggFAmVIVQo0hTjfZxtcxuJe-8mM69Lm9krNJ7PsEuQeUB_9WyxA

Ok, now let’s decode them one by one again!

Decode Part 1:

❯ echo "eyJhbGciOiJSUzI1NiIsImtpZCI6IncwY3FpcXhvZGt1SFlGelNQa1FwenFMcmpoeEFkVi1McjFYcTZVTEh3X1kifQ" | base64 -d | jq .
{
  "alg": "RS256",
  "kid": "w0cqiqxodkuHYFzSPkQpzqLrjhxAdV-Lr1Xq6ULHw_Y"
}

The first key-value pair "alg": "RS256" in the output tells us, that this JWT token (which I will get in the next Part 2 decoding) was signed using a RS256 (RSA + SHA-256) algorithm.
So, if I need to verify the token signature, I know what is the algorithm used to sign it.

And the second key-value pair, the "kid": "w0cqiqxodkuHYFzSPkQpzqLrjhxAdV-Lr1Xq6ULHw_Y" part.
This is a hint to figure out which RS256 (RSA + SHA-256) private/pubic key pair was actually used to sign.
The value I see in front of kid is going to help us to figure out the public key of this pair.

But where to find the Public Key? That information, I will get in the next part.

Decode Part 2:

❯ echo "eyJhdWQiOlsiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwiXSwiZXhwIjoxNzk3Njg0MTc3LCJpYXQiOjE3NjYxNDgxNzcsImlzcyI6Imh0dHBzOi8va3ViZXJuZXRlcy5kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2FsIiwianRpIjoiMGQyMTFiNGEtNjVjMi00ODEyLWIwYjEtNGUzY2I2NzI5ZGMzIiwia3ViZXJuZXRlcy5pbyI6eyJuYW1lc3BhY2UiOiJkZWZhdWx0Iiwibm9kZSI6eyJuYW1lIjoia2luZC1jb250cm9sLXBsYW5lIiwidWlkIjoiNWU0MmM4M2YtMmI2NC00ZjU3LWEyZGMtMjI3M2ZmZjk3ZTBlIn0sInBvZCI6eyJuYW1lIjoiand0LXRlc3QiLCJ1aWQiOiJlNDdmMDVlZi00MWMzLTRmNDctYTdmNC01MDc1ZmIzZGQ2ZDMifSwic2VydmljZWFjY291bnQiOnsibmFtZSI6ImRlZmF1bHQiLCJ1aWQiOiI4Y2ZhNWYxNS0wOWJhLTRmM2QtODE2Ny02OGFhNjE5ZjRmN2YifSwid2FybmFmdGVyIjoxNzY2MTUxNzg0fSwibmJmIjoxNzY2MTQ4MTc3LCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6ZGVmYXVsdDpkZWZhdWx0In0" | base64 -d | jq .

{
  "aud": [
    "https://kubernetes.default.svc.cluster.local"
  ],
  "exp": 1797684177,
  "iat": 1766148177,
  "iss": "https://kubernetes.default.svc.cluster.local",
  "jti": "0d211b4a-65c2-4812-b0b1-4e3cb6729dc3",
  "kubernetes.io": {
    "namespace": "default",
    "node": {
      "name": "kind-control-plane",
      "uid": "5e42c83f-2b64-4f57-a2dc-2273fff97e0e"
    },
    "pod": {
      "name": "jwt-test",
      "uid": "e47f05ef-41c3-4f47-a7f4-5075fb3dd6d3"
    },
    "serviceaccount": {
      "name": "default",
      "uid": "8cfa5f15-09ba-4f3d-8167-68aa619f4f7f"
    },
    "warnafter": 1766151784
  },
  "nbf": 1766148177,
  "sub": "system:serviceaccount:default:default"
}

Ok, this is the JWT token.

The "iss": "https://kubernetes.default.svc.cluster.local" is what issued this JWT token. So, it’s the Issuer.

This is what I will use (later) to figure out the location of the public key.
The "aud": ["https://kubernetes.default.svc.cluster.local"] tells us that who is the intended audience for this token.
The "sub": "system:serviceaccount:default:default" tells us who is the subject of this token.

As in this token represents this service account (called default in the default namespace).
The "iat": 1766148177 part stands for Issued-at, and so the value is a timestamp for when this token was issued (in unix format).
The "nbf": 1766148177 part stands for “not before” meaning this token can’t be used before this time.

In this example, it is matching the “Issued-at” time, but I’m assuming it can be configured (I don’t know how at this point).
The "exp": 1797684177, part stands for “Expiration”, basically again it is a timestamp for when this token will expire.
The "jti": "0d211b4a-65c2-4812-b0b1-4e3cb6729dc3" is a unique ID for this JWT token.

And then the following part is a Kubernetes-specific entity, not really a standard JWT token field.

In this case, it’s giving Kubernetes some metadata information, for which namespace, and which particular instances of node, pod, etc are tied to this ServiceAccount.

And the warnafter bit tells when to automatically rotate this token.

    "kubernetes.io": {
      "namespace": "default",
      "node": {
        "name": "kind-control-plane",
        "uid": "5e42c83f-2b64-4f57-a2dc-2273fff97e0e"
      },
      "pod": {
        "name": "jwt-test",
        "uid": "e47f05ef-41c3-4f47-a7f4-5075fb3dd6d3"
      },
      "serviceaccount": {
        "name": "default",
        "uid": "8cfa5f15-09ba-4f3d-8167-68aa619f4f7f"
      },
      "warnafter": 1766151784
    },

And finally, the Part 3 now:

echo "PA010UKl5ldQCOAk5s-iNRHEsbxkyIscTUsNn1c3hE9TL-uTCTl7_7QnI8-NOmx5Qjj7GPvF2QaHCeynOXlLq-Nt5mcvnOb6IipTfcH0Mfa7OCBufgPo82ggUA7T09kwcs7pmxZoL_lHxBBElFOMl9cMyhYO7I46JZ_AmvmzO4ctD3_ojQ6cyciXx4YZt78IwbM9QdM24e64BjyI_rdCGk3Y8990zodydn447VP9V6UAVQJJV49eleUnWMnQHTc3Z8UGjmawLeSaDQTqxXQ_fr9YTpHwbA_MqmXggFAmVIVQo0hTjfZxtcxuJe-8mM69Lm9krNJ7PsEuQeUB_9WyxA" | base64 -d
5�B��W�$�base64: invalid input

I got some some random binary code here.

This I learnt is a raw RSA signature, that is used to sign the first 2 parts (Header and Payload) of the token.

I learnt it’s something like this:

RSA-SIGN(
  SHA256(
    base64url(header) + "." + base64url(payload)
  )
)

where to find the Public key (used to sign the JWT token)?

I saw in Part 2 decoded output this entry about who issued the token.

"iss": "https://kubernetes.default.svc.cluster.local",

Let’s see if I can find out some information from this Issuer url.

But one thing to note, before I make any request.

The Issuer of a JWT token stores the information that I am looking for at a path:

https://<url-of-the-issuer>/.well-known/openid-configuration

Now, back to our pod container.

❯ kubectl exec -it jwt-test -- sh
/ # wget https://kubernetes.default.svc.cluster.local
Connecting to kubernetes.default.svc.cluster.local (10.96.0.1:443)
wget: note: TLS certificate validation not implemented
wget: server returned error: HTTP/1.1 403 Forbidden

ok, when I tried to hit the https://kubernetes.default.svc.cluster.local url, I got 403 Forbidden.

So, I need some credentials.

You know what, here is what the Service Account token is used for.

I will pass the token as a Autherization header in our request.

Let’s try again.

/ # TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)

/ # wget --header="Authorization: Bearer $TOKEN" https://kubernetes.default.svc.cluster.local/.well-known/openid-configuration --no-check-certificate

Connecting to kubernetes.default.svc.cluster.local (10.96.0.1:443)
saving to 'openid-configuration'
openid-configuration 100% |********************************************************************************************|   236  0:00:00 ETA
'openid-configuration' saved

/ # cat openid-configuration | jq .
{
  "issuer": "https://kubernetes.default.svc.cluster.local",
  "jwks_uri": "https://172.20.0.2:6443/openid/v1/jwks",
  "response_types_supported": [
    "id_token"
  ],
  "subject_types_supported": [
    "public"
  ],
  "id_token_signing_alg_values_supported": [
    "RS256"
  ]
}

Ok, look at this part - "jwks_uri": "https://172.20.0.2:6443/openid/v1/jwks".
The “jwks” in the jwks_url stands for “JSON Web Key Set”.
This is what holds a collection of cryptographic public keys, used primarily for verifying digital signatures on JWT tokens.

So, I have almost gotten what I need.
Let’s hit this JWKS URL now.

/ # wget --header="Authorization: Bearer $TOKEN" https://172.20.0.2:6443/openid/v1/jwks --no-check-certificate
Connecting to 172.20.0.2:6443 (172.20.0.2:6443)
saving to 'jwks'
jwks                 100% |********************************************************************************************|   462  0:00:00 ETA
'jwks' saved

/# cat jwks | jq .
{
  "keys": [
    {
      "use": "sig",
      "kty": "RSA",
      "kid": "w0cqiqxodkuHYFzSPkQpzqLrjhxAdV-Lr1Xq6ULHw_Y",
      "alg": "RS256",
      "n": "vKsQjvpHQWbez2dLiTb2aJp36SKpVWvk-egE1pRertMJmtq3eeDPskb8n_msAWY4GKIMx3RnmfKBMbs_WHAkVt681cH0AzF5CR_oUtJ0Unde1rInUls5nxQcQ7_cCjApyKQlY5x5Z_vASyh7fOvMKUWmfLJt7M20hDoEvlM0WF9kUeqAgexBXlFv106qc-3CoO2-HPN6mlOn8WqHd-Ky_jQaj5xm__A0o04H7JEu09n7_Z9Rws9TFqBHaGCXwio3cozh2Bjv6da7rmyZUSp7ztH_4UcfYQgt5iJnxUdsjD7vXnyWFwvefs-6Wn6vlRp4fVmCfNrkzDL7QPWsjJJoWQ",
      "e": "AQAB"
    }
  ]
}

Voila, I got it.

See, the kid and alg matches exactly what I got in the decoded output of Part 1 of the JWT token.

{
  "alg": "RS256",
  "kid": "w0cqiqxodkuHYFzSPkQpzqLrjhxAdV-Lr1Xq6ULHw_Y"
}

So, I think the last part left for us now is to understand how can i use this to verify the signature now on the token?

let’s verify the token!

So, to verify the token. I will need the following bit I got from the jwks_uri output.

"n": "vKsQjvpHQWbez2dLiTb2aJp36SKpVWvk-egE1pRertMJmtq3eeDPskb8n_msAWY4GKIMx3RnmfKBMbs_WHAkVt681cH0AzF5CR_oUtJ0Unde1rInUls5nxQcQ7_cCjApyKQlY5x5Z_vASyh7fOvMKUWmfLJt7M20hDoEvlM0WF9kUeqAgexBXlFv106qc-3CoO2-HPN6mlOn8WqHd-Ky_jQaj5xm__A0o04H7JEu09n7_Z9Rws9TFqBHaGCXwio3cozh2Bjv6da7rmyZUSp7ztH_4UcfYQgt5iJnxUdsjD7vXnyWFwvefs-6Wn6vlRp4fVmCfNrkzDL7QPWsjJJoWQ",
"e": "AQAB"

Note: all notes this point onwards is me copy/pasting instructions I got from docs or otherwise tinkering with AI.

The n and e are respectively called the “modulus” and “public exponent” which is what I will use to contruct the RSA public key.

Below mathematics is what is used to convert the “n” and “e” to a public key.

What the following process does is 5 things for the “n” and “e” values I got:

convert them from base64url to base64 to binary.
then interpret them as Integers
then wrap them into something called ASN.1 structure
then do something called DER (Distinguished Encoding Rules) encoding this above ASN.1 structure
and finall wrap that DER into a PEM.

Once I got the PEM version, at that point, openssl will be able to use it.

I’m doing the below steps on my host machine, because i need openssl, base64, xxd, and jq, which are not present in the container.

Ok, first step - we decode n and e to binary, replacing the following and adding padding (=):

- to +
_ to /

❯ n="vKsQjvpHQWbez2dLiTb2aJp36SKpVWvk-egE1pRertMJmtq3eeDPskb8n_msAWY4GKIMx3RnmfKBMbs_WHAkVt681cH0AzF5CR_oUtJ0Unde1rInUls5nxQcQ7_cCjApyKQlY5x5Z_vASyh7fOvMKUWmfLJt7M20hDoEvlM0WF9kUeqAgexBXlFv106qc-3CoO2-HPN6mlOn8WqHd-Ky_jQaj5xm__A0o04H7JEu09n7_Z9Rws9TFqBHaGCXwio3cozh2Bjv6da7rmyZUSp7ztH_4UcfYQgt5iJnxUdsjD7vXnyWFwvefs-6Wn6vlRp4fVmCfNrkzDL7QPWsjJJoWQ"

❯ echo $n | tr '_-' '/+' | base64 -d > n.bin

❯ e="AQAB"

❯ echo $e | base64 -d > e.bin

ok, now do:

❯ xxd e.bin
00000000: 0100 01                                  ...

This represents the number 65537.

Now, we need to convert these 2 n.bin and e.bin binaries to Hexadecimal strings.

❯ xxd -p n.bin | tr -d '\n'
bcab108efa474166decf674b8936f6689a77e922a9556be4f9e804d6945eaed3099adab779e0cfb246fc9ff9ac01663818a20cc7746799f28131bb3f58702456debcd5c1f4033179091fe852d27452775ed6b227525b399f141c43bfdc0a3029c8a425639c7967fbc04b287b7cebcc2945a67cb26deccdb4843a04be5334585f6451ea8081ec415e516fd74eaa73edc2a0edbe1cf37a9a53a7f16a8777e2b2fe341a8f9c66fff034a34e07ec912ed3d9fbfd9f51c2cf5316a047686097c22a37728ce1d818efe9d6bbae6c99512a7bced1ffe1471f61082de62267c5476c8c3eef5e7c96170bde7ecfba5a7eaf951a787d59827cdae4cc32fb40f5ac8c926859

❯ xxd -p e.bin
010001

Now, we need to convert these into an ASN.1 format.

so, we create a file called rsa.asn1, with the following contents:

asn1=SEQUENCE:rsa_key

[rsa_key]
modulus=INTEGER:0x<PASTE_HEX_OF_N_HERE>
publicExponent=INTEGER:0x<PASTE_HEX_OF_E_HERE>

so, the final version would look something like:

❯ cat rsa.asn1 
asn1=SEQUENCE:rsa_key

[rsa_key]
modulus=INTEGER:0xbcab108efa474166decf674b8936f6689a77e922a9556be4f9e804d6945eaed3099adab779e0cfb246fc9ff9ac01663818a20cc7746799f28131bb3f58702456debcd5c1f4033179091fe852d27452775ed6b227525b399f141c43bfdc0a3029c8a425639c7967fbc04b287b7cebcc2945a67cb26deccdb4843a04be5334585f6451ea8081ec415e516fd74eaa73edc2a0edbe1cf37a9a53a7f16a8777e2b2fe341a8f9c66fff034a34e07ec912ed3d9fbfd9f51c2cf5316a047686097c22a37728ce1d818efe9d6bbae6c99512a7bced1ffe1471f61082de62267c5476c8c3eef5e7c96170bde7ecfba5a7eaf951a787d59827cdae4cc32fb40f5ac8c926859
publicExponent=INTEGER:0x010001

With this ASN.1 format available now, we can generate a DER encoded value.

❯ openssl asn1parse \
   -genconf rsa.asn1 \
   -out rsa_pub.der
   
    0:d=0  hl=4 l= 266 cons: SEQUENCE          
    4:d=1  hl=4 l= 257 prim: INTEGER           :BCAB108EFA474166DECF674B8936F6689A77E922A9556BE4F9E804D6945EAED3099ADAB779E0CFB246FC9FF9AC01663818A20CC7746799F28131BB3F58702456DEBCD5C1F4033179091FE852D27452775ED6B227525B399F141C43BFDC0A3029C8A425639C7967FBC04B287B7CEBCC2945A67CB26DECCDB4843A04BE5334585F6451EA8081EC415E516FD74EAA73EDC2A0EDBE1CF37A9A53A7F16A8777E2B2FE341A8F9C66FFF034A34E07EC912ED3D9FBFD9F51C2CF5316A047686097C22A37728CE1D818EFE9D6BBAE6C99512A7BCED1FFE1471F61082DE62267C5476C8C3EEF5E7C96170BDE7ECFBA5A7EAF951A787D59827CDAE4CC32FB40F5AC8C926859
  265:d=1  hl=2 l=   3 prim: INTEGER           :010001

Before we move ahead, I want to show why we are doing all this.

Because, the RSA Public key will be of this structure:

RSAPublicKey ::= SEQUENCE {
  modulus           INTEGER (n),
  publicExponent    INTEGER (e)
}

so, if you see the output of our DER encoding command, we see somethings that are needed in the above RSA Public Key structure.

In the output, we have a SEQUENCE consisting of two prime INTEGERS.
That’s exactly what we need.

Now, we are almost close to the final part, of actually converting the “n” and “e” to a RSA Public key.

❯ openssl rsa \
   -pubin \
   -inform DER \
   -in rsa_pub.der \
   -outform PEM \
   -out public.pem
writing RSA key

❯ cat public.pem 
-----BEGIN PUBLIC KEY-----
MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAvKsQjvpHQWbez2dLiTb2
aJp36SKpVWvk+egE1pRertMJmtq3eeDPskb8n/msAWY4GKIMx3RnmfKBMbs/WHAk
Vt681cH0AzF5CR/oUtJ0Unde1rInUls5nxQcQ7/cCjApyKQlY5x5Z/vASyh7fOvM
KUWmfLJt7M20hDoEvlM0WF9kUeqAgexBXlFv106qc+3CoO2+HPN6mlOn8WqHd+Ky
/jQaj5xm//A0o04H7JEu09n7/Z9Rws9TFqBHaGCXwio3cozh2Bjv6da7rmyZUSp7
ztH/4UcfYQgt5iJnxUdsjD7vXnyWFwvefs+6Wn6vlRp4fVmCfNrkzDL7QPWsjJJo
WQIDAQAB
-----END PUBLIC KEY-----

Hurrah! Hurrah! Hurrah! We have the public key! finally!

ok, but let’s verify once

❯ openssl rsa -pubin -in public.pem -text -noout
Public-Key: (2048 bit)
Modulus:
    00:bc:ab:10:8e:fa:47:41:66:de:cf:67:4b:89:36:
    f6:68:9a:77:e9:22:a9:55:6b:e4:f9:e8:04:d6:94:
    5e:ae:d3:09:9a:da:b7:79:e0:cf:b2:46:fc:9f:f9:
    ac:01:66:38:18:a2:0c:c7:74:67:99:f2:81:31:bb:
    3f:58:70:24:56:de:bc:d5:c1:f4:03:31:79:09:1f:
    e8:52:d2:74:52:77:5e:d6:b2:27:52:5b:39:9f:14:
    1c:43:bf:dc:0a:30:29:c8:a4:25:63:9c:79:67:fb:
    c0:4b:28:7b:7c:eb:cc:29:45:a6:7c:b2:6d:ec:cd:
    b4:84:3a:04:be:53:34:58:5f:64:51:ea:80:81:ec:
    41:5e:51:6f:d7:4e:aa:73:ed:c2:a0:ed:be:1c:f3:
    7a:9a:53:a7:f1:6a:87:77:e2:b2:fe:34:1a:8f:9c:
    66:ff:f0:34:a3:4e:07:ec:91:2e:d3:d9:fb:fd:9f:
    51:c2:cf:53:16:a0:47:68:60:97:c2:2a:37:72:8c:
    e1:d8:18:ef:e9:d6:bb:ae:6c:99:51:2a:7b:ce:d1:
    ff:e1:47:1f:61:08:2d:e6:22:67:c5:47:6c:8c:3e:
    ef:5e:7c:96:17:0b:de:7e:cf:ba:5a:7e:af:95:1a:
    78:7d:59:82:7c:da:e4:cc:32:fb:40:f5:ac:8c:92:
    68:59
Exponent: 65537 (0x10001)

We got the Public Key! Let’s verify the JWT token now.

Ok, I’m back to doing things by myself now.

Remember from the Part 1 and Part 2 of the token were actually the “Header” and “Payload”. And “Part3”, the signature.

What we have to verify is Part 1 and Part 2, which is what is signed by Part 3.

So, let’s sort out the the required parts.

❯ TOKEN=eyJhbGciOiJSUzI1NiIsImtpZCI6IncwY3FpcXhvZGt1SFlGelNQa1FwenFMcmpoeEFkVi1McjFYcTZVTEh3X1kifQ.eyJhdWQiOlsiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwiXSwiZXhwIjoxNzk3Njg3ODgxLCJpYXQiOjE3NjYxNTE4ODEsImlzcyI6Imh0dHBzOi8va3ViZXJuZXRlcy5kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2FsIiwianRpIjoiNmJiNzc0MjMtNmE3Yi00MjBmLTg1MDgtOTUzY2Y0YmE5MWZhIiwia3ViZXJuZXRlcy5pbyI6eyJuYW1lc3BhY2UiOiJkZWZhdWx0Iiwibm9kZSI6eyJuYW1lIjoia2luZC1jb250cm9sLXBsYW5lIiwidWlkIjoiNWU0MmM4M2YtMmI2NC00ZjU3LWEyZGMtMjI3M2ZmZjk3ZTBlIn0sInBvZCI6eyJuYW1lIjoiand0LXRlc3QiLCJ1aWQiOiI4M2E5YmE4OC0yOTg5LTRlYmItYTRiZS04ZjA0ODY2ZmM4OGEifSwic2VydmljZWFjY291bnQiOnsibmFtZSI6ImRlZmF1bHQiLCJ1aWQiOiI4Y2ZhNWYxNS0wOWJhLTRmM2QtODE2Ny02OGFhNjE5ZjRmN2YifSwid2FybmFmdGVyIjoxNzY2MTU1NDg4fSwibmJmIjoxNzY2MTUxODgxLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6ZGVmYXVsdDpkZWZhdWx0In0.ByKLRb2376aYeAmOWL1LTHsRmwgjWYp3kklUmoDcAzfMXWOBOcU_R4iXC4UqM5iwpEw_lWhDgEndGghiUg6HFau0rtj5VxFiFWwXkxSfzYNvxzW_nO3uFZlI4R3tWJqIeLMX3hqaWQGb_LvfvjI1My6XNZhkt8UTByUg3nKTmtMWmg-9-XKMylmD078vT4n8f0nSL6YlchJuTFivWc1lGE-FrZmWk6WeiRtH_jTyXp95dhg_Chf566otezUrPE8ern-8sI0rSVDzvLNsF4YvL9IXx2JQn57QR_Pr3otFXpeUTgj5oBUllsCTrA2xpXRmWxUD9qoncjviVkAkcj1fiw

❯ echo -n "$TOKEN" | cut -d. -f1,2 > signed-data.txt

❯ echo "$TOKEN" | cut -d. -f3 | tr '_-' '/+' | base64 -d > signature.bin

ok, now, we verify the signed-data.txt with the signature.bin

❯ openssl dgst -sha256 \
∙   -verify public.pem \
∙   -signature signature.bin \
∙   signed-data.txt
Verification failure
40F7505CB27F0000:error:02000068:rsa routines:ossl_rsa_verify:bad signature:crypto/rsa/rsa_sign.c:442:
40F7505CB27F0000:error:1C880004:Provider routines:rsa_verify_directly:RSA lib:providers/implementations/signature/rsa_sig.c:1043:

and I failed. 😂

Goodness, the process was tiring, so, I’m not repeating it right now.

I’ll come back to it and see at what place, I made mistakes.

But this effort was to learn how it is done, i.e., how a signature verification happens.

And also, I should not forget it all started with me just trying to understand whether a Kubernetes Service Account is a JWT token.

I got my answer and I learnt so much more.

There’s an article that I haven’t read just yet, but I was recommended to read (by Lucas Käldström).
Because, we didn’t end with a shiny green “verified” message, I will leave you to read that brilliant article -
RSA Signing is Not RSA Decryption

Thank you, if you followed so far. :)

Understanding kubeadm Bootstrap Tokens (through Node Bootstrapping)

Mon, 15 Dec 2025 00:00:00 +0000

(This blog is basically a set of notes for self, as I read and try to understand the thesis paper, Usable Access Control in Cloud Management Systems, written by Lucas Käldström.)

For a good while now, I have been wanting to understand how a node joins a Kubernetes cluster using the kubeadm join command.
And especially, the part about the symmetric token (that we generate or we get from the control plane node after a successful kubeadm init run, and then we share it with all other nodes wanting to join the cluster).

So, things I want to understand are:

why there’s a symmetric token?
and what is the role of this token?
and can I try to create a kubeadm token manually and use that to join an existing Kubernetes cluster successfully?

And as I read more and more of the thesis paper (linked at the top), I am understanding that the Access control mechanism(s) (for authentication, and authorization, and admission) used within the Kubernetes clusters are extremly elaborate and thought out from security perspective (and ofcourse, they’re not simple at all, not right away at least).
So, seeing a simple token passed through the command line in plain text format felt a bit off the place (or too simple right away).

And ofcourse, I have a feeling, that it is not. So, following is me trying to figure out some answers for my questions.

Q: What is the role of a kubeadm bootstrap token?

I created a simple Kind cluster, and looked for anything “bootstrap” related on the kube-apiserver-* pod.

❯ kind create cluster

❯ kubectl get pod kube-apiserver-kind-control-plane -n kube-system -o yaml | grep "bootstrap"

   - --enable-bootstrap-token-auth=true    

I got a flag in the output, --enable-bootstrap-token-auth=true.
And I don’t know what this flag actually does.

Now, let’s create a simple docker container with the kube-apiserver image and look at the kube-apiserver --help menu to get the definition of the flag.

❯ kubectl get pod kube-apiserver-kind-control-plane -n kube-system -o yaml | grep "image:"
   image: registry.k8s.io/kube-apiserver:v1.34.0
   image: registry.k8s.io/kube-apiserver-amd64:v1.34.0

❯ docker run -it --rm --entrypoint="" registry.k8s.io/kube-apiserver:v1.34.0 kube-apiserver --help | grep "bootstrap"

     --enable-bootstrap-token-auth                       Enable to allow secrets of type 'bootstrap.kubernetes.io/token' in the 'kube-system' namespace to be used for TLS bootstrapping authentication.

In simple words, how I understand it as:

If I set the flag --enable-bootstrap-token-auth to True, then the API server is configured to trust a list of (kubeadm) bootstrap tokens.
These tokens needs to be stored as a Secret object (of a very specific type - bootstrap.kubernetes.io/token) in the kube-system namespace.
and once that is done, then if a joining node (actually, the kubelet running on the joining node) makes a “TLS bootstrapping authentication” request using the “bootstrap token” in the request header (Authorization: Bearer <token>), then the request will be authenticated.

So, we have some information now about the kubeadm bootstrap token.

But you know what, kubeadm itself explains the role of the token much better:

❯ root@kind-control-plane:/# kubeadm token --help

This command manages bootstrap tokens. It is optional and needed only for advanced use cases.

In short, bootstrap tokens are used for establishing bidirectional trust between a client and a server.
A bootstrap token can be used when a client (for example a node that is about to join the cluster) needs
to trust the server it is talking to. Then a bootstrap token with the "signing" usage can be used.
bootstrap tokens can also function as a way to allow short-lived authentication to the API Server
(the token serves as a way for the API Server to trust the client), for example for doing the TLS Bootstrap.

What is a bootstrap token more exactly?
 - It is a Secret in the kube-system namespace of type "bootstrap.kubernetes.io/token".
 - A bootstrap token must be of the form "[a-z0-9]{6}.[a-z0-9]{16}". The former part is the public token ID,
   while the latter is the Token Secret and it must be kept private at all circumstances!
 - The name of the Secret must be named "bootstrap-token-(token-id)".

You can read more about bootstrap tokens here:
  https://kubernetes.io/docs/admin/bootstrap-tokens/

Also, the link at the bottom doesn’t work anymore.
Correct one (atleast as of writing) is - kubeadm token.

So, at this point we have a verbal answer for the question - what is the role of the kubeadm token?.

Next, I want to try is to create a manual kubeadm token (and now, from the kubeadm --help output, I also know what is the format of a valid kubeadm token).

I am continuing on the Kind cluster we created above.

The control-plane node (docker container) was created with the IP address - 172.18.0.2.
(leaving it here as a note, because we will use it later. And this is the future me talking, I didn’t know it as I was trying things.)

To simulate a new node, I didn’t use kind’s built-in multi-node support.
Instead, I created a plain docker container on the same network (the docker bridge network with the name, kind):

❯ docker run --rm -it --name joining-node \
  --privileged \
  --network kind \
  kindest/node:latest

❯ docker container ps
CONTAINER ID   IMAGE                  COMMAND                  CREATED             STATUS             PORTS                       NAMES
83ab08acf723   kindest/node:latest    "/usr/local/bin/entr…"   7 seconds ago       Up 6 seconds                                   joining-node

This new container has nothing but the basic tools installed (kubeadm, kubelet, kubectl and a clean filesystem, coming from the kindest/node image).
So, as of now, no certificates, no kubeconfig, nothing.

Also note the name of the container, joining-node.
(I will use it later to exec inside the container and use it as a joining node).

Q: Can I create a kubeadm token manually and use it to join an existing Kubernetes cluster successfully?

Back to the rules we got earlier:

What is a bootstrap token more exactly?

It is a Secret in the kube-system namespace of type “bootstrap.kubernetes.io/token”.

A bootstrap token must be of the form “[a-z0-9]{6}.[a-z0-9]{16}”. The former part is the public token ID, while the latter is the Token Secret and it must be kept private at all circumstances!

The name of the Secret must be named “bootstrap-token-(token-id)”.

I am creating the following Secret object in the cluster (from the kind-control-plane node).
Notice the token bits I added in the yaml - token-id: pqrstu and token-secret: abcdef1234567890.
Together, these will give us the full token as pqrstu.abcdef1234567890.
We will use this to make our kubeadm join requests.

To come up with the following template, I referred to existing tokens from other multi-node Kind cluster.
(updated later: the template is also available here - Bootstrap Token Secret Format)

# bootstrap-token.yaml

apiVersion: v1
kind: Secret
metadata:
  name: bootstrap-token-pqrstu
  namespace: kube-system
type: bootstrap.kubernetes.io/token
stringData:
  token-id: pqrstu
  token-secret: abcdef1234567890
  usage-bootstrap-authentication: "true"
  usage-bootstrap-signing: "true"
  expiration: "2030-01-01T00:00:00Z"

Apply to create the token secret:

❯ kubectl apply -f bootstrap-token.yaml

❯ kubectl get secrets -n kube-system

NAME                     TYPE                            DATA   AGE
bootstrap-token-abcdef   bootstrap.kubernetes.io/token   6      58m

Now let’s make some first attempts to make the new docker container node (joining-node) join the Kind cluster. (I have kept the verbosity of the logs very high).

Inside the new node docker container, I started by running kubeadm join with no arguments:

❯ docker exec -it joining-node /bin/bash

root@83ab08acf723:/# kubeadm join --v=9

I1216 07:00:01.797859     164 join.go:423] [preflight] found NodeName empty; using OS hostname as NodeName
I1216 07:00:01.798056     164 initconfiguration.go:122] detected and using CRI socket: unix:///var/run/containerd/containerd.sock
error: discovery: Invalid value: "": bootstrapToken or file must be set
no stack trace

That failed immediately.

Next, let’s try to give it the token we created above:

root@83ab08acf723:/# kubeadm join pqrstu.abcdef1234567890 --v=9

error: [discovery.bootstrapToken.caCertHashes: Invalid value: "": using token-based discovery without caCertHashes can be unsafe. Set unsafeSkipCAVerification as true in your kubeadm config file or pass --discovery-token-unsafe-skip-ca-verification flag to continue, discovery.bootstrapToken.token: Invalid value: "": the bootstrap token is invalid, discovery.bootstrapToken.apiServerEndpoint: Invalid value: "pqrstu.abcdef1234567890": address pqrstu.abcdef1234567890: missing port in address, discovery.tlsBootstrapToken: Invalid value: "": the bootstrap token is invalid]
no stack trace

That produced a more informative error logs.
Now, we atleast know that kubeadm expects an endpoint to the API server first and then the token (maybe).

Also the logs suggested us to pass --discovery-token-unsafe-skip-ca-verification flag to continue.
Let’s do that as well, to make some progress.

(and yes, I could have already looked at the kubeadm --help menu to understand the required format, but above test runs were intentional to understand basic flow).

Ok, let’s provide everything properly that kubeadm actually needs:

root@83ab08acf723:/# kubeadm join 172.18.0.2:6443 \
  --token="pqrstu.abcdef1234567890" \
  --discovery-token-unsafe-skip-ca-verification --v=9

This time things moved some further.

Let’s look at some important bits of the logs part by part.

token.go:229] [discovery] Waiting for the cluster-info ConfigMap to receive a JWS signature for token ID "pqrstu"
type.go:165] "Request Body" body=""
round_trippers.go:527] "Request" curlCommand=<
	curl -v -XGET  -H "Accept: application/vnd.kubernetes.protobuf,application/json" -H "User-Agent: kubeadm/v1.35.0 (linux/amd64) kubernetes/f35f950" 'https://172.18.0.2:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s'
 >
round_trippers.go:562] "HTTP Trace: Dial succeed" network="tcp" address="172.18.0.2:6443"
round_trippers.go:632] "Response" verb="GET" url="https://172.18.0.2:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s" status="200 OK" headers=<...>

From the above bits, it looks like at this point, we are able to successfully authenticate using the manually created token we passed (look at the status="200 OK").

But then after authentication, the API Server refused to let us continue further.

230 round_trippers.go:527] "Request" curlCommand=<
	curl -v -XGET  -H "Accept: application/vnd.kubernetes.protobuf,application/json" -H "User-Agent: kubeadm/v1.35.0 (linux/amd64) kubernetes/f35f950" -H "Authorization: Bearer <masked>" 'https://kind-control-plane:6443/api/v1/namespaces/kube-system/configmaps/kubeadm-config?timeout=10s'
 >

230 round_trippers.go:632] "Response" verb="GET" url="https://kind-control-plane:6443/api/v1/namespaces/kube-system/configmaps/kubeadm-config?timeout=10s" status="403 Forbidden" headers=<...>

230 token.go:249] [discovery] Retrying due to error: could not find a JWS signature in the cluster-info ConfigMap for token ID "pqrstu"

unable to fetch the kubeadm-config ConfigMap:
configmaps "kubeadm-config" is forbidden:
User "system:bootstrap:pqrstu" cannot get resource "configmaps"
in namespace "kube-system"

And we are also able to see where is the problem - User "system:bootstrap:pqrstu" cannot get resource "configmaps" in namespace "kube-system".

So, we know now when we pass the token to the kubeadm join command, the API Server sees our requests as coming from the user system:bootstrap:pqrstu.

ok, let’s try to fix it now by giving it the required permissions.

We know that kubernetes uses RBAC to configure these kind of permissions on the kubernetes objects.
(and once again, I looked at another multi-node kind cluster to see what permissions we are missing and came up with the following template).

I created a ClusterRoleBinding object that allowed the user (system:bootstrap:pqrstu) to read cluster configuration.

AND PLEASE NOTE: This is not something I would do in a real cluster. This is purely just to move forward with this test!

# clusterrolebinding-bootstrap-token.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: allow-bootstrap-read-kubeadm-config
subjects:
- kind: User
  name: system:bootstrap:pqrstu
roleRef:
  kind: ClusterRole
  name: cluster-admin
  apiGroup: rbac.authorization.k8s.io

Apply it to the cluster:

❯ kubectl apply -f clusterrolebinding-bootstrap-token.yaml 
clusterrolebinding.rbac.authorization.k8s.io/allow-bootstrap-read-kubeadm-config created

❯ kubectl get clusterrolebinding allow-bootstrap-read-kubeadm-config -n kube-system 
NAME                                  ROLE                        AGE
allow-bootstrap-read-kubeadm-config   ClusterRole/cluster-admin   49s

Now, we have configured more permissions for our user system:bootstrap:pqrstu.

Let’s re-run the same same command:

root@83ab08acf723:/# kubeadm join 172.18.0.2:6443 \
  --token="pqrstu.abcdef1234567890" \
  --discovery-token-unsafe-skip-ca-verification --v=9

Voila!
This time it worked successfully.
The kubeadm join ran successfully and retured - This node has joined the cluster.

I can confirm it from the Kind cluster’s control-plane node as well.
I can see this new docker container showing up as a node.

❯ kubectl get nodes 
NAME                 STATUS     ROLES           AGE    VERSION
83ab08acf723         NotReady   <none>          108s   v1.35.0-alpha.2.488+f35f9509a69cc6
kind-control-plane   Ready      control-plane   106m   v1.34.0

Please see the full logs here. These contain all the requests showing how and where the Kubeadm token is used, as request's headers. (click to expand)

root@83ab08acf723:/# kubeadm join 172.18.0.2:6443   --token="pqrstu.abcdef1234567890"   --discovery-token-unsafe-skip-ca-verification --v=9
I1215 14:50:15.160788     230 join.go:423] [preflight] found NodeName empty; using OS hostname as NodeName
I1215 14:50:15.161142     230 initconfiguration.go:122] detected and using CRI socket: unix:///var/run/containerd/containerd.sock
[preflight] Running pre-flight checks
I1215 14:50:15.161266     230 preflight.go:93] [preflight] Running general checks
I1215 14:50:15.161321     230 checks.go:315] validating the existence of file /etc/kubernetes/kubelet.conf
I1215 14:50:15.161355     230 checks.go:315] validating the existence of file /etc/kubernetes/bootstrap-kubelet.conf
I1215 14:50:15.161382     230 checks.go:89] validating the container runtime
I1215 14:50:15.167560     230 checks.go:120] validating the container runtime version compatibility
I1215 14:50:15.170013     230 checks.go:685] validating whether swap is enabled or not
	[WARNING Swap]: swap is supported for cgroup v2 only. The kubelet must be properly configured to use swap. Please refer to https://kubernetes.io/docs/concepts/architecture/nodes/#swap-memory, or disable swap on the node
I1215 14:50:15.170160     230 checks.go:405] validating the presence of executable losetup
I1215 14:50:15.170232     230 checks.go:405] validating the presence of executable mount
I1215 14:50:15.170257     230 checks.go:405] validating the presence of executable cp
I1215 14:50:15.170278     230 checks.go:551] running system verification checks
I1215 14:50:15.231856     230 checks.go:436] checking whether the given node name is valid and reachable using net.LookupHost
I1215 14:50:15.232022     230 checks.go:651] validating kubelet version
I1215 14:50:15.276735     230 checks.go:165] validating if the "kubelet" service is enabled and active
I1215 14:50:15.294376     230 checks.go:238] validating availability of port 10250
I1215 14:50:15.294632     230 checks.go:315] validating the existence of file /etc/kubernetes/pki/ca.crt
I1215 14:50:15.294661     230 checks.go:465] validating if the connectivity type is via proxy or direct
I1215 14:50:15.294728     230 checks.go:364] validating the contents of file /proc/sys/net/ipv4/ip_forward
I1215 14:50:15.294770     230 join.go:553] [preflight] Discovering cluster-info
I1215 14:50:15.294792     230 token.go:71] [discovery] Created cluster-info discovery client, requesting info from "172.18.0.2:6443"
I1215 14:50:15.294942     230 envvar.go:172] "Feature gate default state" feature="InOrderInformersBatchProcess" enabled=true
I1215 14:50:15.294959     230 envvar.go:172] "Feature gate default state" feature="InformerResourceVersion" enabled=true
I1215 14:50:15.294967     230 envvar.go:172] "Feature gate default state" feature="WatchListClient" enabled=false
I1215 14:50:15.294976     230 envvar.go:172] "Feature gate default state" feature="ClientsAllowCBOR" enabled=false
I1215 14:50:15.294984     230 envvar.go:172] "Feature gate default state" feature="ClientsPreferCBOR" enabled=false
I1215 14:50:15.294991     230 envvar.go:172] "Feature gate default state" feature="InOrderInformers" enabled=true
I1215 14:50:15.295371     230 token.go:229] [discovery] Waiting for the cluster-info ConfigMap to receive a JWS signature for token ID "pqrstu"
I1215 14:50:15.295452     230 type.go:165] "Request Body" body=""
I1215 14:50:15.295539     230 round_trippers.go:527] "Request" curlCommand=<
	curl -v -XGET  -H "Accept: application/vnd.kubernetes.protobuf,application/json" -H "User-Agent: kubeadm/v1.35.0 (linux/amd64) kubernetes/f35f950" 'https://172.18.0.2:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s'
 >
I1215 14:50:15.295931     230 round_trippers.go:562] "HTTP Trace: Dial succeed" network="tcp" address="172.18.0.2:6443"
I1215 14:50:15.302680     230 round_trippers.go:632] "Response" verb="GET" url="https://172.18.0.2:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s" status="200 OK" headers=<
	Audit-Id: 6c3035a3-0824-415c-8e5d-0db271cdb29f
	Cache-Control: no-cache, private
	Content-Length: 2102
	Content-Type: application/vnd.kubernetes.protobuf
	Date: Mon, 15 Dec 2025 14:50:15 GMT
	X-Kubernetes-Pf-Flowschema-Uid: f54c5d84-c2d8-4cbb-8801-07adb48ce73d
	X-Kubernetes-Pf-Prioritylevel-Uid: 7df0c0a7-2f8a-40ca-9625-0b6635feccd2
 > milliseconds=7 dnsLookupMilliseconds=0 dialMilliseconds=0 tlsHandshakeMilliseconds=4 serverProcessingMilliseconds=2
I1215 14:50:15.302857     230 type.go:165] "Response Body" body=<
	00000000  6b 38 73 00 0a 0f 0a 02  76 31 12 09 43 6f 6e 66  |k8s.....v1..Conf|
	00000010  69 67 4d 61 70 12 9a 10  0a 99 02 0a 0c 63 6c 75  |igMap........clu|
	00000020  73 74 65 72 2d 69 6e 66  6f 12 00 1a 0b 6b 75 62  |ster-info....kub|
	00000030  65 2d 70 75 62 6c 69 63  22 00 2a 24 38 34 37 38  |e-public".*$8478|
	00000040  65 64 38 35 2d 62 61 66  39 2d 34 63 63 61 2d 38  |ed85-baf9-4cca-8|
	00000050  31 65 32 2d 37 35 36 66  62 39 61 61 65 62 62 61  |1e2-756fb9aaebba|
	00000060  32 04 38 34 33 35 38 00  42 08 08 d7 da ff c9 06  |2.84358.B.......|
	00000070  10 00 8a 01 54 0a 07 6b  75 62 65 61 64 6d 12 06  |....T..kubeadm..|
	00000080  55 70 64 61 74 65 1a 02  76 31 22 08 08 d7 da ff  |Update..v1".....|
	00000090  c9 06 10 00 32 08 46 69  65 6c 64 73 56 31 3a 27  |....2.FieldsV1:'|
	000000a0  0a 25 7b 22 66 3a 64 61  74 61 22 3a 7b 22 2e 22  |.%{"f:data":{"."|
	000000b0  3a 7b 7d 2c 22 66 3a 6b  75 62 65 63 6f 6e 66 69  |:{},"f:kubeconfi|
	000000c0  67 22 3a 7b 7d 7d 7d 42  00 8a 01 68 0a 17 6b 75  |g":{}}}B...h..ku|
	000000d0  62 65 2d 63 6f 6e 74 72  6f 6c 6c 65 72 2d 6d 61  |be-controller-ma|
	000000e0  6e 61 67 65 72 12 06 55  70 64 61 74 65 1a 02 76  |nager..Update..v|
	000000f0  31 22 08 08 a8 87 80 ca  06 10 00 32 08 46 69 65  |1".........2.Fie|
	00000100  6c 64 73 56 31 3a 2b 0a  29 7b 22 66 3a 64 61 74  |ldsV1:+.){"f:dat|
	00000110  61 22 3a 7b 22 66 3a 6a  77 73 2d 6b 75 62 65 63  |a":{"f:jws-kubec|
	00000120  6f 6e 66 69 67 2d 70 71  72 73 74 75 22 3a 7b 7d  |onfig-pqrstu":{}|
	00000130  7d 7d 42 00 12 6e 0a 15  6a 77 73 2d 6b 75 62 65  |}}B..n..jws-kube|
	00000140  63 6f 6e 66 69 67 2d 70  71 72 73 74 75 12 55 65  |config-pqrstu.Ue|
	00000150  79 4a 68 62 47 63 69 4f  69 4a 49 55 7a 49 31 4e  |yJhbGciOiJIUzI1N|
	00000160  69 49 73 49 6d 74 70 5a  43 49 36 49 6e 42 78 63  |iIsImtpZCI6InBxc|
	00000170  6e 4e 30 64 53 4a 39 2e  2e 75 5f 4d 74 56 4f 42  |nN0dSJ9..u_MtVOB|
	00000180  69 68 65 4e 5a 37 5a 57  31 5f 5a 36 52 6d 67 78  |iheNZ7ZW1_Z6Rmgx|
	00000190  35 50 41 5f 69 76 6b 77  31 2d 62 46 53 35 49 50  |5PA_ivkw1-bFS5IP|
	000001a0  71 33 34 55 12 8b 0d 0a  0a 6b 75 62 65 63 6f 6e  |q34U.....kubecon|
	000001b0  66 69 67 12 fc 0c 61 70  69 56 65 72 73 69 6f 6e  |fig...apiVersion|
	000001c0  3a 20 76 31 0a 63 6c 75  73 74 65 72 73 3a 0a 2d  |: v1.clusters:.-|
	000001d0  20 63 6c 75 73 74 65 72  3a 0a 20 20 20 20 63 65  | cluster:.    ce|
	000001e0  72 74 69 66 69 63 61 74  65 2d 61 75 74 68 6f 72  |rtificate-author|
	000001f0  69 74 79 2d 64 61 74 61  3a 20 4c 53 30 74 4c 53  |ity-data: LS0tLS|
	00000200  31 43 52 55 64 4a 54 69  42 44 52 56 4a 55 53 55  |1CRUdJTiBDRVJUSU|
	00000210  5a 4a 51 30 46 55 52 53  30 74 4c 53 30 74 43 6b  |ZJQ0FURS0tLS0tCk|
	00000220  31 4a 53 55 52 43 56 45  4e 44 51 57 55 79 5a 30  |1JSURCVENDQWUyZ0|
	00000230  46 33 53 55 4a 42 5a 30  6c 4a 55 46 70 36 4e 31  |F3SUJBZ0lJUFp6N1|
	00000240  5a 44 52 7a 6c 4b 64 6b  31 33 52 46 46 5a 53 6b  |ZDRzlKdk13RFFZSk|
	00000250  74 76 57 6b 6c 6f 64 6d  4e 4f 51 56 46 46 54 45  |tvWklodmNOQVFFTE|
	00000260  4a 52 51 58 64 47 56 45  56 55 54 55 4a 46 52 30  |JRQXdGVEVUTUJFR0|
	00000270  45 78 56 55 55 4b 51 58  68 4e 53 32 45 7a 56 6d  |ExVUUKQXhNS2EzVm|
	00000280  6c 61 57 45 70 31 57 6c  68 53 62 47 4e 36 51 57  |laWEp1WlhSbGN6QW|
	00000290  56 47 64 7a 42 35 54 6c  52 46 65 55 31 55 56 58  |VGdzB5TlRFeU1UVX|
	000002a0  68 4e 56 45 45 30 54 56  52 57 59 55 5a 33 4d 48  |hNVEE0TVRWYUZ3MH|
	000002b0  70 4f 56 45 56 35 54 56  52 4e 65 45 31 55 52 58  |pOVEV5TVRNeE1URX|
	000002c0  70 4e 56 46 5a 68 54 55  4a 56 65 41 70 46 65 6b  |pNVFZhTUJVeApFek|
	000002d0  46 53 51 6d 64 4f 56 6b  4a 42 54 56 52 44 62 58  |FSQmdOVkJBTVRDbX|
	000002e0  51 78 57 57 31 57 65 57  4a 74 56 6a 42 61 57 45  |QxWW1WeWJtVjBaWE|
	000002f0  31 33 5a 32 64 46 61 55  31 42 4d 45 64 44 55 33  |13Z2dFaU1BMEdDU3|
	00000300  46 48 55 30 6c 69 4d 30  52 52 52 55 4a 42 55 56  |FHU0liM0RRRUJBUV|
	00000310  56 42 51 54 52 4a 51 6b  52 33 51 58 64 6e 5a 30  |VBQTRJQkR3QXdnZ0|
	00000320  56 4c 43 6b 46 76 53 55  4a 42 55 55 52 34 4d 6a  |VLCkFvSUJBUUR4Mj|
	00000330  56 79 64 55 35 61 54 6e  42 70 62 6a 68 30 51 54  |VydU5aTnBpbjh0QT|
	00000340  46 4d 52 6d 39 71 4e 31  6c 74 4e 44 56 6b 4d 30  |FMRm9qN1ltNDVkM0|
	00000350  4e 30 65 56 42 71 57 47  73 30 53 6d 4a 31 54 46  |N0eVBqWGs0SmJ1TF|
	00000360  70 53 56 55 55 33 61 6e  6c 59 53 7a 64 6d 4d 6e  |pSVUU3anlYSzdmMn|
	00000370  70 49 53 44 45 35 4d 46  59 4b 54 58 42 36 4e 31  |pISDE5MFYKTXB6N1|
	00000380  46 59 62 7a 64 32 53 31  70 6f 65 46 6c 70 55 7a  |FYbzd2S1poeFlpUz|
	00000390  64 46 61 46 52 4d 65 56  4e 6d 4f 47 68 6f 53 32  |dFaFRMeVNmOGhoS2|
	000003a0  59 34 56 32 55 34 5a 6b  6c 58 55 6b 46 45 4f 56  |Y4V2U4ZklXUkFEOV|
	000003b0  6c 55 56 57 70 61 59 6c  4a 6a 63 6a 4e 6f 52 46  |lUVWpaYlJjcjNoRF|
	000003c0  6f 72 57 44 68 6c 62 47  70 78 52 48 41 31 51 51  |orWDhlbGpxRHA1QQ|
	000003d0  70 52 65 44 4e 34 62 32  52 55 4d 6c 68 4c 4e 31  |pReDN4b2RUMlhLN1|
	000003e0  56 72 54 6e 68 50 64 6d  39 36 5a 58 68 42 4c 32  |VrTnhPdm96ZXhBL2|
	000003f0  74 53 4f 58 6c 78 65 6d  4a 72 57 56 70 7a 61 30  |tSOXlxemJrWVpza0|
	00000400  46 42 59 58 46 5a 51 54  4e 71 52 7a 42 4a 63 56  |FBYXFZQTNqRzBJcV|
	00000410  4e 46 51 58 68 4c 52 56  68 4c 57 6e 64 4d 4d 6d  |NFQXhLRVhLWndMMm|
	00000420  77 77 59 32 45 77 43 6e  70 4b 52 6a 6c 6e 62 54  |wwY2EwCnpKRjlnbT|
	00000430  4e 33 64 33 68 4f 55 47  64 49 53 57 4e 73 4e 57  |N3d3hOUGdISWNsNW|
	00000440  68 68 63 45 56 46 4e 54  56 36 63 30 67 77 65 47  |hhcEVFNTV6c0gweG|
	00000450  68 76 57 58 64 4b 5a 32  78 53 4f 57 31 6d 5a 6c  |hvWXdKZ2xSOW1mZl|
	00000460  64 4e 4e 7a 68 6b 52 32  4d 78 55 33 68 79 52 57  |dNNzhkR2MxU3hyRW|
	00000470  46 30 65 6d 5a 52 4f 44  46 72 59 55 77 4b 56 6b  |F0emZRODFrYUwKVk|
	00000480  52 50 5a 30 56 6b 62 58  4e 4b 65 48 52 4a 55 32  |RPZ0VkbXNKeHRJU2|
	00000490  45 31 53 6d 74 43 54 58  56 52 5a 7a 56 52 57 57  |E1SmtCTXVRZzVRWW|
	000004a0  39 43 51 57 4e 78 55 30  6b 31 64 33 52 30 4e 47  |9CQWNxU0k1d3R0NG|
	000004b0  6b 34 61 6c 6f 76 64 6e  70 45 64 6d 56 36 4e 48  |k4alovdnpEdmV6NH|
	000004c0  67 33 55 45 64 77 63 56  70 5a 56 54 5a 4c 54 33  |g3UEdwcVpZVTZLT3|
	000004d0  6c 48 5a 67 70 75 64 46  42 4a 55 55 49 33 64 55  |lHZgpudFBJUUI3dU|
	000004e0  46 6f 52 57 78 4d 65 46  52 53 4b 7a 4a 68 57 54  |FoRWxMeFRSKzJhWT|
	000004f0  63 7a 53 55 39 73 63 56  6b 76 51 57 64 4e 51 6b  |czSU9scVkvQWdNQk|
	00000500  46 42 52 32 70 58 56 45  4a 59 54 55 45 30 52 30  |FBR2pXVEJYTUE0R0|
	00000510  45 78 56 57 52 45 64 30  56 43 4c 33 64 52 52 55  |ExVWREd0VCL3dRRU|
	00000520  46 33 53 55 4e 77 52 45  46 51 43 6b 4a 6e 54 6c  |F3SUNwREFQCkJnTl|
	00000530  5a 49 55 6b 31 43 51 57  59 34 52 55 4a 55 51 55  |ZIUk1CQWY4RUJUQU|
	00000540  52 42 55 55 67 76 54 55  49 77 52 30 45 78 56 57  |RBUUgvTUIwR0ExVW|
	00000550  52 45 5a 31 46 58 51 6b  4a 53 55 79 39 56 4e 44  |REZ1FXQkJSUy9VND|
	00000560  4e 73 59 6a 51 32 62 45  70 74 52 48 6b 78 57 56  |NsYjQ2bEptRHkxWV|
	00000570  42 7a 56 6e 64 43 4d 31  4a 44 55 44 52 45 51 56  |BzVndCM1JDUDREQV|
	00000580  59 4b 51 6d 64 4f 56 6b  68 53 52 55 56 45 61 6b  |YKQmdOVkhSRUVEak|
	00000590  46 4e 5a 32 64 77 63 6d  52 58 53 6d 78 6a 62 54  |FNZ2dwcmRXSmxjbT|
	000005a0  56 73 5a 45 64 57 65 6b  31 42 4d 45 64 44 55 33  |VsZEdWek1BMEdDU3|
	000005b0  46 48 55 30 6c 69 4d 30  52 52 52 55 4a 44 64 31  |FHU0liM0RRRUJDd1|
	000005c0  56 42 51 54 52 4a 51 6b  46 52 51 33 56 34 64 31  |VBQTRJQkFRQ3V4d1|
	000005d0  46 4a 54 6a 64 6f 61 41  70 59 59 6d 52 4c 63 30  |FJTjdoaApYYmRLc0|
	000005e0  39 72 56 33 42 42 52 46  68 75 59 57 35 4f 57 48  |9rV3BBRFhuYW5OWH|
	000005f0  5a 4b 63 45 52 6b 54 57  38 35 61 47 38 79 54 58  |ZKcERkTW85aG8yTX|
	00000600  70 77 63 55 78 30 4f 58  46 6b 61 55 52 61 59 6b  |pwcUx0OXFkaURaYk|
	00000610  78 76 63 44 52 44 59 6e  67 72 63 30 70 53 59 7a  |xvcDRDYngrc0pSYz|
	00000620  6c 6d 62 33 4e 59 52 7a  56 58 61 6a 46 56 43 6b  |lmb3NYRzVXajFVCk|
	00000630  46 79 62 6e 6c 6a 5a 6b  6c 34 62 6b 5a 56 4d 57  |FybnljZkl4bkZVMW|
	00000640  74 70 53 48 4e 6e 64 32  55 35 4e 47 35 43 59 55  |tpSHNnd2U5NG5CYU|
	00000650  64 6a 59 56 4e 44 63 58  64 54 62 55 56 49 65 44  |djYVNDcXdTbUVIeD|
	00000660  52 73 4d 56 5a 36 53 69  74 6c 64 30 35 57 62 32  |RsMVZ6Sitld05Wb2|
	00000670  34 34 63 6e 42 75 55 30  74 59 65 47 46 52 55 33  |44cnBuU0tYeGFRU3|
	00000680  41 30 4e 6d 55 4b 4b 30  64 69 64 55 31 34 55 48  |A0NmUKK0didU14UH|
	00000690  64 56 5a 46 6c 69 64 6a  5a 32 57 44 4a 43 64 7a  |dVZFlidjZ2WDJCdz|
	000006a0  56 70 64 33 64 43 61 56  42 32 4f 57 78 43 4f 44  |Vpd3dCaVB2OWxCOD|
	000006b0  45 7a 54 6e 4a 6e 61 31  52 5a 4b 30 46 70 65 55  |EzTnJna1RZK0FpeU|
	000006c0  70 47 61 33 64 79 62 6a  46 51 53 6d 4e 55 64 6d  |pGa3dybjFQSmNUdm|
	000006d0  64 74 64 55 4a 4e 53 6b  34 7a 52 77 70 79 63 45  |dtdUJNSk4zRwpycE|
	000006e0  74 4d 55 48 6b 32 55 7a  4a 7a 4f 44 49 35 62 54  |tMUHk2UzJzODI5bT|
	000006f0  64 57 63 45 77 7a 4f 55  4d 30 4d 47 74 54 62 45  |dWcEwzOUM0MGtTbE|
	00000700  39 42 59 6d 78 4a 52 56  70 50 53 55 56 32 63 6d  |9BYmxJRVpPSUV2cm|
	00000710  46 44 65 56 68 58 54 32  68 55 62 47 56 56 56 6c  |FDeVhXT2hUbGVVVl|
	00000720  56 4c 5a 48 42 50 4e 56  5a 34 55 30 67 76 5a 56  |VLZHBPNVZ4U0gvZV|
	00000730  46 45 43 6c 64 48 57 6e  46 76 53 6a 42 35 57 56  |FECldHWnFvSjB5WV|
	00000740  4e 52 54 47 74 48 54 43  38 78 52 79 39 71 52 32  |NRTGtHTC8xRy9qR2|
	00000750  56 6c 51 57 68 4b 62 6e  56 72 4f 56 70 73 4e 54  |VlQWhKbnVrOVpsNT|
	00000760  63 79 4e 6a 63 77 62 48  4e 71 56 48 46 4d 51 55  |cyNjcwbHNqVHFMQU|
	00000770  35 68 55 69 74 58 53 30  64 47 64 7a 46 4a 4d 6e  |5hUitXS0dGdzFJMn|
	00000780  52 7a 57 56 5a 49 54 45  38 4b 62 57 35 4c 54 6e  |RzWVZITE8KbW5LTn|
	00000790  42 49 61 79 74 72 56 56  4a 34 43 69 30 74 4c 53  |BIaytrVVJ4Ci0tLS|
	000007a0  30 74 52 55 35 45 49 45  4e 46 55 6c 52 4a 52 6b  |0tRU5EIENFUlRJRk|
	000007b0  6c 44 51 56 52 46 4c 53  30 74 4c 53 30 4b 0a 20  |lDQVRFLS0tLS0K. |
	000007c0  20 20 20 73 65 72 76 65  72 3a 20 68 74 74 70 73  |   server: https|
	000007d0  3a 2f 2f 6b 69 6e 64 2d  63 6f 6e 74 72 6f 6c 2d  |://kind-control-|
	000007e0  70 6c 61 6e 65 3a 36 34  34 33 0a 20 20 6e 61 6d  |plane:6443.  nam|
	000007f0  65 3a 20 22 22 0a 63 6f  6e 74 65 78 74 73 3a 20  |e: "".contexts: |
	00000800  6e 75 6c 6c 0a 63 75 72  72 65 6e 74 2d 63 6f 6e  |null.current-con|
	00000810  74 65 78 74 3a 20 22 22  0a 6b 69 6e 64 [truncated 178 chars]
 >
I1215 14:50:15.303934     230 token.go:113] [discovery] Cluster info signature and contents are valid and no TLS pinning was specified, will use API Server "172.18.0.2:6443"
I1215 14:50:15.303958     230 discovery.go:53] [discovery] Using provided TLSBootstrapToken as authentication credentials for the join process
I1215 14:50:15.303971     230 join.go:567] [preflight] Fetching init configuration
I1215 14:50:15.303980     230 join.go:681] [preflight] Retrieving KubeConfig objects
[preflight] Reading configuration from the "kubeadm-config" ConfigMap in namespace "kube-system"...
[preflight] Use 'kubeadm init phase upload-config kubeadm --config your-config-file' to re-upload it.
I1215 14:50:15.304427     230 type.go:165] "Request Body" body=""
I1215 14:50:15.304525     230 round_trippers.go:527] "Request" curlCommand=<
	curl -v -XGET  -H "Accept: application/vnd.kubernetes.protobuf,application/json" -H "User-Agent: kubeadm/v1.35.0 (linux/amd64) kubernetes/f35f950" -H "Authorization: Bearer <masked>" 'https://kind-control-plane:6443/api/v1/namespaces/kube-system/configmaps/kubeadm-config?timeout=10s'
 >
I1215 14:50:15.305280     230 round_trippers.go:547] "HTTP Trace: DNS Lookup resolved" host="kind-control-plane" address=[{"IP":"172.18.0.2","Zone":""},{"IP":"fc00:****::2","Zone":""}]
I1215 14:50:15.305519     230 round_trippers.go:562] "HTTP Trace: Dial succeed" network="tcp" address="172.18.0.2:6443"
I1215 14:50:15.312270     230 round_trippers.go:632] "Response" verb="GET" url="https://kind-control-plane:6443/api/v1/namespaces/kube-system/configmaps/kubeadm-config?timeout=10s" status="200 OK" headers=<
	Audit-Id: dc851cb8-00d2-499a-9d93-791b645d3175
	Cache-Control: no-cache, private
	Content-Length: 935
	Content-Type: application/vnd.kubernetes.protobuf
	Date: Mon, 15 Dec 2025 14:50:15 GMT
	X-Kubernetes-Pf-Flowschema-Uid: f54c5d84-c2d8-4cbb-8801-07adb48ce73d
	X-Kubernetes-Pf-Prioritylevel-Uid: 7df0c0a7-2f8a-40ca-9625-0b6635feccd2
 > milliseconds=7 dnsLookupMilliseconds=0 dialMilliseconds=0 tlsHandshakeMilliseconds=4 serverProcessingMilliseconds=2
I1215 14:50:15.312385     230 type.go:165] "Response Body" body=<
	00000000  6b 38 73 00 0a 0f 0a 02  76 31 12 09 43 6f 6e 66  |k8s.....v1..Conf|
	00000010  69 67 4d 61 70 12 8b 07  0a b9 01 0a 0e 6b 75 62  |igMap........kub|
	00000020  65 61 64 6d 2d 63 6f 6e  66 69 67 12 00 1a 0b 6b  |eadm-config....k|
	00000030  75 62 65 2d 73 79 73 74  65 6d 22 00 2a 24 38 66  |ube-system".*$8f|
	00000040  37 61 38 66 35 34 2d 37  39 62 65 2d 34 34 61 35  |7a8f54-79be-44a5|
	00000050  2d 61 63 35 63 2d 34 34  32 34 64 33 66 37 39 39  |-ac5c-4424d3f799|
	00000060  30 39 32 03 32 30 37 38  00 42 08 08 d7 da ff c9  |092.2078.B......|
	00000070  06 10 00 8a 01 5e 0a 07  6b 75 62 65 61 64 6d 12  |.....^..kubeadm.|
	00000080  06 55 70 64 61 74 65 1a  02 76 31 22 08 08 d7 da  |.Update..v1"....|
	00000090  ff c9 06 10 00 32 08 46  69 65 6c 64 73 56 31 3a  |.....2.FieldsV1:|
	000000a0  31 0a 2f 7b 22 66 3a 64  61 74 61 22 3a 7b 22 2e  |1./{"f:data":{".|
	000000b0  22 3a 7b 7d 2c 22 66 3a  43 6c 75 73 74 65 72 43  |":{},"f:ClusterC|
	000000c0  6f 6e 66 69 67 75 72 61  74 69 6f 6e 22 3a 7b 7d  |onfiguration":{}|
	000000d0  7d 7d 42 00 12 cc 05 0a  14 43 6c 75 73 74 65 72  |}}B......Cluster|
	000000e0  43 6f 6e 66 69 67 75 72  61 74 69 6f 6e 12 b3 05  |Configuration...|
	000000f0  61 70 69 53 65 72 76 65  72 3a 0a 20 20 63 65 72  |apiServer:.  cer|
	00000100  74 53 41 4e 73 3a 0a 20  20 2d 20 6c 6f 63 61 6c  |tSANs:.  - local|
	00000110  68 6f 73 74 0a 20 20 2d  20 31 32 37 2e 30 2e 30  |host.  - 127.0.0|
	00000120  2e 31 0a 20 20 65 78 74  72 61 41 72 67 73 3a 0a  |.1.  extraArgs:.|
	00000130  20 20 2d 20 6e 61 6d 65  3a 20 72 75 6e 74 69 6d  |  - name: runtim|
	00000140  65 2d 63 6f 6e 66 69 67  0a 20 20 20 20 76 61 6c  |e-config.    val|
	00000150  75 65 3a 20 22 22 0a 61  70 69 56 65 72 73 69 6f  |ue: "".apiVersio|
	00000160  6e 3a 20 6b 75 62 65 61  64 6d 2e 6b 38 73 2e 69  |n: kubeadm.k8s.i|
	00000170  6f 2f 76 31 62 65 74 61  34 0a 63 61 43 65 72 74  |o/v1beta4.caCert|
	00000180  69 66 69 63 61 74 65 56  61 6c 69 64 69 74 79 50  |ificateValidityP|
	00000190  65 72 69 6f 64 3a 20 38  37 36 30 30 68 30 6d 30  |eriod: 87600h0m0|
	000001a0  73 0a 63 65 72 74 69 66  69 63 61 74 65 56 61 6c  |s.certificateVal|
	000001b0  69 64 69 74 79 50 65 72  69 6f 64 3a 20 38 37 36  |idityPeriod: 876|
	000001c0  30 68 30 6d 30 73 0a 63  65 72 74 69 66 69 63 61  |0h0m0s.certifica|
	000001d0  74 65 73 44 69 72 3a 20  2f 65 74 63 2f 6b 75 62  |tesDir: /etc/kub|
	000001e0  65 72 6e 65 74 65 73 2f  70 6b 69 0a 63 6c 75 73  |ernetes/pki.clus|
	000001f0  74 65 72 4e 61 6d 65 3a  20 6b 69 6e 64 0a 63 6f  |terName: kind.co|
	00000200  6e 74 72 6f 6c 50 6c 61  6e 65 45 6e 64 70 6f 69  |ntrolPlaneEndpoi|
	00000210  6e 74 3a 20 6b 69 6e 64  2d 63 6f 6e 74 72 6f 6c  |nt: kind-control|
	00000220  2d 70 6c 61 6e 65 3a 36  34 34 33 0a 63 6f 6e 74  |-plane:6443.cont|
	00000230  72 6f 6c 6c 65 72 4d 61  6e 61 67 65 72 3a 0a 20  |rollerManager:. |
	00000240  20 65 78 74 72 61 41 72  67 73 3a 0a 20 20 2d 20  | extraArgs:.  - |
	00000250  6e 61 6d 65 3a 20 65 6e  61 62 6c 65 2d 68 6f 73  |name: enable-hos|
	00000260  74 70 61 74 68 2d 70 72  6f 76 69 73 69 6f 6e 65  |tpath-provisione|
	00000270  72 0a 20 20 20 20 76 61  6c 75 65 3a 20 22 74 72  |r.    value: "tr|
	00000280  75 65 22 0a 64 6e 73 3a  20 7b 7d 0a 65 6e 63 72  |ue".dns: {}.encr|
	00000290  79 70 74 69 6f 6e 41 6c  67 6f 72 69 74 68 6d 3a  |yptionAlgorithm:|
	000002a0  20 52 53 41 2d 32 30 34  38 0a 65 74 63 64 3a 0a  | RSA-2048.etcd:.|
	000002b0  20 20 6c 6f 63 61 6c 3a  0a 20 20 20 20 64 61 74  |  local:.    dat|
	000002c0  61 44 69 72 3a 20 2f 76  61 72 2f 6c 69 62 2f 65  |aDir: /var/lib/e|
	000002d0  74 63 64 0a 69 6d 61 67  65 52 65 70 6f 73 69 74  |tcd.imageReposit|
	000002e0  6f 72 79 3a 20 72 65 67  69 73 74 72 79 2e 6b 38  |ory: registry.k8|
	000002f0  73 2e 69 6f 0a 6b 69 6e  64 3a 20 43 6c 75 73 74  |s.io.kind: Clust|
	00000300  65 72 43 6f 6e 66 69 67  75 72 61 74 69 6f 6e 0a  |erConfiguration.|
	00000310  6b 75 62 65 72 6e 65 74  65 73 56 65 72 73 69 6f  |kubernetesVersio|
	00000320  6e 3a 20 76 31 2e 33 34  2e 30 0a 6e 65 74 77 6f  |n: v1.34.0.netwo|
	00000330  72 6b 69 6e 67 3a 0a 20  20 64 6e 73 44 6f 6d 61  |rking:.  dnsDoma|
	00000340  69 6e 3a 20 63 6c 75 73  74 65 72 2e 6c 6f 63 61  |in: cluster.loca|
	00000350  6c 0a 20 20 70 6f 64 53  75 62 6e 65 74 3a 20 31  |l.  podSubnet: 1|
	00000360  30 2e 32 34 34 2e 30 2e  30 2f 31 36 0a 20 20 73  |0.244.0.0/16.  s|
	00000370  65 72 76 69 63 65 53 75  62 6e 65 74 3a 20 31 30  |erviceSubnet: 10|
	00000380  2e 39 36 2e 30 2e 30 2f  31 36 0a 70 72 6f 78 79  |.96.0.0/16.proxy|
	00000390  3a 20 7b 7d 0a 73 63 68  65 64 75 6c 65 72 3a 20  |: {}.scheduler: |
	000003a0  7b 7d 0a 1a 00 22 00                              |{}...".|
 >
I1215 14:50:15.313552     230 kubeproxy.go:55] attempting to download the KubeProxyConfiguration from ConfigMap "kube-proxy"
I1215 14:50:15.313605     230 type.go:165] "Request Body" body=""
I1215 14:50:15.313716     230 round_trippers.go:527] "Request" curlCommand=<
	curl -v -XGET  -H "Authorization: Bearer <masked>" -H "Accept: application/vnd.kubernetes.protobuf,application/json" -H "User-Agent: kubeadm/v1.35.0 (linux/amd64) kubernetes/f35f950" 'https://kind-control-plane:6443/api/v1/namespaces/kube-system/configmaps/kube-proxy?timeout=10s'
 >
I1215 14:50:15.315629     230 round_trippers.go:632] "Response" verb="GET" url="https://kind-control-plane:6443/api/v1/namespaces/kube-system/configmaps/kube-proxy?timeout=10s" status="200 OK" headers=<
	Audit-Id: 7da1f078-b1a7-40b7-8b9d-595fb32485b5
	Cache-Control: no-cache, private
	Content-Length: 2089
	Content-Type: application/vnd.kubernetes.protobuf
	Date: Mon, 15 Dec 2025 14:50:15 GMT
	X-Kubernetes-Pf-Flowschema-Uid: f54c5d84-c2d8-4cbb-8801-07adb48ce73d
	X-Kubernetes-Pf-Prioritylevel-Uid: 7df0c0a7-2f8a-40ca-9625-0b6635feccd2
 > milliseconds=1 getConnectionMilliseconds=0 serverProcessingMilliseconds=1
I1215 14:50:15.315778     230 type.go:165] "Response Body" body=<
	00000000  6b 38 73 00 0a 0f 0a 02  76 31 12 09 43 6f 6e 66  |k8s.....v1..Conf|
	00000010  69 67 4d 61 70 12 8d 10  0a 85 02 0a 0a 6b 75 62  |igMap........kub|
	00000020  65 2d 70 72 6f 78 79 12  00 1a 0b 6b 75 62 65 2d  |e-proxy....kube-|
	00000030  73 79 73 74 65 6d 22 00  2a 24 34 31 37 66 31 65  |system".*$417f1e|
	00000040  38 32 2d 31 33 33 63 2d  34 65 38 32 2d 38 62 66  |82-133c-4e82-8bf|
	00000050  36 2d 38 66 34 36 34 31  33 31 33 61 66 37 32 03  |6-8f4641313af72.|
	00000060  32 33 38 38 00 42 08 08  d8 da ff c9 06 10 00 5a  |2388.B.........Z|
	00000070  11 0a 03 61 70 70 12 0a  6b 75 62 65 2d 70 72 6f  |...app..kube-pro|
	00000080  78 79 8a 01 9a 01 0a 07  6b 75 62 65 61 64 6d 12  |xy......kubeadm.|
	00000090  06 55 70 64 61 74 65 1a  02 76 31 22 08 08 d8 da  |.Update..v1"....|
	000000a0  ff c9 06 10 00 32 08 46  69 65 6c 64 73 56 31 3a  |.....2.FieldsV1:|
	000000b0  6d 0a 6b 7b 22 66 3a 64  61 74 61 22 3a 7b 22 2e  |m.k{"f:data":{".|
	000000c0  22 3a 7b 7d 2c 22 66 3a  63 6f 6e 66 69 67 2e 63  |":{},"f:config.c|
	000000d0  6f 6e 66 22 3a 7b 7d 2c  22 66 3a 6b 75 62 65 63  |onf":{},"f:kubec|
	000000e0  6f 6e 66 69 67 2e 63 6f  6e 66 22 3a 7b 7d 7d 2c  |onfig.conf":{}},|
	000000f0  22 66 3a 6d 65 74 61 64  61 74 61 22 3a 7b 22 66  |"f:metadata":{"f|
	00000100  3a 6c 61 62 65 6c 73 22  3a 7b 22 2e 22 3a 7b 7d  |:labels":{".":{}|
	00000110  2c 22 66 3a 61 70 70 22  3a 7b 7d 7d 7d 7d 42 00  |,"f:app":{}}}}B.|
	00000120  12 d1 0a 0a 0b 63 6f 6e  66 69 67 2e 63 6f 6e 66  |.....config.conf|
	00000130  12 c1 0a 61 70 69 56 65  72 73 69 6f 6e 3a 20 6b  |...apiVersion: k|
	00000140  75 62 65 70 72 6f 78 79  2e 63 6f 6e 66 69 67 2e  |ubeproxy.config.|
	00000150  6b 38 73 2e 69 6f 2f 76  31 61 6c 70 68 61 31 0a  |k8s.io/v1alpha1.|
	00000160  62 69 6e 64 41 64 64 72  65 73 73 3a 20 30 2e 30  |bindAddress: 0.0|
	00000170  2e 30 2e 30 0a 62 69 6e  64 41 64 64 72 65 73 73  |.0.0.bindAddress|
	00000180  48 61 72 64 46 61 69 6c  3a 20 66 61 6c 73 65 0a  |HardFail: false.|
	00000190  63 6c 69 65 6e 74 43 6f  6e 6e 65 63 74 69 6f 6e  |clientConnection|
	000001a0  3a 0a 20 20 61 63 63 65  70 74 43 6f 6e 74 65 6e  |:.  acceptConten|
	000001b0  74 54 79 70 65 73 3a 20  22 22 0a 20 20 62 75 72  |tTypes: "".  bur|
	000001c0  73 74 3a 20 30 0a 20 20  63 6f 6e 74 65 6e 74 54  |st: 0.  contentT|
	000001d0  79 70 65 3a 20 22 22 0a  20 20 6b 75 62 65 63 6f  |ype: "".  kubeco|
	000001e0  6e 66 69 67 3a 20 2f 76  61 72 2f 6c 69 62 2f 6b  |nfig: /var/lib/k|
	000001f0  75 62 65 2d 70 72 6f 78  79 2f 6b 75 62 65 63 6f  |ube-proxy/kubeco|
	00000200  6e 66 69 67 2e 63 6f 6e  66 0a 20 20 71 70 73 3a  |nfig.conf.  qps:|
	00000210  20 30 0a 63 6c 75 73 74  65 72 43 49 44 52 3a 20  | 0.clusterCIDR: |
	00000220  31 30 2e 32 34 34 2e 30  2e 30 2f 31 36 0a 63 6f  |10.244.0.0/16.co|
	00000230  6e 66 69 67 53 79 6e 63  50 65 72 69 6f 64 3a 20  |nfigSyncPeriod: |
	00000240  30 73 0a 63 6f 6e 6e 74  72 61 63 6b 3a 0a 20 20  |0s.conntrack:.  |
	00000250  6d 61 78 50 65 72 43 6f  72 65 3a 20 30 0a 20 20  |maxPerCore: 0.  |
	00000260  6d 69 6e 3a 20 6e 75 6c  6c 0a 20 20 74 63 70 42  |min: null.  tcpB|
	00000270  65 4c 69 62 65 72 61 6c  3a 20 66 61 6c 73 65 0a  |eLiberal: false.|
	00000280  20 20 74 63 70 43 6c 6f  73 65 57 61 69 74 54 69  |  tcpCloseWaitTi|
	00000290  6d 65 6f 75 74 3a 20 6e  75 6c 6c 0a 20 20 74 63  |meout: null.  tc|
	000002a0  70 45 73 74 61 62 6c 69  73 68 65 64 54 69 6d 65  |pEstablishedTime|
	000002b0  6f 75 74 3a 20 6e 75 6c  6c 0a 20 20 75 64 70 53  |out: null.  udpS|
	000002c0  74 72 65 61 6d 54 69 6d  65 6f 75 74 3a 20 30 73  |treamTimeout: 0s|
	000002d0  0a 20 20 75 64 70 54 69  6d 65 6f 75 74 3a 20 30  |.  udpTimeout: 0|
	000002e0  73 0a 64 65 74 65 63 74  4c 6f 63 61 6c 3a 0a 20  |s.detectLocal:. |
	000002f0  20 62 72 69 64 67 65 49  6e 74 65 72 66 61 63 65  | bridgeInterface|
	00000300  3a 20 22 22 0a 20 20 69  6e 74 65 72 66 61 63 65  |: "".  interface|
	00000310  4e 61 6d 65 50 72 65 66  69 78 3a 20 22 22 0a 64  |NamePrefix: "".d|
	00000320  65 74 65 63 74 4c 6f 63  61 6c 4d 6f 64 65 3a 20  |etectLocalMode: |
	00000330  22 22 0a 65 6e 61 62 6c  65 50 72 6f 66 69 6c 69  |"".enableProfili|
	00000340  6e 67 3a 20 66 61 6c 73  65 0a 68 65 61 6c 74 68  |ng: false.health|
	00000350  7a 42 69 6e 64 41 64 64  72 65 73 73 3a 20 22 22  |zBindAddress: ""|
	00000360  0a 68 6f 73 74 6e 61 6d  65 4f 76 65 72 72 69 64  |.hostnameOverrid|
	00000370  65 3a 20 22 22 0a 69 70  74 61 62 6c 65 73 3a 0a  |e: "".iptables:.|
	00000380  20 20 6c 6f 63 61 6c 68  6f 73 74 4e 6f 64 65 50  |  localhostNodeP|
	00000390  6f 72 74 73 3a 20 6e 75  6c 6c 0a 20 20 6d 61 73  |orts: null.  mas|
	000003a0  71 75 65 72 61 64 65 41  6c 6c 3a 20 66 61 6c 73  |queradeAll: fals|
	000003b0  65 0a 20 20 6d 61 73 71  75 65 72 61 64 65 42 69  |e.  masqueradeBi|
	000003c0  74 3a 20 6e 75 6c 6c 0a  20 20 6d 69 6e 53 79 6e  |t: null.  minSyn|
	000003d0  63 50 65 72 69 6f 64 3a  20 31 73 0a 20 20 73 79  |cPeriod: 1s.  sy|
	000003e0  6e 63 50 65 72 69 6f 64  3a 20 30 73 0a 69 70 76  |ncPeriod: 0s.ipv|
	000003f0  73 3a 0a 20 20 65 78 63  6c 75 64 65 43 49 44 52  |s:.  excludeCIDR|
	00000400  73 3a 20 6e 75 6c 6c 0a  20 20 6d 69 6e 53 79 6e  |s: null.  minSyn|
	00000410  63 50 65 72 69 6f 64 3a  20 30 73 0a 20 20 73 63  |cPeriod: 0s.  sc|
	00000420  68 65 64 75 6c 65 72 3a  20 22 22 0a 20 20 73 74  |heduler: "".  st|
	00000430  72 69 63 74 41 52 50 3a  20 66 61 6c 73 65 0a 20  |rictARP: false. |
	00000440  20 73 79 6e 63 50 65 72  69 6f 64 3a 20 30 73 0a  | syncPeriod: 0s.|
	00000450  20 20 74 63 70 46 69 6e  54 69 6d 65 6f 75 74 3a  |  tcpFinTimeout:|
	00000460  20 30 73 0a 20 20 74 63  70 54 69 6d 65 6f 75 74  | 0s.  tcpTimeout|
	00000470  3a 20 30 73 0a 20 20 75  64 70 54 69 6d 65 6f 75  |: 0s.  udpTimeou|
	00000480  74 3a 20 30 73 0a 6b 69  6e 64 3a 20 4b 75 62 65  |t: 0s.kind: Kube|
	00000490  50 72 6f 78 79 43 6f 6e  66 69 67 75 72 61 74 69  |ProxyConfigurati|
	000004a0  6f 6e 0a 6c 6f 67 67 69  6e 67 3a 0a 20 20 66 6c  |on.logging:.  fl|
	000004b0  75 73 68 46 72 65 71 75  65 6e 63 79 3a 20 30 0a  |ushFrequency: 0.|
	000004c0  20 20 6f 70 74 69 6f 6e  73 3a 0a 20 20 20 20 6a  |  options:.    j|
	000004d0  73 6f 6e 3a 0a 20 20 20  20 20 20 69 6e 66 6f 42  |son:.      infoB|
	000004e0  75 66 66 65 72 53 69 7a  65 3a 20 22 30 22 0a 20  |ufferSize: "0". |
	000004f0  20 20 20 74 65 78 74 3a  0a 20 20 20 20 20 20 69  |   text:.      i|
	00000500  6e 66 6f 42 75 66 66 65  72 53 69 7a 65 3a 20 22  |nfoBufferSize: "|
	00000510  30 22 0a 20 20 76 65 72  62 6f 73 69 74 79 3a 20  |0".  verbosity: |
	00000520  30 0a 6d 65 74 72 69 63  73 42 69 6e 64 41 64 64  |0.metricsBindAdd|
	00000530  72 65 73 73 3a 20 22 22  0a 6d 6f 64 65 3a 20 69  |ress: "".mode: i|
	00000540  70 74 61 62 6c 65 73 0a  6e 66 74 61 62 6c 65 73  |ptables.nftables|
	00000550  3a 0a 20 20 6d 61 73 71  75 65 72 61 64 65 41 6c  |:.  masqueradeAl|
	00000560  6c 3a 20 66 61 6c 73 65  0a 20 20 6d 61 73 71 75  |l: false.  masqu|
	00000570  65 72 61 64 65 42 69 74  3a 20 6e 75 6c 6c 0a 20  |eradeBit: null. |
	00000580  20 6d 69 6e 53 79 6e 63  50 65 72 69 6f 64 3a 20  | minSyncPeriod: |
	00000590  30 73 0a 20 20 73 79 6e  63 50 65 72 69 6f 64 3a  |0s.  syncPeriod:|
	000005a0  20 30 73 0a 6e 6f 64 65  50 6f 72 74 41 64 64 72  | 0s.nodePortAddr|
	000005b0  65 73 73 65 73 3a 20 6e  75 6c 6c 0a 6f 6f 6d 53  |esses: null.oomS|
	000005c0  63 6f 72 65 41 64 6a 3a  20 6e 75 6c 6c 0a 70 6f  |coreAdj: null.po|
	000005d0  72 74 52 61 6e 67 65 3a  20 22 22 0a 73 68 6f 77  |rtRange: "".show|
	000005e0  48 69 64 64 65 6e 4d 65  74 72 69 63 73 46 6f 72  |HiddenMetricsFor|
	000005f0  56 65 72 73 69 6f 6e 3a  20 22 22 0a 77 69 6e 6b  |Version: "".wink|
	00000600  65 72 6e 65 6c 3a 0a 20  20 65 6e 61 62 6c 65 44  |ernel:.  enableD|
	00000610  53 52 3a 20 66 61 6c 73  65 0a 20 20 66 6f 72 77  |SR: false.  forw|
	00000620  61 72 64 48 65 61 6c 74  68 43 68 65 63 6b 56 69  |ardHealthCheckVi|
	00000630  70 3a 20 66 61 6c 73 65  0a 20 20 6e 65 74 77 6f  |p: false.  netwo|
	00000640  72 6b 4e 61 6d 65 3a 20  22 22 0a 20 20 72 6f 6f  |rkName: "".  roo|
	00000650  74 48 6e 73 45 6e 64 70  6f 69 6e 74 4e 61 6d 65  |tHnsEndpointName|
	00000660  3a 20 22 22 0a 20 20 73  6f 75 72 63 65 56 69 70  |: "".  sourceVip|
	00000670  3a 20 22 22 12 ae 03 0a  0f 6b 75 62 65 63 6f 6e  |: "".....kubecon|
	00000680  66 69 67 2e 63 6f 6e 66  12 9a 03 61 70 69 56 65  |fig.conf...apiVe|
	00000690  72 73 69 6f 6e 3a 20 76  31 0a 6b 69 6e 64 3a 20  |rsion: v1.kind: |
	000006a0  43 6f 6e 66 69 67 0a 63  6c 75 73 74 65 72 73 3a  |Config.clusters:|
	000006b0  0a 2d 20 63 6c 75 73 74  65 72 3a 0a 20 20 20 20  |.- cluster:.    |
	000006c0  63 65 72 74 69 66 69 63  61 74 65 2d 61 75 74 68  |certificate-auth|
	000006d0  6f 72 69 74 79 3a 20 2f  76 61 72 2f 72 75 6e 2f  |ority: /var/run/|
	000006e0  73 65 63 72 65 74 73 2f  6b 75 62 65 72 6e 65 74  |secrets/kubernet|
	000006f0  65 73 2e 69 6f 2f 73 65  72 76 69 63 65 61 63 63  |es.io/serviceacc|
	00000700  6f 75 6e 74 2f 63 61 2e  63 72 74 0a 20 20 20 20  |ount/ca.crt.    |
	00000710  73 65 72 76 65 72 3a 20  68 74 74 70 73 3a 2f 2f  |server: https://|
	00000720  6b 69 6e 64 2d 63 6f 6e  74 72 6f 6c 2d 70 6c 61  |kind-control-pla|
	00000730  6e 65 3a 36 34 34 33 0a  20 20 6e 61 6d 65 3a 20  |ne:6443.  name: |
	00000740  64 65 66 61 75 6c 74 0a  63 6f 6e 74 65 78 74 73  |default.contexts|
	00000750  3a 0a 2d 20 63 6f 6e 74  65 78 74 3a 0a 20 20 20  |:.- context:.   |
	00000760  20 63 6c 75 73 74 65 72  3a 20 64 65 66 61 75 6c  | cluster: defaul|
	00000770  74 0a 20 20 20 20 6e 61  6d 65 73 70 61 63 65 3a  |t.    namespace:|
	00000780  20 64 65 66 61 75 6c 74  0a 20 20 20 20 75 73 65  | default.    use|
	00000790  72 3a 20 64 65 66 61 75  6c 74 0a 20 20 6e 61 6d  |r: default.  nam|
	000007a0  65 3a 20 64 65 66 61 75  6c 74 0a 63 75 72 72 65  |e: default.curre|
	000007b0  6e 74 2d 63 6f 6e 74 65  78 74 3a 20 64 65 66 61  |nt-context: defa|
	000007c0  75 6c 74 0a 75 73 65 72  73 3a 0a 2d 20 6e 61 6d  |ult.users:.- nam|
	000007d0  65 3a 20 64 65 66 61 75  6c 74 0a 20 20 75 73 65  |e: default.  use|
	000007e0  72 3a 0a 20 20 20 20 74  6f 6b 65 6e 46 69 6c 65  |r:.    tokenFile|
	000007f0  3a 20 2f 76 61 72 2f 72  75 6e 2f 73 65 63 72 65  |: /var/run/secre|
	00000800  74 73 2f 6b 75 62 65 72  6e 65 74 65 73 2e 69 6f  |ts/kubernetes.io|
	00000810  2f 73 65 72 76 69 63 65  61 63 63 6f 75 [truncated 102 chars]
 >
I1215 14:50:15.319974     230 kubelet.go:73] attempting to download the KubeletConfiguration from ConfigMap "kubelet-config"
I1215 14:50:15.320052     230 type.go:165] "Request Body" body=""
I1215 14:50:15.320143     230 round_trippers.go:527] "Request" curlCommand=<
	curl -v -XGET  -H "Accept: application/vnd.kubernetes.protobuf,application/json" -H "User-Agent: kubeadm/v1.35.0 (linux/amd64) kubernetes/f35f950" -H "Authorization: Bearer <masked>" 'https://kind-control-plane:6443/api/v1/namespaces/kube-system/configmaps/kubelet-config?timeout=10s'
 >
I1215 14:50:15.322275     230 round_trippers.go:632] "Response" verb="GET" url="https://kind-control-plane:6443/api/v1/namespaces/kube-system/configmaps/kubelet-config?timeout=10s" status="200 OK" headers=<
	Audit-Id: edacaa09-2a92-49f8-992d-467253d02bac
	Cache-Control: no-cache, private
	Content-Length: 1453
	Content-Type: application/vnd.kubernetes.protobuf
	Date: Mon, 15 Dec 2025 14:50:15 GMT
	X-Kubernetes-Pf-Flowschema-Uid: f54c5d84-c2d8-4cbb-8801-07adb48ce73d
	X-Kubernetes-Pf-Prioritylevel-Uid: 7df0c0a7-2f8a-40ca-9625-0b6635feccd2
 > milliseconds=2 getConnectionMilliseconds=0 serverProcessingMilliseconds=1
I1215 14:50:15.322454     230 type.go:165] "Response Body" body=<
	00000000  6b 38 73 00 0a 0f 0a 02  76 31 12 09 43 6f 6e 66  |k8s.....v1..Conf|
	00000010  69 67 4d 61 70 12 91 0b  0a ac 01 0a 0e 6b 75 62  |igMap........kub|
	00000020  65 6c 65 74 2d 63 6f 6e  66 69 67 12 00 1a 0b 6b  |elet-config....k|
	00000030  75 62 65 2d 73 79 73 74  65 6d 22 00 2a 24 30 32  |ube-system".*$02|
	00000040  36 61 32 32 32 63 2d 39  66 65 34 2d 34 35 39 37  |6a222c-9fe4-4597|
	00000050  2d 39 30 38 61 2d 62 35  65 35 64 62 33 35 32 35  |-908a-b5e5db3525|
	00000060  36 64 32 03 32 31 30 38  00 42 08 08 d7 da ff c9  |6d2.2108.B......|
	00000070  06 10 00 8a 01 51 0a 07  6b 75 62 65 61 64 6d 12  |.....Q..kubeadm.|
	00000080  06 55 70 64 61 74 65 1a  02 76 31 22 08 08 d7 da  |.Update..v1"....|
	00000090  ff c9 06 10 00 32 08 46  69 65 6c 64 73 56 31 3a  |.....2.FieldsV1:|
	000000a0  24 0a 22 7b 22 66 3a 64  61 74 61 22 3a 7b 22 2e  |$."{"f:data":{".|
	000000b0  22 3a 7b 7d 2c 22 66 3a  6b 75 62 65 6c 65 74 22  |":{},"f:kubelet"|
	000000c0  3a 7b 7d 7d 7d 42 00 12  df 09 0a 07 6b 75 62 65  |:{}}}B......kube|
	000000d0  6c 65 74 12 d3 09 61 70  69 56 65 72 73 69 6f 6e  |let...apiVersion|
	000000e0  3a 20 6b 75 62 65 6c 65  74 2e 63 6f 6e 66 69 67  |: kubelet.config|
	000000f0  2e 6b 38 73 2e 69 6f 2f  76 31 62 65 74 61 31 0a  |.k8s.io/v1beta1.|
	00000100  61 75 74 68 65 6e 74 69  63 61 74 69 6f 6e 3a 0a  |authentication:.|
	00000110  20 20 61 6e 6f 6e 79 6d  6f 75 73 3a 0a 20 20 20  |  anonymous:.   |
	00000120  20 65 6e 61 62 6c 65 64  3a 20 66 61 6c 73 65 0a  | enabled: false.|
	00000130  20 20 77 65 62 68 6f 6f  6b 3a 0a 20 20 20 20 63  |  webhook:.    c|
	00000140  61 63 68 65 54 54 4c 3a  20 30 73 0a 20 20 20 20  |acheTTL: 0s.    |
	00000150  65 6e 61 62 6c 65 64 3a  20 74 72 75 65 0a 20 20  |enabled: true.  |
	00000160  78 35 30 39 3a 0a 20 20  20 20 63 6c 69 65 6e 74  |x509:.    client|
	00000170  43 41 46 69 6c 65 3a 20  2f 65 74 63 2f 6b 75 62  |CAFile: /etc/kub|
	00000180  65 72 6e 65 74 65 73 2f  70 6b 69 2f 63 61 2e 63  |ernetes/pki/ca.c|
	00000190  72 74 0a 61 75 74 68 6f  72 69 7a 61 74 69 6f 6e  |rt.authorization|
	000001a0  3a 0a 20 20 6d 6f 64 65  3a 20 57 65 62 68 6f 6f  |:.  mode: Webhoo|
	000001b0  6b 0a 20 20 77 65 62 68  6f 6f 6b 3a 0a 20 20 20  |k.  webhook:.   |
	000001c0  20 63 61 63 68 65 41 75  74 68 6f 72 69 7a 65 64  | cacheAuthorized|
	000001d0  54 54 4c 3a 20 30 73 0a  20 20 20 20 63 61 63 68  |TTL: 0s.    cach|
	000001e0  65 55 6e 61 75 74 68 6f  72 69 7a 65 64 54 54 4c  |eUnauthorizedTTL|
	000001f0  3a 20 30 73 0a 63 67 72  6f 75 70 44 72 69 76 65  |: 0s.cgroupDrive|
	00000200  72 3a 20 73 79 73 74 65  6d 64 0a 63 67 72 6f 75  |r: systemd.cgrou|
	00000210  70 52 6f 6f 74 3a 20 2f  6b 75 62 65 6c 65 74 0a  |pRoot: /kubelet.|
	00000220  63 6c 75 73 74 65 72 44  4e 53 3a 0a 2d 20 31 30  |clusterDNS:.- 10|
	00000230  2e 39 36 2e 30 2e 31 30  0a 63 6c 75 73 74 65 72  |.96.0.10.cluster|
	00000240  44 6f 6d 61 69 6e 3a 20  63 6c 75 73 74 65 72 2e  |Domain: cluster.|
	00000250  6c 6f 63 61 6c 0a 63 6f  6e 74 61 69 6e 65 72 52  |local.containerR|
	00000260  75 6e 74 69 6d 65 45 6e  64 70 6f 69 6e 74 3a 20  |untimeEndpoint: |
	00000270  22 22 0a 63 70 75 4d 61  6e 61 67 65 72 52 65 63  |"".cpuManagerRec|
	00000280  6f 6e 63 69 6c 65 50 65  72 69 6f 64 3a 20 30 73  |oncilePeriod: 0s|
	00000290  0a 63 72 61 73 68 4c 6f  6f 70 42 61 63 6b 4f 66  |.crashLoopBackOf|
	000002a0  66 3a 20 7b 7d 0a 65 76  69 63 74 69 6f 6e 48 61  |f: {}.evictionHa|
	000002b0  72 64 3a 0a 20 20 69 6d  61 67 65 66 73 2e 61 76  |rd:.  imagefs.av|
	000002c0  61 69 6c 61 62 6c 65 3a  20 30 25 0a 20 20 6e 6f  |ailable: 0%.  no|
	000002d0  64 65 66 73 2e 61 76 61  69 6c 61 62 6c 65 3a 20  |defs.available: |
	000002e0  30 25 0a 20 20 6e 6f 64  65 66 73 2e 69 6e 6f 64  |0%.  nodefs.inod|
	000002f0  65 73 46 72 65 65 3a 20  30 25 0a 65 76 69 63 74  |esFree: 0%.evict|
	00000300  69 6f 6e 50 72 65 73 73  75 72 65 54 72 61 6e 73  |ionPressureTrans|
	00000310  69 74 69 6f 6e 50 65 72  69 6f 64 3a 20 30 73 0a  |itionPeriod: 0s.|
	00000320  66 61 69 6c 53 77 61 70  4f 6e 3a 20 66 61 6c 73  |failSwapOn: fals|
	00000330  65 0a 66 69 6c 65 43 68  65 63 6b 46 72 65 71 75  |e.fileCheckFrequ|
	00000340  65 6e 63 79 3a 20 30 73  0a 68 65 61 6c 74 68 7a  |ency: 0s.healthz|
	00000350  42 69 6e 64 41 64 64 72  65 73 73 3a 20 31 32 37  |BindAddress: 127|
	00000360  2e 30 2e 30 2e 31 0a 68  65 61 6c 74 68 7a 50 6f  |.0.0.1.healthzPo|
	00000370  72 74 3a 20 31 30 32 34  38 0a 68 74 74 70 43 68  |rt: 10248.httpCh|
	00000380  65 63 6b 46 72 65 71 75  65 6e 63 79 3a 20 30 73  |eckFrequency: 0s|
	00000390  0a 69 6d 61 67 65 47 43  48 69 67 68 54 68 72 65  |.imageGCHighThre|
	000003a0  73 68 6f 6c 64 50 65 72  63 65 6e 74 3a 20 31 30  |sholdPercent: 10|
	000003b0  30 0a 69 6d 61 67 65 4d  61 78 69 6d 75 6d 47 43  |0.imageMaximumGC|
	000003c0  41 67 65 3a 20 30 73 0a  69 6d 61 67 65 4d 69 6e  |Age: 0s.imageMin|
	000003d0  69 6d 75 6d 47 43 41 67  65 3a 20 30 73 0a 6b 69  |imumGCAge: 0s.ki|
	000003e0  6e 64 3a 20 4b 75 62 65  6c 65 74 43 6f 6e 66 69  |nd: KubeletConfi|
	000003f0  67 75 72 61 74 69 6f 6e  0a 6c 6f 67 67 69 6e 67  |guration.logging|
	00000400  3a 0a 20 20 66 6c 75 73  68 46 72 65 71 75 65 6e  |:.  flushFrequen|
	00000410  63 79 3a 20 30 0a 20 20  6f 70 74 69 6f 6e 73 3a  |cy: 0.  options:|
	00000420  0a 20 20 20 20 6a 73 6f  6e 3a 0a 20 20 20 20 20  |.    json:.     |
	00000430  20 69 6e 66 6f 42 75 66  66 65 72 53 69 7a 65 3a  | infoBufferSize:|
	00000440  20 22 30 22 0a 20 20 20  20 74 65 78 74 3a 0a 20  | "0".    text:. |
	00000450  20 20 20 20 20 69 6e 66  6f 42 75 66 66 65 72 53  |     infoBufferS|
	00000460  69 7a 65 3a 20 22 30 22  0a 20 20 76 65 72 62 6f  |ize: "0".  verbo|
	00000470  73 69 74 79 3a 20 30 0a  6d 65 6d 6f 72 79 53 77  |sity: 0.memorySw|
	00000480  61 70 3a 20 7b 7d 0a 6e  6f 64 65 53 74 61 74 75  |ap: {}.nodeStatu|
	00000490  73 52 65 70 6f 72 74 46  72 65 71 75 65 6e 63 79  |sReportFrequency|
	000004a0  3a 20 30 73 0a 6e 6f 64  65 53 74 61 74 75 73 55  |: 0s.nodeStatusU|
	000004b0  70 64 61 74 65 46 72 65  71 75 65 6e 63 79 3a 20  |pdateFrequency: |
	000004c0  30 73 0a 72 6f 74 61 74  65 43 65 72 74 69 66 69  |0s.rotateCertifi|
	000004d0  63 61 74 65 73 3a 20 74  72 75 65 0a 72 75 6e 74  |cates: true.runt|
	000004e0  69 6d 65 52 65 71 75 65  73 74 54 69 6d 65 6f 75  |imeRequestTimeou|
	000004f0  74 3a 20 30 73 0a 73 68  75 74 64 6f 77 6e 47 72  |t: 0s.shutdownGr|
	00000500  61 63 65 50 65 72 69 6f  64 3a 20 30 73 0a 73 68  |acePeriod: 0s.sh|
	00000510  75 74 64 6f 77 6e 47 72  61 63 65 50 65 72 69 6f  |utdownGracePerio|
	00000520  64 43 72 69 74 69 63 61  6c 50 6f 64 73 3a 20 30  |dCriticalPods: 0|
	00000530  73 0a 73 74 61 74 69 63  50 6f 64 50 61 74 68 3a  |s.staticPodPath:|
	00000540  20 2f 65 74 63 2f 6b 75  62 65 72 6e 65 74 65 73  | /etc/kubernetes|
	00000550  2f 6d 61 6e 69 66 65 73  74 73 0a 73 74 72 65 61  |/manifests.strea|
	00000560  6d 69 6e 67 43 6f 6e 6e  65 63 74 69 6f 6e 49 64  |mingConnectionId|
	00000570  6c 65 54 69 6d 65 6f 75  74 3a 20 30 73 0a 73 79  |leTimeout: 0s.sy|
	00000580  6e 63 46 72 65 71 75 65  6e 63 79 3a 20 30 73 0a  |ncFrequency: 0s.|
	00000590  76 6f 6c 75 6d 65 53 74  61 74 73 41 67 67 50 65  |volumeStatsAggPe|
	000005a0  72 69 6f 64 3a 20 30 73  0a 1a 00 22 00           |riod: 0s...".|
 >
I1215 14:50:15.324583     230 initconfiguration.go:114] skip CRI socket detection, fill with the default CRI socket unix:///var/run/containerd/containerd.sock
I1215 14:50:15.324815     230 interface.go:432] Looking for default routes with IPv4 addresses
I1215 14:50:15.324830     230 interface.go:437] Default route transits interface "eth0"
I1215 14:50:15.324975     230 interface.go:209] Interface eth0 is up
I1215 14:50:15.325021     230 interface.go:257] Interface "eth0" has 3 addresses :[172.18.0.3/16 fc00:****::3/64 fe80::****/64].
I1215 14:50:15.325041     230 interface.go:224] Checking addr  172.18.0.3/16.
I1215 14:50:15.325057     230 interface.go:231] IP found 172.18.0.3
I1215 14:50:15.325072     230 interface.go:263] Found valid IPv4 address 172.18.0.3 for interface "eth0".
I1215 14:50:15.325085     230 interface.go:443] Found active IP 172.18.0.3 
I1215 14:50:15.325133     230 common.go:148] WARNING: tolerating control plane version v1.34.0 as a pre-release version
I1215 14:50:15.325168     230 preflight.go:108] [preflight] Running configuration dependant checks
I1215 14:50:15.325179     230 controlplaneprepare.go:225] [download-certs] Skipping certs download
I1215 14:50:15.325485     230 kubelet.go:147] [kubelet-start] writing bootstrap kubelet config file at /etc/kubernetes/bootstrap-kubelet.conf
I1215 14:50:15.327017     230 kubelet.go:162] [kubelet-start] writing CA certificate at /etc/kubernetes/pki/ca.crt
I1215 14:50:15.327125     230 kubelet.go:178] [kubelet-start] Checking for an existing Node in the cluster with name "5a5adbbfec7a" and status "Ready"
I1215 14:50:15.327178     230 type.go:165] "Request Body" body=""
I1215 14:50:15.327263     230 round_trippers.go:527] "Request" curlCommand=<
	curl -v -XGET  -H "Accept: application/vnd.kubernetes.protobuf,application/json" -H "User-Agent: kubeadm/v1.35.0 (linux/amd64) kubernetes/f35f950" -H "Authorization: Bearer <masked>" 'https://kind-control-plane:6443/api/v1/nodes/5a5adbbfec7a?timeout=10s'
 >
I1215 14:50:15.329075     230 round_trippers.go:632] "Response" verb="GET" url="https://kind-control-plane:6443/api/v1/nodes/5a5adbbfec7a?timeout=10s" status="404 Not Found" headers=<
	Audit-Id: b5dddad2-5b97-4e03-9241-52b266d62df5
	Cache-Control: no-cache, private
	Content-Length: 115
	Content-Type: application/vnd.kubernetes.protobuf
	Date: Mon, 15 Dec 2025 14:50:15 GMT
	X-Kubernetes-Pf-Flowschema-Uid: f54c5d84-c2d8-4cbb-8801-07adb48ce73d
	X-Kubernetes-Pf-Prioritylevel-Uid: 7df0c0a7-2f8a-40ca-9625-0b6635feccd2
 > milliseconds=1 getConnectionMilliseconds=0 serverProcessingMilliseconds=1
I1215 14:50:15.329149     230 type.go:165] "Response Body" body=<
	00000000  6b 38 73 00 0a 0c 0a 02  76 31 12 06 53 74 61 74  |k8s.....v1..Stat|
	00000010  75 73 12 5b 0a 06 0a 00  12 00 1a 00 12 07 46 61  |us.[..........Fa|
	00000020  69 6c 75 72 65 1a 1e 6e  6f 64 65 73 20 22 35 61  |ilure..nodes "5a|
	00000030  35 61 64 62 62 66 65 63  37 61 22 20 6e 6f 74 20  |5adbbfec7a" not |
	00000040  66 6f 75 6e 64 22 08 4e  6f 74 46 6f 75 6e 64 2a  |found".NotFound*|
	00000050  1b 0a 0c 35 61 35 61 64  62 62 66 65 63 37 61 12  |...5a5adbbfec7a.|
	00000060  00 1a 05 6e 6f 64 65 73  28 00 32 00 30 94 03 1a  |...nodes(.2.0...|
	00000070  00 22 00                                          |.".|
 >
I1215 14:50:15.329251     230 kubelet.go:193] [kubelet-start] Stopping the kubelet
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/instance-config.yaml"
[patches] Applied patch of type "application/strategic-merge-patch+json" to target "kubeletconfiguration"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-check] Waiting for a healthy kubelet at http://127.0.0.1:10248/healthz. This can take up to 4m0s
[kubelet-check] The kubelet is healthy after 501.664771ms
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap
I1215 14:50:15.990092     230 loader.go:405] Config loaded from file:  /etc/kubernetes/kubelet.conf
I1215 14:50:15.990751     230 cert_rotation.go:141] "Starting client certificate rotation controller" logger="tls-transport-cache"
I1215 14:50:15.991227     230 loader.go:405] Config loaded from file:  /etc/kubernetes/kubelet.conf

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

So, at this point now.
We’ve answered our second question also.
i.e, can I try to create a kubeadm token manually and use that to join an existing Kubernetes cluster successfully?.
Yes, I can! We saw it above!

But wait, what next now?

We saw the following bit in our output:

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

I have more questions now.

What is the “Certificate signing request”? And what “response” we received?
The Kubelet was informed of the new secure connection details. How?

Let’s try to answer them now.

Q: What is the “Certificate signing request”? And what “response” we received?

So, back to the control-plane node, let’s check the following:

❯ kubectl get certificatesigningrequest -A

NAME        AGE     SIGNERNAME                                    REQUESTOR                        REQUESTEDDURATION   CONDITION
csr-c5j6z   7m46s   kubernetes.io/kube-apiserver-client-kubelet   system:bootstrap:pqrstu          <none>              Approved,Issued
csr-qzbvk   112m    kubernetes.io/kube-apiserver-client-kubelet   system:node:kind-control-plane   <none>              Approved,Issued

OK, so, we see two CertificateSigningRequest (CSR) objects.
One of them was requested by the system:node:kind-control-plane user (our kind-control-plane node).
And the other one was requested by us, through the user system:bootstrap:pqrstu from the docker container node (joining-node).
And both are in condition Approved and Issued.

Let’s also see the body of the CSR object created by our request.

❯ kubectl get certificatesigningrequest csr-c5j6z -o yaml

apiVersion: certificates.k8s.io/v1
kind: CertificateSigningRequest
metadata:
  ...
spec:
  groups:
  - system:bootstrappers
  - system:authenticated
  request: LS0tLS1CRUdJTi****LS0K
  signerName: kubernetes.io/kube-apiserver-client-kubelet
  usages:
  - digital signature
  - client auth
  username: system:bootstrap:pqrstu
status:
  certificate: LS0tLS1CRUdJTiBDR****LS0K
  conditions:
  - ...
    message: Auto approving kubelet client certificate after SubjectAccessReview.
    reason: AutoApproved
    status: "True"
    type: Approved

From the object definition, I understand that the user system:bootstrap:pqrstu, made this CSR request for purposes, usages: {digital signature, client auth}.
And that is AutoApproved (after some process called SubjectAccessReview which I am not exploring in this blog, but I know that is important).

Nice. But what did we receive as response?

I see, we got back a certificate and Approved (with status: "True") in the CSR object’s status section.
So, that means the response we got back is the CSR getting approved. (maybe, I’m still not 100% sure about which response we’re talking).

Q: The Kubelet was informed of the new secure connection details. How?

I do see some relevant logs (truncated to just show the relevant bits):

[kubelet-start] writing CA certificate at /etc/kubernetes/pki/ca.crt

[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"

Let’s check if we see anything new inside our docker container node we joined from (joining-node).

root@83ab08acf723:/# tree /etc/kubernetes/
/etc/kubernetes/
|-- kubelet.conf
|-- manifests
`-- pki
    `-- ca.crt

3 directories, 2 files

We do.
We now have a new directory called /etcd/kubernetes/ which contains a kubelet.conf file as well as pki/ca.crt.
(same as what the logs pointed us.)

Let’s see the contents of the kubelet.conf file:

root@0ecd1b55abc8:/# cat /etc/kubernetes/kubelet.conf 

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: LS0tLS1CRUd****LS0tLS0K
    server: https://kind-control-plane:6443
  name: default-cluster
contexts:
- context:
    cluster: default-cluster
    namespace: default
    user: default-auth
  name: default-context
current-context: default-context
kind: Config
users:
- name: default-auth
  user:
    client-certificate: /var/lib/kubelet/pki/kubelet-client-current.pem
    client-key: /var/lib/kubelet/pki/kubelet-client-current.pem

The certificate-authority-data field stores the public certificate of Cluster’s CA (Which is coming from the control-plane of the cluster).
If you decode the value (using echo "LS0tLS1CRUd****LS0tLS0K" | base64 -d, it will match the contents of the /etc/kubernetes/pki/ca.crt on the kind-control-plane node.

We also see the kubelet.conf file points to the location of the kubelet’s certificates & keys - /var/lib/kubelet/pki/.

As I understand (from reading docs):

The kubelet generates its own certificates/keys locally (/var/lib/kubelet/pki/Kubelet.crt | .key) and then send a CSR (Certificate Signing Request) to the API Server.
The control-plane (controller-manager?) then sign that CSR using the cluster CA keys (which we see from the field - certificate-authority-data).
And then signed certificate is written into kubelet.conf (to be precise here - /var/lib/kubelet/pki/kubelet-client-current.pem, which comes from the status.certificate: field of the CSR Object).

The full tree of /var/lib/kubelet/pki/ looks like:

root@fd5e81a7604b:/# tree /var/lib/kubelet/pki/
/var/lib/kubelet/pki/
|-- kubelet-client-2025-12-16-07-35-31.pem
|-- kubelet-client-current.pem -> /var/lib/kubelet/pki/kubelet-client-2025-12-16-07-35-31.pem
|-- kubelet.crt
`-- kubelet.key

1 directory, 4 files

(Note: The part, “kubelet generate a private key and a CSR for submission to a cluster-level certificate signing process” was originally proposed as part of this design proposal - Kubelet TLS bootstrap)

So, after this point onwards, our (joining-node) node aka the Kubelet has its own set of signed certificates.
And so, the kubeadm bearer token is no longer required.
And all further interactions with the control-plane will use these certificates via mTLS (mutual TLS).

Also, note that - not just certificates, a lot more, was also added to the filesystem of the (joining-node) node.
I don’t understand details about every single listed item, but here’s the full tree:

root@0ecd1b55abc8:/# tree /var/lib/kubelet/
/var/lib/kubelet/
|-- allocated_pods_state
|-- checkpoints
|-- config.yaml
|-- cpu_manager_state
|-- device-plugins
|   `-- kubelet.sock
|-- dra_manager_state
|-- instance-config.yaml
|-- kubeadm-flags.env
|-- memory_manager_state
|-- pki
|   |-- kubelet-client-2025-12-16-10-18-54.pem
|   |-- kubelet-client-current.pem -> /var/lib/kubelet/pki/kubelet-client-2025-12-16-10-18-54.pem
|   |-- kubelet.crt
|   `-- kubelet.key
|-- plugins
|-- plugins_registry
|-- pod-resources
|   `-- kubelet.sock
`-- pods
    |-- 4683e97e-5735-459f-b965-24a5ffd2f63e
    |   |-- plugins
    |   |   `-- kubernetes.io~empty-dir
    |   |       `-- wrapped_kube-api-access-7v75d
    |   |           `-- ready
    |   `-- volumes
    |       `-- kubernetes.io~projected
    |           `-- kube-api-access-7v75d
    |               |-- ca.crt -> ..data/ca.crt
    |               |-- namespace -> ..data/namespace
    |               `-- token -> ..data/token
    `-- d6d5a0a6-4c4a-4cc0-984f-a63129fbefca
        |-- plugins
        |   `-- kubernetes.io~empty-dir
        |       |-- wrapped_kube-api-access-c4l5m
        |       |   `-- ready
        |       `-- wrapped_kube-proxy
        |           `-- ready
        `-- volumes
            |-- kubernetes.io~configmap
            |   `-- kube-proxy
            |       |-- config.conf -> ..data/config.conf
            |       `-- kubeconfig.conf -> ..data/kubeconfig.conf
            `-- kubernetes.io~projected
                `-- kube-api-access-c4l5m
                    |-- ca.crt -> ..data/ca.crt
                    |-- namespace -> ..data/namespace
                    `-- token -> ..data/token

25 directories, 24 files

Q: why there’s a symmetric token used by kubeadm?

OK, I already feel I learnt quite a bit.

Yet, our very first question is still not answered.
i.e., why there’s a symmetric token used by kubeadm?

I was actually having a chat with the creator of Kubeadm himself, Lucas Käldström (yes, the same person who wrote the above linked thesis).

What I learnt is - even though it’s a symmetric and shared string, the token itself has two parts, where the first part (token-id) is supposed to be treated as public entity and the second part (token-secret) to be treated as a private entity.

Becuase, Kubeadm tokens are to be used for establishing bidirectional trust between the client (in our case, joining-node) and the server (the control-plane, api-server).

For the client (joining-node) to establish trust to the server (the control-plane, api-server), we saw the first part (token-id) of the token is used (system:bootstrap:pqrstu and the matching secret and clusterrolebinding).
For the server (the control-plane, api-server) to establish trust to the client (joining-node), the entire shared token (both token-id and token-secret) can be used.

If you look at the full kubeadm join ... logs again, one of the early steps you will see is - attempting to download the KubeletConfiguration from ConfigMap "kubelet-config".

This token (stored as the Secret object in the control-plane aka the server side) can be used to sign the ConfigMap (“kubelet-config”).

And then on the client side (joining-node), the received signed ConfigMap can then be authenticated by using the same shared token.

The process is explained here briefly - ConfigMap Signing,
but the important part is - You can verify the JWS (signature) using the HS256 scheme (HMAC-SHA256) with the full token (e.g. 07401b.f395accd246ae52d) as the shared secret.

So, even though it’s a symmetric and a shared token, but it has similarities to the assymetric key pairs with its public and private entities used for separate purposes.

There’s also this design proposal called, bootstrap discovery, which discusses and infact proposed the flow we see in our logs.

Finally, I will end this post with this diagram which atleast for me, summarises nicely, the entire process we followed from start to end so far:

New node
  │
  │ kubeadm join
  │
  ▼
API Server (unauthenticated)
  │
  │ token-based auth
  ▼
CSR created
  │
  │ CertificateSigningRequest
  ▼
Controller approves CSR
  │
  │ signed by CA
  ▼
kubelet gets client cert
  │
  │ mTLS from now on
  ▼
FULLY TRUSTED NODE

With that, thank you for reading so far.

And I hope you also got to learn a few new things. o/

PS:

Even though the node was able to join the control-plane, but it wasn’t really in a READY state (and I left it there, didn’t troubleshoot it further).

The Kubelet logs read something like:

failed to mount rootfs component: mount source "overlay" ... err: invalid argument

What happens when Kind doesn't have enough IP(s)?

Sat, 06 Dec 2025 00:00:00 +0000

I wanted to write a quick blog to document a tiny experiment I ran last week.
Just dumping my rough notes as it is.

What I want to test is a scenario of creating a (Kind) cluster when it doesn’t have enough IP addresses, to assign internally etc.
Meaning I try to create a Kind cluster and give it only a docker bridge network with 20 or 50 IP addresses (basically a very tiny pool of IP(s)).

Actually, now that I think more, 20-50 IP(s) are actually too much for my experiment.
Because the docker bridge IP pool will only be used for assigning IP(s) to the kind nodes (the control-plane and worker nodes).
Inside these kind nodes - for the Pods and for the Containers IP(s), the node will configure its own pool of private IP addresses, so that doesn’t come from the docker bridge network from the host.

Therefore, the flow is roughly like:

(my host network) sets aside a little set of private IP for the docker bridge ->
(then docker bridge network) assigns an IP to a node ->
(node internal network) which assigns IP(s) to pods and containers

With this understanding now, I feel I should create a docker bridge network with even less IP(s), lets say 5 IP(s).
And with this newly created docker bridge network, if I try to create a Kind cluster with 5 or more nodes, atleast 1 node will never get an IP.
And that is exactly the behavior I want to test.

I also know out of these 5 IP(s) -

one will be used as gateway, so that is gone,
and then the rest 4 will be assigned to 4 nodes

(Note: this above understanding is incomplete right now, will be fixed later in the post.)

So, with that mathematics done now, let’s run the experiment.

First off, create a Kind cluster:

❯ cat kind-config.yaml 
# kind-config.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
networking:
  disableDefaultCNI: true
nodes:
  - role: control-plane
  - role: worker
  - role: worker
  - role: worker
  - role: worker


❯ kind create cluster --name ip-test --retain --config kind-config.yaml
Creating cluster "ip-test" ...
 ✓ Ensuring node image (kindest/node:v1.34.0) 🖼
 ✓ Preparing nodes 📦 📦 📦 📦 📦  
 ✓ Writing configuration 📜 
 ✓ Starting control-plane 🕹️ 
 ✓ Installing StorageClass 💾 
 ✓ Joining worker nodes 🚜 
Set kubectl context to "kind-ip-test"
You can now use your cluster with:

kubectl cluster-info --context kind-ip-test

Have a nice day! 👋

Number one

To create a new docker bridge network, I need to run the following command:

❯ docker network create --driver bridge  \
    --subnet 172.20.0.0/29  \
    --gateway 172.20.0.1 \
    --aux-address "reserved1=172.20.0.6" \
    kind-small-net

Notice the flag I passed, --subnet 172.20.0.0/29.
This translates to a private IP network 172.20.0.0 with a subnet mask of 255.255.255.248/29.
This subnet mask will give me exactly 8 IP addresses.
(and that’s the closest I can get to making an IP pool with exact 5 usable IP(s)).

So, how these 8 IP(s) will be used, is explained below:

172.20.0.0 = network (not usable)
172.20.0.1 → commonly used as gateway (Docker sets a gateway)
172.20.0.2 – 172.20.0..6 = usable host addresses (that’s 5 addresses here)
172.20.0.7 = broadcast (not usable)

Also notice, that I restricted one of the available 5 IP addresses using the flag, --aux-address "reserved1=172.20.0.6" to create an IP contrained scenario.

Number two

Kind always create a default docker network bridge with name “kind” automatically.

So, I can try to create a new docker network bridge (kind-small-net) with constrained IP pool like we did above.
And then try to connect existing Kind cluster nodes to this newly created bridge network.

Like following:

❯ for c in $(docker ps --filter "name=ip-test" -q); do   docker network connect kind-small-net $c; done
Error response from daemon: no available IPv4 addresses on this network's address pools: kind-small-net (15f712efffba69730048b9f826e3e68702646a37df4d09c86288fca47a5a52f6)

Notice, I got an error - no available IPv4 addresses on this network's address pools.
So far, everything is going as expected.

Now, let’s check if anything happened to the cluster:

❯ kubectl get nodes -o wide
NAME                    STATUS     ROLES           AGE     VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                         KERNEL-VERSION     CONTAINER-RUNTIME
ip-test-control-plane   NotReady   control-plane   9m4s    v1.34.0   172.18.0.6    <none>        Debian GNU/Linux 12 (bookworm)   6.17.9-1-default   containerd://2.1.3
ip-test-worker          NotReady   <none>          8m48s   v1.34.0   172.18.0.2    <none>        Debian GNU/Linux 12 (bookworm)   6.17.9-1-default   containerd://2.1.3
ip-test-worker2         NotReady   <none>          8m49s   v1.34.0   172.18.0.4    <none>        Debian GNU/Linux 12 (bookworm)   6.17.9-1-default   containerd://2.1.3
ip-test-worker3         NotReady   <none>          8m49s   v1.34.0   172.18.0.3    <none>        Debian GNU/Linux 12 (bookworm)   6.17.9-1-default   containerd://2.1.3
ip-test-worker4         NotReady   <none>          8m49s   v1.34.0   172.18.0.5    <none>        Debian GNU/Linux 12 (bookworm)   6.17.9-1-default   containerd://2.1.3


❯ docker container ps -a
CONTAINER ID   IMAGE                  COMMAND                  CREATED          STATUS          PORTS                       NAMES
aca02654b39a   kindest/node:v1.34.0   "/usr/local/bin/entr…"   13 minutes ago   Up 13 minutes                               ip-test-worker
a7da65a5a3f7   kindest/node:v1.34.0   "/usr/local/bin/entr…"   13 minutes ago   Up 13 minutes                               ip-test-worker4
4605b3acc669   kindest/node:v1.34.0   "/usr/local/bin/entr…"   13 minutes ago   Up 13 minutes                               ip-test-worker3
856fbf8aec0d   kindest/node:v1.34.0   "/usr/local/bin/entr…"   13 minutes ago   Up 13 minutes   127.0.0.1:46031->6443/tcp   ip-test-control-plane
fcb796be89c8   kindest/node:v1.34.0   "/usr/local/bin/entr…"   13 minutes ago   Up 13 minutes                               ip-test-worker2

NO! All nodes still has an IP assigned to them.
And none of these assigned IP(s) are from our newly created network bridge (atleast not in the kubectl output).
(And yes, I also see the NotReady status, so that is something.)

What’s going on? Let’s inspect both the network bridges:

❯ docker network inspect kind

        "Name": "kind",
         ...
        "Scope": "local",
        "Driver": "bridge",
         ...
        "IPAM": {
            ...
            "Config": [
                {
                    "Subnet": "172.18.0.0/16",
                    "Gateway": "172.18.0.1"
                },
            ]
        },
        ...
        "Containers": {
            "4605b3acc6699173bd975a3d6b74d25e688eecfce644962bd7fb26c50d42f890": {
                "Name": "ip-test-worker3",
                ...
                "IPv4Address": "172.18.0.3/16",
            },
            "856fbf8aec0d287fea50ce9255260a22a1f707e12ea34aac313e3b356ffc3d8d": {
                "Name": "ip-test-control-plane",
                ...
                "IPv4Address": "172.18.0.6/16",
            },
            "a7da65a5a3f7dc78947e33d8f797854c47630adbee68ea30a46590a5238862ac": {
                "Name": "ip-test-worker4",
                ...
                "IPv4Address": "172.18.0.5/16",
            },
            "aca02654b39a931e9d27e313f61a78719eb389cb008e86f078c184fac2bae4e7": {
                "Name": "ip-test-worker",
                ...
                "IPv4Address": "172.18.0.2/16",
            },
            "fcb796be89c800e6e6107026ab04f448b9efa889957c10253f181bd63fa88075": {
                "Name": "ip-test-worker2",
                ...
                "IPv4Address": "172.18.0.4/16",
            }
        },

and

❯ docker network inspect kind-small-net

        "Name": "kind-small-net",
         ...
        "Scope": "local",
        "Driver": "bridge",
         ...
        "IPAM": {
            ...
            "Config": [
                {
                    "Subnet": "172.20.0.0/29",
                    "Gateway": "172.20.0.1",
                    "AuxiliaryAddresses": {
                        "reserved1": "172.20.0.6"
                    }
                }
            ]
        },
        ...
        "Containers": {
            "4605b3acc6699173bd975a3d6b74d25e688eecfce644962bd7fb26c50d42f890": {
                "Name": "ip-test-worker3",
                ...
                "IPv4Address": "172.20.0.4/29",
            },
            "856fbf8aec0d287fea50ce9255260a22a1f707e12ea34aac313e3b356ffc3d8d": {
                "Name": "ip-test-control-plane",
                ...
                "IPv4Address": "172.20.0.5/29",
            },
            "a7da65a5a3f7dc78947e33d8f797854c47630adbee68ea30a46590a5238862ac": {
                "Name": "ip-test-worker4",
                ...
                "IPv4Address": "172.20.0.3/29",
            },
            "aca02654b39a931e9d27e313f61a78719eb389cb008e86f078c184fac2bae4e7": {
                "Name": "ip-test-worker",
                ...
                "IPv4Address": "172.20.0.2/29",
            }
        },

OK, so, 4 out of the 5 nodes (1 control-plane + 3 workers) are assigned an IP from the new kind-small-net network.

But the entire set (1 control-plane + 4 workers) are still assigned an IP from the default Kind network.

Let’s try one more thing.
Just like the docker network connect command, there’s also a command to disconnect containers from a network as well.
Let’s run that as well and see if that makes the Kind cluster nodes fall back to the new kind-small-net network IP(s).

❯ for c in $(docker ps --filter "name=ip-test" -q); do   docker network disconnect kind $c; done

❯ docker network inspect kind

        "Name": "kind",
        ...
        "Scope": "local",
        "Driver": "bridge",
        ...
        "IPAM": {
            ...
            "Config": [
                {
                    "Subnet": "172.18.0.0/16",
                    "Gateway": "172.18.0.1"
                },
                ...
            ]
        },
        ...
        "Containers": {},
        ...


❯ kubectl get nodes -o wide
NAME                    STATUS     ROLES           AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                         KERNEL-VERSION     CONTAINER-RUNTIME
ip-test-control-plane   NotReady   control-plane   28m   v1.34.0   172.18.0.6    <none>        Debian GNU/Linux 12 (bookworm)   6.17.9-1-default   containerd://2.1.3
ip-test-worker          NotReady   <none>          28m   v1.34.0   172.18.0.2    <none>        Debian GNU/Linux 12 (bookworm)   6.17.9-1-default   containerd://2.1.3
ip-test-worker2         NotReady   <none>          28m   v1.34.0   172.18.0.4    <none>        Debian GNU/Linux 12 (bookworm)   6.17.9-1-default   containerd://2.1.3
ip-test-worker3         NotReady   <none>          28m   v1.34.0   172.18.0.3    <none>        Debian GNU/Linux 12 (bookworm)   6.17.9-1-default   containerd://2.1.3
ip-test-worker4         NotReady   <none>          28m   v1.34.0   172.18.0.5    <none>        Debian GNU/Linux 12 (bookworm)   6.17.9-1-default   containerd://2.1.3

❯ for c in $(docker ps --filter "name=ip-test" -q); do   docker network disconnect kind $c; done
Error response from daemon: container aca02654b39a931e9d27e313f61a78719eb389cb008e86f078c184fac2bae4e7 is not connected to network kind
Error response from daemon: container a7da65a5a3f7dc78947e33d8f797854c47630adbee68ea30a46590a5238862ac is not connected to network kind
Error response from daemon: container 4605b3acc6699173bd975a3d6b74d25e688eecfce644962bd7fb26c50d42f890 is not connected to network kind
Error response from daemon: container 856fbf8aec0d287fea50ce9255260a22a1f707e12ea34aac313e3b356ffc3d8d is not connected to network kind
Error response from daemon: container fcb796be89c800e6e6107026ab04f448b9efa889957c10253f181bd63fa88075 is not connected to network kind

Ok! So, all Kind nodes are indeed disconnected from the default “Kind” network now.
But still, they have an IP assigned from the old “Kind” network only (i.e, it didn’t fall back to the newly created bridge “kind-small-net”).

So, looks like Kind only look for the default automatically created docker bridge network (“Kind”) for its cluster configuration.

And therefore, regardless of me creating a new docker network bridge and attaching existing Kind node containers to that, Kind will always assign IP(s) from this default network bridge to the kind nodes.
And so, no IP exhaustion scenaio will happen.

Number three

If I still want the Kind cluster to use a custom IP pool, the way to do that is:

delete the existing docker network bridge, with name “Kind” (if one is existing), and
recreate a new one manually, with the same name “Kind”, with the required custom tiny IP pool I need.

Like following:

❯ docker network rm kind
kind

❯ docker network inspect kind
[]
Error response from daemon: network kind not found


❯ docker network create --driver bridge  \
    --subnet 172.20.0.0/29  \
    --gateway 172.20.0.1 \
    --aux-address "reserved1=172.20.0.6" \
    kind

f64a9d47cf585e9e61c3d25da2b3d3684f02b633b53ee7c053a60c2da0eafd84

❯ docker network inspect kind

        "Name": "kind",
        "Id": "f64a9d47cf585e9e61c3d25da2b3d3684f02b633b53ee7c053a60c2da0eafd84",
        "Created": "2025-12-16T18:35:37.818562813+05:30",
        "Scope": "local",
        "Driver": "bridge",
        ...
        "IPAM": {
            ...
            "Config": [
                {
                    "Subnet": "172.20.0.0/29",
                    "Gateway": "172.20.0.1",
                    "AuxiliaryAddresses": {
                        "reserved1": "172.20.0.6"
                    }
                }
            ]
        },
        ...
        "ConfigOnly": false,
        "Containers": {},
        ...

And with the required IP constrained docker bridge network with name “kind” in place, let’s create the Kind cluster as following:

❯ kind create cluster --name ip-test --retain --config kind-config.yaml
Creating cluster "ip-test" ...
 ✓ Ensuring node image (kindest/node:v1.34.0) 🖼
 ✗ Preparing nodes 📦 📦 📦 📦 📦  
ERROR: failed to create cluster: command "docker run --name ip-test-worker2 --hostname ip-test-worker2 --label io.x-k8s.kind.role=worker --privileged --security-opt seccomp=unconfined --security-opt apparmor=unconfined --tmpfs /tmp --tmpfs /run --volume /var --volume /lib/modules:/lib/modules:ro -e KIND_EXPERIMENTAL_CONTAINERD_SNAPSHOTTER --detach --tty --label io.x-k8s.kind.cluster=ip-test --net kind --restart=on-failure:1 --init=false --cgroupns=private --volume /dev/mapper:/dev/mapper kindest/node:v1.34.0@sha256:7416a61b42b1662ca6ca89f02028ac133a309a2a30ba309614e8ec94d976dc5a" failed with error: exit status 125
Command Output: a14eea4647334a84e95142893a735d1cf97bcb74def2793de4a6653c6f187cc9
docker: Error response from daemon: failed to set up container networking: no available IPv4 addresses on this network's address pools: kind (f64a9d47cf585e9e61c3d25da2b3d3684f02b633b53ee7c053a60c2da0eafd84)

Run 'docker run --help' for more information

OK, we managed to get the scenario working.
This time, the cluster failed at the bootstrap time only, with the expected error:

failed to set up container networking: no available IPv4 addresses on this network's address pools: kind

And once again, docker network inspect also shows that node ip-test-worker2 was the one which failed to get an IP from the pool.
Plus, the cluster is not responding.

❯ docker network inspect kind

        "Name": "kind",
        "Id": "f64a9d47cf585e9e61c3d25da2b3d3684f02b633b53ee7c053a60c2da0eafd84",
        "Created": "2025-12-16T18:35:37.818562813+05:30",
        "Scope": "local",
        "Driver": "bridge",
        ...
        "IPAM": {
            ...
            "Config": [
                {
                    "Subnet": "172.20.0.0/29",
                    "Gateway": "172.20.0.1",
                    "AuxiliaryAddresses": {
                        "reserved1": "172.20.0.6"
                    }
                }
            ]
        },
        ...
        "Containers": {
            "2230663632be9887858ac1037b1f01ec856122bf5ab02e6acf9188c6bfb12b32": {
                "Name": "ip-test-worker3",
                ...
                "IPv4Address": "172.20.0.3/29",
            },
            "6be8bb30074055a2049ccb50d066f0b1cd9cf62243f3d3c619b16c0e555dcf80": {
                "Name": "ip-test-control-plane",
                ...
                "IPv4Address": "172.20.0.4/29",
            },
            "7c545d00e21bbea07ed9a22226c80753acf09d3ed574914e711a8ddc67847013": {
                "Name": "ip-test-worker4",
                ...
                "IPv4Address": "172.20.0.2/29",
            },
            "f7b35fb4319944ba1ef7fb426e1708bac1bad0601b29d6cd1f43a2a4acb41233": {
                "Name": "ip-test-worker",
                ...
                "IPv4Address": "172.20.0.5/29",
            }

❯ kubectl get nodes
E1216 18:40:11.021156  144524 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"http://localhost:8080/api?timeout=32s\": dial tcp [::1]:8080: connect: connection refused"
The connection to the server localhost:8080 was refused - did you specify the right host or port?

Number four

When I ran the “kind create cluster” command above, I learnt there’s a flag called --retain that will retain the nodes (the respective docker containers) even if cluster bootstrap fails, for debugging purposes.

❯ docker container ps -a
CONTAINER ID   IMAGE                  COMMAND                  CREATED         STATUS         PORTS                       NAMES
a14eea464733   kindest/node:v1.34.0   "/usr/local/bin/entr…"   6 minutes ago   Created                                    ip-test-worker2
2230663632be   kindest/node:v1.34.0   "/usr/local/bin/entr…"   6 minutes ago   Up 6 minutes                               ip-test-worker3
6be8bb300740   kindest/node:v1.34.0   "/usr/local/bin/entr…"   6 minutes ago   Up 6 minutes   127.0.0.1:38589->6443/tcp   ip-test-control-plane
7c545d00e21b   kindest/node:v1.34.0   "/usr/local/bin/entr…"   6 minutes ago   Up 6 minutes                               ip-test-worker4
f7b35fb43199   kindest/node:v1.34.0   "/usr/local/bin/entr…"   6 minutes ago   Up 6 minutes                               ip-test-worker

❯ docker exec -it ip-test-control-plane /bin/bash
root@ip-test-control-plane:/# exit

❯ for c in $(docker ps -a --filter "name=ip-test" -q); do   docker inspect --format ' ' "$c"; done
/ip-test-worker2 
/ip-test-worker3 172.20.0.3
/ip-test-control-plane 172.20.0.4
/ip-test-worker4 172.20.0.2
/ip-test-worker 172.20.0.5

We can see that all containers have an IP assigned to them from our new custom “kind” network bridge, but not the ip-test-worker2.

Number five

Ok, let’s finish it with fixing our cluster.

Let’s recreate the “kind” network bridge.
It is going to be a custom network even now, but let’s remove the restriction for that last available and usable IP address.

❯ docker network create --driver bridge      --subnet 172.20.0.0/29      --gateway 172.20.0.1    kind
0272f4e1a9d33eaf31a77a5aec2dece1cf95098345fb1b0bbbcb825901af0c2b

❯ kind create cluster --name ip-test --retain --config kind-config.yaml
Creating cluster "ip-test" ...
 ✓ Ensuring node image (kindest/node:v1.34.0) 🖼
 ✓ Preparing nodes 📦 📦 📦 📦 📦  
 ✓ Writing configuration 📜 
 ✓ Starting control-plane 🕹️ 
 ✓ Installing StorageClass 💾 
 ✓ Joining worker nodes 🚜 
Set kubectl context to "kind-ip-test"
You can now use your cluster with:

kubectl cluster-info --context kind-ip-test

Have a nice day! 👋

❯ kubectl get nodes -o wide
NAME                    STATUS     ROLES           AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                         KERNEL-VERSION     CONTAINER-RUNTIME
ip-test-control-plane   NotReady   control-plane   41s   v1.34.0   172.20.0.6    <none>        Debian GNU/Linux 12 (bookworm)   6.17.9-1-default   containerd://2.1.3
ip-test-worker          NotReady   <none>          26s   v1.34.0   172.20.0.5    <none>        Debian GNU/Linux 12 (bookworm)   6.17.9-1-default   containerd://2.1.3
ip-test-worker2         NotReady   <none>          26s   v1.34.0   172.20.0.3    <none>        Debian GNU/Linux 12 (bookworm)   6.17.9-1-default   containerd://2.1.3
ip-test-worker3         NotReady   <none>          26s   v1.34.0   172.20.0.2    <none>        Debian GNU/Linux 12 (bookworm)   6.17.9-1-default   containerd://2.1.3
ip-test-worker4         NotReady   <none>          26s   v1.34.0   172.20.0.4    <none>        Debian GNU/Linux 12 (bookworm)   6.17.9-1-default   containerd://2.1.3

Done! We have the nodes created with IP addresses assigned from the new custom “kind” network pool.

I know the state of the nodes are NotReady, but that’s not part of this experiment.
(updated later - I know the reason why all nodes stayed in NotReady state. Because I only used a kind-config that disabled default CNI setup 🤦‍♀️. Anyway, removing this should fix it.)

networking:
  disableDefaultCNI: true

Next, I want to try is to see what happens when I constrain “PodCIDR” and “ServiceCIDR” pools. o/

Rescue OpenSUSE Tumbleweed (recreate Grub config from Rescue System)

Sun, 02 Nov 2025 00:00:00 +0000

In last 6 months time, twice I had to get my work machine’s system board (the motherboard) replaced.

First for an “Integrated Graphics Error”. One day I got these very annoying beeps on my work machine, I ran a Lenovo Smartbeep scan using their mobile app, and it suggested to contact (immediately) Lenovo support and request for a system board replacement.

Second time, the Wi-Fi (infact all wireless) stopped working on my machine.

For a few weeks following the first system board replacement, I thought it was some wifi firmware mismatch issue on my OpenSUSE Tumbleweed (TW) machine.
Because TW is a rolling release, once in a while distribution upgrade breaks stuff, so its a normal thing.
But I remember for the first few weeks after the hardware replacement, the Wi-Fi worked sometimes.
The Network manager will detect it but then soon after, it started to drop entirely. And I would get “Wifi adapter not found”.
And then from last ~1.5 months, I have been relying entirely on an Ethernet for Internet on my work machine.
And that won’t work when I’m travelling (I know, I can get an external dongle or something, but still).

So, I tried booting with a live USB stick into a Mint Cinnamon machine, and it was clear, it’s not a TW issue. Mint also didn’t detect any wireless network - zero, nil, nothing.
(Not to say, over last months, when I thought it was a firmware issue, I had tried many things, lots around the iwlwifi firmware, but nothing worked. I have been eying many recent upstream kernel bugzillas related to iwlwifi and I was convinced it was a firmwae issue. And it wasn’t).

Now, very fortunately, Lenovo Premium Support just works.
(for me it did! Twice I contacted them in last 6 months, and both times an engineer visited almost on the next day or in two, basically as soon as they had the replacement component delivered to them.)
Both times, they replaced the mother board.
(My work machine is a ThinkPad Workstation and every thing is just stuck on the mother board, so any tiny chip dies and it requires a full system board replacement).

Both times when the mother board was replaced, it’s almost a new machine, only with the same old storage.
(Very very important storage. Because it still contains my old TW OS partitions and data, and all the precious system configurations which takes a very long time to configure again).
I did run backups before both replacements, but still it’s a pain if I have to do a fresh OS reinstallation and setup everything again, in the middle of a work week.

So, when the system board is replaced, I think it refreshes the BIOS, and my grub menu no longer sees the TW OS partitions and so it just directly boots into the mighty Windows Blue Screen screaming “the system can’t be fixed, and I need to do a fresh install”.

But don’t get fooled by that (not immediately, check once).
Chances are that the old OS partitions are still there, just not being detected by the Grub Bootloader.
And that was the case for me (both times).

And not to my surprise, the OpenSUSE TW “Rescue System” menu came to my resuce!
(well, to my surprise! Because let’s not forget, TW is a rolling release OS. So things can go south very very quickly.)

I did the following:

I created a live USB stick with OpenSUSE Tumbleweed.
(It helped to have a stick with a full Offline image, and not the tiny Network image which will pull every single thing from Internet.
Because remember “the Wi-FI not working on my machine”.
Well I could have connected to Ethernet but still, the lesson is to have a proper stick ready with an offline image so it should just boot.)

Now, put it in the machine, go to the “Boot Menu” (F10, IIRC), and pick the option to boot from the live USB stick.
It will go to a grub menu.
Skip all the immediate menu options like “OpenSUSE Tumbleweed installation”, etc.
Go to “More …” and then “Rescue System”.
It will do the usual “loading basic drivers > hardware detection > ask to pick a keyboard layout, et. al” and then give me the “Resuce Login:” prompt.
Remember the username is “root” and there is no password.
With that, I enter tty1:resuce:/ #.

Now run the following set of commands:

# first things first, check my disk and partitions
## (if they still exists, I move on. Otherwise, all is gone and nothing more to do)

fdisk -l
## gives me something like following (truncated ofcourse, to the important bits)
Disk /dev/nvme0n1: xx.xx GiB, xxxxxx bytes, xxxxxx sectors
Disk model: xxx PC xxxx xxxxx-xxxx-xxxx          
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: xxx
Disk identifier: xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxxxx

Device           Start   End     Sectors   Size       Type
/dev/nvme0n1p1   xxx     xxxx    xxxxx     260M       EFI System
/dev/nvme0n1p2   xxxx    xxxxx   xxxx      xxx.xxG    Linux filesystem
/dev/nvme0n1p3   xxxxx   xxxxx   xxxx      2G Linux   swap

# in my case, the disk that has the OpenSUSE tumbleweed is "/dev/nvme0n1".
# "/dev/nvme0n1p1" is the EFI partition
# "/dev/nvme0n1p2" is the root partition
# "/dev/nvme0n1p3" is the Swap partition
  
# From this step onwards:
# I need "/dev/nvme0n1p1" (EFI System) and "/dev/nvme0n1p2" (Linux Filesystem)
  
# I need to mount these two partitions under "/mnt"
# (make sure the `/mnt` directory is empty before mounting anything to it)

cd /mnt
ls  # should be empty
cd ..

mount /dev/nvme0n1p2 /mnt
mount /dev/nvme0n1p1 /mnt/boot/efi
  
# next I need to mount "/dev", "/proc", "/sys", "/run" from the live environment into the mount directory

mount -B /dev /mnt/dev
mount -B /proc /mnt/proc
mount -B /sys /mnt/sys
mount -B /run /mnt/run

# now chroot into the "/mnt" directory
# the prompt will turn into `resuce:/ #` from the earlier `tty1:resuce:/ #`

chroot /mnt   

# now make the EFI variables available

mount -t efivarfs none /sys/firmware/efi/efivars

# now reinstall grub2, with `grub2-install`

grub2-install --target=x86_64-efi --efi-directory=/boot/efi --bootloader-id=opensuse
## should output something like:
Installing for x86_64-efi platform.
Installation finished. No error reported.

# then probe for other operating systems on this machine
  
os-prober
## should output something like (and because my machine originally came with Windows, it still shows remnants of that)
/dev/nvme0n1p1@/EFI/Microsoft/Boot/bootmgfw.efi:Windows Boot Manager:Windows:efi

# now, create a new grub configuration file using grub2-mkconfig

grub2-mkconfig -o /boot/grub2/grub.cfg
## should output something like:
Generating grub configuraiton file ...
Found theme: /boot/grub2/themes/opneSUSE/theme.txt
Found linux image: /boot/vmlinuz-x.xx.x-x-default
Found initrd image: /boot/initrd-x.xx.x-x-default
Warning: os-prober will be executed to detect other bootable partitions.
Its output will be used to detect bootable binaries on them and create new boot entries.
Found Windows Boot manager on /dev/nvme0n1p1@/EFI/Microsoft/Boot/bootmgfw.efi
Adding boot menu entry for UEFI Firmware Settings ...
done

# If all good so far, exit out of chroot

exit

# Reboot the machine.
# And remove the installation media

reboot

As the system reboots (remember I have removed the installation media at this point), I go to the BIOS to confirm the boot order.
Now, I let it boot normally and it should give me a grub menu with proper “OpenSUSE Tumbleweed” boot entry.
And that should work like before.
Boot and login! (It did, twice I followed this process and it did).

(A note - When I reboot into OpenSUSE TW after this new grub config creation on a new system board, I need to also make sure that “Secure Boot” is disabled in BIOS menu.
Otherwise it will not allow the OpenSUSE TW to boot. It didn’t for me because my Secure Boot was enabled.
So, I had to disable it. And then it worked.
After first successful boot, I think I can enable it again.)

None of this process is my own making.
The whole credit goes to this very useful demo - How To Fix Grub in OpenSUSE | UEFI (KMDTech) (Thank you!)
It worked twice for me without a single hiccup.
(but ofcourse, if you need to follow, please do it carefully, after checking, cross-checking and understanding everything you’re typing.)

Setup (again): HP Laser MFP 136nw

Sat, 01 Nov 2025 00:00:00 +0000

A few weeks back, I had to reset my router for firmware updates.

And because of that, some devices on my local network, in this case my HP printer stopped working on Wi-Fi.

I followed the following steps today, to make it work again.
(MJB helped lots! Thank you so much! Very grateful!)

I realised the IP (http://192.168.1.z) that was assigned to my HP Printer (before router reset) is now taken by some other device on my local network because of DHCP dynamically assigning IPs.
I connect to the HP Printer via Ethernet, access the HP Printer configuration page on the last assigned IP (http://192.168.1.z). (Because now I am connected to the HP Printer via Ethernet, the router give preference to the HP Printer on the above IP, even though DHCP Server had assigned this IP to another device on my network.)
I login to my HP Printer configuration page.
I go to “Settings > Network Settings > Wi-Fi”.
On this menu page, click on “Wi-Fi Settings > Advanced Settings > Network Setup”.
Go to “SSID” and hit “Search List” and “Refresh”.
From the drop down, pick the Wi-Fi SSID I want to connect to, at the bottom, pick “Network Key Setup” and put the updated SSID password in there (both in “Network Key” and “Confirm Network Key”).
Don’t forget to hit “Apply”.
Now other thing I have to fix is that the IP address is still assigned by the Router’s DHCP server to another device on LAN.
I need to assign a proper IP to my HP Printer outside the range of IPs available to DHCP server to assign to devices dynamically.
For that, go to the Router admin page, login and go to “Local Network > LAN > IPv4”.
Then go to the section “DHCP Server” and change “DHCP Start IP Address” and “DHCP End IP Address” respectively to some “192.168.1.a” and “192.168.1.b” and hit “Apply”.
With this the router will now have IP “192.168.1.a-1” and DHCP server will only be able to assign dynamically IPs to devices within the assigned pool only.
(with this, what I am trying is to limit the pool of IPs available to DHCP server so that I can assign an IP (“192.168.1.b+1”) to HP Printer outside the limits of this available DHCP server IP pool manually. So, that the printer IP doesn’t conflict with any other device IP assigned by DHCP server.)
Now login back to the Printer configuration page, go to “Settings > Network Settings > TCP/IPv4”.
Here, in the “General” section, pick “Manual” under the “Assign IPv4 Address” option.
And manually assign the following - (1) “IPv4 Address: 192.168.1.b+1”, (2) “Subnet Mask: 255.255.255.0”, and (3) “Gateway Address: 192.168.1.a-1” (should match the router IP address) to HP Printer.
And hit “Apply”.
With this, the HP Printer configuration page itself will reload to the new assigned IP address url (http://192.168.1.b+1).
After above steps, I then remove the Ethernet from the HP Printer, restart it.
And check if I am still able to access the HP Printer on the assigned IP via Wi-Fi (http://192.168.1.b+1).
Yes! It worked now!
Then now I need to test, whether printing works on Wi-Fi.
I am on an OpenSUSE Tumbleweed machine.
I go to the “Settings > Printers” page.
I have to make sure that my printer is showing up there.
(It wasn’t before, I had to once manually add a printer and pick up the latest available matching model from the available database, but that’s not needed after my steps below.)
Yes, printer shows up. I gave a test print job. Printing on Wi-Fi is working now.
But Scanning still doesn’t work. Neither on Wifi, nor on Ethernet.
My system just doesn’t detect the scanner on my HP Printer at all.
Now, I go back to HP Printer configuration page (http://192.168.1.b+1).
Go to “Settings > Network Settings” and ensure that “AirPrint” and “Bonjour(mDNS)” both are enabled.

Now, I need to do a few things at the OS level.
Install (or ensure if already installed) the following set of packages.

# install required packages 
  
sudo zypper install avahi avahi-utils nss-mdns
sudo zypper install hplip hplip-sane sane-backends
sudo zypper install sane-airscan ipp-usb

# enable, start and verify status of avahi daeomon (make sure I use `-l` flag to have all available information from the service status output)

sudo systemctl enable avahi-daemon
sudo systemctl start avahi-daemon
sudo systemctl status -l avahi-daemon.service

# make sure I have all avahi-related tools as well
  
which avahi-browse
which avahi-resolve
which avahi-publish
rpm -ql avahi | grep bin # gives me `/usr/sbin/avahi-daemon` and `/usr/sbin/avahi-dnsconfd`


# ensure firewall is allowing mdns and ipp

sudo firewall-cmd --permanent --add-service=mdns
sudo firewall-cmd --permanent --add-service=ipp
sudo firewall-cmd --reload
sudo firewall-cmd --info-service=mdns
sudo firewall-cmd --info-service=ipp

# and restart firewall
  
sudo systemctl restart firewalld
sudo systemctl status -l firewalld

# now check if avahi-browse can see devices advertised by my HP Printer

avahi-browse -a | grep HP
## output something like following. I need to make sure it has `_scanner._tcp` and `_uscan._tcp` showing up
  + wlp0s20f3 IPv4 HP Laser MFP 136nw (xx:xx:xx)                 _ipp._tcp            local
  + wlp0s20f3 IPv4 HP Laser MFP 136nw (xx:xx:xx)                 _scanner._tcp        local
  + wlp0s20f3 IPv4 HP Laser MFP 136nw (xx:xx:xx)                 _uscan._tcp          local
  + wlp0s20f3 IPv4 HP Laser MFP 136nw (xx:xx:xx)                 _uscans._tcp         local
  + wlp0s20f3 IPv4 HP Laser MFP 136nw (xx:xx:xx)                 _http._tcp           local
  + wlp0s20f3 IPv4 HP Laser MFP 136nw (xx:xx:xx)                 _pdl-datastream._tcp local
  + wlp0s20f3 IPv4 HP Laser MFP 136nw (xx:xx:xx)                 _printer._tcp        local
  + wlp0s20f3 IPv4 HP Laser MFP 136nw (xx:xx:xx)                 _http-alt._tcp       local
  + wlp0s20f3 IPv4 HP Laser MFP 136nw (xx:xx:xx)                 _privet._tcp         local

# if device shows up, then check is scanner is responding on the network

ping -c3 192.168.1.x

curl http://192.168.1.x:8080/eSCL # any xml ouput is fine as long as there's something

# final check
  
scanimage -L
## it should list something like:
device `airscan:xx:xx Laser MFP 136nw (xx:xx:xx)' is a eSCL HP Laser MFP 136nw (xx:xx:xx) ip=192.168.1.x

At this point, the “Scan Documents” app should be detecting the scanner on my HP printer (it did!)
Also, with Avahi working, my OS system “Settings > Printers” also got a HP Printer added automatically with the correct model name etc.
(Scanner also, although that doesn’t show up as a menu item in the system settings.)

How to run Kubernetes (Node) e2e tests locally?

Sat, 20 Sep 2025 00:00:00 +0000

TL;DR:

To run Node e2e tests

without feature-gate

> KUBE_ROOT=$(pwd) make test-e2e-node FOCUS="a pod failing to mount volumes"

with feature-gate

> KUBE_ROOT=$(pwd) make test-e2e-node FOCUS="a pod failing to mount volumes" TEST_ARGS='--feature-gates=PodReadyToStartContainersCondition=true'

To run rest of the e2e tests:

> ./_output/bin/e2e.test \
--provider=local \
--kubeconfig=$HOME/.kube/config \
--ginkgo.focus="PodReadyToStartContainerCondition"

(logging for my future self!)

Below are a bunch of my raw running notes as I try to figure out how to run Kubernetes e2e tests locally.

I’m trying to add a new e2e test in the Kubernetes codebase (as part of the work required for the GA promotion of KEP-3085 (aka the feature-gate, PodReadyToStartContainerCondition).

And I want to be able to run this locally.

So, starting with the following steps to set up my environment:

clone the kubernetes repository

> git clone git@github.com:kubernetes/kubernetes.git

> cd kubernetes

> git checkout -b some-new-branch-name

# build all the binaries you'll need for running e2e locally
> make WHAT=test/e2e/e2e.test

This will produce the e2e.test binary in ./output/bin/e2e.test path.

Set up a local kubernetes cluster (I’ll use kind)

> kind create cluster --name local-e2e-test

# to make sure my kubeconfig is pointing to the intended cluster
> kubectl cluster-info

Run the e2e test binary (the newly built ./output/bin/e2e.test binary).

The e2e framework runs Ginkgo tests.

I can target the test I want to run, by its --ginkgo.focus flag.

Example:
```
> ./_output/bin/e2e.test \
--provider=local \
--kubeconfig=$HOME/.kube/config \
--ginkgo.focus="PodReadyToStartContainerCondition"
```
I also learnt, that there’s a --help flag also available, if I want to understand the full length of the capablities offered by this e2e.test binary.

I tried a few of them below.

For example, If I want to get a dump of the entire list of e2e tests from the binary:
```
> ./_output/bin/e2e.test --list-tests
```
That prints all 7k+ test names.

So, if I want to check for a specific test, I can grep for some keyword or string used in the test definition (I learnt this needs to some keyword or string from the ginkgo.It(...) block where the test is invoked):
```
> ./_output/bin/e2e.test --list-tests | grep -i "some-string-in-the-test-definition"
```
Also, please note, if I have added a new test after above steps, then IIUC e2e.test --list-tests won’t show it right away.
I’ll need to rebuild the binary to make the new test visible.

You know what? I realised what I did above, doesn’t really help me to run the Node e2e tests (aka the tests defined in the path test/e2d_node/* of the Kubernetes repository).

(And for the KEP-3085, that is exactly what I need - to be able to write and run Node e2e tests locally)

But before I move ahead - How did I realise that I need something else?

Well, because - e2e.test binary wasn’t able to list my intended test (which is defined in the test/e2e_node/pod_conditions_test.go path in the Kubernetes repo).

Ok, moving forward.

I figured out, there’s a different way to run these Node e2e tests locally.

I came across this handy document – Node End-To-End (e2e) tests.
This very thoroughly describes the process for running Node e2e tests locally (as well as, remotely, but that’s for some other day. I want to try the local execution first). (Thank you, to the good folks in the SIG Node, this truly is a gem of a resource!)

Ok, let’s see if this helps me with my usecase!

There’s a bit of work needed (the pre-requisities) before I can start following the instructions in the document to actually run the tests.

I need to install etcd, and make sure it’s available in the path.
(I’m on an openSUSE Tumbleweed machine, so I did zypper install etcd and that did the job. Otherwise, follow the instructions from Etcd documentation here.)
```
> which etcd
/usr/sbin/etcd
  
> etcd --version
etcd Version: 3.6.4
Git SHA: Not provided (use ./build instead of go build)
Go Version: go1.24.5
Go OS/Arch: linux/amd64
```

Next, I need to install Containerd and configure cgroup driver to cgroupfs.
(once again, I did zypper install containerd. Otherwise, follow the instructions from Containerd documentation here.)

> sudo mkdir -p /etc/containerd
> sudo containerd config default | sudo tee /etc/containerd/config.toml
> sudo sed -i 's/SystemdCgroup = true/SystemdCgroup = false/' /etc/containerd/config.toml
> sudo systemctl restart containerd

> containerd --version
containerd github.com/containerd/containerd v1.7.27 05044ec0a9a75232cad458027ca83437aae3f4da

And then finally, I need to ensure that there’s a valid CNI configuration in /etc/cni/net.d/.
For that I ran the following steps.

> sudo mkdir -p /opt/cni/bin

> sudo curl -L https://github.com/containernetworking/plugins/releases/download/v1.3.0/cni-plugins-linux-amd64-v1.3.0.tgz | sudo tar -C /opt/cni/bin -xz

> sudo mkdir -p /etc/cni/net.d

> sudo tee /etc/cni/net.d/10-bridge.conf <<EOF
{
  "cniVersion": "0.4.0",
  "name": "bridge",
  "type": "bridge",
  "bridge": "cni0",
  "isGateway": true,
  "ipMasq": true,
  "ipam": {
      "type": "host-local",
      "subnet": "10.22.0.0/16",
      "routes": [{"dst": "0.0.0.0/0"}]
  }
}
EOF

With these pre-requisties out of my way, for running the Node e2e tests (i.e., to run the ginkgo binary against the subdirectory test/e2e_node), I will use the following make target:

make test-e2e-node

And this also has a handy HELP option:

make test-e2e-node PRINT_HELP=y

Right now, I want to try running a single test, so, I will use the FOCUS arugment to target it.
(I want to test this block invoked via this ginkgo.It(...) description)

So, my make target call would look something like:
(Please note, I also had to set the KUBE_ROOT variable to point to the correct path to my local kubernetes repo clone.)

KUBE_ROOT=$(pwd) make test-e2e-node FOCUS="just the scheduled condition set"

Wait! It ran, but not successfully. I got the following error:

  << Captured StdOut/StdErr Output

  Timeline >>
  I0920 14:50:41.511206   22650 server.go:105] Starting server "services" with command "/home/psaggu/work-upstream/kep-3085-add-condition-for-sandbox-creation/kubernetes/_output/local/go/bin/e2e_node.test --run-services-mode --bearer-token=Ai-zpfsimz4_NvH2 --test.timeout=0 --ginkgo.seed=1758360041 --ginkgo.timeout=23h59m59.999947072s --ginkgo.grace-period=30s --ginkgo.focus=just the scheduled condition set --ginkgo.skip=\\[Flaky\\]|\\[Slow\\]|\\[Serial\\] --ginkgo.parallel.process=1 --ginkgo.parallel.total=8 --ginkgo.parallel.host=127.0.0.1:33109 --v 4 --report-dir=/tmp/_artifacts/250920T145006 --node-name alpha --kubelet-flags=--cluster-domain=cluster.local --dns-domain=cluster.local --prepull-images=false --container-runtime-endpoint=unix:///run/containerd/containerd.sock --runtime-config= --kubelet-config-file=test/e2e_node/jenkins/default-kubelet-config.yaml"
  I0920 14:50:41.511254   22650 util.go:48] Running readiness check for service "services"
  I0920 14:50:41.511354   22650 server.go:133] Output file for server "services": /tmp/_artifacts/250920T145006/services.log
  I0920 14:50:41.511706   22650 server.go:163] Waiting for server "services" start command to complete
  W0920 14:50:43.029782   22650 util.go:106] Health check on "https://127.0.0.1:6443/healthz" failed, status=500
  I0920 14:50:44.032240   22650 services.go:69] Node services started.
  I0920 14:50:44.032254   22650 kubelet.go:157] Starting kubelet
  I0920 14:50:44.032259 22650 kubelet.go:159] Standalone mode: false
  I0920 14:50:44.033031   22650 feature_gate.go:385] feature gates: {map[]}
  I0920 14:50:44.042420   22650 server.go:105] Starting server "kubelet" with command "/usr/bin/systemd-run -p Delegate=true -p StandardError=append:/tmp/_artifacts/250920T145006/kubelet.log --unit=kubelet-20250920T145044.service --slice=runtime.slice --remain-after-exit /home/psaggu/work-upstream/kep-3085-add-condition-for-sandbox-creation/kubernetes/_output/local/go/bin/kubelet --kubeconfig /home/psaggu/work-upstream/kep-3085-add-condition-for-sandbox-creation/kubernetes/_output/local/go/bin/kubeconfig --root-dir /var/lib/kubelet --v 4 --config-dir /home/psaggu/work-upstream/kep-3085-add-condition-for-sandbox-creation/kubernetes/_output/local/go/bin/kubelet.conf.d --hostname-override alpha --container-runtime-endpoint unix:///run/containerd/containerd.sock --config /home/psaggu/work-upstream/kep-3085-add-condition-for-sandbox-creation/kubernetes/_output/local/go/bin/kubelet-config --cluster-domain=cluster.local"
  I0920 14:50:44.042464   22650 util.go:48] Running readiness check for service "kubelet"
  I0920 14:50:44.042571   22650 server.go:133] Output file for server "kubelet": /tmp/_artifacts/250920T145006/kubelet.log
  I0920 14:50:44.043021   22650 server.go:163] Waiting for server "kubelet" start command to complete
  W0920 14:50:45.058069   22650 util.go:104] Health check on "http://127.0.0.1:10248/healthz" failed, error=Head "http://127.0.0.1:10248/healthz": dial tcp 127.0.0.1:10248: connect: connection refused

    ...
    ...

  In [SynchronizedBeforeSuite] at: k8s.io/kubernetes/test/e2e_node/e2e_node_suite_test.go:235 @ 09/20/25 14:52:44.204
------------------------------
[SynchronizedBeforeSuite] [FAILED] [122.848 seconds]
[SynchronizedBeforeSuite] 
k8s.io/kubernetes/test/e2e_node/e2e_node_suite_test.go:235

  [FAILED] SynchronizedBeforeSuite failed on Ginkgo parallel process #1
    The first SynchronizedBeforeSuite function running on Ginkgo parallel process
    #1 failed.  This suite will now abort.

  
  In [SynchronizedBeforeSuite] at: k8s.io/kubernetes/test/e2e_node/e2e_node_suite_test.go:235 @ 09/20/25 14:52:44.211
------------------------------

Summarizing 8 Failures:
  [FAIL] [SynchronizedBeforeSuite] 
  k8s.io/kubernetes/test/e2e_node/e2e_node_suite_test.go:235

  ...
  ...

Ran 0 of 876 Specs in 130.799 seconds
FAIL! -- A BeforeSuite node failed so all tests were skipped.

Ginkgo ran 1 suite in 2m11.106353619s

Test Suite Failed
F0920 14:52:52.136374   21331 run_local.go:101] Test failed: exit status 1
exit status 1
make: *** [Makefile:292: test-e2e-node] Error 1

I did a little bit looking around and realised that my test is not able to start kubelet properly.

But I still couldn’t understand why exactly! Because the Kubelet binary is built as part of the test run and is available in the path - _output/local/go/bin/kubelet.
(I didn’t have to install kubelet binary by myself, it was built as part of the above make target run along with other components like kube-apiserver, etc.)

Some more digging (well, actually reading the above error message once again, properly).
I realised, I can find out the actually issue in the kubelet logs.

Below log helped:

I0920 14:50:44.042571 22650 server.go:133] Output file for server “kubelet”: /tmp/_artifacts/250920T145006/kubelet.log

So, checking the kubelet logs and it gave me the problem right away.

> tail -f /tmp/_artifacts/250920T145006/kubelet.log

I0920 14:50:44.151457   22917 server.go:579] "Sending events to api server"
I0920 14:50:44.151587   22917 swap_util.go:119] "Swap is on" /proc/swaps contents=<
	Filename				Type		Size		Used		Priority
	/dev/nvme0n1p3                          partition	2097472		0		-2
 >
E0920 14:50:44.151610   22917 run.go:72] "command failed" err="failed to run Kubelet: running with swap on is not supported, please disable swap or set --fail-swap-on flag to false"

The swap is ON, on my machine.
The test setup expects me to to turn it OFF atleast for the duration of the execution of the test.


sudo swapoff -a

I turned off the swap and reran the test.

> KUBE_ROOT=$(pwd) make test-e2e-node FOCUS="just the scheduled condition set"

  << Captured StdOut/StdErr Output
------------------------------
SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS

Ran 1 of 876 Specs in 37.704 seconds
SUCCESS! -- 1 Passed | 0 Failed | 0 Pending | 875 Skipped

Ginkgo ran 1 suite in 37.997204389s
Test Suite Passed

It worked! The test ran successfully! Hurrah!

But where to check the logs now?
I can’t see any test execution logs, information like Pod status (pending -> initializing -> running, etc).

Figured out - the make test-e2e-node run captures a bunch of artifacts from the execution of the test.

I can find them in the /tmp/_artifacts path.

❯ ls /tmp/_artifacts/250920T153001

build-log.txt  containerd-installation.log  containerd.log  junit_01.xml  kern.log  kubelet.log  services.log

And in my case, for the Pod status related information, I can check those from the Kubelet logs.

❯ cat /tmp/_artifacts/250920T153001/kubelet.log

So, here we go. I have a solution for running my new node e2e tests locally.

(Off I go to implement the test! o/)

go workspaces usage in kubernetes project

Thu, 19 Jun 2025 00:00:00 +0000

Kubernetes (k/k) recently started using Go Workspaces to manage its multi-module setup more effectively.

The goal is to helps keep things consistent across the many staging repositories and the main k/k project.

At the root of the k/k repo, you’ll find go.work and go.work.sum.

These files are auto-generated by the hack/update-vendor.sh script.

Specifically, the logic lives around lines 198–219 and 293–303.

At a high level, the script runs the following commands from the root of the repo:

go work init
go work edit -go 1.24.0 -godebug default=go1.24
go work edit -use .

git ls-files -z ':(glob)./staging/src/k8s.io/*/go.mod' \
  | xargs -0 -n1 dirname -z \
  | xargs -0 -n1 go work edit -use

go mod download
go work vendor

This creates:

go.work file that includes the root module (.) and all the Kubernetes staging modules under staging/src/k8s.io.

go 1.24.0

godebug default=go1.24

use (
    .
    ./staging/src/k8s.io/api
    ./staging/src/k8s.io/apiextensions-apiserver
    ./staging/src/k8s.io/apimachinery
    ...
    ./staging/src/k8s.io/sample-controller
)

go.work.sum file that tracks checksums for the workspace modules — like go.sum, but at the workspace level.
and the vendor/ directory which is populated via go work vendor and collects all dependencies from the workspace modules.

go_modules: how to create go vendor tarballs from subdirectories

Thu, 10 Apr 2025 00:00:00 +0000

The go_modules OBS service is used to download, verify, and vendor Go module dependency sources.

As described in the source project’s (obs-service-go_modules) README:

Using the go.mod and go.sum files present in a Go application, obs-service-go_modules will call Go tools in sequence:

go mod download

go mod verify

go mod vendor

obs-service-go_modules then creates a vendor.tar.gz archive (or another supported compression format) containing the vendor/ directory generated by go mod vendor.
This archive is produced in the RPM package directory and can be committed to OBS to support offline Go application builds for openSUSE, SUSE, and various other distributions.

The README also provides a few usage examples for packagers.

However, it wasn’t immediately clear how to use the go_modules OBS service to create multiple vendor tarballs from different subdirectories within a single Git source repository.

Below is an example where I create multiple vendor tarballs from a single Git repo— (in this case, the etcd project):

<services>

  <!-- Service #1 -->
  <service name="obs_scm">
    <param name="url">https://github.com/etcd/etcd.git</param>
    <param name="scm">git</param>
    <param name="package-meta">yes</param>
    <param name="versionformat">@PARENT_TAG@</param>
    <param name="versionrewrite-pattern">v(.*)</param>
    <param name="revision">v3.5.21</param>
    <param name="without-version">yes</param>
  </service>

  <!-- Service #2 -->
  <service name="go_modules">
    <param name="archive">*etcd.obscpio</param>
  </service>

  <!-- Service #3 -->
  <service name="go_modules">
    <param name="archive">*etcd.obscpio</param>
    <param name="subdir">server</param>
    <param name="vendorname">vendor-server</param>
  </service>

  <!-- Service #4 -->
  <service name="go_modules">
    <param name="archive">*etcd.obscpio</param>
    <param name="subdir">etcdctl</param>
    <param name="vendorname">vendor-etcdctl</param>
  </service>

</services>

The above _service file defines four services:

Service 1 clones the GitHub repo github.com/etcd/etcd.git into the build root. The resulting output is a cpio archive blob—etcd.cpio.
Service 2 locates the etcd.cpio archive, extracts it, runs go mod download, go mod verify, and go mod vendor from the repo root, and creates the default vendor.tar.gz.
Service 3 and Service 4 work the same as Service 2, with one difference: they run the Go module commands from subdirectories:
- Service 3 changes into the server/ directory before running the Go commands, producing a tarball named vendor-server.tar.gz.
- Service 4 does the same for the etcdctl/ directory, producing vendor-etcdctl.tar.gz.

🔍 Note the subdir and vendorname parameters. These are the key to generating multiple vendor tarballs from various subdirectories, with custom names.

I found the full list of parameters accepted by the go_modules service defined here¹:

...
parser.add_argument("--strategy", default="vendor")
parser.add_argument("--archive")
parser.add_argument("--outdir")
parser.add_argument("--compression", default=DEFAULT_COMPRESSION)
parser.add_argument("--basename")
parser.add_argument("--vendorname", default=DEFAULT_VENDOR_STEM)
parser.add_argument("--subdir")
...

The default values are defined here²:

DEFAULT_COMPRESSION = "gz"
DEFAULT_VENDOR_STEM = "vendor"

Also, while writing this post, I discovered that the final vendor tarball can be compressed in one of the following supported formats³:

.tar.bz2
.tar.gz
.tar.lz
.tar.xz
.tar.zst

And finally, here’s the list of supported source archive formats (the blob from which the vendor tarball is created), powered by the libarchive Python module⁴:

READ_FORMATS = set((
    '7zip', 'all', 'ar', 'cab', 'cpio', 'empty', 'iso9660', 'lha', 'mtree',
    'rar', 'raw', 'tar', 'xar', 'zip', 'warc'
))

https://github.com/openSUSE/obs-service-go_modules/blob/a9bf055557cf024478744fbd7e8621fd03cb2e87/go_modules#L227-L233 ↩
https://github.com/openSUSE/obs-service-go_modules/blob/a9bf055557cf024478744fbd7e8621fd03cb2e87/go_modules#L46C1-L47C31 ↩
https://github.com/openSUSE/obs-service-go_modules/blob/a9bf055557cf024478744fbd7e8621fd03cb2e87/go_modules#L119-L124 ↩
https://github.com/Changaco/python-libarchive-c/blob/1a5b505ab1818686c488b4904445133bcc86fb4d/libarchive/ffi.py#L243-L246 ↩

Access data persisted in Etcd with etcdctl and kubectl

Tue, 25 Feb 2025 00:00:00 +0000

I created the following CRD (Custom Resource Definition) with — kubectl apply -f crd-with-x-validations.yaml:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  # name must be in the form: <plural>.<group>
  name: myapps.example.com
spec:
  # group name to use for REST API: /apis/<group>/<version>
  group: example.com
  scope: Namespaced
  names:
    # kind is normally the CamelCased singular type. 
    kind: MyApp
    # singular name to be used as an alias on the CLI
    singular: myapp
    # plural name in the URL: /apis/<group>/<version>/<plural>
    plural: myapps
  versions:
  - name: v1
    served: true
    storage: true
    schema:
      openAPIV3Schema:
        type: object
        properties:
          spec:
            x-kubernetes-validations: 
              - rule: "self.minReplicas <= self.maxReplicas"
                messageExpression: "'minReplicas (%d) cannot be larger than maxReplicas (%d)'.format([self.minReplicas, self.maxReplicas])"
            type: object
            properties:
              minReplicas:
                type: integer
              maxReplicas:
                type: integer

I want to check how the above CRD is persisted in Etcd.

I have two ways to do the job:

Option 1:

Use etcdctl to directly verify the persisted data in Etcd.¹

My three steps process:

Exec inside the etcd pod in the kube-system namespace of your kubernetes cluster — kubectl exec -it -n kube-system etcd-kep-4595-cluster-control-plane -- /bin/sh
Create alias — alias e="etcdctl --endpoints 127.0.0.1:2379 --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key --cacert=/etc/kubernetes/pki/etcd/ca.crt"
Access the data — e get --prefix /registry/apiextensions.k8s.io/

sh-5.2# e get --prefix /registry/apiextensions.k8s.io/

/registry/apiextensions.k8s.io/customresourcedefinitions/shirts.stable.example.com
{"kind":"CustomResourceDefinition","apiVersion":"apiextensions.k8s.io/v1beta1","metadata":{"name":"shirts.stable.example.com","uid":"09696eb0-d58b-4a21-8820-b2230b13707e","generation":1,"creationTimestamp":"2025-02-21T12:38:19Z","annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"apiextensions.k8s.io/v1\",\"kind\":\"CustomResourceDefinition\",\"metadata\":{\"annotations\":{},\"name\":\"shirts.stable.example.com\"},\"spec\":{\"group\":\"stable.example.com\",\"names\":{\"kind\":\"Shirt\",\"plural\":\"shirts\",\"shortNames\":[\"shrt\"],\"singular\":\"shirt\"},\"scope\":\"Namespaced\",\"versions\":[{\"additionalPrinterColumns\":[{\"jsonPath\":\".spec.color\",\"name\":\"Fruit\",\"type\":\"string\"}],\"name\":\"v1\",\"schema\":{\"openAPIV3Schema\":{\"properties\":{\"spec\":{\"properties\":{\"color\":{\"type\":\"string\"},\"size\":{\"type\":\"string\"}},\"type\":\"object\"}},\"type\":\"object\"}},\"served\":true,\"storage\":true}]}}\n"},"managedFields":[{"manager":"kube-apiserver","operation":"Update","apiVersion":"apiextensions.k8s.io/v1","time":"2025-02-21T12:38:19Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{"f:acceptedNames":{"f:kind":{},"f:listKind":{},"f:plural":{},"f:shortNames":{},"f:singular":{}},"f:conditions":{"k:{\"type\":\"Established\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"NamesAccepted\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}},"subresource":"status"},{"manager":"kubectl-client-side-apply","operation":"Update","apiVersion":"apiextensions.k8s.io/v1","time":"2025-02-21T12:38:19Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:kubectl.kubernetes.io/last-applied-configuration":{}}},"f:spec":{"f:conversion":{".":{},"f:strategy":{}},"f:group":{},"f:names":{"f:kind":{},"f:listKind":{},"f:plural":{},"f:shortNames":{},"f:singular":{}},"f:scope":{},"f:versions":{}}}}]},"spec":{"group":"stable.example.com","version":"v1","names":{"plural":"shirts","singular":"shirt","shortNames":["shrt"],"kind":"Shirt","listKind":"ShirtList"},"scope":"Namespaced","validation":{"openAPIV3Schema":{"type":"object","properties":{"spec":{"type":"object","properties":{"color":{"type":"string"},"size":{"type":"string"}}}}}},"versions":[{"name":"v1","served":true,"storage":true}],"additionalPrinterColumns":[{"name":"Fruit","type":"string","JSONPath":".spec.color"}],"conversion":{"strategy":"None"},"preserveUnknownFields":false},"status":{"conditions":[{"type":"NamesAccepted","status":"True","lastTransitionTime":"2025-02-21T12:38:19Z","reason":"NoConflicts","message":"no conflicts found"},{"type":"Established","status":"True","lastTransitionTime":"2025-02-21T12:38:19Z","reason":"InitialNamesAccepted","message":"the initial names have been accepted"}],"acceptedNames":{"plural":"shirts","singular":"shirt","shortNames":["shrt"],"kind":"Shirt","listKind":"ShirtList"},"storedVersions":["v1"]}}

Option 2:

Use kubectl to access the persisted data from Etcd –

kubectl get --raw /apis/apiextensions.k8s.io/v1/customresourcedefinitions/shirts.stable.example.com

> kubectl get --raw /apis/apiextensions.k8s.io/v1/customresourcedefinitions/shirts.stable.example.com

{"kind":"CustomResourceDefinition","apiVersion":"apiextensions.k8s.io/v1","metadata":{"name":"shirts.stable.example.com","uid":"09696eb0-d58b-4a21-8820-b2230b13707e","resourceVersion":"594","generation":1,"creationTimestamp":"2025-02-21T12:38:19Z","annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"apiextensions.k8s.io/v1\",\"kind\":\"CustomResourceDefinition\",\"metadata\":{\"annotations\":{},\"name\":\"shirts.stable.example.com\"},\"spec\":{\"group\":\"stable.example.com\",\"names\":{\"kind\":\"Shirt\",\"plural\":\"shirts\",\"shortNames\":[\"shrt\"],\"singular\":\"shirt\"},\"scope\":\"Namespaced\",\"versions\":[{\"additionalPrinterColumns\":[{\"jsonPath\":\".spec.color\",\"name\":\"Fruit\",\"type\":\"string\"}],\"name\":\"v1\",\"schema\":{\"openAPIV3Schema\":{\"properties\":{\"spec\":{\"properties\":{\"color\":{\"type\":\"string\"},\"size\":{\"type\":\"string\"}},\"type\":\"object\"}},\"type\":\"object\"}},\"served\":true,\"storage\":true}]}}\n"},"managedFields":[{"manager":"kube-apiserver","operation":"Update","apiVersion":"apiextensions.k8s.io/v1","time":"2025-02-21T12:38:19Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{"f:acceptedNames":{"f:kind":{},"f:listKind":{},"f:plural":{},"f:shortNames":{},"f:singular":{}},"f:conditions":{"k:{\"type\":\"Established\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"NamesAccepted\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}},"subresource":"status"},{"manager":"kubectl-client-side-apply","operation":"Update","apiVersion":"apiextensions.k8s.io/v1","time":"2025-02-21T12:38:19Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:kubectl.kubernetes.io/last-applied-configuration":{}}},"f:spec":{"f:conversion":{".":{},"f:strategy":{}},"f:group":{},"f:names":{"f:kind":{},"f:listKind":{},"f:plural":{},"f:shortNames":{},"f:singular":{}},"f:scope":{},"f:versions":{}}}}]},"spec":{"group":"stable.example.com","names":{"plural":"shirts","singular":"shirt","shortNames":["shrt"],"kind":"Shirt","listKind":"ShirtList"},"scope":"Namespaced","versions":[{"name":"v1","served":true,"storage":true,"schema":{"openAPIV3Schema":{"type":"object","properties":{"spec":{"type":"object","properties":{"color":{"type":"string"},"size":{"type":"string"}}}}}},"additionalPrinterColumns":[{"name":"Fruit","type":"string","jsonPath":".spec.color"}]}],"conversion":{"strategy":"None"}},"status":{"conditions":[{"type":"NamesAccepted","status":"True","lastTransitionTime":"2025-02-21T12:38:19Z","reason":"NoConflicts","message":"no conflicts found"},{"type":"Established","status":"True","lastTransitionTime":"2025-02-21T12:38:19Z","reason":"InitialNamesAccepted","message":"the initial names have been accepted"}],"acceptedNames":{"plural":"shirts","singular":"shirt","shortNames":["shrt"],"kind":"Shirt","listKind":"ShirtList"},"storedVersions":["v1"]}}

I realised while I’m accessing the same CRD data with etcdctl and kubectl, I’m getting a few things different in my output. In case of etcdctl — I get (i) "version":"v1", (ii) the CRD schema is stored in field "validation":{"openAPIV3Schema":{"type":"object","properties":{"spec":{"type":"object","properties":{"color":{"type":"string"},"size":{"type":"string"}}}}}} and (iii) and there’s a top level additionalPrinterColumns. While in case of kubectl— I don’t get the above bits, and instead I get both, the schema and the additionalPrinterColumns stored in the versions array - "versions":[{"name":"v1","served":true,"storage":true,"schema":{"openAPIV3Schema":{"type":"object","properties":{"spec":{"type":"object","properties":{"color":{"type":"string"},"size":{"type":"string"}}}}}},"additionalPrinterColumns":[{"name":"Fruit","type":"string","jsonPath":".spec.color"}]}]. This is (maybe) something to do with how currently (as of writing) Kubernetes stores/persists CRD v1 as v1beta1 in Etcd, because v1 takes more space to represent the same CRD (due to denormalization of fields among multi-version CRDs) and we have CRDs in the wild that are already bumping against the max allowed size (Thank you, Jordan Liggit for explaining this.) Read this² and this³ for some context. ↩
code block, where the encoding version for CRDs is configured ↩
Attempt to bump the storage version from v1beta1 → v1, but was blocked on k/k PR #82292 ↩

slabtop - to check kernel memory usage, and kubelet's container_memory_kernel_usage metrics

Mon, 24 Feb 2025 00:00:00 +0000

Today, I learnt about slabtop¹, a command line utility to check the memory used by kernel
(or as its man page says, it displays the kernel slab cache information in real time²).

(Logging for the future me)

So, what led to me learning about slabtop? 🙂

Jason Braganza taught me this, while we were preparing for our upcoming conference talk (on Kubernetes Metrics)!

Precisely, following is the metrics (exposed by kubelet component within a kubernetes cluster) that led to the discussion.

# HELP container_memory_kernel_usage Size of kernel memory allocated in bytes.
# TYPE container_memory_kernel_usage gauge
container_memory_kernel_usage{container="",id="/",image="",name="",namespace="",pod=""} 0 1732452865827

And how kubelet gets this “kernel memory allocation” information and feeds to the container_memory_kernel_usage metrics?

The answer is (at least to the best of my understanding) –

The kubelet’s server package imports “github.com/google/cadvisor/metrics”³ (aka the cadvisor/metrics) module.

This cadvisor/metrics go module provides a NewPrometheusCollector() function (which kubelet uses here⁴).

The NewPrometheusCollector() function take includedMetrics as one of its many parameters.

  r.RawMustRegister(metrics.NewPrometheusCollector(prometheusHostAdapter{s.host}, containerPrometheusLabelsFunc(s.host), includedMetrics, clock.RealClock{}, cadvisorOpts))

And when this includedMetrics contains cadvisormetrics.MemoryUsageMetrics (which it does in the case in question, check here⁵),

	includedMetrics := cadvisormetrics.MetricSet{
		...
		cadvisormetrics.MemoryUsageMetrics:  struct{}{},
		...
	}

then NewPrometheusCollector() function exposes the container_memory_kernel_usage⁶ metrics.

func NewPrometheusCollector(i infoProvider, f ContainerLabelsFunc, includedMetrics container.MetricSet, now clock.Clock, opts v2.RequestOptions) *PrometheusCollector {
        ...
        ...
	if includedMetrics.Has(container.MemoryUsageMetrics) {
		c.containerMetrics = append(c.containerMetrics, []containerMetric{
		       ...
		       ...
		       {
				name:      "container_memory_kernel_usage",
				help:      "Size of kernel memory allocated in bytes.",
				valueType: prometheus.GaugeValue,
				getValues: func(s *info.ContainerStats) metricValues {
					return metricValues
				},
			},
        ...
        ...

And as we see above in the definition of container_memory_kernel_usage metrics,
the valueType is prometheus.Gaugevalue (so its a gauge type metrics),
and the value is value: float64(s.Memory.KernelUsage) where KernelUsage is defined here⁷ and interpreted here⁸.

I still feel I can go further down a few more steps to find out the true source of this information, but that’s all for now.

which in turn gets the information from /proc/slabinfo. ↩
to me it looks something like, top or htop. ↩
here: https://github.com/kubernetes/kubernetes/blob/d92b99ea637ee67a5c925e5e628f5816a01162ac/pkg/kubelet/server/server.go#L39C2-L39C38 ↩
kubelet registring the cadvisor metrics provided by cdvisor’s metrics.NewPrometheusCollector(...) function – https://github.com/kubernetes/kubernetes/blob/d92b99ea637ee67a5c925e5e628f5816a01162ac/pkg/kubelet/server/server.go#L463 ↩
Kubelet server package creating a (cadvisor metrics based) set of includedMetrics: https://github.com/kubernetes/kubernetes/blob/d92b99ea637ee67a5c925e5e628f5816a01162ac/pkg/kubelet/server/server.go#L441-L451 ↩
codeblock adding container_memory_kernel_usage metrics: https://github.com/kubernetes/kubernetes/blob/d92b99ea637ee67a5c925e5e628f5816a01162ac/vendor/github.com/google/cadvisor/metrics/prometheus.go#L371-L393 ↩
Cadvisor’s MemoryStats struct providing KernelUsage: https://github.com/google/cadvisor/blob/5bd422f9e1cea876ee9d550f2ed95916e1766f1a/info/v1/container.go#L430-L432 ↩
Cadvisor’s setMemoryStats() function, setting value for KernelUsage: https://github.com/google/cadvisor/blob/5bd422f9e1cea876ee9d550f2ed95916e1766f1a/container/libcontainer/handler.go#L799-L803 ↩