slabtop - to check kernel memory usage, and kubelet's container_memory_kernel_usage metrics

kubernetes

Today, I learnt about slabtop¹, a command line utility to check the memory used by kernel
(or as its man page says, it displays the kernel slab cache information in real time²).

(Logging for the future me)

So, what led to me learning about slabtop? 🙂

Jason Braganza taught me this, while we were preparing for our upcoming conference talk (on Kubernetes Metrics)!

Precisely, following is the metrics (exposed by kubelet component within a kubernetes cluster) that led to the discussion.

# HELP container_memory_kernel_usage Size of kernel memory allocated in bytes.
# TYPE container_memory_kernel_usage gauge
container_memory_kernel_usage{container="",id="/",image="",name="",namespace="",pod=""} 0 1732452865827

And how kubelet gets this “kernel memory allocation” information and feeds to the container_memory_kernel_usage metrics?

The answer is (at least to the best of my understanding) –

The kubelet’s server package imports “github.com/google/cadvisor/metrics”³ (aka the cadvisor/metrics) module.

This cadvisor/metrics go module provides a NewPrometheusCollector() function (which kubelet uses here⁴).

The NewPrometheusCollector() function take includedMetrics as one of its many parameters.

  r.RawMustRegister(metrics.NewPrometheusCollector(prometheusHostAdapter{s.host}, containerPrometheusLabelsFunc(s.host), includedMetrics, clock.RealClock{}, cadvisorOpts))

And when this includedMetrics contains cadvisormetrics.MemoryUsageMetrics (which it does in the case in question, check here⁵),

	includedMetrics := cadvisormetrics.MetricSet{
		...
		cadvisormetrics.MemoryUsageMetrics:  struct{}{},
		...
	}

then NewPrometheusCollector() function exposes the container_memory_kernel_usage⁶ metrics.

func NewPrometheusCollector(i infoProvider, f ContainerLabelsFunc, includedMetrics container.MetricSet, now clock.Clock, opts v2.RequestOptions) *PrometheusCollector {
        ...
        ...
	if includedMetrics.Has(container.MemoryUsageMetrics) {
		c.containerMetrics = append(c.containerMetrics, []containerMetric{
		       ...
		       ...
		       {
				name:      "container_memory_kernel_usage",
				help:      "Size of kernel memory allocated in bytes.",
				valueType: prometheus.GaugeValue,
				getValues: func(s *info.ContainerStats) metricValues {
					return metricValues
				},
			},
        ...
        ...

And as we see above in the definition of container_memory_kernel_usage metrics,
the valueType is prometheus.Gaugevalue (so its a gauge type metrics),
and the value is value: float64(s.Memory.KernelUsage) where KernelUsage is defined here⁷ and interpreted here⁸.

I still feel I can go further down a few more steps to find out the true source of this information, but that’s all for now.