Node is a feature introduced in Kubernetes 1.27 that allows administrators to view the logs of services running directly on the node (the server) using the standard kubectl command.

Typically what you'd do to debug issues with the node itself, is that you would have to log in to the server using SSH or RDP, do some journalctling, do some grepping and then log out. This process is made even worse with Windows because you might need to go into the Event Viewer which is a pain etc. etc. etc. Now with the power of Node Log Query we can fetch system-level logs remotely. This is especially helpful for debugging Windows nodes or locked-down environments where direct access is restricted.

How does it work?

Instead of logging into the server, Kubernetes uses the Kubelet (the main agent on the node) to fetch logs for you.

• On Linux nodes: It uses the journalctl command to read system logs.
• On Windows nodes: It uses the Get-WinEvent cmdlet.
• For specific files: It can also read files directly from the /var/log/ directory.

You simply send a request to the Kubernetes API, which talks to the Kubelet, which then grabs the logs and sends them back to you.

Enabling Node Log Query

Since this was introduced as an Alpha feature in Kubernetes 1.27, it is not enabled by default. You must configure it on the nodes where you want to use it.

To enable the feature, you need to make changes to the Kubelet configuration on each node.‍

1. Update Kubelet Configuration

You need to turn on the NodeLogQuery feature gate and you must set two specific options to true in your Kubelet configuration file (often found in a ConfigMap named kubelet-config in the kube-system namespace:

Add the following configs:

2. Restart the Kubelet
For the changes to take effect, you must reload the configuration and restart the kubelet service on the node:

sudo systemctl daemon-reload

sudo systemctl restart kubelet

Note: These steps require administrative access to your Kubernetes cluster nodes.

Why is this useful?

• Security: You don't need to provide SSH or RDP access to developers or administrators just for debugging logs.
• Convenience: You can debug node issues from your local machine using the same tool (kubectl) you already use for pods.
• Windows Support: It simplifies debugging Windows nodes, where accessing logs can sometimes be more difficult.

‍

How to Use It

Once the feature is enabled (see the section below on requirements), you use kubectl with the --raw flag to query the node's API endpoint directly.

1. Basic Service Logs

To fetch logs for a specific service (like kubelet) from a node named node-1:

kubectl get --raw "/api/v1/nodes/node-1/proxy/logs/?query=kubelet"

2. Filter Logs by Keyword

To find specific information, you can filter the logs. For example, to find only lines containing the word "error":

kubectl get --raw "/api/v1/nodes/node-1/proxy/logs/?query=kubelet&pattern=error"

3. View Log Files

You can also fetch the contents of a specific log file from the /var/log/ directory:

kubectl get --raw "/api/v1/nodes/node-1/proxy/logs/?query=/var/log/syslog"

‍