Kubernetes has become the de facto orchestration platform for managing containerized applications, but with its widespread adoption, the security of Kubernetes clusters has come under greater scrutiny. Central to Kubernetes’ architecture is
etcd, a highly-available key-value store used to persist the cluster’s state and its configuration details. While
etcd is essential for the Kubernetes cluster’s functionality, it also presents a tantalizing target for attackers. A new research shows how a compromised
etcd can lead to full control over the cluster, allowing unauthorized changes to resources, tampering with operations, and potentially leading to data breaches or service disruptions. Kubernetes architecture is divided into two main parts: the control-plane and the nodes. The control-plane acts as the central hub and includes components like the kube-apiserver (the brain of the cluster), scheduler (which assigns pods to nodes), control manager (which manages the status of various cluster elements), and etcd (a key-value store for cluster data). Nodes contain components like kubelet (which ensures pods are running correctly) and kube-proxy (which connects services to the network).
Etcd is more than just a storage component in Kubernetes; it’s a critical part of the architecture. It’s a key-value database that stores all the cluster’s information. The data in etcd is stored using a serialization format called Protobuf, developed by Google for efficient data exchange between systems. Kubernetes uses Protobuf to serialize different kinds of data, such as pods and roles, each requiring different parameters and definitions.
The research describes a tool called auger, which can serialize and deserialize data stored in etcd into more readable formats like YAML and JSON. NCC Group has developed a wrapper for auger called kubetcd to showcase how a compromised etcd can be critical.
However, exploiting etcd has limitations. You’d need root access to the host running etcd and have the necessary certificates for authentication. Moreover, this technique mainly applies to self-managed Kubernetes environments, not managed ones offered by cloud providers.
Direct access to etcd could be used to tamper with Kubernetes resources, like changing the startup date of a pod or creating inconsistencies that make pods difficult to manage.
Direct access to
etcd, the key-value store for Kubernetes, could allow an attacker to make unauthorized modifications to the cluster state, which could lead to various security issues. Here are some examples of how this could be exploited:
Changing Pod Timestamps:
- Attackers with access to
etcdcould alter the creation timestamps of pods. This could be used to disguise malicious pods as long-running, trusted processes.
bash kubetcd create pod nginx -t nginx --time 2000-01-31T00:00:00Z
This command sets the timestamp of an nginx pod to January 31, 2000, making it appear as if it has been running for over 20 years.
- By changing the path where a pod’s data is stored in
etcd, an attacker could prevent the pod from being deleted by the regular Kubernetes API commands.
bash kubetcd create pod maliciouspod -t nginx -p randomentry
This command creates a pod and stores its data under a different path, making it difficult for Kubernetes to manage or delete it.
Creating Semi-hidden Pods:
- Attackers could manipulate the namespace entries in
etcdto run pods in a namespace that does not match their manifest. This can cause confusion and make pods difficult to manage.
bash kubetcd create pod hiddenpod -t nginx -n invisible --fake-ns
This command creates a pod that appears to run in the
defaultnamespace but is actually associated with the
etcd. This pod will not be listed under the default namespace, making it semi-hidden.
Bypassing Admission Controllers:
- Admission Controllers enforce security policies in Kubernetes. By injecting resources directly into
etcd, an attacker can bypass these controls and deploy privileged pods that could compromise the cluster.
bash kubetcd create pod privilegedpod -t nginx -n restricted-ns -P
This command injects a privileged pod into a namespace that is supposed to be restricted by Pod Security Admission (PSA) policies.
Tampering with Cluster Roles and Role Bindings:
- Attackers can modify roles and role bindings directly in
etcdto escalate privileges.
bash kubetcd modify rolebinding admin-binding --clusterrole=cluster-admin --user=attacker
This hypothetical command changes a role binding to give the
attackeruser cluster-admin privileges.
These examples show the potential dangers if an attacker gains direct access to
etcd. They can make changes that are not subject to the usual Kubernetes API checks and balances, leading to unauthorized control over the cluster’s resources. This is why securing
etcd access is critical in a Kubernetes environment.
To mitigate the risks associated with
etcd and prevent the kinds of tampering mentioned earlier, several steps and best practices should be implemented:
- Restrict access to
etcdby implementing strong authentication and authorization mechanisms. Use TLS client certificates to secure communication with
- Regularly rotate
etcdaccess credentials and audit access logs to detect unauthorized access attempts.
- Limit network access to
etcdservers, ensuring that only specific, authorized components can communicate with
- Implement firewall rules or Kubernetes network policies to control the traffic towards
- Enable encryption at rest for
etcdto protect sensitive data. Even if attackers gain physical access to the
etcdstorage, they should not be able to read the data without the encryption keys.
- Regularly back up the
etcddata store. In case of a breach, this allows the cluster to be restored to a known good state.
- Ensure backup integrity by verifying and testing backups periodically.
Monitoring and Auditing:
- Implement monitoring to detect abnormal activities, such as unexpected changes in
- Use tools like etcd’s built-in audit capabilities, Falco, or other intrusion detection systems to alert on suspicious behavior.
Least Privilege Principle:
- Apply the principle of least privilege to
etcdaccess. Ensure that only the necessary components have the minimum required access level to perform their functions.
- Regularly update
etcdto the latest version to mitigate vulnerabilities caused by software defects.
- Use Admission Controllers like OPA Gatekeeper or Kyverno to define and enforce policies that can help prevent the creation of unauthorized resources within Kubernetes.
Security Contexts and Policies:
- Apply Security Contexts and Pod Security Policies (or their successors, like Pod Security Admission) to enforce security-related settings in pods and prevent privilege escalation.
Disaster Recovery Plan:
- Have a disaster recovery plan in case
etcdis compromised. This plan should include steps to isolate affected systems, revoke compromised credentials, and restore from backups.
Education and Training:
- Train the operations team to understand the security risks associated with
etcdand Kubernetes, and how to apply best practices for securing the cluster.
By implementing these mitigations, organizations can significantly reduce the risk of unauthorized access and tampering with
etcd, thus securing their Kubernetes clusters. Mitigating the risks associated with
etcd ensures the integrity and reliability of Kubernetes clusters. By implementing industry best practices for security and maintaining a proactive stance on potential vulnerabilities, organizations can confidently deploy and manage their containerized workloads, keeping them secure in an ever-evolving threat landscape.
Information security specialist, currently working as risk infrastructure specialist & investigator.
15 years of experience in risk and control process, security audit support, business continuity design and support, workgroup management and information security standards.