Accéder à un endpoint sans ressources protégé par RBAC

When installing a Kubernetes controller, it is possible that access to the /metrics endpoint may be restricted:

$ curl -k https://<controller-ip>:8443/metrics
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {

  },
  "status": "Failure",
  "message": "forbidden: User \"system:anonymous\" cannot get path \"/metrics\"",
  "reason": "Forbidden",
  "details": {

  },
  "code": 403
}

One of the most common cases is retrieving metrics from Kubernetes control plane components using Prometheus.

Here, the controller identifies us as an anonymous user system:anonymous. This is because we are making an API call without authentication, but it can also occur if we use a JWT token through a ServiceAccount.

curl -k -H "Authorization: Bearer <jwt>" https://<controller-ip>:8443/metrics
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "forbidden: User \"system:serviceaccount:my-namespace:my-serviceaccountname\" cannot get path \"/metrics\"",
  "reason": "Forbidden",
  "details": {},
  "code": 403
}

Our JWT token is identified as belonging to the service account my-serviceaccountname in the namespace my-namespace.

When this error occurs, the API you are querying is protected by access review through Kubernetes’ RBAC system. This review is either done in real-time (as part of the APIServer) or through SelfSubjectAccessReview (SSAR) (API).

Here is a brief summary of what will happen in terms of flow when calling /metrics:

The /metrics endpoint (as well as /healthz and /readyz) does not serve any Kubernetes resource. It is a controller-specific endpoint. Therefore, it will not be possible to access it through a Role since the concept of namespace does not apply to endpoints not related to a Kubernetes resource. However, it is possible to create a ClusterRole that authorizes access to this endpoint.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: controller-metrics
rules:
- apiGroups: [""]
  nonResourceURLs: ["/metrics"]
  verbs: ["get"]

We now link our ClusterRole with a ClusterRoleBinding:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: controller-metrics
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: controller-metrics
subjects:
- kind: ServiceAccount
  name: my-serviceaccountname
  namespace: my-namespace

An important point to note is that, since it’s a ClusterRole, granting this privilege to a service account means it will have access to this path on all APIs implementing Kubernetes RBAC with SubjectAccessReviews. Therefore, this is a privilege that should be granted carefully.

You now know how to handle this error case.