Repository navigation
Error Forbidden: unknown while running kubectl wait #78227
Description
What happened:
I gave a user ([email protected]) very limited permissions in the kube-system namespace, specifically:
- apiGroups: ["extensions"]
resources: ["deployments", "deployments/scale"]
verbs: ["get", "update"]
- apiGroups: [""]
resources: ["pods"]
verbs: ["list", "get"]
I then ran:
kubectl -n kube-system scale deployment my-deployment --replicas=0 --as [email protected] && kubectl wait -n kube-system --for=delete pods -l k8s-app=my-deployment --as [email protected]
This was the output:
deployment.extensions/my-deployment scaled
Error from server (Forbidden): unknown
What you expected to happen:
I would have expected the Forbidden error to describe what permission I lack in order to run the kubectl wait command. In particular this is "watch" on pods, I discovered through trial and error.
How to reproduce it (as minimally and precisely as possible):
Apply the following roles/rolebindings:
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
namespace: kube-system
name: deployment-scaler
rules:
- apiGroups: ["extensions"]
resources: ["deployments", "deployments/scale"]
verbs: ["get", "update"]
- apiGroups: [""]
resources: ["pods"]
verbs: ["list", "get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: scale-deployments
namespace: kube-system
subjects:
- kind: User
name: [email protected]
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: Role
name: deployment-scaler
apiGroup: rbac.authorization.k8s.io
Create a deployment in kube-system called my-deployment with 1 available replica.
Then run
kubectl -n kube-system scale deployment my-deployment --replicas=0 --as [email protected] && kubectl wait -n kube-system --for=delete pods -l k8s-app=my-deployemnt --as [email protected]
Anything else we need to know?:
Environment:
- Kubernetes version (use
kubectl version):
Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.2", GitCommit:"66049e3b21efe110454d67df4fa62b08ea79a19b", GitTreeState:"clean", BuildDate:"2019-05-16T16:23:09Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"12+", GitVersion:"v1.12.7-gke.17", GitCommit:"b1adfec15b29a53b3dcf4af600952861b5819ede", GitTreeState:"clean", BuildDate:"2019-05-01T18:59:13Z", GoVersion:"go1.10.8b4", Compiler:"gc", Platform:"linux/amd64"} - Cloud provider or hardware configuration:
GKE 1.12.7, 3 node cluster - OS (e.g:
cat /etc/os-release):
PRETTY_NAME="Debian GNU/Linux buster/sid"
NAME="Debian GNU/Linux"
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/" - Kernel (e.g.
uname -a):
Linux 4.19.28-2rodete1-amd64 Unit test coverage in Kubelet is lousy. (~30%) #1 SMP Debian 4.19.28-2rodete1 (2019-03-18 > 2018) x86_64 GNU/Linux - Install tools:
- Network plugin and version (if this is a network-related bug):
- Others:
Activity
brycecr commentedon May 23, 2019
@kubernetes/sig-cli
pswica commentedon Jun 15, 2019
I can confirm, on test, that this issue exists. It ends up being pretty difficult, in my opinion.
As stated above, it only affects pods and not deployments.
Error is caused here: https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/client-go/rest/request.go#L1142
When I tested, I got
r.err == niland!errors.IsUnexpectedServerError(r.err). Even if I forced to the next block, I had issues decodingr.body.This was the error message:
no kind "PodList" is registered for version "v1" in scheme "k8s.io/client-go/dynamic/scheme.go:29"fejta-bot commentedon Sep 13, 2019
Issues go stale after 90d of inactivity.
Mark the issue as fresh with
/remove-lifecycle stale.Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with
/close.Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale
brycecr commentedon Sep 17, 2019
/remove-lifecycle stale
caarlos0 commentedon Oct 22, 2019
BTW: happens to jobs too.
had to give watch permission to jobs.
fejta-bot commentedon Jan 20, 2020
Issues go stale after 90d of inactivity.
Mark the issue as fresh with
/remove-lifecycle stale.Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with
/close.Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale
fejta-bot commentedon Feb 19, 2020
Stale issues rot after 30d of inactivity.
Mark the issue as fresh with
/remove-lifecycle rotten.Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with
/close.Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
fejta-bot commentedon Mar 20, 2020
Rotten issues close after 30d of inactivity.
Reopen the issue with
/reopen.Mark the issue as fresh with
/remove-lifecycle rotten.Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close
k8s-ci-robot commentedon Mar 20, 2020
@fejta-bot: Closing this issue.
Details
In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
lexsys27 commentedon Nov 27, 2020
I see this issue in
1.18.8fornodes.SA has following permissions:
Script should wait until node is ready:
Instead I get
Error from server (Forbidden): unknownmessage. It is doesn't say what permissionskubectlmiss so is not very helpful.I believe this issue should be reopened.
alitoufighi commentedon Oct 3, 2021
It seems that the API verb to access for
waiting for a resource is not documented and not mentioned anywhere.YevheniiPokhvalii commentedon Feb 3, 2022
You need to add
get, list, watchto your role to avoid this error.