[KUBE-APISERVER] Error from server (RequestEntityTooLarge Request entity too large: limit is 3145728

What happened?

I'm trying to push a large .yaml file (over 3 MB, less than 4 MB) with kubectl.
kubectl apply -f viya4-sasdeployment.yaml -n sasoperator
In response I've got :
Error from server (RequestEntityTooLarge): error when creating "viya4-sasdeployment.yaml": Request entity too large: limit is 3145728
To get proper debug I've done this with verbosity level 7 do determine what service is a bottleneck.

root@vmkub01:/home/gabos# kubectl apply -f viya4-sasdeployment.yaml -n sasoperator -v 7
I1024 15:26:54.488727 3449581 loader.go:372] Config loaded from file: /root/.kube/config
I1024 15:26:54.490443 3449581 round_trippers.go:432] GET https://10.0.110.99:6443/openapi/v2?timeout=32s
I1024 15:26:54.490463 3449581 round_trippers.go:438] Request Headers:
I1024 15:26:54.490481 3449581 round_trippers.go:442] User-Agent: kubectl/v1.21.5 (linux/amd64) kubernetes/aea7bba
I1024 15:26:54.490494 3449581 round_trippers.go:442] Accept: application/[email protected]+protobuf
I1024 15:26:54.543938 3449581 round_trippers.go:457] Response Status: 200 OK in 53 milliseconds
I1024 15:26:55.184873 3449581 round_trippers.go:432] GET https://10.0.110.99:6443/api/v1/namespaces/sasoperator/secrets/sas-viya
I1024 15:26:55.186073 3449581 round_trippers.go:438] Request Headers:
I1024 15:26:55.186107 3449581 round_trippers.go:442] Accept: application/json
I1024 15:26:55.186140 3449581 round_trippers.go:442] User-Agent: kubectl/v1.21.5 (linux/amd64) kubernetes/aea7bba
I1024 15:26:55.191114 3449581 round_trippers.go:457] Response Status: 200 OK in 4 milliseconds
I1024 15:26:55.198460 3449581 round_trippers.go:432] PATCH https://10.0.110.99:6443/api/v1/namespaces/sasoperator/secrets/sas-viya?fieldManager=kubectl-client-side-apply
I1024 15:26:55.198511 3449581 round_trippers.go:438] Request Headers:
I1024 15:26:55.198539 3449581 round_trippers.go:442] Accept: application/json
I1024 15:26:55.198562 3449581 round_trippers.go:442] Content-Type: application/strategic-merge-patch+json
I1024 15:26:55.198585 3449581 round_trippers.go:442] User-Agent: kubectl/v1.21.5 (linux/amd64) kubernetes/aea7bba
I1024 15:26:55.208301 3449581 round_trippers.go:457] Response Status: 200 OK in 9 milliseconds
I1024 15:26:55.260091 3449581 round_trippers.go:432] GET https://10.0.110.99:6443/apis/orchestration.sas.com/v1alpha1/namespaces/sasoperator/sasdeployments/sas-viya
I1024 15:26:55.260157 3449581 round_trippers.go:438] Request Headers:
I1024 15:26:55.260193 3449581 round_trippers.go:442] Accept: application/json
I1024 15:26:55.260206 3449581 round_trippers.go:442] User-Agent: kubectl/v1.21.5 (linux/amd64) kubernetes/aea7bba
I1024 15:26:55.267102 3449581 round_trippers.go:457] Response Status: 404 Not Found in 6 milliseconds
I1024 15:26:55.267538 3449581 round_trippers.go:432] GET https://10.0.110.99:6443/api/v1/namespaces/sasoperator
I1024 15:26:55.267555 3449581 round_trippers.go:438] Request Headers:
I1024 15:26:55.267564 3449581 round_trippers.go:442] Accept: application/json
I1024 15:26:55.267573 3449581 round_trippers.go:442] User-Agent: kubectl/v1.21.5 (linux/amd64) kubernetes/aea7bba
I1024 15:26:55.270827 3449581 round_trippers.go:457] Response Status: 200 OK in 3 milliseconds
I1024 15:26:55.331689 3449581 round_trippers.go:432] POST https://10.0.110.99:6443/apis/orchestration.sas.com/v1alpha1/namespaces/sasoperator/sasdeployments?fieldManager=kubectl-client-side-apply
I1024 15:26:55.331778 3449581 round_trippers.go:438] Request Headers:
I1024 15:26:55.331790 3449581 round_trippers.go:442] Accept: application/json
I1024 15:26:55.331808 3449581 round_trippers.go:442] Content-Type: application/json
I1024 15:26:55.331835 3449581 round_trippers.go:442] User-Agent: kubectl/v1.21.5 (linux/amd64) kubernetes/aea7bba
I1024 15:26:55.347992 3449581 round_trippers.go:457] Response Status: 413 Request Entity Too Large in 16 milliseconds
I1024 15:26:55.348328 3449581 helpers.go:216] server response object: [{
"kind": "Status",
"apiVersion": "v1",
"metadata": {},
"status": "Failure",
"message": "error when creating "viya4-sasdeployment.yaml": Request entity too large: limit is 3145728",
"reason": "RequestEntityTooLarge",
"code": 413
}]
F1024 15:26:55.348394 3449581 helpers.go:115] Error from server (RequestEntityTooLarge): error when creating "viya4-sasdeployment.yaml": Request entity too large: limit is 3145728

In my opinion the limit is hardcoded and I cannot change this.
As you can probably see, I'm trying to push a SAS Viya 4 deployment which generates files over 3 MB.

There is no way to split the file in smaller ones, there are 2 main sections "secret" and "deployment"
Secret is about 4-5 KB.

Is there any workaround ?

What did you expect to happen?

To allow apply files bigger than 3 MB

How can we reproduce it (as minimally and precisely as possible)?

Try to push random .yaml file with size 3MB + via kubectl

Anything else we need to know?

No response

Kubernetes version

Details

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.5", GitCommit:"aea7bbadd2fc0cd689de94a54e5b7b758869d691", GitTreeState:"clean", BuildDate:"2021-09-15T21:10:45Z", GoVersion:"go1.16.8", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.5", GitCommit:"aea7bbadd2fc0cd689de94a54e5b7b758869d691", GitTreeState:"clean", BuildDate:"2021-09-15T21:04:16Z", GoVersion:"go1.16.8", Compiler:"gc", Platform:"linux/amd64"}

Cloud provider

Details

Self-Hosted Ubuntu 20.04 LTS

OS version

Details

# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here

Install tools

Details

Container runtime (CRI) and and version (if applicable)

Details

Related plugins (CNI, CSI, ...) and versions (if applicable)

Details

this is by design and not a bug. the limit is hardcoded i don't think there are any plans to increase it or make it configurable.

/kind feature
/remove-kind bug
/sig api-machinery

Sorry, 1st time on github :|

@neolit123

+1 to @neolit123 's comment.

/triage accepted

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue or PR as fresh with /remove-lifecycle stale
Mark this issue or PR as rotten with /lifecycle rotten
Close this issue or PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue or PR as fresh with /remove-lifecycle rotten
Close this issue or PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Reopen this issue or PR with /reopen
Mark this issue or PR as fresh with /remove-lifecycle rotten
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

@k8s-triage-robot

@k8s-triage-robot: Closing this issue.

Details

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied

After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied

After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Reopen this issue or PR with /reopen

Mark this issue or PR as fresh with /remove-lifecycle rotten

Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

I'm facing the same thing and, although regarding the configmaps size limitation I found something documented, I didn't find anything documented about the other Kuberntes yaml manifests.

So is this documented somewhere? And what would be the source cause of this limitation?

@mrbobbytables

As Adam Acosta answered on slack:

It's an etcd limit. See https://github.com/etcd-io/etcd/blob/v3.5.11/server/embed/config.go#L59. Kubernetes allows double this to account for the fact that the request to the API server may be in json and it get converted to a more compact protobuf format before sending to etcd. That fact is documented in the code here but could potentially stand to be added to the website somewhere, but given the issue hit triage three years ago then got closed by a stale bot for lack of contributors, your best bet of that happening is almost certainly updating the docs yourself and submitting a pull request.

Thank you Adam Acosta.
Thanks also to @mrbobbytables for replying on the community forum.

Navigation Menu

Repository navigation

[KUBE-APISERVER] Error from server (RequestEntityTooLarge Request entity too large: limit is 3145728 #105863

What happened?

What did you expect to happen?

How can we reproduce it (as minimally and precisely as possible)?

Anything else we need to know?

Kubernetes version

Cloud provider

OS version

Install tools

Container runtime (CRI) and and version (if applicable)

Related plugins (CNI, CSI, ...) and versions (if applicable)

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[KUBE-APISERVER] Error from server (RequestEntityTooLarge Request entity too large: limit is 3145728 #105863

Description

What happened?

What did you expect to happen?

How can we reproduce it (as minimally and precisely as possible)?

Anything else we need to know?

Kubernetes version

Cloud provider

OS version

Install tools

Container runtime (CRI) and and version (if applicable)

Related plugins (CNI, CSI, ...) and versions (if applicable)

Activity

neolit123 commented on Oct 24, 2021

n2243 commented on Oct 24, 2021

caesarxuchao commented on Oct 26, 2021

k8s-triage-robot commented on Jan 24, 2022

k8s-triage-robot commented on Feb 23, 2022

k8s-triage-robot commented on Mar 25, 2022

k8s-ci-robot commented on Mar 25, 2022

MotaOcimar commented on Dec 7, 2023

MotaOcimar commented on Dec 8, 2023

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions