Operations: Please allow setting nodeSelector, tolerations, and affinity on all Pods

Most companies with complex enough kubernetes deployments that have multiple different node groups require some sort of topology selection so pods will run on the correct group of nodes. Please allow setting nodeSelector, tolerations, and affinity on all Pods to accomplish this. This includes adding it to any CRDs that create pods so these values are settable.

https://kubernetes.io/docs/concepts/configuration/assign-pod-node/
https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/

See below for example:

apiVersion: v1
kind: Pod
metadata:
  name: sample
spec:
  nodeSelector:
    disktype: sad
  tolerations:
  - key: "example-key"
    operator: "Exists"
    effect: "NoSchedule"
  affinity:
    nodeAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 1
        preference:
          matchExpressions:
          - key: another-node-label-key
            operator: In
            values:
            - another-node-label-value

Related Feature Request: knative/eventing#2883

This is what is implemented in other operator:

NodeSelector: https://github.com/helm/charts/blob/master/stable/prometheus-operator/crds/crd-prometheus.yaml#L3223

Affinity: https://github.com/helm/charts/blob/master/stable/prometheus-operator/crds/crd-prometheus.yaml#L119

Tolerations: https://github.com/helm/charts/blob/master/stable/prometheus-operator/crds/crd-prometheus.yaml#L4523

The difference for knative operator could be that they should be defined within an array section for CRD, since we have got multiple deployments in knative.

We found PodNodeSelector and PodTolerationRestriction admission controllers.
https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/#podnodeselector
https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/#podtolerationrestriction
kubernetes/kubernetes#57424

But most hosted K8s control plane providers do not support them because their annotations are alpha.
https://docs.aws.amazon.com/eks/latest/userguide/platform-versions.html

We also found this open-source project that allows for setting node selector in namespace annotations using a webhook.
https://github.com/liangrog/admission-webhook-server

We will likely try to contribute the other tolerations, and affinity back to this project assuming the maintainer is open to it.

Adjust 0.14 serving manifest with Openshift specifica. (knative#5)

This issue is stale because it has been open for 90 days with no
activity. It will automatically close after 30 more days of
inactivity. Reopen the issue with /reopen. Mark the issue as
fresh by adding the comment /remove-lifecycle stale.

/remove-lifecycle stale

Is it possible to follow a pattern like in serving for setting node selectors on the serving pods such as activator, autoscale, controller, etc...

knative/serving#1816

This issue is stale because it has been open for 90 days with no
activity. It will automatically close after 30 more days of
inactivity. Reopen the issue with /reopen. Mark the issue as
fresh by adding the comment /remove-lifecycle stale.

Bump

@AceHack

@AceHack you mean setting tolerations and node selectors for the knative serving components (autoscaler, activator etc.) via the KnativeServing CRD from the Operator, correct?

This issue is stale because it has been open for 90 days with no
activity. It will automatically close after 30 more days of
inactivity. Reopen the issue with /reopen. Mark the issue as
fresh by adding the comment /remove-lifecycle stale.

I think we should start with a global config like current spec.high-availability.
I am thinking that we can create spec.global then add some configurations under it like:

apiVersion: operator.knative.dev/v1alpha1
kind: KnativeServing
metadata:
  name: knative-serving
  namespace: knative-serving
spec:
  global:
    nodeSelector:
      environment: dev

Users might want to customize for a specific deployment but we should start with this global one.
If no objections or other opinions, I will start creating the pull request.

@nak3

@nak3 shouldn't this be part of the existing deployments section maybe?

@markusthoemmes

@markusthoemmes Ah yes, deployments section would be good.
So it will be needed to configure each deployment like:

apiVersion: operator.knative.dev/v1alpha1
kind: KnativeServing
metadata:
  name: knative-serving
  namespace: knative-serving
spec:
  deployments:
  - name: webhook
    nodeSelector:
      environment: dev
  - name: controller
    nodeSelector:
      environment: dev

I guess we could add a global or all convenience section to deployments, which'd still allow to set them to all at once.

Umm... The deployments is a []array type so I think we cannot add the global or all section there?

operator/pkg/apis/operator/v1alpha1/common.go

Line 134 in 1bbdd90

DeploymentOverride []DeploymentOverride `json:"deployments,omitempty"`

operator/pkg/apis/operator/v1alpha1/common.go

Line 220 in 1bbdd90

type DeploymentOverride struct {

Allow to set NodeSelector via `spec.deployments.nodeSelector` (#658)

@nak3

Hello @nak3 @evankanderson guys,

Do you have any ETA to allow us put tolerations in the Knative operator?

For us it's blocking because we have the same issue of @AceHack

Thanks

@isaac88

@isaac88 Thank you for the information. I think we are alright to add it. I sent the PR #747 so it will be available on the next release (v0.26) if it was merged.

Allow to set Tolerations via `spec.deployments.tolerations` (#747)

All three nodeSelector, tolerations and affinity are now implemented. (tolerations and affinity will be available after 0.26+).

This issue is stale because it has been open for 90 days with no
activity. It will automatically close after 30 more days of
inactivity. Reopen the issue with /reopen. Mark the issue as
fresh by adding the comment /remove-lifecycle stale.

Allow to set TopologySpreadConstraints via `spec.deployments.topology…

Navigation Menu

Repository navigation

Operations: Please allow setting nodeSelector, tolerations, and affinity on all Pods #5

5 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Operations: Please allow setting nodeSelector, tolerations, and affinity on all Pods #5

Description

Activity

houshengbo commented on Apr 8, 2020

AceHack commented on Apr 21, 2020

github-actions commented on Sep 17, 2020

AceHack commented on Sep 17, 2020

AceHack commented on Sep 22, 2020

github-actions commented on Dec 21, 2020

AceHack commented on Dec 21, 2020

adamkgray commented on Jan 26, 2021

github-actions commented on Apr 27, 2021

5 remaining items

nak3 commented on Jun 9, 2021

markusthoemmes commented on Jun 9, 2021

nak3 commented on Jun 9, 2021

markusthoemmes commented on Jun 9, 2021

nak3 commented on Jun 9, 2021

isaac88 commented on Aug 31, 2021

nak3 commented on Sep 1, 2021

nak3 commented on Sep 16, 2021

github-actions commented on Dec 16, 2021

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions