[{"content":"You know how to build a kubernetes operator? Cool. Now, it\u0026rsquo;s time to get serious!\nLet\u0026rsquo;s write some tests to use our operator in production!\nIn my previous article, I showed how to build a Kubernetes operator in about ten minutes. I\u0026rsquo;ve also described the functioning of an operator, the custom resource definition and custom resources, the controllers, and the manager. If you want to learn more about these concepts, I highly recommend you read the article.\nBut now it\u0026rsquo;s time to write some tests to use our operator in production! It\u0026rsquo;s a subject in its own right that is less complicated than it seems. Most people don\u0026rsquo;t like writing tests, but they are necessary. They help to improve code quality, detect bugs, and save time and money. So, let\u0026rsquo;s do some testing!\n✨ The article was updated to use the latest version of kubebuilder (v4.2.0), released in August 2024!\nThe different types of tests There are three major types of testing in software development: unit, integration, and end-to-end testing. It is even more complex than that in theory, but it gives an idea of what we\u0026rsquo;ll see in this article.\nUnit tests are used to test small pieces of code, for example, methods or functions. Integration tests are used to test the integration of the different parts of an application. For example, the functioning of an operator in a Kubernetes environment with an API server and other resources. End-to-end tests, also known as e2e, aim to simulate a user\u0026rsquo;s step-by-step experience. They should cover the main functionalities of the application. In the practical work that follows, we\u0026rsquo;ll see how to write unit tests in Go as well as integration tests for Kubernetes controllers. It\u0026rsquo;s interesting, you\u0026rsquo;ll see! Practical work 1. Set up your environment I choose to use the Kubebuilder framework to build Kubernetes operators. It\u0026rsquo;s very easy to use, the documentation is easy to read, and it\u0026rsquo;s a battle-tested product. It\u0026rsquo;s one of the two most popular Go frameworks to build operators, along with Operator SDK.\nI assume you already have the necessary tools to design an operator (go, docker, kubectl, kubebuilder and a small local Kubernetes cluster). But if this is not the case, I strongly recommend that you follow the following installation steps so that you can follow along with me during this tutorial.\nTo focus only on the testing part, I have prepared a simple Kubernetes operator, designed with Kubebuilder. This practical work will be to write tests to validate that the operator\u0026rsquo;s code meets our expectations and ensure that it does not contain any bugs. I\u0026rsquo;m going to ask you to clone the project and move to the operator-v2 directory so we all start from the same base.\n$ git clone git@github.com:leovct/kubernetes-operator-tutorial.git \\ \u0026amp;\u0026amp; cd kubernetes-operator-tutorial/operator-v2 If you try to run tests, you\u0026rsquo;ll see that the coverage equals 0%. Indeed, we haven\u0026rsquo;t written any tests yet! But this is going to change!\n$ make test ... KUBEBUILDER_ASSETS=\u0026#34;/Users/leovct/Documents/projects/kubernetes-operator-tutorial/operator-v2/bin/k8s/1.28.0-darwin-amd64\u0026#34; go test ./... -coverprofile cover.out ? my.domain/tutorial/api/v1 [no test files] ? my.domain/tutorial/cmd [no test files] ? my.domain/tutorial/internal/color [no test files] ok my.domain/tutorial/internal/controller 2.129s coverage: 0.0% of statements 2. Some context on the operator The \u0026ldquo;Foo\u0026rdquo; operator you cloned is similar to the one I built in my \u0026ldquo;Build a Kubernetes operator in 10 minutes\u0026rdquo; article. It has no real purpose except to show how you can use operators to perform simple tasks in Kubernetes.\nHow does the operator work?\nBecause a diagram explains ideas way better than three paragraphs, here it is.\nFig 2. Overview of the operator\u0026rsquo;s functioning\nFirst, there is the Foo Custom Resource Definition or CRD (see api/v1/foo_types.go). It has a name field in its specification which refers to the name of the friend Foo is looking for. If Foo finds a friend (a Pod with the same name as his friend), its happy status will be set to true. A small addition since the previous tutorial, the status of Foo also contains a color field, determined according to its name and the namespace in which it evolves.\nSecond, there is the Foo Controller (see controllers/foo_controller.go). It fetches the Foo resource that triggered the reconciliation request to get the name of Foo\u0026rsquo;s friend. Then, it lists all the pods with the same name as Foo\u0026rsquo;s friend. If at least one friend has been found, it updates Foo\u0026rsquo;s happy status to true, else we set it to false. It also updates Foo\u0026rsquo;s color status.\nNote that the controller also reacts to Pod events (see mapPodsReqToFooReq). Indeed, if a new pod is created, we want the Foo resource to be able to update its status accordingly. This method will be triggered each time a Pod event happens (creation, update, or deletion). It then triggers a reconciliation loop of the Foo controller only if the Pod\u0026rsquo;s name is a \u0026ldquo;friend\u0026rdquo; of one of the Foo custom resources deployed in the cluster.\nNow that we know how the operator works, we can get on with testing it.\n3. Unit tests At this stage, we will test small pieces of code like methods and functions. Of course, we won\u0026rsquo;t test the controller Reconcile method here because it involves setting up a testing Kubernetes environment with a local Kubernetes API server, the CRD, the controller, the manager, etc. We\u0026rsquo;ll see this in the 4. Integration tests section instead.\nWhat are we going to test then? If you take the time to read the code of the Foo controller, you\u0026rsquo;ll notice that I use the ConvertStrToColor function from the color package to update Foo\u0026rsquo;s color status. Let\u0026rsquo;s write the unit tests of this simple method!\nOur goal here is to make sure that the function returns what is expected given the input parameters. Here are the three tests I\u0026rsquo;d like to write to test this function:\nConvert an empty string to a colour of the colour wheel. Convert a short string to a colour of the colour wheel. Convert a very long string (with numbers and dashes) to a colour of the colour wheel. To write unit tests in Go, you create a list of tests describing the name, input parameters and expected result. Then you run a loop and compare the function\u0026rsquo;s output with the expected result. Here is my test function to check that ConvertStrToColor works correctly:\nNow, if you attempt to run the tests a second time, you\u0026rsquo;ll see that the code coverage of the color package is equal to 100%! Awesome, we did it! The function works as expected.\n$ make test ... KUBEBUILDER_ASSETS=\u0026#34;/Users/leovct/Documents/projects/kubernetes-operator-tutorial/operator-v2/bin/k8s/1.28.0-darwin-amd64\u0026#34; go test ./... -coverprofile cover.out ? my.domain/tutorial/api/v1 [no test files] ? my.domain/tutorial/cmd [no test files] ok my.domain/tutorial/internal/color 0.319s coverage: 100.0% of statements ok my.domain/tutorial/internal/controller 2.109s coverage: 0.0% of statements Now, let\u0026rsquo;s test the operator\u0026rsquo;s reconciliation loop.\n4. Integration tests Kubebuilder provided a test boilerplate (see controllers/suite_test.go) that sets up a testing environment with a local Kubernetes API server. The only thing we need to do is first instantiate and run the controller and then write tests with Ginkgo (framework used by Kubebuilder) to verify that our operator behaves well when it evolves in a Kubernetes environment.\nOf course, this test environment is very simple and often does not represent the environment of a production cluster. This is why running the operator\u0026rsquo;s tests in an existing cluster is also possible to get closer to a real production environment. You can learn more about that here.\nFirst, we\u0026rsquo;ll need to instantiate and start the Foo controller. To do this, we need to create another client. Why do we need two clients? Because when running tests, you want to assert against the live state of the API server. If you use the client from the manager in your tests, you\u0026rsquo;d end up asserting against the content of the cache instead. First, this is slower. Second, this can introduce flaws. That\u0026rsquo;s why we\u0026rsquo;ve got two clients: k8sclient, the client we\u0026rsquo;ll use in our tests, and k8sManager.getClient(), the manager\u0026rsquo;s client.\nHere is the modified source code of the test environment setup.\nNow that our test environment is properly set up, we can write integration tests! Let\u0026rsquo;s create a new test file called foo_controller_test.go under the controllers folder. In this file, we\u0026rsquo;ll define our tests using the Ginkgo framework. Here\u0026rsquo;s a simple test example that creates two Foo custom resources and ensures that the resources have been created.\nGreat! Now let\u0026rsquo;s write a test that creates a pod with the same name as one of the Foo custom resources\u0026rsquo; friends. The controller should update the status of the custom resource and set it to true. The other custom resource should still have its status set to false.\nLet\u0026rsquo;s write another test that updates the name of the second Foo custom resource\u0026rsquo;s friend to the name of the previously created pod. The controller should update the status of the custom resource and set it to true. Again, the other custom resource should still have its status set to false.\nOur final test for today will be to delete the pod we created and check that the controller updates the status of all the custom resources to false.\nThese are fairly simple tests, but they show you the possibilities. You can do very complex things using the various tools provided by Ginkgo. If you paid attention during the tutorial, you should have seen various instructions such as Expect, Eventually, or Consistently, which are very useful methods to test the behaviour of the operator and the resources running in Kubernetes.\nWe\u0026rsquo;ve written many great tests that cover our controller\u0026rsquo;s scope. Now it\u0026rsquo;s time to run them to see if the operator passes them all.\n$ make test ... KUBEBUILDER_ASSETS=\u0026#34;/Users/leovct/Documents/projects/kubernetes-operator-tutorial/operator-v2/bin/k8s/1.28.0-darwin-amd64\u0026#34; go test ./... -coverprofile cover.out ? my.domain/tutorial/api/v1 [no test files] ? my.domain/tutorial/cmd [no test files] ok my.domain/tutorial/internal/color 0.443s coverage: 100.0% of statements ok my.domain/tutorial/internal/controller 16.166s coverage: 81.8% of statements Nice, we\u0026rsquo;ve got no errors! In addition, we went from 0% coverage to over 82%! The last 18% corresponds to the parts of the code that we can\u0026rsquo;t test (or maybe using mocks but it would be a lot of work for not much). For example, when the controller can\u0026rsquo;t find the custom resource that triggered the reconciliation loop or when it can\u0026rsquo;t list the pods in the cluster. It doesn\u0026rsquo;t matter because we know that we have tested all the operator\u0026rsquo;s scope.\nHere\u0026rsquo;s the link to the GitHub repository if you need to browse the code. The operator-v2 directory contains the source code of the operator without the tests, while the operator-v2-with-tests directory contains the source code with all the tests.\nTo go further We\u0026rsquo;ve seen how to write unit tests and integration tests for Kubernetes controllers using the Ginkgo framework. It is a crucial step in developing Kubernetes operators as it allows for validating the correct functioning of the reconciliation logic.\nAlso, here\u0026rsquo;s a list of resources you can use to dig deeper into the subject.\nhttps://book.kubebuilder.io/cronjob-tutorial/writing-tests.html https://book.kubebuilder.io/reference/envtest.html http://onsi.github.io/ginkgo Have fun! :)\nNB: This article was published on Medium in Better Programming, you can find it here.\n","permalink":"https://leovct.github.io/posts/kubernetes/write-tests-for-operators/","summary":"\u003cp\u003eYou know how to build a kubernetes operator? Cool. Now, it\u0026rsquo;s time to get serious!\u003c/p\u003e\n\u003cp\u003eLet\u0026rsquo;s write some tests to use our operator in production!\u003c/p\u003e\n\u003cp\u003eIn my previous \u003ca href=\"https://leovct.github.io/posts/kubernetes/build-an-operator/\"\u003earticle\u003c/a\u003e, I showed how to build a Kubernetes operator in about ten minutes. I\u0026rsquo;ve also described the functioning of an operator, the custom resource definition and custom resources, the controllers, and the manager. If you want to learn more about these concepts, I highly recommend you read the article.\u003c/p\u003e","title":"🧪 Write tests for Kubernetes operators"},{"content":"You\u0026rsquo;re probably familiar with Kubernetes, but do you know what operators are, how they work, and how to build one? If you want to know more, you\u0026rsquo;ve come to the right place!\nKubernetes operators is a complicated subject but fortunately, since their creation in 2016, many tools have been developed to simplify the life of engineers.\nWithout further ado, let\u0026rsquo;s dive in and learn more about operators!\n✨ The article was updated to use the latest version of kubebuilder (v4.2.0), released in August 2024!\ntl;dr Kubernetes operators allow to incorporate custom logic into Kubernetes to automate a large number of tasks, beyond what the software can do natively. While one could build an operator from scratch, it\u0026rsquo;s highly recommended to use a framework like Kubebuilder or OperatorSDK, as shown in this article.\nWhat is an operator? Wait a minute, do you know what Kubernetes (or k8s) is? Just a quick reminder for everyone, it\u0026rsquo;s an \u0026ldquo;open source system to deploy, scale, and manage containerized applications anywhere\u0026rdquo; developed by Google Cloud.\nMost people use Kubernetes by deploying their applications using native resources such as pods, deployments, services, etc. However, it is possible to extend the capabilities of the software to incorporate its logic to meet specific needs. That\u0026rsquo;s where the operator comes into place.\nThe main goal of an operator is to translate an engineer\u0026rsquo;s logic into code in order to automate certain tasks beyond what Kubernetes can do natively.\nEngineers dealing with applications or services have a deep knowledge of how the system should behave, how it should be deployed, and how to react in case of a problem. The ability to encapsulate this technical knowledge in code and automate actions means less time is spent on repetitive tasks and more on important issues.\nFor example, one can imagine an operator deploying and maintaining tools such as MySQL, Elasticsearch, or Gitlab runners in Kubernetes. An operator could configure these tools, adjust the state of the system according to events and react to failures.\nSounds interesting, right? Let\u0026rsquo;s get our hands dirty.\nPractical work You could either build an operator from scratch using the controller-runtime project developed by Kubernetes or you could use one of the most popular frameworks to accelerate your development cycle and reduce the complexity (Kubebuilder or OperatorSDK). I\u0026rsquo;d choose the Kubebuilder framework because it\u0026rsquo;s very easy to use, the documentation is good, and it\u0026rsquo;s a battle-tested product. In any case, the two projects are currently being aligned to merge into a single project.\n1. Set up your environment You\u0026rsquo;ll need some tools to develop your operator. Here are the requisites:\ngo version v1.20.0+ docker version 17.03+ kubectl version v1.11.3+ You\u0026rsquo;ll also need access to a Kubernetes v1.11.3+ cluster. I highly suggest using kind to set up your own local k8s cluster, it\u0026rsquo;s very easy to use!\nWe can then install kubebuilder.\n$ curl -L -o kubebuilder \\ https://go.kubebuilder.io/dl/latest/$(go env GOOS)/$(go env GOARCH) \\ \u0026amp;\u0026amp; chmod +x kubebuilder \\ \u0026amp;\u0026amp; mv kubebuilder /usr/local/bin/ If everything goes fine, you should see a similar output (the version might have changed depending on when you read this article).\n$ kubebuilder version Version: main.version{KubeBuilderVersion:\u0026#34;3.13.0\u0026#34;, KubernetesVendor:\u0026#34;1.27.1\u0026#34;, GitCommit:\u0026#34;c8a7cc58eeb56586c019cf8845dad37286d077ff\u0026#34;, BuildDate:\u0026#34;2023-11-02T20:43:44Z\u0026#34;, GoOs:\u0026#34;darwin\u0026#34;, GoArch:\u0026#34;amd64\u0026#34;} Awesome, now we can get started!\n2. Create a simple operator Let\u0026rsquo;s do a little exercise: we\u0026rsquo;ll build a simple foo operator which has no real use, except to demonstrate the capabilities of an operator.\nInitialise a new project by running the following command. It will download the controller-runtime binary and scaffold a project that\u0026rsquo;s ready for us to customise.\n$ kubebuilder init --domain my.domain --repo my.domain/tutorial INFO Writing kustomize manifests for you to edit... INFO Writing scaffold for you to edit... INFO Get controller runtime: $ go get sigs.k8s.io/controller-runtime@v0.16.3 ... INFO Update dependencies: $ go mod tidy Next: define a resource with: $ kubebuilder create api Here\u0026rsquo;s the structure of the project (as you can notice, it\u0026rsquo;s a Go project):\n$ ls -la total 120 drwxr-xr-x 17 leovct staff 544 Nov 3 08:49 . drwxr-xr-x 10 leovct staff 320 Nov 3 08:49 .. -rw------- 1 leovct staff 120 Nov 3 08:49 .dockerignore -rw------- 1 leovct staff 384 Nov 3 08:49 .gitignore -rw------- 1 leovct staff 692 Nov 3 08:49 .golangci.yml -rw------- 1 leovct staff 1278 Nov 3 08:49 Dockerfile -rw------- 1 leovct staff 7966 Nov 3 08:49 Makefile -rw------- 1 leovct staff 508 Nov 3 08:49 PROJECT -rw------- 1 leovct staff 2419 Nov 3 08:49 README.md drwx------ 3 leovct staff 96 Nov 3 08:49 api drwxr-xr-x 3 leovct staff 96 Nov 3 08:50 bin drwx------ 3 leovct staff 96 Nov 3 08:49 cmd drwx------ 8 leovct staff 256 Nov 3 08:49 config -rw------- 1 leovct staff 3156 Nov 3 08:49 go.mod -rw-r--r-- 1 leovct staff 20603 Nov 3 08:49 go.sum drwx------ 3 leovct staff 96 Nov 3 08:49 hack drwx------ 3 leovct staff 96 Nov 3 08:49 internal Let\u0026rsquo;s go through the most important components of the operator:\nDockerfile is the container file used to build the manager\u0026rsquo;s image. Makefile contains handy helper commands. api/v1 contains the definition of the Foo CRD. cmd/main.go is the entry point of the project; it sets up and runs the manager. config/ contains the manifests to deploy the operator and CRDs in Kubernetes. internal/controller contains the logic of the Foo controller. Wait, what\u0026rsquo;s this controller component?!\nThis is going to be a bit theoretical. Hang on!\nAn operator is made of two components, a custom resource definition (CRD) and a controller.\nA CRD is a \u0026ldquo;Kubernetes custom type\u0026rdquo; or a blueprint of the resource, used to describe its specification and status. We can define instances of the CRD, called custom resources (or CR).\nFig 2. Custom Resource Definition (CRD) and Custom Resources (CR)\nThe controller (also called the control loop) continuously monitors the state of the cluster and, depending on events, makes changes. Its goal is to bring the current state of a resource towards its desired state, defined by the user in the specification of the custom resource.\nFig 3. High-level operation of a controller by Stefanie Lai\nIn general, a controller is specific to a type of resource but it can perform CRUD (Create, Read, Update and Delete) operations on a set of different resources.\nAn example of a controller, presented in the Kubernetes documentation, is the thermostat. When we set the temperature, we tell the thermostat the desired state. The actual state is determined by the actual temperature of the room. The thermostat then reacts to bring the actual state closer to the desired state by turning the heat on or off.\nWhat about the manager then? The goal of this component is to start all the various controllers and makes the set of control loops coexist. Let\u0026rsquo;s say you have two CRDs in your project. You\u0026rsquo;ll then have two controllers: one for each CRD. The manager will start these two controllers and make them coexist.\nIf you want more details on how operators work, you can leave questions in the comments or explore the list of resources provided at the end of the article. I will certainly do a detailed article on this concept in the future.\nNow that we know how an operator works, we can start to create one using the Kubebuilder framework. We\u0026rsquo;ll start by creating a new API (group/version) and a new Kind (CRD). Press yes when asked to create a CRD and a controller.\n$ kubebuilder create api --group tutorial --version v1 --kind Foo INFO Create Resource [y/n] y INFO Create Controller [y/n] y INFO Writing kustomize manifests for you to edit... INFO Writing scaffold for you to edit... INFO api/v1/foo_types.go INFO api/v1/groupversion_info.go INFO internal/controller/suite_test.go INFO internal/controller/foo_controller.go INFO Update dependencies: $ go mod tidy INFO Running make: $ make generate ... Next: implement your new API and generate the manifests (e.g. CRDs,CRs) with: $ make manifests This is where the fun starts now! We\u0026rsquo;ll customise the CRD and the controller to meet our needs. You\u0026rsquo;ll notice that two new folders have been created:\napi/v1 which contains our Foo CRD (see foo_types.go). internal/controllers which contain the Foo controller (see foo_controller.go). 3. Customize the CRD and the controller Here\u0026rsquo;s our lovely Foo CRD customised (see api/v1/foo_types.go). As I said previously, this CRD has no purpose. It simply shows how you can use operators to perform simple tasks in Kubernetes.\nThe Foo CRD has a name field in its specification which refers to the name of the friend Foo is looking for. If Foo finds a friend (a pod with the same name as his friend), its happy status will be set to true.\nNow, let\u0026rsquo;s implement the logic of the controller. Nothing very complicated here. We fetch the Foo resource that triggered the reconciliation request to get the name of Foo\u0026rsquo;s friend. Then, we list all the pods that have the same name as Foo\u0026rsquo;s friend. If we find one (or more), we update Foo\u0026rsquo;s happy status to true, else we set it to false.\nNote that the controller also reacts to Pod events (see mapPodsReqToFooReq). Indeed, if a new pod is created, we want the Foo resource to be able to update its status accordingly. This method will be triggered each time a Pod event happens (creation, update, or deletion). It then triggers a reconciliation loop of the Foo controller only if the name of the Pod is a \u0026ldquo;friend\u0026rdquo; of one of the Foo custom resources deployed in the cluster.\nA picture is worth more than 1000 words so here is an overview of how the operator works.\nFig 4. Overview of the operator\u0026rsquo;s functioning\nAnd now, here is the implementation of the controller.\nWe\u0026rsquo;re done editing the API definitions and the controller so we can run the following command to update the operator manifests. If you pay attention, you\u0026rsquo;ll see that some manifest files have been updated.\n$ make manifests /Users/leovct/Documents/projects/kubernetes-operator-tutorial/operator-v1/bin/controller-gen rbac:roleName=manager-role crd webhook paths=\u0026#34;./...\u0026#34; output:crd:artifacts:config=config/crd/bases 4. Run the controller I\u0026rsquo;m using a local Kubernetes cluster set up with kind, and I advise you to do the same. It\u0026rsquo;s very easy to use.\nFirst, we install the CRDs into the cluster.\n$ make install test -s /Users/leovct/Documents/projects/kubernetes-operator-tutorial/operator-v1/bin/controller-gen \u0026amp;\u0026amp; /Users/leovct/Documents/projects/kubernetes-operator-tutorial/operator-v1/bin/controller-gen --version | grep -q v0.13.0 || \\ GOBIN=/Users/leovct/Documents/projects/kubernetes-operator-tutorial/operator-v1/bin go install sigs.k8s.io/controller-tools/cmd/controller-gen@v0.13.0 ... /Users/leovct/Documents/projects/kubernetes-operator-tutorial/operator-v1/bin/controller-gen rbac:roleName=manager-role crd webhook paths=\u0026#34;./...\u0026#34; output:crd:artifacts:config=config/crd/bases /Users/leovct/Documents/projects/kubernetes-operator-tutorial/operator-v1/bin/kustomize build config/crd | kubectl apply -f - customresourcedefinition.apiextensions.k8s.io/foos.tutorial.my.domain created You can see that the Foo CRD has been created.\n$ kubectl get crds NAME CREATED AT foos.tutorial.my.domain 2023-11-03T08:14:04Z Then we run the controller in the terminal. Keep in mind that we can also deploy it as deployment in the Kubernetes cluster.\n$ make run test -s /Users/leovct/Documents/projects/kubernetes-operator-tutorial/operator-v1/bin/controller-gen \u0026amp;\u0026amp; /Users/leovct/Documents/projects/kubernetes-operator-tutorial/operator-v1/bin/controller-gen --version | grep -q v0.13.0 || \\ GOBIN=/Users/leovct/Documents/projects/kubernetes-operator-tutorial/operator-v1/bin go install sigs.k8s.io/controller-tools/cmd/controller-gen@v0.13.0 /Users/leovct/Documents/projects/kubernetes-operator-tutorial/operator-v1/bin/controller-gen rbac:roleName=manager-role crd webhook paths=\u0026#34;./...\u0026#34; output:crd:artifacts:config=config/crd/bases /Users/leovct/Documents/projects/kubernetes-operator-tutorial/operator-v1/bin/controller-gen object:headerFile=\u0026#34;hack/boilerplate.go.txt\u0026#34; paths=\u0026#34;./...\u0026#34; go fmt ./... go vet ./... go run ./cmd/main.go 2023-11-03T09:15:42+01:00 INFO setup starting manager 2023-11-03T09:15:42+01:00 INFO starting server {\u0026#34;kind\u0026#34;: \u0026#34;health probe\u0026#34;, \u0026#34;addr\u0026#34;: \u0026#34;[::]:8081\u0026#34;} 2023-11-03T09:15:42+01:00 INFO controller-runtime.metrics Starting metrics server 2023-11-03T09:15:42+01:00 INFO controller-runtime.metrics Serving metrics server {\u0026#34;bindAddress\u0026#34;: \u0026#34;:8080\u0026#34;, \u0026#34;secure\u0026#34;: false} 2023-11-03T09:15:42+01:00 INFO Starting EventSource {\u0026#34;controller\u0026#34;: \u0026#34;foo\u0026#34;, \u0026#34;controllerGroup\u0026#34;: \u0026#34;tutorial.my.domain\u0026#34;, \u0026#34;controllerKind\u0026#34;: \u0026#34;Foo\u0026#34;, \u0026#34;source\u0026#34;: \u0026#34;kind source: *v1.Foo\u0026#34;} 2023-11-03T09:15:42+01:00 INFO Starting EventSource {\u0026#34;controller\u0026#34;: \u0026#34;foo\u0026#34;, \u0026#34;controllerGroup\u0026#34;: \u0026#34;tutorial.my.domain\u0026#34;, \u0026#34;controllerKind\u0026#34;: \u0026#34;Foo\u0026#34;, \u0026#34;source\u0026#34;: \u0026#34;kind source: *v1.Pod\u0026#34;} 2023-11-03T09:15:42+01:00 INFO Starting Controller {\u0026#34;controller\u0026#34;: \u0026#34;foo\u0026#34;, \u0026#34;controllerGroup\u0026#34;: \u0026#34;tutorial.my.domain\u0026#34;, \u0026#34;controllerKind\u0026#34;: \u0026#34;Foo\u0026#34;} 2023-11-03T09:15:42+01:00 INFO Starting workers {\u0026#34;controller\u0026#34;: \u0026#34;foo\u0026#34;, \u0026#34;controllerGroup\u0026#34;: \u0026#34;tutorial.my.domain\u0026#34;, \u0026#34;controllerKind\u0026#34;: \u0026#34;Foo\u0026#34;, \u0026#34;worker count\u0026#34;: 1} As you can see, the manager started and then the Foo controller started. The controller is now running and listening to events!\n5. Test the controller To test that everything works properly, we\u0026rsquo;ll create two Foo custom resources and some pods just to see how the controller behaves. You can modify config/samples/tutorial_v1_foo.yaml.\nFirst, create the Foo custom resources manifests in config/samples and run the following command to create the resources in your local Kubernetes cluster.\n$ kubectl apply -k config/samples foo.tutorial.my.domain/foo-1 created foo.tutorial.my.domain/foo-2 created You should see that the controller triggered two reconciliation loops for each Foo custom resource creation event. You may wonder why two loops were triggered for each custom resource and not one, this is a more technical topic, I invite you to read this thread.\n2023-11-03T09:16:40+01:00 INFO reconciling foo custom resource {\u0026#34;controller\u0026#34;: \u0026#34;foo\u0026#34;, \u0026#34;controllerGroup\u0026#34;: \u0026#34;tutorial.my.domain\u0026#34;, \u0026#34;controllerKind\u0026#34;: \u0026#34;Foo\u0026#34;, \u0026#34;Foo\u0026#34;: {\u0026#34;name\u0026#34;:\u0026#34;foo-sample\u0026#34;,\u0026#34;namespace\u0026#34;:\u0026#34;default\u0026#34;}, \u0026#34;namespace\u0026#34;: \u0026#34;default\u0026#34;, \u0026#34;name\u0026#34;: \u0026#34;foo-sample\u0026#34;, \u0026#34;reconcileID\u0026#34;: \u0026#34;c98a2469-159f-4c3f-b9bb-aa02b6dd5bf9\u0026#34;} 2023-11-03T09:16:40+01:00 INFO foo\u0026#39;s happy status updated {\u0026#34;controller\u0026#34;: \u0026#34;foo\u0026#34;, \u0026#34;controllerGroup\u0026#34;: \u0026#34;tutorial.my.domain\u0026#34;, \u0026#34;controllerKind\u0026#34;: \u0026#34;Foo\u0026#34;, \u0026#34;Foo\u0026#34;: {\u0026#34;name\u0026#34;:\u0026#34;foo-sample\u0026#34;,\u0026#34;namespace\u0026#34;:\u0026#34;default\u0026#34;}, \u0026#34;namespace\u0026#34;: \u0026#34;default\u0026#34;, \u0026#34;name\u0026#34;: \u0026#34;foo-sample\u0026#34;, \u0026#34;reconcileID\u0026#34;: \u0026#34;c98a2469-159f-4c3f-b9bb-aa02b6dd5bf9\u0026#34;, \u0026#34;status\u0026#34;: false} 2023-11-03T09:16:40+01:00 INFO foo custom resource reconciled {\u0026#34;controller\u0026#34;: \u0026#34;foo\u0026#34;, \u0026#34;controllerGroup\u0026#34;: \u0026#34;tutorial.my.domain\u0026#34;, \u0026#34;controllerKind\u0026#34;: \u0026#34;Foo\u0026#34;, \u0026#34;Foo\u0026#34;: {\u0026#34;name\u0026#34;:\u0026#34;foo-sample\u0026#34;,\u0026#34;namespace\u0026#34;:\u0026#34;default\u0026#34;}, \u0026#34;namespace\u0026#34;: \u0026#34;default\u0026#34;, \u0026#34;name\u0026#34;: \u0026#34;foo-sample\u0026#34;, \u0026#34;reconcileID\u0026#34;: \u0026#34;c98a2469-159f-4c3f-b9bb-aa02b6dd5bf9\u0026#34;} 2023-11-03T09:16:40+01:00 INFO reconciling foo custom resource {\u0026#34;controller\u0026#34;: \u0026#34;foo\u0026#34;, \u0026#34;controllerGroup\u0026#34;: \u0026#34;tutorial.my.domain\u0026#34;, \u0026#34;controllerKind\u0026#34;: \u0026#34;Foo\u0026#34;, \u0026#34;Foo\u0026#34;: {\u0026#34;name\u0026#34;:\u0026#34;foo-sample\u0026#34;,\u0026#34;namespace\u0026#34;:\u0026#34;default\u0026#34;}, \u0026#34;namespace\u0026#34;: \u0026#34;default\u0026#34;, \u0026#34;name\u0026#34;: \u0026#34;foo-sample\u0026#34;, \u0026#34;reconcileID\u0026#34;: \u0026#34;d747fc5a-3465-4c56-843f-0f4d87d2e3f0\u0026#34;} 2023-11-03T09:16:40+01:00 INFO foo\u0026#39;s happy status updated {\u0026#34;controller\u0026#34;: \u0026#34;foo\u0026#34;, \u0026#34;controllerGroup\u0026#34;: \u0026#34;tutorial.my.domain\u0026#34;, \u0026#34;controllerKind\u0026#34;: \u0026#34;Foo\u0026#34;, \u0026#34;Foo\u0026#34;: {\u0026#34;name\u0026#34;:\u0026#34;foo-sample\u0026#34;,\u0026#34;namespace\u0026#34;:\u0026#34;default\u0026#34;}, \u0026#34;namespace\u0026#34;: \u0026#34;default\u0026#34;, \u0026#34;name\u0026#34;: \u0026#34;foo-sample\u0026#34;, \u0026#34;reconcileID\u0026#34;: \u0026#34;d747fc5a-3465-4c56-843f-0f4d87d2e3f0\u0026#34;, \u0026#34;status\u0026#34;: false} 2023-11-03T09:16:40+01:00 INFO foo custom resource reconciled {\u0026#34;controller\u0026#34;: \u0026#34;foo\u0026#34;, \u0026#34;controllerGroup\u0026#34;: \u0026#34;tutorial.my.domain\u0026#34;, \u0026#34;controllerKind\u0026#34;: \u0026#34;Foo\u0026#34;, \u0026#34;Foo\u0026#34;: {\u0026#34;name\u0026#34;:\u0026#34;foo-sample\u0026#34;,\u0026#34;namespace\u0026#34;:\u0026#34;default\u0026#34;}, \u0026#34;namespace\u0026#34;: \u0026#34;default\u0026#34;, \u0026#34;name\u0026#34;: \u0026#34;foo-sample\u0026#34;, \u0026#34;reconcileID\u0026#34;: \u0026#34;d747fc5a-3465-4c56-843f-0f4d87d2e3f0\u0026#34;} If you check the status of the Foo custom resources, you can see that their status is empty. That\u0026rsquo;s exactly what we expect so everything\u0026rsquo;s good so far!\n$ kubectl describe foos Name: foo-01 Namespace: default Labels: app.kubernetes.io/created-by=operator-v1 app.kubernetes.io/instance=foo-sample app.kubernetes.io/managed-by=kustomize app.kubernetes.io/name=foo app.kubernetes.io/part-of=operator-v1 Annotations: \u0026lt;none\u0026gt; API Version: tutorial.my.domain/v1 Kind: Foo Metadata: Creation Timestamp: 2023-11-03T08:18:01Z Generation: 1 Resource Version: 808 UID: fbda3c29-83d4-4e03-92e0-78f33f900b25 Spec: Name: jack Status: Events: \u0026lt;none\u0026gt; Name: foo-02 Namespace: default Labels: app.kubernetes.io/created-by=operator-v1 app.kubernetes.io/instance=foo-sample app.kubernetes.io/managed-by=kustomize app.kubernetes.io/name=foo app.kubernetes.io/part-of=operator-v1 Annotations: \u0026lt;none\u0026gt; API Version: tutorial.my.domain/v1 Kind: Foo Metadata: Creation Timestamp: 2023-11-03T08:18:01Z Generation: 1 Resource Version: 810 UID: ca34e5dc-f2d9-4caf-9239-bbd190bc9c9d Spec: Name: joe Status: Events: \u0026lt;none\u0026gt; Now, let\u0026rsquo;s spice things up! We\u0026rsquo;ll deploy a pod named jack to see how the system reacts.\nOnce done, you should see that the controller reacts to the pod creation event. It then updates the status of the first Foo custom resource as expected. You can verify by yourself by describing the Foo custom resources.\n2023-11-03T09:26:25+01:00 INFO pod linked to a foo custom resource issued an event {\u0026#34;name\u0026#34;: \u0026#34;jack\u0026#34;} 2023-11-03T09:26:25+01:00 INFO pod linked to a foo custom resource issued an event {\u0026#34;name\u0026#34;: \u0026#34;jack\u0026#34;} 2023-11-03T09:26:25+01:00 INFO reconciling foo custom resource {\u0026#34;controller\u0026#34;: \u0026#34;foo\u0026#34;, \u0026#34;controllerGroup\u0026#34;: \u0026#34;tutorial.my.domain\u0026#34;, \u0026#34;controllerKind\u0026#34;: \u0026#34;Foo\u0026#34;, \u0026#34;Foo\u0026#34;: {\u0026#34;name\u0026#34;:\u0026#34;foo-01\u0026#34;,\u0026#34;namespace\u0026#34;:\u0026#34;default\u0026#34;}, \u0026#34;namespace\u0026#34;: \u0026#34;default\u0026#34;, \u0026#34;name\u0026#34;: \u0026#34;foo-01\u0026#34;, \u0026#34;reconcileID\u0026#34;: \u0026#34;ec16b29b-4bae-4909-8392-8966c3e10ad4\u0026#34;} 2023-11-03T09:26:25+01:00 INFO pod linked to a foo custom resource found {\u0026#34;controller\u0026#34;: \u0026#34;foo\u0026#34;, \u0026#34;controllerGroup\u0026#34;: \u0026#34;tutorial.my.domain\u0026#34;, \u0026#34;controllerKind\u0026#34;: \u0026#34;Foo\u0026#34;, \u0026#34;Foo\u0026#34;: {\u0026#34;name\u0026#34;:\u0026#34;foo-01\u0026#34;,\u0026#34;namespace\u0026#34;:\u0026#34;default\u0026#34;}, \u0026#34;namespace\u0026#34;: \u0026#34;default\u0026#34;, \u0026#34;name\u0026#34;: \u0026#34;foo-01\u0026#34;, \u0026#34;reconcileID\u0026#34;: \u0026#34;ec16b29b-4bae-4909-8392-8966c3e10ad4\u0026#34;, \u0026#34;name\u0026#34;: \u0026#34;jack\u0026#34;} 2023-11-03T09:26:25+01:00 INFO foo\u0026#39;s happy status updated {\u0026#34;controller\u0026#34;: \u0026#34;foo\u0026#34;, \u0026#34;controllerGroup\u0026#34;: \u0026#34;tutorial.my.domain\u0026#34;, \u0026#34;controllerKind\u0026#34;: \u0026#34;Foo\u0026#34;, \u0026#34;Foo\u0026#34;: {\u0026#34;name\u0026#34;:\u0026#34;foo-01\u0026#34;,\u0026#34;namespace\u0026#34;:\u0026#34;default\u0026#34;}, \u0026#34;namespace\u0026#34;: \u0026#34;default\u0026#34;, \u0026#34;name\u0026#34;: \u0026#34;foo-01\u0026#34;, \u0026#34;reconcileID\u0026#34;: \u0026#34;ec16b29b-4bae-4909-8392-8966c3e10ad4\u0026#34;, \u0026#34;status\u0026#34;: true} 2023-11-03T09:26:25+01:00 INFO foo custom resource reconciled {\u0026#34;controller\u0026#34;: \u0026#34;foo\u0026#34;, \u0026#34;controllerGroup\u0026#34;: \u0026#34;tutorial.my.domain\u0026#34;, \u0026#34;controllerKind\u0026#34;: \u0026#34;Foo\u0026#34;, \u0026#34;Foo\u0026#34;: {\u0026#34;name\u0026#34;:\u0026#34;foo-01\u0026#34;,\u0026#34;namespace\u0026#34;:\u0026#34;default\u0026#34;}, \u0026#34;namespace\u0026#34;: \u0026#34;default\u0026#34;, \u0026#34;name\u0026#34;: \u0026#34;foo-01\u0026#34;, \u0026#34;reconcileID\u0026#34;: \u0026#34;ec16b29b-4bae-4909-8392-8966c3e10ad4\u0026#34;} Let\u0026rsquo;s update the specification of the second Foo custom resource and change the value of its name field from joe to jack. The controller should catch the update event and trigger a reconciliation loop.\n2023-11-03T09:27:18+01:00 INFO pod linked to a foo custom resource issued an event {\u0026#34;name\u0026#34;: \u0026#34;joe\u0026#34;} 2023-11-03T09:27:18+01:00 INFO pod linked to a foo custom resource issued an event {\u0026#34;name\u0026#34;: \u0026#34;joe\u0026#34;} 2023-11-03T09:27:18+01:00 INFO reconciling foo custom resource {\u0026#34;controller\u0026#34;: \u0026#34;foo\u0026#34;, \u0026#34;controllerGroup\u0026#34;: \u0026#34;tutorial.my.domain\u0026#34;, \u0026#34;controllerKind\u0026#34;: \u0026#34;Foo\u0026#34;, \u0026#34;Foo\u0026#34;: {\u0026#34;name\u0026#34;:\u0026#34;foo-02\u0026#34;,\u0026#34;namespace\u0026#34;:\u0026#34;default\u0026#34;}, \u0026#34;namespace\u0026#34;: \u0026#34;default\u0026#34;, \u0026#34;name\u0026#34;: \u0026#34;foo-02\u0026#34;, \u0026#34;reconcileID\u0026#34;: \u0026#34;d7e94353-94ae-4883-b5de-1d50c966448f\u0026#34;} 2023-11-03T09:27:18+01:00 INFO pod linked to a foo custom resource found {\u0026#34;controller\u0026#34;: \u0026#34;foo\u0026#34;, \u0026#34;controllerGroup\u0026#34;: \u0026#34;tutorial.my.domain\u0026#34;, \u0026#34;controllerKind\u0026#34;: \u0026#34;Foo\u0026#34;, \u0026#34;Foo\u0026#34;: {\u0026#34;name\u0026#34;:\u0026#34;foo-02\u0026#34;,\u0026#34;namespace\u0026#34;:\u0026#34;default\u0026#34;}, \u0026#34;namespace\u0026#34;: \u0026#34;default\u0026#34;, \u0026#34;name\u0026#34;: \u0026#34;foo-02\u0026#34;, \u0026#34;reconcileID\u0026#34;: \u0026#34;d7e94353-94ae-4883-b5de-1d50c966448f\u0026#34;, \u0026#34;name\u0026#34;: \u0026#34;joe\u0026#34;} 2023-11-03T09:27:18+01:00 INFO foo\u0026#39;s happy status updated {\u0026#34;controller\u0026#34;: \u0026#34;foo\u0026#34;, \u0026#34;controllerGroup\u0026#34;: \u0026#34;tutorial.my.domain\u0026#34;, \u0026#34;controllerKind\u0026#34;: \u0026#34;Foo\u0026#34;, \u0026#34;Foo\u0026#34;: {\u0026#34;name\u0026#34;:\u0026#34;foo-02\u0026#34;,\u0026#34;namespace\u0026#34;:\u0026#34;default\u0026#34;}, \u0026#34;namespace\u0026#34;: \u0026#34;default\u0026#34;, \u0026#34;name\u0026#34;: \u0026#34;foo-02\u0026#34;, \u0026#34;reconcileID\u0026#34;: \u0026#34;d7e94353-94ae-4883-b5de-1d50c966448f\u0026#34;, \u0026#34;status\u0026#34;: true} 2023-11-03T09:27:18+01:00 INFO foo custom resource reconciled {\u0026#34;controller\u0026#34;: \u0026#34;foo\u0026#34;, \u0026#34;controllerGroup\u0026#34;: \u0026#34;tutorial.my.domain\u0026#34;, \u0026#34;controllerKind\u0026#34;: \u0026#34;Foo\u0026#34;, \u0026#34;Foo\u0026#34;: {\u0026#34;name\u0026#34;:\u0026#34;foo-02\u0026#34;,\u0026#34;namespace\u0026#34;:\u0026#34;default\u0026#34;}, \u0026#34;namespace\u0026#34;: \u0026#34;default\u0026#34;, \u0026#34;name\u0026#34;: \u0026#34;foo-02\u0026#34;, \u0026#34;reconcileID\u0026#34;: \u0026#34;d7e94353-94ae-4883-b5de-1d50c966448f\u0026#34;} Yeah, it worked! Enough tests for today; I think you got it! If you delete the pod named jack, the custom resources\u0026rsquo; happy status will be set back to false.\nWe can confirm that the operator works as expected! It would be better to write unit and e2e tests but that\u0026rsquo;s not the purpose of this article—I\u0026rsquo;ll cover this topic in another article.\nYou can be proud of yourself. You have designed, deployed, and tested your very first operator. Congratulations!\nHere\u0026rsquo;s the link to the GitHub repository if you need to browse the code.\nTo go further We\u0026rsquo;ve seen how to create a very basic Kubernetes operator and it\u0026rsquo;s far from being perfect. There are many ways to improve it. Here\u0026rsquo;s a list of topics you can explore if you want.\nOptimise event filtering (sometimes, events are submitted twice…). Refine RBAC permissions. Improve the logging system. Issue Kubernetes events when resources are updated by the operator. Add custom fields when getting Foo custom resources (display the happy status maybe?). Write unit tests and e2e tests. Here\u0026rsquo;s a list of resources you can use to dig deeper into the subject.\nhttps://youtu.be/i9V4oCa5f9I https://book.kubebuilder.io/ https://cloudark.medium.com/kubernetes-custom-controllers-b6c7d0668fdf https://medium.com/swlh/kubernetes-operator-for-beginners-what-why-how-21b23f0cb9b1 https://medium.com/swlh/advanced-kubernetes-operators-development-988edad5f58a https://operatorhub.io/ Have fun! :)\nNB: A first version of this article was published on Medium in Better Programming, you can find it here. This article is an improved version with a little more content, including the diagram of the Foo operator\u0026rsquo;s functioning.\n","permalink":"https://leovct.github.io/posts/kubernetes/build-an-operator/","summary":"\u003cp\u003eYou\u0026rsquo;re probably familiar with Kubernetes, but do you know what operators are, how they work, and how to build one?\nIf you want to know more, you\u0026rsquo;ve come to the right place!\u003c/p\u003e\n\u003cp\u003eKubernetes operators is a complicated subject but fortunately, since \u003ca href=\"https://web.archive.org/web/20170129131616/https://coreos.com/blog/introducing-operators.html\"\u003etheir creation\u003c/a\u003e in 2016, many tools have been developed to simplify the life of engineers.\u003c/p\u003e\n\u003cp\u003eWithout further ado, let\u0026rsquo;s dive in and learn more about operators!\u003c/p\u003e\n\u003cblockquote\u003e\n\u003cp\u003e\u003cstrong\u003e✨ The article was updated to use the latest version of kubebuilder (\u003ca href=\"https://github.com/kubernetes-sigs/kubebuilder/releases/tag/v4.2.0\"\u003ev4.2.0\u003c/a\u003e), released in August 2024!\u003c/strong\u003e\u003c/p\u003e","title":"🛠️ Build a Kubernetes operator in 10 minutes"},{"content":"Some of the projects I have been working on lately, ranging from blockchain infrastructure to developer tooling and Kubernetes platforms.\n2023-2026 Deep dive into blockchain infrastructure and developer tooling at Polygon Labs\n✅ Reliability Defining user journeys and establishing SLIs and SLOs to measure and improve service reliability Implementing synthetic monitoring for Polygon\u0026rsquo;s critical infrastructure (blockchain networks, AggLayer, OMS/BPN, web apps, etc.) 📦 Kurtosis Packages Packages for reproducible blockchain devnets, enabling faster local development and integration testing across multiple stacks.\n0xPolygon/kurtosis-cdk: Polygon CDK stack - maintainer + core contributor 0xPolygon/kurtosis-pos: Polygon PoS stack - maintainer + core contributor agglayer/optimism-package: Optimism stack - maintainer 🔧 Blockchain Tools 0xPolygon/polygon-cli: CLI toolkit for blockchain development and operations - occasional contributor 💖 Open-source Contributions kurtosis-tech/kurtosis: Platform for packaging and launching blockchain infrastructure foundry-rs/foundry: Blockchain toolkit for Ethereum paradigmxyz/reth: Rust execution layer implementation of the Ethereum protocol mds1/evm-diff: Web UI to differentiate EVM-compatible chains 2021-2022 A fresh start in the world of platform engineering at Renault Software Factory\n☸️ Kubernetes Operators Designed and implemented a Kubernetes operator to standardize configurations and automate deployments and updates of GitLab runners Production deployment managing 50+ runners, operating 24/7, and executing thousands of jobs per week Wrote technical articles on building and testing Kubernetes operators 👀 Monitoring Managed Kubernetes monitoring stack using Prometheus, Grafana, Karma, and Thanos Infrastructure and application metrics monitoring 🌸 Chainlink Spring Hackathon Everest Protocol: Attempted to build a DeFi protocol for trading synthetic assets (crypto following traditional asset prices like $TSLA) with two other crypto nerds (cc @yvesbou) Top 40 out of 400+ projects and 1,500+ participants Based on Chainlink oracles and Uniswap Deployed on Avalanche Fuji testnet 6-week hackathon ","permalink":"https://leovct.github.io/projects/","summary":"projects","title":"Projects"}]