I spent a couple of months interviewing, and eventually landed up at SUSE working on Rancher.

In some ways, Rancher is similar to Weave GitOps, but without almost all of the GitOps, and a lot less declarative, most things are done as “ClickOps”.

It’s been tough finding a place at SUSE, the Rancher codebase is not what I’d write, and there are bits that are painfully bad, but there are a lot of smart folks working on it, hammering it into shape.

Hackweek is an annual thing at SUSE (we had a very similar week at Heroku) in which we get to hack on whatever we want.

I opted to work on a number of different things this year:

• Getting gitopssets going again, it’s been in maintenance mode since last December.
• Replacing the old Tekton Poller which I wrote about in July 2020.
• Writing TinyGo for my Keyboard Featherwing I write a lot of Go, but I’ve never written any “Tiny” Go for embedded devices, despite having written a fair bit of Arduino C++ and some CircuitPython.
• Implementing a small Forth interpreter.

That’s a lot of things to get on with…how’d it go?

GitOpsSets

GitOpsSets had been in maintenance-only for almost a year, only dependencies being updated.

I had forked it when WeaveWorks folded, and rearranged the packages, and some minor tweaks, but I hadn’t actually run it, so…getting it going took a little bit of effort, but it’s working now.

I fixed the GitRepository generator to work with post 2.2 Flux.

The CEL integration branch which would replace JSON-path in places where it makes sense is still active, I will do the work to get this to land, it will benefit from recent work I’ve done elsewhere in the CEL ecosystem.

I started a few separate explorations in GitOpsSets to see what is possible.

Configurable Cluster generation

Is it possible to generate from either GitOps Clusters (WeaveWorks) or Rancher Clusters?

I got quite far down the route, replacing the mechanism for querying clusters, and allowing a GVK to be provided. I stopped at the point where I realised that this could go in a few different directions, and one of them was a generic resource querying mechanism.

The caveat is that it becomes harder to select “ready” clusters, I will come back to this, and maybe it’s time to give up on WeaveWorks GitopsClusters (but there is now movement in the weave-gitops repo!), I’ll figure out the right way forward for this, as it’s a core element of the cluster-reflector, cluster-bootstrapper and cluster-generator workflow that I think is crucial for managing clusters at scale.

Generation from Keycloak

Over the past few years, I’ve seen a large number of enterprise users running Keycloak, while I’m not its biggest fan, it definitely does the job for folks, and so, can we generate from Keycloak users and groups within a realm?

This would allow generation of users and groups by querying they Keycloak API.

Ideally this would allow declarative generation of RBAC-type resources, e.g. RoleBindings by querying Keycloak as the source of truth.

I need to figure out what a generic version of this looks like, for example, is it valuable to expose a UsersGenerator which can query Keycloak and other things like OpenLDAP?

External generators

I made a start on external-generators, i.e. support for referencing generators which are Services that accept a specific format of request, and return a specific response format.

I’ve got the API defined, and implemented an example, I just need to hook this up to the generator resolution mechanism, effectively you should be able to reference an external generator by name, and the code will find a Generator resource that is an API endpoint that we can submit the generator parameters to, and receive a response from.

This is very, very similar to the approach in Tekton.

Most of this is already available via the APIClient generator in GitOpsSets, but external generators could integrate paginated responses in a standardised way which would be much more complicated to add to the APIClient generator.

I’m not even particularly wedded to the idea of an external Generator resource type, this might just be possible via an ExternalGenerator generator, which takes a URL and queries it rather than going via a Generator resource.

If done well, this might negate the need for a specific Keycloak generator, as it could just be an external generator.

GitPoller

Four and a half years ago I wrote a tool to allow you to trigger Tekton pipelines by polling a Git API for changes.

This was written because I saw a lot of requests by folks who wanted to use Tekton Triggers, but couldn’t easily expose an EventListener to the internet.

The design for the tool was not great, it was too tightly coupled to Tekton Pipelines, which added maintenance cost, at the time I thought this would reduce the cost of setting it up, but…the tradeoff wasn’t worth it.

I had started working on a replacement fairly early on, and after some experimentation I had decided on the shape of it with a reduction in the coupling the goal.

GitPoller had sat unreleased for the best part of three years, it needed a bit of modernisation, and translation of the original experimental code into more modern thinking.

It polls an upstream GitHub or GitLab repository, and when the commit changes it sends a CloudEvent with the response body from the detected change, this required a change to Tekton Triggers to respond appropriately to CloudEvents.

apiVersion: polling.gitops.tools/v1alpha1
kind: PolledRepository
metadata:
  name: polledrepository-sample
  namespace: poller-demo
spec:
  url: https://github.com/bigkevmcd/go-demo.git
  ref: main
  type: github
  frequency: 5m
  endpoint: https://el-polling-listener.poller-demo.svc.cluster.local:8080

This example will poll the main branch in that repo every 5 minutes, and send notifications to the tekton event listener in the endpoint.

It isn’t complete yet, there are a few things I need to do including providing support for custom CAs so that you can talk to a private GitHub/GitLab installation, but instead of talking to Tekton Pipelines, it now talks to Tekton Triggers, so you can use the full power of Triggers to drive your pipelines, including parsing the body using CEL.

TinyGo

I’ve written a fair bit of Arduino code, and some CircuitPython too.

I have a lot of micro-controllers, they’re really useful for teaching kids how computers work, and building small projects.

Despite this, I have never written any Go for a micro-controller, despite having looked at TinyGo several times.

I decided to change that, and I set a goal of getting some code to drive my Keyboard Featherwing which is a small keyboard/screen based board that accepts an Adafruit Feather micro-controller.

The device has an SD Card, screen with touch-capability, a 35 button keyboard, and 5 buttons including a joystick.

Debugging embedded code is hard, I can’t just write a test and stick strategic log.Printf calls in to see what’s failing, it needs to be compiled and flashed to the device and then the serial device monitored for those same logging messages.

I did manage to get the touch screen working, it’s a TI 2004 and there’s Arduino code to work from.

The conversion from Arduino C++ to Go wasn’t very hard, it took a few hours.

#include "TSC2004.h"

TSC2004 ts;

void setup() {
  Serial.begin(9600);

  ts.begin();
}

void loop() {
  if (!ts.touched()) {
    return;
  }

  const TS_Point p = ts.getPoint();

  Serial.print("(");
  Serial.print(p.x);
  Serial.print(", ");
  Serial.print(p.y);
  Serial.print(", ");
  Serial.print(p.z);
  Serial.println(")");
}

Gets converted to this Go:

package main

import (
	"fmt"
	"machine"
	"time"

	"github.com/bigkevmcd/tiny-go/tsc2004"
)

func main() {
	machine.I2C0.Configure(machine.I2CConfig{
		SDA: machine.SDA_PIN,
		SCL: machine.SCL_PIN,
	})

	tsc := tsc2004.NewI2C(machine.I2C0)
	if err := tsc.Begin(); err != nil {
		fmt.Println("failed to initialise", err)
	}

	for {
		touched, err := tsc.Touched()
		if err != nil {
			fmt.Println("error reading touch state ", err)
			continue
		}

		if !touched {
			continue
		}

		point, err := tsc.Point()
		if err != nil {
			fmt.Println("error reading touch point ", err)
			continue
		}

		fmt.Println(point)
	}
}

The Go code is a little bit messier because of the error handling, but the C++ code isn’t doing any error handling, I know which I prefer.

This is basically talking I2C to the correct port and accessing the registers to send appropriate events.

I have started on code to read the keyboard, and hopefully I can get that working, which would let me connect it to my other project…

Implementing a small Forth interpreter

I like the simplicity and explicitness of Forth, it works nicely on very low power machines…thus making it ideal for running on micro-controllers.

It’s a rite of passage for Forth developers to write their own Forth, to understand the combination of parsing, dictionaries and stacks that combine to make an implementation.

The TSC2004 code I struggled to TDD, because I didn’t really know where I was going with the code, I did design it in a way that does make it easy to test (and I’ve written tests).

The tiny-forth code is TDDed, it’s nice to actually create some code from scratch, with no framework or anything, just straight up Go stdlib code.

While the dictionary and stack are fairly easy, I use stacks a fair bit in my Go code for things, and linked-lists are trivial, it turns out the parsing and tokenisation of stdin is the hardest part.

While it’s trivial to read a line and easy enough to tokenise it and my tests can use strings.NewReader() as a stand-in for the console, it turns out parsing Forth input isn’t quite that simple, especially once you get past the first steps of parsing tokens and figuring out if they’re known words or integers to be put on the stack.

Parsing strings, and floats and other things adds complexity, and I haven’t gotten as far as “defining words” i.e. CREATE...DOES>.

It’s still really interesting, and there’s not a massive amount of code (which is still littered with TODOs that mark out design decisions not yet solidified).

Summary

All in all it’s been a fun week, I enjoyed all the projects I worked on.

It’s easy to forget how much I prefer working on TDD written code, the designs are flexible, changing things is easier, and I find that I make more explicit design decisions.

For example, these are two of the outstanding TODOs.

func newInterpreter(output consoleOutput) *forthInterpreter {
	i := &forthInterpreter{stack: newStack[int](), output: output}
	i.addWord("+", i.word_add)
    // `add.addWord` adds more words which are methods on the `forthInterpreter`
    // struct. 
}

// TODO: Shift this to a separate dictionary struct?
func (i *forthInterpreter) find(s string) *executionToken {
    // Iterate through the linked-list of execution tokens.
}

// TODO: Should these be standalone functions that accept an interpreter?
func (i *forthInterpreter) word_add() error {

Both of them have design implications, not major ones, but I can refactor with the tests behind me to ensure I don’t break anything.

I have decided to take on maintenance of GitOpsSets again - I really, really want to land support for SSA and the external generation mechanism.

I’ve watched as folks fork the old Tekton Poller, and probably realise that it needs too much work to get going while I had a replacement in progress and it needed a push to get it over the line.

The work to harden the GitPoller shouldn’t be massive, and I can do that in a few evenings, it turns out to be pretty useful for testing things in Tekton, because you don’t need to change anything to trigger a push, just delete and recreate the GitPoller.

I enjoyed the TinyGo, it’s slightly different from the main Go compiler, but the same techniques for testing it work, so once I get the hang of it, I should be able to TDD it as everything else. Integration tests will be a bit trickier, but I think they’re doable.

I did write a little bit of Zig, reimplementing a simple CPU in Zig, it’s definitely different from most of my current toolkit of languages, but I didn’t write a lot of code - I can see me coming back to this.

test "SyntheticCpu step" {
    const multiply: []const u8 = &.{ 0x04, 0x00, 0xc9, 0xff, 0xff, 0xff, 0x12, 0x00, 0xff, 0xff, 0xff, 0xff, 0x08, 0x00, 0x00, 0x00 };
    var cpu = SyntheticCpu.init(multiply);
    try expectEqual(0, cpu.ip);

    cpu.step();

    try expectEqual(6, cpu.ip);
    try expectEqual(0xc9, cpu.regs[0]);

    cpu.step();

    try expectEqual(12, cpu.ip);
    try expect(cpu.flags[CPU_FLAG_OF] == false);
    try expectEqual(55, cpu.regs[0]);
}

Finally, the Forth interpreter might fall by the wayside, it was an interesting experiment, but there are already micro-controller Forths, if anything has to get dropped from this week, that’ll be the thing.

I really enjoyed Hackweek, and I’m looking forward to getting back to customer problems, hopefully finally get a few branches over the line.

It’s originally written for Scheme which is definitely a high-level language.

When I first started writing software for a living, I wrote a fair bit of code in C and 8086 Assembly, specifically I wrote code for interfacing with serial terminals, this is before the Internet, and before TCP/IP become ubiquitous and so “green screens” were common in settings, which were TTYs connected to Unix systems via terminal interfacing hardware.

At this time, C was the dominant language for this kind of work, with some bits dropping into assembly where the C wasn’t quite fast enough, which on base level 80286 and 802386 hardware of the era, wasn’t that uncommon.

Around this time, I learned of a language called Forth, and I remember experimenting with Pygmy for DOS mostly out of curiosity, it was easy enough to talk to the 8250/16550 serial devices that were prevalent pre-USB.

Forth is an interesting language, it’s designed to be very low-level, for interfacing with hardware, and so, like C it’s easy to drop into assembly.

The syntax is definitely unlike anything I’m familiar with, everything is done with RPN and there are multiple stacks in use to use for passing parameters around, this use of the stack leads to a lot of stack manipulation, and designing small functions involves designing the layout of the variables on the stack, and how they’re moved around in the code, this leads to small functions as it becomes increasingly hard to reason about the state of the stack in code as functions get larger.

I’m not going to go into Forth syntax too heavily here, the best books on the subject are freely available:

• Starting Forth by Leo Brodie https://www.forth.com/starting-forth/
• Thinking Forth by Leo Brodie https://thinking-forth.sourceforge.net/

Looking at the examples in the HtDP book…

(define (area-of-disk r) 
  (* 3.14 (* r r)))

This declares a function area-of-disk that takes a parameter r it then multiplies r by itself, and then by 3.14 which is an approximation of Pi.

The simple approach to this looks like this:

: area-of-disk ( n -- n )
  dup * 314 * 100 / ;

Trying this in Gforth

Gforth 0.7.3, Copyright (C) 1995-2008 Free Software Foundation, Inc.
Gforth comes with ABSOLUTELY NO WARRANTY; for details type `license'
Type `bye' to exit
: area-of-disk ( n1 -- n2 )  compiled
  dup * 314 * 100 / ;  ok
5 area-of-disk . 78  ok

In the example, I’m pushing 5 onto the stack, then calling area-of-disk which executes and pushes the result onto the stack, running . pops the top element off the stack, with the result 78.

Looking at each operation, and how it affects the stack:

In the book, the result should be 78.5, but this code uses purely integer arithmetic which isn’t ideal.

Reimplementing this in Forth to use the floating-point stack:

: farea-of-disk ( r1 -- r2 )
  fdup f* pi f* ;

Trying this in Gforth looks slightly different:

Gforth 0.7.3, Copyright (C) 1995-2008 Free Software Foundation, Inc.
Gforth comes with ABSOLUTELY NO WARRANTY; for details type `license'
Type `bye' to exit
: farea-of-disk ( r1 -- r2 )  compiled
  fdup f* pi f* ;  ok
  ok
5e farea-of-disk f. 78.5398163397448  ok

The stack operations look almost identical:

This definitely looks like a more acceptable result.

There are two user stacks in Gforth, the stack and the floating-point stack, these are maintained separately, and the e notation pushes the number 5.0 onto the floating-point stack.

There is a mirrored set of functionality for operating on the floating-point stack.

pi is defined as a floating-point constant with 3.14159265358979.

The next part involves calculating the area of a ring, which is one circle inside of another:

(define (area-of-ring outer inner) 
  (- (area-of-disk outer)
     (area-of-disk inner)))

Reimplementing this in Forth is fairly simple and short:

: farea-of-ring ( r1 r2 -- r3 )
 farea-of-disk fswap farea-of-disk fswap f- ;

This time there’s a bit of stack rearrangement to get this to work:

Running the code:

Gforth 0.7.3, Copyright (C) 1995-2008 Free Software Foundation, Inc.
Gforth comes with ABSOLUTELY NO WARRANTY; for details type `license'
Type `bye' to exit
: farea-of-disk ( r1 -- r2 )  compiled
  fdup f* pi f* ;  ok
: farea-of-ring ( r1 r2 -- r3 )  compiled
 farea-of-disk fswap farea-of-disk fswap f- ;  ok
5e 3e farea-of-ring f. 50.2654824574367  ok

There are two fswaps in this code, fswap swaps the top two elements on the floating-point stack.`

Without this, the second area-of-disk would use the top element on the stack, which would be the result of area-of-disk on the first item.

The second fswap orders the areas of the two disks for the f- call.

This could be avoided by changing the ordering of the parameters on the stack, but this would put the onus on the caller to remember that the inner circle radius comes first which is not ideal.

Currently, I’m exploring git repository directory structures, trying to detect structures in the files and directories, in the past I’ve looked at ways of describing this in a “manifest”, and while I think this is useful, it leads to duplication of the data.

So, what can we figure out from a git repository?

If a git repository contains this simplified directory structure, I want to detect the single app demo-app.

apps
└── demo-app

To do this, I created a tarball of this, because I’m working with Flux GitRepository objects, and write this test…

func TestFindApplications(t *testing.T) {
	ts := httptest.NewServer(http.FileServer(http.Dir("testdata/archives")))
	t.Cleanup(func() {
		ts.Close()
	})

	apps, err := FindApplications(ts.URL+"/first.tar.gz")
	if err != nil {
		t.Fatal(err)
	}
	want := sets.NewString("demo-app")
	if diff := cmp.Diff(want, apps); diff != "" {
		t.Fatalf("failed to parse apps:\n%s", diff)
	}
}

This basically sets up a Server to serve the generated tarball from a testdata directory.

The simplest thing I could write to implement this looks like this:

func FindApplications(archiveURL string) (sets.String, error) {
	resp, err := http.Get(archiveURL)
	// error handling trimmed for brevity
	defer resp.Body.Close()

	gzipReader, err := gzip.NewReader(resp.Body)
	// error handling trimmed for brevity

	tarReader := tar.NewReader(gzipReader)
	apps := sets.NewString()
	for true {
		header, err := tarReader.Next()
		if err == io.EOF {
			break
		}
		// error handling trimmed for brevity
		switch header.Typeflag {
		case tar.TypeDir:
			if strings.HasPrefix(header.Name, "apps") {
				appPath := strings.TrimPrefix(header.Name, "apps")
				pathElements := strings.Split(appPath, "/")
				if len(pathElements) == 2 {
					apps.Insert(pathElements[1])
				}
			}
		}
	}
	return apps, nil
}

I am removing most error handling and logging from this code to clarify it

I could download the tarball and unarchive it to a temporary directory, but I’ll parse the streamed tarball in this case.

This gets the test passing, great!

From here, we need tests for error handling, what happens if the file is a 404, or isn’t a valid GZIP stream?

But, the main goal of this code, is parsing directory trees, and I don’t want to create tarballs for every case I want to test…it’s too expensive.

I could write a tarball generation mechanism, and I might need to, but the first thing I really care about, is making sure that it parses the correct things from the names (the code doesn’t read the files that are in the archive yet).

So, I refactor the FindApplications function, to separate out the functionality, this gets me code that looks like this:

func FindApplications(archiveURL string) (sets.String, error) {
	// identical to the earlier code, opening the stream and ddecoding
	return parseTarball(tarReader, appsPrefix)
}

func parseTarball(tarReader *tar.Reader, base string) (sets.String, error) {
	apps := sets.NewString()
	for true {
		header, err := tarReader.Next()
		if err == io.EOF {
			break
		}
		// trim error handling
		switch header.Typeflag {
		// case tar.TypeReg:
		case tar.TypeDir:
			if strings.HasPrefix(header.Name, appsPrefix) {
				appPath := strings.TrimPrefix(header.Name, appsPrefix)
				pathElements := strings.Split(appPath, "/")
				if len(pathElements) == 2 {
					apps.Insert(pathElements[1])
				}
			}
		}
	}
	return apps, nil
}

My tests still pass, but now I can isolate the behaviour of the directory parsing.

All I need from the *tar.Reader is this simple interface.

type tarIterator interface {
	Next() (*tar.Header, error)
}

Because of this, I can build a fake tar reader…

type fakeTarReader struct {
	headers []*tar.Header
	sync.Mutex
}

// Next() implements the tarReader interface.
func (s *fakeTarReader) Next() (*tar.Header, error) {
	s.Lock()
	defer s.Unlock()
	if len(s.headers) == 0 {
		return nil, io.EOF
	}
	r := s.headers[len(s.headers)-1]
	s.headers = s.headers[0 : len(s.headers)-1]
	return r, nil
}

This is a simple stack implementation which is a technique I use a fair bit in test fakes.

I can then cheaply set up a set of table tests…

func Test_parseTarball(t *testing.T) {
	parseTests := []struct {
		headers []*tar.Header
		prefix  string
		want    sets.String
	}{
		{
			headers: []*tar.Header,
			want:    sets.NewString("demo-app"),
			prefix:  "apps",
		},
		{
			headers: []*tar.Header,
			want:    sets.NewString("test-app"),
			prefix:  "team/apps",
		},
	}

	for i, tt := range parseTests {
		t.Run(fmt.Sprintf("test-%d", i), func(t *testing.T) {
			reader := &fakeTarReader{headers: tt.headers}
			apps, err := parseTarball(reader, tt.prefix)
			if err != nil {
				t.Fatal(err)
			}
			if diff := cmp.Diff(tt.want, apps); diff != "" {
				t.Fatalf("failed to parse apps:\n%s", diff)
			}
		})
	}
}

The second test, with the team/apps prefix fails, and I’m into the classic TDD “write a failing test”, “make the tests pass”, “refactor” cycle.

There’s a lot of discussion about whether or not you should test “private methods”, but I’m pragmatic :-)

In this case, it’s not exported, and so, I don’t feel too bad, but this is a tricky line to tread in TDD, as this test is strongly tied to the underlying tar parsing.

Moving amountFor:

After the success of extracting amountFor: I opted to use the refactoring browser’s “Move to another class” functionality to move the method across to Rental.

After choosing the destination class…

I ended up with amountFor: on the Rental class.

Unfortunately, this broke my tests:

The Pharo traceback was fairly clear:

But, by clicking on the Create button I got to fix it fairly easily.

And adding an interim method to keep my tests passing:

On to the refactoring…

Extracting the amount calculation

The first step, is to extract the logic that calculates the price of a Rental.

Customer >> amountFor: aRental [
	"I calculate the cost to rent the movie for a period."

	| thisAmount |
	thisAmount := 0.
	aRental movie priceCode = #regular ifTrue: [ thisAmount := thisAmount + 2 ].
	(aRental movie priceCode = #regular and: [ aRental daysRented > 2 ]) 
		ifTrue: [ thisAmount := thisAmount + (aRental daysRented - 2 * 1.5) ].

	aRental movie priceCode = #new ifTrue: [ 
		thisAmount := thisAmount + aRental daysRented * 3 ].

	aRental movie priceCode = #childrens ifTrue: [ 
		thisAmount := thisAmount + 1.5 ].
	(aRental movie priceCode = #childrens and: [ aRental daysRented > 3 ]) 
		ifTrue: [ thisAmount := thisAmount + (aRental daysRented - 3) * 1.5 ].
	^ thisAmount
]

This leads to a small change in the statement method:

		  rentals do: [ :rental | 
			  thisAmount := self amountFor: rental.

All the tests pass at this point.

At this point in the book, Fowler talks about the Smalltalk refactoring browser, and how he was “looking forward to a Java version!”. I’ll be honest, the state of the art in refactoring tools is definitely now in the Java camp.

But Pharo does have a refactoring browser, so, I reverted the change above and used it:

This lead to the same result (minus the comment of course).

Renaming result

The next step is renaming thisAmount to result.

This is trivial with the refactoring browser (although, ‘Rename temp’ is not obvious).

This was a nice step forward.

I had a go at the VideoStore refactoring exercise from the original Refactoring book.

(It was revisited in 2018 in JavaScript for the 2nd Edition)

It involves refactoring a method for generating a statement for a customers video rentals (yeah, I guess they were still a thing in 1999).

I’ll be honest, it’s hard to replicate the code from the book, not because Smalltalk isn’t as expressive as the Java from the book, but because it’s hard to write the code that is a switch statement in the book in Smalltalk without the urge to replace it with something better…

The original Java…

class Customer {
    public String statement() {
        double totalAmount = 0;
        int frequentRenterPoints = 0;
        Enumeration rentals = _rentals.elements();
        String result = "Rental Record for " + getName() + "\n";

        while (rentals.hasMoreElements()) {
            double thisAmount = 0;
            Rental each = (Rental) rentals.nextElement();

            //determine amounts for each line
            switch (each.getMovie().getPriceCode()) {
                case Movie.REGULAR: thisAmount += 2;
                    if (each.getDaysRented() > 2)
                        thisAmount += (each.getDaysRented() - 2) * 1.5;
                    break;
                case Movie.NEW_RELEASE:
                    thisAmount += each.getDaysRented() * 3;
                    break;
                case Movie.CHILDRENS:
                    thisAmount += 1.5;
                    if (each.getDaysRented() > 3)
                        thisAmount += (each.getDaysRented() - 3) * 1.5;
                    break;
            }
            // add frequent renter points
            frequentRenterPoints ++;
            // add bonus for a two day new release rental
            if ((each.getMovie().getPriceCode() == Movie.NEW_RELEASE) && each.getDaysRented() > 1)
                frequentRenterPoints ++;
            //show figures for this rental
            result += "\t" + each.getMovie().getTitle()+ "\t" + String.valueOf(thisAmount) + "\n";
            totalAmount += thisAmount;
        }
        //add footer lines
        result += "Amount owed is " + String.valueOf(totalAmount) + \n";
        result += "You earned " + String.valueOf(frequentRenterPoints) + " frequent renter points";
        return result;
    }
}

Which is fairly simply translated into Smalltalk (I used symbols instead of Enums for the rental types).

Customer >> statement [
	"I calculate and return a statement showing the costs of renting films."
	| totalAmount points thisAmount |
	totalAmount := 0.
	points := 0.
	^ String streamContents: [ :aStream | 
		  aStream << ('Rental record for {1}' format: { name }); cr.

		  rentals do: [ :rental | 
			  thisAmount := 0.		  
			  rental movie priceCode = #regular ifTrue: [ 
				  thisAmount := thisAmount + 2 ].
			  (rental movie priceCode = #regular and: [ rental daysRented > 2 ]) 
				  ifTrue: [ 
				  thisAmount := thisAmount + (rental daysRented - 2 * 1.5) ].

			  rental movie priceCode = #new ifTrue: [ 
				  thisAmount := thisAmount + rental daysRented * 3 ].

			  rental movie priceCode = #childrens ifTrue: [ 
				  thisAmount := thisAmount + 1.5 ].
			  (rental movie priceCode = #childrens and: [ 
				   rental daysRented > 3 ]) ifTrue: [ 
				  thisAmount := thisAmount + (rental daysRented - 3) * 1.5 ].

			  aStream
				  tab;
				  nextPutAll: rental movie title;
				  tab;
				  tab;
				  nextPutAll: thisAmount printString;
				  cr.
			  totalAmount := totalAmount + thisAmount.

			  points := points + 1.

			  (rental movie priceCode = #new and: [ rental daysRented > 1 ]) 
				  ifTrue: [ points := points + 1 ]
			].

		  aStream << ('Amount owed is ${1}' format: { totalAmount }); cr.
		  aStream << ('You earned {1} frequent renter points' format: { points }); cr
	]
]

Of course, it starts with tests…

CustomerTest >> testWithoutAnyRentals [
	self
		assert: customer statement
		equals:
			'Rental record for Martin' , String cr , 'Amount owed is $0'
			, String cr , 'You earned 0 frequent renter points' , String cr
]

CustomerTest >> testWithOneRegularRentalForThreeDays [
	| statement |
	customer addRental: (Rental withMovie: regularMovie daysRented: 3).

	statement := customer statement.

	self assert: (statement includesSubstring: 'Amount owed is $3.5').
	self assert:
		(statement includesSubstring: 'You earned 1 frequent renter points')
]

CustomerTest >> testWithOneRegularRentalForTwoDays [
	| statement |
	customer addRental: (Rental withMovie: regularMovie daysRented: 2).

	statement := customer statement.

	self assert: (statement includesSubstring: 'Amount owed is $2').
	self assert:
		(statement includesSubstring: 'You earned 1 frequent renter points')
]

I’ll look at how to step through the same refactoring process as Martin Fowler covers in the book, using Pharo in a series of steps.

I like the approach they’re taking, each of the parts is combined with their flux tool to build GitOps workflows, but they’re useful standalone too if you want to build your own workflow.

Not unreasonably, it doesn’t come with any sort of “pipeline” mechanism for building CI or CD pipelines, but with the Notification Controller you can drive Tekton pipelines, for example, maybe you want to trigger some process when a new commit is available.

The following configuration assumes you’re reasonably familiar with Tekton.

Requirements

1. Install Flux from the Bootstrap instructions.

2. Install Tekton Pipeline using the “Installation One-liner”.

   $ kubectl apply -f https://storage.googleapis.com/tekton-releases/pipeline/previous/v0.18.1/release.yaml
   

3. Install Tekton Triggers.

   $ kubectl apply -f https://storage.googleapis.com/tekton-releases/triggers/previous/v0.10.1/release.yaml
   

$ flux bootstrap github --owner=$GITHUB_USER --repository=fleet-infra --branch=main --path=staging-cluster --personal
► connecting to github.com
✔ repository created
✔ repository cloned
✚ generating manifests
✔ components manifests pushed
► installing components in flux-system namespace
<snip>
✔ install completed
► configuring deploy key
✔ deploy key configured
► generating sync manifests
✔ sync manifests pushed
► applying sync manifests
◎ waiting for cluster sync
✔ bootstrap finished

$ kubectl apply -f https://storage.googleapis.com/tekton-releases/pipeline/previous/v0.18.1/release.yaml
namespace/tekton-pipelines created
<snip>
service/tekton-pipelines-webhook created

$ kubectl apply -f https://storage.googleapis.com/tekton-releases/triggers/previous/v0.10.1/release.yaml
podsecuritypolicy.policy/tekton-triggers created
<snip>
deployment.apps/tekton-triggers-webhook created

Configuring Tekton

For this example, I’m going to create a really simple Pipeline which is triggered by notifications from the Source Controller.

The event documentation could do with a JSON example for folks who don’t read Go natively, but the requests look like this:

POST / HTTP/1.1
Host: example.com
Accept-Encoding: gzip
Content-Length: 452
Content-Type: application/json
Gotk-Component: source-controller
User-Agent: Go-http-client/1.1

With a body that looks like this:

{
  "involvedObject": {
    "kind":"GitRepository",
    "namespace":"flux-system",
    "name":"flux-system",
    "uid":"cc4d0095-83f4-4f08-98f2-d2e9f3731fb9",
    "apiVersion":"source.toolkit.fluxcd.io/v1beta1",
    "resourceVersion":"56921"
  },
  "severity":"info",
  "timestamp":"2020-11-27T15:52:21Z",
  "message":"Fetched revision: main/731f7eaddfb6af01cb2173e18f0f75b0ba780ef1",
  "reason":"info",
  "reportingController":"source-controller",
  "reportingInstance":"source-controller-7c7b47f5f-8bhrp"
}

This event is emitted when the Source Controller detects a change to the watched GitRepository.

Tekton Pipeline

apiVersion: tekton.dev/v1beta1
kind: Pipeline
metadata:
  name: flux-demo-pipeline
spec:
  params:
    - name: message
      description: incoming notification message
      type: string
    - name: repository
      description: The name of the notification GitRepository
      type: string
    - name: namespace
      description: The namespace of the notification GitRepository
      type: string
  tasks:
    - name: task-echo-message
      taskSpec:
        params:
          - name: message
            type: string
          - name: repository
            type: string
          - name: namespace
            type: string
        steps:
          - name: echo-pipeline
            image: alpine
            command:
              - echo
            args:
              - "notification from $(params.namespace)/$(params.repository)"
          - name: echo-message
            image: alpine
            command:
              - echo
            args:
              - "$(params.message)"
      params:
        - name: message
          value: $(params.message)
        - name: repository
          value: $(params.repository)
        - name: namespace
          value: $(params.namespace)

This is a really simple Pipeline that accepts a param with a message to be echoed to the logs, and the name of the affected GitRepository CR, you could parse out the branch/revision and fetch the GitRepository to do more work, or perhaps these could be provided in the Metadata field of the event, which would definitely simplify the process.

Triggering the Pipeline from an EventListener

I created a simple EventListener to receive the notifications.

apiVersion: triggers.tekton.dev/v1alpha1
kind: EventListener
metadata:
  name: flux-event-listener
spec:
  serviceAccountName: flux-notifications-sa
  triggers:
    - name: flux-notification-trigger
      interceptors:
        - cel:
            filter: header.canonical('Gotk-Component') == 'source-controller' &&
              body.involvedObject.kind == 'GitRepository'
      template:
        name: flux-notification-template
      bindings:
        - ref: flux-notification-binding

This is matching on notifications from the Source Controller.

apiVersion: triggers.tekton.dev/v1alpha1
kind: TriggerTemplate
metadata:
  name: flux-notification-template
spec:
  params:
    - name: message
      description: message
    - name: repository
      description: the notifying repository
    - name: namespace
      description: the namespace for the notifying repository
  resourcetemplates:
    - apiVersion: tekton.dev/v1beta1
      kind: PipelineRun
      metadata:
        annotations:
        name: flux-notification-run-$(uid)
      spec:
        params:
          - name: message
            value: $(tt.params.message)
          - name: repository
            value: $(tt.params.repository)
          - name: namespace
            value: $(tt.params.namespace)
        pipelineRef:
          name: flux-demo-pipeline

This template is creating a PipelineRun with a generateName with the three parameters that the Pipeline requires from the TriggerBinding below:.

apiVersion: triggers.tekton.dev/v1alpha1
kind: TriggerBinding
metadata:
  name: flux-notification-binding
spec:
  params:
    - name: message
      value: $(body.message)
    - name: repository
      value: $(body.involvedObject.name)
    - name: namespace
      value: $(body.involvedObject.namespace)

There is some permission configuration to allow an EventListener to run in a namespace, this is a long set of configuration items, which while important for getting the EventListener to work isn’t really connected to setting up the notifications, I’ve provided the yaml here.

Configuring Flux

First off, this needs a random token secret for the receiver, I use this script to generate a secret with a token:

#!/bin/sh
# Put this in generate_secret.sh and chmod +x
TOKEN=$(head -c 12 /dev/urandom | shasum | cut -d ' ' -f1)
kubectl create secret generic $1 -n flux-system \
  --from-literal=token=$TOKEN

Creating a secret is as simple as:

$ ./generate_secret.sh generic-receiver-token
secret/generic-receiver-token created

Creating a Receiver

Toolkit components send HTTP requests to an HTTP endpoint that is part of the notification-controller, so I have to configure a receiver for this:

apiVersion: notification.toolkit.fluxcd.io/v1beta1
kind: Receiver
metadata:
  name: generic-receiver
  namespace: flux-system
spec:
  type: generic
  resources:
    - kind: GitRepository
      name: flux-system
      namespace: flux-system
  secretRef:
    name: generic-receiver-token

I also need a Provider, this is the element that defines how messages transmitted, there are providers for Slack and GitHub, but for this case, I’ll use a generic webhook provider which just sends an HTTP body.

apiVersion: notification.toolkit.fluxcd.io/v1beta1
kind: Provider
metadata:
  name: tekton-el-provider
  namespace: flux-system  
spec:
  type: generic
  address: http://el-flux-event-listener.default.svc.cluster.local:8080/

This is using standard Kubernetes DNS resolution to direct events to the Tekton EventListener defined above, when Tekton Triggers creates an EventListener process to receive HTTP requests, it creates a Service and Deployment with the prefix el- and the name of the EventListener object, and I placed it in the default namespace for this demo.

Next, there’s an Alert which connects the notification to the Provider.

apiVersion: notification.toolkit.fluxcd.io/v1beta1
kind: Alert
metadata:
  name: tekton-alert
  namespace: flux-system
spec:
  providerRef:
    name: tekton-el-provider
  eventSeverity: info
  eventSources:
    - kind: GitRepository
      name: flux-system
      namespace: flux-system

This configures an alert for info-level notifications for the GitRepository object that’s created during the installation of flux, to be sent via the Provider defined above.

Modifying a file in the repository pointed to by the flux-system GitRepository triggers a notification.

I can see the logs of the notification being triggered:

$ kubectl logs deploy/notification-controller -n flux-system
{"level":"info","ts":"2020-12-05T13:15:53.235Z","logger":"event-server","msg":"Dispatching event","object":"flux-system/flux-system","kind":"GitRepository","message":"Fetched revision: main/6fca464c6c124156b9cb3913c229b59148323703"}
{"level":"info","ts":"2020-12-05T13:15:56.408Z","logger":"event-server","msg":"Discarding event, no alerts found for the involved object","object":"flux-system/flux-system","kind":"Kustomization"}

And in the logs of the EventListener, the EventListener is logging out that it’s creating a new resource called flux-notification-run-rqswx.

$ kubectl logs deploy/el-flux-event-listener
{"level":"info","ts":"2020-12-05T13:15:53.265Z","logger":"eventlistener","caller":"sink/sink.go:237","msg":"ResolvedParams : [{Name:message Value:Fetched revision: main/6fca464c6c124156b9cb3913c229b59148323703} {Name:repository Value:flux-system} {Name:namespace Value:flux-system}]","knative.dev/controller":"eventlistener","/triggers-eventid":"wvd9c","/trigger":"flux-notification-trigger"}
{"level":"info","ts":"2020-12-05T13:15:53.268Z","logger":"eventlistener","caller":"resources/create.go:95","msg":"Generating resource: kind: &APIResource{Name:pipelineruns,Namespaced:true,Kind:PipelineRun,Verbs:[delete deletecollection get list patch create update watch],ShortNames:[pr prs],SingularName:pipelinerun,Categories:[tekton tekton-pipelines],Group:tekton.dev,Version:v1beta1,StorageVersionHash:RcAKAgPYYoo=,}, name: flux-notification-run-rqswx","knative.dev/controller":"eventlistener"}
{"level":"info","ts":"2020-12-05T13:15:53.268Z","logger":"eventlistener","caller":"resources/create.go:103","msg":"For event ID \"wvd9c\" creating resource tekton.dev/v1beta1, Resource=pipelineruns","knative.dev/controller":"eventlistener"}

And finally, I get to see the message from the event:

$ tkn pipelinerun logs --last
[task-echo-message : echo-pipeline] notification from flux-system/flux-system

[task-echo-message : echo-message] Fetched revision: main/6fca464c6c124156b9cb3913c229b59148323703

Right now, the Notifications Controller is functional, and hopefully it will become easier to integrate components.

Yes, opening firewall ports in the modern enterprise is often a trial of patience, so exposing your Tekton EventListener to GitHub is often not possible.

For this reason, I created tekton-poller-operator.

It’s not a perfect replacement for an EventListener, but if you want to drive Pipelines then this might help you!.

This is still in early stages, so I’m looking for feedback.

The obligatory YAML

apiVersion: polling.tekton.dev/v1alpha1
kind: Repository
metadata:
  name: example-repository
spec:
  url: https://github.com/my-org/my-repo.git
  ref: main
  frequency: 5m
  type: github # can also be gitlab
  pipelineRef:
    name: github-poll-pipeline
    namespace: test-ns # optional: if provided, the pipelinerun will be created in this namespace to reference the pipeline.
    params:
    - name: sha
      expression: commit.sha
    - name: repoURL
      expression: repoURL

This defines a Repository, every 5m, the GitHub repo https://github.com/myorg/my-repo.git is polled, if the SHA changes, then the pipeline github-poll-pipeline is executed with the new SHA, and the repoURL.

Params

params:
- name: sha
  expression: commit.sha
- name: repoURL
  expression: repoURL

The expressions here are CEL expressions, these are parsed from the commit that’s fetched by the Poller.

The repoURL is the URL from the spec in the repository, and the commit is the JSON body of the commit that was fetched.

TriggerBindings and TriggerTemplates

Currently, these are not supported, it will take some refactoring of Tekton triggers to make this happen, but this is the goal.

For example, kubectl provides really good information in its output:

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.4", GitCommit:"c96aede7b5205121079932896c4ad89bb93260af", GitTreeState:"clean", BuildDate:"2020-06-18T07:39:54Z", GoVersion:"go1.14.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.3", GitCommit:"2e7996e3e2712684bc73f0dec0200d64eec7fe40", GitTreeState:"clean", BuildDate:"2020-05-20T12:43:34Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

So, how do you do this in your compilations?

Let’s start with the simplest possible Go file.

package main

import "fmt"

var gitRevision = "unknown"

func main() {
    fmt.Printf("version %s\n", gitRevision)
}

Quickly put this into a git repo so that I can actually get the version

$ git init . && git add main.go && git commit -m "Initial commit."

Building at the command-line

$ go build main.go
$ ./main
version unknown

Ok, nothing totally unexpected there…but…

$ GIT_COMMIT=$(git rev-parse HEAD) && go build -ldflags "-X main.gitRevision=${GIT_COMMIT}" ./main.go
$ ./main
version 49e5416e820fb376fdb277243dbec8e2e81fbc29

The "-X main.gitRevision=${GIT_COMMIT}" embeds the environment variable GIT_COMMIT into the binary, by replacing the path main.gitRevision.

The -X parameter is passed to the Go link tool and from that documentation:

-X importpath.name=value Set the value of the string variable in importpath named name to value. This is only effective if the variable is declared in the source code either uninitialized or initialized to a constant string expression. -X will not work if the initializer makes a function call or refers to other variables. Note that before Go 1.5 this option took two separate arguments.

That importpath is fairly crucial to understanding the next part, for the simple case, the Go package is main, but we can also write into any package in a Go source tree.

Adding a go.mod path to the path.

That was fairly easy, but what if our package is in a go.mod package?

$ go mod init github.com/bigkevmcd/demo
go: creating new go.mod: module github.com/bigkevmcd/demo

And I’ll move the version into a specific package.

Editing pkg/git/git.go

package git

var Revision = "unknown"

And changing the original main.go to

package main

import (
	"fmt"

	"github.com/bigkevmcd/demo/pkg/git"
)

func main() {
	fmt.Printf("version %s\n", git.Revision)
}

So, I’m importing the version from a package.

I need to slightly adjust the command-line for building this:

$ GIT_COMMIT=$(git rev-parse HEAD) && go build -ldflags "-X github.com/bigkevmcd/demo/pkg/git.Revision=${GIT_COMMIT}" ./main.go
$ ./main
version abd2bcd0497f7a0f4e04aa157e879b478f1b0f95

NOTE that the full Go “package path” to the value I want to update is provided.

Building in a Dockerfile

So, now that I’ve got the basics right, building it in a Dockerfile is fairly simple:

FROM golang:latest AS build
WORKDIR /go/src
COPY . /go/src
RUN ls -la
RUN GIT_COMMIT=$(git rev-parse HEAD) && \
    CGO_ENABLED=0 GOOS=linux go build \
    -ldflags "-X github.com/bigkevmcd/demo/pkg/git.Revision=${GIT_COMMIT}" ./main.go

FROM alpine
WORKDIR /root/
COPY --from=build /go/src/main .
ENTRYPOINT ["./main"]

I can easily build and push this to an image repository.

$ docker build -t bigkevmcd/demo .
$ docker push bigkevmcd/demo

And I can run this image in Kubernetes with a one-liner:

$ kubectl run --image=bigkevmcd/demo demo-pod --restart=Never
pod/demo-pod created
$ kubectl logs pod/demo-pod
version abd2bcd0497f7a0f4e04aa157e879b478f1b0f95

Short post to introduce a tiny service called “quay-imager”.

This is an HTTP server that receives Quay.io Webhook notifications and automates editing of a YAML file, by replacing an image reference.

How does this work?

Deployment

This is packaged and deployed as a Kubernetes deployment.

Before deploying the release, you will need to do some minimal configuration.

Then deploy the release.

$ kubectl apply -f https://github.com/bigkevmcd/image-hooks/releases/download/v0.0.1/release.yaml

NOTE: There’s nothing in this code that requires to be run in a Kubernetes cluster, so you are free to build and deploy in other ways, for example Heroku.

Configuration

Given this configuration:

repositories:
  - name: <Quay.io username>/<Quay.io repository>
    sourceRepo: <github user>/<github repository>
    sourceBranch: master
    filePath: deploy/deployment.yaml
    updateKey: spec.template.spec.containers.0.image
    branchGenerateName: repo-imager-

This is an array of repositories, the name is used to match on the incoming Quay.io webhooks

The sourceRepo and sourceBranch reference the file to be updated, the filePath is the actual file to change.

The updateKey is a JSON path to update with the incoming key.

Finally, the branchGenerateName is used to generate a random branch name.

Quay.io configuration

You’ll need to expose your service so it can be accessed by Quay.io, and create a Webhook Notification for your repository.

This must be a “Push to Repository” notification, with a type of “Webhook POST”.

Fill in the Webhook URL with with the exposed URL (don’t forget things like ngrok if you’re testing this locally).

Finally, push an image to Quay.io:

$ docker push quay.io/<Quay.io username>/<Quay.io repository>:v1

This should trigger an update to the GitHub repository, creating a new branch (named with the prefix repo-imager-), and opening a PR with the change.

The spec.template.spec.containers.0.image field, in the deploy/deployment.yaml field is updated with the image that was pushed, for example, above, this would be replaced with something like quay.io/myuser/example:v1.

By approving this PR, ideally, you’d trigger a deployment (if you’re following GitOps processes).

IMPORTANT CAVEAT

Quay.io has no support for authenticating hooks, so unfortunately you can’t trust Webhooks, so I wouldn’t recommend this in production without some mechanism to prevent randoms on the internet triggering builds.

API

You could see this as an API for automating updating GitOps files, these are the fields you’d need to send.

It would certainly be possible to have an optional authentication header, and require API requests to send it, and block non-authenticated requests.

{
  "repository": "mynamespace/repository",
  "docker_url": "quay.io/mynamespace/repository",
  "updated_tags": [
    "latest"
  ]
}