Joshua Zenn

Introduction to GitOps with Flux v2

Sun, 07 Nov 2021 01:00:00 -0500

Overview

GitOps is a word that gets thrown around a lot these days. People are talking about it a lot becuase it solves a lot of problems that have plagued the industry for a long time. Deploying to production can be an insanely complicated process for some people, requiring multiple build steps and established procedures requiring human intervention or hacky scripts somebody wrote at 11 PM who just wanted to go home for the day. GitOps allows you to syncronize your collection of kubernetes manifest files between a repository and your cluster. That way if things go south, you can destroy your cluster and rebuild in just a few minutes. Additionally you can bring over your code review, change control, and audit procedures that you use with your application development to your cluster configuration, since now changes are no different than a PR to your application’s codebase.

When I first learned about GitOps, I really wanted to try it out on my homelab. I initially went with Argo CD, simply because that’s what everyone else was using. Although Argo CD is cool and all, there was one major deal-breaker for me. Argo CD requires nearly 8GB of RAM for normal operation and a lot of CPU power, which was more than double what I had in my test k3s cluster. What’s the point of a GitOps tool if there’s no room left on the cluster to deploy applications? As a broke college student I figured there had to be a better (and cheaper) way. The solution I found was Flux CD. It’s very lightweight, and all of it’s configuration is stored in your source control, so no need to mess with any fancy PV setups in k8s.

Flux works on top of any existing k3s or k8s cluster. I’ll be using k3s for this tutorial. You’ll need about 512MB of RAM and a few milli-CPU available in your cluster. One important thing that I found is that the Flux documentation is a little confusing at times, and I often had to delete the Flux deployment and re-deploy to implement the change. This guide will attempt to get it right the first time (assuming your needs are the same as mine).

Setup

This tutorial assumes that you already have a Kubernetes cluster of some kind already up and running. If you do not have a cluster, there are several options available to you. Minikube is an excellent choice if you only have one (somewhat-powerful) machine. k3sup is a good tool if you have a few spare computers and want to quickly turn them into Kubernetes nodes. And finally there are always hosted options such as EKS and GKE, but be aware these can get pricy very quickly. (If you enjoy a challenge and have a couple free weekends, there’s also Kubernetes The Hard Way).

All the commands below should be run using a computer with a Kubeconfig file that can connect to your cluster. In my case I have a dedicated LXC container that is inside an isolated network that I have created for my k3s cluster. Start by installing the Flux CLI tool:

# Linux
curl -s https://fluxcd.io/install.sh | sudo bash
# MacOS
brew install fluxcd/tap/flux
# Windows (requires Chocolatey)
choco install flux

Next go to GitHub and create a personal access token. Make sure that you check all the boxes under repo and read:packages. Fill in the appropriate values in the script below to bootstrap Flux into your cluster.

# On Windows, replace this with 'set'.
export GITHUB_TOKEN=<gh_token> # Your GitHub personal access token
export GITHUB_USER=<gh_username> # Your GitHub username
export GITHUB_REPO=<gh_repository> # Name of an empty or non-existant repository on your GitHub account (e.g. cluster-iac)
flux bootstrap github --components-extra=image-reflector-controller,image-automation-controller --owner=$GITHUB_USER --repository=$GITHUB_REPO --branch=main --path=./clusters/prod --personal --read-write-key

Now what does this command do? First we tell Flux that we want to bootstrap a new cluster, and we want to store our Infrastucture-as-Code (IaC) on GitHub. We then list some extra components we want to install. These will be important later when we set up automatic image update scanning. We then pass some information about GitHub to Flux. Flux will automatically create $GITHUB_REPO and watch the main branch for updates. At the very end we note that the key we attached is a --read-write-key. This extra argument is critical for Flux to be able to write back to GitHub when image updates are found. Once you run this command give Flux a few minutes to deploy everything and fully initialize. In the meantime, you should see a new private repository on GitHub that has some Kubernetes manifest files in it. Congratulations, you now have a functioning GitOps setup!

Files Walkthrough

If you log in to GitHub.com, you should see your new repository. In this repository there should be some new files. What we’re interested in is the clusters/prod directory, as this is where configuration specific to your cluster will live. It’s important to know that you can have multiple Flux clusters use the same repository. This way you can re-use your manifests and configuration across all your clusters.

In this directory, you can add several kinds of files. If you add a regular Kubernetes manifest file (*.yaml), Flux will automatically apply it when you commit, and will ensure that it stays in sync (one important caveat to this is that if you manually edit a resource using kubectl, it will get overwritten the next time Flux tries to reconcile your manifest files).

Another kind of file you can add is a Kustomization file. I have a kustomization file that points to another directory in the repository.

apiVersion: kustomize.toolkit.fluxcd.io/v1beta1
kind: Kustomization
metadata:
  name: apps
  namespace: flux-system
spec:
  interval: 10m0s
  dependsOn:
    - name: infrastructure
  sourceRef:
    kind: GitRepository
    name: flux-system
  path: ./apps/prod
  prune: true
  validation: client

This Kustomization file will apply all the manifest/Kustomization files located in apps/prod. In turn I have a Kustomization file there that loads whatever apps I want from apps/base like so:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../base/app1
  - ../base/app2
  - ../base/app3

Each of these app directories has a Kustomization files that lists all the regular Kubernetes manifest files that are needed for that app, located in the same directory:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - namespace.yaml
  - pgcluster.yaml
  - deployment.yaml
  - secrets.yaml
  - service.yaml
  - ingress.yaml
namespace: app1

This may seem a little redundant, but this gives me the flexability to keep all my app definitions in one place. Then I can enable and disable apps by commenting out one line in apps/prod/kustomization.yaml. There is no “right” way to organize your files as long as it works for you.

Wrap-up

You are now ready to jump into the world of GitOps. Coming soon is another tutorial on how to deploy your own app, completely from scratch, using only GitHub, GitHub Actions, and Flux.

Quick Bare-Bones Pastebin for Home Labs

Wed, 13 May 2020 13:00:00 -0400

As I have spun up and used hundreds of containers in my home cluster, I have run into a very annoying problem. Frequently I need to quickly copy initialization URLs, temporary authentication keys, or configuration files from machine to machine. My machines are configured for key-based authentication through SSH, so SCP is usually a no-go since I don’t want random containers with access to other containers. Uploading to Dropbox requires extra tools as well as introducing security risks. My Samba shares also have the same problem.

Eventually, I found Fiche. It’s a completely bare-bones pastebin with ZERO pre-requisites on the client machines. You only need netcat, which is installed by default on all Linux distributions, and equivilents are found in Windows and Mac OS X.

If you just want a quick Ansible or Bash script, scroll to the end.

Pre-Requisites

A Linux server to host Fiche (I will be using Debian 9 on top of LXC).
A static IP address for your Fiche server.

Download Packages

First, run this command to download all the needed packages. LXC users will need to remove sudo at the beginning of any command in this tutorial.

1
sudo apt-get install git nginx build-essential automake autoconf autotools-dev curl

Now, we’ll move to a temporary folder to download and compile Fiche.

1
2
3
4
5
cd /usr/src
git clone https://github.com/solusipse/fiche.git
cd fiche
make
make install

Security Stuff

Now move to the folder that make just created. We need to create a folder for the uploaded data. Additionally, we’ll create a fiche user so that we don’t need to use root for a public-facing service.

1
2
3
4
5
cd /usr/local/bin/fiche
mkdir fiche-data
adduser fiche
chown -R fiche /usr/local/bin/fiche
chmod o+r /usr/local/bin/fiche/fiche-data

Configure Services

First we will create a service for Fiche so that it will start automatically after a reboot. Next, we will configure Nginx to point to the fiche-data directory. In both of these files, replace fiche.firecore.lan with the DNS-resolvable hostname of your server.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
cat << EOF > /etc/systemd/system/fiche.service
[Unit]
Description=FICHE-SERVER

[Service]
ExecStart=/usr/local/bin/fiche -d fiche.firecore.lan -o /usr/local/bin/fiche-data/ -u fiche

[Install]
WantedBy=multi-user.target
EOF
rm /etc/nginx/sites-enabled/default
cat << EOF > /etc/nginx/sites-enabled/fiche
server {
	listen 80;
	server_name fiche.firecore.lan;
	charset utf-8;

	location / {
		root /usr/local/bin/fiche-data;
		index index.txt index.html;
	}
}
EOF
systemctl enable fiche
systemctl start fiche

Testing

That’s it! Now to test it. From any computer connected to the same network as your Fiche server, execute the following command (replacing fiche.firecore.lan with your server’s DNS name or IP address).

1
echo "test" | nc fiche.firecore.lan 9999

You should see the above code spit out a URL. Now to pull it up on another computer, execute the following from another computer (or the same terminal if you don’t have another one). Replace the URL with the one you recieved in the above command.

1
curl -L http://niche.firecore.lan/xxxxxxx

You should see test. That’s it. A bare-bones pastebin for moving those long URLs and tokens around your network. One word of advice, keep your Fiche server behind a firewall. The trade-off for this simplicity is no security. Anyone with HTTP access to your system can read or write to the pastebins on your system.

For the record, I am not the creator of Fiche. The original GitHub repo can be found on solusipse’s profile.

Ansible + Bash Scripts

For those of you that are looking for it, here’s the command to execute an all-in-one Bash script and also an Ansible playbook.

Bash

1
wget -O https://gist.githubusercontent.com/ARMmaster17/94dcbaddd06de8df11fd294d5014d18e/raw/52f1289efccf0427048f6e72f5257c7eca9ce6c0/install-fiche.sh | bash

Ansible Playbook

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
# Execute with ansible fiche.yml --limit hostname.example.org
---
- hosts: all
  tasks:
  - name: install pre-req software
    package:
      name:
        - git
        - nginx
        - build-essential
        - automake
        - autoconf
        - autotools-dev
        - curl
      state: present
  - name: checkout fiche
    git:
      repo: 'https://github.com/solusipse/fiche.git'
      dest: /usr/src/fiche
  - name: compile fiche
    make:
      chdir: /usr/src/fiche
  - name: install fiche
    make:
      chdir: /usr/src/fiche
      target: install
  - name: create fiche user
    user:
      name: fiche
  - name: set directory permissions
    file:
      path: /usr/local/bin/fiche
      owner: fiche
      state: directory
      mode: o=rwx
  - name: create fiche-data directory
    file:
      path: /usr/local/bin/fiche/fiche-data
      state: directory
      owner: fiche
      mode: o=rw
  - name: create fiche service
    blockinfile:
      path: /etc/systemd/system/fiche.service
      block: |
        [Unit]
        Description=FICHE-SERVER
        [Service]
        ExecStart=/usr/local/bin/fiche -d fiche.firecore.lan -o /usr/local/bin/fiche-data/ -u fiche
        [Install]
        WantedBy=multi-user.target
  - name: create nginx site for fiche
    blockinfile:
      path: /etc/nginx/sites-enabled/fiche
      block: |
        server {
        listen 80;
        charset utf-8;
        location / {
          root /usr/local/bin/fiche-data;
        index index.txt index.html;
        }
        }
  - name: remove default nginx site
    file:
      path: /etc/nginx/sites-enabled/default
      state: absent
  - name: start fiche service
    service:
      name: fiche
      enabled: yes
      state: started
  - name: restart nginx
    service:
      name: nginx
      state: restarted

Writing a Markov-Chain Chatbot

Sat, 23 Sep 2017 13:00:00 -0400

A while back, I wrote Jeffbot as a fun side project because I was bored between classes in college. JeffBot is simply a chatbot with the intelligence of a two-year-old. It learns words and what words can/can’t be used sequentially in a sentence. At first, it spews out pure garbage or just repeats back what you just said. However, over time, JeffBot learns how to make what appears to be intelligent english.

JeffBot itself is a complex app split over three different Heroku dynos and a particularly large database running on top of Rails/Sinatra/ASP.NET/I’m not even sure what anymore. In this article, I’m going to show you how to rebuild an N2 version of JeffBot in pure Ruby that can run locally with no external service/library dependencies.

First, you will need to set up a directory for JeffBot. Create a directory called JeffBot and cd into it. From now on, this directory will be referred to as ~/. Inside this directory, create a file called main.rb and paste this code in.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
# ~/main.rb

def learn(input, training_data)
    grams = input.split(' ').each_cons(2).to_a
    grams.each do |gram|
        entries = training_data.select { |entry| entry.w1.eql?(gram[0]) && entry.w2.eql?(gram[1]) }
        if entries.count == 0 || entries.nil?
            training_data << Struct::Ngram.new(gram[0], gram[1], 1)
        else
            entries.first.count += 1
        end
    end
end

def respond(input, training_data)
    word = input.split(" ").sample
    sentence = word
    for i in 0..10
        entries = training_data.select { |entry| entry.w1.eql?(word) }
        if entries.count == 0 || entries.nil?
            sentence << "."
            break
        else
            new_word = entries.sample.w2
            sentence << " " + new_word
            word = new_word
        end
    end
    return sentence
end

Struct.new("Ngram", :w1, :w2, :count)

learning_data = []

while true
    print ">"
    input = gets.chomp
    if input.eql?("exit")
        exit(0)
    end
    learn(input, learning_data)
    puts respond(input, learning_data)
end

Let’s go through this code block by block to see what it does. Let’s start in the same place that the interpreter starts, line 32. This defines a struct called an N-gram. This represents a relationship between how likely one word is to appear after another. Take a look at the following input string:

This is a sentence.

Our learn() function that we are going to cover in a minute is going to split that sentence into 2n N-grams like so.

[This, is]
[is, a]
[a, sentence]

Assuming this is the only training data provided, if we start with “a”, our chatbot would choose “sentence” as the next word. If multiple relations exist for one word, a random selection is made based on the choices (utilizing the count attribute is left as an exercise for the reader).

The remainder of the code creates a learning_data[] array that we use to store training data. It also provides a loop to take input from the user, and exit the application if they have requested to do so by typing “exit” at the prompt. Towards the bottom of the code block, we see calls to learn() and respond(). These two methods form the backbone logic of our chatbot.

Now let’s take a look at the learn() method. We call it with two parameters, input and training_data[]. The first thing we need to do is convert input into a series of 2N-grams like so.

1
grams = input.split(' ').each_cons(2).to_a

Next we iterate over the list of grams, with gram representing the relation we are currently looking at.

1
entries = training_data.select { |entry| entry.w1.eql?(gram[0]) && entry.w2.eql?(gram[1]) }

This line pulls out an N-gram relation if it already exists in our training data where both the first word and the second word match in the gram we are currently looking at.

1
2
3
4
5
if entries.count == 0 || entries.nil?
    training_data << Struct::Ngram.new(gram[0], gram[1], 1)
else
    entries.first.count += 1
end

This if block runs some logic on the results from the previous query of training_data[]. If a relation already exists, we increment the count. Otherwise we create a new Ngram Struct and shove it into training_data[].

Now let’s move on to the respond() method. We call this with the same parameters as learn(). Our first line picks the first word based on the words that the user wrote at the prompt. We use this word to start our sentence and to initialize word that we will utilize in a loop.

1
2
    word = input.split(" ").sample
    sentence = word

Next we enter a for loop. I set an arbitrary limit of 12 words in a sentence, you can increment this by increasing the range that the for loop iterates over.

1
entries = training_data.select { |entry| entry.w1.eql?(word) }

Within this for loop, we use similar logic from the learn() method. The difference is that we only have the first part of the N-gram (w1). If the number of matchine entries is zero, we add a period to the sentence and return it to the prompt loop.

1
2
3
if entries.count == 0 || entries.nil?
    sentence << "."
    break

If matches are found, we assign it to new_word. Then we add it to sentence and move it to word so we can use it as our new w1 on the next loop iteration.

That’s all there is to it. Run it by cd-ing into your project directory and running ruby ./main.rb from the terminal. When you recieve the > prompt, enter a string of words. Your new chatbot should reply back with some garbage. Talk with it some more, the more input you provide, the faster it can learn and respond with some more intelligent-sounding english.

To quit, just type “exit” at the > prompt. Note that learning data is stored in memory. This means that when you quit, all training data is lost. When you run the program again, you are starting with a fresh slate. For more permanent storage, take a look at the pg ruby gem paired with the PostgreSQL database service.

Getting Started With Disco MapReduce

Thu, 27 Apr 2017 13:00:00 -0400

A while back, I wanted to learn how to use Hadoop. I prefer to self-host my stuff as it’s free (as in pizza), and free (as in speech).

I almost immediately ran into issues getting it set up. First off, it turns out Hadoop uses Java, which I hate. It also has insane memory requirements, which my homelab couldn’t handle. To top it off, the installation docs were too confusing for something that I would use for 10 minutes and then never touch again.

Then I found Disco.

Unlike Hadoop, Disco has libraries for use in Python, which I find slightly more acceptable than Java. It also has an extremely small footprint. I was able to run a testing worker with 256mb of RAM, 1 CPU, and 1 GB of hard drive space (including all of its dependencies).

The one remaining problem, is that following the installation instructions for Disco has probably been one of the most frustrating pieces of software I’ve ever had to install. You know you’ve spent too much time trying when you’re reading a 5 year old GitHub issue, tracking down an Erlang issue related to a cross-language pipe library, all just by following the “Getting Started” guide. Of course, there’s an XKCD for that.

This guide is written as a fool-proof guide for anyone else who wants to get up and running with Disco with minimal hassle.

1. Prerequisites

There are several ways of accomplishing this. I personally used my Proxmox cluster split out over 6 different machines, all behind a private subnet with a firewall. You can do whatever as long as you meet the following requirements.

At least one machine (preferably more, dedicated boxes like docker containers or VMs)
CentOS 7 (anything else and you are on your own)
A configurable DNS server in your network (if you think you can cheat with the HOSTS file, you are in for a world of hurt)
A free afternoon (Seriously, this could take a while)

2. Hello bash my old friend

Go ahead and turn your machines on or provision your VMs. You can do one, or two, (or twenty, as I did). Make sure all of them are running CentOS 7. The minimum specs I got away with was 1 CPU, 512mb of RAM, and 1GB of hard drive space. If you are creating multiple “workers”, each one only needs 256mb of RAM. 512mb is only needed for the “master” node. Of course, if you intend on running bigger datasets than me, you will need to increase this number.

For each box you create, you will need to create a corresponding entry on your DNS server. How to do this is beyond the scope of this article and depends on what software you choose to use for this. I was able to just add the entries to my pfSense router using the configuration page. If your router can’t do this, take a look at a BIND server.

Now boot all of the boxes and run the following command.

1
2
3
$ yum install openssh-server openssh-client openssl-devel -y
$ ssh-keygen -N '' -f ~/.ssh/id_dsa
$ service sshd restart

This sets everything up for SSH connections between the master node and the workers.

3. Installing Erlang on the first try (sometimes)

Now we have to install Erlang. This is a pain to set up on CentOS for some reason. All the tutorials I found were either 5 years old or not for my distro. The solution I found was Kerl. Type the following commands into the terminals on all the machines.

1
2
$ curl -O https://raw.githubusercontent.com/kerl/kerl/master/kerl
$ chmod a+x kerl

This gets Kerl all set up so we can install Erlang. Before we clone and build Erlang though, we need to install some additional dependencies.

1
$ yum install tar gcc make perl ncurses-devel git -y

Now we can install Kerl. Run the following commands.

1
2
3
4
$ ./kerl build 17.5 17.5
$ ./kerl install 17.5 erlang/17_5/
$ . /root/erlang/17_5/activate
$ erl -version

If that works without any errors, you are now good to go. Now we can install Disco.

4. Time for the (actual) installation

1
2
$ git clone git://github.com/discoproject/disco.git
$ cd disco

At this point, what you need to do next depends on the type of installation you are performing. If you are installing the master node (or doing a single-machine install), run this:

1
$ make install

Otherwise, if you are creating a worker node that doesn’t need the web GUI, run this instead:

1
$ make install-node

Now, on both the master and worker nodes, set up the python libraries.

1
2
3
$ cd lib
$ python setup.py install --user
$ cd ..

5. Developing trust in your cluster (using math and crypto-stuff)

The last part is to set up the Erlang cookie and SSH keys. On the master server, run the disco service and stop it by running the following.

1
2
3
$ bin/disco start
// Wait a few seconds...
$ bin/disco stop

Now run the following on your master node. Replace NODE with the DNS-resolvable name of each of your worker boxes.

1
2
3
4
5
6
$ ssh-copy-id localhost
$ scp ~/.erlang.cookie localhost:
// Now run these two commands for every worker box you have,
// replacing NODE with the DNS resolvable name.
$ ssh-copy-id NODE
$ scp ~/.erlang.cookie NODE:

Finally, run the following on all the nodes:

1
$ bin/disco nodaemon

At this point, you should be able to follow the instructions in the documentation on adding all of your nodes to the web GUI and running the test wordcount program.