staticnotes.org

Encrypted cloud backups with restic

Mon, 31 Aug 2026 00:00:00 +0100

My current backup strategy is pretty inconsistent. I run a weekly system backup of my MacBook onto an external hardrive sysbackup using CarbonCopyCloner . I also have another external hardrive photos1 to store files that don’t fit onto my MacBook and which isn’t backed up at all. Unfortunately, this includes most of my photos and videos. Storing those photos in the cloud would be a convenient option. However, I am not comfortable uploading important personal documents and photos to the cloud. Even if Google Photos reminds me daily that I should.

This status quo is eventually going to fail, e.g. if the photos1 drive breaks or both my MacBook and the sysbackup drive get destroyed at the same time (burglary, fire, flooding, ransomware attack). Recently, my photo hardrive made unusually loud noises when plugged in, which prompted me to finally adopt a more solid backup approach.

Generally for backups the 3-2-1 rule is recommended (3 copies, 2 different storage media, 1 off-site backup). To accomplish this I will periodically clone the external hardrive photos1 to a new external hardrive photos2 and also set up the off-site backup. This post is mostly about how I set up the cloud backup, but I will summarize the overall approach at the end as well.

Requirements for my cloud backup

To find the right approach I wrote down my requirements:

reliable and cheap cloud storage
backups encrypted locally before upload
open-source software to prevent vendor lock-in or software disappearing in 5 years.
simple to use for someone comfortable with the command line

In the end I settled on restic which works on Linux, macOS, Windows, encrypts data locally by default, and doesn’t need to run on a server. I also looked into BorgBackup which probably would have worked as well. However, that requires running borg on the server. Moreover, the commands and basic configuration are straightforward.

Getting cheap space in the cloud

I already have a Hetzner account to host my Nextcloud instance. Luckily, they offer very cheap storage. I got the Hetzner 1TB Storage Box for 3.83€/month. This means my data is stored in their datacenter in Falkenstein, Germany. By activating automatic snapshots of the storage box in the Hetzner console, I get another backup layer. Here you can see the inside of their datacenter.

To set up the Hetzner Storage Box to work with restic, I had to enable External Reachability, enable SSH, and set a password in the Hetzner console. The communication between my local machine and the storage box will be via SSH and the file upload happens over SFTP which is based on SSH.

First I create a new pair of SSH keys I have more detailed notes on setting up SSH keys here . and then upload the public key to the storage box. In short:

Create new SSH keys:

$ ssh-keygen -t ed25519 -f ~/.ssh/storagebox

Add a host entry for the storagebox to your ~/.ssh/config file:

Host storagebox
    HostName uXXXXXX.your-storagebox.de
    User uXXXXXX
    Port 23
    IdentityFile ~/.ssh/storagebox
    WarnWeakCrypto no-pq-kex

Copy the public SSH key to the storagebox:

$ ssh-copy-id -p 23 -s -i ~/.ssh/storagebox.pub uXXXXXX@uXXXXXX.your-storagebox.de

I should now be able to interact via sftp without providing a separate password. I can test that it works with:

$ sftp storagebox

Installing and configuring `restic`

I installed restic via homebrew with brew install restic. Next, I create a restic password that restic uses to encrypt my data locally before it is uploaded. To not forget it, I stored it in my password manager and also wrote it down in my notes. I also store this password in my Apple Keychain with:

$ security add-generic-password -a "$USER" -s restic-password -w

I then configured two important environment variables in a new restic config file ~/.config/restic/env:

export RESTIC_REPOSITORY="sftp:storagebox:/home/restic-backup"
export RESTIC_PASSWORD_COMMAND="security find-generic-password -a yourusername -s restic-password -w"

RESTIC_REPOSITORY is the director on the storage box where the backups are stored. This is called the restic repository. RESTIC_PASSWORD_COMMAND tells restic which command to run to get the restic password from the Apple Keychain (all of this avoids storing it in plain text in an environment variable).

Next I created the restic repository on the storage box with:

$ restic -r sftp:storagebox:/home/restic-backup init

This is all that is needed for the initial configuration. I can now start making backups.

Basic restic backup commands

To backup the folder ~/important_docs/ to the restic repository on the storage box I run:

$ restic backup ~/important_docs --verbose

Restic calls the content of a directory at a specific point in time a snapshot. This snapshot is now available in the repository and I can restore the whole snapshot or selected files of the directory back to my local machine.

If I run the above backup command again, restic will create a second snapshot, but not upload any files as none have changed. This deduplication makes sure that the data is stored efficiently.

Inspecting the storage box

To inspect the files in my storage box I use Cyberduck . Create a new SFTP connection with the storage box details:

Server: uXXXXXX.your-storagebox.de
Port: 22
Username: uXXXXXX
SSH Private Key: ~/.ssh/storagebox

This opens the file viewer:

Figure 1. Restic repository files visible in my storage box.

I can see that (encrypted) restic files have been uploaded to the subfolder restic-backup. Of course I can also store other things in the storage box, e.g. I have a separate sync folder that I use with rsync.

Working with snapshots

Here are useful commands to inspect and manage snapshots taken in the past:

To list all available snapshots in the repository: restic snapshots.
To list all files in a specific snapshot with id 073a90db: restic ls 073a90db.
To view the size of the restic repository, run restic stats --mode raw-data.

Restoring backups

It is worth understanding how to restore a backed-up directory. Let’s say I want to restore snapshot a798b76e. If I run

$ restic restore a798b76e --target ~/restore-test

restic will reproduce all directories and files that are part of that snapshot in the provided target folder restore-test. If I just want to restore a single file from a snapshot I can use the --include flag, e.g.

$ restic restore a798b76e --include photo_of_grandma.jpg --target ~/restore-test

This will restore photo_of_grandma.jpg from snapshot a798b76e and make it available in the folder ~/restore-test. Important: To restore you will implicitely or explicitely need my restic password. Make sure you have it available.

Managing the repository’s size and integrity

Two additional commands help me manage the repository and its size. To make sure that the data on the storage box hasn’t been corrupted I periodically run:

$ restic check

This performs a structural integrity check of the data in the repository. Do the available files and the index match? Are there files not in the index? Check that it can decrypt snapshot metadata. This check is fast because it doesn’t actually read any data. If I want to also check that files are correct I need to append --read-data for a full repository read. This will likely take some time. Alternatively I can only check a subset of data with --read-data-subset=5% or --read-data-subset=500M.

The other important command is forget. My storage box only has a size of 1TB, so if I keep accumulating snapshots, I will eventually run out of space. forget and prune can be used to delete snapshots.

To manually delete a snapshot with id a798b76e I run:

$ restic forget a798b76e

This only deletes the snapshot. It doesn’t actually delete the data that was exclusive to that snapshot (not part of another snapshot). To delete this data I need to follow up with prune.

$ restic prune

or do it as part of the forget command:

$ restic forget a798b76e --prune

The restic docs recommend running a restic check after pruning operations to flag any issues.

Instead of deleting a specific snapshot, I want to specify a deletion policy, e.g. “keep one backup for the last 6 month, one for the last 4 weeks, and daily for the current week.” Restic uses the --keep-* flag to specify such a policy. Use the --dry-run flag when trying this out the first time. Some example policies:

--keep-last n keep the n last (most recent) snapshots.
--keep-daily n for the last n days which have one or more snapshots, keep only the most recent one for each day.
--keep-weekly n for the last n weeks which have one or more snapshots, keep only the most recent one for each week.

These can be combined (union of snapshots). So for my desired policy I could use:

$ restic forget --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --prune --verbose

Note that one snapshot could fulfil multiple conditions.

Scheduling and profiles with `resticprofile`

So far I only have a set of CLI commands that I would have to remember to run periodically. To actually make restic work, I want scheduling and profiles.

Scheduling to reliably run the backup daily and profiles to have different command configurations for different backup sources, e.g. I want daily backups of core folders on my MacBook and only weekly backups for my external hard drive photos1.

I am using resticprofile for both which I install via homebrew

$ brew tap creativeprojects/tap
$ brew install resticprofile

With resticprofile I can use a yml or toml configuration file to specify commands and configurations for specific paths. I created two profiles here ~/.config/resticprofile/profiles.yml:

version: "1"

# default profile set up for files on my MacBook
default:
  repository: "sftp:storagebox:/home/restic-backup"
  password-command: "security find-generic-password -a yourusername -s restic-password -w"
  initialize: false 
  force-inactive-lock: true
  retry-lock: 10m
  
  backup:
    source:
      - "~/core_data_from_my_macbook"  
    exclude:
      - "~/stuff_I_want_to_exclude/"
    exclude-file:
      - "~/.config/restic/excludes.txt"
    verbose: true
    schedule: "11:00"
  forget:
    # path: true reuses the source list from backup command
    path: true
    keep-daily: 7
    keep-weekly: 4
    keep-monthly: 6
    prune: true
    schedule: "12:00"
  check:
    read-data-subset: "5%"
    schedule: "Mon 12:30"

# for `photos1` external hard drive
photos:
  repository: "sftp:storagebox:/home/restic-backup"
  password-command: "security find-generic-password -a yourusername -s restic-password -w"
  initialize: false 
  backup:
    source:
      - "/Volumes/photos1/Photos"
  forget:
    path: true
    prune: true
    keep-last: 5

The first profile default configures my MacBook backups. It specifies which folders to backup and is configured to run daily at 11am. At 12pm restic executes my forget policy on the specified paths and at 12:30 restic runs an integrity check.

I have set up a second profile photos to configure backups from my external hard drive to the same remote restic repository. The backup command is not scheduled. I only run it after I have copied new photos onto the harddrive, which only really happens every few weeks. The forget policy ensures the last 5 snapshots are kept.

To schedule this configuration, I run resticprofile schedule. On macOS this schedules the jobs in the launchd service manager. The first time this runs, macOS will ask you to confirm permissions. I verified the schedule is set with launchctl list | grep -i restic. Moreover, I can show all configured profiles with

$ resticprofile profiles

I run the backup command for the photos profile manually with

$ resticprofile photos.backup

Conclusion

My final backup approach is show in Figure 2. My MacBook is backed up daily to the remote restic repository on the Hetzner Storage Box and weekly to an external hard drive. My photos hard drive is backed up weekly to the restic repository and monthly to a second external hard drive.

Figure 2. Final backup approach to restic cloud repository and external hard drives.

I have been running this setup for a couple of weeks and so far I am happy with it. The important files on my MacBook are about 15GB which took 20–30min for the first uploaded snapshot and only a few seconds since. The photos1 hard drive has about 250GB of data which took 4–5h for the initial upload.

It’s also nice that I can change storage vendor seamlessly, e.g. I can switch to AWS S3 or Backblaze B2 by modifying the config file and setting up new SSH keys.

Appendix

Launchd schedules with sleeping MacBook (closed lid)

I noticed that some scheduled backups failed when my MacBook was asleep (lid closed). I think that it’s because launchd resumes scheduled jobs once the MacBook awakes but a network connection isn’t always immediately available. I fixed this by checking for a network connection befor running the backup command using the run-before setting. I added the following line for backup forget and check in my profiles.yml:

# [...]
backup:
    # before running backup try 30x to reach the storage box, this should give time for network access to be available after sleep
    run-before: 
      - "for i in $(seq 1 30); do nc -z -G 5 uXXXXXX.your-storagebox.de 23 && exit 0; sleep 10; done; exit 1"
# [...]

Locks

Certain restic commands put a lock onto the repository to prevent multiple clients from working on the same files. I can list all locks with

$ restic list locks

Sometimes failed commands leave the repository locked. If you are sure it’s safe to unlock it, run:

$ restic unlock

A common failure mode is for example for the check command to run and fail and leave an exclusive lock behind on the repository, which then makes subsequent backups fail. To avoid this issue I added the following settings:

# [...]
default:
  repository: "sftp:storagebox:/home/restic-backup"
  # remove stale restic locks older than 1h
  force-inactive-lock: true
  # when restic finds a blocking lock, try locking again in 10min
  retry-lock: 10m
# [...]

Exclusion file

My profile uses an exclusion file ~/.config/restic/excludes.txt to ignore certain types of files I don’t want to back up, e.g. build artefacts or temporary files:

# Python
.venv
venv
__pycache__
*.pyc
.pytest_cache
.mypy_cache
.ruff_cache
.tox
*.egg-info

# Node / JS
node_modules
.next
.nuxt
.parcel-cache

# Editors & tooling
.idea
.vscode
*.swp
*~

# Caches & logs
.cache
*.log
*.tmp
.terraform

# OS junk
.DS_Store
._*
.Spotlight-V100
.fseventsd
.Trashes
.TemporaryItems
.DocumentRevisions-V100

Additionally I use the flag exclude-caches: true, which ignores all folders that have a CACHEDIR.TAG file, e.g. Rust cache directories.

TIL how to trigger notifications in Thunderbird for specific emails

Sun, 16 Aug 2026 00:00:00 +0100

Some training sessions in my triathlon club are very popular. The weekly swim sessions in the winter and some of the longer rides on summer weekends are limited to a few people per group. There are usually five groups for different fitness levels. People in my club are really fast at signing up, but there are always some who drop out a day or a few hours before the training session. If I didn’t sign up quickly enough I usually try to grab one of those last-minute spots. But even for those you have to be quick. I usually have between 1-5 min to react after a drop out before it gets snatched by somebody else.

Problem

The notifications for available spots usually come during work hours via email from the booking system. I want to set up the email client, Thunderbird, on my MacBook to display a notification as soon as such an email arrives.

Solution

There is no built-in feature in Thunderbird to get notified for specific emails. But I managed to set it up using the Thunderbird Extension FiltaQuilla and an osascript one-liner.

FiltaQuilla extension

In Thunderbird install the FiltaQuilla extension. This adds a number of new email filter actions to Thunderbird. After installation, navigate to the FiltaQuilla preferences and check the Run Program filter action. Then restart Thunderbird.

Notification display script

Create the script that later runs when a certain filter rule fires. I created ~/scripts/thunderbird_alert.sh with the following content

#!/bin/bash
osascript -e 'display notification "A new spot opened up. Check your inbox." with title "Mail from Triathlon Club" sound name "Glass"'

Then add execution rights to the script with chmod +x ~/scripts/thunderbird_alert.sh. I tested the notification by running the script: ~/scripts/thunderbird_alert.sh. You should see it appear in the top right corner of your screen. Mine looks like this:

Figure 1. Notification about new spots for Triathlon training sessions.

Set up filter rule

Back in Thunderbird, go to Tools -> Message Filters -> New.... Give the new rule a name, then set match on From is no_reply@triathlon.club. For the action select Run Program and paste the path to the notification script /Users/myname/scripts/thunderbird_alert.sh.

That’s it. Every time a new email with that address arrives in Thunderbird, the filter rule fires the script and I see the alert that a spot is available. Compared to previous weeks my booking rate for popular sessions has gone up a lot.

Visualising London rental bike location data

Wed, 15 Jul 2026 12:00:00 +0000

London has a great rental bike scheme that makes it easy to get around the city. Rather than free-floating you can rent bikes from fixed bike stations like the one shown below. If you get the yearly membership you even get a key that you can use to rent the bike without an app.

Image(filename='map_images/station_photo.jpg', width='400px')

Moreover, Transport for London makes the live rental station data available on this endpoint . I collected hourly snapshots of bike availability data over a month so I can play around with it with DuckDB’s new spatial features and QGIS.

Collecting the data

I created a simple script fetch_bike_data.sh and a cronjob on a Hetzner VPS to request and store the data every hour. Here is the script:

#!/bin/bash
DATE=\\((date +"%Y.%m.%d.%H.%M.%S")
DIR=/home/projects/bike\_data

mkdir -p \\)DIR
curl -o santander.$DATE.xml https://tfl.gov.uk/tfl/syndication/feeds/cycle-hire/livecyclehireupdates.xml

After adding permissions to the script with chmod +x fetch_bike_data.sh I modify crontab:

crontab -e

and add the following line to run this script:

0 * * * * /home/projects/bike_data/fetch_bike_data.sh

Each request is stored to an XML-file on disk which has the following format:

<stations lastUpdate="1784031061002" version="2.0">
    <station>
        <id>1</id>
        <name>River Street , Clerkenwell</name>
        <terminalName>001023</terminalName>
        <lat>51.52916347</lat>
        <long>-0.109970527</long>
        <installed>true</installed>
        <locked>false</locked>
        <installDate>1278947280000</installDate>
        <removalDate/>
        <temporary>false</temporary>
        <nbBikes>0</nbBikes>
        <nbStandardBikes>0</nbStandardBikes>
        <nbEBikes>0</nbEBikes>
        <nbEmptyDocks>19</nbEmptyDocks>
        <nbDocks>19</nbDocks>
    </station>
    <station>
        <id>2</id>
        <name>Phillimore Gardens, Kensington</name>
        <terminalName>001018</terminalName>
        <lat>51.49960695</lat>
        <long>-0.197574246</long>
        <installed>true</installed>
        <locked>false</locked>
        <installDate>1278585780000</installDate>
        <removalDate/>
        <temporary>false</temporary>
        <nbBikes>6</nbBikes>
        <nbStandardBikes>6</nbStandardBikes>
        <nbEBikes>0</nbEBikes>
        <nbEmptyDocks>28</nbEmptyDocks>
        <nbDocks>37</nbDocks>
    </station>
[...]

As you can see the raw data contains per-station information about its name, location, and bike availability.

Convert to DuckDB table

I load and combine all the XML-files into a DuckDB database london_bike_data so that I can use SQL to analyse it. Before I do that I install the extensions spatial and h3 for the geospatial features and webbed to read xml files.

import duckdb
from IPython.display import Image

# Connect to DuckDB
conn = duckdb.connect('london_bike_data.duckdb')

conn.execute("INSTALL spatial;")
conn.execute("INSTALL h3 FROM community;")
conn.execute("INSTALL webbed FROM community;")
conn.execute("LOAD spatial;")
conn.execute("LOAD h3;")
conn.execute("LOAD webbed;");

The next query loads all collected xml-files into one DuckDB table and enforces the schema, adds metadata, and handles missing data issues. Note the Hetzner instance was based in Helsinki which I account for when converting the filename into a timestamp with the correct time zone.

conn.execute("""
    CREATE OR REPLACE TABLE londonbikestations AS
     SELECT
            strptime(
            regexp_extract(filename, 'santander\\.(\\d{4}\\.\\d{2}\\.\\d{2}\\.\\d{2}\\.\\d{2}\\.\\d{2})\\.xml', 1),
            '%Y.%m.%d.%H.%M.%S'
        ) AT TIME ZONE 'Europe/Helsinki' AT TIME ZONE 'Europe/London' AS downloaded_at,
                 id::INTEGER AS id,
                 name,
                 terminalName AS terminal_name,
                 lat::DOUBLE AS lat,
                 long::DOUBLE AS lng,
                 installed::BOOLEAN AS installed,
                 locked::BOOLEAN AS locked,
                 temporary::BOOLEAN AS temporary,
                 nbBikes::INTEGER AS nb_bikes,
                 nbStandardBikes::INTEGER AS nb_standard_bikes,
                 nbEBikes::INTEGER AS nb_ebikes,
                 nbEmptyDocks::INTEGER AS nb_empty_docks,
                 nbDocks::INTEGER AS nb_docks,
                epoch_ms(installDate::BIGINT) AS install_date,
                CASE WHEN removalDate = '' THEN NULL ELSE epoch_ms(removalDate::BIGINT) END AS removal_date
             FROM read_xml('data/santander.*.xml', all_varchar=true, filename=true)
""")
conn.sql("select count(*) from londonbikestations")

┌──────────────┐
│ count_star() │
│    int64     │
├──────────────┤
│       707197 │
└──────────────┘

The query below shows the station data that is available. The table contains the station id, name and coordinates (lat, lng) and when it was installed and/or removed. For every station I have the capacity nb_docks and the current number of rentable bikes nb_bikes (split into standard and ebikes).

conn.sql("describe table londonbikestations")

┌───────────────────┬─────────────┬─────────┬─────────┬─────────┬─────────┐
│    column_name    │ column_type │  null   │   key   │ default │  extra  │
│      varchar      │   varchar   │ varchar │ varchar │ varchar │ varchar │
├───────────────────┼─────────────┼─────────┼─────────┼─────────┼─────────┤
│ downloaded_at     │ TIMESTAMP   │ YES     │ NULL    │ NULL    │ NULL    │
│ id                │ INTEGER     │ YES     │ NULL    │ NULL    │ NULL    │
│ name              │ VARCHAR     │ YES     │ NULL    │ NULL    │ NULL    │
│ terminal_name     │ VARCHAR     │ YES     │ NULL    │ NULL    │ NULL    │
│ lat               │ DOUBLE      │ YES     │ NULL    │ NULL    │ NULL    │
│ lng               │ DOUBLE      │ YES     │ NULL    │ NULL    │ NULL    │
│ installed         │ BOOLEAN     │ YES     │ NULL    │ NULL    │ NULL    │
│ locked            │ BOOLEAN     │ YES     │ NULL    │ NULL    │ NULL    │
│ temporary         │ BOOLEAN     │ YES     │ NULL    │ NULL    │ NULL    │
│ nb_bikes          │ INTEGER     │ YES     │ NULL    │ NULL    │ NULL    │
│ nb_standard_bikes │ INTEGER     │ YES     │ NULL    │ NULL    │ NULL    │
│ nb_ebikes         │ INTEGER     │ YES     │ NULL    │ NULL    │ NULL    │
│ nb_empty_docks    │ INTEGER     │ YES     │ NULL    │ NULL    │ NULL    │
│ nb_docks          │ INTEGER     │ YES     │ NULL    │ NULL    │ NULL    │
│ install_date      │ TIMESTAMP   │ YES     │ NULL    │ NULL    │ NULL    │
│ removal_date      │ TIMESTAMP   │ YES     │ NULL    │ NULL    │ NULL    │
└───────────────────┴─────────────┴─────────┴─────────┴─────────┴─────────┘
  16 rows                                                       6 columns

The removal_date column seems not trustworthy since there is definitely activity (changes in nb_bikes) happening at stations that have a removal date, e.g.

conn.sql("select downloaded_at, name, terminal_name, nb_bikes, removal_date from londonbikestations where terminal_name = '000968' order by downloaded_at")

┌─────────────────────┬──────────────────────────┬───────────────┬──────────┬─────────────────────┐
│    downloaded_at    │           name           │ terminal_name │ nb_bikes │    removal_date     │
│      timestamp      │         varchar          │    varchar    │  int32   │      timestamp      │
├─────────────────────┼──────────────────────────┼───────────────┼──────────┼─────────────────────┤
│ 2026-05-08 22:00:01 │ Warwick Row, Westminster │ 000968        │        6 │ 2018-10-18 06:56:00 │
│ 2026-05-08 23:00:01 │ Warwick Row, Westminster │ 000968        │        6 │ 2018-10-18 06:56:00 │
│ 2026-05-09 00:00:01 │ Warwick Row, Westminster │ 000968        │        6 │ 2018-10-18 06:56:00 │
│ 2026-05-09 01:00:01 │ Warwick Row, Westminster │ 000968        │        6 │ 2018-10-18 06:56:00 │
│ 2026-05-09 02:00:01 │ Warwick Row, Westminster │ 000968        │        6 │ 2018-10-18 06:56:00 │
│ 2026-05-09 03:00:01 │ Warwick Row, Westminster │ 000968        │        6 │ 2018-10-18 06:56:00 │
│ 2026-05-09 04:00:01 │ Warwick Row, Westminster │ 000968        │        6 │ 2018-10-18 06:56:00 │
│ 2026-05-09 05:00:01 │ Warwick Row, Westminster │ 000968        │        6 │ 2018-10-18 06:56:00 │
│ 2026-05-09 06:00:01 │ Warwick Row, Westminster │ 000968        │        6 │ 2018-10-18 06:56:00 │
│ 2026-05-09 07:00:01 │ Warwick Row, Westminster │ 000968        │        6 │ 2018-10-18 06:56:00 │
│          ·          │            ·             │   ·           │        · │          ·          │
│          ·          │            ·             │   ·           │        · │          ·          │
│          ·          │            ·             │   ·           │        · │          ·          │
│ 2026-06-14 12:00:01 │ Warwick Row, Westminster │ 000968        │        3 │ 2018-10-18 06:56:00 │
│ 2026-06-14 13:00:01 │ Warwick Row, Westminster │ 000968        │        5 │ 2018-10-18 06:56:00 │
│ 2026-06-14 14:00:01 │ Warwick Row, Westminster │ 000968        │        9 │ 2018-10-18 06:56:00 │
│ 2026-06-14 15:00:01 │ Warwick Row, Westminster │ 000968        │        4 │ 2018-10-18 06:56:00 │
│ 2026-06-14 16:00:01 │ Warwick Row, Westminster │ 000968        │        2 │ 2018-10-18 06:56:00 │
│ 2026-06-14 17:00:01 │ Warwick Row, Westminster │ 000968        │        7 │ 2018-10-18 06:56:00 │
│ 2026-06-14 18:00:01 │ Warwick Row, Westminster │ 000968        │        9 │ 2018-10-18 06:56:00 │
│ 2026-06-14 19:00:01 │ Warwick Row, Westminster │ 000968        │       10 │ 2018-10-18 06:56:00 │
│ 2026-06-14 20:00:01 │ Warwick Row, Westminster │ 000968        │       11 │ 2018-10-18 06:56:00 │
│ 2026-06-14 21:00:01 │ Warwick Row, Westminster │ 000968        │        8 │ 2018-10-18 06:56:00 │
└─────────────────────┴──────────────────────────┴───────────────┴──────────┴─────────────────────┘
  887 rows (20 shown)                                                                   5 columns

I have confirmed that this station still exists (here ) so I will just ignore this column.

I want to get an overview of the total capacity across all stations. The exact number fluctuates slightly between snapshots but the capacity seems to be around 20.9k. The maximum number of docked bikes was 7708 standard bikes and 1131 eBikes. The official numbers given by TfL are 10,000 standard bikes and 2,000 eBikes.

conn.sql("select downloaded_at, sum(nb_standard_bikes) as sum_docked_standard_bikes, sum(nb_ebikes) as sum_docked_ebikes, sum(nb_docks) as sum_station_capacity from londonbikestations group by downloaded_at order by sum_docked_standard_bikes")

┌─────────────────────┬───────────────────────────┬───────────────────┬──────────────────────┐
│    downloaded_at    │ sum_docked_standard_bikes │ sum_docked_ebikes │ sum_station_capacity │
│      timestamp      │          int128           │      int128       │        int128        │
├─────────────────────┼───────────────────────────┼───────────────────┼──────────────────────┤
│ 2026-06-04 08:00:01 │                      6008 │               795 │                20963 │
│ 2026-06-01 08:00:01 │                      6047 │               585 │                20934 │
│ 2026-06-03 08:00:01 │                      6056 │               948 │                20951 │
│ 2026-06-02 08:00:01 │                      6082 │               927 │                20921 │
│ 2026-06-04 17:00:01 │                      6122 │              1185 │                20963 │
│ 2026-06-09 08:00:02 │                      6127 │               830 │                20962 │
│ 2026-05-28 08:00:01 │                      6191 │               692 │                20950 │
│ 2026-06-01 17:00:01 │                      6201 │              1086 │                20892 │
│ 2026-05-27 17:00:01 │                      6203 │               829 │                20950 │
│ 2026-06-04 10:00:01 │                      6210 │               983 │                20963 │
│          ·          │                        ·  │                ·  │                  ·   │
│          ·          │                        ·  │                ·  │                  ·   │
│          ·          │                        ·  │                ·  │                  ·   │
│ 2026-05-16 06:00:01 │                      7668 │              1081 │                20969 │
│ 2026-05-15 01:00:01 │                      7671 │              1182 │                20949 │
│ 2026-05-15 03:00:01 │                      7674 │              1169 │                20949 │
│ 2026-05-16 01:00:01 │                      7678 │              1172 │                20942 │
│ 2026-05-15 02:00:01 │                      7678 │              1178 │                20949 │
│ 2026-05-16 00:00:01 │                      7679 │              1186 │                20942 │
│ 2026-05-16 02:00:01 │                      7699 │              1157 │                20942 │
│ 2026-05-16 05:00:01 │                      7703 │              1106 │                20969 │
│ 2026-05-16 03:00:01 │                      7705 │              1143 │                20942 │
│ 2026-05-16 04:00:01 │                      7708 │              1131 │                20942 │
└─────────────────────┴───────────────────────────┴───────────────────┴──────────────────────┘
  887 rows (20 shown)                                                              4 columns

The largest station is Jubilee Plaza in Canary Wharf.

conn.sql("select name, nb_bikes, nb_docks from londonbikestations order by nb_docks desc limit 1")

┌─────────────────────────────┬──────────┬──────────┐
│            name             │ nb_bikes │ nb_docks │
│           varchar           │  int32   │  int32   │
├─────────────────────────────┼──────────┼──────────┤
│ Jubilee Plaza, Canary Wharf │       42 │       63 │
└─────────────────────────────┴──────────┴──────────┘

Busiest stations

To approximate the busiest stations I sum the absolute hour-to-hour change in nb_bikes per station. This approximates total churn (bikes leaving and arriving) since simultaneous arrivals and departures within the same hour cancel out.

conn.execute("""
    CREATE OR REPLACE TABLE busy_stations AS   
    WITH hourly_changes AS (
        SELECT
            terminal_name,
            ST_POINT(lng, lat) AS geometry,
            name,
            downloaded_at,
            nb_bikes,
            abs(nb_bikes - lag(nb_bikes) OVER w) AS abs_change
        FROM londonbikestations
        WINDOW w AS (PARTITION BY terminal_name ORDER BY downloaded_at)
    )
    SELECT
        terminal_name,
        geometry,
        any_value(name) AS name,
        sum(abs_change) AS total_churn,
        count(*) AS nb_snapshots
    FROM hourly_changes
    GROUP BY terminal_name, geometry
    ORDER BY total_churn DESC
""")
conn.sql("""select name, total_churn from busy_stations limit 20""")

┌──────────────────────────────────────────┬─────────────┐
│                   name                   │ total_churn │
│                 varchar                  │   int128    │
├──────────────────────────────────────────┼─────────────┤
│ Hop Exchange, The Borough                │        3218 │
│ Soho Square , Soho                       │        3160 │
│ Argyle Street, Kings Cross               │        2930 │
│ Waterloo Station 2, Waterloo             │        2753 │
│ Hyde Park Corner, Hyde Park              │        2605 │
│ Waterloo Station 1, Waterloo             │        2554 │
│ Moorfields, Moorgate                     │        2527 │
│ Brushfield Street, Liverpool Street      │        2514 │
│ Waterloo Station 3, Waterloo             │        2389 │
│ St. James's Square, St. James's          │        2360 │
│ Jubilee Plaza, Canary Wharf              │        2289 │
│ Eagle Wharf Road, Hoxton                 │        2263 │
│ Cheapside, Bank                          │        2202 │
│ Imperial College, Knightsbridge          │        2181 │
│ Worship Street, Shoreditch               │        2167 │
│ Natural History Museum, South Kensington │        2142 │
│ Whitehall Place, Strand                  │        2042 │
│ Queen's Gate (North), Kensington         │        2019 │
│ Malet Street, Bloomsbury                 │        2013 │
│ Exhibition Road, Knightsbridge           │        1987 │
└──────────────────────────────────────────┴─────────────┘
  20 rows                                      2 columns

None of these stations are surprising. As we can see later on the map all of these stations are in the city centre or close to commuter train stations.

Busiest hours of the day

Using the same total-churn metric I group the data by hour of day to see when bike activity peaks.

conn.sql("""
    WITH hourly_changes AS (
        SELECT
            terminal_name,
            downloaded_at,
            nb_bikes,
            abs(nb_bikes - lag(nb_bikes) OVER w) AS abs_change
        FROM londonbikestations
        WINDOW w AS (PARTITION BY terminal_name ORDER BY downloaded_at)
    )
    SELECT
        hour(downloaded_at) AS hour_of_day,
        sum(abs_change) AS total_churn,
        count(*) AS nb_snapshots
    FROM hourly_changes
    GROUP BY hour_of_day
    ORDER BY total_churn
""")

┌─────────────┬─────────────┬──────────────┐
│ hour_of_day │ total_churn │ nb_snapshots │
│    int64    │   int128    │    int64     │
├─────────────┼─────────────┼──────────────┤
│           3 │        3911 │        29501 │
│           4 │        4111 │        29501 │
│           2 │        5398 │        29501 │
│           5 │        7441 │        29502 │
│           1 │        8518 │        29501 │
│           0 │       13462 │        29501 │
│           6 │       20333 │        29502 │
│          23 │       21172 │        29500 │
│          22 │       29554 │        29500 │
│          21 │       33808 │        28702 │
│           · │         ·   │          ·   │
│           · │         ·   │          ·   │
│           · │         ·   │          ·   │
│          13 │       40336 │        29497 │
│          20 │       40433 │        29499 │
│          10 │       41575 │        29495 │
│           7 │       45237 │        29503 │
│          19 │       47598 │        28707 │
│          16 │       54813 │        29498 │
│           9 │       57626 │        29498 │
│          17 │       64213 │        29498 │
│          18 │       64873 │        30300 │
│           8 │       78166 │        29503 │
└─────────────┴─────────────┴──────────────┘
  24 rows (20 shown)             3 columns

We see the highest activity from 7:00-8:00, 8:00-9:00, 16:00-17:00, 17:00-18:00 which are peak commuter hours and the lowest activity in the early morning.

Storing geometric data as parquet files

Before I can use QGIS for map-based visualisations I need to store the relevant data in a compatible format.

I want to show the number of bikes available across the city at different times of the day (I picked Wednesday 27.5.2026). To aggregate the number of available bikes spatially, I group by H3 cells . If the coordinates of the bike station fall into the cell I sum all available bikes.

conn.execute("""
    CREATE OR REPLACE TABLE bike_station_geo AS
        SELECT *, ST_POINT(lng, lat) AS geometry FROM londonbikestations where downloaded_at::date = '2026-05-27'   
""")
conn.execute("""
    CREATE OR REPLACE TABLE h3_cell_covering AS
        SELECT
            downloaded_at,
            H3_LATLNG_TO_CELL(
                ST_Y(geometry),
                ST_X(geometry),
                7
            ) AS hexagon,
            sum(nb_bikes) AS nb_bikes
        FROM bike_station_geo
        GROUP BY 1, 2
""");

I then store the h3 bike counts table as a parquet file. This involves conversion of the hexagon data into WKB(well-known-binary) that GeoParquet uses to store geometries. Note: By default the geometry in DuckDB is assumed to be EPSG:4326.

conn.execute("""
COPY (
      SELECT geometry: ST_ASWKB(H3_CELL_TO_BOUNDARY_WKT(hexagon)::geometry),
              date_trunc('hour', downloaded_at) AS downloaded_at,
             nb_bikes
      FROM   h3_cell_covering
  ) TO 'available_bikes_per_h3_7_cell.parquet';
  """);

I also want to visualise the location of the busy stations on the map, so I store the busy stations as a separate file.

conn.execute("""
COPY (
      SELECT *  FROM busy_stations
  ) TO 'busy_stations.parquet';
  """);

Visualising availability using QGIS

QGIS (3.44.10) is an open-source geographic information system. It allows combining, viewing, and analysing tabular data with geographic and mapping data.

I want to display our parquet data over an actual map of London. To get this map, I install the plugin NextGIS MapServices and then restart QGIS to make it available. I use OpenStreetMap as the base map. In the toolbar click on QuickMapService(Globe icon) and add the OpenstreetMap OSM Standard. Next, in Project Properties (Cmd+Shift+P)-> CRS change the Projection or Coordinate Reference System to EPSG:3857 (Pseudo-Mercator).

Visualising busy stations

Drag and drop the busy_stations.parquet file into the Layers field (bottom left). Ensure under Layer Properties –> Assigned Coordinate Reference System that EPSG:4326 - WGS 84 is selected. This allows QGIS to correctly reproject the geography data from the file onto the map.

Under Layer Properties –> Symbology I can change the style of the location marker to show all the stations. You can see all the coordinates of stations in the dataset below.

Image(filename='map_images/station_locations.jpg')

To only show the top 20 busiest stations as calculated in the previous section I add a filter on the busy stations layer. (Right click on layer –> Filter, then add total_churn > 1900). I also want to show the station names, so I go to Layer Properties –> Label –> Single labels and select for value the Expression Builder and then use the expression: "name" || ' (' || to_string("total_churn") || ')' to concatenate the churn value to the station name in brackets.

Image(filename='map_images/busy_stations.jpg')

Unsurprisingly the busiest stations are in the centre where many Londoners work (Bank, Canary Wharf, Soho, Westminster) or close to major train stations (Liverpool Street, King’s Cross, Waterloo, Hop Exchange).

Rental bike availability during commute hours

I also want to visualise how the rental bikes move before and after morning rush hour. My expectation is that fewer bikes are initially in the city centre (zone 1) and more bikes in the outer residential areas (zones 2 and 3). Then around 11am we should see a concentration of bikes in zone 1.

I drag and drop the relevant data file available_bikes_per_h3_7_cell.parquet to create a new Layer. Again, I first ensure that the projection for this dataset is set to EPSG:4326 - WGS 84. The hexagons should now appear on top of the map. Remember that the file has availability data for every hour of the day, so I add a Layer filter: "downloaded_at" = '2026-05-27T06:00:00.000' to only use data from one point in time before the start of the rush hour.

To style the hex fields according to the number of available bikes in the hexagon area I make the following changes in the Layer properties:

Symbology –> Graduated –> Value: nb_bikes, Color Ramp: RdYlGr, Mode: Equal Interval, Opacity: 60%
Labels –> Single Labels –> Value: nb_bikes, 18Pt, Bold.

I have filtered the layer once for 6am and once for 11am and here are the two outputs:

display(Image(filename='map_images/bike_availibility_6am.jpg'))

display(Image(filename='map_images/bike_availibility_11am.jpg'))

As expected you can see that a lot of the residential districts along the water, West Kensington, Fulham, Wandsworth, Battersea, Elephant and Castle and in the east around Limehouse, Mile End, Bethnal Green see an outflow of available bikes. The central districts around the City of London, Mayfair, King’s Cross and Westminster accumulate a lot of the bikes by 11am.

Conclusion

It turns out to be pretty easy to collect, explore and visualise the London bike station data. DuckDB makes it really easy to ingest hundreds of XML files and the H3 and spatial extensions allow easy geographical aggregation. This is my first time using QGIS. It took 20min to get up to speed with it, but it’s really cool how it fuses mapping data and geographical and analytical data. I will try to use it in other projects as well.

References

I was inspired by Mark Litwintschik’s ebike fleet monitoring blog post
A useful QGIS Tutorial is available here

Jupyter Notebook

You can find the Jupyter notebook and the datasets for this post here .

TIL how to profile memory in Python

Tue, 21 Apr 2026 00:00:00 +0100

In my job I often work on pipelines that process a large amount of data in batches. Sometimes the batches are defined outside the task and sometimes the task itself processes its data in batches.

Problem

A common question is how to choose the batch size that is used to break up the data to be processed into smaller manageable parts. In many cases increasing batch size means decreasing the time for the whole dataset to be processedThis is of course not always the case, but often holds true in practice.. At the same time it increases the memory usage of the task at any point in time.

When these tasks run in containers in Kubernetes they tend to have a fixed amount of memory available. Increasing the batch size too much means that the container will run out of memory and the task will fail.

So the question that often comes up is “How do I choose the largest batch that will safely fit on the container?”

Solution

Our data transformations are written in Python. I can use the memory_profiler package to sample memory consumption while the task runs and stash the data in a file.

Install the package (in this case in a poetry project):
```
$ poetry add --group dev memory-profiler
```
(Ideally:) Make your Python code locally runnable.
Prepend your Python run with mprof run:
```
$ poetry run mprof run dataprocessing_task.py
```
This will record memory usage over time and stash it as a .dat file in the current directory.
You can then plot the memory usage over time with
```
$ poetry run mprof plot
```
which plots the last found file with matplotlib.

The .dat files are pretty simple:

CMDLINE /Users/dir/.venv/bin/python dataprocessing_task.py
MEM 0.187500 1776791347.5081
MEM 40.484375 1776791347.6133
MEM 59.218750 1776791347.7185
MEM 89.812500 1776791347.8203
MEM 114.078125 1776791347.9246
MEM 121.906250 1776791348.0287
MEM 133.640625 1776791348.1339
MEM 140.921875 1776791348.2365
MEM 151.546875 1776791348.3406
MEM 156.515625 1776791348.4457
MEM 167.468750 1776791348.5508
MEM 178.625000 1776791348.6560
MEM 194.421875 1776791348.7611
MEM 211.218750 1776791348.8663
MEM 225.031250 1776791348.9714
MEM 236.187500 1776791349.0765
MEM 252.156250 1776791349.1817

where the first column is the memory usage in MiB and the second column is a unix timestamp.

In my use case I wanted to compare memory consumption of two different batch sizes (30k and 300k). So I profiled the task twice and stored both .dat files.

I then use this claude-generated Python script to visualise and compare memory consumption of the batch size options:

import glob
import sys
from datetime import datetime
from pathlib import Path
import matplotlib.pyplot as plt


def load_mprof(path: str) -> tuple[list[datetime], list[float]]:
    """Parse an mprof .dat file. Returns (timestamps, memory_in_MiB)."""
    times, mems = [], []
    with open(path) as f:
        for line in f:
            parts = line.split()
            if not parts or parts[0] != "MEM":
                continue
            mems.append(float(parts[1]))
            times.append(datetime.fromtimestamp(float(parts[2])))
    return times, mems


def main(paths: list[str]) -> None:
    if not paths:
        paths = sorted(glob.glob("mprofile_*.dat"))
    if not paths:
        raise SystemExit("No .dat files found.")

    fig, ax = plt.subplots(figsize=(10, 5))
    for path in paths:
        times, mems = load_mprof(path)
        if not times:
            print(f"skip {path}: no MEM samples")
            continue
        # plot seconds since start of this run, so runs align on x-axis
        t0 = times[0]
        xs = [(t - t0).total_seconds() for t in times]
        ax.plot(xs, mems, label=Path(path).stem)

    ax.set_xlabel("Time since run start (s)")
    ax.set_ylabel("Memory (MiB)")
    ax.set_title("Memory usage over time")
    ax.legend()
    ax.grid(True, alpha=0.3)
    plt.tight_layout()
    plt.show()


if __name__ == "__main__":
    main(sys.argv[1:])

Here is the plot:

Figure 1. Memory usage of python task over time for different batch sizes.

In this case my container had 6GB of memory available for the Python task, so I could safely increase the batch size to 300k and reduce the runtime of the task by 5x.

UK house price changes in real terms

Sun, 11 Jan 2026 00:00:00 +0000

This post is a response to a newspaper article I read this morning. Every month the FT is reporting on how the housing market in the UK is performing as people love to talk about it:

UK house prices rise less than expected in 2025 as growth slows

UK house prices rose by 0.6 per cent in 2025 after a slowdown at the end of the year, according to lender Nationwide. […] Prices fell 0.4 per cent between November and December to an average of £271,068. Both figures were below analysts’ expectations of a 1.2 per cent annual rise and a 0.1 per cent month-on-month expansion. FT Weekend (02.01.2026)

I am asking myself how useful this statistic is for the average reader, especially for a reader who wants to buy or sell a property.

Two issues are immediately obvious:

The average yearly house price change across all of the UK might not agree with the house price performance in the areas I live in or I want to move to. There are also differences by house type, e.g. flat vs. detached house vs. terraced house. Different types of property might even perform in opposite directions.
British people often view property, at least partially, as an investment opportunity. However, in those cases it would help to at least report the real (inflation-adjusted) house price change, not just the nominal change. An increase of 0.6% in nominal house prices sounds different when one doesn’t mention that inflation in 2025 was around 3.6% (UK CPI). Meanwhile other forms of investments did quite well. The FTSE 100 nominally returned 21.4% (which to be fair was a positive outlier this year compared to an average of 6% over the last 10 years).

Fortunately, it is not that difficult to compute real house prices more closely to my situation. The raw data is openly available via API:

The HM Land Registry registers the ownership of property in the UK and provides aggregate monthly data of the average transactions by region and property type.
The Office for National Statistics (ONS) provides monthly inflation data (CPI) for the UK.

So, in this post I want to compute and visualise how real flat and detached house prices in specific areas performed both nominally and in real terms.

Querying the housing price data

I am using the UK House Price Index of the Land Registry API:

http://landregistry.data.gov.uk/data/ukhpi/region

This allows me to query the monthly average house price value for different regions and property types. Important to note here is that this data is based on the successful property transactions in a region which we use here as an approximation of the average value of similar properties in the region (that were or weren’t sold).

I am based in London and interested in how the housing market performs close to me or areas that I am interested in living in. So I will query the data for all London boroughs as well as some South England regions around London: Kent, Surrey, Buckinghamshire, and Oxfordshire.

I then resample the monthly data to quarterly data by taking the mean across the 3 months of a quarter.

I then store all the data in parquet files. You can find them here .

Inspecting the data

Let’s have a look at how the data is stored in the file:

import pandas as pd
import warnings
import matplotlib.pyplot as plt
pd.set_option('display.float_format', lambda x: '%.2f' % x)
warnings.simplefilter('always', category=UserWarning)

Here I am loading the file for the different counties that I am interested in:

df_regions = pd.read_parquet('./data/uk_house_prices.parquet')
df_regions['is_london_borough'] = False
df_regions.sample(5,ignore_index=True)

	Quarter	Region	Price	PropertyType	is_london_borough
0	2023Q4	Surrey	517233.67	All	False
1	2011Q3	Kent	192827.67	All	False
2	2014Q2	Oxfordshire	277195.00	Semi_detached	False
3	2013Q4	London	415838.00	Semi_detached	False
4	2021Q3	Buckinghamshire	431147.67	All	False

and here I am loading the file for the London boroughs:

df_boroughs = pd.read_parquet('./data/london_borough_prices.parquet')
df_boroughs['is_london_borough'] = True
df_boroughs.sample(5,ignore_index=True)

	Quarter	Region	Price	PropertyType	is_london_borough
0	2020Q4	Hillingdon	406610.67	Terraced	True
1	2024Q4	Redbridge	490087.67	All	True
2	2017Q2	Lewisham	910228.33	Detached	True
3	2011Q3	Islington	369223.67	Flat	True
4	2010Q3	Enfield	661067.33	Detached	True

You can see that the dataset shows the average quarterly housing transaction values by region. The column PropertyType shows what type of property was sold (Flat, Terraced, Detached, Semi_detached).

Computing real house prices

Next we adjust the nominal house prices from the Land Registry for inflation and compute the real house prices in terms of “2025 pounds”. I have queried the Office for National Statistics API to get the UK’s quarterly CPI data and stored it in a parquet file. I am loading this data here:

cpi_df = pd.read_parquet('./data/cpi_data.parquet')
cpi_df.tail(5)

	Quarter	CPI
146	2024Q3	134.10
147	2024Q4	135.20
148	2025Q1	136.00
149	2025Q2	138.50
150	2025Q3	139.20

With the consumer price index, we can compute the real price of housing. This real price answers the question: “How much was the property worth in last quarter’s pounds?” If you experienced inflation, your money is worth less, and you will need to pay more for the same house in today’s money terms. If you experienced deflation, your money is worth more, and you will need to pay less for a house in today’s terms (all else equal).

The formula to compute the real house price in terms of the current CPI:

real_price_current = nominal_price_t * (CPI_current / CPI_t)

# Merging the dataframes here to do real price calculation in one step
df = pd.concat([df_regions, df_boroughs])

# Get the most recent CPI value
current_cpi = cpi_df['CPI'].iloc[-1]
latest_quarter = cpi_df['Quarter'].iloc[-1]

# Merge CPI df with house prices df
df = df.merge(cpi_df, on='Quarter', how='left')

# For quarters without CPI data yet, use the most recent CPI
df['CPI'] = df['CPI'].fillna(current_cpi)

# Calculate real prices using above formula 
df['RealPrice'] = df['Price'] * (current_cpi / df['CPI'])

df.sample(5,ignore_index=True)

	Quarter	Region	Price	PropertyType	is_london_borough	CPI	RealPrice
0	2012Q3	City Of Westminster	2138137.67	Semi_detached	True	96.10	3097073.50
1	2011Q3	Croydon	280930.00	Semi_detached	True	93.80	416902.52
2	2016Q3	Bromley	898828.67	Detached	True	100.90	1240009.42
3	2017Q4	Ealing	1160573.67	Detached	True	104.60	1544472.80
4	2025Q2	City Of London	792392.33	Flat	True	138.50	796397.20

As you can see we now have the column RealPrice which gives us the mean house price transaction per quarter and region in real terms. Now we can visualise the data to finally see what happened.

Visualising nominal and real house prices over time

My goal is to plot the nominal and real house prices over time. To keep things visually simpler on the charts I am only selecting a few London boroughs that I am interested in and only look at data since 2015. For London boroughs I am looking at Flats and Terraced houses and for the counties I am only looking at detached houses.

df = df[df['Quarter'] >= '2015Q1'].copy()

SELECTED_BOROUGHS = [
    'Ealing',
    'Hackney',
    'Hammersmith And Fulham',
    'Hounslow',
    'Islington',
    'Richmond Upon Thames',
    'Wandsworth',
]
df_boroughs = df[df['Region'].isin(SELECTED_BOROUGHS) & (df['PropertyType'].isin(['Flat', 'Terraced']))].copy()
df_regions = df[(df['is_london_borough'] == False) & (df['PropertyType'] == 'Detached')].copy()

In the file utils.py I added some code to create the plots and tables. You can safely skip this if not interested. Importing them here in the notebook:

from utils import create_table, plot_prices_by_type

Detached Houses in South England Counties

I am first looking at the house price performance of detached houses in five different English counties. I am plotting both the nominal prices (dashed line) and the real prices (solid line).

I am also creating a table with the real price changes vs. 1 year ago, 2 years ago, 5 years ago, 10 years ago.

fig, axes = plt.subplots(1, 1, figsize=(14, 8))
plot_prices_by_type(df_regions, 'Detached', axes, 'Price', 'Detached Houses in South England counties')

You can see that within the last 10 years most counties saw a decent increase in nominal terms and a decline in real house prices for detached properties. The only exception is Kent with a small real increase of 0.9%. London and Surrey both lost around 10% in real terms.

regions = sorted(df['Region'].unique())
gt_regions_flat = create_table(
        df_regions, 'Detached',
        'South England Counties',
        regions
    )
gt_regions_flat

	2025 (Current)	2024 (1y ago)	2023 (2y ago)	2020 (5y ago)	2015 (10y ago)
South England Counties
Detached properties - Real prices (CPI-adjusted) - Q4 2025 vs 1, 2, 5, 10 years ago
Price (£)	%	Price (£)	%	Price (£)	%	Price (£)	%	Price (£)	%
Buckinghamshire	£853,221	0.0	£883,178	−3.4	£894,115	−4.6	£974,110	−12.4	£924,706	−7.7
Kent	£605,363	0.0	£620,638	−2.5	£633,552	−4.4	£692,871	−12.6	£599,898	0.9
London	£1,132,911	0.0	£1,178,785	−3.9	£1,179,862	−4.0	£1,331,199	−14.9	£1,265,328	−10.5
Oxfordshire	£680,124	0.0	£675,726	0.7	£676,613	0.5	£757,910	−10.3	£723,218	−6.0
Surrey	£975,634	0.0	£1,007,309	−3.1	£1,023,827	−4.7	£1,118,116	−12.7	£1,083,197	−9.9

Flats in London Boroughs

Next, I am looking at flats in London. Again I am plotting nominal and real quarterly prices for selected London boroughs and create a table with the corresponding numbers.

fig, axes = plt.subplots(1, 1, figsize=(14, 8))
plot_prices_by_type(df_boroughs, 'Flat', axes, 'Price', 'Flats in selected London boroughs')

Both the chart and the table show somewhat stable or slightly increasing nominal prices, but drastic reductions in real prices. Both Hammersmith/Fulham and Wandsworth are down about 30% in real terms since 2015. Moreover compared to last year and two years ago we still see price declines in real terms.

regions = sorted(df['Region'].unique())
gt_regions_flat = create_table(
        df_boroughs, 'Flat',
        'Selected London Boroughs',
        regions
    )
gt_regions_flat

	2025 (Current)	2024 (1y ago)	2023 (2y ago)	2020 (5y ago)	2015 (10y ago)
Selected London Boroughs
Flat properties - Real prices (CPI-adjusted) - Q4 2025 vs 1, 2, 5, 10 years ago
Price (£)	%	Price (£)	%	Price (£)	%	Price (£)	%	Price (£)	%
Ealing	£404,408	0.0	£419,269	−3.5	£419,733	−3.7	£489,258	−17.3	£521,670	−22.5
Hackney	£546,932	0.0	£554,931	−1.4	£583,301	−6.2	£685,095	−20.2	£667,622	−18.1
Hammersmith And Fulham	£577,801	0.0	£639,856	−9.7	£707,815	−18.4	£760,674	−24.0	£883,098	−34.6
Hounslow	£367,011	0.0	£375,391	−2.2	£373,890	−1.8	£441,401	−16.9	£419,450	−12.5
Islington	£566,442	0.0	£599,386	−5.5	£623,593	−9.2	£725,791	−22.0	£771,081	−26.5
Richmond Upon Thames	£483,018	0.0	£524,730	−7.9	£520,433	−7.2	£601,897	−19.8	£633,108	−23.7
Wandsworth	£540,034	0.0	£568,240	−5.0	£603,492	−10.5	£715,552	−24.5	£761,426	−29.1

Terraced houses in London Boroughs

Next, I am looking at terraced houses in the same London boroughs. My expectation is that they did better than flats, especially since they were more popular during the COVID pandemic, where people wanted a garden.

fig, axes = plt.subplots(1, 1, figsize=(14, 8))
plot_prices_by_type(df_boroughs, 'Terraced', axes, 'Price', 'Terraced houses in selected London boroughs')

Indeed, terraced houses did better in nominal terms with all of them up. Again in real terms we see price decreases over all considered comparison years (1,2,5,10). However, it is less drastic than for flats in the same locations.

regions = sorted(df['Region'].unique())
gt_regions_flat = create_table(
        df_boroughs, 'Terraced',
        'Selected London Boroughs',
        regions
    )
gt_regions_flat

	2025 (Current)	2024 (1y ago)	2023 (2y ago)	2020 (5y ago)	2015 (10y ago)
Selected London Boroughs
Terraced properties - Real prices (CPI-adjusted) - Q4 2025 vs 1, 2, 5, 10 years ago
Price (£)	%	Price (£)	%	Price (£)	%	Price (£)	%	Price (£)	%
Ealing	£690,513	0.0	£697,849	−1.1	£690,002	0.1	£788,622	−12.4	£800,316	−13.7
Hackney	£972,274	0.0	£966,520	0.6	£1,006,975	−3.4	£1,158,935	−16.1	£1,073,030	−9.4
Hammersmith And Fulham	£1,111,278	0.0	£1,221,472	−9.0	£1,334,009	−16.7	£1,392,109	−20.2	£1,547,731	−28.2
Hounslow	£625,089	0.0	£622,316	0.4	£612,967	2.0	£704,956	−11.3	£637,839	−2.0
Islington	£1,124,856	0.0	£1,175,162	−4.3	£1,214,208	−7.4	£1,380,766	−18.5	£1,380,624	−18.5
Richmond Upon Thames	£890,118	0.0	£935,936	−4.9	£918,218	−3.1	£1,036,584	−14.1	£1,033,227	−13.9
Wandsworth	£965,788	0.0	£1,000,163	−3.4	£1,050,418	−8.1	£1,214,261	−20.5	£1,219,378	−20.8

Winners and Losers in real terms (last 3 years)

Lastly I want to see how the housing market in the different London boroughs compares to each other. I am looking at the prices 3 years ago across all property types. (At first I wanted to use 5 years, but I wanted to stay clear of the COVID years which might introduce short-term anomalies into the result). I am comparing real prices even though for a relative comparison between boroughs nominal prices would be fine as well (they all experienced the same inflation).

selected = df[(df['is_london_borough'] == True) & (df['Quarter'].isin(['2025Q4','2022Q4'])) & (df['PropertyType'] == 'All')]

df_wide = selected.pivot(index=['Region', 'PropertyType'], columns='Quarter', values='RealPrice').reset_index()
df_wide['%_real_price_change'] = (df_wide['2025Q4'] - df_wide['2022Q4']) / df_wide['2022Q4']
df_wide.sort_values('%_real_price_change', ascending=False,ignore_index=True).head(10)

Quarter	Region	PropertyType	2022Q4	2025Q4	%_real_price_change
0	Lewisham	All	529841.12	503542.00	-0.05
1	Havering	All	479610.47	450760.00	-0.06
2	Southwark	All	647219.21	607299.00	-0.06
3	Waltham Forest	All	569427.61	533922.00	-0.06
4	Haringey	All	697080.71	649650.00	-0.07
5	Hackney	All	681376.13	627538.00	-0.08
6	Hounslow	All	588525.22	540799.00	-0.08
7	Sutton	All	493375.92	452726.00	-0.08
8	Hillingdon	All	526316.99	482852.00	-0.08
9	Barking And Dagenham	All	387895.94	354709.00	-0.09

You can see that even the best-performing borough Lewisham saw decreasing real house prices of 5% in the last 3 years. Overall a lot of these top 10 boroughs are further outside of London and generally more affordable areas. Both Hackney and Southwark are more central, but have recently undergone gentrification which might have counteracted the price pressure in other central boroughs.

df_wide.sort_values('%_real_price_change', ascending=False,ignore_index=True).tail(10)

Quarter	Region	PropertyType	2022Q4	2025Q4	%_real_price_change
23	Barnet	All	706765.75	606006.00	-0.14
24	Camden	All	928639.42	792454.00	-0.15
25	Croydon	All	464581.92	393479.00	-0.15
26	Lambeth	All	654875.76	550720.00	-0.16
27	Hammersmith And Fulham	All	883837.60	741308.00	-0.16
28	Wandsworth	All	834257.35	695867.00	-0.17
29	Tower Hamlets	All	595217.88	470209.00	-0.21
30	Kensington And Chelsea	All	1637183.90	1194726.00	-0.27
31	City Of Westminster	All	1273329.16	889935.00	-0.30
32	City Of London	All	993657.65	607399.00	-0.39

Some of the worst-performing boroughs over the last 3 years are the City of London, Westminster, Kensington & Chelsea, and Hammersmith and Fulham, which are all very expensive ‘prestige’ areas.

Conclusion

This analysis shows that a prospective home buyer or seller only gains limited information from a national-level house price statistic. Instead, I showed that there are, unsurprisingly, vast differences in price developments between property types and locations.

Moreover, in real terms almost none of the considered regions saw increasing house prices over the last 10 years (notable exception: Kent).

Appendix

Since someone asked, here are the real price changes compared to 3 years ago for all London boroughs:

df_wide.sort_values('%_real_price_change', ascending=False,ignore_index=True)

Quarter	Region	PropertyType	2022Q4	2025Q4	%_real_price_change
0	Lewisham	All	529841.12	503542.00	-0.05
1	Havering	All	479610.47	450760.00	-0.06
2	Southwark	All	647219.21	607299.00	-0.06
3	Waltham Forest	All	569427.61	533922.00	-0.06
4	Haringey	All	697080.71	649650.00	-0.07
5	Hackney	All	681376.13	627538.00	-0.08
6	Hounslow	All	588525.22	540799.00	-0.08
7	Sutton	All	493375.92	452726.00	-0.08
8	Hillingdon	All	526316.99	482852.00	-0.08
9	Barking And Dagenham	All	387895.94	354709.00	-0.09
10	Merton	All	677772.16	618798.00	-0.09
11	Harrow	All	590633.55	538433.00	-0.09
12	Bromley	All	581127.96	529749.00	-0.09
13	Redbridge	All	524996.77	476764.00	-0.09
14	Richmond Upon Thames	All	870795.06	785086.00	-0.10
15	Enfield	All	533218.77	480183.00	-0.10
16	Bexley	All	456886.19	410002.00	-0.10
17	Islington	All	757742.76	677305.00	-0.11
18	Ealing	All	645069.13	572575.00	-0.11
19	Greenwich	All	533193.50	472599.00	-0.11
20	Brent	All	639480.63	558093.00	-0.13
21	Kingston Upon Thames	All	657636.32	573489.00	-0.13
22	Newham	All	471479.67	405808.00	-0.14
23	Barnet	All	706765.75	606006.00	-0.14
24	Camden	All	928639.42	792454.00	-0.15
25	Croydon	All	464581.92	393479.00	-0.15
26	Lambeth	All	654875.76	550720.00	-0.16
27	Hammersmith And Fulham	All	883837.60	741308.00	-0.16
28	Wandsworth	All	834257.35	695867.00	-0.17
29	Tower Hamlets	All	595217.88	470209.00	-0.21
30	Kensington And Chelsea	All	1637183.90	1194726.00	-0.27
31	City Of Westminster	All	1273329.16	889935.00	-0.30
32	City Of London	All	993657.65	607399.00	-0.39

Jupyter Notebook

You can find the Jupyter notebook for this post here .

TIL how to set up multiple SSH keys (for multiple GitHub accounts)

Fri, 28 Nov 2025 16:20:00 +0000

Setting up SSH keys is one of these things I need to do every couple of months. Inevitably, I don’t remember how to do it and then follow some tutorial, e.g. the GitHub Docs . It kind of always works but I haven’t fully internalised how to properly manage those different SSH keys.

Disclaimer: This learning was provoked by setting up multiple SSH keys for different GitHub accounts (I know how to do that) and then being able to git clone {private-repository} a private repository with the correct ssh key (I didn’t know how to do that). An example use case could be a personal and professional GitHub account that don’t have access to the same private repositories. So ideally when we git clone we want to automatically pick the right ssh key.

1. Generate the SSH keys for each account

Let’s assume we want to generate new separate ssh keys for each account. I like to have them all in my ~/.ssh folder. We can use the key generation command

$ cd ~/.ssh
$ ssh-keygen -t ed25519 -C "your_email or your_name"

where we specify the encryption type (-t) as ed25519 which is Edwards-curve Digital Signature Algorithm and generally recommended over RSA-based keys . The comment (-C) argument allows us to optionally add a comment that is added at the back of the key so that you remember who created it. People often use their email address. However, this is not used anywhere to select the key.

$ Generating public/private ed25519 key pair.
$ Enter file in which to save the key (/Users/youruser/.ssh/id_ed25519): /Users/youruser/.ssh/github_personal

$ Enter passphrase for "github_personal" (empty for no passphrase):

This generates both the private and public key github_personal (private) and github_personal.pub (public) in the specified path. If we entered a passphrase the private key will be encrypted as well using 3DES . We can check out what the keys look like:

$ cat github_personal.pub
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDycaSFXnekEe32AZFhhDp3Q14JRXF7bu8CNfGf7Z6hN your_email

and

$ cat github_personal

-----BEGIN OPENSSH PRIVATE KEY-----
somegibberylongkeyperhapsfurtherencryptedusingpassphrase
-----END OPENSSH PRIVATE KEY-----

Let’s also use the same method to create a keypair for work (github_work, github_work.pub). Once we have these keys, we can add them to our two GitHub accounts.

2. Associate the public keys with the GitHub accounts

In both GitHub accounts, navigate to Access:SSH and GPG Keys -> New SSH key and add the public key as an Authentication Key to the account. GitHub now has the ability to verify messages that were signed by the corresponding private key of the pair.

3. Set up OpenSSH client config file

The file ~/.ssh/config allows us to configure “Host” specifications that we can refer to when using SSH (e.g. when git clone uses the SSH protocol for data transfer). The file looks like this:

Host hostname1
    SSH_OPTION value
    SSH_OPTION value

Host hostname2
    SSH_OPTION value

In our case, we define the two different github identities that we want to be able to select:

Host github-personal
    User git
    PreferredAuthentications publickey
    Hostname github.com
    IdentityFile ~/.ssh/github_personal

Host github-work
    User git
    PreferredAuthentications publickey
    Hostname github.com
    IdentityFile ~/.ssh/github_work

This defines which keys to use when we connect using SSH via user@host. So when we test the config by running ssh -T git@github-personal SSH will use the SSH key ~/.ssh/github_personal as specified in the file. (Note: We can name the host and user whatever we want.)

4. Use the right hostname when cloning repositories

Let’s say we want to git clone a repository that only our work account has access to, then I can use the git clone command git clone {user}@{host}:{repo_path}.git:

$ git clone git@github-work:/org1/project1.git

or more explicitly:

$ git clone ssh://git@github-work/org1/project.git

git/ssh will look into the config file and select the configured private key for the provided host.

GitHub will send a challenge (message) to our machine for authentication. The private key will be used to sign this challenge and our machine then sends the signed message back to the GitHub server. The GitHub server will use the stored public key (from step 2) to verify the signature and allow a connection.

Note: If in step 1) we decided to encrypt the private key with a passphrase we will be prompted to provide it. Normally, at that point MacOS will offer us to add this passphrase to our Keychain .

TIL how to re-run only failed tests

Fri, 21 Nov 2025 13:00:00 +0000

This is a super small thing I stumbled upon. When I implement a new feature, there is inevitably a step where I go through the failing pytest tests and fix or clarify them.

Unless I use an IDE, my work loop is like this:

Make code change
Run python -m pytest
Check which tests are failing.
Go back to 1.

Now, 2. runs all tests which can take quite some time. There is always the option to run a specific test only, but multiple failing tests can be scattered across different files.

So instead, what you can do is to run only the tests that failed at your last test run. Simply use:

python -m pytest --last-failed

Once you have fixed those tests, it’s worth running all tests one more time to check that nothing else was broken by the new changes.

Another option is to use the test view of your IDE, if you have access to one in that environment.

TIL how to investigate missing data easily with DuckDB

Fri, 16 May 2025 14:00:00 +0000

In data engineering pipelines we commonly store output data in S3 buckets at various stages. In this particular case we stored the input data received from customers as csv files in the S3 bucket s3://customer-ingested/customerA.csv and after some transformations we stored the transformed data as csv files in another S3 bucket s3://customer-transformed/customerA.csv.

Problem

For some of the customers the csv file was missing in the first bucket customer-ingested while we were sure that all customers had a file in customer-transformed. However, there was a legitimate reason why the file could be missing which was that in the application the customer had a state of is_deactivated = True.

The goal of this investigation was to find all the active customers for which we were missing data in the bucket customer-ingested.

Solution

The nice thing about DuckDB in this scenario is that I can combine filesystem data with information from an application table and work in (familiar) SQL land.

Here is the query:

with transformed as (
	select regexp_extract(file, '/([^/]+)\.csv$', 1) AS filenames_transformed from glob('s3://customer-transformed/*.csv')
	),
ingested as (
	select regexp_extract(file, '/([^/]+)\.csv$', 1) AS filenames_ingested from glob('s3://customer-ingested/*.csv')
	),
customer_status as (
	select customer_name, is_deactivated from read_parquet('s3://application_table_snapshot/customers.parquet')
)

select * from transformed
left join ingested on transformed.filenames_transformed = ingested.filenames_ingested 
left join customer_status as cs on transformed.filenames_transformed = cs.customer_name
where is_deactivated = False and filenames_ingested is null;

┌───────────────────────┬────────────────────┬────────────────────┬────────────────┐
│ filenames_transformed │ filenames_ingested │   customer_name    │ is_deactivated │
│        varchar        │      varchar       │      varchar       │    varchar     │
├───────────────────────┼────────────────────┼────────────────────┼────────────────┤
│ customerT             │ NULL               │ customerT          │ False          │
│ customerW             │ NULL               │ customerW          │ False          │
│ customerZ             │ NULL               │ customerZ          │ False          │
└───────────────────────┴────────────────────┴────────────────────┴────────────────┘

In the CTE, I am creating a table transformed with all the filenames in the S3 bucket customer-transformed. I am using DuckDB’s utility function glob(search_path) to retrieve the filenames. I do the same for the bucket customer-ingested.

To get the customer status I read from a recent replication snapshot of the application database to get customer_status.

Finally, I join the three expressions to identify missing files (missing values) in the column filenames_ingested for active customers.

TIL that DuckDB now supports Avro files

Wed, 12 Mar 2025 16:00:00 +0000

In the data engineering pipelines of my current company we use two main file formats:

Parquet files for analytical workloads (columnar-storage)
Avro files for transactional / event-based messages (row-storage)

In Querying remote S3 files I wrote about how I use DuckDB to query parquet files stored in S3. Recently, I noticed that DuckDB started supporting reading of Avro files in the same way.

Install extension

To install the extension open DuckDB and run:

INSTALL avro FROM community;
LOAD avro;

Read local files

Local files can be queried with the read_avro function:

select * FROM read_avro('my_dataset.avro');

Read files stored in S3

Remote files can be queried by providing the S3 URI (after logging into the AWS account ):

select * FROM read_avro('s3://my_bucket/my_remote_dataset.avro');

A few limitations of the extension are listed here .

Building a local data warehouse with DuckDB, dbt, and Superset

Sun, 09 Mar 2025 15:00:00 +0000

In previous blog posts, I described two DuckDB use cases for data scientists and data engineers: Querying remote parquet files and processing larger-than-memory datasets .

Now I want to explore if I can set up a local and open-source version of the analytics data stack that we use at my company. At work, we are using Snowflake, dbt cloud, and Google’s Looker which cost us several thousand EUR per month. I am going to use the following open-source tools in my local setup:

	cloud	local
analytical database	snowflake	DuckDB
data modelling tool	dbt cloud	local dbt
dashboard tool	Looker	Apache Superset

Figure 1 shows how the components work together.

Figure 1. Components of my local data warehouse stack.

I will use a toy visualisation problem to demonstrate the setup. My goal is to load the race time dataset that I used in Data analysis: Vätternrundan 2024 results into DuckDB. I then use dbt to create data models and Superset to create an interactive dashboard to visualise the data.

Loading the raw data into DuckDB

I want to use DuckDB for all data processing. Therefore, I will load the raw race time data into DuckDB.

I start by installing DuckDB as my analytical database. Since I use homebrew as a package manager, I can run:

brew install duckdb

I then create a persistent database (a file on my machine):

duckdb database.duckdb

and create a new DuckDB table from the parquet file that contains the raw data.

CREATE TABLE vatternrundan AS select startnumber, city, country, result_time, start_time from read_parquet('results_vatternrundan24.parquet');

To verify that everything works, I run:

describe table vatternrundan;

┌─────────────┬─────────────┬─────────┬─────────┬─────────┬─────────┐
│ column_name │ column_type │  null   │   key   │ default │  extra  │
│   varchar   │   varchar   │ varchar │ varchar │ varchar │ varchar │
├─────────────┼─────────────┼─────────┼─────────┼─────────┼─────────┤
│ startnumber │ BIGINT      │ YES     │ NULL    │ NULL    │ NULL    │
│ city        │ VARCHAR     │ YES     │ NULL    │ NULL    │ NULL    │
│ country     │ VARCHAR     │ YES     │ NULL    │ NULL    │ NULL    │
│ result_time │ BIGINT      │ YES     │ NULL    │ NULL    │ NULL    │
│ start_time  │ VARCHAR     │ YES     │ NULL    │ NULL    │ NULL    │
└─────────────┴─────────────┴─────────┴─────────┴─────────┴─────────┘

Setting up dbt for data transformations

I want to use dbt to create data models on top of the raw data. I start by creating a new poetry project to install dependencies:

poetry init
poetry add duckdb-dbt

Then I initialise the dbt project local_warehouse with

poetry run dbt init local_warehouse

Next, I configure dbt to use DuckDB as the data processing backend. In my work setup, this would point at a Snowflake instance instead. I point at the local database file in the dbt profiles file profiles.yml:

local_warehouse:
  outputs:
    dev:
      type: duckdb
      path: ../duckdb/database.duckdb
  target: dev

dbt models

I define the DuckDB table vatternrundan as a dbt source in sources.yml:

version: 2
sources:
  - name: vatternrundan_db
    schema: main 
    tables:
    - name: vatternrundan

This means I can now refer to this raw data in dbt. Moreover, I want to create two more models. One thin staging layer over the raw data stg_vatternrundan.sql, and one refined model that aggregates the rider data by country average_speed_by_country.sql.

The staging model is stored in the dbt project as models/staging/stg_vatternrundan.sql:

select 
    startnumber::int64 as startnumber,
    city::varchar as city,
    country::varchar as country,
    to_microseconds((result_time/1000)::int64) as result_time,
from {{ source('vatternrundan_db', 'vatternrundan') }}

As you can see, I refer to the DuckDB source table, define the column types, and do some light conversions.

The aggregation model is stored in models/refined/average_speed_by_country.sql:

with speeds as (
    select startnumber,
    country,
    60 * datepart('hours', result_time) + datepart('minutes', result_time) as result_time_minutes,
    315 * 60 / result_time_minutes as average_speed,
    from {{ref('stg_vatternrundan')}}
)

select country, count(*) as num_riders, mean(average_speed) as average_speed_of_country 
from speeds
group by country
order by average_speed_of_country desc

I use this dbt model to compute the average rider speed for every country in the dataset.

Now that I have defined my dbt models, I configure dbt to run them by adding the following to dbt_project.yml:

models:
  local_warehouse:
    staging:
      +schema: staging
      +materialized: table
    refined:
      +schema: refined
      +materialized: table

This tells dbt how to materialise the dbt models in DuckDB. So let’s run dbt to build the models:

poetry run dbt run --profiles-dir=.

Check tables were created

I can verify in DuckDB that these tables have been created

Run

duckdb database.duckdb

followed by:

show all tables;

which should now show:

database.main_staging.stg_vatternrundan
database.main_refined.average_speed_by_country

Superset for data analytics dashboards

After modelling my data, I want to create a dashboard and visualise the data. I use Apache Superset as my dashboarding tool. Superset works with many analytical databases. However, setting it up to work with DuckDB is a bit clunky and I had to troubleshoot quite a bit.

Install Superset with DuckDB support

I follow the Quickstart Guide with some modifications. First, I download the git repository and checkout the last tagged commit:

git clone https://github.com/apache/superset
cd superset
git checkout tags/4.1.1

I need to make two modifications before building the container.

Install duckdb-engine inside the container. This allows us later to select DuckDB as a database in Superset. To do this, add

RUN pip install duckdb-engine

to the Dockerfile.

I need to make the DuckDB database file available in the Docker container. I add its path as a volume. Modify docker-compose-image-tag.yml:

superset:
[...]
    volumes:
        - ./docker:/app/docker
        - superset_home:/app/superset_home
        - <local-machine-path-to-duckdb-database>:/app/duckdb

and force the Superset container to build from the Dockerfile and not from the image:

superset:
[...]
    build:
        context: .
        dockerfile: Dockerfile
#   image: apache/superset: comment out or remove

Then build the container with

docker compose build

and start Superset with

docker compose -f docker-compose-image-tag.yml up

Then, I can access Superset with my browser under http://localhost:8088 and log in with username: admin and password: admin.

Configure Superset

Now, I can add DuckDB as a database in the Superset UI. I navigate to Settings -> Data -> Database Connections + Database -> DuckDB

and add the link to the DuckDB database file as the SQLAlchemy URI:

duckdb:////app/duckdb/database.duckdb?access_mode=READ_ONLY

Clicking on Test Connection should respond with “Connection looks good!”.

Add datasets in Superset

In the navbar under Datasets I can add the DuckDB tables stg_vatternrundan and average_speed_by_country as new Superset datasets.

Create dashboard in Superset

Creating the dashboard is self-explanatory. In the navbar, I select Dashboards and add a new Dashboard called “Vätternrundan Dashboard”. Next, I create two charts:

number of riders per country
average speed of riders per country

Just select the relevant Superset Dataset, build the charts, and add them to the dashboard. Figure 2 shows how I did it.

Figure 2. Visualization of Vätternrundan rider speed by country in Apache Superset.

Conclusion

The combination of DuckDB, dbt, and Apache Superset is a local version of “the modern datastack”. This can be useful for personal projects, and to experiment with data transformations and visualisations.

I found that the integration of DuckDB and dbt worked seamlessly. On the other hand, connecting Superset to DuckDB was clunky. Superset would benefit from better DuckDB support out-of-the-box.

Troubleshooting

DuckDB only allows one connection with read-write-access, but multiple concurrent read-only connections. Make sure you are not connected to your DuckDB database with more than one client as access mode allows write by default, e.g. Superset and the DuckDB CLI client in your terminal.
Ensure to connect Superset to DuckDB in read-only mode, i.e. add access_mode=READ_ONLY to the SQLAlchemy URI to connect.

Book notes: Poor Charlie’s Almanack

Fri, 21 Feb 2025 00:00:00 +0000

The book is a collection of speeches that Charlie Munger, partner of Warren Buffett at Berkshire Hathaway, gave over the years (1986 - 2007) at universities and institutions.

I found four interesting themes across the speeches:

Advocating interdisciplinary collaboration between university departments

He believes that the social sciences, especially economics and psychology, are focusing on too narrow theoretical problems. Academics suffer from a man-with-a-hammer-syndrome. He suggests that they collaborate more across departments, e.g. economics borrowing from psychology, and within departments, e.g. macroeconomists shouldn’t avoid microeconomic explanations. Moreover, the social sciences should try to incorporate findings / models from the hard sciences (physics, chemistry, etc.) and attribute them properly.

Fraud and advice for endowment funds

In some of his talks, he uses fictional examples of companies to show the problems with accounting and financial engineering fraud.

He thinks that charitable foundations and endowment funds should as much as possible avoid wasteful investment practices. He criticises the trend of university endowment funds to employ layers of analysts and consultants to use funds of funds. These multilayered systems erode a large chunk of the endowment funds’ returns when compared to simpler alternatives, like unlevered domestic equity indices. He directly criticises the practices of his audience of consultants and fund managers in talk six, but he uses humour and self-deprecation to get away with it.

(Inverted) Advice for Graduates

In his commencement speeches for university graduates, he gives advice on how to guarantee misery in life.

Be unreliable.
Don’t learn from others’ mistakes. Instead, make common mistakes of others again (join a cult, drive while drunk, gamble) and don’t learn from people that came before you.
Give up after adversity and failures. Just give up when the inevitable hard times occur.
Don’t ever invert. Don’t attempt to learn from thinking about achieving the opposite of your goals. Don’t try to be objective.

Mental checklist to analyse problems

A recurring topic in his speeches is a checklist of mental models and human biases. He developed this checklist over time and iterated through it when analysing a problem or evaluating an investment. Using appropriate checklists and the inversion technique are two main tools that he promotes for better thinking.

Here are 25 human biases from talk eleven: The Psychology of Human Misjudgment .

Reward and Punishment Superresponse Tendency: If you want to persuade people, appeal to their own interests. Ensure the incentives of people you work with are aligned with the outcome you want. Don’t reward them for metrics that they can easily game.
Liking/Loving Tendency: People are seeking love and approval from other people. Moreover, we favour people and products that are merely associated with the target of our affection. We can use this to our advantage by liking truly admirable people or ideas.
Disliking/Hating Tendency: People can have a tendency to dislike things different from them or products or people that are associated with the object of their dislike.
Doubt-Avoidance Tendency: Our brains are conditioned to quickly remove doubt after reaching a first decision. This tendency should be countered by forcing a delay for reflection before an important decision, e.g. jury decisions in court.
Inconsistency-Avoidance Tendency: A tendency to stick to previous conclusions, habits, and ideas to avoid change. Practically, this means that it is much easier to prevent a bad habit than to change it. To counter this tendency, we should force the discussion of counterarguments before a decision can be made. This tendency can be used to manipulate people (see also Ben Franklin effect , Cialdini’s consistency principle ).
Curiosity Tendency: Humans have a general tendency to be curious, which can be supercharged with today’s access to information. This fortunate tendency should be used to counteract other psychological tendencies.
Kantian Fairness Tendency: People have a tendency in direct interactions to behave fairly (following Kant’s categorical imperative).
Envy/Jealousy Tendency: People’s tendency to envy someone else’s status, wealth, or compensation. “It is not greed that drives the world but envy.” (Buffett)
Reciprocation Tendency: People have a tendency to reciprocate favours and disfavours. This can also be used for manipulation, e.g. a salesman could do you a small favour to get a much better outcome in negotiation.
Influence-from-Mere-Association Tendency: Valuing something by the association with another unrelated factor/idea/concept. Examples:
- association of quality with the highest price
- purchasing of luxury items to boost status
- advertising of products with unrelated but positive images
- associating one’s ability with past successes and making bad decisions
- thinking of someone worse because they are a competitor
Simple Pain-Avoiding Psychological Denial: Denying reality because it is too painful to accept, e.g. addiction, bankruptcy.
- Denial of a reality that’s too painful to accept, e.g. addiction to alcohol.
Excessive Self-Regard Tendency: People’s tendency to overestimate their abilities, decisions, and possessions. Moreover, their preference of people that are similar to them.
Overoptimism Tendency: A tendency to be overly optimistic of the future, especially if one has done well in the past.
Deprival-Superreaction Tendency: Reacting more strongly to losses than to gains, e.g. losing $10 is considered worse than gaining $10. Irrational overreaction to threatened loss of status, territory, love, friendship, or property.
Social-Proof Tendency: A tendency to act and think the same way as people around you (Group-think).
Contrast-Misreaction Tendency: Making bad decisions by anchoring on an irrelevant comparison. Examples:
- Adding $1000 of useless add-ons to a car only because the car costs $65k.
- A real estate agent presenting 3 terrible and expensive houses, then showing a merely bad house to make it look more desirable.
Stress-Influence Tendency: Light stress can increase performance temporarily, while heavy stress can cause dysfunctional thinking and bad decision-making.
Availability-Misweighting Tendency: Tendency to overweight or overvalue things (people, decisions, work, ideas) that are close or readily accessible to you. This also holds for metrics that are easy to measure and actions that are easy to take. This can be countered by following checklists of actions and by considering more difficult (or less accessible) alternatives.
Use-It-or-Lose-It Tendency: Over time, our skills and knowledge fade. Therefore we should deliberately train and repeat the skills we want to retain. Write them down as a checklist and work through them regularly.
Drug-Misinfluence Tendency: Most people can’t handle drugs responsibly over a long period of time. Not worth trying to prove that you can.
Senescence-Misinfluence Tendency: Older people have a harder time learning new skills. Knowing this, the best counter is to actively maintain the accumulated knowledge, see 19).
Authority-Misinfluence Tendency: Tendency to blindly follow the leader (see also: HiPPO effect ).
Twaddle Tendency: Some people waste time talking about things they are not an expert in. Try to separate these people from the experts and follow the latter.
Reason-Respecting Tendency: People can learn better when they can think through the reasons behind a directive or action. Therefore, when giving orders explain your reasoning.
Lollapalooza Tendency: Often multiple human biases act together to drive a certain behaviour or outcome.

Conclusion

I gave only 3/5 stars because the format of reading his talks didn’t appeal to me. I was already aware of most of the human biases listed by Munger. Some are discussed in Thinking Fast and Slow (2011) and Influence: Science and Practice (2001) . However, what I found impressive is that he talked about and used these biases already 20 years ago, before they were widely discussed. I also liked that in many speeches he directly criticises the audience that invited him to speak. However, he does it in a charming and humorous way.

TIL how to fix a broken Joplin sync

Mon, 17 Feb 2025 00:00:00 +0000

I use Joplin for all my note-taking as it’s open-source, markdown-based, and has decent Desktop and Android apps. I use note encryption and sync my notes via WebDAV with my remote Nextcloud instance.

Problem

After not syncing my notes for several weeks, I encountered a couple of errors both on my Desktop app and on my Android app. I don’t have access to the error message anymore, but the errors referred to files and file IDs that caused inconsistencies on the remote server, e.g. broken references:

f1231gdffgasd1asd2.md TypeError: Cannot read property 'type_ of null.

Solution

To solve the issue, I had to delete the offending file on the remote server. However, to not lose it, it’s best to make a local copy of the file’s content if it is present on one of your local devices.

First, I had to determine which file caused the issue, which is a bit harder here as filenames are encrypted. The easiest way to identify the note from the filename is to paste a local markdown reference into a new Joplin note like this:

[Note I am looking for](:/f1231gdffgasd1asd2)

and then click the link in the rendered note. This took me to the offending note.

Second, copy the content into a new note.

Third, I had to delete the file on the remote. I connected on my Mac to the remote WebDAV server via Finder. In Finder, I clicked on Go -> Connect to Server -> Add address: https://yourserveraddress.org/remote.php/webdav/joplin (look up the link in your Joplin settings), and authenticated with my Nextcloud credentials. Then I was able to search for the file f1231gdffgasd1asd2 and delete it.

The sync now succeeds.

TIL how to debug a broken Nextcloud login (again)

Sun, 16 Feb 2025 00:00:00 +0000

The symptoms of this problem were the same as the ones I described in TIL how to debug a broken Nextcloud login .

Problem

Phone and Mac apps stopped syncing data with the Nextcloud instance. In my browser I could access the login page, but after entering the details I was being sent back to the same login page without any error message.

Investigation

Last time the issue was caused by excessive logging. However, I reduced the loglevel in the Nextcloud config, so this was not the issue here. I am no expert in web admin, PHP, or Nextcloud, so I used Claude 3.5 to pair on debugging:

Prompt

I am self-hosting a nextcloud instance on a Hetzner VPC. I currently have the following problem. I can’t log in to nextcloud via my account. The screen just reloads and shows me the login page again. It is not a login detail issue. They are correct. Moreover, other nextcloud related syncs (files, calendar) stopped working. Walk me through how I can debug this issue and get nextcloud up and running again.

I can’t say that the suggestions worked right away, but they put me on the right path and helped me with some useful debugging commands.

The command

df -h

showed me that I didn’t have any free space. While the command

du -sh /var/www/nextcloud/data/*

indicated that in Nextcloud a lot of space was occupied by old file versions and deleted files from the Nextcloud file sync.

Solution

You can use the Nextcloud console occ (ownCloud Console) tool to run maintenance tasks on the Nextcloud instance.

The command

sudo -u www-data php /var/www/nextcloud/occ list

lists all available commands.

A couple of cleanup commands to remove old file versions and files in the trashbin worked for me:

# Check Nextcloud trashbin size
sudo -u www-data php /var/www/nextcloud/occ trashbin:cleanup --all-users

# Clean up file versions
sudo -u www-data php /var/www/nextcloud/occ versions:cleanup

# Clear Nextcloud cache:
sudo -u www-data php /var/www/nextcloud/occ cache:clear

This freed up enough space for Nextcloud to work again.

Unfortunately, the occ commands didn’t work right away because there was apparently not enough space to run any of them. Each one returned the same error message:

fwrite(): write of 4963 bytes failed with errno=28 No space left on device

So I first had to free up a few hundred megabytes:

Clear local repository of retrieved package files:
```
sudo apt-get clean
```

Remove old log files:

sudo find /var/log -type f -name "*.gz" -delete
sudo find /var/log -type f -name "*.1" -delete
sudo truncate -s 0 /var/log/*.log

Remove temporary files:
```
sudo rm -rf /tmp/*
```

This cleared enough space to run the occ commands.

About this blog and its microfeatures

Wed, 12 Feb 2025 00:00:00 +0000

No blog would be complete without a post explaining the code and frameworks used to generate it. In fact, often tweaking the blog code is more fun than actually writing something.

Static site generator

I try to keep the tech of this blog simple. I am using the Hugo static site generator and chose one of the simplest themes Bear Cub to get started. What I liked about the theme was its small code footprint, no JavaScript/ads/trackers, and it was entirely focused on text. The generated pages are small and load fast. The simple layout made it very easy for me to understand and modify it over time.

The source code is hosted on GitLab and the static sites are published to GitLab Pages. This means that to publish a new post, I just have to push a commit with the changes to the remote repository.

Supporting math, code, Jupyter notebooks

At the time of starting this blog I worked as a data scientist. I wanted to be able to share reproducible Jupyter notebooks that I could easily convert to HTML and embed in this blog. All notebooks are reproducible unless the data used in the blog post is too large to be uploaded to GitLab. I put my notebooks into the /notebooks folder of the repository and use the script nb2hugo to convert the notebooks to markdown files. At the bottom of the post you can usually find the link to the Jupyter notebook, the poetry environment, and the data. I wrote about this here .

The equations on this blog are rendered with KaTeX .

Content types

The content of this blog has four different types:

Posts are normal blog posts.
TILs, or Today-I-Learned, are notes documenting something small I have learned on a day. The idea is based on Simon Willison’s TILs .
Links appear in Feeds and are links that I want to share, usually with a short comment. They can be thought of as a retweet.
Book reviews.

1-3 are shown in reverse-chronological order in the main page’s feed.

Microfeatures

I like Daniel Fedorin’s post on blog microfeatures . In the same spirit, here are the microfeatures that I have added:

Recommendations

Most pages have post recommendations at the bottom. I compute the (cosine) similarity of posts after summarizing them with a local LLM, computing embeddings, and storing them in a local Chroma vector database. The two most similar posts and their cosine similarity are then displayed. Note that all of this is pre-generated and static. I wrote in more detail about how this works here .

For this post, they look like this:

Sidenotes

I can add sidenotes like this one. to a blog post by using the {{% sidenote %}} shortcode. I really like to use this for comments that are additional to the primary content or later updates to a blog post.

Markers for external links

Internal links on this blog have no marker and open in the same window. External links are appended with an icon and open in a new window. Inspired by gwern.net , I also added specific icons for particular sites: tweet on x.com, book link on goodreads, video on youtube, and article on wikipedia.

Link-blog / retweets / boosts

The main page of this blog has a section Recent which displays a feed of the content being posted. It’s a mix of posts, TILs and link-blog entries and should mimic a social media profile feed. Shared links have a reshare/boost symbol and usually a short comment about them. They also have a via link to the place where I found this link. This is also inspired by Simon Willison’s main page.

Figure 1. A shared link.

RSS

I provide RSS feeds for all content types and also separate feeds for Posts , TIL , Feed , and Books only. I wrote more about this here .

Longer blog posts have a table of contents section at the beginning of the post.

Figure 2. Some longer posts show a table of contents at the top.

Linkable Headers

The main headers of blog posts have a linkable §-symbol in front of them. This means you can easily link to a specific section of the blog post.

Drawings

I added the ability to provide an optional SVG image that is added to the background next to the post title, e.g. here and here .

Figure 3. Blog posts can have an SVG drawing next to the title.

Prompts / LLM responses

When I first wrote about using an LLM to build something I thought that it would be quite common to share a prompt and the LLM’s output in your blog post. So I built a small feature to display them in a chat-like fashion. You can use {{% prompt %}} to put text into a prompt chat bubble, and {{% llm %}} for the model response (colored by model). Turns out, this is not something I regularly need and nobody wants to read someone else’s prompt or the LLM’s answer anymore.

Prompt

How many Rs are in the word strawberry?

Model: claude-opus-5-5 - response

There are 3 Rs in “strawberry”: strawberry.

Model: openai-sol-5-6 - response

There are 3 Rs in “strawberry”: strawberry.

Model: qwen-4-3 - response

There are 3 Rs in “strawberry”: strawberry.

How related posts are computed

Sun, 09 Feb 2025 00:00:00 +0000

The number next to related posts at the bottom of each page is the advertised post’s “similarity” to the currently viewed page (from 1.0 to -1.0).

I am using the following process to compute related posts locally:

Summarise every post and TIL using local qwen (qwen3.5:9b via Ollama) with the following prompt:

Prompt

You are an analyst and editor with many years of experience in reading and synthesizing content.

Here is a blog post:

<BLOGPOST>

{ blog_post }

</BLOGPOST>

Please create a comprehensive and concise summary of the blog post. Focus on the main concepts, key details, and central arguments.

<INSTRUCTIONS>

Include any specific technologies, methods, or frameworks mentioned.
Don’t use more than 7 sentences.
Respond in plaintext. Don’t add formatting or linebreak characters to your response.
Don’t repeat the instructions of the task. Respond directly.

</INSTRUCTIONS>

Embed the summary using a qwen embedding model qwen3-embedding:0.6b and store the embedding and metadata about the post in a persistent ChromaDB vector database (a file on my computer).
Compute the cosine similarity between the embeddings for each pair of posts. A score of 1.0 indicates proportional vectors, a score of 0.0 orthogonal vectors, and a score of -1.0 opposite vectors.
Write a yaml file that includes for every post a link to the most similar post and their similarity.
Use a Hugo partial to include the data in the yaml file about the most relevant posts at the bottom of each page.

Noteworthy

When using Llama, I had to explicitly instruct the model not to restate the task. Otherwise, every summary would have started with “Here is your concise blog post summary in not more than 7 sentences.” which would create some artificial similarity in the embeddings.
Most blog posts are shorter than 15000 characters, or 4000 tokens. I had to work this out to configure the model context correctly.
ChromaDB’s default embedding model all-MiniLM-L6-v2 has 384 dimensions and a maximum input sequence length of 256 BERT tokens (~100–150 words). I checked some of the summaries and they were around 160 – 220 BERT tokens. I have since switched to a qwen embedding model which has much longer input context. A good embedding model leaderboard to help make a choice is available here .

Example

For the blog post UK house price changes in real terms the LLM came up with this summary which was then embedded:

Model: qwen3.5:9b - response

The author critiques standard UK housing market reports for presenting nominal price changes that obscure local realities and ignore inflation, arguing instead for real house prices adjusted using CPI data from ONS alongside Land Registry transaction records. To address this gap, the post demonstrates a Python-based workflow utilizing pandas to merge regional property datasets with quarterly consumer price index figures to calculate purchasing power-adjusted values in 2025 pounds. The analysis focuses on specific areas around London and South England, visualizing trends for detached houses in counties like Surrey and Kent as well as flats within selected boroughs since 2015. Results reveal that while nominal prices may show modest growth or stagnation, real house prices have actually declined significantly over the last decade in many regions due to inflation outpacing asset appreciation. The author provides interactive tables comparing current valuations against historical data points from one, two, five, and ten years prior to highlight these divergences across different property types. By leveraging open APIs for Land Registry and ONS data stored as parquet files, the post offers a reproducible method for investors and buyers to assess true market performance beyond headline statistics. Ultimately, this approach clarifies that British housing has often underperformed relative to other investments like the FTSE 100 when viewed through an inflation-adjusted lens.

Embeddings

I can visualise the embeddings in two dimensions using t-SNE :

Figure 1. Plot of blog post summary embeddings in t-SNE space.

The plot looks somewhat reasonable. The code-heavy Jupyter notebook posts are at the bottom, clustered around duckdb-large-datasets. The book reviews (4000-weeks, how-big-things-get-done, how-to-win-friends) are fairly close together. SQL related posts are clustered at the top. On the other hand, I would have expected reading-and-note-taking to be closer to writing-well.

Code

I am using this langchain script to compute the recommendations.

Update

A previous version of the code and blog post used Llama 3.1 to summarise the text and ChromaDB’s default embedding model all-MiniLM-L6-v2.

Book notes: How Big Things Get Done by Bent Flyvbjerg and Dan Gardner

Sun, 02 Feb 2025 00:00:00 +0000

Bent Flyvbjerg is an economics professor at IT University of Copenhagen. He maintains a database of megaprojects (power plants, opera halls, tunnels, airports) and their planned and realised timelines and budgets. He researches the reasons why modern megaprojects often fail to deliver on time and on budget. It has recently become quite popular to discuss the apparent decrease in speed with which large-scale projects are realised. Examples are easy to find, e.g. the delayed High Speed Rail 2 (HS2) project or the time and cost overruns of the Berlin Brandenburg Airport (BER) . Counter-examples are less common (or less newsworthy), e.g. the 2023 Notre-Dame Reconstruction.

In their book Flyvbjerg/Gardner explain the common reasons for cost and time overruns of megaprojects and how to mitigate them. I think we can apply the majority of their learnings for smaller home (kitchen renovation) and work projects (cloud migration) as well.

Planning is cheap, building is expensive

In large construction projects, planning is a lot cheaper than building. Once a badly planned project is underway, unforeseen problems will be discovered. Delays tend to cause further delays. The total project time can be in itself a source of additional risk and cost.

When politicians plan prestigious infrastructure projects, they have a bias for strategic misrepresentation of the cost and time to get support for it. Therefore, the government and the public need to scrutinise the project’s goal and its planning to ensure that the plan is realistic and enough alternatives have been considered. Only get involved in such a project if it has the people and funds, including contingencies, to succeed.

Since planning is a lot cheaper than building for large projects, Flyvbjerg/Gardner advise investing significant time into the planning phase:

understand the objective behind the proposed project (assume an outside view)
explore alternatives and don’t commit to the first available solution (resist quick action bias)
don’t forecast using the best-case scenario, instead use similar past projects to anchor your estimate
experiment and iterate in the planning phase using digital modelling and simulations

Understand the why

The project is not a goal in itself, it is how the goal is achieved. We want to talk to the stakeholder to understand the goal. Is the project even the right approach? Good planning needs to explore the problem before jumping to the solution. This includes considering alternatives.

There is a good example in the book about a bridge project. The original goal of the project is to connect an island with the mainland. By jumping to the bridge idea, the stakeholders ignored alternatives, e.g. a tunnel, ferries, a helipad. Or maybe a physical connection is not required. If it’s about improved communication, maybe a high-speed broadband connection achieves the goal.

Another framing of this idea in product development is to work backwards from the customer. Understand their needs and problems, before coming up with any solution.

Reduce uncertainty via experimentation

A crucial part of the planning phase is to experiment with the solution. Ideally, we can simulate and iterate on the project, e.g. use 3D models of the building, low fidelity designs to show customers, or low effort versions of the animation movie we are trying to produce.

The simulation ensures that the majority of aspects of our project are scrutinised before the building phase begins. We can assume that the project will run into problems, so we want most of the problems to occur during the cheap planning phase.

Experience in people and technology

To maximise the chance of a successful project we should maximise experience. Flyvbjerg/Gardner consider both experience in key people and in technology. We should try to ensure that key people have experience in similar projects. Ask: “Have they done it before?”.

“Technology is ‘frozen experience’.”

All things being equal, we should use tried and tested off-the-shelf technology instead of shiny new technology. If we can, we should use existing designs and operational processes.This is the same idea behind McKinley’s advice to Choose Boring Technology .

The construction of the Empire State Building is used throughout the book as a positive example. One reason for this was the architect William Lamb who insisted on using only proven technology, and who created a design that allowed repeatable non-custom work steps. Moreover, the construction company Starrett Brothers and Eken had built several similar skyscrapers on time and budget before.

Reference-class forecasting

After the project is planned we need to forecast the project duration and its cost. Unfortunately, planners often forecast too optimistically. Instead of deriving it from the project alone, the authors suggest using reference-class forecasting. The idea is to find a set of comparable projects, and to anchor our forecast based on their outcomes. Then adjust from that anchor.

However, good forecasting cannot protect the project from fat-tailed risks. Since those risks can kill our project, we need to identify the (known) high risk events and try to mitigate them.

“Successful project leaders focus every day on not losing, while keeping a keen eye on the […] goal they are trying to achieve.”

After the planning is done, build quickly

The project duration is in itself a source of budget and time risk. After planning, simulating, and forecasting, we therefore need to act fast once the building phase starts. To do this we should ideally use an experienced team and a modular building pattern.

Build with an experienced team

If possible, hire, what the authors call, a masterbuilder. That is someone who has experience in similar projects and can pick the right team. If such a team doesn’t exist, you or them need to create it.

Moreover, we need to ensure that the incentives of participating contractors are aligned with ours by sharing risks and rewards, e.g. pay a bonus for early completion. Don’t always pick the contractor that submits the lowest bid, because the lowest bid doesn’t necessarily lead to the lowest cost. Moreover, we will try to choose companies that we have successfully collaborated with before.

What’s our Lego?

Nuclear power plant constructions belong to the class of projects that most likely exceed cost and time budgets. Flyvbjerg/Gardner argue that you are building one large thing that has few repeatable, standardised parts. Solar power plants are the opposite. The core ingredient, the solar modules, can be produced rapidly and repeatedly in factories, and then assembled in a modular fashion.

When possible we should try to build modules. We can then produce and assemble the modules in a repeatable process. This delivers value in stages. After we have completed a module, we can use the learnings to iterate on the module’s design and assembly process. So we should always ask ourselves:

“What’s our basic building block, the thing we will repeatedly make, becoming smarter and better each time. What’s our Lego?”

Additional thoughts about the book

Incentives and identity

There is a section in the book that discusses why the Heathrow Terminal 5 project was completed on time. The authors argue that a large part was the alignment of incentives between the project managers, the contractors, and the workers. Contractors were aligned via contractual bonuses and established working relationships. Workers were treated well, their feedback was actively encouraged, and they felt like they were contributing to a historic project in their country. Not part of the book, but the Notre-Dame reconstruction after the fire, was also accomplished on time. I believe that the history of the building, and the feeling of being part of a national project are very powerful motivators that aligned the participants of the project.

Cost overruns by project type

I found the table in the book that shows the mean cost overrun by category interesting.

The table shows the mean (base rate) cost overrun for each project type, the fatness of the distribution (% of projects in the upper tail), and the cost overrun for projects in that tail.

Project type	Mean cost overrun (%)	% of projects in tail (>= 50% overrun)	Mean overrun of projects in tail (%)
Nuclear storage	238	48	427
Olympic Games	157	76	200
Nuclear power	120	55	204
IT	73	18	447
…	…	…	…
Buildings	62	39	206
Rail	39	28	116
Airport	39	43	88
Tunnels	37	28	103
…	…	…	…
Wind energy	13	7	97
Energy transmission	8	4	166
Solar power	1	2	50

It makes sense that (non-standard, non-modular) nuclear projects have the highest base rate for cost increase. On the opposite side are wind energy and solar power projects, that can be pre-produced in factories and assembled on-site. Organising Olympic Games suffers from the fact that they are highly complex, and usually held in a city that hasn’t hosted them before (inexperience). As a tech worker, I am intrigued to see IT projects at the top of the list. Moreover, when IT projects overrun they incur a 447% cost overrun, the highest among all project types.

Learnings for software projects

How applicable is the advice for day-to-day data and software projects? I think much of the advice applies. I found that chunking larger projects into small value-delivering modules is a great way to ensure continuous progress (especially in large refactoring projects).

The emphasis on planning vs. building is probably not as relevant in normal day-to-day software projects. There is less of a cost difference between planning and building a feature (in both cases mostly the software engineers’ time) compared to construction projects.

LLM prompt heuristics that definitely maybe work

Fri, 11 Oct 2024 14:56:00 +0000

Effective prompt writing for large language models continues to be a dark art. Having read the prompt engineering blogs from Meta, Anthropic, and OpenAI and watched some of Anthropic’s prompt discussions online, it does feel more like design than engineering. Even employees from the same lab don’t agree what tricks actually work. If you have time watch this video of one of Anthropic’s prompt engineers running a prompting masterclass. While giving good heuristics to the audience he is careful not to make any definite statements.

I don’t believe it is a good investment to try to become a world-class prompter. All heuristics are highly dependent on the model architecture, training data, and training procedure. This means that with every iteration of the models, heuristics could become obsolete or harmful to performance.

However, the current attention-based architecture doesn’t seem to be going anywhere soon. Therefore, it is reasonable to expect that we can continue to use the prompt context to help the model to move the embeddings of the user task into directions that contain a lot of nuance and information about the domain of the task.

Therefore, I am collecting the advice into a few guidelines that I can use with the current models. I will caveat this by saying that in 90% of my use-cases the model response is parsed by a human: me. Therefore, I am not as worried about hallucinations as someone that puts the model outputs in front of their customers or inside data parsing pipelines.

Here is what seems to work in October 2024 with Claude Sonnet 3.5, GPT-4o, and Llama 3.2.

Prompt specificity

Make the prompt as specific to the task as you can. This is probably the biggest return on your time. Good advice given in this discussion is to imagine printing out the prompt and giving it to a new hire at your company. Then see if they could solve the task. This forces you to give all necessary context and constraints of the task.
Components of a role prompt: The role, its place in an organisation, its perspective and the perspective of the person or organisation being addressed.

No lazy role prompts. Make the role and the context clear.

Bad: You are a cab driver.
Good: You are a cab driver, driving people as your full-time job in London for twenty years and are knowledgeable about the city, its roads, and its sights.

If the model is used inside a product, tell it about it:

Bad: You are an assistant writing document summaries.
Good: You are an assistant used in a product for law firms that summarizes
legal documents.

Order of prompt components

Order the components of your prompts in the following order:

Specific role and context description (put into system prompt if part of interactive use of the model)
Input data, e.g. documents, code snippets, CSV files
Task description
(Optional:) Good examples and bad examples
Task constraints

Use XML tags for prompt components

Use XML tags to separate different components of a prompt.

You are an experienced post-doc at a reputable research institute in the US. You are an expert in the research field of the following paper:

<Research_paper>
{{research_paper}}  
</Research_paper>

Your task is to summarize the findings of the research paper given to you in
<Research_paper> tags.

<Instructions>
- List between 1 - 5 most important findings of the paper. Don't list more
  than 5 findings.
- more instructions...
</Instructions>

Using examples (few shot prompting)

It can help to provide both good and bad examples for a specific task with an explanation why they are good or bad. In the above example we could provide an example research paper with a summary of findings we wrote ourselves as a good example.

Instructions

Response length limit. Instead of writing “Be concise”, tell the model what that means for your context, e.g. “Answer in no more than 4 sentences.”
Avoid open-ended instructions.
Instruct for style, formatting, and restrictions.
Instruct to ask for sources of evidence to reduce hallucinations.

Ask it to respond in a chain-of-thought to increase performance.

You are a logician and love to solve logic puzzles. Carefully read the following puzzle.

<PUZZLE>
Simon is looking at Charlie. Charlie is looking at Sarah. You know that Simon is married and Sarah is not married. Is a married person looking at an unmarried person?
</PUZZLE>
Let's think step by step before giving an answer.

Grammar and style

Avoid typos and wrong punctuation as that can deteriorate the quality of the response.

Parse-able output

Often LLMs add a preamble at the beginning or an epilogue at the end of a response. If you want to force the model to respond only with valid json, you can:

Use the “Prefill Claude’s response” feature
Ask the model to put the json into tags and then extract that block from the response
Prefill the response yourself by adding at the end of your prompt: Here is the JSON: {. The open bracket conditions the model to start the answer with the first JSON key. You then need to prepend the “{” to the response to make it valid.

Other tricks

If your prompt includes logic that could be handled in code, handle it in code.

Bad: You are part of a role-playing game that is used for training of customer support agents in a Fortune 500 company. 

You can assume any of the following roles based on user input. 

If the user asks for Role1 assume the role of a customer asking questions about the company's products. If the user asks for Role2, assume the role of a helpful customer support agent.

Good: Define two different prompts for each role and use code to switch prompts based on what role the operator wants the model to assume.

Give the model a way out if it doesn’t know the answer.

Good: {{prompt}}
If something weird happens and you are unsure about what to do, simply print out
"UNSURE".

Use a temperature of 0 for fact-based, less creative tasks.

Iterative prompt design

You can use the LLM to generate examples for a task and select the good examples. Then use those examples in the prompt that is used “in production”.
If the model responds incorrectly, tell it about the mistake and ask it how you should modify the prompt.

TIL (from Claude) how to create separate RSS feeds for my Hugo blog

Thu, 03 Oct 2024 00:00:00 +0100

I really like the idea of providing different content types of my blog in separate RSS feeds. For example a blog could have:

essays
today-I-learned posts
link-list or bloglink posts
short social media style posts
etc.

Providing different RSS feeds for them allows my readers to subscribe to all or a subset of these, depending on their interest. I am using Hugo, as the static site generator for this website. Setting up these different feeds for a framework that I am only partially familiar with is a great use-case to work with an LLM.

I knew that I would need to:

Distinguish different content types with a flag
Specify different RSS templates that filter on these flags
Tell Hugo to generate separate xml files for each content type and use the right templates.

I just didn’t know how to piece these parts together. I first tried googling the answer and read the typical mix of

stackoverflow posts that were not solving the exact same problem
Hugo forum discussions that did not have a clear solution or where the solution wasn’t transferable to my unique setup.

I think with those first pointers, spending some time on the Hugo documentation, and trial-and-erroring, I could have worked out how to do it. I think that these shallow but unknown problems are the ideal use case for an LLM.

I think of using LLMs as an interface for search + context, where I don’t have to do the laborious work of pre-filtering and connecting the sources. Instead, it helps me get a quick overview and a potential draft of a solution. I can then go deeper and understand the problem better by asking about parts of the solution that I don’t understand. I can then mix this with more googling and get to a solution.

My Goal:

Prompt

I have a hugo blog with the following folder structure:

/content/posts/post1.md
/content/posts/post2.md
/content/posts/_index.md
/content/til/_index.md
/content/til/til1.md
/content/til/til2.md

Markdown pages written in the posts subfolder have type: post in the frontmatter and pages written in the til subfolder have type: til in the frontmatter.

Explain to me step by step how I can create three separate rss feeds on my website.

Only showing content coming from the til subfolder on til.xml
Only showing content coming from the post subfolder posts.xml
One showing everything on index.xml

Solution

Model: claude-3-5-sonnet-20241022 - response

I’d be happy to explain how you can create three separate RSS feeds for your Hugo blog with the structure you’ve described. Let’s go through this step-by-step:

Create RSS template files: First, you’ll need to create three separate RSS template files in your Hugo project’s layouts folder.

Now, let me explain each step in more detail:

We create three separate RSS template files in the layouts/_default/ directory. Each file corresponds to one of the RSS feeds you want to create.
In each RSS template, we use Hugo’s templating language to generate the RSS feed. The main difference between the templates is in the range statement:
- For til.xml, we use {{ range where .Site.RegularPages "Type" "til" }}
- For posts.xml, we use {{ range where .Site.RegularPages "Type" "post" }}
- For index.xml, we use {{ range .Site.RegularPages }} to include all content
We update the Hugo configuration file to define new output formats for TIL and Posts, and to include these new formats in the home page outputs.
We add links to the new RSS feeds in the site’s HTML, typically in the <head> section or footer.
Finally, we build the site to generate the new RSS feed files.

After following these steps, your Hugo site will generate three separate RSS feeds:

til.xml will only include content from the til subfolder
posts.xml will only include content from the posts subfolder
index.xml will include all content

Would you like me to explain any part of this process in more detail?

I followed these steps, except for using my already existing xml template for til.xml, posts.xml, index.xml. I already read in some Hugo discussions that I need to add [outputs] and [outputFormats] to my Hugo config.toml file. However, I didn’t really understand the difference between home, page, section and what exactly home = ["HTML", "RSS", "JSON", "TIL", "POSTS"] does. So I asked for clarification:

Prompt

Can you explain to me the config [outputs] and its purpose as well as [outputFormats]?

Model: claude-3-5-sonnet-20241022 - response

Certainly! I’d be happy to explain the [outputs] and [outputFormats] configurations in Hugo and their purposes.

Now, let me provide a more detailed explanation of these configurations:

[outputs]: This configuration section determines what types of files Hugo will generate for different types of pages on your site. In our case, we’re telling Hugo to generate additional output formats for the home page. The line home = ["HTML", "RSS", "TIL", "POSTS"] means:

For the home page, generate the standard HTML and RSS outputs
Also generate our custom TIL and POSTS outputs This is crucial for creating our multiple RSS feeds, as it instructs Hugo to actually generate these files when building the site.

[outputFormats]: This section allows you to define new output formats or modify existing ones. In our case, we’re defining two new output formats: TIL and POSTS. For each new format, we specify:

mediaType: This tells Hugo what kind of content this is. We’re using “application/rss+xml” because we’re creating RSS feeds.
baseName: This determines the filename of the output file. For example, baseName = "til" will result in a file named til.xml.

By defining these custom output formats, we’re giving Hugo the information it needs to generate our specialized RSS feeds.

The interaction between these two configuration sections is key:

[outputFormats] defines what the TIL and POSTS formats are
[outputs] tells Hugo to actually generate these formats for the home page

This setup allows you to have fine-grained control over what files Hugo generates and how they’re named, which is exactly what we need to create multiple, specialized RSS feeds. Is there any part of this you’d like me to elaborate on further?

This helped but I still didn’t really get why I would have to add this to home and not page because I thought that I want to generate rss for page-types.

Prompt

What’s the difference between adding TIL and POSTS to home instead of page?

Model: claude-3-5-sonnet-20241022 - response

That’s an excellent question! The difference between adding output formats to home versus page in Hugo’s configuration is quite significant and affects how and where these outputs are generated. Let me explain:

Now, let me elaborate on the key points:

Scope and Purpose:
- Adding TIL and POSTS to home creates site-wide RSS feeds that include all relevant content from across your site. This is ideal for your use case of creating separate feeds for TIL entries and blog posts.
- If you added these to page, it would create individual RSS feeds for each content page, which is not what you’re aiming for.
Number of Files Generated:
- With home, you get one til.xml and one posts.xml file for your entire site.
- If you used page, you’d get a til.xml and posts.xml for every single page on your site, which would be unnecessary and potentially confusing.
Content Aggregation:
- The home outputs can access and aggregate content from across your entire site, which is perfect for creating comprehensive RSS feeds.
- page outputs typically only contain information from that specific page.
Location of Generated Files:
- home outputs are usually placed at the root of your site, making them easy to find and link to (e.g., https://yoursite.com/til.xml).
- page outputs would be scattered throughout your site structure, mirroring your content organization.
Practical Implementation:
- For your RSS feeds, using home allows you to create the three distinct feeds you want (TIL, posts, and everything) in a clean, organized manner.
- Using page for this purpose would be impractical and wouldn’t achieve your goal of separate, comprehensive feeds.

In summary, for your specific need of creating site-wide RSS feeds for different content types, adding TIL and POSTS to the home outputs is the correct approach. It generates the feeds at the site level, includes all relevant content, and produces a single file for each feed type, which is exactly what you’re looking to achieve.

I followed the suggested approach and added to my config.toml:

# I want to top-level output html, rss, and my two custom outputs
[outputs]
  home = ["html", "rss", "til", "post"]

# Here I define how to process these custom feeds
[outputFormats.til]
  mediaType = "application/rss+xml"
  baseName = "til"

[outputFormats.post]
  mediaType = "application/rss+xml"
  baseName = "posts"

Then I built the page and Hugo would build three different xml files:

public/index.xml
public/posts.xml
public/til.xml

However, the files all had the same content as if they were built from the same template instead of the individual templates that filter by page type.

Prompt

This generates the three xml files but they all seem to use the same template file

Model: claude-3-5-sonnet-20241022 - response

I apologize for the confusion in my previous explanation. You’re right to point out that issue. Let’s correct this and explain how to properly set up distinct templates for each RSS feed.

Now, let me explain the key changes and why they solve the problem:

Template File Naming: The crucial change is in how we name and place the template files. Instead of putting them in layouts/_default/, we’re now creating:
- layouts/index.til.xml
- layouts/index.posts.xml
- layouts/index.xml
This naming convention is important because Hugo uses it to match the output formats we defined in the configuration.
Template Content: The content of each template remains largely the same, but we’ve made sure that each one is filtering for the correct content type:
- index.til.xml uses {{ range where .Site.RegularPages "Type" "til" }}
- index.posts.xml uses {{ range where .Site.RegularPages "Type" "post" }}
- index.xml continues to use {{ range .Site.RegularPages }} for all content
Configuration: The configuration in config.toml remains the same as before. The [outputFormats] and [outputs] sections are correctly set up to generate these files.
How It Works:
- When Hugo sees TIL in the [outputs] section, it looks for a template named index.til.xml
- Similarly, for POSTS, it looks for index.posts.xml
- The default RSS output uses index.xml

This setup ensures that each RSS feed uses its own template, allowing you to customize the content for each feed independently.

The reason your previous attempt generated three XML files with the same content was likely because Hugo was falling back to a default RSS template when it couldn’t find specific templates for TIL and POSTS.

With these changes, you should now have three distinct RSS feeds, each with its appropriate content. Does this explanation help clarify the issue? Would you like me to elaborate on any part of this solution?

So I moved the template layout files as suggested to:

layouts/index.til.xml
layouts/index.posts.xml
layouts/index.xml

and they were correctly selected. You can find the correct feeds here:

https://staticnotes.org/index.xml

https://staticnotes.org/posts.xml

https://staticnotes.org/til.xml

Conclusion

A common criticism is that LLMs will confidently hallucinate incorrect information. I think this will likely be true for the next couple of years. However, when I use the old-school google + stackoverflow + documentation + online discussion workflow I also might generate an incorrect approach. This can happen because I misunderstand the content or because I can’t piece the information from different sources together correctly for my use case.

I think viewing an LLM as an interactive interface to search and combine information and context is the right mental model to use them productively.

Processing 112M rows of steam reviews locally with DuckDB

Sat, 28 Sep 2024 13:43:40 +0000

In DuckDB use cases for data scientists: Querying remote S3 files I wrote about how I use DuckDB as a convenient way to query data from CSV or Parquet files in S3. Another use case for data scientists and data engineers is DuckDB’s ability to process larger-than-memory data on your local machine. For us data scientists this fills the gap between

small data that you can transform with pandas
big data that typically requires a multi-node processing engine like PySpark.

In this post I am going to run some exploratory queries against this kaggle Steam review dataset . It contains 112M rows of Steam game reviews and comes as an uncompressed 46GB CSV file (17GB compressed). Let’s find out how my MacBook copes with it. Later I am going to run a small comparison against polars , a DataFrame library, which is often mentioned for out-of-memory processing.I am not comparing against pandas because its inability to work with large datasets was one of the reasons why I explored this topic. However, you could do this with chunking .

Useful commands and settings

Before I start, here are some useful DuckDB commands and settings that I often use:

Command	What is it good for?
`.timer on`	Prints the execution time after each SQL command
`FROM duckdb_memory();`	Gives an overview of how much memory is used by DuckDB
`SET enable_progress_bar = true;`	Prints a progress bar for query runs
`SET memory_limit = '2GB';`	Controls how much data DuckDB can keep in RAM.
`.mode line`	Prints query results one at a time (useful if many columns). Default: `.mode duckbox`
`SUMMARIZE (select *)`	Summarizes the contents of a table.
`.exit`	Stops DuckDB.

Steam review dataset

First, I want to familiarise myself with the dataset. I downloaded the 46GB file all_reviews.csv to my 2023 MacBook Pro M3 with 18GB RAM. After starting a non-persistent session with the duckdb command, I use the above commands to activate the timer and line mode. I then take a look at an example row with:

 select * from read_csv('all_reviews.csv', ignore_errors=true) limit 1;

              recommendationid = 148919893
                         appid = 10
                          game = Counter-Strike
                author_steamid = 76561199036724879
        author_num_games_owned = 0
            author_num_reviews = 3
       author_playtime_forever = 197
author_playtime_last_two_weeks = 197
     author_playtime_at_review = 197
            author_last_played = 1698336369
                      language = russian
                        review = старость
             timestamp_created = 1698336397
             timestamp_updated = 1698336397
                      voted_up = 1
                      votes_up = 0
                   votes_funny = 0
           weighted_vote_score = 0.0
                 comment_count = 0
                steam_purchase = 1
             received_for_free = 0
   written_during_early_access = 0
         hidden_in_steam_china = 1
          steam_china_location =

This gives me an overview of the file columns and an idea of their content.

Processing the file directly

Rather than slowly loading all data into DuckDB, I want to test DuckDB’s stream processing capabilities. Instead of fully materialising the data in memory, the execution engine reads and processes the data in chunks.This is a useful feature if you want to convert larger-than-memory files from one format to another format, e.g. CSV to Parquet. But that’s not what we are here for today.

I am going to use two aggregation queries for my benchmark:

The first query counts the number of English reviews per steam account and sorts them from highest to lowest

D select author_steamid, 
COUNT(*) AS num_reviews  
FROM read_csv('all_reviews.csv', ignore_errors = true)  
WHERE language = 'english' 
GROUP BY author_steamid 
ORDER BY num_reviews DESC;
100% ▕████████████████████████████████████████████████████████████▏ 
┌───────────────────┬─────────────┐
│  author_steamid   │ num_reviews │
│       int64       │    int64    │
├───────────────────┼─────────────┤
│ 76561198030784015 │        9674 │
│ 76561198024340430 │        5930 │
│ 76561198067298289 │        5534 │
│ 76561198094803808 │        4341 │
│ 76561198125392509 │        4124 │
│ 76561198027267313 │        4124 │
│ 76561197960373660 │        3350 │
│ 76561197970602587 │        3212 │
│ 76561197961017729 │        2810 │
│ 76561198155150242 │        2554 │
│ 76561198066590240 │        2490 │
│ 76561198045381877 │        2280 │
│ 76561198069159152 │        2216 │
│ 76561197960319772 │        1978 │
│ 76561198062813911 │        1940 │
│ 76561198025731804 │        1928 │
│ 76561198137285867 │        1898 │
│ 76561198043135631 │        1883 │
│ 76561198036629241 │        1875 │
│ 76561198055119582 │        1843 │
│         ·         │           · │
│         ·         │           · │
│         ·         │           · │
│ 76561198111175247 │           1 │
├───────────────────┴─────────────┤
│    15324507 rows (40 shown)     │
└─────────────────────────────────┘
Run Time (s): real 22.556 user 166.382276 sys 9.815767

This query processes the 112M rows in a surprisingly short 22.5s. The most active Steam user has written 9674 game reviews. You can find them here .

I made the second query intentionally more complex. I filter for reviews that contain the word love and I also compute the mean of the votes_up score and the sum of the votes_funny score for each user. I then filter the aggregation using having for only users that have a sum_votes_funny score of more than 100.

D SELECT author_steamid, 
    COUNT(*) AS num_reviews,
    avg(votes_up) as mean_votes_up, 
    sum(votes_funny) as sum_votes_funny,
    FROM read_csv('all_reviews.csv', ignore_errors = true) 
    WHERE language = 'english' and review LIKE '%love%'
    GROUP BY author_steamid 
    having sum_votes_funny > 100 
    ORDER BY num_reviews DESC; 
100% ▕████████████████████████████████████████████████████████████▏ 
┌───────────────────┬─────────────┬────────────────────┬─────────────────┐
│  author_steamid   │ num_reviews │   mean_votes_up    │ sum_votes_funny │
│       int64       │    int64    │       double       │     int128      │
├───────────────────┼─────────────┼────────────────────┼─────────────────┤
│ 76561198155150242 │         777 │   1.09009009009009 │             362 │
│ 76561198042406453 │         665 │ 13.478195488721804 │             341 │
│ 76561198043135631 │         475 │ 2.9410526315789474 │             116 │
│ 76561198149437416 │         352 │ 1.2386363636363635 │             256 │
│ 76561198007343154 │         342 │  4.038011695906433 │             375 │
│ 76561198066590240 │         273 │  9.293040293040294 │             247 │
│ 76561197970314107 │         241 │  38.15767634854772 │             319 │
│ 76561197961017729 │         237 │ 10.278481012658228 │             251 │
│ 76561197981638563 │         231 │ 26.372294372294373 │             509 │
│ 76561197972040704 │         223 │  8.560538116591928 │             123 │
│ 76561197992694498 │         223 │   73.1390134529148 │             759 │
│ 76561197970761123 │         222 │ 22.603603603603602 │             150 │
│ 76561198043609914 │         217 │ 18.792626728110598 │             129 │
│ 76561198053422627 │         202 │  17.04950495049505 │             284 │
│ 76561198007888370 │         196 │  93.96938775510205 │            3345 │
│ 76561198040884867 │         191 │  38.41884816753927 │             509 │
│ 76561198817597644 │         166 │ 22.542168674698797 │             119 │
│ 76561198356141989 │         166 │ 19.246987951807228 │             216 │
│ 76561198031599084 │         165 │ 17.163636363636364 │             171 │
│ 76561198011647032 │         159 │  6.345911949685535 │             192 │
│         ·         │           · │                ·   │              ·  │
│         ·         │           · │                ·   │              ·  │
│         ·         │           · │                ·   │              ·  │
│ 76561198106232693 │           1 │               96.0 │             155 │
├───────────────────┴─────────────┴────────────────────┴─────────────────┤
│ 2383 rows (40 shown)                                         4 columns │
└────────────────────────────────────────────────────────────────────────┘
Run Time (s): real 26.648 user 213.432443 sys 7.281861

With ~26s this is on par with the previous query. User 76561198155150242 used the word “love” in 777 reviews and collected 362 funny upvotes across their reviews.

These timings show how incredibly convenient DuckDB is to analyse large datasets. <30s is still in the realm where I can interactively work with the dataset and explore different queries.You can push DuckDB to the limit by using a blocking operator, e.g. a rank() window function. This query will be slow because the entire input needs to be buffered to compute the result.

When I explore data it is more common that I work in a notebook instead of the CLI. Fortunately, I can run the same queries using the DuckDB Python client . In the next section I am going to compare it against polars, another relatively new kid on the block for out-of-memory analytics.

Working in a Jupyter notebook

DuckDB Python client

I am going to write the code to run the above queries using the duckdb Python package and return the results as a pandas dataframe.

import duckdb
import pandas as pd

# Connect to DuckDB
conn = duckdb.connect(':memory:')


# Define the queries
query1 = """
SELECT author_steamid, COUNT(*) AS num_reviews 
FROM read_csv('./steam_reviews/all_reviews/all_reviews.csv', ignore_errors = true) 
WHERE language = 'english' 
GROUP BY author_steamid 
ORDER BY num_reviews DESC 
"""

query2 = """
SELECT author_steamid, 
COUNT(*) AS num_reviews,
avg(votes_up) as mean_votes_up, 
sum(votes_funny) as sum_votes_funny,
FROM read_csv('./steam_reviews/all_reviews/all_reviews.csv', ignore_errors = true) 
WHERE language = 'english' and review LIKE '%love%'
GROUP BY author_steamid 
having sum_votes_funny > 100 
ORDER BY num_reviews DESC 
"""

%%timeit

# Execute the query
result1 = conn.execute(query1)
# Convert the result to a pandas DataFrame
df1 = result1.df()

16.6 s ± 627 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

df1.head(5)

	author_steamid	num_reviews
0	76561198030784015	9674
1	76561198024340430	5930
2	76561198067298289	5534
3	76561198094803808	4341
4	76561198027267313	4124

%%timeit

# Execute the query
result2 = conn.execute(query2)
# Convert the result to a pandas DataFrame
df2 = result2.df()

19.9 s ± 343 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

df2.head(5)

	author_steamid	num_reviews	mean_votes_up	sum_votes_funny
0	76561198155150242	777	1.090090	362.0
1	76561198042406453	665	13.478195	341.0
2	76561198043135631	475	2.941053	116.0
3	76561198149437416	352	1.238636	256.0
4	76561198007343154	342	4.038012	375.0

Not surprisingly this gives the same results as the queries run from the CLI. Having the aggregated results as a pandas dataframe allows me to use it in downstream work. I don’t need to learn another syntax.

## clear memory 
import gc 
del result1, result2, df1, df2;
gc.collect();

Polars

Since polars is often brought up as a faster and better pandas, I want to do a performance comparison. Polars supports lazy dataframes, which allows me to define operations on the dataframe without loading it fully into memory first.

Below are the two equivalent queries written using polars syntax.

import  polars as pl

# Create a lazy DataFrame
df_lazy = pl.scan_csv("./steam_reviews/all_reviews/all_reviews.csv", ignore_errors=True)

%%time

# Define the query using lazy operations
result_polars1 = (
    df_lazy.filter(pl.col("language") == "english")
    .group_by("author_steamid")
    .agg(pl.len().alias("num_reviews"))
    .sort("num_reviews", descending=True)
    
)

# Execute the query and collect the results
df_polars1 = result_polars1.collect()
df_polars1.head(5)

CPU times: user 1min 3s, sys: 22.1 s, total: 1min 26s
Wall time: 2min 7s

shape: (5, 2)

author_steamid	num_reviews
i64	u32
76561198030784015	9822
76561198024340430	5983
76561198067298289	5577
76561198094803808	4408
76561198125392509	4203

%%time

result_polars2 = (
    df_lazy
    .filter(
        (pl.col("language") == "english") & 
        pl.col("review").str.contains("love")
    )
    .group_by("author_steamid")
    .agg([
        pl.len().alias("num_reviews"),
        pl.col("votes_up").mean().alias("mean_votes_up"),
        pl.col("votes_funny").sum().alias("sum_votes_funny")
    ])
    .filter(pl.col("sum_votes_funny") > 100)
    .sort("num_reviews", descending=True)
)


# Execute the query and collect the results
df_polars2 = result_polars2.collect()
df_polars2.head(5)

<timed exec>:9: DeprecationWarning: `pl.count()` is deprecated. Please use `pl.len()` instead.


CPU times: user 1min 21s, sys: 42.7 s, total: 2min 4s
Wall time: 5min 14s

shape: (5, 4)

author_steamid	num_reviews	mean_votes_up	sum_votes_funny
i64	u32	f64	i64
76561198155150242	786	1.094148	369
76561198042406453	673	13.43685	345
76561198043135631	480	2.922917	116
76561198149437416	360	1.216667	256
76561198007343154	354	4.014124	381

Now we can compare DuckDB vs. polars execution times for the two queries:

wall time	DuckDB	polars
query 1	17s	2min 7s
query 2	20s	5min 14s

Conclusion

This investigation shows that DuckDB is a powerful and convenient tool to process larger-than-memory datasets on a single machine. As a data scientist this is useful, because I can focus on exploring the data in the early exploration phase. I don’t yet have to spend time setting up more complex tools or work on a remote machine. The quick comparison with polars also shows its speed advantages and that it can be used as a drop-in when aggregating data in Jupyter notebooks.

TIL about the QUALIFY statement in SQL

Mon, 16 Sep 2024 00:00:00 +0100

Today I came across the QUALIFY clause which is supported in some SQL dialects. It’s not part of the SQL standard but supported by the main analytical databases like BigQuery, Snowflake, Oracle, Databricks, DuckDB, etc. It’s part of Snowflake’s dialect which is the data warehouse that I use at work. The QUALIFY statement lets me filter the result of a query based on the result of a window function.

The order of execution is the following:

-- SELECT <columns>
--   FROM <table>
--   [GROUP BY ...]
--   [HAVING ...]
--   QUALIFY <window function predicate>

QUALIFY is for window functions what HAVING is for group-by statements. It allows me to filter based on the outputs of these operations without having to use a subquery or CTE. Therefore I can write more concise SQL.

I often deduplicate data using the row_number() window function. Sometimes I just want the latest event for a particular user in a dataset.

An example would be:

-- logs with some user events and timestamps
with logs_annotated as (
    select
        logs.*,
        row_number() over (partition by user_id order by event_timestamp desc) as row_number
    from logs
)
select * from logs_annotated where row_number = 1

With QUALIFY I can write one statement instead:

select * from logs
qualify row_number() over (partition by user_id order by event_timestamp desc) = 1

This works because the QUALIFY statement is evaluated after the window function result has been computed and can therefore be used in filtering.

Interest rate expectations

Sat, 14 Sep 2024 00:00:00 +0100

This morning, the top headline on the front page of my weekly newspaper was:

Bets rise on bumper rate cut by Fed

Investors have sharply increased their bets on half percentage point interest rate cut by the Federal reserve next week as the US central bank prepares to lower borrowing cost for the first time in more than four years. Traders in swaps markets are pricing in a 43 percent chance the Fed will opt for a bumper cut in a bid to prevent high rates damaging the economy. - FT Weekend (14.9.2024)

I often read news about expectations on how the Fed will set interest rates. I have a vague sense that it is related to the price of some interest rate related futures. I just asked myself if I wanted to look up what the market expectation is next week, where would I actually go to look? This detail is regularly glossed over in news articles. It turns out that the maths behind the expectation approximation is fairly simple. Let us work it out.

The Federal Funds Rate

Banks are required to put a certain percentage of their deposits into accounts at a Federal Reserve Bank. This is to maintain liquidity to cover depositors’ withdrawals in the short-term. This reserve requirement is loosely a percentage of the bank’s deposits at the end of the day averaged over a two-week period. If at the end of the day a bank has excess reserve balances it can lend it overnight to another bank that is undercapitalised. Banks do that a lot and negotiate an interest rate for this overnight loan. The weighted average of all these deals for the day is the effective federal funds rate (EFFR).

The Fed’s federal funds rate (FFR) target range is set by the Federal Open Market Committee and is the desired range for the EFFR. While the Fed can’t directly impact the EFFR it can influence it via their own deposit interest rates or changes to the monetary supply. This tells us what today’s FFR range and EFFR are (both are published by the Fed). But how do we compute the market’s expectations on changes to the FFR range?

Federal Funds Futures

We can infer the expectations of FFR range changes at the next committee meeting by looking into how FFR future contracts are priced. Those are traded on the Chicago Mercantile Exchange (CME). You can view the monthly future quotes on the CME website .

These futures are used by banks and fixed-income portfolio managers to hedge against short-term interest rate fluctuations. The 30-day futures are monthly contracts that are settled on the last business day of every month. The contract price payable is the arithmetic mean of the daily EFFR during the contract month as reported by the Federal Reserve Bank of New York, subtracted from 100. For example if the average interest rate was 3.5% for a given month, then the contract price would be 100 - 3.5 = $96.5. Unfortunately, the minimum contract size is the price times 4167, which for this example would be $96.5 * 4167 = $416,796.5. A bit too high for my personal hedging needs.

What do the Federal Funds Future(s) hold?

Now that we know about the EFFR and Federal Funds futures, we can collect all ingredients to compute the expectations:

The current FFR range was set in the July meeting to 5.25% - 5.5%.
The dates of the next committee meetings are published here on federalreserve.gov . The remaining meetings this year are scheduled for: 18.9.2024, 7.11.2024, 18.12.2024.
The expected EFFR from the futures contracts for the month of the next committee meeting and the following month. For the remaining year they were on the 14.9.2024:

import pandas as pd

df = pd.DataFrame(
    {
        'MONTH' : ['09-2024', '10-2024', '11-2024', '12-2024'],
        'PRICE' : [94.81, 95.03, 95.315, 95.58],
        'MEETING': [18, None, 7, 18]
     }
).set_index('MONTH')
display(df)

	PRICE	MEETING
MONTH
09-2024	94.810	18.0
10-2024	95.030	NaN
11-2024	95.315	7.0
12-2024	95.580	18.0

The next month without a committee meeting is October. This means that October’s average EFFR can only be impacted by the September committee meeting. This means that the October future contract price of $95.03 is the expected average EFFR for October.

r_october_avg = 100 - df.at['10-2024','PRICE']
r_september_avg = 100 - df.at['09-2024','PRICE']

It is also the expected EFFR on the last day of September (and the first day of November).

r_september_end = r_october_avg

We are interested in the implied difference in EFFR at the beginning and the end of the month. If there is a difference, then this would imply that the FED committee changed the FFR range.

Keep in mind that the contract price is the arithmetic mean of the realised EFFR on every day in September. Assume that the FED committee does lower the price on the 18th of the month, then we will have 18 days at a higher rate and 30-18 = 12 days at a lower rate. All of that is captured in the future price (the average EFFR) for September.

Therefore we can use:

\[ EFFR(\text{average of Sep}) = \frac{N}{30} EFFR(\text{start of Sep}) +\frac{30-N}{30} EFFR(\text{end of Sep}) \]

\[ \Leftrightarrow EFFR(\text{start of Sep}) = \left( EFFR(\text{average of Sep}) - \frac{30-N}{30}\cdot EFFR(\text{end of Sep}))\right) \cdot \frac{30}{N} \]

where $N$ is the number of days before the committee meeting in September.

days_month = 30
days_before_meeting = df.at['09-2024','MEETING']
days_after_meeting = days_month - days_before_meeting
r_september_start = (r_september_avg - (days_after_meeting/days_month * r_september_end) ) * (days_month/days_before_meeting)
print(f'The expected EFFR for the start of September is {r_september_start:.2f}%.')

The expected EFFR for the start of September is 5.34%.

Since today is the 14th of September, we can already look up the realised EFFR up until today. If today were the 25th of August, then we need to calculate the expected $EFFR(\text{start of Sep})$ with the above formula.

r_september_change = r_september_end - r_september_start
print(f'Comparing the expected EFFR at beginning and end of month we have a delta of {r_september_change:.2f}.')

Comparing the expected EFFR at beginning and end of month we have a delta of -0.37.

The market’s expectation for September is that we will have 37bps lower EFFR at the end of the month. If we buy the futures contract today we will make money if the FED committee cuts the rate in such a way that the realised EFFR will decrease more than 37bps compared to the beginning of the month.

Traditionally, the committee changes the FFR range by 25bps (however 50 and 100 basis point changes do occur, e.g. after 9/11, after the 2007-2008 housing market crash and during the COVID-19 pandemic).

However, we can see that the market prices in a 37bps lower EFFR. This means we can calculate the probability of a 25bps cut vs. a 50bps cut.

print(f'The price implies {r_september_change / -0.25:.2f} x 25bps rate cuts.')

The price implies 1.47 x 25bps rate cuts.

probability_of_25_bps_cut = 1 - (r_september_change/(-0.25) % 1)
print(f'This implies a probability of {100 * probability_of_25_bps_cut:.2f}% that the committee announces a 25bps rate cut.')

This implies a probability of 53.33% that the committee announces a 25bps rate cut.

probability_of_50_bps_cut =  (r_september_change/(-0.25) % 1)
print(f'This implies a probability of {100 * probability_of_50_bps_cut:.2f}% that the committee announces a 50bps rate cut.')

This implies a probability of 46.67% that the committee announces a 50bps rate cut.

The calculation confirms the newspaper article’s claim that the rate decision is priced as a coin flip between a 25bps and a 50bps rate cut.

Update 18.9.2024: The FOMC decided on a 50bps cut to a target range of 4.75% - 5.0%.

What about Europe?

If you are interested in the expectations for the interest rates set by the Bank of England, you can use One Month SONIA Index Futures . Similarly, for the ECB interest rates expectations you can check ECB Dated ESTR Futures .

Jupyter Notebook

You can find the Jupyter notebook for this post here .

TIL about just and justfiles

Fri, 13 Sep 2024 00:00:00 +0100

just is just a command runner that I recently started using to define aliases for commands in my projects, e.g. build, test, or deploy commands.

Installation

just can be installed via homebrew:

brew install just

Example

I want to create alias commands for my dbt project to build my dbt models in production vs. development environments. At the moment, I need to run the commands:

# prod warehouse
poetry run dbt run --target prod
# dev warehouse
poetry run dbt run --target dev

To define shorter just commands for this project I add a justfile into the project folder.

# justfile
run-prod *ARGS:
    poetry run dbt run --target prod {{ARGS}}

run-dev *ARGS:
    poetry run dbt run --target dev {{ARGS}}

If I am inside the project folder I can now run just run-prod and just run-dev. Moreover, I can pass in any dbt keyword arguments, e.g.

just run-dev --select mymodel

will resolve to

poetry run dbt run --target dev --select mymodel

Is this not just `make`?

just is inspired by the Unix build tool make from 1978. However, just is not a build system and focuses solely on command running. This avoids having to deal with things like .PHONY that can catch you out in make recipes. I also like two additional features of just:

I can run my just commands from any subfolder of the project folder that contains the justfile. This is useful when I work on this blog. I have to run hugo serve from the top-level project folder and Jupyter-to-markdown converter nb2hugo from the notebooks/project_x/ subfolder. With just command aliases it doesn’t matter from which folder I run the command.

just allows you to load environment variables from an .env file.

Example .env:

DATABASE=localhost:1111
USERACCOUNT=rob1234

Add set dotenv-load to your justfile and the environment variables will be available in your shell:

set dotenv-load

check-vars:
    @echo "Env vars are DATABASE:$DATABASE and USER:$USERACCOUNT"

Running just check-vars will output:

Env vars are DATABASE:localhost:1111 and USER:rob1234

Clearing up confusion around IPython, ipykernel, and Jupyter notebooks

Mon, 26 Aug 2024 00:00:00 +0100

One of my big recurring time sinks while doing data science work used to be trying to get my colleagues’ Jupyter notebooks to run on my machine. The main contributing factors:

My team generally uses poetry environments to improve reproducibility, but sometimes dependencies aren’t specified
I use VS code and the Jupyter extension to edit notebooks in VS code which requires more configuration than running the Jupyter web UI.
I lacked a clear understanding of the differences between IPython, ipykernel, jupyter and which Python environments are being used when running a notebook.

So this is my attempt at a friendcatcher . I hope this saves you a few minutes the next time you run into similar issues.

The different components

Let’s distinguish the components that play a role in running a Jupyter notebook:

Jupyter Notebook platform: A web-based interactive computing platform that supports different languages via different kernels, e.g. for Python (ipykernel), Julia (IJulia), R (IRKernel)
Jupyter notebook (extension .ipynb): is a document in json format that holds metadata and cell code.
IPython command shell: The shell has two components:
1. An interactive Python shell. You can start it with ipython. It’s like the default Python REPL but with enhanced features, e.g. object introspection, tab completion, input history, magic commands, etc.
2. A Jupyter kernel ipykernel. This is the backend process where user Python code runs and which can be connected to different frontends. One frontend is indeed the IPython shell, another one a Jupyter notebook. You can install ipykernel as a standalone package into your Python environment.
jupyter Python package. This is a metapackage which installs the notebook, qtconsole, and ipykernel.

graph TD;

colab(Google Colab UI) <--> ipykernel(ipykernel)
vs(VS Code UI) <--> ipykernel
ui(jupyter notebook UI) <-->  server
server(jupyter server) <-->  ipykernel
ipykernel <-->  ipython[IPython]

Jupyter kernels vs. shell environment

One source of confusion is that a Jupyter kernel can point to a different Python executable than your shell environment.

To get an overview of available Jupyter executables you can use:

List all available Jupyter executables in your system:
```
$ type -a jupyter
```
List all available Jupyter kernels:
```
$ jupyter kernelspec list
```

Every Jupyter kernel folder includes a kernel.json file that links to the Python executable that is being used. Note this can be different from the Python executable referenced by your current shell. Moreover, the shell environment of a Jupyter notebook uses the Python executable used to launch the notebook.

Print path of currently used Python executable:
```
$ type python
```
or in a notebook cell:
```
!type python
```
Print path of Python executable of current kernel: In a notebook cell:
```
import sys
sys.executable
```

You can create new kernels using the ipykernel package:

$ python -m ipykernel install --user --name envname --display-name "Python (envname)"

Dependency management

Since I use VS Code as my frontend I just need to add the ipykernel package into the virtual environment that I use to manage all other dependencies used to run the notebook. This ensures that the same Python executable is used for the kernel and the shell environment.This is well explained here .

These are the steps to create a new environment for a Jupyter notebook:

Create project folder:
```
$ mkdir notebook_project
```

Create new virtual environment in the folder, then activate it

$ cd notebook_project
$ python3 -m venv .venv
$ source .venv/bin/activate

Install ipykernel (and other dependencies) using pip (make sure the venv is activated):
```
$ python3 -m pip install ipykernel
$ python3 -m pip install pandas
```
Create a new notebook
```
$ touch mynotebook.ipynb
```
Open notebook in VS Code and in the top right corner select Select Kernel –> Python Environment –> .venv (.venv/bin/python)
You should now be able to run the notebook and use the pandas package inside the notebook.
To add new dependencies:
- Use the terminal: $ python3 -m pip install <package_name>
- Install from within a notebook cell:
```
import sys
!{sys.executable} -m pip install <package_name>
```
- Specify your dependencies in a requirements.txt

I use poetry for virtual environments and dependency management. So in step 2 I would instead use:

$ cd notebook_project
$ poetry init

and install packages via:

$ poetry add ipykernel
$ poetry add pandas

If I want to use the default Jupyter UI, I can install the jupyter metapackage into my environment and then start the UI with:

$ poetry add jupyter
$ poetry run jupyter notebook

Links

TIL to create continuously numbered lists with content between items

Wed, 21 Aug 2024 00:00:00 +0100

When writing notes in markdown I commonly encounter the problem that I want to create an automatically numbered list with content between the items. However, this messes up the numbering as it creates two separate numbered lists, e.g.

My first point
My second point

code block or image

My third point
My fourth point

The solution to this is embarrassingly obvious once you know it. Just move the image or code block one level down by indenting it with four spaces. This keeps the list intact.

My first point
My second point

code block or image
My third point
My fourth point

TIL how to set a lower threshold on a column in SQL

Wed, 07 Aug 2024 00:00:00 +0100

For the last couple of months I have come across multiple instances where I wanted to write an SQL query to apply a lower threshold to all the values in a particular column in one of our Snowflake tables.

I used quite clunky workarounds to do that. It turns out you can use the SQL functions GREATEST or LEAST for it.

Example

I am creating a temporary table in the first CTE with a column DATA_COLUMN that has positive and negative numbers.

with example_table as (
SELECT * 
FROM (
    VALUES 
        (1, -100.423), 
        (2, 5.0), 
        (3, -0.001), 
        (4, 100.0), 
        (5, 12.0)
) AS t (row_nr, data_column)
)

select GREATEST(DATA_COLUMN, 0.) as thresholded 
from example_table

Output:

The function’s primary purpose is to take the greatest value among a number of referenced columns, e.g. GREATEST(column1, column2, column3).

staticnotes.org

Encrypted cloud backups with restic

Requirements for my cloud backup

Getting cheap space in the cloud

Installing and configuring restic

Basic restic backup commands

Inspecting the storage box

Working with snapshots

Restoring backups

Managing the repository’s size and integrity

Scheduling and profiles with resticprofile

Conclusion

Appendix

Launchd schedules with sleeping MacBook (closed lid)

Locks

Exclusion file

TIL how to trigger notifications in Thunderbird for specific emails

Problem

Solution

FiltaQuilla extension

Notification display script

Set up filter rule

Visualising London rental bike location data

Collecting the data

Convert to DuckDB table

Busiest stations

Busiest hours of the day

Storing geometric data as parquet files

Visualising availability using QGIS

Visualising busy stations

Rental bike availability during commute hours

Conclusion

References

Jupyter Notebook

TIL how to profile memory in Python

Problem

Solution

UK house price changes in real terms

Querying the housing price data

Inspecting the data

Computing real house prices

Visualising nominal and real house prices over time

Detached Houses in South England Counties

Flats in London Boroughs

Terraced houses in London Boroughs

Winners and Losers in real terms (last 3 years)

Conclusion

Appendix

Jupyter Notebook

TIL how to set up multiple SSH keys (for multiple GitHub accounts)

1. Generate the SSH keys for each account

2. Associate the public keys with the GitHub accounts

3. Set up OpenSSH client config file

4. Use the right hostname when cloning repositories

TIL how to re-run only failed tests

TIL how to investigate missing data easily with DuckDB

Problem

Solution

TIL that DuckDB now supports Avro files

Install extension

Read local files

Read files stored in S3

Building a local data warehouse with DuckDB, dbt, and Superset

Loading the raw data into DuckDB

Setting up dbt for data transformations

dbt models

Check tables were created

Superset for data analytics dashboards

Install Superset with DuckDB support

Configure Superset

Add datasets in Superset

Create dashboard in Superset

Conclusion

Troubleshooting

Book notes: Poor Charlie’s Almanack

Advocating interdisciplinary collaboration between university departments

Fraud and advice for endowment funds

(Inverted) Advice for Graduates

Mental checklist to analyse problems

Conclusion

Installing and configuring `restic`

Scheduling and profiles with `resticprofile`