Posts on staticnotes.org

Encrypted cloud backups with restic

Mon, 31 Aug 2026 00:00:00 +0100

My current backup strategy is pretty inconsistent. I run a weekly system backup of my MacBook onto an external hardrive sysbackup using CarbonCopyCloner . I also have another external hardrive photos1 to store files that don’t fit onto my MacBook and which isn’t backed up at all. Unfortunately, this includes most of my photos and videos. Storing those photos in the cloud would be a convenient option. However, I am not comfortable uploading important personal documents and photos to the cloud. Even if Google Photos reminds me daily that I should.

This status quo is eventually going to fail, e.g. if the photos1 drive breaks or both my MacBook and the sysbackup drive get destroyed at the same time (burglary, fire, flooding, ransomware attack). Recently, my photo hardrive made unusually loud noises when plugged in, which prompted me to finally adopt a more solid backup approach.

Generally for backups the 3-2-1 rule is recommended (3 copies, 2 different storage media, 1 off-site backup). To accomplish this I will periodically clone the external hardrive photos1 to a new external hardrive photos2 and also set up the off-site backup. This post is mostly about how I set up the cloud backup, but I will summarize the overall approach at the end as well.

Requirements for my cloud backup

To find the right approach I wrote down my requirements:

reliable and cheap cloud storage
backups encrypted locally before upload
open-source software to prevent vendor lock-in or software disappearing in 5 years.
simple to use for someone comfortable with the command line

In the end I settled on restic which works on Linux, macOS, Windows, encrypts data locally by default, and doesn’t need to run on a server. I also looked into BorgBackup which probably would have worked as well. However, that requires running borg on the server. Moreover, the commands and basic configuration are straightforward.

Getting cheap space in the cloud

I already have a Hetzner account to host my Nextcloud instance. Luckily, they offer very cheap storage. I got the Hetzner 1TB Storage Box for 3.83€/month. This means my data is stored in their datacenter in Falkenstein, Germany. By activating automatic snapshots of the storage box in the Hetzner console, I get another backup layer. Here you can see the inside of their datacenter.

To set up the Hetzner Storage Box to work with restic, I had to enable External Reachability, enable SSH, and set a password in the Hetzner console. The communication between my local machine and the storage box will be via SSH and the file upload happens over SFTP which is based on SSH.

First I create a new pair of SSH keys I have more detailed notes on setting up SSH keys here . and then upload the public key to the storage box. In short:

Create new SSH keys:

$ ssh-keygen -t ed25519 -f ~/.ssh/storagebox

Add a host entry for the storagebox to your ~/.ssh/config file:

Host storagebox
    HostName uXXXXXX.your-storagebox.de
    User uXXXXXX
    Port 23
    IdentityFile ~/.ssh/storagebox
    WarnWeakCrypto no-pq-kex

Copy the public SSH key to the storagebox:

$ ssh-copy-id -p 23 -s -i ~/.ssh/storagebox.pub uXXXXXX@uXXXXXX.your-storagebox.de

I should now be able to interact via sftp without providing a separate password. I can test that it works with:

$ sftp storagebox

Installing and configuring `restic`

I installed restic via homebrew with brew install restic. Next, I create a restic password that restic uses to encrypt my data locally before it is uploaded. To not forget it, I stored it in my password manager and also wrote it down in my notes. I also store this password in my Apple Keychain with:

$ security add-generic-password -a "$USER" -s restic-password -w

I then configured two important environment variables in a new restic config file ~/.config/restic/env:

export RESTIC_REPOSITORY="sftp:storagebox:/home/restic-backup"
export RESTIC_PASSWORD_COMMAND="security find-generic-password -a yourusername -s restic-password -w"

RESTIC_REPOSITORY is the director on the storage box where the backups are stored. This is called the restic repository. RESTIC_PASSWORD_COMMAND tells restic which command to run to get the restic password from the Apple Keychain (all of this avoids storing it in plain text in an environment variable).

Next I created the restic repository on the storage box with:

$ restic -r sftp:storagebox:/home/restic-backup init

This is all that is needed for the initial configuration. I can now start making backups.

Basic restic backup commands

To backup the folder ~/important_docs/ to the restic repository on the storage box I run:

$ restic backup ~/important_docs --verbose

Restic calls the content of a directory at a specific point in time a snapshot. This snapshot is now available in the repository and I can restore the whole snapshot or selected files of the directory back to my local machine.

If I run the above backup command again, restic will create a second snapshot, but not upload any files as none have changed. This deduplication makes sure that the data is stored efficiently.

Inspecting the storage box

To inspect the files in my storage box I use Cyberduck . Create a new SFTP connection with the storage box details:

Server: uXXXXXX.your-storagebox.de
Port: 22
Username: uXXXXXX
SSH Private Key: ~/.ssh/storagebox

This opens the file viewer:

Figure 1. Restic repository files visible in my storage box.

I can see that (encrypted) restic files have been uploaded to the subfolder restic-backup. Of course I can also store other things in the storage box, e.g. I have a separate sync folder that I use with rsync.

Working with snapshots

Here are useful commands to inspect and manage snapshots taken in the past:

To list all available snapshots in the repository: restic snapshots.
To list all files in a specific snapshot with id 073a90db: restic ls 073a90db.
To view the size of the restic repository, run restic stats --mode raw-data.

Restoring backups

It is worth understanding how to restore a backed-up directory. Let’s say I want to restore snapshot a798b76e. If I run

$ restic restore a798b76e --target ~/restore-test

restic will reproduce all directories and files that are part of that snapshot in the provided target folder restore-test. If I just want to restore a single file from a snapshot I can use the --include flag, e.g.

$ restic restore a798b76e --include photo_of_grandma.jpg --target ~/restore-test

This will restore photo_of_grandma.jpg from snapshot a798b76e and make it available in the folder ~/restore-test. Important: To restore you will implicitely or explicitely need my restic password. Make sure you have it available.

Managing the repository’s size and integrity

Two additional commands help me manage the repository and its size. To make sure that the data on the storage box hasn’t been corrupted I periodically run:

$ restic check

This performs a structural integrity check of the data in the repository. Do the available files and the index match? Are there files not in the index? Check that it can decrypt snapshot metadata. This check is fast because it doesn’t actually read any data. If I want to also check that files are correct I need to append --read-data for a full repository read. This will likely take some time. Alternatively I can only check a subset of data with --read-data-subset=5% or --read-data-subset=500M.

The other important command is forget. My storage box only has a size of 1TB, so if I keep accumulating snapshots, I will eventually run out of space. forget and prune can be used to delete snapshots.

To manually delete a snapshot with id a798b76e I run:

$ restic forget a798b76e

This only deletes the snapshot. It doesn’t actually delete the data that was exclusive to that snapshot (not part of another snapshot). To delete this data I need to follow up with prune.

$ restic prune

or do it as part of the forget command:

$ restic forget a798b76e --prune

The restic docs recommend running a restic check after pruning operations to flag any issues.

Instead of deleting a specific snapshot, I want to specify a deletion policy, e.g. “keep one backup for the last 6 month, one for the last 4 weeks, and daily for the current week.” Restic uses the --keep-* flag to specify such a policy. Use the --dry-run flag when trying this out the first time. Some example policies:

--keep-last n keep the n last (most recent) snapshots.
--keep-daily n for the last n days which have one or more snapshots, keep only the most recent one for each day.
--keep-weekly n for the last n weeks which have one or more snapshots, keep only the most recent one for each week.

These can be combined (union of snapshots). So for my desired policy I could use:

$ restic forget --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --prune --verbose

Note that one snapshot could fulfil multiple conditions.

Scheduling and profiles with `resticprofile`

So far I only have a set of CLI commands that I would have to remember to run periodically. To actually make restic work, I want scheduling and profiles.

Scheduling to reliably run the backup daily and profiles to have different command configurations for different backup sources, e.g. I want daily backups of core folders on my MacBook and only weekly backups for my external hard drive photos1.

I am using resticprofile for both which I install via homebrew

$ brew tap creativeprojects/tap
$ brew install resticprofile

With resticprofile I can use a yml or toml configuration file to specify commands and configurations for specific paths. I created two profiles here ~/.config/resticprofile/profiles.yml:

version: "1"

# default profile set up for files on my MacBook
default:
  repository: "sftp:storagebox:/home/restic-backup"
  password-command: "security find-generic-password -a yourusername -s restic-password -w"
  initialize: false 
  force-inactive-lock: true
  retry-lock: 10m
  
  backup:
    source:
      - "~/core_data_from_my_macbook"  
    exclude:
      - "~/stuff_I_want_to_exclude/"
    exclude-file:
      - "~/.config/restic/excludes.txt"
    verbose: true
    schedule: "11:00"
  forget:
    # path: true reuses the source list from backup command
    path: true
    keep-daily: 7
    keep-weekly: 4
    keep-monthly: 6
    prune: true
    schedule: "12:00"
  check:
    read-data-subset: "5%"
    schedule: "Mon 12:30"

# for `photos1` external hard drive
photos:
  repository: "sftp:storagebox:/home/restic-backup"
  password-command: "security find-generic-password -a yourusername -s restic-password -w"
  initialize: false 
  backup:
    source:
      - "/Volumes/photos1/Photos"
  forget:
    path: true
    prune: true
    keep-last: 5

The first profile default configures my MacBook backups. It specifies which folders to backup and is configured to run daily at 11am. At 12pm restic executes my forget policy on the specified paths and at 12:30 restic runs an integrity check.

I have set up a second profile photos to configure backups from my external hard drive to the same remote restic repository. The backup command is not scheduled. I only run it after I have copied new photos onto the harddrive, which only really happens every few weeks. The forget policy ensures the last 5 snapshots are kept.

To schedule this configuration, I run resticprofile schedule. On macOS this schedules the jobs in the launchd service manager. The first time this runs, macOS will ask you to confirm permissions. I verified the schedule is set with launchctl list | grep -i restic. Moreover, I can show all configured profiles with

$ resticprofile profiles

I run the backup command for the photos profile manually with

$ resticprofile photos.backup

Conclusion

My final backup approach is show in Figure 2. My MacBook is backed up daily to the remote restic repository on the Hetzner Storage Box and weekly to an external hard drive. My photos hard drive is backed up weekly to the restic repository and monthly to a second external hard drive.

Figure 2. Final backup approach to restic cloud repository and external hard drives.

I have been running this setup for a couple of weeks and so far I am happy with it. The important files on my MacBook are about 15GB which took 20–30min for the first uploaded snapshot and only a few seconds since. The photos1 hard drive has about 250GB of data which took 4–5h for the initial upload.

It’s also nice that I can change storage vendor seamlessly, e.g. I can switch to AWS S3 or Backblaze B2 by modifying the config file and setting up new SSH keys.

Appendix

Launchd schedules with sleeping MacBook (closed lid)

I noticed that some scheduled backups failed when my MacBook was asleep (lid closed). I think that it’s because launchd resumes scheduled jobs once the MacBook awakes but a network connection isn’t always immediately available. I fixed this by checking for a network connection befor running the backup command using the run-before setting. I added the following line for backup forget and check in my profiles.yml:

# [...]
backup:
    # before running backup try 30x to reach the storage box, this should give time for network access to be available after sleep
    run-before: 
      - "for i in $(seq 1 30); do nc -z -G 5 uXXXXXX.your-storagebox.de 23 && exit 0; sleep 10; done; exit 1"
# [...]

Locks

Certain restic commands put a lock onto the repository to prevent multiple clients from working on the same files. I can list all locks with

$ restic list locks

Sometimes failed commands leave the repository locked. If you are sure it’s safe to unlock it, run:

$ restic unlock

A common failure mode is for example for the check command to run and fail and leave an exclusive lock behind on the repository, which then makes subsequent backups fail. To avoid this issue I added the following settings:

# [...]
default:
  repository: "sftp:storagebox:/home/restic-backup"
  # remove stale restic locks older than 1h
  force-inactive-lock: true
  # when restic finds a blocking lock, try locking again in 10min
  retry-lock: 10m
# [...]

Exclusion file

My profile uses an exclusion file ~/.config/restic/excludes.txt to ignore certain types of files I don’t want to back up, e.g. build artefacts or temporary files:

# Python
.venv
venv
__pycache__
*.pyc
.pytest_cache
.mypy_cache
.ruff_cache
.tox
*.egg-info

# Node / JS
node_modules
.next
.nuxt
.parcel-cache

# Editors & tooling
.idea
.vscode
*.swp
*~

# Caches & logs
.cache
*.log
*.tmp
.terraform

# OS junk
.DS_Store
._*
.Spotlight-V100
.fseventsd
.Trashes
.TemporaryItems
.DocumentRevisions-V100

Additionally I use the flag exclude-caches: true, which ignores all folders that have a CACHEDIR.TAG file, e.g. Rust cache directories.

Visualising London rental bike location data

Wed, 15 Jul 2026 12:00:00 +0000

London has a great rental bike scheme that makes it easy to get around the city. Rather than free-floating you can rent bikes from fixed bike stations like the one shown below. If you get the yearly membership you even get a key that you can use to rent the bike without an app.

Image(filename='map_images/station_photo.jpg', width='400px')

Moreover, Transport for London makes the live rental station data available on this endpoint . I collected hourly snapshots of bike availability data over a month so I can play around with it with DuckDB’s new spatial features and QGIS.

Collecting the data

I created a simple script fetch_bike_data.sh and a cronjob on a Hetzner VPS to request and store the data every hour. Here is the script:

#!/bin/bash
DATE=\\((date +"%Y.%m.%d.%H.%M.%S")
DIR=/home/projects/bike\_data

mkdir -p \\)DIR
curl -o santander.$DATE.xml https://tfl.gov.uk/tfl/syndication/feeds/cycle-hire/livecyclehireupdates.xml

After adding permissions to the script with chmod +x fetch_bike_data.sh I modify crontab:

crontab -e

and add the following line to run this script:

0 * * * * /home/projects/bike_data/fetch_bike_data.sh

Each request is stored to an XML-file on disk which has the following format:

<stations lastUpdate="1784031061002" version="2.0">
    <station>
        <id>1</id>
        <name>River Street , Clerkenwell</name>
        <terminalName>001023</terminalName>
        <lat>51.52916347</lat>
        <long>-0.109970527</long>
        <installed>true</installed>
        <locked>false</locked>
        <installDate>1278947280000</installDate>
        <removalDate/>
        <temporary>false</temporary>
        <nbBikes>0</nbBikes>
        <nbStandardBikes>0</nbStandardBikes>
        <nbEBikes>0</nbEBikes>
        <nbEmptyDocks>19</nbEmptyDocks>
        <nbDocks>19</nbDocks>
    </station>
    <station>
        <id>2</id>
        <name>Phillimore Gardens, Kensington</name>
        <terminalName>001018</terminalName>
        <lat>51.49960695</lat>
        <long>-0.197574246</long>
        <installed>true</installed>
        <locked>false</locked>
        <installDate>1278585780000</installDate>
        <removalDate/>
        <temporary>false</temporary>
        <nbBikes>6</nbBikes>
        <nbStandardBikes>6</nbStandardBikes>
        <nbEBikes>0</nbEBikes>
        <nbEmptyDocks>28</nbEmptyDocks>
        <nbDocks>37</nbDocks>
    </station>
[...]

As you can see the raw data contains per-station information about its name, location, and bike availability.

Convert to DuckDB table

I load and combine all the XML-files into a DuckDB database london_bike_data so that I can use SQL to analyse it. Before I do that I install the extensions spatial and h3 for the geospatial features and webbed to read xml files.

import duckdb
from IPython.display import Image

# Connect to DuckDB
conn = duckdb.connect('london_bike_data.duckdb')

conn.execute("INSTALL spatial;")
conn.execute("INSTALL h3 FROM community;")
conn.execute("INSTALL webbed FROM community;")
conn.execute("LOAD spatial;")
conn.execute("LOAD h3;")
conn.execute("LOAD webbed;");

The next query loads all collected xml-files into one DuckDB table and enforces the schema, adds metadata, and handles missing data issues. Note the Hetzner instance was based in Helsinki which I account for when converting the filename into a timestamp with the correct time zone.

conn.execute("""
    CREATE OR REPLACE TABLE londonbikestations AS
     SELECT
            strptime(
            regexp_extract(filename, 'santander\\.(\\d{4}\\.\\d{2}\\.\\d{2}\\.\\d{2}\\.\\d{2}\\.\\d{2})\\.xml', 1),
            '%Y.%m.%d.%H.%M.%S'
        ) AT TIME ZONE 'Europe/Helsinki' AT TIME ZONE 'Europe/London' AS downloaded_at,
                 id::INTEGER AS id,
                 name,
                 terminalName AS terminal_name,
                 lat::DOUBLE AS lat,
                 long::DOUBLE AS lng,
                 installed::BOOLEAN AS installed,
                 locked::BOOLEAN AS locked,
                 temporary::BOOLEAN AS temporary,
                 nbBikes::INTEGER AS nb_bikes,
                 nbStandardBikes::INTEGER AS nb_standard_bikes,
                 nbEBikes::INTEGER AS nb_ebikes,
                 nbEmptyDocks::INTEGER AS nb_empty_docks,
                 nbDocks::INTEGER AS nb_docks,
                epoch_ms(installDate::BIGINT) AS install_date,
                CASE WHEN removalDate = '' THEN NULL ELSE epoch_ms(removalDate::BIGINT) END AS removal_date
             FROM read_xml('data/santander.*.xml', all_varchar=true, filename=true)
""")
conn.sql("select count(*) from londonbikestations")

┌──────────────┐
│ count_star() │
│    int64     │
├──────────────┤
│       707197 │
└──────────────┘

The query below shows the station data that is available. The table contains the station id, name and coordinates (lat, lng) and when it was installed and/or removed. For every station I have the capacity nb_docks and the current number of rentable bikes nb_bikes (split into standard and ebikes).

conn.sql("describe table londonbikestations")

┌───────────────────┬─────────────┬─────────┬─────────┬─────────┬─────────┐
│    column_name    │ column_type │  null   │   key   │ default │  extra  │
│      varchar      │   varchar   │ varchar │ varchar │ varchar │ varchar │
├───────────────────┼─────────────┼─────────┼─────────┼─────────┼─────────┤
│ downloaded_at     │ TIMESTAMP   │ YES     │ NULL    │ NULL    │ NULL    │
│ id                │ INTEGER     │ YES     │ NULL    │ NULL    │ NULL    │
│ name              │ VARCHAR     │ YES     │ NULL    │ NULL    │ NULL    │
│ terminal_name     │ VARCHAR     │ YES     │ NULL    │ NULL    │ NULL    │
│ lat               │ DOUBLE      │ YES     │ NULL    │ NULL    │ NULL    │
│ lng               │ DOUBLE      │ YES     │ NULL    │ NULL    │ NULL    │
│ installed         │ BOOLEAN     │ YES     │ NULL    │ NULL    │ NULL    │
│ locked            │ BOOLEAN     │ YES     │ NULL    │ NULL    │ NULL    │
│ temporary         │ BOOLEAN     │ YES     │ NULL    │ NULL    │ NULL    │
│ nb_bikes          │ INTEGER     │ YES     │ NULL    │ NULL    │ NULL    │
│ nb_standard_bikes │ INTEGER     │ YES     │ NULL    │ NULL    │ NULL    │
│ nb_ebikes         │ INTEGER     │ YES     │ NULL    │ NULL    │ NULL    │
│ nb_empty_docks    │ INTEGER     │ YES     │ NULL    │ NULL    │ NULL    │
│ nb_docks          │ INTEGER     │ YES     │ NULL    │ NULL    │ NULL    │
│ install_date      │ TIMESTAMP   │ YES     │ NULL    │ NULL    │ NULL    │
│ removal_date      │ TIMESTAMP   │ YES     │ NULL    │ NULL    │ NULL    │
└───────────────────┴─────────────┴─────────┴─────────┴─────────┴─────────┘
  16 rows                                                       6 columns

The removal_date column seems not trustworthy since there is definitely activity (changes in nb_bikes) happening at stations that have a removal date, e.g.

conn.sql("select downloaded_at, name, terminal_name, nb_bikes, removal_date from londonbikestations where terminal_name = '000968' order by downloaded_at")

┌─────────────────────┬──────────────────────────┬───────────────┬──────────┬─────────────────────┐
│    downloaded_at    │           name           │ terminal_name │ nb_bikes │    removal_date     │
│      timestamp      │         varchar          │    varchar    │  int32   │      timestamp      │
├─────────────────────┼──────────────────────────┼───────────────┼──────────┼─────────────────────┤
│ 2026-05-08 22:00:01 │ Warwick Row, Westminster │ 000968        │        6 │ 2018-10-18 06:56:00 │
│ 2026-05-08 23:00:01 │ Warwick Row, Westminster │ 000968        │        6 │ 2018-10-18 06:56:00 │
│ 2026-05-09 00:00:01 │ Warwick Row, Westminster │ 000968        │        6 │ 2018-10-18 06:56:00 │
│ 2026-05-09 01:00:01 │ Warwick Row, Westminster │ 000968        │        6 │ 2018-10-18 06:56:00 │
│ 2026-05-09 02:00:01 │ Warwick Row, Westminster │ 000968        │        6 │ 2018-10-18 06:56:00 │
│ 2026-05-09 03:00:01 │ Warwick Row, Westminster │ 000968        │        6 │ 2018-10-18 06:56:00 │
│ 2026-05-09 04:00:01 │ Warwick Row, Westminster │ 000968        │        6 │ 2018-10-18 06:56:00 │
│ 2026-05-09 05:00:01 │ Warwick Row, Westminster │ 000968        │        6 │ 2018-10-18 06:56:00 │
│ 2026-05-09 06:00:01 │ Warwick Row, Westminster │ 000968        │        6 │ 2018-10-18 06:56:00 │
│ 2026-05-09 07:00:01 │ Warwick Row, Westminster │ 000968        │        6 │ 2018-10-18 06:56:00 │
│          ·          │            ·             │   ·           │        · │          ·          │
│          ·          │            ·             │   ·           │        · │          ·          │
│          ·          │            ·             │   ·           │        · │          ·          │
│ 2026-06-14 12:00:01 │ Warwick Row, Westminster │ 000968        │        3 │ 2018-10-18 06:56:00 │
│ 2026-06-14 13:00:01 │ Warwick Row, Westminster │ 000968        │        5 │ 2018-10-18 06:56:00 │
│ 2026-06-14 14:00:01 │ Warwick Row, Westminster │ 000968        │        9 │ 2018-10-18 06:56:00 │
│ 2026-06-14 15:00:01 │ Warwick Row, Westminster │ 000968        │        4 │ 2018-10-18 06:56:00 │
│ 2026-06-14 16:00:01 │ Warwick Row, Westminster │ 000968        │        2 │ 2018-10-18 06:56:00 │
│ 2026-06-14 17:00:01 │ Warwick Row, Westminster │ 000968        │        7 │ 2018-10-18 06:56:00 │
│ 2026-06-14 18:00:01 │ Warwick Row, Westminster │ 000968        │        9 │ 2018-10-18 06:56:00 │
│ 2026-06-14 19:00:01 │ Warwick Row, Westminster │ 000968        │       10 │ 2018-10-18 06:56:00 │
│ 2026-06-14 20:00:01 │ Warwick Row, Westminster │ 000968        │       11 │ 2018-10-18 06:56:00 │
│ 2026-06-14 21:00:01 │ Warwick Row, Westminster │ 000968        │        8 │ 2018-10-18 06:56:00 │
└─────────────────────┴──────────────────────────┴───────────────┴──────────┴─────────────────────┘
  887 rows (20 shown)                                                                   5 columns

I have confirmed that this station still exists (here ) so I will just ignore this column.

I want to get an overview of the total capacity across all stations. The exact number fluctuates slightly between snapshots but the capacity seems to be around 20.9k. The maximum number of docked bikes was 7708 standard bikes and 1131 eBikes. The official numbers given by TfL are 10,000 standard bikes and 2,000 eBikes.

conn.sql("select downloaded_at, sum(nb_standard_bikes) as sum_docked_standard_bikes, sum(nb_ebikes) as sum_docked_ebikes, sum(nb_docks) as sum_station_capacity from londonbikestations group by downloaded_at order by sum_docked_standard_bikes")

┌─────────────────────┬───────────────────────────┬───────────────────┬──────────────────────┐
│    downloaded_at    │ sum_docked_standard_bikes │ sum_docked_ebikes │ sum_station_capacity │
│      timestamp      │          int128           │      int128       │        int128        │
├─────────────────────┼───────────────────────────┼───────────────────┼──────────────────────┤
│ 2026-06-04 08:00:01 │                      6008 │               795 │                20963 │
│ 2026-06-01 08:00:01 │                      6047 │               585 │                20934 │
│ 2026-06-03 08:00:01 │                      6056 │               948 │                20951 │
│ 2026-06-02 08:00:01 │                      6082 │               927 │                20921 │
│ 2026-06-04 17:00:01 │                      6122 │              1185 │                20963 │
│ 2026-06-09 08:00:02 │                      6127 │               830 │                20962 │
│ 2026-05-28 08:00:01 │                      6191 │               692 │                20950 │
│ 2026-06-01 17:00:01 │                      6201 │              1086 │                20892 │
│ 2026-05-27 17:00:01 │                      6203 │               829 │                20950 │
│ 2026-06-04 10:00:01 │                      6210 │               983 │                20963 │
│          ·          │                        ·  │                ·  │                  ·   │
│          ·          │                        ·  │                ·  │                  ·   │
│          ·          │                        ·  │                ·  │                  ·   │
│ 2026-05-16 06:00:01 │                      7668 │              1081 │                20969 │
│ 2026-05-15 01:00:01 │                      7671 │              1182 │                20949 │
│ 2026-05-15 03:00:01 │                      7674 │              1169 │                20949 │
│ 2026-05-16 01:00:01 │                      7678 │              1172 │                20942 │
│ 2026-05-15 02:00:01 │                      7678 │              1178 │                20949 │
│ 2026-05-16 00:00:01 │                      7679 │              1186 │                20942 │
│ 2026-05-16 02:00:01 │                      7699 │              1157 │                20942 │
│ 2026-05-16 05:00:01 │                      7703 │              1106 │                20969 │
│ 2026-05-16 03:00:01 │                      7705 │              1143 │                20942 │
│ 2026-05-16 04:00:01 │                      7708 │              1131 │                20942 │
└─────────────────────┴───────────────────────────┴───────────────────┴──────────────────────┘
  887 rows (20 shown)                                                              4 columns

The largest station is Jubilee Plaza in Canary Wharf.

conn.sql("select name, nb_bikes, nb_docks from londonbikestations order by nb_docks desc limit 1")

┌─────────────────────────────┬──────────┬──────────┐
│            name             │ nb_bikes │ nb_docks │
│           varchar           │  int32   │  int32   │
├─────────────────────────────┼──────────┼──────────┤
│ Jubilee Plaza, Canary Wharf │       42 │       63 │
└─────────────────────────────┴──────────┴──────────┘

Busiest stations

To approximate the busiest stations I sum the absolute hour-to-hour change in nb_bikes per station. This approximates total churn (bikes leaving and arriving) since simultaneous arrivals and departures within the same hour cancel out.

conn.execute("""
    CREATE OR REPLACE TABLE busy_stations AS   
    WITH hourly_changes AS (
        SELECT
            terminal_name,
            ST_POINT(lng, lat) AS geometry,
            name,
            downloaded_at,
            nb_bikes,
            abs(nb_bikes - lag(nb_bikes) OVER w) AS abs_change
        FROM londonbikestations
        WINDOW w AS (PARTITION BY terminal_name ORDER BY downloaded_at)
    )
    SELECT
        terminal_name,
        geometry,
        any_value(name) AS name,
        sum(abs_change) AS total_churn,
        count(*) AS nb_snapshots
    FROM hourly_changes
    GROUP BY terminal_name, geometry
    ORDER BY total_churn DESC
""")
conn.sql("""select name, total_churn from busy_stations limit 20""")

┌──────────────────────────────────────────┬─────────────┐
│                   name                   │ total_churn │
│                 varchar                  │   int128    │
├──────────────────────────────────────────┼─────────────┤
│ Hop Exchange, The Borough                │        3218 │
│ Soho Square , Soho                       │        3160 │
│ Argyle Street, Kings Cross               │        2930 │
│ Waterloo Station 2, Waterloo             │        2753 │
│ Hyde Park Corner, Hyde Park              │        2605 │
│ Waterloo Station 1, Waterloo             │        2554 │
│ Moorfields, Moorgate                     │        2527 │
│ Brushfield Street, Liverpool Street      │        2514 │
│ Waterloo Station 3, Waterloo             │        2389 │
│ St. James's Square, St. James's          │        2360 │
│ Jubilee Plaza, Canary Wharf              │        2289 │
│ Eagle Wharf Road, Hoxton                 │        2263 │
│ Cheapside, Bank                          │        2202 │
│ Imperial College, Knightsbridge          │        2181 │
│ Worship Street, Shoreditch               │        2167 │
│ Natural History Museum, South Kensington │        2142 │
│ Whitehall Place, Strand                  │        2042 │
│ Queen's Gate (North), Kensington         │        2019 │
│ Malet Street, Bloomsbury                 │        2013 │
│ Exhibition Road, Knightsbridge           │        1987 │
└──────────────────────────────────────────┴─────────────┘
  20 rows                                      2 columns

None of these stations are surprising. As we can see later on the map all of these stations are in the city centre or close to commuter train stations.

Busiest hours of the day

Using the same total-churn metric I group the data by hour of day to see when bike activity peaks.

conn.sql("""
    WITH hourly_changes AS (
        SELECT
            terminal_name,
            downloaded_at,
            nb_bikes,
            abs(nb_bikes - lag(nb_bikes) OVER w) AS abs_change
        FROM londonbikestations
        WINDOW w AS (PARTITION BY terminal_name ORDER BY downloaded_at)
    )
    SELECT
        hour(downloaded_at) AS hour_of_day,
        sum(abs_change) AS total_churn,
        count(*) AS nb_snapshots
    FROM hourly_changes
    GROUP BY hour_of_day
    ORDER BY total_churn
""")

┌─────────────┬─────────────┬──────────────┐
│ hour_of_day │ total_churn │ nb_snapshots │
│    int64    │   int128    │    int64     │
├─────────────┼─────────────┼──────────────┤
│           3 │        3911 │        29501 │
│           4 │        4111 │        29501 │
│           2 │        5398 │        29501 │
│           5 │        7441 │        29502 │
│           1 │        8518 │        29501 │
│           0 │       13462 │        29501 │
│           6 │       20333 │        29502 │
│          23 │       21172 │        29500 │
│          22 │       29554 │        29500 │
│          21 │       33808 │        28702 │
│           · │         ·   │          ·   │
│           · │         ·   │          ·   │
│           · │         ·   │          ·   │
│          13 │       40336 │        29497 │
│          20 │       40433 │        29499 │
│          10 │       41575 │        29495 │
│           7 │       45237 │        29503 │
│          19 │       47598 │        28707 │
│          16 │       54813 │        29498 │
│           9 │       57626 │        29498 │
│          17 │       64213 │        29498 │
│          18 │       64873 │        30300 │
│           8 │       78166 │        29503 │
└─────────────┴─────────────┴──────────────┘
  24 rows (20 shown)             3 columns

We see the highest activity from 7:00-8:00, 8:00-9:00, 16:00-17:00, 17:00-18:00 which are peak commuter hours and the lowest activity in the early morning.

Storing geometric data as parquet files

Before I can use QGIS for map-based visualisations I need to store the relevant data in a compatible format.

I want to show the number of bikes available across the city at different times of the day (I picked Wednesday 27.5.2026). To aggregate the number of available bikes spatially, I group by H3 cells . If the coordinates of the bike station fall into the cell I sum all available bikes.

conn.execute("""
    CREATE OR REPLACE TABLE bike_station_geo AS
        SELECT *, ST_POINT(lng, lat) AS geometry FROM londonbikestations where downloaded_at::date = '2026-05-27'   
""")
conn.execute("""
    CREATE OR REPLACE TABLE h3_cell_covering AS
        SELECT
            downloaded_at,
            H3_LATLNG_TO_CELL(
                ST_Y(geometry),
                ST_X(geometry),
                7
            ) AS hexagon,
            sum(nb_bikes) AS nb_bikes
        FROM bike_station_geo
        GROUP BY 1, 2
""");

I then store the h3 bike counts table as a parquet file. This involves conversion of the hexagon data into WKB(well-known-binary) that GeoParquet uses to store geometries. Note: By default the geometry in DuckDB is assumed to be EPSG:4326.

conn.execute("""
COPY (
      SELECT geometry: ST_ASWKB(H3_CELL_TO_BOUNDARY_WKT(hexagon)::geometry),
              date_trunc('hour', downloaded_at) AS downloaded_at,
             nb_bikes
      FROM   h3_cell_covering
  ) TO 'available_bikes_per_h3_7_cell.parquet';
  """);

I also want to visualise the location of the busy stations on the map, so I store the busy stations as a separate file.

conn.execute("""
COPY (
      SELECT *  FROM busy_stations
  ) TO 'busy_stations.parquet';
  """);

Visualising availability using QGIS

QGIS (3.44.10) is an open-source geographic information system. It allows combining, viewing, and analysing tabular data with geographic and mapping data.

I want to display our parquet data over an actual map of London. To get this map, I install the plugin NextGIS MapServices and then restart QGIS to make it available. I use OpenStreetMap as the base map. In the toolbar click on QuickMapService(Globe icon) and add the OpenstreetMap OSM Standard. Next, in Project Properties (Cmd+Shift+P)-> CRS change the Projection or Coordinate Reference System to EPSG:3857 (Pseudo-Mercator).

Visualising busy stations

Drag and drop the busy_stations.parquet file into the Layers field (bottom left). Ensure under Layer Properties –> Assigned Coordinate Reference System that EPSG:4326 - WGS 84 is selected. This allows QGIS to correctly reproject the geography data from the file onto the map.

Under Layer Properties –> Symbology I can change the style of the location marker to show all the stations. You can see all the coordinates of stations in the dataset below.

Image(filename='map_images/station_locations.jpg')

To only show the top 20 busiest stations as calculated in the previous section I add a filter on the busy stations layer. (Right click on layer –> Filter, then add total_churn > 1900). I also want to show the station names, so I go to Layer Properties –> Label –> Single labels and select for value the Expression Builder and then use the expression: "name" || ' (' || to_string("total_churn") || ')' to concatenate the churn value to the station name in brackets.

Image(filename='map_images/busy_stations.jpg')

Unsurprisingly the busiest stations are in the centre where many Londoners work (Bank, Canary Wharf, Soho, Westminster) or close to major train stations (Liverpool Street, King’s Cross, Waterloo, Hop Exchange).

Rental bike availability during commute hours

I also want to visualise how the rental bikes move before and after morning rush hour. My expectation is that fewer bikes are initially in the city centre (zone 1) and more bikes in the outer residential areas (zones 2 and 3). Then around 11am we should see a concentration of bikes in zone 1.

I drag and drop the relevant data file available_bikes_per_h3_7_cell.parquet to create a new Layer. Again, I first ensure that the projection for this dataset is set to EPSG:4326 - WGS 84. The hexagons should now appear on top of the map. Remember that the file has availability data for every hour of the day, so I add a Layer filter: "downloaded_at" = '2026-05-27T06:00:00.000' to only use data from one point in time before the start of the rush hour.

To style the hex fields according to the number of available bikes in the hexagon area I make the following changes in the Layer properties:

Symbology –> Graduated –> Value: nb_bikes, Color Ramp: RdYlGr, Mode: Equal Interval, Opacity: 60%
Labels –> Single Labels –> Value: nb_bikes, 18Pt, Bold.

I have filtered the layer once for 6am and once for 11am and here are the two outputs:

display(Image(filename='map_images/bike_availibility_6am.jpg'))

display(Image(filename='map_images/bike_availibility_11am.jpg'))

As expected you can see that a lot of the residential districts along the water, West Kensington, Fulham, Wandsworth, Battersea, Elephant and Castle and in the east around Limehouse, Mile End, Bethnal Green see an outflow of available bikes. The central districts around the City of London, Mayfair, King’s Cross and Westminster accumulate a lot of the bikes by 11am.

Conclusion

It turns out to be pretty easy to collect, explore and visualise the London bike station data. DuckDB makes it really easy to ingest hundreds of XML files and the H3 and spatial extensions allow easy geographical aggregation. This is my first time using QGIS. It took 20min to get up to speed with it, but it’s really cool how it fuses mapping data and geographical and analytical data. I will try to use it in other projects as well.

References

I was inspired by Mark Litwintschik’s ebike fleet monitoring blog post
A useful QGIS Tutorial is available here

Jupyter Notebook

You can find the Jupyter notebook and the datasets for this post here .

UK house price changes in real terms

Sun, 11 Jan 2026 00:00:00 +0000

This post is a response to a newspaper article I read this morning. Every month the FT is reporting on how the housing market in the UK is performing as people love to talk about it:

UK house prices rise less than expected in 2025 as growth slows

UK house prices rose by 0.6 per cent in 2025 after a slowdown at the end of the year, according to lender Nationwide. […] Prices fell 0.4 per cent between November and December to an average of £271,068. Both figures were below analysts’ expectations of a 1.2 per cent annual rise and a 0.1 per cent month-on-month expansion. FT Weekend (02.01.2026)

I am asking myself how useful this statistic is for the average reader, especially for a reader who wants to buy or sell a property.

Two issues are immediately obvious:

The average yearly house price change across all of the UK might not agree with the house price performance in the areas I live in or I want to move to. There are also differences by house type, e.g. flat vs. detached house vs. terraced house. Different types of property might even perform in opposite directions.
British people often view property, at least partially, as an investment opportunity. However, in those cases it would help to at least report the real (inflation-adjusted) house price change, not just the nominal change. An increase of 0.6% in nominal house prices sounds different when one doesn’t mention that inflation in 2025 was around 3.6% (UK CPI). Meanwhile other forms of investments did quite well. The FTSE 100 nominally returned 21.4% (which to be fair was a positive outlier this year compared to an average of 6% over the last 10 years).

Fortunately, it is not that difficult to compute real house prices more closely to my situation. The raw data is openly available via API:

The HM Land Registry registers the ownership of property in the UK and provides aggregate monthly data of the average transactions by region and property type.
The Office for National Statistics (ONS) provides monthly inflation data (CPI) for the UK.

So, in this post I want to compute and visualise how real flat and detached house prices in specific areas performed both nominally and in real terms.

Querying the housing price data

I am using the UK House Price Index of the Land Registry API:

http://landregistry.data.gov.uk/data/ukhpi/region

This allows me to query the monthly average house price value for different regions and property types. Important to note here is that this data is based on the successful property transactions in a region which we use here as an approximation of the average value of similar properties in the region (that were or weren’t sold).

I am based in London and interested in how the housing market performs close to me or areas that I am interested in living in. So I will query the data for all London boroughs as well as some South England regions around London: Kent, Surrey, Buckinghamshire, and Oxfordshire.

I then resample the monthly data to quarterly data by taking the mean across the 3 months of a quarter.

I then store all the data in parquet files. You can find them here .

Inspecting the data

Let’s have a look at how the data is stored in the file:

import pandas as pd
import warnings
import matplotlib.pyplot as plt
pd.set_option('display.float_format', lambda x: '%.2f' % x)
warnings.simplefilter('always', category=UserWarning)

Here I am loading the file for the different counties that I am interested in:

df_regions = pd.read_parquet('./data/uk_house_prices.parquet')
df_regions['is_london_borough'] = False
df_regions.sample(5,ignore_index=True)

	Quarter	Region	Price	PropertyType	is_london_borough
0	2023Q4	Surrey	517233.67	All	False
1	2011Q3	Kent	192827.67	All	False
2	2014Q2	Oxfordshire	277195.00	Semi_detached	False
3	2013Q4	London	415838.00	Semi_detached	False
4	2021Q3	Buckinghamshire	431147.67	All	False

and here I am loading the file for the London boroughs:

df_boroughs = pd.read_parquet('./data/london_borough_prices.parquet')
df_boroughs['is_london_borough'] = True
df_boroughs.sample(5,ignore_index=True)

	Quarter	Region	Price	PropertyType	is_london_borough
0	2020Q4	Hillingdon	406610.67	Terraced	True
1	2024Q4	Redbridge	490087.67	All	True
2	2017Q2	Lewisham	910228.33	Detached	True
3	2011Q3	Islington	369223.67	Flat	True
4	2010Q3	Enfield	661067.33	Detached	True

You can see that the dataset shows the average quarterly housing transaction values by region. The column PropertyType shows what type of property was sold (Flat, Terraced, Detached, Semi_detached).

Computing real house prices

Next we adjust the nominal house prices from the Land Registry for inflation and compute the real house prices in terms of “2025 pounds”. I have queried the Office for National Statistics API to get the UK’s quarterly CPI data and stored it in a parquet file. I am loading this data here:

cpi_df = pd.read_parquet('./data/cpi_data.parquet')
cpi_df.tail(5)

	Quarter	CPI
146	2024Q3	134.10
147	2024Q4	135.20
148	2025Q1	136.00
149	2025Q2	138.50
150	2025Q3	139.20

With the consumer price index, we can compute the real price of housing. This real price answers the question: “How much was the property worth in last quarter’s pounds?” If you experienced inflation, your money is worth less, and you will need to pay more for the same house in today’s money terms. If you experienced deflation, your money is worth more, and you will need to pay less for a house in today’s terms (all else equal).

The formula to compute the real house price in terms of the current CPI:

real_price_current = nominal_price_t * (CPI_current / CPI_t)

# Merging the dataframes here to do real price calculation in one step
df = pd.concat([df_regions, df_boroughs])

# Get the most recent CPI value
current_cpi = cpi_df['CPI'].iloc[-1]
latest_quarter = cpi_df['Quarter'].iloc[-1]

# Merge CPI df with house prices df
df = df.merge(cpi_df, on='Quarter', how='left')

# For quarters without CPI data yet, use the most recent CPI
df['CPI'] = df['CPI'].fillna(current_cpi)

# Calculate real prices using above formula 
df['RealPrice'] = df['Price'] * (current_cpi / df['CPI'])

df.sample(5,ignore_index=True)

	Quarter	Region	Price	PropertyType	is_london_borough	CPI	RealPrice
0	2012Q3	City Of Westminster	2138137.67	Semi_detached	True	96.10	3097073.50
1	2011Q3	Croydon	280930.00	Semi_detached	True	93.80	416902.52
2	2016Q3	Bromley	898828.67	Detached	True	100.90	1240009.42
3	2017Q4	Ealing	1160573.67	Detached	True	104.60	1544472.80
4	2025Q2	City Of London	792392.33	Flat	True	138.50	796397.20

As you can see we now have the column RealPrice which gives us the mean house price transaction per quarter and region in real terms. Now we can visualise the data to finally see what happened.

Visualising nominal and real house prices over time

My goal is to plot the nominal and real house prices over time. To keep things visually simpler on the charts I am only selecting a few London boroughs that I am interested in and only look at data since 2015. For London boroughs I am looking at Flats and Terraced houses and for the counties I am only looking at detached houses.

df = df[df['Quarter'] >= '2015Q1'].copy()

SELECTED_BOROUGHS = [
    'Ealing',
    'Hackney',
    'Hammersmith And Fulham',
    'Hounslow',
    'Islington',
    'Richmond Upon Thames',
    'Wandsworth',
]
df_boroughs = df[df['Region'].isin(SELECTED_BOROUGHS) & (df['PropertyType'].isin(['Flat', 'Terraced']))].copy()
df_regions = df[(df['is_london_borough'] == False) & (df['PropertyType'] == 'Detached')].copy()

In the file utils.py I added some code to create the plots and tables. You can safely skip this if not interested. Importing them here in the notebook:

from utils import create_table, plot_prices_by_type

Detached Houses in South England Counties

I am first looking at the house price performance of detached houses in five different English counties. I am plotting both the nominal prices (dashed line) and the real prices (solid line).

I am also creating a table with the real price changes vs. 1 year ago, 2 years ago, 5 years ago, 10 years ago.

fig, axes = plt.subplots(1, 1, figsize=(14, 8))
plot_prices_by_type(df_regions, 'Detached', axes, 'Price', 'Detached Houses in South England counties')

You can see that within the last 10 years most counties saw a decent increase in nominal terms and a decline in real house prices for detached properties. The only exception is Kent with a small real increase of 0.9%. London and Surrey both lost around 10% in real terms.

regions = sorted(df['Region'].unique())
gt_regions_flat = create_table(
        df_regions, 'Detached',
        'South England Counties',
        regions
    )
gt_regions_flat

	2025 (Current)	2024 (1y ago)	2023 (2y ago)	2020 (5y ago)	2015 (10y ago)
South England Counties
Detached properties - Real prices (CPI-adjusted) - Q4 2025 vs 1, 2, 5, 10 years ago
Price (£)	%	Price (£)	%	Price (£)	%	Price (£)	%	Price (£)	%
Buckinghamshire	£853,221	0.0	£883,178	−3.4	£894,115	−4.6	£974,110	−12.4	£924,706	−7.7
Kent	£605,363	0.0	£620,638	−2.5	£633,552	−4.4	£692,871	−12.6	£599,898	0.9
London	£1,132,911	0.0	£1,178,785	−3.9	£1,179,862	−4.0	£1,331,199	−14.9	£1,265,328	−10.5
Oxfordshire	£680,124	0.0	£675,726	0.7	£676,613	0.5	£757,910	−10.3	£723,218	−6.0
Surrey	£975,634	0.0	£1,007,309	−3.1	£1,023,827	−4.7	£1,118,116	−12.7	£1,083,197	−9.9

Flats in London Boroughs

Next, I am looking at flats in London. Again I am plotting nominal and real quarterly prices for selected London boroughs and create a table with the corresponding numbers.

fig, axes = plt.subplots(1, 1, figsize=(14, 8))
plot_prices_by_type(df_boroughs, 'Flat', axes, 'Price', 'Flats in selected London boroughs')

Both the chart and the table show somewhat stable or slightly increasing nominal prices, but drastic reductions in real prices. Both Hammersmith/Fulham and Wandsworth are down about 30% in real terms since 2015. Moreover compared to last year and two years ago we still see price declines in real terms.

regions = sorted(df['Region'].unique())
gt_regions_flat = create_table(
        df_boroughs, 'Flat',
        'Selected London Boroughs',
        regions
    )
gt_regions_flat

	2025 (Current)	2024 (1y ago)	2023 (2y ago)	2020 (5y ago)	2015 (10y ago)
Selected London Boroughs
Flat properties - Real prices (CPI-adjusted) - Q4 2025 vs 1, 2, 5, 10 years ago
Price (£)	%	Price (£)	%	Price (£)	%	Price (£)	%	Price (£)	%
Ealing	£404,408	0.0	£419,269	−3.5	£419,733	−3.7	£489,258	−17.3	£521,670	−22.5
Hackney	£546,932	0.0	£554,931	−1.4	£583,301	−6.2	£685,095	−20.2	£667,622	−18.1
Hammersmith And Fulham	£577,801	0.0	£639,856	−9.7	£707,815	−18.4	£760,674	−24.0	£883,098	−34.6
Hounslow	£367,011	0.0	£375,391	−2.2	£373,890	−1.8	£441,401	−16.9	£419,450	−12.5
Islington	£566,442	0.0	£599,386	−5.5	£623,593	−9.2	£725,791	−22.0	£771,081	−26.5
Richmond Upon Thames	£483,018	0.0	£524,730	−7.9	£520,433	−7.2	£601,897	−19.8	£633,108	−23.7
Wandsworth	£540,034	0.0	£568,240	−5.0	£603,492	−10.5	£715,552	−24.5	£761,426	−29.1

Terraced houses in London Boroughs

Next, I am looking at terraced houses in the same London boroughs. My expectation is that they did better than flats, especially since they were more popular during the COVID pandemic, where people wanted a garden.

fig, axes = plt.subplots(1, 1, figsize=(14, 8))
plot_prices_by_type(df_boroughs, 'Terraced', axes, 'Price', 'Terraced houses in selected London boroughs')

Indeed, terraced houses did better in nominal terms with all of them up. Again in real terms we see price decreases over all considered comparison years (1,2,5,10). However, it is less drastic than for flats in the same locations.

regions = sorted(df['Region'].unique())
gt_regions_flat = create_table(
        df_boroughs, 'Terraced',
        'Selected London Boroughs',
        regions
    )
gt_regions_flat

	2025 (Current)	2024 (1y ago)	2023 (2y ago)	2020 (5y ago)	2015 (10y ago)
Selected London Boroughs
Terraced properties - Real prices (CPI-adjusted) - Q4 2025 vs 1, 2, 5, 10 years ago
Price (£)	%	Price (£)	%	Price (£)	%	Price (£)	%	Price (£)	%
Ealing	£690,513	0.0	£697,849	−1.1	£690,002	0.1	£788,622	−12.4	£800,316	−13.7
Hackney	£972,274	0.0	£966,520	0.6	£1,006,975	−3.4	£1,158,935	−16.1	£1,073,030	−9.4
Hammersmith And Fulham	£1,111,278	0.0	£1,221,472	−9.0	£1,334,009	−16.7	£1,392,109	−20.2	£1,547,731	−28.2
Hounslow	£625,089	0.0	£622,316	0.4	£612,967	2.0	£704,956	−11.3	£637,839	−2.0
Islington	£1,124,856	0.0	£1,175,162	−4.3	£1,214,208	−7.4	£1,380,766	−18.5	£1,380,624	−18.5
Richmond Upon Thames	£890,118	0.0	£935,936	−4.9	£918,218	−3.1	£1,036,584	−14.1	£1,033,227	−13.9
Wandsworth	£965,788	0.0	£1,000,163	−3.4	£1,050,418	−8.1	£1,214,261	−20.5	£1,219,378	−20.8

Winners and Losers in real terms (last 3 years)

Lastly I want to see how the housing market in the different London boroughs compares to each other. I am looking at the prices 3 years ago across all property types. (At first I wanted to use 5 years, but I wanted to stay clear of the COVID years which might introduce short-term anomalies into the result). I am comparing real prices even though for a relative comparison between boroughs nominal prices would be fine as well (they all experienced the same inflation).

selected = df[(df['is_london_borough'] == True) & (df['Quarter'].isin(['2025Q4','2022Q4'])) & (df['PropertyType'] == 'All')]

df_wide = selected.pivot(index=['Region', 'PropertyType'], columns='Quarter', values='RealPrice').reset_index()
df_wide['%_real_price_change'] = (df_wide['2025Q4'] - df_wide['2022Q4']) / df_wide['2022Q4']
df_wide.sort_values('%_real_price_change', ascending=False,ignore_index=True).head(10)

Quarter	Region	PropertyType	2022Q4	2025Q4	%_real_price_change
0	Lewisham	All	529841.12	503542.00	-0.05
1	Havering	All	479610.47	450760.00	-0.06
2	Southwark	All	647219.21	607299.00	-0.06
3	Waltham Forest	All	569427.61	533922.00	-0.06
4	Haringey	All	697080.71	649650.00	-0.07
5	Hackney	All	681376.13	627538.00	-0.08
6	Hounslow	All	588525.22	540799.00	-0.08
7	Sutton	All	493375.92	452726.00	-0.08
8	Hillingdon	All	526316.99	482852.00	-0.08
9	Barking And Dagenham	All	387895.94	354709.00	-0.09

You can see that even the best-performing borough Lewisham saw decreasing real house prices of 5% in the last 3 years. Overall a lot of these top 10 boroughs are further outside of London and generally more affordable areas. Both Hackney and Southwark are more central, but have recently undergone gentrification which might have counteracted the price pressure in other central boroughs.

df_wide.sort_values('%_real_price_change', ascending=False,ignore_index=True).tail(10)

Quarter	Region	PropertyType	2022Q4	2025Q4	%_real_price_change
23	Barnet	All	706765.75	606006.00	-0.14
24	Camden	All	928639.42	792454.00	-0.15
25	Croydon	All	464581.92	393479.00	-0.15
26	Lambeth	All	654875.76	550720.00	-0.16
27	Hammersmith And Fulham	All	883837.60	741308.00	-0.16
28	Wandsworth	All	834257.35	695867.00	-0.17
29	Tower Hamlets	All	595217.88	470209.00	-0.21
30	Kensington And Chelsea	All	1637183.90	1194726.00	-0.27
31	City Of Westminster	All	1273329.16	889935.00	-0.30
32	City Of London	All	993657.65	607399.00	-0.39

Some of the worst-performing boroughs over the last 3 years are the City of London, Westminster, Kensington & Chelsea, and Hammersmith and Fulham, which are all very expensive ‘prestige’ areas.

Conclusion

This analysis shows that a prospective home buyer or seller only gains limited information from a national-level house price statistic. Instead, I showed that there are, unsurprisingly, vast differences in price developments between property types and locations.

Moreover, in real terms almost none of the considered regions saw increasing house prices over the last 10 years (notable exception: Kent).

Appendix

Since someone asked, here are the real price changes compared to 3 years ago for all London boroughs:

df_wide.sort_values('%_real_price_change', ascending=False,ignore_index=True)

Quarter	Region	PropertyType	2022Q4	2025Q4	%_real_price_change
0	Lewisham	All	529841.12	503542.00	-0.05
1	Havering	All	479610.47	450760.00	-0.06
2	Southwark	All	647219.21	607299.00	-0.06
3	Waltham Forest	All	569427.61	533922.00	-0.06
4	Haringey	All	697080.71	649650.00	-0.07
5	Hackney	All	681376.13	627538.00	-0.08
6	Hounslow	All	588525.22	540799.00	-0.08
7	Sutton	All	493375.92	452726.00	-0.08
8	Hillingdon	All	526316.99	482852.00	-0.08
9	Barking And Dagenham	All	387895.94	354709.00	-0.09
10	Merton	All	677772.16	618798.00	-0.09
11	Harrow	All	590633.55	538433.00	-0.09
12	Bromley	All	581127.96	529749.00	-0.09
13	Redbridge	All	524996.77	476764.00	-0.09
14	Richmond Upon Thames	All	870795.06	785086.00	-0.10
15	Enfield	All	533218.77	480183.00	-0.10
16	Bexley	All	456886.19	410002.00	-0.10
17	Islington	All	757742.76	677305.00	-0.11
18	Ealing	All	645069.13	572575.00	-0.11
19	Greenwich	All	533193.50	472599.00	-0.11
20	Brent	All	639480.63	558093.00	-0.13
21	Kingston Upon Thames	All	657636.32	573489.00	-0.13
22	Newham	All	471479.67	405808.00	-0.14
23	Barnet	All	706765.75	606006.00	-0.14
24	Camden	All	928639.42	792454.00	-0.15
25	Croydon	All	464581.92	393479.00	-0.15
26	Lambeth	All	654875.76	550720.00	-0.16
27	Hammersmith And Fulham	All	883837.60	741308.00	-0.16
28	Wandsworth	All	834257.35	695867.00	-0.17
29	Tower Hamlets	All	595217.88	470209.00	-0.21
30	Kensington And Chelsea	All	1637183.90	1194726.00	-0.27
31	City Of Westminster	All	1273329.16	889935.00	-0.30
32	City Of London	All	993657.65	607399.00	-0.39

Jupyter Notebook

You can find the Jupyter notebook for this post here .

Building a local data warehouse with DuckDB, dbt, and Superset

Sun, 09 Mar 2025 15:00:00 +0000

In previous blog posts, I described two DuckDB use cases for data scientists and data engineers: Querying remote parquet files and processing larger-than-memory datasets .

Now I want to explore if I can set up a local and open-source version of the analytics data stack that we use at my company. At work, we are using Snowflake, dbt cloud, and Google’s Looker which cost us several thousand EUR per month. I am going to use the following open-source tools in my local setup:

	cloud	local
analytical database	snowflake	DuckDB
data modelling tool	dbt cloud	local dbt
dashboard tool	Looker	Apache Superset

Figure 1 shows how the components work together.

Figure 1. Components of my local data warehouse stack.

I will use a toy visualisation problem to demonstrate the setup. My goal is to load the race time dataset that I used in Data analysis: Vätternrundan 2024 results into DuckDB. I then use dbt to create data models and Superset to create an interactive dashboard to visualise the data.

Loading the raw data into DuckDB

I want to use DuckDB for all data processing. Therefore, I will load the raw race time data into DuckDB.

I start by installing DuckDB as my analytical database. Since I use homebrew as a package manager, I can run:

brew install duckdb

I then create a persistent database (a file on my machine):

duckdb database.duckdb

and create a new DuckDB table from the parquet file that contains the raw data.

CREATE TABLE vatternrundan AS select startnumber, city, country, result_time, start_time from read_parquet('results_vatternrundan24.parquet');

To verify that everything works, I run:

describe table vatternrundan;

┌─────────────┬─────────────┬─────────┬─────────┬─────────┬─────────┐
│ column_name │ column_type │  null   │   key   │ default │  extra  │
│   varchar   │   varchar   │ varchar │ varchar │ varchar │ varchar │
├─────────────┼─────────────┼─────────┼─────────┼─────────┼─────────┤
│ startnumber │ BIGINT      │ YES     │ NULL    │ NULL    │ NULL    │
│ city        │ VARCHAR     │ YES     │ NULL    │ NULL    │ NULL    │
│ country     │ VARCHAR     │ YES     │ NULL    │ NULL    │ NULL    │
│ result_time │ BIGINT      │ YES     │ NULL    │ NULL    │ NULL    │
│ start_time  │ VARCHAR     │ YES     │ NULL    │ NULL    │ NULL    │
└─────────────┴─────────────┴─────────┴─────────┴─────────┴─────────┘

Setting up dbt for data transformations

I want to use dbt to create data models on top of the raw data. I start by creating a new poetry project to install dependencies:

poetry init
poetry add duckdb-dbt

Then I initialise the dbt project local_warehouse with

poetry run dbt init local_warehouse

Next, I configure dbt to use DuckDB as the data processing backend. In my work setup, this would point at a Snowflake instance instead. I point at the local database file in the dbt profiles file profiles.yml:

local_warehouse:
  outputs:
    dev:
      type: duckdb
      path: ../duckdb/database.duckdb
  target: dev

dbt models

I define the DuckDB table vatternrundan as a dbt source in sources.yml:

version: 2
sources:
  - name: vatternrundan_db
    schema: main 
    tables:
    - name: vatternrundan

This means I can now refer to this raw data in dbt. Moreover, I want to create two more models. One thin staging layer over the raw data stg_vatternrundan.sql, and one refined model that aggregates the rider data by country average_speed_by_country.sql.

The staging model is stored in the dbt project as models/staging/stg_vatternrundan.sql:

select 
    startnumber::int64 as startnumber,
    city::varchar as city,
    country::varchar as country,
    to_microseconds((result_time/1000)::int64) as result_time,
from {{ source('vatternrundan_db', 'vatternrundan') }}

As you can see, I refer to the DuckDB source table, define the column types, and do some light conversions.

The aggregation model is stored in models/refined/average_speed_by_country.sql:

with speeds as (
    select startnumber,
    country,
    60 * datepart('hours', result_time) + datepart('minutes', result_time) as result_time_minutes,
    315 * 60 / result_time_minutes as average_speed,
    from {{ref('stg_vatternrundan')}}
)

select country, count(*) as num_riders, mean(average_speed) as average_speed_of_country 
from speeds
group by country
order by average_speed_of_country desc

I use this dbt model to compute the average rider speed for every country in the dataset.

Now that I have defined my dbt models, I configure dbt to run them by adding the following to dbt_project.yml:

models:
  local_warehouse:
    staging:
      +schema: staging
      +materialized: table
    refined:
      +schema: refined
      +materialized: table

This tells dbt how to materialise the dbt models in DuckDB. So let’s run dbt to build the models:

poetry run dbt run --profiles-dir=.

Check tables were created

I can verify in DuckDB that these tables have been created

Run

duckdb database.duckdb

followed by:

show all tables;

which should now show:

database.main_staging.stg_vatternrundan
database.main_refined.average_speed_by_country

Superset for data analytics dashboards

After modelling my data, I want to create a dashboard and visualise the data. I use Apache Superset as my dashboarding tool. Superset works with many analytical databases. However, setting it up to work with DuckDB is a bit clunky and I had to troubleshoot quite a bit.

Install Superset with DuckDB support

I follow the Quickstart Guide with some modifications. First, I download the git repository and checkout the last tagged commit:

git clone https://github.com/apache/superset
cd superset
git checkout tags/4.1.1

I need to make two modifications before building the container.

Install duckdb-engine inside the container. This allows us later to select DuckDB as a database in Superset. To do this, add

RUN pip install duckdb-engine

to the Dockerfile.

I need to make the DuckDB database file available in the Docker container. I add its path as a volume. Modify docker-compose-image-tag.yml:

superset:
[...]
    volumes:
        - ./docker:/app/docker
        - superset_home:/app/superset_home
        - <local-machine-path-to-duckdb-database>:/app/duckdb

and force the Superset container to build from the Dockerfile and not from the image:

superset:
[...]
    build:
        context: .
        dockerfile: Dockerfile
#   image: apache/superset: comment out or remove

Then build the container with

docker compose build

and start Superset with

docker compose -f docker-compose-image-tag.yml up

Then, I can access Superset with my browser under http://localhost:8088 and log in with username: admin and password: admin.

Configure Superset

Now, I can add DuckDB as a database in the Superset UI. I navigate to Settings -> Data -> Database Connections + Database -> DuckDB

and add the link to the DuckDB database file as the SQLAlchemy URI:

duckdb:////app/duckdb/database.duckdb?access_mode=READ_ONLY

Clicking on Test Connection should respond with “Connection looks good!”.

Add datasets in Superset

In the navbar under Datasets I can add the DuckDB tables stg_vatternrundan and average_speed_by_country as new Superset datasets.

Create dashboard in Superset

Creating the dashboard is self-explanatory. In the navbar, I select Dashboards and add a new Dashboard called “Vätternrundan Dashboard”. Next, I create two charts:

number of riders per country
average speed of riders per country

Just select the relevant Superset Dataset, build the charts, and add them to the dashboard. Figure 2 shows how I did it.

Figure 2. Visualization of Vätternrundan rider speed by country in Apache Superset.

Conclusion

The combination of DuckDB, dbt, and Apache Superset is a local version of “the modern datastack”. This can be useful for personal projects, and to experiment with data transformations and visualisations.

I found that the integration of DuckDB and dbt worked seamlessly. On the other hand, connecting Superset to DuckDB was clunky. Superset would benefit from better DuckDB support out-of-the-box.

Troubleshooting

DuckDB only allows one connection with read-write-access, but multiple concurrent read-only connections. Make sure you are not connected to your DuckDB database with more than one client as access mode allows write by default, e.g. Superset and the DuckDB CLI client in your terminal.
Ensure to connect Superset to DuckDB in read-only mode, i.e. add access_mode=READ_ONLY to the SQLAlchemy URI to connect.

Book notes: Poor Charlie’s Almanack

Fri, 21 Feb 2025 00:00:00 +0000

The book is a collection of speeches that Charlie Munger, partner of Warren Buffett at Berkshire Hathaway, gave over the years (1986 - 2007) at universities and institutions.

I found four interesting themes across the speeches:

Advocating interdisciplinary collaboration between university departments

He believes that the social sciences, especially economics and psychology, are focusing on too narrow theoretical problems. Academics suffer from a man-with-a-hammer-syndrome. He suggests that they collaborate more across departments, e.g. economics borrowing from psychology, and within departments, e.g. macroeconomists shouldn’t avoid microeconomic explanations. Moreover, the social sciences should try to incorporate findings / models from the hard sciences (physics, chemistry, etc.) and attribute them properly.

Fraud and advice for endowment funds

In some of his talks, he uses fictional examples of companies to show the problems with accounting and financial engineering fraud.

He thinks that charitable foundations and endowment funds should as much as possible avoid wasteful investment practices. He criticises the trend of university endowment funds to employ layers of analysts and consultants to use funds of funds. These multilayered systems erode a large chunk of the endowment funds’ returns when compared to simpler alternatives, like unlevered domestic equity indices. He directly criticises the practices of his audience of consultants and fund managers in talk six, but he uses humour and self-deprecation to get away with it.

(Inverted) Advice for Graduates

In his commencement speeches for university graduates, he gives advice on how to guarantee misery in life.

Be unreliable.
Don’t learn from others’ mistakes. Instead, make common mistakes of others again (join a cult, drive while drunk, gamble) and don’t learn from people that came before you.
Give up after adversity and failures. Just give up when the inevitable hard times occur.
Don’t ever invert. Don’t attempt to learn from thinking about achieving the opposite of your goals. Don’t try to be objective.

Mental checklist to analyse problems

A recurring topic in his speeches is a checklist of mental models and human biases. He developed this checklist over time and iterated through it when analysing a problem or evaluating an investment. Using appropriate checklists and the inversion technique are two main tools that he promotes for better thinking.

Here are 25 human biases from talk eleven: The Psychology of Human Misjudgment .

Reward and Punishment Superresponse Tendency: If you want to persuade people, appeal to their own interests. Ensure the incentives of people you work with are aligned with the outcome you want. Don’t reward them for metrics that they can easily game.
Liking/Loving Tendency: People are seeking love and approval from other people. Moreover, we favour people and products that are merely associated with the target of our affection. We can use this to our advantage by liking truly admirable people or ideas.
Disliking/Hating Tendency: People can have a tendency to dislike things different from them or products or people that are associated with the object of their dislike.
Doubt-Avoidance Tendency: Our brains are conditioned to quickly remove doubt after reaching a first decision. This tendency should be countered by forcing a delay for reflection before an important decision, e.g. jury decisions in court.
Inconsistency-Avoidance Tendency: A tendency to stick to previous conclusions, habits, and ideas to avoid change. Practically, this means that it is much easier to prevent a bad habit than to change it. To counter this tendency, we should force the discussion of counterarguments before a decision can be made. This tendency can be used to manipulate people (see also Ben Franklin effect , Cialdini’s consistency principle ).
Curiosity Tendency: Humans have a general tendency to be curious, which can be supercharged with today’s access to information. This fortunate tendency should be used to counteract other psychological tendencies.
Kantian Fairness Tendency: People have a tendency in direct interactions to behave fairly (following Kant’s categorical imperative).
Envy/Jealousy Tendency: People’s tendency to envy someone else’s status, wealth, or compensation. “It is not greed that drives the world but envy.” (Buffett)
Reciprocation Tendency: People have a tendency to reciprocate favours and disfavours. This can also be used for manipulation, e.g. a salesman could do you a small favour to get a much better outcome in negotiation.
Influence-from-Mere-Association Tendency: Valuing something by the association with another unrelated factor/idea/concept. Examples:
- association of quality with the highest price
- purchasing of luxury items to boost status
- advertising of products with unrelated but positive images
- associating one’s ability with past successes and making bad decisions
- thinking of someone worse because they are a competitor
Simple Pain-Avoiding Psychological Denial: Denying reality because it is too painful to accept, e.g. addiction, bankruptcy.
- Denial of a reality that’s too painful to accept, e.g. addiction to alcohol.
Excessive Self-Regard Tendency: People’s tendency to overestimate their abilities, decisions, and possessions. Moreover, their preference of people that are similar to them.
Overoptimism Tendency: A tendency to be overly optimistic of the future, especially if one has done well in the past.
Deprival-Superreaction Tendency: Reacting more strongly to losses than to gains, e.g. losing $10 is considered worse than gaining $10. Irrational overreaction to threatened loss of status, territory, love, friendship, or property.
Social-Proof Tendency: A tendency to act and think the same way as people around you (Group-think).
Contrast-Misreaction Tendency: Making bad decisions by anchoring on an irrelevant comparison. Examples:
- Adding $1000 of useless add-ons to a car only because the car costs $65k.
- A real estate agent presenting 3 terrible and expensive houses, then showing a merely bad house to make it look more desirable.
Stress-Influence Tendency: Light stress can increase performance temporarily, while heavy stress can cause dysfunctional thinking and bad decision-making.
Availability-Misweighting Tendency: Tendency to overweight or overvalue things (people, decisions, work, ideas) that are close or readily accessible to you. This also holds for metrics that are easy to measure and actions that are easy to take. This can be countered by following checklists of actions and by considering more difficult (or less accessible) alternatives.
Use-It-or-Lose-It Tendency: Over time, our skills and knowledge fade. Therefore we should deliberately train and repeat the skills we want to retain. Write them down as a checklist and work through them regularly.
Drug-Misinfluence Tendency: Most people can’t handle drugs responsibly over a long period of time. Not worth trying to prove that you can.
Senescence-Misinfluence Tendency: Older people have a harder time learning new skills. Knowing this, the best counter is to actively maintain the accumulated knowledge, see 19).
Authority-Misinfluence Tendency: Tendency to blindly follow the leader (see also: HiPPO effect ).
Twaddle Tendency: Some people waste time talking about things they are not an expert in. Try to separate these people from the experts and follow the latter.
Reason-Respecting Tendency: People can learn better when they can think through the reasons behind a directive or action. Therefore, when giving orders explain your reasoning.
Lollapalooza Tendency: Often multiple human biases act together to drive a certain behaviour or outcome.

Conclusion

I gave only 3/5 stars because the format of reading his talks didn’t appeal to me. I was already aware of most of the human biases listed by Munger. Some are discussed in Thinking Fast and Slow (2011) and Influence: Science and Practice (2001) . However, what I found impressive is that he talked about and used these biases already 20 years ago, before they were widely discussed. I also liked that in many speeches he directly criticises the audience that invited him to speak. However, he does it in a charming and humorous way.

About this blog and its microfeatures

Wed, 12 Feb 2025 00:00:00 +0000

No blog would be complete without a post explaining the code and frameworks used to generate it. In fact, often tweaking the blog code is more fun than actually writing something.

Static site generator

I try to keep the tech of this blog simple. I am using the Hugo static site generator and chose one of the simplest themes Bear Cub to get started. What I liked about the theme was its small code footprint, no JavaScript/ads/trackers, and it was entirely focused on text. The generated pages are small and load fast. The simple layout made it very easy for me to understand and modify it over time.

The source code is hosted on GitLab and the static sites are published to GitLab Pages. This means that to publish a new post, I just have to push a commit with the changes to the remote repository.

Supporting math, code, Jupyter notebooks

At the time of starting this blog I worked as a data scientist. I wanted to be able to share reproducible Jupyter notebooks that I could easily convert to HTML and embed in this blog. All notebooks are reproducible unless the data used in the blog post is too large to be uploaded to GitLab. I put my notebooks into the /notebooks folder of the repository and use the script nb2hugo to convert the notebooks to markdown files. At the bottom of the post you can usually find the link to the Jupyter notebook, the poetry environment, and the data. I wrote about this here .

The equations on this blog are rendered with KaTeX .

Content types

The content of this blog has four different types:

Posts are normal blog posts.
TILs, or Today-I-Learned, are notes documenting something small I have learned on a day. The idea is based on Simon Willison’s TILs .
Links appear in Feeds and are links that I want to share, usually with a short comment. They can be thought of as a retweet.
Book reviews.

1-3 are shown in reverse-chronological order in the main page’s feed.

Microfeatures

I like Daniel Fedorin’s post on blog microfeatures . In the same spirit, here are the microfeatures that I have added:

Recommendations

Most pages have post recommendations at the bottom. I compute the (cosine) similarity of posts after summarizing them with a local LLM, computing embeddings, and storing them in a local Chroma vector database. The two most similar posts and their cosine similarity are then displayed. Note that all of this is pre-generated and static. I wrote in more detail about how this works here .

For this post, they look like this:

Sidenotes

I can add sidenotes like this one. to a blog post by using the {{% sidenote %}} shortcode. I really like to use this for comments that are additional to the primary content or later updates to a blog post.

Markers for external links

Internal links on this blog have no marker and open in the same window. External links are appended with an icon and open in a new window. Inspired by gwern.net , I also added specific icons for particular sites: tweet on x.com, book link on goodreads, video on youtube, and article on wikipedia.

Link-blog / retweets / boosts

The main page of this blog has a section Recent which displays a feed of the content being posted. It’s a mix of posts, TILs and link-blog entries and should mimic a social media profile feed. Shared links have a reshare/boost symbol and usually a short comment about them. They also have a via link to the place where I found this link. This is also inspired by Simon Willison’s main page.

Figure 1. A shared link.

RSS

I provide RSS feeds for all content types and also separate feeds for Posts , TIL , Feed , and Books only. I wrote more about this here .

Longer blog posts have a table of contents section at the beginning of the post.

Figure 2. Some longer posts show a table of contents at the top.

Linkable Headers

The main headers of blog posts have a linkable §-symbol in front of them. This means you can easily link to a specific section of the blog post.

Drawings

I added the ability to provide an optional SVG image that is added to the background next to the post title, e.g. here and here .

Figure 3. Blog posts can have an SVG drawing next to the title.

Prompts / LLM responses

When I first wrote about using an LLM to build something I thought that it would be quite common to share a prompt and the LLM’s output in your blog post. So I built a small feature to display them in a chat-like fashion. You can use {{% prompt %}} to put text into a prompt chat bubble, and {{% llm %}} for the model response (colored by model). Turns out, this is not something I regularly need and nobody wants to read someone else’s prompt or the LLM’s answer anymore.

Prompt

How many Rs are in the word strawberry?

Model: claude-opus-5-5 - response

There are 3 Rs in “strawberry”: strawberry.

Model: openai-sol-5-6 - response

There are 3 Rs in “strawberry”: strawberry.

Model: qwen-4-3 - response

There are 3 Rs in “strawberry”: strawberry.

How related posts are computed

Sun, 09 Feb 2025 00:00:00 +0000

The number next to related posts at the bottom of each page is the advertised post’s “similarity” to the currently viewed page (from 1.0 to -1.0).

I am using the following process to compute related posts locally:

Summarise every post and TIL using local qwen (qwen3.5:9b via Ollama) with the following prompt:

Prompt

You are an analyst and editor with many years of experience in reading and synthesizing content.

Here is a blog post:

<BLOGPOST>

{ blog_post }

</BLOGPOST>

Please create a comprehensive and concise summary of the blog post. Focus on the main concepts, key details, and central arguments.

<INSTRUCTIONS>

Include any specific technologies, methods, or frameworks mentioned.
Don’t use more than 7 sentences.
Respond in plaintext. Don’t add formatting or linebreak characters to your response.
Don’t repeat the instructions of the task. Respond directly.

</INSTRUCTIONS>

Embed the summary using a qwen embedding model qwen3-embedding:0.6b and store the embedding and metadata about the post in a persistent ChromaDB vector database (a file on my computer).
Compute the cosine similarity between the embeddings for each pair of posts. A score of 1.0 indicates proportional vectors, a score of 0.0 orthogonal vectors, and a score of -1.0 opposite vectors.
Write a yaml file that includes for every post a link to the most similar post and their similarity.
Use a Hugo partial to include the data in the yaml file about the most relevant posts at the bottom of each page.

Noteworthy

When using Llama, I had to explicitly instruct the model not to restate the task. Otherwise, every summary would have started with “Here is your concise blog post summary in not more than 7 sentences.” which would create some artificial similarity in the embeddings.
Most blog posts are shorter than 15000 characters, or 4000 tokens. I had to work this out to configure the model context correctly.
ChromaDB’s default embedding model all-MiniLM-L6-v2 has 384 dimensions and a maximum input sequence length of 256 BERT tokens (~100–150 words). I checked some of the summaries and they were around 160 – 220 BERT tokens. I have since switched to a qwen embedding model which has much longer input context. A good embedding model leaderboard to help make a choice is available here .

Example

For the blog post UK house price changes in real terms the LLM came up with this summary which was then embedded:

Model: qwen3.5:9b - response

The author critiques standard UK housing market reports for presenting nominal price changes that obscure local realities and ignore inflation, arguing instead for real house prices adjusted using CPI data from ONS alongside Land Registry transaction records. To address this gap, the post demonstrates a Python-based workflow utilizing pandas to merge regional property datasets with quarterly consumer price index figures to calculate purchasing power-adjusted values in 2025 pounds. The analysis focuses on specific areas around London and South England, visualizing trends for detached houses in counties like Surrey and Kent as well as flats within selected boroughs since 2015. Results reveal that while nominal prices may show modest growth or stagnation, real house prices have actually declined significantly over the last decade in many regions due to inflation outpacing asset appreciation. The author provides interactive tables comparing current valuations against historical data points from one, two, five, and ten years prior to highlight these divergences across different property types. By leveraging open APIs for Land Registry and ONS data stored as parquet files, the post offers a reproducible method for investors and buyers to assess true market performance beyond headline statistics. Ultimately, this approach clarifies that British housing has often underperformed relative to other investments like the FTSE 100 when viewed through an inflation-adjusted lens.

Embeddings

I can visualise the embeddings in two dimensions using t-SNE :

Figure 1. Plot of blog post summary embeddings in t-SNE space.

The plot looks somewhat reasonable. The code-heavy Jupyter notebook posts are at the bottom, clustered around duckdb-large-datasets. The book reviews (4000-weeks, how-big-things-get-done, how-to-win-friends) are fairly close together. SQL related posts are clustered at the top. On the other hand, I would have expected reading-and-note-taking to be closer to writing-well.

Code

I am using this langchain script to compute the recommendations.

Update

A previous version of the code and blog post used Llama 3.1 to summarise the text and ChromaDB’s default embedding model all-MiniLM-L6-v2.

Book notes: How Big Things Get Done by Bent Flyvbjerg and Dan Gardner

Sun, 02 Feb 2025 00:00:00 +0000

Bent Flyvbjerg is an economics professor at IT University of Copenhagen. He maintains a database of megaprojects (power plants, opera halls, tunnels, airports) and their planned and realised timelines and budgets. He researches the reasons why modern megaprojects often fail to deliver on time and on budget. It has recently become quite popular to discuss the apparent decrease in speed with which large-scale projects are realised. Examples are easy to find, e.g. the delayed High Speed Rail 2 (HS2) project or the time and cost overruns of the Berlin Brandenburg Airport (BER) . Counter-examples are less common (or less newsworthy), e.g. the 2023 Notre-Dame Reconstruction.

In their book Flyvbjerg/Gardner explain the common reasons for cost and time overruns of megaprojects and how to mitigate them. I think we can apply the majority of their learnings for smaller home (kitchen renovation) and work projects (cloud migration) as well.

Planning is cheap, building is expensive

In large construction projects, planning is a lot cheaper than building. Once a badly planned project is underway, unforeseen problems will be discovered. Delays tend to cause further delays. The total project time can be in itself a source of additional risk and cost.

When politicians plan prestigious infrastructure projects, they have a bias for strategic misrepresentation of the cost and time to get support for it. Therefore, the government and the public need to scrutinise the project’s goal and its planning to ensure that the plan is realistic and enough alternatives have been considered. Only get involved in such a project if it has the people and funds, including contingencies, to succeed.

Since planning is a lot cheaper than building for large projects, Flyvbjerg/Gardner advise investing significant time into the planning phase:

understand the objective behind the proposed project (assume an outside view)
explore alternatives and don’t commit to the first available solution (resist quick action bias)
don’t forecast using the best-case scenario, instead use similar past projects to anchor your estimate
experiment and iterate in the planning phase using digital modelling and simulations

Understand the why

The project is not a goal in itself, it is how the goal is achieved. We want to talk to the stakeholder to understand the goal. Is the project even the right approach? Good planning needs to explore the problem before jumping to the solution. This includes considering alternatives.

There is a good example in the book about a bridge project. The original goal of the project is to connect an island with the mainland. By jumping to the bridge idea, the stakeholders ignored alternatives, e.g. a tunnel, ferries, a helipad. Or maybe a physical connection is not required. If it’s about improved communication, maybe a high-speed broadband connection achieves the goal.

Another framing of this idea in product development is to work backwards from the customer. Understand their needs and problems, before coming up with any solution.

Reduce uncertainty via experimentation

A crucial part of the planning phase is to experiment with the solution. Ideally, we can simulate and iterate on the project, e.g. use 3D models of the building, low fidelity designs to show customers, or low effort versions of the animation movie we are trying to produce.

The simulation ensures that the majority of aspects of our project are scrutinised before the building phase begins. We can assume that the project will run into problems, so we want most of the problems to occur during the cheap planning phase.

Experience in people and technology

To maximise the chance of a successful project we should maximise experience. Flyvbjerg/Gardner consider both experience in key people and in technology. We should try to ensure that key people have experience in similar projects. Ask: “Have they done it before?”.

“Technology is ‘frozen experience’.”

All things being equal, we should use tried and tested off-the-shelf technology instead of shiny new technology. If we can, we should use existing designs and operational processes.This is the same idea behind McKinley’s advice to Choose Boring Technology .

The construction of the Empire State Building is used throughout the book as a positive example. One reason for this was the architect William Lamb who insisted on using only proven technology, and who created a design that allowed repeatable non-custom work steps. Moreover, the construction company Starrett Brothers and Eken had built several similar skyscrapers on time and budget before.

Reference-class forecasting

After the project is planned we need to forecast the project duration and its cost. Unfortunately, planners often forecast too optimistically. Instead of deriving it from the project alone, the authors suggest using reference-class forecasting. The idea is to find a set of comparable projects, and to anchor our forecast based on their outcomes. Then adjust from that anchor.

However, good forecasting cannot protect the project from fat-tailed risks. Since those risks can kill our project, we need to identify the (known) high risk events and try to mitigate them.

“Successful project leaders focus every day on not losing, while keeping a keen eye on the […] goal they are trying to achieve.”

After the planning is done, build quickly

The project duration is in itself a source of budget and time risk. After planning, simulating, and forecasting, we therefore need to act fast once the building phase starts. To do this we should ideally use an experienced team and a modular building pattern.

Build with an experienced team

If possible, hire, what the authors call, a masterbuilder. That is someone who has experience in similar projects and can pick the right team. If such a team doesn’t exist, you or them need to create it.

Moreover, we need to ensure that the incentives of participating contractors are aligned with ours by sharing risks and rewards, e.g. pay a bonus for early completion. Don’t always pick the contractor that submits the lowest bid, because the lowest bid doesn’t necessarily lead to the lowest cost. Moreover, we will try to choose companies that we have successfully collaborated with before.

What’s our Lego?

Nuclear power plant constructions belong to the class of projects that most likely exceed cost and time budgets. Flyvbjerg/Gardner argue that you are building one large thing that has few repeatable, standardised parts. Solar power plants are the opposite. The core ingredient, the solar modules, can be produced rapidly and repeatedly in factories, and then assembled in a modular fashion.

When possible we should try to build modules. We can then produce and assemble the modules in a repeatable process. This delivers value in stages. After we have completed a module, we can use the learnings to iterate on the module’s design and assembly process. So we should always ask ourselves:

“What’s our basic building block, the thing we will repeatedly make, becoming smarter and better each time. What’s our Lego?”

Additional thoughts about the book

Incentives and identity

There is a section in the book that discusses why the Heathrow Terminal 5 project was completed on time. The authors argue that a large part was the alignment of incentives between the project managers, the contractors, and the workers. Contractors were aligned via contractual bonuses and established working relationships. Workers were treated well, their feedback was actively encouraged, and they felt like they were contributing to a historic project in their country. Not part of the book, but the Notre-Dame reconstruction after the fire, was also accomplished on time. I believe that the history of the building, and the feeling of being part of a national project are very powerful motivators that aligned the participants of the project.

Cost overruns by project type

I found the table in the book that shows the mean cost overrun by category interesting.

The table shows the mean (base rate) cost overrun for each project type, the fatness of the distribution (% of projects in the upper tail), and the cost overrun for projects in that tail.

Project type	Mean cost overrun (%)	% of projects in tail (>= 50% overrun)	Mean overrun of projects in tail (%)
Nuclear storage	238	48	427
Olympic Games	157	76	200
Nuclear power	120	55	204
IT	73	18	447
…	…	…	…
Buildings	62	39	206
Rail	39	28	116
Airport	39	43	88
Tunnels	37	28	103
…	…	…	…
Wind energy	13	7	97
Energy transmission	8	4	166
Solar power	1	2	50

It makes sense that (non-standard, non-modular) nuclear projects have the highest base rate for cost increase. On the opposite side are wind energy and solar power projects, that can be pre-produced in factories and assembled on-site. Organising Olympic Games suffers from the fact that they are highly complex, and usually held in a city that hasn’t hosted them before (inexperience). As a tech worker, I am intrigued to see IT projects at the top of the list. Moreover, when IT projects overrun they incur a 447% cost overrun, the highest among all project types.

Learnings for software projects

How applicable is the advice for day-to-day data and software projects? I think much of the advice applies. I found that chunking larger projects into small value-delivering modules is a great way to ensure continuous progress (especially in large refactoring projects).

The emphasis on planning vs. building is probably not as relevant in normal day-to-day software projects. There is less of a cost difference between planning and building a feature (in both cases mostly the software engineers’ time) compared to construction projects.

LLM prompt heuristics that definitely maybe work

Fri, 11 Oct 2024 14:56:00 +0000

Effective prompt writing for large language models continues to be a dark art. Having read the prompt engineering blogs from Meta, Anthropic, and OpenAI and watched some of Anthropic’s prompt discussions online, it does feel more like design than engineering. Even employees from the same lab don’t agree what tricks actually work. If you have time watch this video of one of Anthropic’s prompt engineers running a prompting masterclass. While giving good heuristics to the audience he is careful not to make any definite statements.

I don’t believe it is a good investment to try to become a world-class prompter. All heuristics are highly dependent on the model architecture, training data, and training procedure. This means that with every iteration of the models, heuristics could become obsolete or harmful to performance.

However, the current attention-based architecture doesn’t seem to be going anywhere soon. Therefore, it is reasonable to expect that we can continue to use the prompt context to help the model to move the embeddings of the user task into directions that contain a lot of nuance and information about the domain of the task.

Therefore, I am collecting the advice into a few guidelines that I can use with the current models. I will caveat this by saying that in 90% of my use-cases the model response is parsed by a human: me. Therefore, I am not as worried about hallucinations as someone that puts the model outputs in front of their customers or inside data parsing pipelines.

Here is what seems to work in October 2024 with Claude Sonnet 3.5, GPT-4o, and Llama 3.2.

Prompt specificity

Make the prompt as specific to the task as you can. This is probably the biggest return on your time. Good advice given in this discussion is to imagine printing out the prompt and giving it to a new hire at your company. Then see if they could solve the task. This forces you to give all necessary context and constraints of the task.
Components of a role prompt: The role, its place in an organisation, its perspective and the perspective of the person or organisation being addressed.

No lazy role prompts. Make the role and the context clear.

Bad: You are a cab driver.
Good: You are a cab driver, driving people as your full-time job in London for twenty years and are knowledgeable about the city, its roads, and its sights.

If the model is used inside a product, tell it about it:

Bad: You are an assistant writing document summaries.
Good: You are an assistant used in a product for law firms that summarizes
legal documents.

Order of prompt components

Order the components of your prompts in the following order:

Specific role and context description (put into system prompt if part of interactive use of the model)
Input data, e.g. documents, code snippets, CSV files
Task description
(Optional:) Good examples and bad examples
Task constraints

Use XML tags for prompt components

Use XML tags to separate different components of a prompt.

You are an experienced post-doc at a reputable research institute in the US. You are an expert in the research field of the following paper:

<Research_paper>
{{research_paper}}  
</Research_paper>

Your task is to summarize the findings of the research paper given to you in
<Research_paper> tags.

<Instructions>
- List between 1 - 5 most important findings of the paper. Don't list more
  than 5 findings.
- more instructions...
</Instructions>

Using examples (few shot prompting)

It can help to provide both good and bad examples for a specific task with an explanation why they are good or bad. In the above example we could provide an example research paper with a summary of findings we wrote ourselves as a good example.

Instructions

Response length limit. Instead of writing “Be concise”, tell the model what that means for your context, e.g. “Answer in no more than 4 sentences.”
Avoid open-ended instructions.
Instruct for style, formatting, and restrictions.
Instruct to ask for sources of evidence to reduce hallucinations.

Ask it to respond in a chain-of-thought to increase performance.

You are a logician and love to solve logic puzzles. Carefully read the following puzzle.

<PUZZLE>
Simon is looking at Charlie. Charlie is looking at Sarah. You know that Simon is married and Sarah is not married. Is a married person looking at an unmarried person?
</PUZZLE>
Let's think step by step before giving an answer.

Grammar and style

Avoid typos and wrong punctuation as that can deteriorate the quality of the response.

Parse-able output

Often LLMs add a preamble at the beginning or an epilogue at the end of a response. If you want to force the model to respond only with valid json, you can:

Use the “Prefill Claude’s response” feature
Ask the model to put the json into tags and then extract that block from the response
Prefill the response yourself by adding at the end of your prompt: Here is the JSON: {. The open bracket conditions the model to start the answer with the first JSON key. You then need to prepend the “{” to the response to make it valid.

Other tricks

If your prompt includes logic that could be handled in code, handle it in code.

Bad: You are part of a role-playing game that is used for training of customer support agents in a Fortune 500 company. 

You can assume any of the following roles based on user input. 

If the user asks for Role1 assume the role of a customer asking questions about the company's products. If the user asks for Role2, assume the role of a helpful customer support agent.

Good: Define two different prompts for each role and use code to switch prompts based on what role the operator wants the model to assume.

Give the model a way out if it doesn’t know the answer.

Good: {{prompt}}
If something weird happens and you are unsure about what to do, simply print out
"UNSURE".

Use a temperature of 0 for fact-based, less creative tasks.

Iterative prompt design

You can use the LLM to generate examples for a task and select the good examples. Then use those examples in the prompt that is used “in production”.
If the model responds incorrectly, tell it about the mistake and ask it how you should modify the prompt.

Processing 112M rows of steam reviews locally with DuckDB

Sat, 28 Sep 2024 13:43:40 +0000

In DuckDB use cases for data scientists: Querying remote S3 files I wrote about how I use DuckDB as a convenient way to query data from CSV or Parquet files in S3. Another use case for data scientists and data engineers is DuckDB’s ability to process larger-than-memory data on your local machine. For us data scientists this fills the gap between

small data that you can transform with pandas
big data that typically requires a multi-node processing engine like PySpark.

In this post I am going to run some exploratory queries against this kaggle Steam review dataset . It contains 112M rows of Steam game reviews and comes as an uncompressed 46GB CSV file (17GB compressed). Let’s find out how my MacBook copes with it. Later I am going to run a small comparison against polars , a DataFrame library, which is often mentioned for out-of-memory processing.I am not comparing against pandas because its inability to work with large datasets was one of the reasons why I explored this topic. However, you could do this with chunking .

Useful commands and settings

Before I start, here are some useful DuckDB commands and settings that I often use:

Command	What is it good for?
`.timer on`	Prints the execution time after each SQL command
`FROM duckdb_memory();`	Gives an overview of how much memory is used by DuckDB
`SET enable_progress_bar = true;`	Prints a progress bar for query runs
`SET memory_limit = '2GB';`	Controls how much data DuckDB can keep in RAM.
`.mode line`	Prints query results one at a time (useful if many columns). Default: `.mode duckbox`
`SUMMARIZE (select *)`	Summarizes the contents of a table.
`.exit`	Stops DuckDB.

Steam review dataset

First, I want to familiarise myself with the dataset. I downloaded the 46GB file all_reviews.csv to my 2023 MacBook Pro M3 with 18GB RAM. After starting a non-persistent session with the duckdb command, I use the above commands to activate the timer and line mode. I then take a look at an example row with:

 select * from read_csv('all_reviews.csv', ignore_errors=true) limit 1;

              recommendationid = 148919893
                         appid = 10
                          game = Counter-Strike
                author_steamid = 76561199036724879
        author_num_games_owned = 0
            author_num_reviews = 3
       author_playtime_forever = 197
author_playtime_last_two_weeks = 197
     author_playtime_at_review = 197
            author_last_played = 1698336369
                      language = russian
                        review = старость
             timestamp_created = 1698336397
             timestamp_updated = 1698336397
                      voted_up = 1
                      votes_up = 0
                   votes_funny = 0
           weighted_vote_score = 0.0
                 comment_count = 0
                steam_purchase = 1
             received_for_free = 0
   written_during_early_access = 0
         hidden_in_steam_china = 1
          steam_china_location =

This gives me an overview of the file columns and an idea of their content.

Processing the file directly

Rather than slowly loading all data into DuckDB, I want to test DuckDB’s stream processing capabilities. Instead of fully materialising the data in memory, the execution engine reads and processes the data in chunks.This is a useful feature if you want to convert larger-than-memory files from one format to another format, e.g. CSV to Parquet. But that’s not what we are here for today.

I am going to use two aggregation queries for my benchmark:

The first query counts the number of English reviews per steam account and sorts them from highest to lowest

D select author_steamid, 
COUNT(*) AS num_reviews  
FROM read_csv('all_reviews.csv', ignore_errors = true)  
WHERE language = 'english' 
GROUP BY author_steamid 
ORDER BY num_reviews DESC;
100% ▕████████████████████████████████████████████████████████████▏ 
┌───────────────────┬─────────────┐
│  author_steamid   │ num_reviews │
│       int64       │    int64    │
├───────────────────┼─────────────┤
│ 76561198030784015 │        9674 │
│ 76561198024340430 │        5930 │
│ 76561198067298289 │        5534 │
│ 76561198094803808 │        4341 │
│ 76561198125392509 │        4124 │
│ 76561198027267313 │        4124 │
│ 76561197960373660 │        3350 │
│ 76561197970602587 │        3212 │
│ 76561197961017729 │        2810 │
│ 76561198155150242 │        2554 │
│ 76561198066590240 │        2490 │
│ 76561198045381877 │        2280 │
│ 76561198069159152 │        2216 │
│ 76561197960319772 │        1978 │
│ 76561198062813911 │        1940 │
│ 76561198025731804 │        1928 │
│ 76561198137285867 │        1898 │
│ 76561198043135631 │        1883 │
│ 76561198036629241 │        1875 │
│ 76561198055119582 │        1843 │
│         ·         │           · │
│         ·         │           · │
│         ·         │           · │
│ 76561198111175247 │           1 │
├───────────────────┴─────────────┤
│    15324507 rows (40 shown)     │
└─────────────────────────────────┘
Run Time (s): real 22.556 user 166.382276 sys 9.815767

This query processes the 112M rows in a surprisingly short 22.5s. The most active Steam user has written 9674 game reviews. You can find them here .

I made the second query intentionally more complex. I filter for reviews that contain the word love and I also compute the mean of the votes_up score and the sum of the votes_funny score for each user. I then filter the aggregation using having for only users that have a sum_votes_funny score of more than 100.

D SELECT author_steamid, 
    COUNT(*) AS num_reviews,
    avg(votes_up) as mean_votes_up, 
    sum(votes_funny) as sum_votes_funny,
    FROM read_csv('all_reviews.csv', ignore_errors = true) 
    WHERE language = 'english' and review LIKE '%love%'
    GROUP BY author_steamid 
    having sum_votes_funny > 100 
    ORDER BY num_reviews DESC; 
100% ▕████████████████████████████████████████████████████████████▏ 
┌───────────────────┬─────────────┬────────────────────┬─────────────────┐
│  author_steamid   │ num_reviews │   mean_votes_up    │ sum_votes_funny │
│       int64       │    int64    │       double       │     int128      │
├───────────────────┼─────────────┼────────────────────┼─────────────────┤
│ 76561198155150242 │         777 │   1.09009009009009 │             362 │
│ 76561198042406453 │         665 │ 13.478195488721804 │             341 │
│ 76561198043135631 │         475 │ 2.9410526315789474 │             116 │
│ 76561198149437416 │         352 │ 1.2386363636363635 │             256 │
│ 76561198007343154 │         342 │  4.038011695906433 │             375 │
│ 76561198066590240 │         273 │  9.293040293040294 │             247 │
│ 76561197970314107 │         241 │  38.15767634854772 │             319 │
│ 76561197961017729 │         237 │ 10.278481012658228 │             251 │
│ 76561197981638563 │         231 │ 26.372294372294373 │             509 │
│ 76561197972040704 │         223 │  8.560538116591928 │             123 │
│ 76561197992694498 │         223 │   73.1390134529148 │             759 │
│ 76561197970761123 │         222 │ 22.603603603603602 │             150 │
│ 76561198043609914 │         217 │ 18.792626728110598 │             129 │
│ 76561198053422627 │         202 │  17.04950495049505 │             284 │
│ 76561198007888370 │         196 │  93.96938775510205 │            3345 │
│ 76561198040884867 │         191 │  38.41884816753927 │             509 │
│ 76561198817597644 │         166 │ 22.542168674698797 │             119 │
│ 76561198356141989 │         166 │ 19.246987951807228 │             216 │
│ 76561198031599084 │         165 │ 17.163636363636364 │             171 │
│ 76561198011647032 │         159 │  6.345911949685535 │             192 │
│         ·         │           · │                ·   │              ·  │
│         ·         │           · │                ·   │              ·  │
│         ·         │           · │                ·   │              ·  │
│ 76561198106232693 │           1 │               96.0 │             155 │
├───────────────────┴─────────────┴────────────────────┴─────────────────┤
│ 2383 rows (40 shown)                                         4 columns │
└────────────────────────────────────────────────────────────────────────┘
Run Time (s): real 26.648 user 213.432443 sys 7.281861

With ~26s this is on par with the previous query. User 76561198155150242 used the word “love” in 777 reviews and collected 362 funny upvotes across their reviews.

These timings show how incredibly convenient DuckDB is to analyse large datasets. <30s is still in the realm where I can interactively work with the dataset and explore different queries.You can push DuckDB to the limit by using a blocking operator, e.g. a rank() window function. This query will be slow because the entire input needs to be buffered to compute the result.

When I explore data it is more common that I work in a notebook instead of the CLI. Fortunately, I can run the same queries using the DuckDB Python client . In the next section I am going to compare it against polars, another relatively new kid on the block for out-of-memory analytics.

Working in a Jupyter notebook

DuckDB Python client

I am going to write the code to run the above queries using the duckdb Python package and return the results as a pandas dataframe.

import duckdb
import pandas as pd

# Connect to DuckDB
conn = duckdb.connect(':memory:')


# Define the queries
query1 = """
SELECT author_steamid, COUNT(*) AS num_reviews 
FROM read_csv('./steam_reviews/all_reviews/all_reviews.csv', ignore_errors = true) 
WHERE language = 'english' 
GROUP BY author_steamid 
ORDER BY num_reviews DESC 
"""

query2 = """
SELECT author_steamid, 
COUNT(*) AS num_reviews,
avg(votes_up) as mean_votes_up, 
sum(votes_funny) as sum_votes_funny,
FROM read_csv('./steam_reviews/all_reviews/all_reviews.csv', ignore_errors = true) 
WHERE language = 'english' and review LIKE '%love%'
GROUP BY author_steamid 
having sum_votes_funny > 100 
ORDER BY num_reviews DESC 
"""

%%timeit

# Execute the query
result1 = conn.execute(query1)
# Convert the result to a pandas DataFrame
df1 = result1.df()

16.6 s ± 627 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

df1.head(5)

	author_steamid	num_reviews
0	76561198030784015	9674
1	76561198024340430	5930
2	76561198067298289	5534
3	76561198094803808	4341
4	76561198027267313	4124

%%timeit

# Execute the query
result2 = conn.execute(query2)
# Convert the result to a pandas DataFrame
df2 = result2.df()

19.9 s ± 343 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

df2.head(5)

	author_steamid	num_reviews	mean_votes_up	sum_votes_funny
0	76561198155150242	777	1.090090	362.0
1	76561198042406453	665	13.478195	341.0
2	76561198043135631	475	2.941053	116.0
3	76561198149437416	352	1.238636	256.0
4	76561198007343154	342	4.038012	375.0

Not surprisingly this gives the same results as the queries run from the CLI. Having the aggregated results as a pandas dataframe allows me to use it in downstream work. I don’t need to learn another syntax.

## clear memory 
import gc 
del result1, result2, df1, df2;
gc.collect();

Polars

Since polars is often brought up as a faster and better pandas, I want to do a performance comparison. Polars supports lazy dataframes, which allows me to define operations on the dataframe without loading it fully into memory first.

Below are the two equivalent queries written using polars syntax.

import  polars as pl

# Create a lazy DataFrame
df_lazy = pl.scan_csv("./steam_reviews/all_reviews/all_reviews.csv", ignore_errors=True)

%%time

# Define the query using lazy operations
result_polars1 = (
    df_lazy.filter(pl.col("language") == "english")
    .group_by("author_steamid")
    .agg(pl.len().alias("num_reviews"))
    .sort("num_reviews", descending=True)
    
)

# Execute the query and collect the results
df_polars1 = result_polars1.collect()
df_polars1.head(5)

CPU times: user 1min 3s, sys: 22.1 s, total: 1min 26s
Wall time: 2min 7s

shape: (5, 2)

author_steamid	num_reviews
i64	u32
76561198030784015	9822
76561198024340430	5983
76561198067298289	5577
76561198094803808	4408
76561198125392509	4203

%%time

result_polars2 = (
    df_lazy
    .filter(
        (pl.col("language") == "english") & 
        pl.col("review").str.contains("love")
    )
    .group_by("author_steamid")
    .agg([
        pl.len().alias("num_reviews"),
        pl.col("votes_up").mean().alias("mean_votes_up"),
        pl.col("votes_funny").sum().alias("sum_votes_funny")
    ])
    .filter(pl.col("sum_votes_funny") > 100)
    .sort("num_reviews", descending=True)
)


# Execute the query and collect the results
df_polars2 = result_polars2.collect()
df_polars2.head(5)

<timed exec>:9: DeprecationWarning: `pl.count()` is deprecated. Please use `pl.len()` instead.


CPU times: user 1min 21s, sys: 42.7 s, total: 2min 4s
Wall time: 5min 14s

shape: (5, 4)

author_steamid	num_reviews	mean_votes_up	sum_votes_funny
i64	u32	f64	i64
76561198155150242	786	1.094148	369
76561198042406453	673	13.43685	345
76561198043135631	480	2.922917	116
76561198149437416	360	1.216667	256
76561198007343154	354	4.014124	381

Now we can compare DuckDB vs. polars execution times for the two queries:

wall time	DuckDB	polars
query 1	17s	2min 7s
query 2	20s	5min 14s

Conclusion

This investigation shows that DuckDB is a powerful and convenient tool to process larger-than-memory datasets on a single machine. As a data scientist this is useful, because I can focus on exploring the data in the early exploration phase. I don’t yet have to spend time setting up more complex tools or work on a remote machine. The quick comparison with polars also shows its speed advantages and that it can be used as a drop-in when aggregating data in Jupyter notebooks.

Interest rate expectations

Sat, 14 Sep 2024 00:00:00 +0100

This morning, the top headline on the front page of my weekly newspaper was:

Bets rise on bumper rate cut by Fed

Investors have sharply increased their bets on half percentage point interest rate cut by the Federal reserve next week as the US central bank prepares to lower borrowing cost for the first time in more than four years. Traders in swaps markets are pricing in a 43 percent chance the Fed will opt for a bumper cut in a bid to prevent high rates damaging the economy. - FT Weekend (14.9.2024)

I often read news about expectations on how the Fed will set interest rates. I have a vague sense that it is related to the price of some interest rate related futures. I just asked myself if I wanted to look up what the market expectation is next week, where would I actually go to look? This detail is regularly glossed over in news articles. It turns out that the maths behind the expectation approximation is fairly simple. Let us work it out.

The Federal Funds Rate

Banks are required to put a certain percentage of their deposits into accounts at a Federal Reserve Bank. This is to maintain liquidity to cover depositors’ withdrawals in the short-term. This reserve requirement is loosely a percentage of the bank’s deposits at the end of the day averaged over a two-week period. If at the end of the day a bank has excess reserve balances it can lend it overnight to another bank that is undercapitalised. Banks do that a lot and negotiate an interest rate for this overnight loan. The weighted average of all these deals for the day is the effective federal funds rate (EFFR).

The Fed’s federal funds rate (FFR) target range is set by the Federal Open Market Committee and is the desired range for the EFFR. While the Fed can’t directly impact the EFFR it can influence it via their own deposit interest rates or changes to the monetary supply. This tells us what today’s FFR range and EFFR are (both are published by the Fed). But how do we compute the market’s expectations on changes to the FFR range?

Federal Funds Futures

We can infer the expectations of FFR range changes at the next committee meeting by looking into how FFR future contracts are priced. Those are traded on the Chicago Mercantile Exchange (CME). You can view the monthly future quotes on the CME website .

These futures are used by banks and fixed-income portfolio managers to hedge against short-term interest rate fluctuations. The 30-day futures are monthly contracts that are settled on the last business day of every month. The contract price payable is the arithmetic mean of the daily EFFR during the contract month as reported by the Federal Reserve Bank of New York, subtracted from 100. For example if the average interest rate was 3.5% for a given month, then the contract price would be 100 - 3.5 = $96.5. Unfortunately, the minimum contract size is the price times 4167, which for this example would be $96.5 * 4167 = $416,796.5. A bit too high for my personal hedging needs.

What do the Federal Funds Future(s) hold?

Now that we know about the EFFR and Federal Funds futures, we can collect all ingredients to compute the expectations:

The current FFR range was set in the July meeting to 5.25% - 5.5%.
The dates of the next committee meetings are published here on federalreserve.gov . The remaining meetings this year are scheduled for: 18.9.2024, 7.11.2024, 18.12.2024.
The expected EFFR from the futures contracts for the month of the next committee meeting and the following month. For the remaining year they were on the 14.9.2024:

import pandas as pd

df = pd.DataFrame(
    {
        'MONTH' : ['09-2024', '10-2024', '11-2024', '12-2024'],
        'PRICE' : [94.81, 95.03, 95.315, 95.58],
        'MEETING': [18, None, 7, 18]
     }
).set_index('MONTH')
display(df)

	PRICE	MEETING
MONTH
09-2024	94.810	18.0
10-2024	95.030	NaN
11-2024	95.315	7.0
12-2024	95.580	18.0

The next month without a committee meeting is October. This means that October’s average EFFR can only be impacted by the September committee meeting. This means that the October future contract price of $95.03 is the expected average EFFR for October.

r_october_avg = 100 - df.at['10-2024','PRICE']
r_september_avg = 100 - df.at['09-2024','PRICE']

It is also the expected EFFR on the last day of September (and the first day of November).

r_september_end = r_october_avg

We are interested in the implied difference in EFFR at the beginning and the end of the month. If there is a difference, then this would imply that the FED committee changed the FFR range.

Keep in mind that the contract price is the arithmetic mean of the realised EFFR on every day in September. Assume that the FED committee does lower the price on the 18th of the month, then we will have 18 days at a higher rate and 30-18 = 12 days at a lower rate. All of that is captured in the future price (the average EFFR) for September.

Therefore we can use:

\[ EFFR(\text{average of Sep}) = \frac{N}{30} EFFR(\text{start of Sep}) +\frac{30-N}{30} EFFR(\text{end of Sep}) \]

\[ \Leftrightarrow EFFR(\text{start of Sep}) = \left( EFFR(\text{average of Sep}) - \frac{30-N}{30}\cdot EFFR(\text{end of Sep}))\right) \cdot \frac{30}{N} \]

where $N$ is the number of days before the committee meeting in September.

days_month = 30
days_before_meeting = df.at['09-2024','MEETING']
days_after_meeting = days_month - days_before_meeting
r_september_start = (r_september_avg - (days_after_meeting/days_month * r_september_end) ) * (days_month/days_before_meeting)
print(f'The expected EFFR for the start of September is {r_september_start:.2f}%.')

The expected EFFR for the start of September is 5.34%.

Since today is the 14th of September, we can already look up the realised EFFR up until today. If today were the 25th of August, then we need to calculate the expected $EFFR(\text{start of Sep})$ with the above formula.

r_september_change = r_september_end - r_september_start
print(f'Comparing the expected EFFR at beginning and end of month we have a delta of {r_september_change:.2f}.')

Comparing the expected EFFR at beginning and end of month we have a delta of -0.37.

The market’s expectation for September is that we will have 37bps lower EFFR at the end of the month. If we buy the futures contract today we will make money if the FED committee cuts the rate in such a way that the realised EFFR will decrease more than 37bps compared to the beginning of the month.

Traditionally, the committee changes the FFR range by 25bps (however 50 and 100 basis point changes do occur, e.g. after 9/11, after the 2007-2008 housing market crash and during the COVID-19 pandemic).

However, we can see that the market prices in a 37bps lower EFFR. This means we can calculate the probability of a 25bps cut vs. a 50bps cut.

print(f'The price implies {r_september_change / -0.25:.2f} x 25bps rate cuts.')

The price implies 1.47 x 25bps rate cuts.

probability_of_25_bps_cut = 1 - (r_september_change/(-0.25) % 1)
print(f'This implies a probability of {100 * probability_of_25_bps_cut:.2f}% that the committee announces a 25bps rate cut.')

This implies a probability of 53.33% that the committee announces a 25bps rate cut.

probability_of_50_bps_cut =  (r_september_change/(-0.25) % 1)
print(f'This implies a probability of {100 * probability_of_50_bps_cut:.2f}% that the committee announces a 50bps rate cut.')

This implies a probability of 46.67% that the committee announces a 50bps rate cut.

The calculation confirms the newspaper article’s claim that the rate decision is priced as a coin flip between a 25bps and a 50bps rate cut.

Update 18.9.2024: The FOMC decided on a 50bps cut to a target range of 4.75% - 5.0%.

What about Europe?

If you are interested in the expectations for the interest rates set by the Bank of England, you can use One Month SONIA Index Futures . Similarly, for the ECB interest rates expectations you can check ECB Dated ESTR Futures .

Jupyter Notebook

You can find the Jupyter notebook for this post here .

Clearing up confusion around IPython, ipykernel, and Jupyter notebooks

Mon, 26 Aug 2024 00:00:00 +0100

One of my big recurring time sinks while doing data science work used to be trying to get my colleagues’ Jupyter notebooks to run on my machine. The main contributing factors:

My team generally uses poetry environments to improve reproducibility, but sometimes dependencies aren’t specified
I use VS code and the Jupyter extension to edit notebooks in VS code which requires more configuration than running the Jupyter web UI.
I lacked a clear understanding of the differences between IPython, ipykernel, jupyter and which Python environments are being used when running a notebook.

So this is my attempt at a friendcatcher . I hope this saves you a few minutes the next time you run into similar issues.

The different components

Let’s distinguish the components that play a role in running a Jupyter notebook:

Jupyter Notebook platform: A web-based interactive computing platform that supports different languages via different kernels, e.g. for Python (ipykernel), Julia (IJulia), R (IRKernel)
Jupyter notebook (extension .ipynb): is a document in json format that holds metadata and cell code.
IPython command shell: The shell has two components:
1. An interactive Python shell. You can start it with ipython. It’s like the default Python REPL but with enhanced features, e.g. object introspection, tab completion, input history, magic commands, etc.
2. A Jupyter kernel ipykernel. This is the backend process where user Python code runs and which can be connected to different frontends. One frontend is indeed the IPython shell, another one a Jupyter notebook. You can install ipykernel as a standalone package into your Python environment.
jupyter Python package. This is a metapackage which installs the notebook, qtconsole, and ipykernel.

graph TD;

colab(Google Colab UI) <--> ipykernel(ipykernel)
vs(VS Code UI) <--> ipykernel
ui(jupyter notebook UI) <-->  server
server(jupyter server) <-->  ipykernel
ipykernel <-->  ipython[IPython]

Jupyter kernels vs. shell environment

One source of confusion is that a Jupyter kernel can point to a different Python executable than your shell environment.

To get an overview of available Jupyter executables you can use:

List all available Jupyter executables in your system:
```
$ type -a jupyter
```
List all available Jupyter kernels:
```
$ jupyter kernelspec list
```

Every Jupyter kernel folder includes a kernel.json file that links to the Python executable that is being used. Note this can be different from the Python executable referenced by your current shell. Moreover, the shell environment of a Jupyter notebook uses the Python executable used to launch the notebook.

Print path of currently used Python executable:
```
$ type python
```
or in a notebook cell:
```
!type python
```
Print path of Python executable of current kernel: In a notebook cell:
```
import sys
sys.executable
```

You can create new kernels using the ipykernel package:

$ python -m ipykernel install --user --name envname --display-name "Python (envname)"

Dependency management

Since I use VS Code as my frontend I just need to add the ipykernel package into the virtual environment that I use to manage all other dependencies used to run the notebook. This ensures that the same Python executable is used for the kernel and the shell environment.This is well explained here .

These are the steps to create a new environment for a Jupyter notebook:

Create project folder:
```
$ mkdir notebook_project
```

Create new virtual environment in the folder, then activate it

$ cd notebook_project
$ python3 -m venv .venv
$ source .venv/bin/activate

Install ipykernel (and other dependencies) using pip (make sure the venv is activated):
```
$ python3 -m pip install ipykernel
$ python3 -m pip install pandas
```
Create a new notebook
```
$ touch mynotebook.ipynb
```
Open notebook in VS Code and in the top right corner select Select Kernel –> Python Environment –> .venv (.venv/bin/python)
You should now be able to run the notebook and use the pandas package inside the notebook.
To add new dependencies:
- Use the terminal: $ python3 -m pip install <package_name>
- Install from within a notebook cell:
```
import sys
!{sys.executable} -m pip install <package_name>
```
- Specify your dependencies in a requirements.txt

I use poetry for virtual environments and dependency management. So in step 2 I would instead use:

$ cd notebook_project
$ poetry init

and install packages via:

$ poetry add ipykernel
$ poetry add pandas

If I want to use the default Jupyter UI, I can install the jupyter metapackage into my environment and then start the UI with:

$ poetry add jupyter
$ poetry run jupyter notebook

Links

Analysing the Vätternrundan 2024 cycling results

Sat, 03 Aug 2024 00:00:00 +0100

In 2024, I was foolish enough to participate in the Vätternrundan , which is a 315km distance bike sportive around Sweden’s second biggest lake. It took me and my friends a little less than 11.5 hours of cycling (15h including breaks).

Image(filename='vatternrundan_map_lake.jpg', width='600px')

The above image shows the route. The event starts and ends in Motala (the big black dot on the map) and the riders ride clock-wise around the lake.

Since this event is a sportive the organizers only publish finishing times via a bib number search on their website . However, I was curious to understand how well my group did overall. To get my hands on the data I wrote a small scraping script to collect all finish times from the results page.

With this data I then analyse:

summary statistics about the participants, e.g. what countries had a lot of starters?
distribution of race times, e.g. how fast is the median rider?
what percentage of people finished the race

Scraping the dataset

Since getting the timing information from the results page involves several clicks for each start number, I had to write a script that can click through the browser pages. I use selenium for the browser automation. The script performs the actions for each start number between 1 and 30,000 and collects the timing information. Some start numbers are not used and many people didn’t finish the race. All the data is stored in a single parquet file that is used for the data analysis in the next section. I had to do some light data cleaning, e.g. converting scraped strings to timedelta, removing data for unused startnumbers, etc.

Loading the dataset

First, I load the data from the parquet file into a pandas dataframe. I then perform a few data cleaning steps to make the dataset easier to use.

import pandas as pd
import numpy as np
import datetime as dt
import matplotlib.pyplot as plt
import seaborn as sns
from matplotlib.ticker import FuncFormatter
from IPython.display import Image

df = pd.read_parquet("results_vatternrundan24.parquet")
df['result_time_minutes'] = df['result_time'].dt.total_seconds() / 60
df['average_speed'] = 1/df['result_time_minutes'] * 315 * 60
df['has_finished'] = ~df['result_time'].isna() & (df['timing_consistent']);

print(f'The dataset contains {df.shape[0]} entries.')

The dataset contains 15813 entries.

Analysing the dataset

The dataset has the following columns:

startnumber
initials: initials of the rider
club: club of the rider
city: home city of the rider
country: home country of the rider
result_time: full duration between crossing start and finish lines (includes breaks at food stations)
average_speed
is_sub9: whether the rider was part of a sub9 group that started later in the day (after 11:30 on Saturday)
has_finished: a boolean whether a rider completed the event or dropped out

You can see a few example rows of rider information below:

display(df[['startnumber', 'initials', 'club','city', 'country', 'result_time', 'average_speed', 'has_finished']].head(5))
df.dtypes

	startnumber	initials	club	city	country	result_time	average_speed	has_finished
40	5121	JK	-	Mantorp	SE	0 days 15:11:00	20.746432	True
41	5122	MI	-	Skärblacka	SE	0 days 15:11:00	20.746432	True
42	5123	TP	-	Höör	SE	0 days 14:35:00	21.600000	True
43	5124	KH	Trimgutta	Løvenstad	NO	0 days 14:47:00	21.307779	True
44	5125	HN	-	Jessheim	NO	NaT	NaN	False

startnumber                      int64
initials                        object
club                            object
city                            object
country                         object
result_time            timedelta64[ns]
start_time                      object
station_records                 object
timing_consistent                 bool
is_sub9                           bool
result_time_minutes            float64
average_speed                  float64
has_finished                      bool
dtype: object

The riders

Let’s first look at the top countries, cities, and clubs that are in this dataset. Not surprisingly, Sweden (SE) has the most participants, followed by Germany (DE), Norway (NO), and Finland (FI).

df.groupby('country').agg(num_starters = ('startnumber', 'count'), num_finishers = ('has_finished', 'sum'), perc_finished = ('has_finished', 'mean'), mean_result_time = ('result_time', 'mean'), fastest_finisher = ('result_time', 'min')).sort_values('num_starters', ascending=False).head(10)

	num_starters	num_finishers	perc_finished	mean_result_time	fastest_finisher
country
SE	12035	9904	0.822933	0 days 13:38:44.486094316	0 days 05:15:00
DE	1437	1174	0.816980	0 days 14:30:54.331914893	0 days 07:37:00
NO	846	658	0.777778	0 days 11:15:29.272727272	0 days 07:16:00
FI	402	335	0.833333	0 days 12:56:00.537313432	0 days 08:09:00
DK	346	281	0.812139	0 days 13:24:26.501766784	0 days 07:57:00
GB	269	194	0.721190	0 days 14:24:04.307692307	0 days 02:54:00
NL	87	77	0.885057	0 days 14:25:03.896103896	0 days 08:53:00
	43	36	0.837209	0 days 15:36:35	0 days 09:01:00
US	38	25	0.657895	0 days 13:39:55.384615384	0 days 09:19:00
PL	38	30	0.789474	0 days 12:52:26	0 days 09:20:00

The cities with most participants are all Swedish: Stockholm, Göteborg, Linköping, Uppsala, Malmö.

df.groupby('city').agg(num_starters = ('startnumber', 'count'), num_finishers = ('has_finished', 'sum'), perc_finished = ('has_finished', 'mean'), mean_result_time = ('result_time', 'mean'), fastest_finisher = ('result_time', 'min')).sort_values('num_starters', ascending=False).head(10)

	num_starters	num_finishers	perc_finished	mean_result_time	fastest_finisher
city
Stockholm	851	726	0.853114	0 days 13:37:09.752066115	0 days 07:31:00
Göteborg	613	516	0.841762	0 days 13:30:09.266409266	0 days 07:35:00
Linköping	378	313	0.828042	0 days 13:48:16.815286624	0 days 07:31:00
Uppsala	323	268	0.829721	0 days 13:50:19.029850746	0 days 07:37:00
Malmö	233	191	0.819742	0 days 13:52:36.125654450	0 days 07:31:00
Örebro	230	184	0.800000	0 days 13:34:19.677419354	0 days 08:19:00
Motala	223	186	0.834081	0 days 14:07:49.354838709	0 days 07:35:00
Västerås	186	142	0.763441	0 days 13:25:54.929577464	0 days 07:37:00
Lund	168	138	0.821429	0 days 13:56:18.260869565	0 days 07:16:00
Jönköping	147	124	0.843537	0 days 13:49:49.354838709	0 days 08:55:00

The following clubs had the most riders.

df.groupby('club').agg(city = ('city', 'first'), num_starters = ('startnumber', 'count'), num_finishers = ('has_finished', 'sum'), perc_finished = ('has_finished', 'mean'), mean_result_time = ('result_time', 'mean'), fastest_finisher = ('result_time', 'min')).sort_values('num_starters', ascending=False)[1:10]

	city	num_starters	num_finishers	perc_finished	mean_result_time	fastest_finisher
club
IMOVEFORCANCER	Karlskrona	69	58	0.840580	0 days 09:54:26.896551724	0 days 07:37:00
Fredrikshof	Stockholm	56	48	0.857143	0 days 10:46:42.500000	0 days 09:20:00
Örebrocyklisterna	Kumla	55	42	0.763636	0 days 11:31:47.142857142	0 days 08:04:00
Team Kungälv	Kungälv	43	29	0.674419	0 days 10:37:20	0 days 08:53:00
schulz sportreisen	Dresden	40	36	0.900000	0 days 13:47:45	0 days 10:20:00
KCE - Kävlinge Cykelentusiaster	Kävlinge	38	32	0.842105	0 days 11:50:01.875000	0 days 09:41:00
Fredrikshofs IF Cykelklubb	Enskede	35	30	0.857143	0 days 11:29:52	0 days 09:44:00
Försvarsmakten	Örebro	32	29	0.906250	0 days 13:51:00	0 days 09:21:00
Team Sportia Uppsala	Uppsala	31	29	0.935484	0 days 10:50:47.586206896	0 days 09:24:00

Number of finishers

315km is a long event. Let’s check how many of the participants finished the event:

num_starters = df.shape[0]
num_finishers = df.query('has_finished').shape[0]
print(f"Among the {num_starters} starting riders in the dataset, we found recorded and consistent finishing times for {num_finishers}. \
This is a finishing rate of {100 * num_finishers / num_starters:.2f}%.")

Among the 15813 starting riders in the dataset, we found recorded and consistent finishing times for 12920. This is a finishing rate of 81.70%.

The podium and top 10

fastest_rider = df.query('has_finished').sort_values('result_time').head(1)
print(f'The fastest rider {fastest_rider.initials.values[0]} from {fastest_rider.city.values[0]} only needed {dt.timedelta(microseconds= float(fastest_rider.result_time.values[0])/1000)} to complete the 315km. \
That is a mindblowing average speed of {(fastest_rider.average_speed.values[0]):.1f} km/h.')

The fastest rider JL from Lund only needed 7:16:00 to complete the 315km. That is a mindblowing average speed of 43.3 km/h.

The 10 fastest riders were below 7h 32min.

df.query('has_finished')[['startnumber', 'initials', 'city', 'club', 'country', 'result_time', 'average_speed']].sort_values('result_time', ascending=True).head(10)

	startnumber	initials	city	club	country	result_time	average_speed
16311	27012	JL	Lund	Kjekkas IF	SE	0 days 07:16:00	43.348624
16301	27002	TJ	Slattum	-	NO	0 days 07:16:00	43.348624
16304	27005	EO	Nittedal	-	NO	0 days 07:16:00	43.348624
16308	27009	BHB	Follebu	-	NO	0 days 07:16:00	43.348624
16310	27011	DT	Oslo	-	NO	0 days 07:16:00	43.348624
16316	27017	JK	Lillehammer	-	NO	0 days 07:16:00	43.348624
16314	27015	SD	Oslo - Norway	-	NO	0 days 07:16:00	43.348624
16781	27482	HD	Kalmar	-	SE	0 days 07:31:00	41.906874
16782	27483	TE	Rockneby	-	SE	0 days 07:31:00	41.906874
16783	27484	JL	Ekerö	-	SE	0 days 07:31:00	41.906874

The 10 fastest riders that didn’t ride as part of a sub-9 registered team were:

df.query('has_finished and not is_sub9')[['startnumber', 'initials', 'city', 'club', 'country', 'result_time', 'average_speed']].sort_values('result_time', ascending=True).head(10)

	startnumber	initials	city	club	country	result_time	average_speed
8860	16661	MN	Haslum	Team Tøff i Tryne	NO	0 days 08:02:00	39.211618
8850	16651	HL	Oslo	Team Tøff i Trynet	NO	0 days 08:02:00	39.211618
28298	18529	MS	Otalampi	Team Wassu	FI	0 days 08:12:00	38.414634
28297	18528	JK	Kotka	Team Wassu	FI	0 days 08:12:00	38.414634
28295	18526	JP	Helsinki	Team Wassu	FI	0 days 08:12:00	38.414634
8849	16650	SG	Eiksmarka	Team Tøff i Trynet	NO	0 days 08:17:00	38.028169
28296	18527	JS	Hamina	Team Wassu	FI	0 days 08:21:00	37.724551
8861	16662	FA	Oslo	Team Tøff i Trynet	NO	0 days 08:23:00	37.574553
11817	23218	WG	Lemgo	RC Sprintax Bielefeld	DE	0 days 08:24:00	37.500000
11819	23220	CB	Bielefeld	RC Sprintax Bielefeld	DE	0 days 08:24:00	37.500000

Lanterne rouge

The slowest finisher took almost 28h.

df.query('has_finished')[['startnumber', 'initials', 'city', 'country', 'result_time', 'average_speed']].sort_values('result_time', ascending=True).tail(5)

	startnumber	initials	city	country	result_time	average_speed
3416	2367	AH	Danderyd	SE	1 days 02:37:00	11.834690
1464	415	MP	Östersund	SE	1 days 02:49:00	11.746426
1465	416	ZYP	Stockholm	SE	1 days 02:49:00	11.746426
1260	211	HB	Svedala	SE	1 days 03:36:00	11.413043
1109	60	KH	Skara	SE	1 days 03:44:00	11.358173

Finishing times (excluding sub-9 groups)

To get a better sense of the finishing times, I plot the histogram and get some summary statistics of the result_time column for every finisher. I only consider the times of the regular riders below. This means excluding the sub-9 cycling teams that start separately later on Saturday. With a little less than 15h total (11.5h moving time) my group was slower than the median rider. Next time, we might want to take shorter breaks.

times = df.query('has_finished and not is_sub9')
times['result_time'].describe()

count                        12175
mean     0 days 13:55:05.829979466
std      0 days 02:57:47.200342638
min                0 days 08:02:00
25%                0 days 11:37:00
50%                0 days 13:44:00
75%                0 days 15:48:00
max                1 days 03:44:00
Name: result_time, dtype: object

print(f"To be among the top 5% fastest finishers, you need to beat {times['result_time'].quantile(q=0.05)}.")
print(f"To be among the top 10% fastest finishers, you need to beat {times['result_time'].quantile(q=0.1)}.")
print(f"To be among the top 25% fastest finishers, you need to beat {times['result_time'].quantile(q=0.25)}.")

To be among the top 5% fastest finishers, you need to beat 0 days 09:44:00.
To be among the top 10% fastest finishers, you need to beat 0 days 10:10:00.
To be among the top 25% fastest finishers, you need to beat 0 days 11:37:00.

To plot the distribution of result times, I am binning the result time in minutes and also add some vertical lines indicating the 0.1- and 0.25- percentiles. It’s not surprising that the distribution is right-skewed with many riders that have very long finishing times, but nobody faster than 8 hours.


def minutes_to_hours_minutes(x, pos):
    """Define a function to convert minutes to HH:MM. Used for the axis labelling."""
    hours = int(x // 60)
    minutes = int(x % 60)
    return f"{hours:02d}:{minutes:02d}"

# create histogram plot of result time distribution
plt.figure(figsize=(10, 6)) 
sns.histplot(data=times, x="result_time_minutes", binwidth=15)

plt.gca().xaxis.set_major_formatter(FuncFormatter(minutes_to_hours_minutes))

# draw colored vertical percentile lines
plt.axvline(times.result_time_minutes.median(), color='red', linestyle='--', linewidth=1.5)
plt.axvline(times.result_time_minutes.quantile(0.25), color='green', linestyle='--', linewidth=1.5)
plt.axvline(times.result_time_minutes.quantile(0.1), color='orange', linestyle='--', linewidth=1.5)
plt.text(times.result_time_minutes.median(), plt.ylim()[1] * 0.95, f'Median',
         color='red', ha='left', va='bottom')
plt.text(times.result_time_minutes.quantile(0.25), plt.ylim()[1] * 0.95, f'Top 25%',
         color='green', ha='center', va='bottom')
plt.text(times.result_time_minutes.quantile(0.1), plt.ylim()[1] * 0.95, f'Top 10%',
         color='orange', ha='right', va='bottom');
plt.xlabel('Finish time (incl. breaks)');

Participating in this race with a group of friends was great fun and the Swedish locals are very supportive. I can highly recommend this event if you are a confident cyclist. If you are planning to race next year’s Vatternrundan, this distribution might give you an indication of what to expect and a time to aim for.

Jupyter Notebook

You can find the jupyter notebook and the datasets for this post here .

How to better remember books

Sun, 26 May 2024 00:00:00 +0100

In this article I am going to describe the system that I use to retain more information from the non-fiction books that I read. There seem to be two schools of thought about how to best retain the content of non-fiction books. The first approach is best described by the quote:

“I cannot remember the books I’ve read any more than the meals I have eaten; even so, they have made me.”

- Ralph Waldo Emerson

By reading a lot, our brain retains what it finds interesting. Our prior beliefs are updated and specific ideas are reinforced within us when we encounter the same or related concepts in different sources.

Some argue that proper retention can only be achieved by actively working through the material while reading it. This second approach commonly involves taking notes on the major ideas of the book.

While I do experience surprise connection moments occasionally, there are two reasons why I invest time into taking good notes:

The first approach has an inherent survivorship bias. I do remember concepts from books and I am able to link them to other things. This always feels great. However, I don’t know about the missed opportunities of ideas that I have forgotten and that will never resurface.
Reading a non-fiction book is quite a significant time investment. For me it takes between a few days and several weeks. Taking at least basic notes seems to be a marginal additional cost for the certainty of improved retention and better ability to digest the book in the future.

I use a simple multi-pass approach that involves both analog and digital notes. I mostly read physical books, because I like to browse books on my shelf that I have read in the past. However, you can easily adapt this approach to ebooks.

Skim the table of contents, and try to understand the structure of the book.
Read the book and use a pencil to mark interesting sentences or paragraphs with a line in the margin (Fig. 1). In rare cases I use double lines to highlight especially noteworthy ideas. This helps later to quickly re-familiarise myself with the main concepts of a chapter without having to fully re-read it. I don’t typically write a lot of notes into the margin because there is usually not a lot of space available.

Figure 1. I use a pencil to mark relevant sentences in the margin and occasionally add some notes.

After finishing one or several chapters, I review the marked paragraphs and summarise the main points of each chapter on the empty space at the beginning or end of the chapter (Fig. 2). By writing the notes into the book I get two benefits. Firstly, there is a higher chance for these analog notes to still be accessible to me in a decade. Secondly, these notes might be interesting to friends that I lend the book to.

Figure 2. I summarize the main points of each chapter on the blank space at the beginning or end of the chapter.

Once I have finished the book, I transfer the chapter notes into a markdown file (in Joplin ). While copying the notes I sometimes rework or modify them. Since 2025 I actually take a photo of the handwritten notes and ask Gemini/Claude/GPT to produce a markdown file for me. I might also link to other digital notes that have a connection to the book.
Optional: If I want to share my notes with an audience I will write about the major learnings from the notes and bring them into context with concepts from other books/articles/experiences.
Optional: Most books will have some learnings that I want to periodically and actively recall. For those I create atomic Anki cards which will prompt me to regularly review the concepts. I am trying to write good prompts for these cards. This can be a mix of facts like “Where was Napoleon born?” or higher level concepts I don’t want to forget, e.g. “Give an example of an infinite game as defined by J. P. Carse”.

The most important rule for this note taking system is that I arbitrarily break the rules for certain books and skip some of these steps when I feel they are not worth the time. However, this system ensures that I do multiple passes of the content, that I have both durable analog and searchable digital notes, and that I am periodically prompted for the most interesting concepts.

A great weekend

Sun, 25 Feb 2024 00:00:00 +0000

When you start working you suddenly put much higher value on the “free” weekend hours. At the beginning this sometimes meant that I was quite restless and felt like those valuable hours had to be used optimally. Of course this way of thinking was not relaxing at all.

Over time I realised that the most rewarding/recharging weekends are those that involve random unplanned activities, other people/community, and hobbies done for their own sake (atelic activities).

My great weekends often involve one or more of the following activities:

spending time in nature, e.g. hiking on a local trail or going to the nearest park or forest
exploring a new neighbourhood in my city without a particular goal in mind
cooking a nice meal for myself, my wife and/or friends
completing a household chore at the beginning of your day
meeting a friend for a coffee/museum walk
organising a day trip to somewhere else
small talk with someone in my local community, e.g. one of my neighbours
messaging or calling a friend who doesn’t live close
exercising (running, cycling, swimming, gym)
playing or learning a board game (my recommendations: Hive , Robinson Crusoe , Wingspan , Spirit Island )
watching a documentary with full attention
fixing something broken at home, e.g. my bicycle, water filter, squeaky door, broken light
reading a fiction or history book
reading the weekly edition of a newspaper (especially a section of the paper I would normally skip)
listening to the radio or a music genre that I would not normally listen to
following a yoga / stretching / meditation routine
learning and taking notes about a topic that is not relevant for my work or career (learning for the sake of learning)
practicing a different language
calling my parents
volunteering at a local event

DuckDB use cases for data scientists: Querying remote S3 files

Sun, 25 Feb 2024 00:00:00 +0000

DuckDB is a pretty cool in-process OLAP analytical database that I started to spin up on the fly for quick data analysis. What SQLite is to Postgres, DuckDB is to Snowflake. It is a single executable without dependencies and stores databases in local files.

I can think of four use cases for data science work:

DuckDB supports larger-than-memory workloads by loading data sequentially. You can use it to analyse datasets that are too large for Pandas (and too small to justify PySpark).
I can query CSV, parquet, and JSON files directly from remote endpoints, e.g. S3, using SQL.
I can replace Snowflake queries with DuckDB queries in unit / integration tests.
I can set up a (DuckDB + dbt) data warehouse for local development.

I want to share here my workflow for the second use case. Inspecting parquet files in AWS S3 is a pain because I can’t easily inspect them in the AWS console. For a few months now I have used DuckDB to load, inspect, and analyse parquet files from the command line. I found this reduced my cognitive load in situations where I quickly want to check a remote file, because I don’t have to download the parquet file and write a Python script to inspect it.

Installing DuckDB

I use macOS and can install DuckDB via homebrew: brew install duckdb. To work with remote files I also install the httpfs extension.

I start the DuckDB shell with duckdb and run the SQL commands:

INSTALL httpfs;
LOAD httpfs;

Authentication with AWS

To load files from S3, I need to configure AWS credentials. I am assuming here that your workplace has configured AWS SSO temporary credentials, but this works also with static credentials (ACCESS_KEY_ID, SECRET_ACCESS_KEY). There are two ways of doing this:

1. Set AWS credentials in the session:

I can set AWS credentials inside a DuckDB session like this:

LOAD httpfs;
SET s3_region = 'eu-west-2';
SET s3_access_key_id ='???';
SET s3_secret_access_key ='???';

2. Let aws vault handle credentials

I use the AWS credentials management tool aws vault (see here ) which manages my temporary AWS credentials in the background and exposes them to my shell sub-process. I create the DuckDB shell with aws-vault exec [profile-name] -- duckdb which ensures that AWS credentials are automatically set and updated.

Querying parquet files

Let’s assume that I have multiple parquet files stored in an S3 bucket work-project and they all have the same schema:

s3://work-project/clickdata_001.parquet
s3://work-project/clickdata_002.parquet
[...]
s3://work-project/clickdata_100.parquet

These files might be placed in this S3 bucket as part of an ETL pipeline before they are loaded into a table in a data warehouse. Let’s imagine I want to investigate a bug in the pipeline and need to inspect the files.

Inspect schema of parquet file

To familiarise myself with the schema I run parquet_schema

LOAD httpfs; -- once at the beginning of the session
SELECT * FROM parquet_schema('s3://work-project/clickdata_001.parquet');

which will return:

clicked_at::timestamp
user_id::int
event_type::string

Query a parquet file

I can use postgres SQL dialect to query the parquet files with the read_parquet function, e.g.

--- count number of events in the file
SELECT count(*) FROM read_parquet('s3://work-project/clickdata_001.parquet');

--- find the last event time of a particular user
SELECT max(clicked_at) FROM read_parquet('s3://work-project/clickdata_001.parquet') where user_id = 1234;

Query multiple parquet files

Say I want to find the users with the highest number of events across all files. I can use glob syntax with * to run a query against all files as if they were one table:

SELECT user_id, count(*) as num_events FROM read_parquet('s3://work-project/*.parquet') group by user_id order by num_events desc;

Create table from files

Let’s assume that all clickdata files together have a size of 500MB. It would be annoying if I had to download these files for every query that I want to run. Let’s instead create a table from the files.

create table clickdata as select * from read_parquet('s3://work-project/*.parquet');

This table is stored in memory until we close the shell, which allows me to run different queries against it until I am done with my analysis, e.g.

select * from clickdata;

I can also store the table in a local database on my computer in case I need to work with the data for a longer time. The following SQL statement

EXPORT DATABASE 'mydatabase' (FORMAT PARQUET);

will create a local folder mydatabase which stores the tables currently in memory. In a later DuckDB session I can reload the table from the database using the following command

IMPORT DATABASE 'mydatabase';

Query column statistics

A nice feature of the parquet file format is that it stores statistics about each column , e.g. min, max, null_count, distinct_count. If you are interested, you can inspect parquet file metadata using select * from parquet_metadata('file.parquet'); Assume the click event dataset is several GB large and I want to identify the earliest clicked_at time. I would run the query below.

SELECT min(clicked_at) FROM read_parquet('s3://work-project/*.parquet');

Fortunately, DuckDB uses the column statistics stored in the parquet files to compute the answer without having to download the whole dataset from S3. Simon Willison shows a more extreme example of this feature.

DuckDB is a great tool for quick investigations of remotely hosted files. Especially after having configured automatic handling of AWS credentials I can spin up a DuckDB shell with one command. I hope it becomes a time saver for you too.

How to display Jupyter notebooks on your Hugo blog

Sat, 20 Jan 2024 00:00:00 +0000

I like the simplicity and ease of use of Hugo, the static site generator that powers this blog. However, as a data scientist I want to be able to make an argument using code, data, and graphs in a Jupyter notebook. This post explains how I convert a Jupyter notebook into a blog post for this website, such as this . Before I started, my requirements were:

I wanted to be able to easily share both the original notebook (and accompanying data) as well as the website version.
I wanted to regenerate the website version from the notebook file with at most one command and without any post-processing. This is important because I usually work iteratively on notebooks.

Writing the notebook

The source code for this blog is available in this Gitlab repository . I created a new top-level folder notebooks/ and a subfolder for each notebook project I want to post about. This allows me to easily link to the Jupyter notebook and data to make it reproducible.

It is important that you start the notebook with a cell that contains the front matter of the blog post, e.g.

# My notebook title

Date: 2018-06-01  
Author: firstname lastname  
Categories: category1, category2  
Tags: tag1, tag2, tag3  
<!--eofm-->

The  is important to divide the front matter from the rest of the notebook.

Converting to markdown

To align with the markdown-based workflow of Hugo we need to convert the notebook to markdown. The script nb2hugo does exactly that. We can install it for example via: pip install nb2hugo.

To convert the notebook to markdown we navigate into the notebook folder and run

nb2hugo covid_bipartisan_bayesian.ipynb --site-dir /Users/rob/hugoblog/ --section posts

This command writes the output markdown file covid_bipartisan_bayesian.md into the specified folder posts/.

Adjusting the website CSS

I made two more adjustments to the style of this blog to better display the “markdown-ified” notebook. Both pandas code and displayed dataframes are quite wide so I:

increased the content width slightly to 800px,
decreased the font-size for the text in the code cells to 0.75rem.

While still not perfect, I think this gives a decent result.

Reproducing Nate Silver's regression analysis on COVID death rates

Mon, 20 Nov 2023 00:00:00 +0000

I spent some time recently studying causal inference methods. Two great resources for this are:

Causal Inference for The Brave and True (frequentist view)
Statistical Rethinking (Bayesian view).

When I learn about an unfamiliar topic, I will always try to look for real-world examples. In this case I keep my eyes open about articles sharing statistical findings, where I can attempt to reproduce the results.

In Nate Silver’s recent article Fine, I’ll run a regression analysis. But it won’t make you happy he shows the effect of state partisanship and COVID vaccination rates on COVID death rates. The models used are simple linear regressions with one to three independent variables (partisanship, age, vaccination rates).

Silver’s point in the article is only partially about his finding that state partisanship is a good predictor for COVID death rates in that state. Instead, he tries to dismiss critics that argue that simple models are not valid when they leave out additional variables, e.g. age, co-morbidities. While he agrees that one has to justify a particular model design, simple models can deliver true insights while standing up to scrutiny. Moreover he adds that “there’s a general tendency [in the profession] to overfit models.”

This post is not about Silver’s point or the political aspect of the result. I simply wanted to reproduce the findings in the article.

In his article Silver runs four linear regressions:

impact of state partisanship on COVID death rates
impact of state partisanship and state age on COVID death rates
impact of state partisanship, state age, and state vaccination rates on COVID death rates
impact of state age and state vaccination rates on COVID death rates

We need to collect data for every US state on partisanship, age structure, vaccination rates, and COVID death rates. It turned out, perhaps not surprisingly, that finding and assembling the dataset from open sources took significantly more time than setting up the regressions. I had three issues when I tried to assemble the data from the sources that the article links to:

one source didn’t allow me to download the data as a file and I didn’t want to scrape or copy-paste from the website
one source didn’t allow me to view the data on the same reference day as in the article
one source didn’t provide the data as a file for free

So I had to use in some cases different sources.

Building the data set

We are building the data set from four sources:

US states COVID cases and deaths: https://github.com/nytimes/covid-19-data/blob/62ef34cfcb60214be873a38d73619da9ea57d50b/us-states.csv
US states age statistics: https://www.prb.org/resources/which-us-states-are-the-oldest/
US states vaccination rates: ourworldindata.org (archive.org version)
US states election results 2020: https://www.kaggle.com/code/paultimothymooney/2020-usa-election-vote-percentages-by-state/output

COVID death rates

We want to calculate the COVID death rates (COVID deaths per 1M population) between two points in time (1. February 2021 when vaccines became widely available and 23. March 2023 as in the article). Silver links to worldometer.info as his dataset for covid deaths per state. However, I couldn’t find a way to download the data in timeseries form (numbers for each day) without scraping the website. Instead we are using the equivalent numbers published by the New York Times. Let’s load the CSV file into a pandas dataframe:

import pandas as pd
import numpy as np
import statsmodels.api as sm
import matplotlib.pyplot as plt

df = pd.read_csv("us-states.csv")

df['date'] = pd.to_datetime(df['date'])
# create two series for the two dates of interest and rename the columns
deaths_february = df[df.date == '2021-02-01'][["state", "deaths"]].rename(columns = {"deaths" : "deaths_2021_02_01"})
deaths_last = df[df.date == '2023-03-23'][["state", "deaths"]].rename(columns = {"deaths" : "deaths_2023_03_23"})

With the two pandas series deaths_february and deaths_last we can calculate the number of deaths in this time period.

deaths = deaths_last.merge(deaths_february, how = 'left', on = 'state').dropna(axis = 'index')
deaths["deaths_after_vaccine"] = deaths["deaths_2023_03_23"] - deaths["deaths_2021_02_01"]

Now we can look at the state numbers for COVID deaths between 01.02.2021 and 23.03.2023:

deaths

	state	deaths_2023_03_23	deaths_2021_02_01	deaths_after_vaccine
0	Alabama	21631	7688.0	13943.0
1	Alaska	1438	253.0	1185.0
3	Arizona	33190	13124.0	20066.0
4	Arkansas	13068	4895.0	8173.0
5	California	104277	41284.0	62993.0
6	Colorado	14245	5737.0	8508.0
7	Connecticut	12270	7119.0	5151.0
8	Delaware	3352	1101.0	2251.0
9	District of Columbia	1432	916.0	516.0
10	Florida	87141	26684.0	60457.0
11	Georgia	41055	13821.0	27234.0
12	Guam	416	130.0	286.0
13	Hawaii	1851	407.0	1444.0
14	Idaho	5456	1737.0	3719.0
15	Illinois	41618	21273.0	20345.0
16	Indiana	26179	9989.0	16190.0
17	Iowa	10770	4906.0	5864.0
18	Kansas	10232	3809.0	6423.0
19	Kentucky	18348	3995.0	14353.0
20	Louisiana	18835	8912.0	9923.0
21	Maine	2981	595.0	2386.0
22	Maryland	16672	7154.0	9518.0
23	Massachusetts	24441	14607.0	9834.0
24	Michigan	42311	15527.0	26784.0
25	Minnesota	14964	6270.0	8694.0
26	Mississippi	13431	6056.0	7375.0
27	Missouri	23998	7182.0	16816.0
28	Montana	3701	1235.0	2466.0
29	Nebraska	5068	2031.0	3037.0
30	Nevada	12093	4281.0	7812.0
31	New Hampshire	3018	1059.0	1959.0
32	New Jersey	36097	21513.0	14584.0
33	New Mexico	9110	3297.0	5813.0
34	New York	80109	43354.0	36755.0
35	North Carolina	29746	9397.0	20349.0
36	North Dakota	2529	1447.0	1082.0
37	Northern Mariana Islands	41	2.0	39.0
38	Ohio	42061	11233.0	30828.0
39	Oklahoma	16549	3564.0	12985.0
40	Oregon	9451	1980.0	7471.0
41	Pennsylvania	50701	21741.0	28960.0
42	Puerto Rico	5848	1836.0	4012.0
43	Rhode Island	3915	2173.0	1742.0
44	South Carolina	20192	7283.0	12909.0
45	South Dakota	3222	1778.0	1444.0
46	Tennessee	29035	9660.0	19375.0
47	Texas	94518	37405.0	57113.0
48	Utah	5316	1669.0	3647.0
49	Vermont	939	175.0	764.0
50	Virgin Islands	130	24.0	106.0
51	Virginia	23782	6474.0	17308.0
52	Washington	15905	4404.0	11501.0
53	West Virginia	8132	2028.0	6104.0
54	Wisconsin	16485	6439.0	10046.0
55	Wyoming	2014	596.0	1418.0

Age characteristics

We load data for US states population age characteristics and calculate the proportion of the state population aged 65 or older. We would expect that states with an older population are more severely affected by COVID (all else being equal).

states_population = pd.read_csv("states_population_age_2020.csv", thousands=',').drop(["Rank", "Population Ages 65+ (percent of state population)"], axis = 1)
states_population = states_population.rename(columns = {"State" : "state", "Total Resident Population (thousands)": "total_population_thousands", "Population Ages 65+ (thousands)" : "population_age65_thousands"})
states_population["perc_over_65"] =  states_population["population_age65_thousands"] / states_population["total_population_thousands"] * 100.0

Next we add the perc_over_65 column to the deaths dataframe.

# drop due to missing population data for DC, Guam, Puerto Rico, Northern Mariana Islands, Virgin Islands
deaths_and_states = deaths.merge(states_population[["state", "total_population_thousands", "perc_over_65"]], on = 'state', how = 'left').dropna()

Moreover, to use the same variable scaling as in the article, we scale the deaths_after_vaccine variable from absolute numbers to be COVID deaths per million population.

deaths_and_states["deaths_per_million"] = deaths_and_states["deaths_after_vaccine"] / ((deaths_and_states["total_population_thousands"] * 1000) / 1_000_000 )

# dataframe with deaths, population, and age variable
deaths_and_states.head(5)

	state	deaths_2023_03_23	deaths_2021_02_01	deaths_after_vaccine	total_population_thousands	perc_over_65	deaths_per_million
0	Alabama	21631	7688.0	13943.0	4922.0	17.757009	2832.791548
1	Alaska	1438	253.0	1185.0	731.0	13.132695	1621.067031
2	Arizona	33190	13124.0	20066.0	7421.0	18.515025	2703.948255
3	Arkansas	13068	4895.0	8173.0	3031.0	17.683933	2696.469812
4	California	104277	41284.0	62993.0	39368.0	15.179841	1600.106686

Presidential election results

Similar to the article we want to generate a variable biden which indicates Joe Biden’s margin of victory over Donald Trump in the 2020 elections.

# load election results by state for 2020 presidential election
election_results = pd.read_csv("democratic_vs_republican_votes_by_usa_state_2020.csv").drop("usa_state_code", axis = 1)
election_results["biden"] = election_results["percent_democrat"] - 50
election_results.head(10)

	state	DEM	REP	usa_state	percent_democrat	biden
0	Alabama	843473	1434159	Alabama	37.032892	-12.967108
1	Alaska	45758	80999	Alaska	36.098993	-13.901007
2	Arizona	1643664	1626679	Arizona	50.259682	0.259682
3	Arkansas	420985	761251	Arkansas	35.609218	-14.390782
4	California	9315259	4812735	California	65.934760	15.934760
5	Colorado	1753416	1335253	Colorado	56.769307	6.769307
6	Connecticut	1059252	699079	Connecticut	60.241900	10.241900
7	Delaware	295413	199857	Delaware	59.646859	9.646859
8	District of Columbia	258561	14449	District of Columbia	94.707520	44.707520
9	Florida	5294767	5667834	Florida	48.298456	-1.701544

# merge to deaths dataframe
death_and_election = deaths_and_states.merge(election_results[["state", "biden"]], how = "left", on = 'state')

Vaccination rates

Lastly, we add the vaccination rates per state after the pandemic was over. I use the vaccination rates from 10.5.2023 as this data was easily available.

vaccinations_raw = pd.read_csv("us_state_vaccinations.csv").rename(columns={"location": "state"})
vaccinations_raw.replace({'New York State': 'New York'}, inplace=True)
# chose date and relevant columns
vaccinations = vaccinations_raw[vaccinations_raw.date == '2023-05-10'][["state","people_vaccinated_per_hundred"]]

Next, we merge the vaccination rates with the dataframe to get our final dataset.

dataset = death_and_election.merge(vaccinations, how = "left", on = 'state')
dataset[["state", "deaths_per_million", "biden", "perc_over_65", "people_vaccinated_per_hundred"]].head(5)

	state	deaths_per_million	biden	perc_over_65	people_vaccinated_per_hundred
0	Alabama	2832.791548	-12.967108	17.757009	65.12
1	Alaska	1621.067031	-13.901007	13.132695	73.23
2	Arizona	2703.948255	0.259682	18.515025	78.37
3	Arkansas	2696.469812	-14.390782	17.683933	70.09
4	California	1600.106686	15.934760	15.179841	85.07

I am also storing the final dataset as a parquet file in case you want to play with it. To find it, follow the link at the bottom of the page.

dataset[["state", "deaths_per_million", "biden", "perc_over_65", "people_vaccinated_per_hundred"]].to_parquet("death_partisanship_final_dataset.parquet", engine = 'pyarrow')

Regressions

With the assembled dataset we can now reproduce Nate Silver’s findings.

1. Regression: state partisanship on death rates

Silver starts with a one-variable baseline model. And indeed biden, Joe Biden’s winning margin, is a statistically significant predictor of COVID deaths. According to the model, a one-percentage-point increase in Biden’s margin reduced expected COVID deaths by ~30 covid deaths per million population.

x = dataset[['biden']]
y = dataset['deaths_per_million']
# add a constant term to the model
x = sm.add_constant(x)
# run regression
model = sm.OLS(y, x).fit()
print_model = model.summary()
print(print_model)

                            OLS Regression Results                            
==============================================================================
Dep. Variable:     deaths_per_million   R-squared:                       0.286
Model:                            OLS   Adj. R-squared:                  0.272
Method:                 Least Squares   F-statistic:                     19.27
Date:                Sun, 14 Jan 2024   Prob (F-statistic):           6.22e-05
Time:                        18:05:09   Log-Likelihood:                -381.14
No. Observations:                  50   AIC:                             766.3
Df Residuals:                      48   BIC:                             770.1
Df Model:                           1                                         
Covariance Type:            nonrobust                                         
==============================================================================
                 coef    std err          t      P>|t|      [0.025      0.975]
------------------------------------------------------------------------------
const       2031.3050     72.204     28.133      0.000    1886.129    2176.481
biden        -29.7773      6.784     -4.389      0.000     -43.418     -16.137
==============================================================================
Omnibus:                        0.718   Durbin-Watson:                   2.263
Prob(Omnibus):                  0.698   Jarque-Bera (JB):                0.732
Skew:                          -0.266   Prob(JB):                        0.694
Kurtosis:                       2.737   Cond. No.                         10.8
==============================================================================

Notes:
[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.

We can also plot the data set and the regression line to get a more intuitive overview of the result:

fig, ax = plt.subplots(figsize = (9, 9))
ax.scatter(dataset["biden"],  dataset["deaths_per_million"])
plt.xlabel("Margin % for Biden (positive values mean Democrats won)")
plt.ylabel("Post vaccine (Feb 2021) deaths per 1M population")
b, m = model.params
ax.axline(xy1=(0, b), slope=m, color='red')
plt.grid()

2. Regression: state partisanship and age on death rates

Silver’s article is a response to criticism that the model above is not valuable because different states have different age structures and that may explain most of the variation in death rates. Silver remarks that the results hold even when controlling for age. We can add age perc_over_65 to the model and re-run the regression and indeed we get the same result. biden is still significant with a similar coefficient.

x2 = dataset[['biden', 'perc_over_65']]
y2 = dataset['deaths_per_million']

x2 = sm.add_constant(x2)
model2 = sm.OLS(y2, x2).fit()
print_model2 = model2.summary()
print(print_model2)

                            OLS Regression Results                            
==============================================================================
Dep. Variable:     deaths_per_million   R-squared:                       0.397
Model:                            OLS   Adj. R-squared:                  0.371
Method:                 Least Squares   F-statistic:                     15.44
Date:                Sun, 14 Jan 2024   Prob (F-statistic):           6.99e-06
Time:                        18:05:12   Log-Likelihood:                -376.95
No. Observations:                  50   AIC:                             759.9
Df Residuals:                      47   BIC:                             765.6
Df Model:                           2                                         
Covariance Type:            nonrobust                                         
================================================================================
                   coef    std err          t      P>|t|      [0.025      0.975]
--------------------------------------------------------------------------------
const          256.0210    609.788      0.420      0.677    -970.714    1482.756
biden          -32.9481      6.397     -5.151      0.000     -45.817     -20.079
perc_over_65   101.6118     34.690      2.929      0.005      31.824     171.400
==============================================================================
Omnibus:                        1.194   Durbin-Watson:                   2.318
Prob(Omnibus):                  0.551   Jarque-Bera (JB):                1.061
Skew:                          -0.162   Prob(JB):                        0.588
Kurtosis:                       2.365   Cond. No.                         162.
==============================================================================

Notes:
[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.

3. Regression: state partisanship, age, and vaccination rates on death rates

Clearly your political leanings are not a predictor for your chance of dying from COVID. The assumption behind the model is that Republicans have been less likely to get vaccinated, and unvaccinated humans have a higher mortality w.r.t COVID. We can check what happens if we add the vaccination rates as a variable to the model.

We can see below that vaccination rate is significant and explains away the effect of the biden variable.

x3 = dataset[['biden', 'perc_over_65', 'people_vaccinated_per_hundred']]
y3 = dataset['deaths_per_million']
x3 = sm.add_constant(x3)
model3 = sm.OLS(y3, x3).fit()
print_model3 = model3.summary()
print(print_model3)

                            OLS Regression Results                            
==============================================================================
Dep. Variable:     deaths_per_million   R-squared:                       0.450
Model:                            OLS   Adj. R-squared:                  0.414
Method:                 Least Squares   F-statistic:                     12.55
Date:                Sun, 14 Jan 2024   Prob (F-statistic):           4.01e-06
Time:                        18:05:13   Log-Likelihood:                -374.63
No. Observations:                  50   AIC:                             757.3
Df Residuals:                      46   BIC:                             764.9
Df Model:                           3                                         
Covariance Type:            nonrobust                                         
=================================================================================================
                                    coef    std err          t      P>|t|      [0.025      0.975]
-------------------------------------------------------------------------------------------------
const                          1528.3900    841.133      1.817      0.076    -164.723    3221.503
biden                           -15.9990     10.110     -1.583      0.120     -36.349       4.351
perc_over_65                    114.7298     34.042      3.370      0.002      46.207     183.253
people_vaccinated_per_hundred   -18.5072      8.743     -2.117      0.040     -36.106      -0.908
==============================================================================
Omnibus:                        0.262   Durbin-Watson:                   2.192
Prob(Omnibus):                  0.877   Jarque-Bera (JB):                0.454
Skew:                           0.027   Prob(JB):                        0.797
Kurtosis:                       2.537   Cond. No.                     1.08e+03
==============================================================================

Notes:
[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.
[2] The condition number is large, 1.08e+03. This might indicate that there are
strong multicollinearity or other numerical problems.

4. Regression: age and vaccination rates on death rates

In a final step, we remove the correlated variable biden from the model to demonstrate the impact of age and vaccination rate on death rate:

x4 = dataset[['perc_over_65', 'people_vaccinated_per_hundred']]
y4 = dataset['deaths_per_million']
x4 = sm.add_constant(x4)

model4 = sm.OLS(y4, x4).fit()

print_model4 = model4.summary()
print(print_model4)

                            OLS Regression Results                            
==============================================================================
Dep. Variable:     deaths_per_million   R-squared:                       0.420
Model:                            OLS   Adj. R-squared:                  0.396
Method:                 Least Squares   F-statistic:                     17.03
Date:                Sun, 14 Jan 2024   Prob (F-statistic):           2.74e-06
Time:                        18:05:14   Log-Likelihood:                -375.95
No. Observations:                  50   AIC:                             757.9
Df Residuals:                      47   BIC:                             763.6
Df Model:                           2                                         
Covariance Type:            nonrobust                                         
=================================================================================================
                                    coef    std err          t      P>|t|      [0.025      0.975]
-------------------------------------------------------------------------------------------------
const                          2386.5532    653.203      3.654      0.001    1072.478    3700.629
perc_over_65                    117.0242     34.551      3.387      0.001      47.516     186.532
people_vaccinated_per_hundred   -29.4653      5.423     -5.434      0.000     -40.374     -18.556
==============================================================================
Omnibus:                        0.653   Durbin-Watson:                   2.211
Prob(Omnibus):                  0.722   Jarque-Bera (JB):                0.527
Skew:                           0.244   Prob(JB):                        0.768
Kurtosis:                       2.881   Cond. No.                         829.
==============================================================================

Notes:
[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.

Jupyter Notebook

You can find the Jupyter notebook and the datasets for this post here .

On Writing Well - 80/20 Checklist

Mon, 14 Aug 2023 00:00:00 +0100

This is a checklist that I use at work to quickly sense check my drafts of strategy documents, tech proposals, PR reviews, project feedback, and other places where effective writing is helpful. It lists suggestions from the book On Writing Well that I covered in this blog post . As I mentioned in the blog post I am not striving for perfection. Instead I want to be able to identify the main parts of my first draft that I can improve within 5–20 minutes of editing time.

Clear Thinking

I am clear about what the main point of the article is, who my audience is, and why they should care.

Structure

I have paid special attention to the first sentence and made it interesting.
The first paragraph hooks the reader by being fresh, novel, paradoxical, humorous, surprising, unusual or starting with an interesting fact or question.
The first paragraph tells the reader what the article is about and why they should care.
Every paragraph is kept reasonably short and captures one logical idea.
The last sentence of each paragraph entices the reader to keep reading.
The article is not longer than it needs to be. It does not attempt to cover every aspect of the topic.
I paid attention to the end. The end happens in a fitting, unexpected or surprising way that should keep the reader thinking about the text.

Unity

The text is mostly written in one tense.
The reader is addressed with the same pronoun.
The tone is not changed (casual vs. formal, neutral vs. involved).

Words

Every word is essential for the sentence and doing new work.
Unnecessary adverbs are avoided.
Unnecessary adjectives are avoided, e.g. “diligent code review”.
Words that have shorter alternatives are replaced, e.g. “assistance” (help), “numerous” (many), “facilitate” (ease), “sufficient” (enough), “attempt” (try).
Words that inflate importance are avoided, e.g. “with the possible exception of” (except), “due to the fact that” (because).
Small qualifier words are removed: “a bit,” “a little,” “sort of,” “kind of,” “rather,” “quite,” “very,” “too,” “pretty much,” “in a sense”. Be confident in what you write.
Active verbs are used over passive verbs, e.g. “Joe documented the architecture” over “The architecture was documented by Joe”.
Concept nouns are replaced by active verbs. Instead of “The monitoring system is used to detect data drift.” use “We monitor our data to detect drift.”
Verbs are precise: “The CEO resigned” instead of “The CEO left”.

Style

Sentences are kept short.
Sentences link logically to the next. If not, a link is explicitly provided.
Avoid exclamation points unless for effect.
Use contractions like “I’ll”, “I’ve”, but not “I’d” as this can mean both “I had” and “I would”.
Always use “that” over “which”.

Tone

Avoid business lingo and concept nouns, e.g. “incident management procedures”. Replace them with active verbs and plain talk.
Try to write in a human way. Make people do things using active verbs.
Resist trying to sound smart in work documents.
Avoid sexism in language: “Software Engineers can spend more time with their families.” instead of “Software Engineers can spend more time with their wives and children.”

What I learned from On Writing Well

Sat, 12 Aug 2023 00:00:00 +0100

Writing a good blog post that summarises William Zinsser’s classic book On Writing Well is stressful. I don’t want to break his rules while writing about them. I will attempt this anyway since I learned a lot from his book.

Notes that I write for myself are pretty sloppy, unstructured, and stream-of-consciousness, but I care about my writing when others read it. Unfortunately, my most regular readers at the moment are probably my work colleagues. I often draft project or strategy documents that others need to understand and engage with. I am trying hard to keep the style non-corporate and maybe even a bit fun to read.I probably use too many self-made memes.I don’t think William Zinsser considered tech guys writing data strategy documents part of his target audience, but his advice is universal.

William Zinsser argues in his book that an average reader has an attention span of 30 secondsAn estimate he made in the pre-social media year of 2006. and that their attention is competing with a dozen alternative activities. He argues that the key to good writing is clear and engaging structure, simplicity, and, in the context of business writing, humanity. I will share my takeaways and a checklist that I use during my writing process.

Structure - Clear thinking produces clear writing

When writing, you should think clearly about what you want to say. Illogical and bad writing is a result of being unsure about the purpose and structure of the text. We all can relate. Writing is an iterative process. You often start an article with different ideas and arguments than when you reach the end. During the writing process you come up with better arguments that you want to include or that you can explain better. This is natural and you should enjoy the process of editing and deleting to ensure your first draft gets a coherent structure.

Ask yourself: “Did I write what I wanted to say?” and “Can my reader easily follow my narrative from the first paragraph to the last?” The second question relates to the earlier idea of a lazy reader, who will stop reading if they are not hooked or need to spend brain power to follow your train of thought. How do you avoid that?

The most important sentence of your writing is the first one. It must induce the reader to read the second sentence. The second sentence must do the same for the third sentence and so on. The goal is to hook the reader in the first paragraph. Zinsser calls this “the lead” and it should achieve two things:

To be an effective hook it should be: fresh, novel, paradoxical, humorous, surprising, unusual or be an interesting fact or question. It needs to force the reader to keep reading.
It needs to tell the reader what the article is about and why they should care.

Once the reader is hooked, ensure that the last sentence of each paragraph entices them to continue to the next. The end of a paragraph is often a natural stopping point. The paragraphs should be kept short and reflect one idea. Once you have said what you wanted to say, stop. Be comfortable dropping material. Decide which part of the subject you want to cover. Cover it well and then stop.

How do you end your piece the right way? Try to encapsulate the main idea of the text and end in a fitting, unexpected or surprising way. Like a good dessert, the last paragraph or sentence should be a joy in itself and linger for a moment after the end.

Simplicity - Make it easy for your readers

Like good structure, simplicity helps your reader stay engaged. Unfortunately, while you write you will accumulate clutter. This can be words that do not add value or sentences that are difficult to follow. You can achieve simplicity by ensuring that:

every word in a sentence is doing new work
sentences are short and logically linked together.
you have unity in choice of tense, pronouns, and style

Zinsser argues that writing improves proportionally to the number of things you can keep out of it that should not be there. Be critical of the words you choose and avoid:

Words that do not do extra work, e.g. “a personal friend of mine” does not add more than “a friend”.
Unnecessary adverbs. “I wrote up the documentation”. “up” is not required. “My algorithm is decidedly better than brute-force”. “decidedly” does not add anything.
Avoid adjectives unless absolutely necessary, e.g. do not write “diligent code review” unless your company does not care about code reviews.
Words that have shorter alternatives, e.g. “assistance” (help), “numerous” (many), “facilitate” (ease), “sufficient” (enough), “attempt” (try).
Words that inflate importance, e.g. “with the possible exception of” (except), “due to the fact that” (because), “he totally lacked the ability to” (he couldn’t), “for the purpose of” (for).
Remove the small words that qualify how you feel and how you think and what you saw: “a bit,” “a little,” “sort of,” “kind of,” “rather,” “quite,” “very,” “too,” “pretty much,” “in a sense”.
Prefer active verbs over passive, e.g. “Joe documented the architecture” over “The architecture was documented by Joe”.
Nouns that express a concept are commonly used in bad writing instead of verbs that tell what somebody did. Instead of “The monitoring system is used to detect data drift.” use “We monitor our data to detect drift.”
Use precise verbs: “Start a company” instead of “Set up a company”. “The CEO resigned” or “The CEO was fired” instead of “The CEO stepped down”.

After applying these rules your sentences should be clear and stripped of clutter. Now your job is to ensure that sentence B follows logically from sentence A. Also ensure that sentence F does not repeat the same argument made in sentence A. If the connections are not clear, provide the missing link. My favourite quote in the book is on the question of sentence length. Zinsser advises: “If you want to write long sentences, be a genius.”

Aside from a careful choice of words and logical sentences, you should strive for unity in:

tense. Stick to one tense.
pronouns. Use the same pronoun to address your reader.
tone. Are you writing casually or formally, involved or detached, ironic or amused?
style. Are you writing a Wikipedia entry, a personal travel story, or a tech strategy?

Decide on these points at the beginning and do not change unless necessary.

Humanity - Plain talk, not vanity

After you have internalised the advice from the previous two sections you can start applying it at work. What could go wrong?

People at work have a tendency to write in a pretentious style, e.g. “The company uses evaluative procedures for our objectives based on our KPIs.”

Instead of dead concept nouns like “evaluative procedures” use active verbs and plain talk, e.g. a better way is “We will evaluate our progress based on our KPIs.” You should aim to stay natural and write how you talk. A good test is to ensure your colleagues can visualise who is doing what when they read your sentence.

To incorporate Zinsser’s advice in my own writing, I created this checklist that I refer to after completing a first draft. I am not striving for perfection. I don’t want to become an author for The New Yorker magazine or win the Nobel Prize in Literature. Instead, I want to focus on the ~20% of changes that lead to the largest quality improvement.

Posts on staticnotes.org

Encrypted cloud backups with restic

Requirements for my cloud backup

Getting cheap space in the cloud

Installing and configuring restic

Basic restic backup commands

Inspecting the storage box

Working with snapshots

Restoring backups

Managing the repository’s size and integrity

Scheduling and profiles with resticprofile

Conclusion

Appendix

Launchd schedules with sleeping MacBook (closed lid)

Locks

Exclusion file

Visualising London rental bike location data

Collecting the data

Convert to DuckDB table

Busiest stations

Busiest hours of the day

Storing geometric data as parquet files

Visualising availability using QGIS

Visualising busy stations

Rental bike availability during commute hours

Conclusion

References

Jupyter Notebook

UK house price changes in real terms

Querying the housing price data

Inspecting the data

Computing real house prices

Visualising nominal and real house prices over time

Detached Houses in South England Counties

Flats in London Boroughs

Terraced houses in London Boroughs

Winners and Losers in real terms (last 3 years)

Conclusion

Appendix

Jupyter Notebook

Building a local data warehouse with DuckDB, dbt, and Superset

Loading the raw data into DuckDB

Setting up dbt for data transformations

dbt models

Check tables were created

Superset for data analytics dashboards

Install Superset with DuckDB support

Configure Superset

Add datasets in Superset

Create dashboard in Superset

Conclusion

Troubleshooting

Book notes: Poor Charlie’s Almanack

Advocating interdisciplinary collaboration between university departments

Fraud and advice for endowment funds

(Inverted) Advice for Graduates

Mental checklist to analyse problems

Conclusion

About this blog and its microfeatures

Static site generator

Supporting math, code, Jupyter notebooks

Content types

Microfeatures

Recommendations

Sidenotes

Markers for external links

Link-blog / retweets / boosts

RSS

Table of Contents

Linkable Headers

Drawings

Prompts / LLM responses

How related posts are computed

Noteworthy

Example

Embeddings

Code

Update

Book notes: How Big Things Get Done by Bent Flyvbjerg and Dan Gardner

Planning is cheap, building is expensive

Installing and configuring `restic`

Scheduling and profiles with `resticprofile`