Florian Marending

Self hosting maps

Sat, 22 Aug 2026 22:00:00 GMT

23 Aug 2026

Self hosting maps

PMTiles are delightfully boring

For an upcoming project, I’m going to need a world map in a web app to display location data. I’ve done that before, using mapbox as the tile server, but this time I want to serve the map data myself. A couple of years ago I stumbled across PMTiles, at the time a rather novel way to serve maps. I didn’t have a need for it then, so it just sat in the back of my mind as something to take a closer look at later. Now the time has come, so in this note I’m going to play around with PMTiles and see if they fit my needs.

Background

First, a short explainer: PMTiles is a file format crafted so that individual mapping tiles can be retrieved using HTTP Range requests. You can chuck such a file onto S3, or serve it from an HTTP server like Caddy, and a suitable JS library can immediately start retrieving the tiles it needs for the current map location and zoom level, all from that one file.

Compare that to a tiling server, which has map data on disk but needs a runtime component to serve it in the appropriate format. The advantage is clear. Mind you, that comes with some disadvantages. For example, PMTiles are read-only, so you can’t update part of a map in place. Protomaps has a good article about the tradeoffs if you want to read more.

Boring technology

For my usage, I’ll prepare a map file to my liking and serve it from my server with Caddy. If I want to include the latest OpenStreetMap changes, I just recreate the map and that’s it. It’s delightful. No additional moving parts to manage and operate. More technologies should be like this.

Creating PMTiles

With the musings out of the way, let’s get practical. First, grab the latest build from this page on Protomaps. It’s essentially a dump of all OpenStreetMap data for the whole planet. We don’t want to download it, just note the filename.

Next, using the PMTiles CLI, we can generate our own PMTiles file of the world, capped at a maximum zoom level. That limits how much detail you see when zooming in close, and in turn keeps the file much smaller.

./pmtiles extract https://build.protomaps.com/20260823.pmtiles world.pmtiles --maxzoom=4

This gives us a 6 MB file. Super small, but also not so detailed.

Next, let’s create a high-fidelity file for a single region. For that, we need a bounding box around the region we care about. To avoid juggling coordinates by hand, bbox finder lets you draw a box easily.

Then we can generate a full-fidelity extract of that region like so:

./pmtiles extract https://build.protomaps.com/20260823.pmtiles zurich.pmtiles --bbox=8.497925,47.339055,8.588905,47.395444

This results in a 6.3 MB file. Viewing it on pmtiles.io shows every detail in Zurich, but nothing outside the bounding box.

What’s really nice is that we can now combine the two files into one with the advantages of both. That only works if the files don’t overlap, though. So far, we have the whole world up to zoom level 4, but Zurich at every zoom level. To make it work, we need Zurich only from zoom level 5 and up:

./pmtiles extract zurich.pmtiles zurich-hi.pmtiles --minzoom=5

Resulting in a slightly smaller file at 5.8 MB. Now, the two files are non-overlapping and can be combined like so:

./pmtiles merge world.pmtiles zurich-hi.pmtiles combined.pmtiles

This results in a 12 MB file with high detail around Zurich and a low-fidelity basemap for the rest. Neat!

There’s one catch, depending on the renderer: in the plain world.pmtiles basemap, I can zoom past maxzoom and still see a low-fidelity map. With the combined map, however, zooming past maxzoom turns everything outside Zurich black. That’s because the combined file declares maxzoom: 15 in its header: the renderer expects zoom level 15 everywhere and paints black wherever that data is missing.

To fix this, configure the JS rendering lib with two sources at different zoom levels, both pointing to the same file.

Verdict

It’s truly as simple and easy as it says on the box. Consider me impressed. For my purposes, I might explore automatically deriving a set of high-fidelity bounding boxes from my location data and merging them into one ultimate file. But that may well be overengineering.

Apple M1 vs Hetzner CAX21

Fri, 07 Aug 2026 22:00:00 GMT

08 Aug 2026

Apple M1 vs Hetzner CAX21

Benchmarking homeserver hardware

I’m in the process of migrating my VPS to a MacBook Air M1 running Asahi in my closet. Before I decommission the VPS though, I’m curious to see what the performance difference is. So here are a couple of non-representative benchmarks comparing the M1 to an Ampere-based CAX21 at Hetzner.

By the way, I’ve benchmarked these Arm-based Hetzner machines before. Last time, I was in the process of migrating from an x86 machine to the CAX31. In the meantime, I’ve downgraded to the CAX21, which is why these numbers don’t exactly match up.

Anyway, here are the specs of the two machines:

CAX21: 4 vCPU, 8 GB RAM

M1: 4 performance cores, 4 efficiency cores, 16 GB RAM

Note that we’re comparing a VPS with noisy neighbors and a modest production load to an idle M1. Also, the tests are rather short, so the impact of thermal throttling on the passively cooled M1 is not visible. I did say these would be non-representative benchmarks.

sysbench cpu --threads=x run

[chart — see article]

Figure 1. sysbench CPU benchmark results

sysbench memory --threads=x run

[chart — see article]

Figure 2. sysbench memory benchmark results

I have no clue what happened here at 8 threads for the M1. I guess the performance cores don’t like to share.

sysbench fileio --threads=x --file-test-mode=seqwr run

[chart — see article]

Figure 3. sysbench fileio benchmark results (--file-test-mode=seqwr)

And finally, I wanted to see how they compare when compiling one of my Rust code bases.

time cargo build --release

[chart — see article]

Figure 4. Compilation time of Rust application

Verdict

It’s simultaneously unsurprising that a dedicated M1 outperforms a tiny slice of a server CPU, and yet I’m always blown away by what this little 6-year-old machine can do. Needless to say, this migration will be a nice little upgrade, although I’m more excited about the energy efficiency of the Air (<2W idle) than its raw power.

Asahi Linux on a Macbook Air M1

Mon, 22 Jun 2026 22:00:00 GMT

23 Jun 2026

Asahi Linux on a Macbook Air M1

Setting up a headless server

I’ve been wanting to move my server from a VPS hosted at Hetzner to my own hardware for a while. After scouring the internet for suitable low-power devices, think Raspberry Pi but more powerful, I landed on a weird idea: What if I just repurpose my Macbook Air M1 as a server? It’s still a super performant machine and consumes barely any power idling. After some research it seems like this should be well possible thanks to the Asahi Linux project. So here we go.

I’ll use this chance to revamp my server setup again, but that’s not the focus of this note. Here I just want to run through the fundamental software tweaks necessary to make this device capable of acting as a server. All the other basic hardening that should be applied to any server I leave out as that is better found in a serious server setup guide.

Step 1: Install Asahi from macOS as described in their documentation.

Step 2: Ensure device doesn’t sleep when lid is closed.

# /etc/systemd/logind.conf.d/lid-ignore.conf
[Login]
HandleLidSwitch=ignore
HandleLidSwitchDocked=ignore
HandleLidSwitchExternalPower=ignore

Step 3: Turn screen off when lid is closed. Don’t want to waste power.

sudo grubby --update-kernel=ALL --args="consoleblank=30"

Step 4: Set battery charge limit to 80%. Let’s keep the battery healthy and avoid it exploding in my closet.

echo 'SUBSYSTEM=="power_supply", KERNEL=="macsmc-battery", ATTR{charge_control_end_threshold}="80"' | sudo tee /etc/udev/rules.d/10-battery.rules

Voilà

With those basic tweaks I can plug my Macbook into power and Ethernet and set it aside. It acts as a neat little arm64 server while sipping around 1.75 Watts at idle. Great success.

Known issues

One inconvenience I found is that my USB-C to Ethernet adapter doesn’t get listed when the server reboots. I have to physically plug it in again for it to be picked up. None of the common workarounds work for me. I have tried a different dongle, which seems to work. That one is however slower and seems to consume about 3 Watts just by itself. It also gets pretty warm just sitting there. So I’ll just live with having to be home when I want to reboot the machine after updates. That’ll do for my purposes. If it turns out to be annoying I may go hunting for another adapter.

Towards approachable observability with wide events

Sat, 16 May 2026 22:00:00 GMT

17 May 2026

Towards approachable observability with wide events

A tragedy in four acts

I like to think about how to get operational insights into the services I host. It’s become kind of a hobby, to be honest. Maybe this whole thing is just bike shedding. What better way to avoid the crushing reality that an actual project you had in mind doesn’t live up to your expectations than by not even attempting it, and instead building infrastructure around your existing applications? Maybe that’s a topic for therapy.

Anyhow, I realized during my latest manic observability episode that I barely even remember how I got here. So for posterity, I’m outlining the history and motivations for all the different phases in this note.

Act 1: Metrics and Prometheus

A couple of years ago I got the itch to play around with embedded devices. I got myself an ESP32 and a CO2 sensor and got to work. With a little elbow grease, I had the current CO2 concentration displayed on a small seven segment display. Soon, I graduated to wanting to store and chart this data.

At this point I had little knowledge on how to approach this, so I went the well-trodden path: Prometheus to store time series data and Grafana to visualize it. Since my microcontroller was not internet accessible, I had to put a push gateway in front of Prometheus to allow pushing data into the system, instead of having Prometheus scrape the metrics off the device on a regular schedule.

Around this time I also built and deployed this website. I had seen a cool statistics section on another blog, where one could view how many page views the blog was getting and which pages were popular. Wanting the same for this site, I initially transformed the static site into one with a NodeJS backend just for this feature. After a short while I realized I wasn’t happy with that and ripped it out again, planning instead to track visitors with an external service. You can watch this unfold in this old note.

I could have used my existing Prometheus setup to scrape Caddy for access metrics, but that wouldn’t have given me all the data I wanted. I’d need a dedicated service serving the site and exposing metrics. So this kicked off the realization that I needed a broader observability system than what Prometheus could give me. What was always clear to me, by the way, is that I wanted one unified system for metrics and service monitoring.

In hindsight, I also realize there was another push factor: PromQL. I got annoyed with learning a language with such a narrow use case.

Act 2: Metrics with DuckDB

With all this in place, the time was ripe to venture out. And boy did I. The requirements for my next system were clear: it should accept arbitrary JSON payloads (like sensor readings or access logs), store them efficiently, and allow querying them in an ergonomic fashion.

Notice the shift away from time series data to something more general, something that could cover multiple use cases. You can follow the journey from musing on how to store that data to benchmarking suitable databases in my notes.

Coming away from PromQL, I was craving more expressive power for querying and transforming data. So I started looking into building a WebAssembly plugin system to safely execute efficient transformations server-side. For visualization, I wasn’t satisfied with Grafana anymore either — I wanted to go custom there as well. So I built a SolidJS frontend with responsive plots using Observable Plot. All this started taking shape under the name observatory.

You might already be guessing that this would not go so well. Indeed, the sensor readings and GPS location data from my phone were handled fine, but I was missing a puzzle piece on how access logs fit in. How would I instrument my services in such a way as to get the data into observatory in a queryable form? I found the answer in the form of traces. At first I wanted to build a tracing collector into observatory, but that proved to be difficult. Instead, I turned towards a more off-the-shelf stack to ingest tracing data.

Act 3: Tracing with ClickHouse

When I learned about tracing — not just in the context of distributed systems, but the general concept of spans and events — I felt like I had hit the jackpot. Events you emit within spans mimic log lines, except they are naturally associated with a particular request. No correlation id needed (it’s the span id, and the tooling handles it for you). Metrics are just hardcoded aggregations done in the application, which lends itself to very space-efficient storage in exchange for limited flexibility. With traces, you can generate any metrics you might care about after the fact. Tracing is kind of a one-stop-shop for observability.

So I set up an OpenTelemetry collector that would receive traces and store them in a ClickHouse database. That part made a lot of sense, and instrumenting my services with the excellent tracing crate is a walk in the park. Getting that sent off not so much, though. But now the sensor and GPS data didn’t fit the bill so cleanly anymore. I didn’t want to have to send that data in a format the collector would understand, so I built a small service I called events that accepts the data and simply declares a span with the received attributes. This way, the tracing machinery in that service takes care of getting the data into ClickHouse (via the collector).

So with this, I finally realized the dream of having one central system where all my data flows together:

So this is great. Just a couple of problems:

I don’t like OpenTelemetry because I don’t understand it. I would prefer simpler protocols, something I could build myself in a pinch. The docs are a mess, too.

I don’t like operating ClickHouse and Grafana on my server. They’re great at what they do, but I have a need for simpler software. I suffer from not-invented-here syndrome, ok?

I don’t like the way the data is laid out in the DB. I can put any data into the DB just by annotating a function with a span, no problem. But it’s very inefficient at storing, say, an 8 byte sensor value. The fact that each span is its own row also makes aggregating data across all spans belonging to one request quite tedious.

I’m craving simplicity.

Act 4: Wide events with Parquet

For more background on this, please read the preceding note on Parquet files.

Apparently other people have had the same thoughts. Some call it Observability 2.0, others wide events. I’ll give you a brief rundown; for more details and a good list of further reading, consult this blog post.

The idea is simple: for a “unit of useful work”, whatever that means for your system, collect an object with relevant properties and send it off to your observability system upon completion of the work. For a web backend, the unit of work you care about most is almost certainly the handling of a request. But it may also be the execution of a scheduled task. So for the request, you would track url, method, response status, headers, and all the usual suspects. But if you go all the way, you may even add current memory utilisation and other environmental factors.

Fundamentally, this is not that different from traces applied to a monolith. The data is just more easily queryable. And what I particularly appreciate: no need for sophisticated machinery.

The plan is, once again, simple: build a system that accepts arbitrary JSON payloads. Regularly write the buffered payloads into Parquet files. That’s it. That’s the write path. To visualize the data, I’ll build custom dashboards just like in Act 2. The storage engine enabling fast queries will be DuckDB. Instrumentation on the services should be rather straight-forward as well. All I need to do is keep track of some context throughout the handling of a request to write properties into.

What I’m hoping to achieve with this approach: a simpler wire protocol that any service and device can directly talk - just JSON. A simpler data model, plain files for easier backup. A simpler operating model, no queries, no load. More powerful and elegant visualizations. The simplicity I’ll gain of course comes at the cost of significant effort to build this system. But I’m more than happy to pay for it.

I’m writing this as I gear up to make Act 4 a reality, sorry to leave you hanging like this. Once it is done I’ll add a more satisfying resolution here. Or maybe Act 5. We’ll see.

Exploring the Variant type in Parquet

Sun, 10 May 2026 22:00:00 GMT

11 May 2026

Exploring the Variant type in Parquet

Let's get shredding

Observability has been on my mind again lately. I already touched on how to store unstructured data a while back. After a brief stint building my own system on top of DuckDB, I’ve been running an off-the-shelf combination of ClickHouse and Grafana to collect and visualize tracing data. It has served me well, but a planned move of my side projects to my own hardware has me re-evaluating the setup. ClickHouse is by far the heaviest service I run on my server, and it has a habit of filling up my disk with logging tables. Look, ClickHouse is an engineering marvel, but it’s not my marvel. It’s someone else’s marvel. And I have a need to create my own. Thing, not marvel, I mean. So here we are.

Now, I’ve stumbled across the Parquet file format, and in particular its new data type that looks like it was custom-made to solve all my problems. Maybe not all of them. In this note I want to explore how DuckDB can write and query these files. If this works out, I may be tempted to build a new observability system around wide events on top of these technologies.

But before we get into the weeds, let me set the scene. Let’s say I have a service that sends structured logs (or wide events) to my observability system. Today they might look like this:

{
  "timestamp": "2026-05-05T08:00:00Z",
  "service": "marending-dev",
  "severity": "error",
  "message": "User Bobby Tables doesn't exist"
}

But then I want to track more information.

{
  "timestamp": "2026-05-05T08:00:00Z",
  "service": "marending-dev",
  "severity": "error",
  "message": "User Bobby Tables doesn't exist",
  "user_id": "null",
  "ip": "85.0.1.42",
  "path": "/user/login"
}

You get the picture. The question is, how can we store these JSON blobs such that we can then efficiently compute, say, error rate grouped by user_id? We can’t have a static schema, since we might want to add properties to this event at any time. We can chuck this as a JSON string into any database, file format or what have you. But then any query that attempts to read from properties of that JSON string is going to incur the significant cost of extracting the appropriate fields for every row of interest. One way to get reasonable performance here is to ensure severity and user_id are each contiguously laid out so that they can be efficiently aggregated.

Parquet and Variant

That brings us to Apache Parquet. Parquet is a column-oriented file format. While DuckDB has its own file format for storing data, one of its strengths is its support for a wide variety of other formats, including Parquet. So you can write and query Parquet files natively with SQL within DuckDB.

Recently Parquet added the Variant type, a binary encoding of primitives, arrays or objects of arbitrary types. It allows representing structured data in such a way that it can be queried more efficiently than if the same data was serialized and stored as a string type. But the real kicker is “Variant shredding”. It’s the process of extracting some of the properties within the Variant into their own columns. This should then give us maximum performance when aggregating on those columns. And DuckDB apparently supports this!

Once I learned about this, a path towards my dream observability system emerged. Step 1: Write a service that accepts arbitrary JSON payloads, buffers them in memory and writes them to Parquet files with the payload being of type Variant every 10 seconds or so. Step 2: Every hour, read all the small Parquet files and shred them into a larger hourly file. Step 3: Possibly aggregate the hourly files into even broader files, but I’ll see when I get to it.

Now at query time, I will have at most an hour’s worth of data in inefficient, unshredded form. That should be easy enough to deal with. Data older than that will be in highly efficient form. And the best part? DuckDB should be able to query across many Parquet files using glob syntax, even when some have concrete, shredded columns and some don’t. But we’ll see this for ourselves.

Exploration

Let’s start with the example given in DuckDB’s documentation.

CREATE TABLE events (id INTEGER, data VARIANT);
INSERT INTO events VALUES
    (1, 42::VARIANT),
    (2, 'hello world'::VARIANT),
    (3, [1, 2, 3]::VARIANT),
    (4, {'name': 'Alice', 'age': 30}::VARIANT);

Querying the table presents the expected result.

SELECT * FROM events;
┌───────┬────────────────────────────┐
│  id   │            data            │
│ int32 │          variant           │
├───────┼────────────────────────────┤
│     1 │ 42                         │
│     2 │ hello world                │
│     3 │ [1, 2, 3]                  │
│     4 │ {'name': Alice, 'age': 30} │
└───────┴────────────────────────────┘

SELECT id, data, variant_typeof(data) AS vtype FROM events;
┌───────┬────────────────────────────┬───────────────────┐
│  id   │            data            │       vtype       │
│ int32 │          variant           │      varchar      │
├───────┼────────────────────────────┼───────────────────┤
│     1 │ 42                         │ INT32             │
│     2 │ hello world                │ VARCHAR           │
│     3 │ [1, 2, 3]                  │ ARRAY(3)          │
│     4 │ {'name': Alice, 'age': 30} │ OBJECT(name, age) │
└───────┴────────────────────────────┴───────────────────┘

We can now also query nested fields in a Variant.

SELECT data.name FROM events;
┌─────────┐
│  name   │
│ variant │
├─────────┤
│ NULL    │
│ NULL    │
│ NULL    │
│ Alice   │
└─────────┘

Writing this data as-is to a parquet file is now as simple as COPY events TO 'events.parquet' (FORMAT parquet);. And indeed, having DuckDB describe the parquet file’s shape yields the expected.

DESCRIBE SELECT * FROM 'events.parquet';
┌────────────────┐
│ events.parquet │
│                │
│ id     integer │
│ data   variant │
└────────────────┘

But hold on. According to the docs, writing to a Parquet file should automatically shred the variant type. Shouldn’t I see the individual columns then? Turns out shredding is only a storage detail, not a logical schema change. The type stays as is. To see how the data is actually stored, we have to inspect the file’s metadata.

SELECT name, type, converted_type FROM parquet_schema('events.parquet');
┌───────────────┬────────────┬────────────────┐
│     name      │    type    │ converted_type │
│    varchar    │  varchar   │    varchar     │
├───────────────┼────────────┼────────────────┤
│ duckdb_schema │ NULL       │ NULL           │
│ id            │ INT32      │ INT_32         │
│ data          │ NULL       │ NULL           │
│ metadata      │ BYTE_ARRAY │ NULL           │
│ value         │ BYTE_ARRAY │ NULL           │
│ typed_value   │ INT32      │ INT_32         │
└───────────────┴────────────┴────────────────┘

Well, the way this is supposed to be read is as follows. id is of type INT32, the data variant itself doesn’t really have a type but it comprises three parts: metadata, value and typed_value. Of interest are value and typed_value. Each row with the data logical column is either going to have the (fallback) byte-array value representation set, or the shredded typed_value column. In this case, DuckDB’s autoshredder apparently decided that row 1 with the integer data variant deserves to have its representation shredded, but not the other rows.

If, instead, we instruct DuckDB to shred data to a struct like so

COPY events TO 'events.parquet' (
    FORMAT parquet,
    SHREDDING {'data': 'STRUCT(name VARCHAR, age INTEGER)'}
);

we get this

SELECT name, type, converted_type, logical_type FROM parquet_schema('events.parquet');
┌───────────────┬────────────┬────────────────┐
│     name      │    type    │ converted_type │
│    varchar    │  varchar   │    varchar     │
├───────────────┼────────────┼────────────────┤
│ duckdb_schema │ NULL       │ NULL           │
│ id            │ INT32      │ INT_32         │
│ data          │ NULL       │ NULL           │
│ metadata      │ BYTE_ARRAY │ NULL           │
│ value         │ BYTE_ARRAY │ NULL           │
│ typed_value   │ NULL       │ NULL           │
│ name          │ NULL       │ NULL           │
│ value         │ BYTE_ARRAY │ NULL           │
│ typed_value   │ BYTE_ARRAY │ UTF8           │
│ age           │ NULL       │ NULL           │
│ value         │ BYTE_ARRAY │ NULL           │
│ typed_value   │ INT32      │ INT_32         │
└───────────────┴────────────┴────────────────┘

Now we see data with its metadata, value and typed_value properties and also value and typed_value logical columns for both name and age. For these the shredded types are UTF8 and INT32 respectively.

To get an intuitive understanding, it is interesting to look at the output of parquet-tools meta events.parquet from this tool:

Output

{
  "NumRowGroups": 1,
  "RowGroups": [
    {
      "NumRows": 4,
      "TotalByteSize": 725,
      "Columns": [
        {
          "PathInSchema": ["id"],
          "Type": "INT32",
          "ConvertedType": "convertedtype=INT_32",
          "Encodings": ["PLAIN"],
          "CompressedSize": 41,
          "UncompressedSize": 39,
          "NumValues": 4,
          "NullCount": 0,
          "MaxValue": 4,
          "MinValue": 1,
          "CompressionCodec": "SNAPPY"
        },
        {
          "PathInSchema": ["data", "metadata"],
          "Type": "BYTE_ARRAY",
          "Encodings": ["PLAIN"],
          "CompressedSize": 52,
          "UncompressedSize": 60,
          "NumValues": 4,
          "NullCount": 0,
          "CompressionCodec": "SNAPPY"
        },
        {
          "PathInSchema": ["data", "value"],
          "Type": "BYTE_ARRAY",
          "Encodings": ["PLAIN"],
          "CompressedSize": 81,
          "UncompressedSize": 137,
          "NumValues": 4,
          "NullCount": 1,
          "CompressionCodec": "SNAPPY"
        },
        {
          "PathInSchema": ["data", "typed_value", "name", "value"],
          "Type": "BYTE_ARRAY",
          "Encodings": ["PLAIN"],
          "CompressedSize": 34,
          "UncompressedSize": 119,
          "NumValues": 4,
          "NullCount": 4,
          "CompressionCodec": "SNAPPY"
        },
        {
          "PathInSchema": ["data", "typed_value", "name", "typed_value"],
          "Type": "BYTE_ARRAY",
          "ConvertedType": "convertedtype=UTF8",
          "Encodings": ["PLAIN"],
          "CompressedSize": 44,
          "UncompressedSize": 128,
          "NumValues": 4,
          "NullCount": 3,
          "CompressionCodec": "SNAPPY"
        },
        {
          "PathInSchema": ["data", "typed_value", "age", "value"],
          "Type": "BYTE_ARRAY",
          "Encodings": ["PLAIN"],
          "CompressedSize": 34,
          "UncompressedSize": 119,
          "NumValues": 4,
          "NullCount": 4,
          "CompressionCodec": "SNAPPY"
        },
        {
          "PathInSchema": ["data", "typed_value", "age", "typed_value"],
          "Type": "INT32",
          "ConvertedType": "convertedtype=INT_32",
          "Encodings": ["PLAIN"],
          "CompressedSize": 39,
          "UncompressedSize": 123,
          "NumValues": 4,
          "NullCount": 3,
          "CompressionCodec": "SNAPPY"
        }
      ]
    }
  ]
}

Let’s take this line for example ["data", "typed_value", "name", "typed_value"]. A row in our dataset is going to populate this logical column when the name property in the variant is a UTF8 string, otherwise it will fall back to the BYTE_ARRAY data.value. And indeed we see that 3 out of 4 values in this column are NULL, since row 4 is the only one that can populate this typed_value. Conversely, the ["data", "value"] entry only has a single NULL value: only row 4 does not populate the fallback column, populating the typed values instead.

If we look at ["data", "typed_value", "name", "value"] we can see that not a single non-NULL value is here. I suppose this would be populated if we had a row with an object that contains the name property, but with a type other than string.

I’ll just leave this blog post from the official Parquet website here for further reading. It has some nice visualizations that further explain how this works.

Autoshredding

So far, so good. But we don’t want to have to specify how the values should be shredded. The whole point would be for DuckDB to figure it out from the data, but somehow in our first attempt it just chose to “materialize” the integer and not our object.

Well, it looks like DuckDB decides what to shred based on how often a specific type occurs in the table. If I add two more rows that have the same shape as the object in row 4

INSERT INTO events VALUES (5, {'name': 'Bob', 'age': 27}::VARIANT);
INSERT INTO events VALUES (6, {'name': 'Flo', 'age': 31}::VARIANT);
COPY events TO 'events-more-structs.parquet' (FORMAT parquet);

I get our familiar shredding, this time without having to specify the type.

SELECT name, type, converted_type FROM parquet_schema('events-more-structs.parquet');
┌───────────────┬────────────┬────────────────┐
│     name      │    type    │ converted_type │
│    varchar    │  varchar   │    varchar     │
├───────────────┼────────────┼────────────────┤
│ duckdb_schema │ NULL       │ NULL           │
│ id            │ INT32      │ INT_32         │
│ data          │ NULL       │ NULL           │
│ metadata      │ BYTE_ARRAY │ NULL           │
│ value         │ BYTE_ARRAY │ NULL           │
│ typed_value   │ NULL       │ NULL           │
│ age           │ NULL       │ NULL           │
│ value         │ BYTE_ARRAY │ NULL           │
│ typed_value   │ INT32      │ INT_32         │
│ name          │ NULL       │ NULL           │
│ value         │ BYTE_ARRAY │ NULL           │
│ typed_value   │ BYTE_ARRAY │ UTF8           │
└───────────────┴────────────┴────────────────┘

If I add a bunch more string columns now, I can get the autoshredder to only shred the UTF8 type. Initially I would have expected that it should be possible to have both shredded age and name values as well as a fallback not to an untyped BYTE_ARRAY but to a UTF8 string. But thinking about this, I guess the typed_value for data is now an object of name and age (although it’s not readily visible in the above representation), so it can’t be a string simultaneously.

It should be possible, however, if the string columns were in a new property inside an object. Let me give this a shot.

INSERT INTO events VALUES (10, {'message': 'Hello world'}::VARIANT);
INSERT INTO events VALUES (11, {'message': 'message'}::VARIANT);
INSERT INTO events VALUES (12, {'message': '42'}::VARIANT);

COPY events TO 'events-message.parquet' (FORMAT parquet);
SELECT name, type, converted_type FROM parquet_schema('events-message.parquet');
┌───────────────┬────────────┬────────────────┐
│     name      │    type    │ converted_type │
│    varchar    │  varchar   │    varchar     │
├───────────────┼────────────┼────────────────┤
│ duckdb_schema │ NULL       │ NULL           │
│ id            │ INT32      │ INT_32         │
│ data          │ NULL       │ NULL           │
│ metadata      │ BYTE_ARRAY │ NULL           │
│ value         │ BYTE_ARRAY │ NULL           │
│ typed_value   │ NULL       │ NULL           │
│ message       │ NULL       │ NULL           │
│ value         │ BYTE_ARRAY │ NULL           │
│ typed_value   │ BYTE_ARRAY │ UTF8           │
│ age           │ NULL       │ NULL           │
│ value         │ BYTE_ARRAY │ NULL           │
│ typed_value   │ INT32      │ INT_32         │
│ name          │ NULL       │ NULL           │
│ value         │ BYTE_ARRAY │ NULL           │
│ typed_value   │ BYTE_ARRAY │ UTF8           │
└───────────────┴────────────┴────────────────┘

Indeed we can see that it now materialized age, name and message. Neat!

Real-world data

With my mental model starting to get less foggy, I think it’s time to start digging into some real-world data from my existing observability setup. Luckily ClickHouse makes it really easy to get my data out: SELECT * FROM events.otel_traces INTO OUTFILE './otel_traces.parquet' FORMAT Parquet. This dumps all of the data into a single Parquet file with just over 10 million rows, weighing in at 800 MB.

The shape of this file is determined by the standard schema the OpenTelemetry collector created when I started ingesting data. In ClickHouse it looks like this:

CREATE TABLE default.otel_traces
(
        `Timestamp` DateTime64(9) CODEC(Delta(8), ZSTD(1)),
        `TraceId` String CODEC(ZSTD(1)),
        `SpanId` String CODEC(ZSTD(1)),
        `ParentSpanId` String CODEC(ZSTD(1)),
        `TraceState` String CODEC(ZSTD(1)),
        `SpanName` LowCardinality(String) CODEC(ZSTD(1)),
        `SpanKind` LowCardinality(String) CODEC(ZSTD(1)),
        `ServiceName` LowCardinality(String) CODEC(ZSTD(1)),
        `ResourceAttributes` Map(LowCardinality(String), String) CODEC(ZSTD(1)),
        `ScopeName` String CODEC(ZSTD(1)),
        `ScopeVersion` String CODEC(ZSTD(1)),
        `SpanAttributes` Map(LowCardinality(String), String) CODEC(ZSTD(1)),
        `Duration` Int64 CODEC(ZSTD(1)),
        `StatusCode` LowCardinality(String) CODEC(ZSTD(1)),
        `StatusMessage` String CODEC(ZSTD(1)),
        `Events.Timestamp` Array(DateTime64(9)) CODEC(ZSTD(1)),
        `Events.Name` Array(LowCardinality(String)) CODEC(ZSTD(1)),
        `Events.Attributes` Array(Map(LowCardinality(String), String)) CODEC(ZSTD(1)),
        `Links.TraceId` Array(String) CODEC(ZSTD(1)),
        `Links.SpanId` Array(String) CODEC(ZSTD(1)),
        `Links.TraceState` Array(String) CODEC(ZSTD(1)),
        `Links.Attributes` Array(Map(LowCardinality(String), String)) CODEC(ZSTD(1)),
        INDEX idx_trace_id TraceId TYPE bloom_filter(0.001) GRANULARITY 1,
        INDEX idx_res_attr_key mapKeys(ResourceAttributes) TYPE bloom_filter(0.01) GRANULARITY 1,
        INDEX idx_res_attr_value mapValues(ResourceAttributes) TYPE bloom_filter(0.01) GRANULARITY 1,
        INDEX idx_span_attr_key mapKeys(SpanAttributes) TYPE bloom_filter(0.01) GRANULARITY 1,
        INDEX idx_span_attr_value mapValues(SpanAttributes) TYPE bloom_filter(0.01) GRANULARITY 1,
        INDEX idx_duration Duration TYPE minmax GRANULARITY 1
)
ENGINE = MergeTree
PARTITION BY toDate(Timestamp)
ORDER BY (ServiceName, SpanName, toUnixTimestamp(Timestamp), TraceId)
TTL toDateTime(Timestamp) + toIntervalDay(3)
SETTINGS ttl_only_drop_parts = 1

Once dumped to the Parquet file, we can see the shape as read by DuckDB here:

DESCRIBE SELECT * FROM 'otel_traces.parquet';
┌───────────────────────────────────────────────┐
│              otel_traces.parquet              │
│                                               │
│ Timestamp          timestamp with time zone   │
│ TraceId            varchar                    │
│ SpanId             varchar                    │
│ ParentSpanId       varchar                    │
│ TraceState         varchar                    │
│ SpanName           varchar                    │
│ SpanKind           varchar                    │
│ ServiceName        varchar                    │
│ ResourceAttributes map(varchar, varchar)      │
│ ScopeName          varchar                    │
│ ScopeVersion       varchar                    │
│ SpanAttributes     map(varchar, varchar)      │
│ Duration           ubigint                    │
│ StatusCode         varchar                    │
│ StatusMessage      varchar                    │
│ Events.Timestamp   timestamp with time zone[] │
│ Events.Name        varchar[]                  │
│ Events.Attributes  map(varchar, varchar)[]    │
│ Links.TraceId      varchar[]                  │
│ Links.SpanId       varchar[]                  │
│ Links.TraceState   varchar[]                  │
│ Links.Attributes   map(varchar, varchar)[]    │
└───────────────────────────────────────────────┘

You can see a clear correspondence between the Parquet and ClickHouse types. Notably, this file is not in the shape we want yet. For this we’ll need to slice and dice it into different forms to benchmark how queries behave.

Benchmarks

To get an overview of the performance of the Variant type, I’m going to wrangle the payload into the following types to see how the different approaches stack up:

JSON
Map / Array
Variant
Fully materialized

For simplicity, I’m going to focus on two types of data I have stored in my system, GPS location data and access logs to this website. For both datasets I’ll run a query that can benefit from prefiltering data down to the relevant rows and another one that has to run expensive aggregations across the whole dataset.

For the GPS data, the first query should get distinct days I’ve visited a particular location. The second query will get average speed and altitude hourly across the whole timespan.

For the access logs, the first query will count error responses per URI, and the second one will aggregate daily traffic per browser.

Danger zone! The queries compared here were generated / translated by Claude and may not be the pinnacle of optimization.

GPS details

JSON

This nifty query gets the relevant data from my GPS entries and stores it into a new Parquet file with the data stored as JSON.

COPY (
    SELECT
    Timestamp,
    'gps-location' AS service,
    CAST(SpanAttributes AS JSON) AS data
    FROM 'otel_traces.parquet'
    WHERE SpanName = 'gps-location'
) TO 'gps_location_json.parquet' (FORMAT PARQUET);

DESCRIBE SELECT * FROM 'gps_location_json.parquet';
┌────────────────────────────────────┐
│        gps_location.parquet        │
│                                    │
│ Timestamp timestamp with time zone │
│ service   varchar                  │
│ data      json                     │
└────────────────────────────────────┘

The nested structures inside the data field are unfortunately not parsed into JSON as well, but rather stay as string values of the top-level properties. But as the DuckDB JSON type is anyhow just backed by a VARCHAR, I don’t expect too much impact for this.

Now here is the query to list distinct days where I’ve visited a particular location. Pasting these coordinates into a map is left as an exercise to the reader.

SELECT DISTINCT
  DATE_TRUNC('day',
    CAST(json_extract_string(
           CAST(json_extract_string(data, '$.location') AS JSON),
           '$.properties.timestamp'
         ) AS TIMESTAMP)
  ) AS day
FROM 'gps_location_json.parquet'
WHERE CAST(json_extract(
        CAST(json_extract_string(data, '$.location') AS JSON),
        '$.geometry.coordinates[0]'
      ) AS DOUBLE) BETWEEN 8.486986 AND 8.493896
  AND CAST(json_extract(
        CAST(json_extract_string(data, '$.location') AS JSON),
        '$.geometry.coordinates[1]'
      ) AS DOUBLE) BETWEEN 47.348284 AND 47.353111
ORDER BY day DESC;

The second query to aggregate some data into hourly buckets.

WITH g AS (
  SELECT
    CAST(json_extract_string(data, '$.location') AS JSON) AS loc
  FROM 'gps_location_json.parquet'
)
SELECT
  TIME_BUCKET(INTERVAL '1 hour',
    CAST(json_extract_string(loc, '$.properties.timestamp') AS TIMESTAMP)
  ) AS hour,
  AVG(CAST(json_extract(loc, '$.properties.speed') AS DOUBLE))               AS avg_speed,
  AVG(CAST(json_extract(loc, '$.properties.altitude') AS DOUBLE))            AS avg_altitude,
  MAX(CAST(json_extract(loc, '$.properties.horizontal_accuracy') AS DOUBLE)) AS worst_accuracy,
  COUNT(*) AS points
FROM g
GROUP BY hour
ORDER BY hour;

Note how in those two queries we use the nested timestamp from within the payload instead of the top-level Timestamp. That’s due to getting those proper timestamps messed up when migrating data from an even older system into the one I use currently. If what I’m trying here comes to fruition, I’m going to fix this up.

The benchmark results will follow at the end.

Map

For the Map type I could just leave the data as-is, since the original Parquet file already has the SpanAttributes property as map(varchar, varchar). But due to the nesting of the data I care about, this would actually end up being the JSON type just with a detour. Instead, I’m flattening the structure into a proper Map below. Of course, this kind of defeats the purpose of not having to care about the exact schema of the data coming my way. But for benchmarking the data types it will do.

COPY (
  WITH src AS (
    SELECT
      Timestamp,
      CAST(SpanAttributes['location'] AS JSON) AS loc
    FROM 'otel_traces.parquet'
    WHERE SpanName = 'gps-location'
  )
  SELECT
    Timestamp,
    'gps-location' AS service,
    MAP {
      'location.type':                           json_extract_string(loc, '$.type'),
      'location.geometry.type':                  json_extract_string(loc, '$.geometry.type'),
      'location.geometry.coordinates[0]':        json_extract_string(loc, '$.geometry.coordinates[0]'),
      'location.geometry.coordinates[1]':        json_extract_string(loc, '$.geometry.coordinates[1]'),
      'location.properties.altitude':            json_extract_string(loc, '$.properties.altitude'),
      'location.properties.battery_level':       json_extract_string(loc, '$.properties.battery_level'),
      'location.properties.battery_state':       json_extract_string(loc, '$.properties.battery_state'),
      'location.properties.course':              json_extract_string(loc, '$.properties.course'),
      'location.properties.course_accuracy':     json_extract_string(loc, '$.properties.course_accuracy'),
      'location.properties.horizontal_accuracy': json_extract_string(loc, '$.properties.horizontal_accuracy'),
      'location.properties.motion[0]':           json_extract_string(loc, '$.properties.motion[0]'),
      'location.properties.speed':               json_extract_string(loc, '$.properties.speed'),
      'location.properties.speed_accuracy':      json_extract_string(loc, '$.properties.speed_accuracy'),
      'location.properties.timestamp':           json_extract_string(loc, '$.properties.timestamp'),
      'location.properties.vertical_accuracy':   json_extract_string(loc, '$.properties.vertical_accuracy'),
      'location.properties.wifi':                json_extract_string(loc, '$.properties.wifi')
    } AS data
  FROM src
) TO 'gps_location_map.parquet' (FORMAT PARQUET);

DESCRIBE SELECT * FROM 'gps_location_map.parquet';
┌────────────────────────────────────┐
│      gps_location_map.parquet      │
│                                    │
│ Timestamp timestamp with time zone │
│ service   varchar                  │
│ data      map(varchar, varchar)    │
└────────────────────────────────────┘

SELECT DISTINCT
  DATE_TRUNC('day', CAST(data['location.properties.timestamp'] AS TIMESTAMP)) AS day
FROM 'gps_location_map.parquet'
WHERE CAST(data['location.geometry.coordinates[0]'] AS DOUBLE)
        BETWEEN 8.486986 AND 8.493896
  AND CAST(data['location.geometry.coordinates[1]'] AS DOUBLE)
        BETWEEN 47.348284 AND 47.353111
ORDER BY day DESC;

SELECT
  TIME_BUCKET(INTERVAL '1 hour',
    CAST(data['location.properties.timestamp'] AS TIMESTAMP)) AS hour,
  AVG(CAST(data['location.properties.speed'] AS DOUBLE))               AS avg_speed,
  AVG(CAST(data['location.properties.altitude'] AS DOUBLE))            AS avg_altitude,
  MAX(CAST(data['location.properties.horizontal_accuracy'] AS DOUBLE)) AS worst_accuracy,
  COUNT(*) AS points
FROM 'gps_location_map.parquet'
GROUP BY hour
ORDER BY hour;

Variant

Finally, let’s get to the real meat of the story. Here we store the gps location data into a Parquet file where the relevant data is stored as a Variant.

COPY (
  SELECT
    Timestamp,
    'gps-location' AS service,
    CAST(CAST(SpanAttributes['location'] AS JSON) AS VARIANT) AS data
  FROM 'otel_traces.parquet'
  WHERE SpanName = 'gps-location'
) TO 'gps_location_variant.parquet' (FORMAT PARQUET);

DESCRIBE SELECT * FROM 'gps_location_variant.parquet';
┌────────────────────────────────────┐
│    gps_location_variant.parquet    │
│                                    │
│ Timestamp timestamp with time zone │
│ service   varchar                  │
│ data      variant                  │
└────────────────────────────────────┘

SELECT DISTINCT
  DATE_TRUNC('day', CAST(data.properties.timestamp AS TIMESTAMP)) AS day
FROM 'gps_location_variant.parquet'
WHERE CAST(data.geometry.coordinates[1] AS DOUBLE) BETWEEN 8.486986 AND 8.493896
  AND CAST(data.geometry.coordinates[2] AS DOUBLE) BETWEEN 47.348284 AND 47.353111
ORDER BY day DESC;

SELECT
  TIME_BUCKET(INTERVAL '1 hour',
    CAST(data.properties.timestamp AS TIMESTAMP)) AS hour,
  AVG(CAST(data.properties.speed AS DOUBLE))               AS avg_speed,
  AVG(CAST(data.properties.altitude AS DOUBLE))            AS avg_altitude,
  MAX(CAST(data.properties.horizontal_accuracy AS DOUBLE)) AS worst_accuracy,
  COUNT(*) AS points
FROM 'gps_location_variant.parquet'
GROUP BY hour
ORDER BY hour;

The double casting to JSON and then Variant is apparently doing the heavy lifting to ensure nested values are also properly typed. I can confirm that everything is nicely shredded after I save this file.

Materialized

This one again is not going to be an actual option for my use case as I need to know the schema up front, but it serves as a comparison.

COPY (
  WITH src AS (
    SELECT
      Timestamp,
      CAST(SpanAttributes['location'] AS JSON) AS loc
    FROM 'otel_traces.parquet'
    WHERE SpanName = 'gps-location'
  )
  SELECT
    Timestamp,
    'gps-location' AS service,
    json_extract_string(loc, '$.type')                                      AS location_type,
    json_extract_string(loc, '$.geometry.type')                             AS geometry_type,
    CAST(json_extract(loc, '$.geometry.coordinates[0]') AS DOUBLE)          AS lon,
    CAST(json_extract(loc, '$.geometry.coordinates[1]') AS DOUBLE)          AS lat,
    CAST(json_extract(loc, '$.properties.altitude') AS DOUBLE)              AS altitude,
    CAST(json_extract(loc, '$.properties.battery_level') AS DOUBLE)         AS battery_level,
    json_extract_string(loc, '$.properties.battery_state')                  AS battery_state,
    CAST(json_extract(loc, '$.properties.course') AS DOUBLE)                AS course,
    CAST(json_extract(loc, '$.properties.course_accuracy') AS DOUBLE)       AS course_accuracy,
    CAST(json_extract(loc, '$.properties.horizontal_accuracy') AS DOUBLE)   AS horizontal_accuracy,
    json_extract_string(loc, '$.properties.motion[0]')                      AS motion,
    CAST(json_extract(loc, '$.properties.speed') AS DOUBLE)                 AS speed,
    CAST(json_extract(loc, '$.properties.speed_accuracy') AS DOUBLE)        AS speed_accuracy,
    CAST(json_extract_string(loc, '$.properties.timestamp') AS TIMESTAMP)   AS gps_timestamp,
    CAST(json_extract(loc, '$.properties.vertical_accuracy') AS DOUBLE)     AS vertical_accuracy,
    json_extract_string(loc, '$.properties.wifi')                           AS wifi
  FROM src
) TO 'gps_location_materialized.parquet' (FORMAT PARQUET);

DESCRIBE SELECT * FROM 'gps_location_materialized.parquet';
┌──────────────────────────────────────────────┐
│      gps_location_materialized.parquet       │
│                                              │
│ Timestamp           timestamp with time zone │
│ service             varchar                  │
│ location_type       varchar                  │
│ geometry_type       varchar                  │
│ lon                 double                   │
│ lat                 double                   │
│ altitude            double                   │
│ battery_level       double                   │
│ battery_state       varchar                  │
│ course              double                   │
│ course_accuracy     double                   │
│ horizontal_accuracy double                   │
│ motion              varchar                  │
│ speed               double                   │
│ speed_accuracy      double                   │
│ gps_timestamp       timestamp                │
│ vertical_accuracy   double                   │
│ wifi                varchar                  │
└──────────────────────────────────────────────┘

The query here is very straight-forward.

SELECT DISTINCT DATE_TRUNC('day', gps_timestamp) AS day
FROM 'gps_location_materialized.parquet'
WHERE lon BETWEEN 8.486986 AND 8.493896
  AND lat BETWEEN 47.348284 AND 47.353111
ORDER BY day DESC;

SELECT
  TIME_BUCKET(INTERVAL '1 hour', gps_timestamp) AS hour,
  AVG(speed)               AS avg_speed,
  AVG(altitude)            AS avg_altitude,
  MAX(horizontal_accuracy) AS worst_accuracy,
  COUNT(*) AS points
FROM 'gps_location_materialized.parquet'
GROUP BY hour
ORDER BY hour;

Access logs details

Now let’s look at another dataset I have in my observability system: Access logs to this website. I build this page into a static bundle which I then serve using a Rust service. The only reason I have this service and reverse proxy via Caddy instead of just letting Caddy serve the static page directly is exactly these access logs. Caddy doesn’t give you insights per domain, but only aggregated metrics. So this code sets up an OpenTelemetry exporter and then logs events on request and on response.

Given the way OpenTelemetry ingests this data, the Events.Attributes property takes the following form. The two objects correspond to the two info! invocations in the code.

[
  {
    "code.filepath": "src/main.rs",
    "code.lineno": "67",
    "code.namespace": "marending_dev",
    "level": "INFO",
    "referrer": "",
    "request": "HEAD",
    "target": "marending_dev",
    "uri": "/",
    "user_agent": "updown.io daemon 2.11"
  },
  {
    "code.filepath": "src/main.rs",
    "code.lineno": "85",
    "code.namespace": "marending_dev",
    "latency": "424164",
    "level": "INFO",
    "status": "200",
    "target": "marending_dev"
  }
]

JSON

Now, going through the same spiel again, we extract the data into a Parquet file with the relevant fields cast to JSON.

COPY (
  SELECT
    Timestamp,
    'marending' AS service,
    CAST("Events.Attributes" AS JSON) AS data
  FROM 'otel_traces.parquet'
  WHERE ServiceName = 'marending'
) TO 'marending_json.parquet' (FORMAT PARQUET);

DESCRIBE SELECT * FROM 'marending_json.parquet';
┌────────────────────────────────────┐
│       marending_json.parquet       │
│                                    │
│ Timestamp timestamp with time zone │
│ service   varchar                  │
│ data      json                     │
└────────────────────────────────────┘

Here’s the first query that counts error responses by URI. The top result is /robots.txt if you care to know. .env is not much further down the list though.

SELECT
  json_extract_string(data, '$[0].uri') AS uri,
  COUNT(*) AS error_count
FROM 'marending_json.parquet'
WHERE json_extract_string(data, '$[1].status') LIKE '4%'
   OR json_extract_string(data, '$[1].status') LIKE '5%'
GROUP BY uri
ORDER BY error_count DESC
LIMIT 50;

And the second query to get the daily traffic by browser:

WITH r AS (
  SELECT
    Timestamp,
    json_extract_string(data, '$[0].user_agent') AS user_agent,
    CAST(json_extract(data, '$[1].latency') AS DOUBLE) AS latency_us
  FROM 'marending_json.parquet'
)
SELECT
  DATE_TRUNC('day', Timestamp) AS day,
  CASE
    WHEN user_agent LIKE '%Firefox%' THEN 'Firefox'
    WHEN user_agent LIKE '%Chrome%' AND user_agent NOT LIKE '%Edg%' THEN 'Chrome'
    WHEN user_agent LIKE '%Safari%' AND user_agent NOT LIKE '%Chrome%' THEN 'Safari'
    WHEN user_agent LIKE '%Edg%' THEN 'Edge'
    WHEN user_agent LIKE '%bot%' OR user_agent LIKE '%Bot%' THEN 'Bot'
    ELSE 'Other'
  END AS browser,
  COUNT(*) AS requests,
  AVG(latency_us) AS avg_latency_us
FROM r
GROUP BY day, browser
ORDER BY day, requests DESC;

Map

COPY (
  SELECT
    Timestamp,
    'marending' AS service,
    "Events.Attributes" AS data
  FROM 'otel_traces.parquet'
  WHERE ServiceName = 'marending'
) TO 'marending_map.parquet' (FORMAT PARQUET);

DESCRIBE SELECT * FROM 'marending_map.parquet';
┌────────────────────────────────────┐
│       marending_map.parquet        │
│                                    │
│ Timestamp timestamp with time zone │
│ service   varchar                  │
│ data      map(varchar, varchar)[]  │
└────────────────────────────────────┘

SELECT
  data[1]['uri'] AS uri,
  COUNT(*) AS error_count
FROM 'marending_map.parquet'
WHERE data[2]['status'] LIKE '4%'
   OR data[2]['status'] LIKE '5%'
GROUP BY uri
ORDER BY error_count DESC
LIMIT 50;

WITH r AS (
  SELECT
    Timestamp,
    data[1]['user_agent'] AS user_agent,
    CAST(data[2]['latency'] AS DOUBLE) AS latency_us
  FROM 'marending_map.parquet'
)
SELECT
  DATE_TRUNC('day', Timestamp) AS day,
  CASE
    WHEN user_agent LIKE '%Firefox%' THEN 'Firefox'
    WHEN user_agent LIKE '%Edg%'     THEN 'Edge'
    WHEN user_agent LIKE '%Chrome%'  THEN 'Chrome'
    WHEN user_agent LIKE '%Safari%'  THEN 'Safari'
    WHEN user_agent ILIKE '%bot%'    THEN 'Bot'
    ELSE 'Other'
  END AS browser,
  COUNT(*) AS requests,
  AVG(latency_us) AS avg_latency_us
FROM r
GROUP BY day, browser
ORDER BY day, requests DESC;

Variant

COPY (
  SELECT
    Timestamp,
    'marending' AS service,
    CAST(CAST("Events.Attributes" AS JSON) AS VARIANT) AS data
  FROM 'otel_traces.parquet'
  WHERE ServiceName = 'marending'
) TO 'marending_variant.parquet' (FORMAT PARQUET);

DESCRIBE SELECT * FROM 'marending_variant.parquet';
┌────────────────────────────────────┐
│     marending_variant.parquet      │
│                                    │
│ Timestamp timestamp with time zone │
│ service   varchar                  │
│ data      variant                  │
└────────────────────────────────────┘

SELECT
  CAST(data[1].uri AS VARCHAR) AS uri,
  COUNT(*) AS error_count
FROM 'marending_variant.parquet'
WHERE CAST(data[2].status AS VARCHAR) LIKE '4%'
   OR CAST(data[2].status AS VARCHAR) LIKE '5%'
GROUP BY uri
ORDER BY error_count DESC
LIMIT 50;

WITH r AS (
  SELECT
    Timestamp,
    CAST(data[1].user_agent AS VARCHAR) AS user_agent,
    CAST(data[2].latency AS DOUBLE) AS latency_us
  FROM 'marending_variant.parquet'
)
SELECT
  DATE_TRUNC('day', Timestamp) AS day,
  CASE
    WHEN user_agent LIKE '%Firefox%' THEN 'Firefox'
    WHEN user_agent LIKE '%Edg%'     THEN 'Edge'
    WHEN user_agent LIKE '%Chrome%'  THEN 'Chrome'
    WHEN user_agent LIKE '%Safari%'  THEN 'Safari'
    WHEN user_agent ILIKE '%bot%'    THEN 'Bot'
    ELSE 'Other'
  END AS browser,
  COUNT(*) AS requests,
  AVG(latency_us) AS avg_latency_us
FROM r
GROUP BY day, browser
ORDER BY day, requests DESC;

Materialized

COPY (
  SELECT
    Timestamp,
    'marending' AS service,
    -- Request event (index 1)
    "Events.Attributes"[1]['code.filepath']  AS request_code_filepath,
    "Events.Attributes"[1]['code.lineno']    AS request_code_lineno,
    "Events.Attributes"[1]['code.namespace'] AS request_code_namespace,
    "Events.Attributes"[1]['level']          AS request_level,
    "Events.Attributes"[1]['referrer']       AS referrer,
    "Events.Attributes"[1]['request']        AS request_method,
    "Events.Attributes"[1]['target']         AS request_target,
    "Events.Attributes"[1]['uri']            AS uri,
    "Events.Attributes"[1]['user_agent']     AS user_agent,
    -- Response event (index 2)
    "Events.Attributes"[2]['code.filepath']            AS response_code_filepath,
    "Events.Attributes"[2]['code.lineno']              AS response_code_lineno,
    "Events.Attributes"[2]['code.namespace']           AS response_code_namespace,
    CAST("Events.Attributes"[2]['latency'] AS BIGINT)  AS latency_us,
    "Events.Attributes"[2]['level']                    AS response_level,
    CAST("Events.Attributes"[2]['status']  AS SMALLINT) AS status,
    "Events.Attributes"[2]['target']                   AS response_target
  FROM 'otel_traces.parquet'
  WHERE ServiceName = 'marending'
) TO 'marending_materialized.parquet' (FORMAT PARQUET);

SELECT uri, COUNT(*) AS error_count
FROM 'marending_materialized.parquet'
WHERE status >= 400
GROUP BY uri
ORDER BY error_count DESC
LIMIT 50;

SELECT
  DATE_TRUNC('day', Timestamp) AS day,
  CASE
    WHEN user_agent LIKE '%Firefox%' THEN 'Firefox'
    WHEN user_agent LIKE '%Edg%'     THEN 'Edge'
    WHEN user_agent LIKE '%Chrome%'  THEN 'Chrome'
    WHEN user_agent LIKE '%Safari%'  THEN 'Safari'
    WHEN user_agent ILIKE '%bot%'    THEN 'Bot'
    ELSE 'Other'
  END AS browser,
  COUNT(*) AS requests,
  AVG(latency_us) AS avg_latency_us
FROM 'marending_materialized.parquet'
GROUP BY day, browser
ORDER BY day, requests DESC;

Results

[chart — see article]

Figure 1. Duration of GPS queries against different Parquet files

JSON, Map and materialized columns behave pretty much exactly like I would have expected. Only the one I actually cared about, Variant, is disappointing. I might be hitting this issue. Although at the time of writing, I get similar performance on a nightly build of DuckDB even though it’s supposedly fixed on main. I think this must truly be a limitation that will be addressed at some point. Looking at the query plan with EXPLAIN ANALYZE, it’s clear that a lot of variant extractions happen, even though in my understanding it shouldn’t have to do that.

In general, I’m impressed with the performance of DuckDB here. There are just over 30’000 rows in this dataset and it churns through that in no time even with JSON types.

Also interesting are the file sizes of the different parquet files, one can clearly see how the faster formats also compress the data better.

JSON: 2.2MB
Map: 1.3MB
Variant: 1.1MB
Materialized: 1.0MB

For the access logs, it looks similar. Although here this DuckDB limitation around Variant hits even harder. We must be hitting a highly unoptimized code path here to be that much slower than even JSON handling.

[chart — see article]

Figure 2. Duration of access log queries against different Parquet files

ClickHouse comparison

I don’t want to get too deep into this, as the goal here is primarily to establish the feasibility of querying my data from a Parquet file. But just to get a feel for the landscape of query engines, I’m going to translate the four queries I used here to ClickHouse equivalents and run them against the live system. Note that this is very much an apples-to-oranges comparison: The numbers thus far have been achieved on an M1 MacBook Air, the ClickHouse ones on a Hetzner CAX21. And the Parquet files have first been isolated to their respective datasets, while ClickHouse has an assortment of data in its otel_traces table. Not to mention that the Hetzner server is a noisy shared VPS.

[chart — see article]

Figure 3. Duration of queries against ClickHouse

Still, I’m honestly a bit surprised. I was expecting ClickHouse to do better. Looking back to this comparison where I looked at rather similar queries, ClickHouse was pulling way ahead of DuckDB. But there I was checking throughput rather than latency, maybe parallelism is where the client-server model shines. Either way, this only reinforces that DuckDB should be up to the task as I was satisfied with the performance of ClickHouse.

For fairness it should be mentioned that the schema used in ClickHouse is quite a bit more complicated than our neat Parquet files and uses the Map type. I would suspect we could go considerably faster by using ClickHouse’s own JSON type, which behaves similarly to Variant in my understanding.

Verdict

I think I’m reaching a different conclusion than what I set out to see, but I’m still not complaining. I was expecting to see subpar performance on JSON strings, with Variant saving the day and making my undertaking feasible. But I got the opposite: JSON handling is surprisingly fast in DuckDB and the Variant support is unfortunately too young for real-world use.

But the bigger take-away is that the general pattern of writing and querying Parquet files could actually be quite elegant. While I would be very nervous trying to alter the otel_traces table in a running system right now, rewriting Parquet files to use Variant down the line should be a walk in the park (save for rewriting queries of course). In a follow-up note I’ll have to home in on Parquet file handling, e.g. is there an impact when I have a single file that mixes data sources? And how to simultaneously write files periodically and query across different files?

Being terrible at leaving my side-projects be comes with plenty of downsides, but I do enjoy that the iteration tends to result in leaner and simpler systems.

Update from August 2026:

I just read the DuckDB 2.0 preview blog post. In it, they mention that Parquet becomes a first-class citizen, and excitingly, that JSON is going to be backed by Variant some time after 2.0. So JSON is probably the way to go: at first with pretty good performance, and then a sudden jump in performance as it becomes backed by Variant.

I was reinvigorated to check whether my benchmark queries would now be faster in the preview version of DuckDB 2.0. But unfortunately, it was not meant to be. Instead, I had DeepSeek v4 Flash experiment around to find the issue, and it didn’t dissappoint. Apparently projection pushdown in array types isn’t implemented yet. Those are, anyway, just an artifact from mapping the OpenTelemetry span attributes (an array) directly to the Variant type. After a quick cleanup unifying the objects inside the array into one big object, I was delighted to find performance within a factor of 2 of the “materialized” version. Great success.

So in my wide-event vision, this problem would not occur anyway.

ePaper air quality dashboard

Mon, 13 Apr 2026 22:00:00 GMT

14 Apr 2026

ePaper air quality dashboard

Should have just used a kindle

I’ve been collecting CO2 concentration, humidity, temperature and other properties of our apartment’s air for some time now. That data only lived in a ClickHouse instance with a Grafana dashboard to view it, but I’ve been missing a physical display for other people to view (and appreciate) the data. So I’ve finally set out to build a little ePaper dashboard.

At first I wanted to use an old Kindle Paperwhite I had lying around from years ago and build a little frontend application that renders charts using a plotting library. Then I realized that my Kindle was so old that it couldn’t run JavaScript to the extent required to chart data. Instead, I found this ePaper device by Elecrow for 30 bucks. It includes an ESP32-S3 to control the display. The original plan was to expose an endpoint from a service connected to the ClickHouse DB that serves the data I want to show on the dashboard. Then, the ePaper device would consume this data and render charts and values.

Once I started looking at the library the manufacturer ships to draw to the screen, it didn’t take long to realize that this wasn’t going to be so easy. Between all the comments in Chinese, it became clear that rendering an image on the server and merely painting that to the ePaper screen is the preferable way. If I went this way from the start, I probably could have made the Kindle work after all, although maybe still requiring jailbreaking.

Either way, I’ve now built an endpoint in my Rust service that renders an image comprised of a main CO2 chart as well as a couple of scalar values like current temperature and humidity, as well as - crucially - the current humidity in our laundry room.

The CrowPanel device then loads this image every 2 minutes and paints it to the screen. Unfortunately I didn’t get deep sleep working without the screen somehow showing weird banding, so the device currently just sits around idle in between cycles. If you care to see the code on the device, you can find it here. The code that renders the image can be found here. All said and done, I’ve now set up the device in kind of a hallway in our apartment (in an admittedly janky way; I’ll think of a more permanent fixture one day, surely).

On a side note, both the rendering code and the embedded code were heavily vibe-coded. Iterating on the rendering in particular worked very well. On the embedded side, despite having access to the whole drawing library and a bunch of example code, it took a lot of churning to get usable code. And even then, the device occasionally just seizes up — probably the error handling with Wi-Fi connectivity is flaky, or there may even be a memory leak. I’ll need to manually look into this.

My first firefox extension

Sat, 21 Mar 2026 23:00:00 GMT

22 Mar 2026

My first firefox extension

Personalize your software

I wouldn’t say I have an addictive personality, but I do spend too much time on YouTube. It’s a devious pattern where I watch educational content, learn a bunch of useless stuff and feel like I’m not wasting my time doing it. I don’t want to stop watching videos on YouTube entirely, there is some genuinely fantastic content on there. But I need to limit my time on it.

I’ve been aware of this pattern for years, but didn’t do anything about it. Then, I installed LeechBlock, a Firefox extension that allows you to set up block rules for certain websites. Like say, at most 30 minutes of YouTube per day and after that it will block further access. Depending on my mood, I would then habitually overrule the block. So it wasn’t exactly working.

Then, a couple of weeks ago I had an idea. What if I don’t just allow myself a maximum duration of YouTube per day that resets every day but instead keep a rolling balance that increases by X minutes every day and decreases by however much I use of it, effectively allowing a roll over of unused time? I postulated that this may help psychologically as not spending time on YouTube today is not a lost opportunity, but rather delayed gratification.

While LeechBlock has the option to roll over time, it doesn’t quite work the way I need it to. So I set out to build my own.

Starting out with a tutorial on MDN, I was surprised how easy it is to get a first demo extension built and installed. From there, I iterated with Claude to get the behaviour I needed. And boy did it take some prompting. While Claude understands the APIs available to an extension just fine, it struggled with concepts like timers not firing when the device is sleeping.

The result is this tiny extension. It keeps track of the remaining time budget and when it was last updated in local storage. Whenever YouTube is in focus, it updates the budget and the timestamp every second. When there is a visibility change, it checks when the budget was last updated and adds time as appropriate. I currently have it afford 2.5 minutes every hour, which comes to 1 hour of YouTube per day. You gotta start somewhere.

It’s worked well so far. As theorized, I have an easier time doing something else and not feeling like I’m missing out, as I’m just postponing watching that video where the guy explains how dishwashers work. I should build software that fits my needs exactly more often.

Update from the future: Overall, this has worked well. One loophole I started to exploit is that the hourly refill of 2.5 minutes gives enough time to find an interesting video (which is almost always longer than 2.5 minutes). Combined with the fact that access is only blocked on page reload, not mid-video, this is problematic. Instead, I’ve now resorted to allotting 45 minutes at midnight. This way, the budget is used up for the day, not just for the hour.

Storing SSH and encryption keys in 1Password

Sat, 17 Jan 2026 23:00:00 GMT

18 Jan 2026

Storing SSH and encryption keys in 1Password

My laptop is not a one-stop shop for attackers

I like to keep my hosting setup simple. That includes not operating a service for secret management on my server. Instead, I’ve opted to keep secrets in local .env files that are transmitted to the server during deployment. I also don’t run continuous integration pipelines for deployment, but just issue docker commands via a remote context to the server, which uses SSH for authentication. Naturally, for maximum comfort I don’t have a passphrase on my key. This simplicity comes at a security cost that I haven’t explicitly thought about before. But after looking into passkeys lately, security has been more present in my mind, so I’m writing this note to reason my way through this whole mess.

First, my approach of keeping secrets locally comes with a very practical concern: it’s very easy to accidentally commit .env files to the repo, exposing your API keys and such to everybody. This risk is quite obvious and, dare I say, not something I think would happen to me. Regardless, a mechanism where it’s so easy to do the catastrophically wrong thing is arguably badly designed either way. But more importantly, I’ve always considered my laptop, my local files, a sanctuary. A server hosting services that are exposed to the internet obviously need to be protected from all the bad actors out there, but my laptop? How could anyone possibly gain access to it? I could have easily listed a bunch of ways how that could happen theoretically, but I wouldn’t seriously have considered that I would have to take precautions against it. “Times of peace …” and all that.

A string of supply chain attacks have made me reconsider my stance. Not only do I myself use a bunch of dependencies on my projects, software I use have a bunch of dependencies. Did you know that an NPM package can run arbitrary code in lifecycle scripts? I.e. any one of your NPM dependencies can declare a postinstall script that is run on your machine after installing the package. Such a script could easily read your .env file or even your .ssh/id_rsa key and send it off to wherever! Or maybe it could install a little hook on your project that will modify the generated code subtly even after you’ve removed the dependency. The possibilities are endless. Even if you have your dependencies under control, any piece of software that runs on your system could have an exploit snuck in via one of their dependencies. Conversely not only does access to my computer compromise my productive secrets, it also allows an attacker to deploy malicious software on my behalf. It’s kind of depressing to think about.

After the “Shai-Hulud” compromise, of course the entire industry is trying to find solutions to this issue. In the meantime, I’m primarily concerned with trying to avoid having sensitive files lie around on my system. I don’t want to give up on the simplicity of my setup though, I still want to directly deploy software from my laptop to my server, and I also want to avoid secrets management on my server. So it’s time for one of my signature simple-yet-complicated solutions.

I use 1Password as my password manager of choice. It’s solid. Only recently did I learn that there is a CLI as well as an SSH agent. I immediately connected the dots with the conundrum in the back of my head.

Let’s consider the SSH key I use to log in to my server first. I created a new key in 1Password, set up the agent, transferred the public key to the server and then deleted the old key. Now, whenever I want to ssh into the server, the ssh client asks the agent to perform cryptographic operations using the private key. At that point, 1Password pops up a dialog that has me unlock the vault (using password or biometrics). Neat! To my surprise, this also works flawlessly when I run docker commands towards a remote ssh context. I did the same for my Github SSH key, as that could also be used to push malicious code to my projects.

As for my .env files, I now started using SOPS with age to encrypt the files and commit them to their respective repos. Here’s how this works. First, I generate a new age keypair.

> age-keygen
# created: 2026-01-18T19:10:14+01:00
# public key: age1kasp3na0g9vmp9htuduhn6fdxu2yc7am45lvg48ta97t66nypqys0zvm70
AGE-SECRET-KEY-1KKX7L4J7EXQQJA4K6XCS4SLZANGXEVKAX3H0T2L8Z7QYCUV5T3KQ5LX7TW

I store public and private keys in 1Password. Then, I create a new .sops.yaml file in the project.

# .sops.yaml
creation_rules:
  - path_regex: \.env.*$
    age: 'age1kasp3na0g9vmp9htuduhn6fdxu2yc7am45lvg48ta97t66nypqys0zvm70'

Now, when you run a SOPS encryption in this project, it will take the configured public key from this file.

> sops -e --input-type dotenv --output-type dotenv .env.prod > .env.prod.enc

# .env.prod.enc
OTEL_SERVICE_NAME=ENC[AES256_GCM,data:RfR0yTaqEbhh,iv:Lkx6qiB46iKye3ZUKPTXzjSv1VctQ0qWfZXYHmxOE1Q=,tag:IMxdFIF4aDX1AxuFEG9ZAg==,type:str]
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=ENC[AES256_GCM,data:vkHPgvhiLrjYAZeVGzPcxy+yMtLcdHQXehaW2XI1GJAIIPMp,iv:5SL6MO9HTKbsGh0U9bR7Xa1MzKP8FDBanaLHkXqAzRg=,tag:Mo3N1qes8ZLpuflkP1a+MA==,type:str]
OTEL_BSP_MAX_QUEUE_SIZE=ENC[AES256_GCM,data:PIETWdtc,iv:3aoQpSb9EDOAb8Qa0hU/LIafxa5t5VnKM6cRjmGcD6o=,tag:oCR8/cIbffN4lP410WpHcg==,type:str]
OTEL_BSP_MAX_EXPORT_BATCH_SIZE=ENC[AES256_GCM,data:xSdryjU=,iv:0xs74ILRRzpbUkWMhx5t3MYzODZYrjwAiXZPg9QbCaI=,tag:KXT93vSRIUxJQW2DkjcEEA==,type:str]
SERVE_PORT=ENC[AES256_GCM,data:I/XPtQ==,iv:+vjGknnpXIHX0HCThT/8SzBiFs7IsYjsl66faf78cGk=,tag:XnK5jT/hTOVijajfl53AUg==,type:str]
sops_age__list_0__map_enc=-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBRQ0hObVJ0VS91cktzcHZG\nM0xHZFZLdDVLSUlocVFpck8wM1g2YlBNdVZvClovbWs5UmVpVVo1MDlpZFg3U29i\nNFk0ZFZ2cUMwWUIyS1JpczdVTHlJdG8KLS0tIFNGNzl2V1RqN2s2SG9tbUEwc2or\nNzBncVg4L2s4MHNMUjB3MUU3WHJrVzAKoQUYuztqDD4Z2HJYoUhABXJmJGAGbGIr\nKnfZWZRpjawijjkPdW8WNslrxCQx5j3FFr4Fr1h+V1ueX14aHAvyKg==\n-----END AGE ENCRYPTED FILE-----\n
sops_age__list_0__map_recipient=age1tlteuzp4uwd7ld28s2ncstgmh92jja84y4a4frrl0hhwc04vpyzsqpqkyl
sops_lastmodified=2026-01-16T21:08:17Z
sops_mac=ENC[AES256_GCM,data:JjD3VlHY0ih4kItPRH7zmMOT5Qvklx8+xJxmTKKqa8OH6+j5bpM3hFXStcyXZ9qpNM/DNu50Hn6y7QKW3IrrLylF5EYLeFWk9Ys6/VmtbsjWUXuoEmsmHDcERtHo8AwBatzqwny+M1YRs/hJEqmgRQydKxg1c6ducIPv6fmlXfM=,iv:J0i6WWiccFtBbyUPvAU+KC3EmlYg+fIr3xXxjcmg+Rc=,tag:a2RmQ7P3XcuCnHTNOtcz/A==,type:str]
sops_unencrypted_suffix=_unencrypted
sops_version=3.11.0

The dotenv type is so that the resulting file is easier to diff in git. You can try without it and see the json format it defaults to (unless the input filename fits a pattern, don’t ask me). As long as the age private key is not compromised, this file is now safe to commit to the repo and the original .env.prod file is deleted.

At this point, there is nothing sensitive on disk in plaintext. But how can my deployment script still see the secrets? This brings us to the 1Password CLI. At the appropriate spot in my script (again, go read this for details) I insert the following lines.

trap "rm -f .env.prod" EXIT
export SOPS_AGE_KEY=$(op item get "SOPS age key - marending.dev" --reveal --fields "private key")
sops -d --input-type dotenv --output-type dotenv .env.prod.enc > .env.prod

The op command is the 1Password CLI that allows retrieving items from my vault programmatically. In this case, I’m reading out the “private key” field from the appropriate item. Then, with the key in an environment variable, I decrypt .env.prod.enc and store it in a plaintext .env.prod file. After this the actual deployment takes place. Finally, the trap we set up in the first line is going to ensure that the plaintext file is deleted when the script exits, be that cleanly or failing somewhere. If you’re wondering about the environment variable, rest assured that it’s only visible to the script and any child processes it spawns, not to unrelated process on the system.

So for a brief moment during deployment, the plaintext secrets are on disk, but then promptly deleted again. This is a big improvement over having them always lie around. My attack surface does expand to include 1Password, but if that were breached, I, and my software would be toast anyway with all my passkeys to Github, Hetzner, Porkbun etc. in there. In conclusion, I think this approach improves my security (and that of my users) significantly, at least in the face of opportunistic and automated attacks. But now that I think about this, an attacker that has read-write access to my system could modify my deployment script to send the private key off somewhere and I might not notice as I expect the verification prompt from 1Password.

If I get too pensive about these things, I start to realize all hope is lost anyway and I might as well go off-grid with some chickens.

Understanding passkeys

Tue, 13 Jan 2026 23:00:00 GMT

14 Jan 2026

Understanding passkeys

And improving my online security in the process

I’ve been meaning to look into passkeys for a while now. I always enjoy being informed on new developments in the industry, but for some reason this particular technology has been eluding me. A couple of misconceptions combined with entirely too much misinformation on the internet made passkeys unapproachable for me. Finally, I want to dive deep, properly understand them, and start using them. Luckily, I already have two Yubikeys lying about that I bought years ago for hardware-based OTP but never ended up using. They support the standards in question, so they will come in handy when playing around with passkeys.

What are passkeys

First, it’s important to establish a broad view of passkeys and their underlying standards. In practical terms, passkeys are cryptographic key pairs that are generated and handled by authenticators. These can be hardware-based like security keys or software-based like platform keychains or password managers. The WebAuthn web standard specifies how websites interact with the browser to register new passkeys and authenticate with existing passkeys. How the browser then talks to authenticators is captured in the Client to Authenticator Protocol (CTAP). Together, WebAuthn and CTAP make up the broader FIDO2 standard.

Registration and authentication flows

Let’s now have a simplified look at how clients, browsers and authenticators interact. First, the registration flow to establish new credentials:

Following this, existing credentials are used to authenticate:

Authenticators

Authenticators are worth having a closer look at, as they play a central role in passkey handling. They are commonly separated into platform and cross-platform (or roaming). Platform authenticators come built into your device, like Apple Face ID or Windows Hello. Cross-platform authenticators are portable. For instance, security keys or password managers are cross-platform, as they can be used from multiple devices. Smartphones can simultaneously be platform authenticators when signing in on device or roaming authenticators when used to sign in on another device.

The FIDO2 standard requires authenticators to at least support testing for user presence. A security key for instance will require the user to touch a capacitive button before a challenge is signed. The Apple Face ID authenticator will additionally support user verification via biometrics or PIN.

Notice that how key pairs are stored is not part of the specification. Some authenticators, like security keys, will generate a key pair and never hand out a private key. This means the key is tied to the physical possession of the device. Password managers on the other hand will offer to sync passkeys between your devices. This added comfort comes at the cost of having to trust your password manager to keep your keys safe in transit.

Advantages of passkeys

The security industry has learned from the way traditional passwords have been exploited and has thus built a bunch of improvements into the standards behind passkeys:

They are generated by the authenticator, not the user. This makes it impossible to accidentally generate weak passkeys, as is commonly done with passwords. As a result, passkeys are also automatically unique, as the user has no chance to reuse one.
The key to prove your identity and the key to check your proof are not the same. This means that the server does not need to protect a shared secret. If a site is breached, nothing of value about your ability to authenticate can be stolen. Passwords on the other hand are a shared secret between user and server, so the server must take great care to handle them responsibly.
They offer protection against phishing. Passkeys are bound to an origin, that means an attacker cloning a website cannot coax your authenticator into signing a challenge. Users with passwords on the other hand readily type them into a fake site that looks real.
They are easy to use. While most of us use password managers, the process of auto-filling username and password fields is error-prone, as it relies on parsing the web page for fields with appropriate attributes. Many sites do this wrong, leading to having to copy and paste passwords. Passkeys on the other hand build on proper programmatic APIs that can be easily implemented to specification by website, browser and authenticator. This should make the process seamless and robust.

Misconceptions

With the basics out of the way, let’s look at some common misconceptions to clarify our understanding.

If you lose the device with the passkey, you’re locked out

When I first learned about passkeys and wanted to try them, they were often used synonymously with security keys. So my biggest misunderstanding that prevented me from adopting passkeys was the belief that if I lose my Yubikey, I’m going to be locked out of my accounts. The common advice at that time was to get a second Yubikey and register a passkey on each device for every account. Then, deposit one security key in a safe spot so that if you lose the one that you have with you, there is a fallback.

This misses two crucial pieces of information. First, passkeys simply replace passwords, they don’t claim much more than that. So what do you do if you misplace your password? You enter a “Forgot my password” flow on the auth page of a service. Turns out you can do exactly the same if you lost your passkey. Most services will allow you to register a new passkey this way if you can prove your identity, usually by accessing your email account. Second, services typically don’t disable other authentication methods just because you registered a passkey. In the case of Github, for instance, you have the option to sign in via passkey or via username / password + two factor authentication.

Naturally, this has security implications. Your shiny new passkey is only as secure as your email account or your alternative authentication methods. For this reason, some services will allow you to disable other authentication modes and fallback mechanisms. Then, and only then, does the original fear of getting locked out materialize. You better keep those passkeys safe in this case.

Finally, most people are going to be saving their passkeys in synchronizing authenticators, like platform keychains or password managers. For these, losing a device doesn’t matter much anyway as long as you have a way to get back into the authenticator.

Passkeys don’t replace 2FA

This is quite a contentious take that illustrates how poor the general understanding of security is around the internet. You’ll read many misguided arguments that don’t hold up to scrutiny. After some research, it seems a reasonable argument goes as follows: If the authenticator employs user verification during authentication, passkeys do indeed provide similar security guarantees as passwords with a second factor. Access to the authenticator is something you have, and the biometrics or password you use to unlock your authenticator are something you are or know. Thus, you have two factors. Now, if your authenticator is not verifying you during authentication, but only tests for your presence (as the standard requires), you only have one factor: the physical possession of the authenticator. An example of such an authenticator is a security key that is not configured with a PIN and hands out signatures at the touch of a button.

Now, this line of reasoning feels suspect to me. By this logic, my use of strong passwords stored in a password manager would also qualify as two factors: access to the authenticator and biometrics to unlock. I suppose the big difference is that for passwords, a service cannot assume that they are handled correctly—strong, unique, and stored in a password manager—whereas passkeys are guaranteed to have these properties. Another iffy aspect of the argument is what happens when your password manager is breached. In that scenario, you’re immediately compromised. A password in a password manager and an OTP code provided by an authenticator app on your phone would save you from this.

So counting factors is not all that matters here. Separate password storage and OTP generation give you two trust anchors, versus just the one with passkeys. Instead, we have to consider that 2FA methods were conceived to reduce a user’s attack surface by making passwords no longer sufficient to authenticate alone. This way, attackers who gain access to passwords via phishing, database breaches, etc. can’t log in without also tricking you into providing the second factor. From this perspective, it’s clear why services like Github consider passkeys sufficiently strong to replace password + 2FA: they reduce this attack surface by binding keys to origins and eliminating shared secrets.

Also note that passkeys can be used as the second factor to strengthen passwords.

How do I use passkeys now?

Now that a decent understanding of passkeys has been established, what do I do with this information? First, let me elaborate on where I’m coming from. I was always very lazy when it comes to 2FA; to this date, I have 2FA enabled only on the services that force me to use it. As someone who considers himself a well-informed power user, this is a point of shame for me (hence I’m writing this note finally). However, after the above revelation that properly managed passwords are actually not that much worse than passkeys, it would be easy to feel validated and continue to do business as usual.

Instead, now that the uncertainty of how this technology works exactly is gone, I can focus on the ease of use of passkeys. Especially discoverable credentials (passkeys that contain metadata about the user) are just magical: You navigate to a login page, get a popup from your authenticator, put your finger on the fingerprint sensor, click on “Sign in”, and that’s it. No entering usernames, no scrambling to get your phone for the OTP code. That ease of use, coupled with the improved security, actually makes it a no-brainer to use passkeys.

But which authenticator to use? For me, built-in platform authenticators are not an option due to vendor lock-in. There is a draft specification now that will allow exchanging credentials between authenticators, but I’d rather use an authenticator that is portable from the get-go.

Further, there is also the opposite problem of vendor “lock-out”: Apple may ban your Apple ID just for buying a couple of gift cards, and Google may ban your entire account for buying Youtube Premium via a VPN in Turkey. This is also the reason I don’t use Gmail anymore, I would be putting too many of my digital eggs into one basket. An independent authenticator sounds more reasonable to me.

So password managers and security keys are still in the race. Password managers boast the following advantages:

Convenience & usability. They are easy to access via biometrics or password and are available on all devices.
Easy recovery. They synchronize passkeys across devices and into the cloud, ensuring they aren’t lost when a device breaks.

Security keys, on the other hand, have one main strength: security. Passkeys on a Yubikey are hardware-bound. No breach of any online service is going to compromise credentials.

Threat model considerations

The more I think about this, the more I realize that (apart from getting phished, of course) I worry more about getting locked out of accounts than about trusting nobody. I could only use security keys and eliminate trust in a password manager, but I believe the chances that I lose two security keys are higher than 1Password getting compromised.

Ultimately, I think hardware-based security is not worth the effort for me. The usability of password managers wins out for my use cases. But security keys can still have their place in my security theater. Not as exclusive auth methods, but as additional ones.

For critical services I really don’t want to lose access to, like email, password manager, and domain registrar, I’m going to register additional passkeys on both of my security keys. This way, I always have backup access if I lose access to my primary passkey. This could also serve as a nice way into my accounts for a spouse should something happen to me. Does this increase my overall attack surface? Probably. An attacker can now choose between hacking my password manager or breaking into my place, stealing my Yubikey, and somehow getting their hands on the PIN—whichever is easier. But again, I consider getting locked out more likely than someone targeting me to this degree. And if they do, the good old wrench method is anyway the most effective.

For the accounts that don’t support passkeys, I’ll now enable 2FA with a TOTP authenticator on my phone everywhere. Why not use the password manager’s TOTP functionality and do away with the phone altogether? Honestly, I’m not sure. I suppose when I trust the password manager not to be compromised, there is little benefit in having a second device.

Passkeys in the wild

First impressions of using passkeys with a password manager are good. It’s fast and smooth. The first problem arises when wanting to use a security key: first, one must dismiss the password manager prompt, then, when the native prompt presents itself with Apple’s keychain front and center, one must click on a small “Other options” button, and only then is there the option to use security keys.

Next, let’s have a look at specific services around the internet.

Github

When registering secondary passkeys on security keys, I have to set up a PIN. It looks like Github requires user verification, so just touching the capacitive button on the Yubikey is not sufficient, unlike with the other services below. The usability when logging in is kind of weird. When I already entered the PIN before, I can just touch the button on the login page and I’m in. However, when I’ve newly plugged the key in, touching the button doesn’t do anything; instead, I have to dismiss the password manager prompt, select security key, and then I’m prompted for the PIN and can log in. And now that I’ve set PINs on my keys for Github, I have to enter them for other accounts as well. I suppose most services set user verification to preferred, which doesn’t require it, but now that it’s set, I will get prompted.

Porkbun

Once a passkey is registered, password login doesn’t work anymore. Unfortunately, there is no option to influence this behavior, and it’s also not mentioned anywhere. Porkbun also offers the option to force user verification on the passkey.

Fastmail

Nothing special here, other than the fact that there is no direct way to add an arbitrary passkey. Instead, there are “on this device” and “on another device” options. Only with the latter does one get the option to use security keys.

Hetzner

Doesn’t support WebAuthn, only 2FA using OTP via mobile device or Yubikey.

Exoscale

This is an S3 object storage provider I use for backups. It doesn’t support passkeys and does support OTP via a mobile app, though it requires a phone number to serve as a backup second factor via SMS. Kind of disappointing for developer tooling. It’s widely known at this point how insecure SMS-based 2FA is. And if it’s required as a backup, well, the whole chain is only as strong as the weakest link.

1Password

Added OTP for password login and passkeys on both security keys as second factors. 1Password doesn’t seem to offer registering passkeys as the only authentication mechanism, only as second factors. Weirdly, even if I sign out everywhere, 1Password never asks for any second factors. Apparently, this is only required on new devices.

Google

This is the first service where I get a popup from Firefox warning me that the website is requesting extended information about my security key. Apparently, this exposes device identifiers of the security key to enable security-critical websites to disallow insecure or compromised keys. A (surely only accidentally useful for Google) side effect is that your keys could be tracked across multiple websites this way.

Closing thoughts

I’m glad I finally looked into passkeys. Not only are they convenient as a user, but I’m inclined to also look into WebAuthn as the sole authentication method for future side projects. To this end, I’ll probably write up a proof-of-concept implementation in a future note.

This was also helpful to finally get my security hygiene in order. I’d say I’m now in a fairly good spot: every account of value has at least 2FA enabled. While I’m at it, I’ll also look into putting my SSH keys into 1Password or a security key to avoid having them lying around on my system.

As the resident tech person in my circle, I’ll recommend people start adopting passkeys using a password manager going forward.

Visual regression tests for my website

Mon, 05 Jan 2026 23:00:00 GMT

06 Jan 2026

Visual regression tests for my website

Gaining confidence in refactorings

This website is built using Astro to generate static pages. I author the notes themselves with mdx, a nice extension to markdown to include inline html and other components. The static html after building this site is then styled using some rather convoluted CSS. All this is to say that if I change, for example, a margin of a list item only if it precedes an image element, this may have unintended consequences on an older note I don’t look at often.

Whenever I do such changes, I find myself sampling older notes to see if something is broken. Lately, I’ve had the idea to use Playwright to do visual regression testing. For the uninitiated, this type of testing simply takes automated screenshots of pages (using a headless browser typically) and compares them against an earlier, considered golden, snapshot. Should the image deviate by more than some configurable threshold, the test is considered a failure. Then you either fix your application or in case of a legitimate change, you simply update the golden snapshot for that particular test.

These tests would come with the obvious upside of increasing confidence that changes don’t have unintended side-effects. Especially for a static website where the visual appearance is really all there is to it. But further, because I check the screenshots into the git repo, I get an automatic history of what the site looks like at the time of the commit.

Technical implementation

Playwright makes this quite easy out of the box. After initializing a new test project using npm init playwright@latest in the same repo as the website itself, I add this single test file:

import { test, expect } from '@playwright/test';

const notes = [
  '/',
  '/projects/',
  '/about/',
  '/notes/launchd/',
  '/notes/jour/',
  '/notes/reflective/',
  '/notes/monitoring/',
  '/notes/otel/',
  '/notes/llm/',
  '/notes/clickhouse/',
  '/notes/server-setup/',
  '/notes/jpeg-raw/',
  '/notes/go-rest-quest/',
  '/notes/responsive-plots/',
  '/notes/co2-loft/',
  '/notes/sqlite-vs-duckdb/',
  '/notes/unstructured-data/',
  '/notes/rest-quest/',
  '/notes/fieldnotes/',
  '/notes/rust-spa/',
  '/notes/16-hour-projects/',
  '/notes/wasm-benchmark/',
  '/notes/vps-benchmarks/',
  '/notes/sqlite-benchmarks/',
  '/notes/league-rating/',
  '/notes/league-data/',
  '/notes/co2-bedroom/',
  '/notes/esp-protocol/',
  '/notes/esp-power/',
  '/notes/performance/',
  '/notes/website/',
  '/feedback/',
];

test.describe('Visual regression', () => {
  const baseUrl = 'https://marending.dev';

  for (const note of notes) {
    test(`capture page: ${note}`, async ({ page }) => {
      const url = `${baseUrl}${note}`;

      await page.goto(url);
      await page.waitForTimeout(200);

      const pageHeight = await page.evaluate(() => document.body.scrollHeight);

      for (let scrolled = 0; scrolled < pageHeight; scrolled += 200) {
        await page.mouse.wheel(0, 200);
        await page.waitForTimeout(200);
      }

      await page.waitForLoadState('networkidle');

      const screenshotName =
        note
          .replace(/^\//, '')
          .replace(/\/$/, '')
          .replace(/[^a-z0-9]/gi, '-')
          .toLowerCase() || 'index';

      await expect(page).toHaveScreenshot(`${screenshotName}.png`, {
        fullPage: true,
      });
    });
  }
});

There are a couple of things to note here. The magic happens on the await expect(page).toHaveScreenshot line. This makes Playwright take a screenshot and compare it against a stored screenshot. If no screenshot with this name exists, it will fail the test and you have to first generate a screenshot by running your suite with --update-snapshots.

Second, there is some complication involved with taking full page screenshots. My website lazy-loads images, which means some images aren’t loaded when the page is sufficiently long. I wouldn’t care so much about that if it wasn’t flaky whether some images are loaded or not. I noticed that sometimes particular images were loaded and sometimes not, which defeats the purpose when trying to look for pixel differences between snapshots. For this purpose, you’ll notice the whole scrolling logic in the test: I scroll down the whole page 200 pixels at a time to ensure all images are loaded in.

Lastly, the list of pages I want to test are statically listed in the notes variable. At first, I actually generated this dynamically by programmatically visiting the index page and then extracting all linked targets. In the current design of the site, this yields exactly all subpages. Another way would be to expose an “endpoint” in the site that produces all the pages in the notes collection. Both approaches have the benefit of not requiring me to update the list manually when I publish a new note, but come with the downside that I need to execute all tests in a single Playwright test.

You see, this test-inside-for-loop you can see above only works as expected when the array to iterate over is statically known. In the dynamic approaches I can’t do that. And then you have to deal with the test failing once the first screenshot doesn’t match, instead of getting a nice summary in the case where each page is its own clean test.

Workflow

So how do I use this? It would be easy to over-engineer it and run this in CI periodically or build it into my deployment script. Instead, I decided to keep it simple and stupid. I have this setup with the images checked into the same repo as the website itself and I run the tests whenever I feel like I’ve made changes that could affect some other part of the site. There is no point in burning energy by running them on every commit or constantly failing my deployment just to confirm that changing a typo on a page does in fact cause visual changes.

With such simplicity in mind, it’s easy to add real value to my workflow with maybe 2 hours of effort. I need to do more things like it.

Using launchd on macOS to run periodic tasks

Thu, 01 Jan 2026 23:00:00 GMT

02 Jan 2026

Using launchd on macOS to run periodic tasks

Surveilling myself

Some months ago I had the idea to record some data about my computer use for later analysis. I’m recording the window title of the focused application every 2 minutes and I take screenshots of my display (main as well as secondary if connected) every 10 minutes. Once this has been running for a while longer I can analyze how I spend my time on my computer.

In this note I just want to briefly go into how to achieve this easily on macOS natively using launchd. It’s quite easy to do but the documentation is lacking, hopefully this write-up clears things up.

First, I have a bash script I want to execute periodically. Below is my script for recording the application and window title in focus. Don’t ask me why AppleScript (the thing passed to osascript) is so weird. If you execute this script, it will append the desired data to a file.

#!/usr/bin/env bash
application=$(osascript -e 'tell application "System Events" to tell (first process whose frontmost is true) to return {name}')
title=$(osascript -e 'tell application "System Events" to tell (first process whose frontmost is true) to return {name of window 1}')
date=$(date +"%Y-%m-%dT%H:%M:%S")

if [ -n "$title" ]; then
  echo "$date,$application,$title" >> /Users/florian/Projects/aware/window-focus/windows.csv
fi

The other script I have running is the one to take screenshots. The two code blocks are there to capture two displays (if connected). Within each block you can see the weird mean conditional, this is there to filter out black screens. It seems like the way we’ll schedule these scripts sometimes runs them while the device is actually sleeping and thus has the screen off. The magick invocation compresses the screenshot for storage.

#!/usr/bin/env bash
date=$(date +"%Y-%m-%dT%H:%M:%S")
filename="screenshot_$date"

# Screenshot of primary display
if screencapture "/Users/florian/Projects/aware/screenshots/data/screenshot.png"; then
  mean=$(/opt/homebrew/bin/magick identify -format "%[fx:mean]" "/Users/florian/Projects/aware/screenshots/data/screenshot.png")
  if (( $(echo "$mean > 0.26" | bc -l) )); then
    /opt/homebrew/bin/magick -quality 80% "/Users/florian/Projects/aware/screenshots/data/screenshot.png" "/Users/florian/Projects/aware/screenshots/data/$filename.jpg"
  fi
  rm "/Users/florian/Projects/aware/screenshots/data/screenshot.png"
fi

# Screenshot of secondary display
if screencapture -D 2 "/Users/florian/Projects/aware/screenshots/data/screenshot-2.png"; then
  mean=$(/opt/homebrew/bin/magick identify -format "%[fx:mean]" "/Users/florian/Projects/aware/screenshots/data/screenshot-2.png")
  if (( $(echo "$mean > 0.26" | bc -l) )); then
    /opt/homebrew/bin/magick -quality 80% "/Users/florian/Projects/aware/screenshots/data/screenshot-2.png" "/Users/florian/Projects/aware/screenshots/data/$filename-2.jpg"
  fi
  rm "/Users/florian/Projects/aware/screenshots/data/screenshot-2.png"
fi

Next, we need to describe our schedule in an xml file. Here’s the one for the window title recording. The script is called windows.sh. Make sure to use absolute paths everywhere, including the shell scripts as the whole thing will run under a different user.

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
	<key>Label</key>
	<string>local.aware.window-focus</string>
	<key>Program</key>
	<string>/Users/florian/Projects/aware/window-focus/windows.sh</string>
  <key>StartInterval</key>
  <integer>120</integer>
	<key>StandardOutPath</key>
	<string>/Users/florian/Projects/aware/window-focus/log.stdout</string>
	<key>StandardInPath</key>
	<string>/Users/florian/Projects/aware/window-focus/log.stdin</string>
  <key>StandardErrorPath</key>
	<string>/Users/florian/Projects/aware/window-focus/log.stderr</string>
</dict>
</plist>

Finally, to install the job I run the following commands in the terminal. If you’re doing it for the first time you can probably omit the first line. From what I can tell there is no easy way to determine if a job was actually installed correctly. I just set the interval low at first and see if it works, then increase it later. Also, in the log.stderr file you can see if your script didn’t execute successfully.

launchctl unload local.aware.window.plist
cp local.aware.window.plist ~/Library/LaunchAgents/local.aware.window.plist
launchctl load -w ~/Library/LaunchAgents/local.aware.window.plist

A last bit of advice: If you want to do stuff like take screenshots, this requires accessibility permissions on macOS. You have to grant those permissions in the settings to the /usr/bin/env process.

jour: An opinionated lab journal

Wed, 31 Dec 2025 23:00:00 GMT

01 Jan 2026

jour: An opinionated lab journal

A project report

Years ago I stumbled across a blog post on lab notebooks. The author argues that practitioners in many fields keep a journal where they detail what they’re up to, what’s working and what’s not etc. in their daily work. For some fields such a journal even has legal implications when patents were disputed. In software engineering however, this is not widespread. At the time I read the post I knew of no one who kept such notes.

Sure, we have documentation (maybe), but that’s not the same thing. Documentation is there to explain how a system works, how to interact with it, maybe what constraints it operates under. But documentation typically doesn’t explain how and why we arrived at this particular solution. Did we try to solve the problem in a different way first but failed, leading to the current approach? You won’t find that in the documentation. Old attempts either weren’t committed to the code base or were squashed away.

This blog post opened my eyes to an issue I had at work that I couldn’t even put my finger on before. In my current project at work, I often run into situations where we’re reevaluating a design decision from two years ago because someone thought of a better way to do it. Inevitably, no one remembers why it was built the way it is. But I would often have this nagging feeling that we considered the proposed solution back then but decided against it for reasons. Now the big question (possibly worth weeks of effort) is whether we just didn’t think of it back then or whether we’re missing the reason now for why we didn’t do it. Or, some surrounding factor changed in the meantime, now enabling the better approach. If I had kept notes on my attempts, failures, assumptions etc. I could simply read up on this now and we would have clarity. Alas, those notes don’t exist.

Naturally, I resolved to start a lab journal. But instead of using my usual notes app, I had the feeling I needed something more specialized for the occasion. An application that would keep entries in a chronological feed, rather than siloed notes. Enter jour, a simple web app for keeping journal entries with no hassle. It’s local-first with optional S3 sync. A hosted version is available at jour.rest.quest. As you can tell by the URL, this application is another instance of my local-first series kicked off by rest.quest. I would recommend reading that project report first for the technical details, here I will only go into the feature set and learnings from this project.

Features

If you’ve seen the project reports for rest.quest or go you know the drill by now: Brutalist, minimalist UIs are my jam. Below is the default view of jour, a complete feed of every entry I’ve written in reverse chronological order. Each entry has a text body, some tags and a date. The tags are parsed out of the content when submitting an entry simply by looking at words starting with a # in the last line. Tags do not get special treatment other than being displayed in a lighter tone below the content.

On the top, there is a search bar that searches through all entries and filters down the feed accordingly. I started using this application a while ago and was quite satisfied with it. The one gripe I had was that when I was writing entries at work, I wanted to automatically have the entries tagged by a special tag (nca in my case). Instead of always having to type out this tag manually, I later added the things you see on the left. When clicking on nca, there is an implicit filter for this term that doesn’t show up in the search bar, and every new entry automatically gets tagged correspondingly. These “autofilters” can be created via shortcuts and deleted via a button that is revealed on hover.

Next, by hitting n, a new entry is created at the top with a textbox and a date picker. By default the datepicker shows the current date and time, but I quickly found that sometimes one wants to create entries in the past, so I added the ability to set the date. With cmd+Enter the entry is saved.

That’s pretty much the functionality that I use every day. One additional bit of eye-candy I added later is a statistics screen that can be toggled. It shows a GitHub-style heatmap of all entries laid out over time.

Here you can see that I’m quite consistent in using jour. The week(s)-long gaps are almost always vacations.

At the time of writing there are 848 entries saved. Many of them quite short, some are longer. There is no progressive loading of entries, everything is shown on one page. Turns out you can go quite far with such a simple UI when you build it to your specifications. If the sheer number of entries grows to a point where this becomes laggy, I’ll reconsider.

Learnings

Technologically, jour is nothing to write home about. It’s a slight variation of my rest.quest-winning-formula. That’s also the reason I could build this out in a mere couple of hours. The interesting learning for me in this project is more in the direction of disjoint, use-case-specific applications vs integrated multi-purpose apps. At first, I was very happy with my new-found trifecta of productivity applications: jour as a lab journal, rest.quest as a todo list and fieldnotes as the notes app. Over time, I started noticing an indecisiveness about where to record some things. For instance, after figuring out a shell one-liner to fix an issue I was experiencing, I wouldn’t know whether to jot this down in the lab journal or as a new note in the notepad. Or taking notes on what’s being discussed in a sync meeting to refer to later. Dedicated note or quick entry in jour?

This conundrum has kicked off thoughts I can’t fully verbalize yet. In classical me fashion, one option in the room is to build an integrated application that allows bringing all this data together. But then again, moving away from single-purpose, focused applications to bloated monoliths doing too many things badly is not exactly a mistake I want to repeat. We’ll see where this goes.

Reflective: A simple self-hosted image gallery

Mon, 29 Dec 2025 23:00:00 GMT

30 Dec 2025

Reflective: A simple self-hosted image gallery

A project report

Let me take you back to around 6 years ago. I was starting to de-Google my life, replacing Google Drive and Google Photos with pCloud. Now, while pCloud works fine as a cloud storage provider, it completely fails at the photo experience. At that time, viewing images was painfully slow, thumbnails took 10 seconds to load. It felt like they were regenerating thumbnails on every view.

This made me wonder if hosting and efficiently serving images was really that hard, or if pCloud simply didn’t care about the user experience. Google was pulling it off, but maybe that required hundreds of engineers and specialized compression schemes.

So I started my first real side project (apart from some smaller CLI tools) with foti. This is code I wrote with around 1 month of on-the-job frontend experience. View it at your own risk. I built a Python server that eagerly downsampled images for thumbnails and served them via a WebSocket connection (for whatever reason). The whole thing ran on a Raspberry Pi 4 in a box in my cupboard.

While the result wasn’t pretty, it validated my suspicion: Serving images at reasonable speeds wasn’t that hard. With a proof-of-concept done and my appetite for side projects awakened, I decided to first hone my web dev skills on an “easier” target: note taking. I thought I’d quickly build a note-taking application to satisfy a personal need, learn a ton in the process, then swiftly return to my actual goal — the image gallery.

If you’ve read about fieldnotes, you’ll know this became much more than a short intermezzo. I lost myself there for a couple of years, then a bunch of other projects snuck in, and here we are, quite a while later. When I finally returned to it, I started sketching out the feature set and architecture, only to be distracted again by another project. The second time around, I drastically cut the scope and finally pushed it over the line. This is the version I’m writing about now. As always, there are plenty of learnings here. Reflecting on those is the main reason I write these project reports.

Features

Reflective is currently a simple gallery for distraction-free viewing of my images. When you open the web app, you’re greeted with a grid of all the images ingested into the system:

No buttons or labels to distract from the images. When scrolling, the application seamlessly loads more images. All images are cropped to a 4:3 landscape aspect ratio.

There’s no categorization into albums or folders, just a simple feed of images in reverse chronological order.

Images can have any number of tags associated with them. To find specific images, you can open the search bar via shortcut (cmd+k) and search for any tag. In the following screenshot, the images are filtered to ones tagged japan. There are 1860 images matching that query.

To tag images, open the tag view (cmd+e) and select individual images, or select all images between two selections by holding down Shift. The tag bar shows tags that all selected images share, tags that only some images have (in lighter gray), and an input field to add new tags to all selected images.

Back in normal view mode, clicking an image opens it in fullscreen. A lightbox fills the screen with shortcuts to navigate between images and load the full-resolution version for zooming into details.

Hitting cmd+i displays selected metadata below the image.

To ingest images into reflective, I copy files into a folder on my server and create a marker file in a specific location. The server recognizes the new files and starts downsampling them for thumbnails. Once complete, the new images appear in the view.

Architecture

As with rest.quest, I first brainstormed on paper what the application should do and how it should work. For reflective, this became quite a ramble, but I enjoy this process. In hindsight, I once again over-engineered everything — it’s always easy to dream up mechanisms and features, but pragmatism only kicks in when it’s time to implement.

In the scans below, you’ll find my entire thought process detailed.

Full concept

See if you can find the angry cow in prison.

At first, I was convinced I should store image data in S3. After all, storing images on my server wasn’t scalable. Architecturally, collocating that much data with the application server is a bad idea. But storing the data and metadata in separate places creates a synchronization problem. You’ll find plenty of musing about this issue in my notes.

Once again, I expertly ignored the fact that I’m the only user and could just skip this whole problem. After implementing the entire system with upload functionality, presigned S3 URLs, and all that complexity, I unceremoniously ripped it out. Instead, I now store images on the server, have the server asynchronously process them for thumbnails, and serve the images directly.

The architecture has been simplified to the point where there’s nothing left to discuss.

Learnings

Reflective continues my eternal battle against over-engineering. By now, I have the concept of MVP scope down — I always try to simplify to a barely usable application and iterate from there. The problem this time was that my scope included functionality for an imaginary user base. I hadn’t yet internalized the idea of home-cooked software.

Halfway through the concept, I even wrote this note to myself:

Reminder to build for yourself and not over-engineer for an imaginary user-base!

Me (to myself)

And yet I carried on designing a system that would scale to lots of users instead of focusing on what I needed. Building significant functionality only to rip it out later hopefully taught me to recognize these situations earlier next time.

Cheap and simple service monitoring

Fri, 24 Oct 2025 22:00:00 GMT

25 Oct 2025

Cheap and simple service monitoring

An ode to updown.io

I’m hosting a bunch of software on a VPS. Among them a todo list, a note-taking application, and of course my personal website. It would be a hassle if that were to go down without me noticing for an extended period of time. Especially my notes app contains information that I sometimes urgently need access to. Naturally, I want to know sooner rather than later if my server or individual deployments go down. For this reason, I was looking around for uptime monitoring tools some time ago.

Now, 2 years of using updown.io later, I want to highlight this neat little service. I wish more software was built like it.

Feature set

updown effectively comes with two features:

uptime monitoring
pulse monitoring

Uptime monitoring is the core feature you expect from any monitoring service. It hits your website/webapp with a request regularly, checking the response status, response time, SSL certificate expiration, etc. in the process.

Pulse monitoring works the other way around. You’re expected to hit an endpoint at updown regularly to indicate that you’re alive and well. This is very useful to me to ensure backups are running: I have backup scripts that upload DB dumps to S3 and send a pulse to updown upon successful completion. If that script stops working for some reason (e.g., expired S3 credentials), the monitor would alert me.

And then there is some other stuff you would expect but I don’t use: status pages, API access, etc. That’s pretty much it. No automated browser tests, no integrated tracing, no bells and whistles. Just the rock-solid essentials.

And I’ve noticed that this kind of minimalism encourages more home-grown solutions. Say I not only want to check that a service is up, but also that the API is working correctly. Some monitoring services offer automated API testing for this purpose. To achieve this with updown, I would instead deploy a container that regularly runs these tests and reports success via a pulse check.

The hard thing you don’t want to do yourself is the alerting, that’s the part that requires independent hardware that won’t go down along with your infrastructure. That’s the part updown takes over. The rest— the testing framework I use, how I run them, etc.—I don’t want to marry to a monitoring solution. Some platform may offer a way to write API tests in JS, but I may prefer hurl. The fact that I’m left to construct my own solution around the essentials is getting more attractive the more experience I gain.

Of course, one should always carefully consider the maintenance burden of building your own solution. But the cost of the hosted, locked-in solution is often underestimated. And I’m not just talking about monetary cost.

Pricing

Right in line with the philosophy behind this service, the pricing is extremely reasonable. You buy credits upfront, and then your checks consume the credits. For my modest usage, that comes out to less than 50 cents per month. How refreshing.

I often feel annoyed with the pricing services go with: a free tier that has limited functionality, followed by a paid tier that goes well beyond my usage but costs, say, 15 bucks a month. I want to give them money; I want them to have a stable, sustainable business. But I also want to pay a fair amount based on my usage. It doesn’t make sense that I pay the same amount of money whether I check 1 website once an hour or 10 websites every minute. There needs to be some gradation.

Anyhow, I’m sure I’m preaching to the choir. I also understand the business’s perspective, subscription models make way more money, and small fish customers are probably just not worth the support obligation. All the more reason to celebrate the businesses that do offer such friendly pricing.

Rust tracing to ClickHouse

Fri, 18 Jul 2025 22:00:00 GMT

19 Jul 2025

Rust tracing to ClickHouse

This shouldn't be this hard

I don’t like the OpenTelemetry ecosystem. It’s complicated. Not just because it’s solving a hard problem, but it feels needlessly complicated. I get the impression the observability vendors are begrudgingly adopting an open standard but want to make it a daunting task to host your own observability stack. Anyway, I’ve recently struggled for a bit to instrument a Rust application with tracing and send the data to ClickHouse. So here is a short write-up to save you a hot minute—or provide some inspiration to self-host your observability platform.

Rust application

The Rust crates involved here are unstable and moving fast, part of the reason this was trickier than expected. But at the time of writing, here is what works.

// Cargo.toml
tracing = "0.1.40"
tracing-subscriber = "0.3.18"
opentelemetry_sdk = { version = "0.30.0", features = ["rt-tokio"] }
opentelemetry = "0.30.0"
opentelemetry-otlp = "0.30.0"
tracing-opentelemetry = "0.31.0"
opentelemetry-stdout = "0.30.0"
opentelemetry-semantic-conventions = "0.30.0"

// main.rs
use opentelemetry::{global, trace::TracerProvider};
use opentelemetry_otlp::WithExportConfig;
use opentelemetry_sdk::trace::SdkTracerProvider;
use tracing::{error, info, span, Span};
use tracing_opentelemetry::OpenTelemetryLayer;
use tracing_subscriber::{layer::SubscriberExt, util::SubscriberInitExt, Layer};

std::env::set_var("OTEL_SERVICE_NAME", "your-application");

let tracer = opentelemetry_otlp::SpanExporter::builder()
    .with_http()
    .with_endpoint("http://localhost:4318/v1/traces")
    .build()?;

let provider = SdkTracerProvider::builder()
    .with_batch_exporter(tracer)
    .build();

global::set_tracer_provider(provider.clone());

tracing_subscriber::registry()
    .with(
        tracing_subscriber::fmt::layer()
            .with_filter(tracing_subscriber::filter::LevelFilter::WARN),
    )
    .with(
        OpenTelemetryLayer::new(provider.tracer("your-application"))
            .with_filter(tracing_subscriber::filter::LevelFilter::INFO),
    )
    .init();

Observability compose stack

Next, we want to run the OpenTelemetry Collector from the contrib repository, a ClickHouse instance as well as Grafana.

// compose.yaml
services:
  clickhouse:
    image: "clickhouse/clickhouse-server:25.6"
    container_name: clickhouse
    ports:
      - 18123:8123
      - 19000:9000
    restart: unless-stopped
    environment:
      - CLICKHOUSE_USER=default
      - CLICKHOUSE_PASSWORD=default
      - CLICKHOUSE_DB=your-db
    ulimits:
      nofile:
        soft: 262144
        hard: 262144
    healthcheck:
      test: ["CMD", "wget", "--no-verbose", "--tries=1", "--spider", "http://localhost:8123/ping"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 40s

  otel-collector:
    image: otel/opentelemetry-collector-contrib:0.128.0
    container_name: otel-collector
    ports:
      - 4317:4317
      - 4318:4318
      - 55679:55679
    volumes:
      - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml
    command: ["--config=/etc/otel-collector-config.yaml"]
    restart: unless-stopped
    depends_on:
      clickhouse:
        condition: service_healthy

  grafana:
    image: grafana/grafana:latest
    container_name: grafana
    ports:
      - "127.0.0.1:3000:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin
    volumes:
      - grafana-data:/var/lib/grafana
    restart: unless-stopped
    depends_on:
      clickhouse:
        condition: service_healthy

volumes:
  grafana-data:

// otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:

exporters:
  clickhouse:
    endpoint: tcp://clickhouse:9000
    database: your-db
    username: default
    password: default
    ttl: 72h
    create_schema: true
    timeout: 5s
    retry_on_failure:
      enabled: true
      initial_interval: 5s
      max_interval: 30s
      max_elapsed_time: 300s

extensions:
  health_check:
    endpoint: 0.0.0.0:55679

service:
  extensions: [health_check]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [clickhouse]

Once you bring up these containers, you should be able to run your Rust application and start sending tracing data to the collector. By manually connecting to the ClickHouse instance with something like TablePlus, you should see an otel_traces table with your spans in there.

Grafana

Navigate to http://localhost:3000 for the Grafana frontend. Here you must install the appropriate data source plugin. Finally, you can start writing queries like this to visualize your tracing data:

SELECT
  toStartOfInterval("Timestamp", toIntervalSecond(${__interval_ms}/50)) as time,
  count(*) as trace_count
FROM
  "your-db"."otel_traces"
WHERE
  (
    Timestamp >= $__fromTime
    AND Timestamp <= $__toTime
  )
  AND (ParentSpanId = '')
  AND (Duration > 0)
GROUP BY
  toStartOfInterval("Timestamp", toIntervalSecond(${__interval_ms}/50))
ORDER BY
  time ASC

Exploring LLMs

Thu, 17 Jul 2025 22:00:00 GMT

18 Jul 2025

Exploring LLMs

A skeptic reconsiders

I’ve been a long-time skeptic of large language models and their applicability to intellectual work. Too readily do they hallucinate information, too amateurish are their programming mistakes. So for too long, I didn’t give them a proper chance to redeem themselves. Now, I’ve spent a couple of focused days reevaluating my stance, first by catching up on the latest advancements in transformer architecture and then by putting these models to the test.

This is mostly a worklog I was writing as I went along. The structure might be a bit weird as a result.

Goals and summary

Learn technical details of how LLMs work: Excellent videos by 3Blue1Brown and Andrej Karpathy.

Explore modern local models: Underwhelming. Models small enough to run on my laptop leave much to be desired.

Explore modern LLM tools and agents: Claude Code phenomenally useful for coding. Claude also useful for a bunch of other tasks.

Developing an intuition for how LLMs work

I came into this with some knowledge of machine learning and neural networks from university courses. I even read the OG transformer paper back then but apparently didn’t take much away from it. So previously I would consider LLMs as the classical stochastic token predictors that don’t think, but only produce text that conforms to previous patterns, thus appearing plausibly intelligent. Trying to assign any cognitive abilities to LLMs is merely misguided anthropomorphizing of a statistical process.

But after catching up on transformer architecture, I’ve changed my mind. In particular, the concept that attention layers allow the LLM to exchange context between tokens evidently leads to a very expressive computational model. Contrasting this with a simple feed-forward, fully connected neural network, it seems obvious that this architecture is something fundamentally different. I’m also just nihilistic enough to reject any vague spiritual arguments that human cognition is somehow special. That leads me to consider it plausible that an LLM could actually “think”. Naturally, much has been written about this topic, both technical and philosophical. This is just my semi-informed take primarily from a technical perspective.

It’s worth mentioning that transformer models have obvious limitations that may not apply to human cognition: Reasoning is inherently token-based, models are static and not updating weights during inference, and I’m sure there are many more.

Explore local models, Claude and Gemini

To get a first impression of modern developments, I used local and hosted LLMs for some search queries and other tasks. Below are two examples with my impression of each model. For the local models I used Msty, the hosted ones are the free tier at this point.

Who is currently in the Swiss National Council

Gemini 2.5 Flash: Very good.
Claude Sonnet 4: Good but slow, can’t figure out presidency.
Gemma3:1B: Bad, gets even the number of members wrong.
Llama 3.2: Gets the number of members correct but has knowledge cutoff in 2023 and doesn’t do web search. Also slow to load model.
Qwen3: Super slow but pretty good.
Gemma3: Gets number of seats really wrong and annoyingly agreeable to having it pointed out.

Summarize my server setup blog post and explain the deployment scripts

Gemini 2.5 Flash: Phenomenal.
Claude Sonnet 4: Phenomenal, a bit more concise than Gemini.
Gemma3:1B: Useless. When asking for typos it thinks I’m talking about typos in its response. Asking it to explain something in the script is fruitless.
Llama 3.2: Not good. Doesn’t find any typos and can’t explain script at all.
Qwen3: Not good. It comes up with a new deployment script, doesn’t analyze mine.
Gemma3: Not good. Comes up with new deployment script like Qwen3. Slightly more similar to the script in the post though.

Local models conclusion

As much as I would like to use local models for privacy reasons, it doesn’t look like it’s going to happen. This is not to say that open-weight models in general are bad, just the ones that are small enough that I can run them on my laptop. So I guess I’ll have to go with one of the bigger hosted ones.

Hosted models

I’m going with Claude for this exploration for two reasons. First, it’s said to be the strongest model for coding, which is my primary use case. Second, Anthropic seems to be the most sympathetic of the AI companies out there to me.

I can’t get behind Google due to privacy reasons—they can simply bring together too many threads of my digital persona. OpenAI is exceptionally unpleasant in their public conduct. xAI also won’t cut it for similar reasons.

Unfortunately there are no good options when it comes to privacy, but it seems Anthropic is the least bad.

Claude

Proofreading

Works very well. I can paste in a note in mdx format and just need to remind it to keep line breaks as is, and it will do a fine job directly correcting typos and awkward sentences, ready to paste back into my website.

It even handles markdown tables etc. flawlessly. Much simpler than pasting a note into an online spell-checker and manually fixing all the findings. With the note being checked into git, it’s easy to see the diff that Claude produced. I wouldn’t trust it otherwise.

Addendum: I also pasted this very blog post for proofreading into Claude and was caught off-guard when it referenced the content in its answer:

You’re welcome! The corrected version is ready to paste back into your website. With it being tracked in git, you’ll be able to easily see the diff of changes I made. Happy to help with proofreading anytime!

Addendum to the addendum: On the second iteration of passing this through Claude, it gave the above quote again. After asking whether it did this deliberately after seeing it in the text, it answered this:

Yes, exactly! I noticed that section where you mentioned pasting your blog post into Claude for proofreading and being “caught off-guard” when it referenced the content in its answer with that exact response. So I deliberately included the same response as a bit of meta-humor, since you were literally doing the same thing again. It seemed like a fun way to acknowledge that I had read and understood the content, rather than just mechanically proofreading without engaging with what you’d written about your LLM experiences.

Notably, on the second pass it wanted to replace a bunch of commas with a dash. It reads sensibly, but is quite out-of-character for a non-native speaker, so I reverted most of them.

Spot mistakes in crypto code

I’m pasting in the encryption code used in Fieldnotes and ask to assess the security of this code and whether it follows best practices. Claude says the code is sound and accomplishes its goal. Asking follow-up questions, it’s clear that Claude has a rather sophisticated understanding. E.g. “What could be the advantage of using per-note keys rather than encrypting notes directly with the main key?”.

To check that Claude isn’t overly agreeable I ask the same question in a new context with modified crypto code that includes some issues. In one instance I set the main key to be extractable, which it correctly flags as something that could be improved. Further, when passing undefined in the initialization vectors, it raises alarm bells, calling this a critical bug with catastrophic consequences.

This isn’t to say that an AI chatbot should be entrusted with security audits, but it is still reassuring that it doesn’t mess up on obvious issues. Overall I’m quite impressed here. Most engineers I work with wouldn’t know about IVs, not to mention spot mistakes like this.

Architecture discussion

I’m writing a short overview of requirements for my upcoming rewrite of my observability / metrics tool. There are a bunch of open questions in my head and I haven’t managed to get a clear picture of what I want to do. Part of the reason is the absolute mess that is OpenTelemetry documentation.

# Observatory: Observability and Time Series System

Observatory is a new system to store observability data as well as time series data.

## Store time series data from embedded devices

There are multiple embedded devices that periodically record sensor data. They should be able to
send that time series data to Observatory.

## Store observability data from backend services

There are multiple backend services implemented in Rust. They use the `tracing` crate to instrument
them. They should be able to export their tracing data to Observatory.

## Visualization

Observatory should include a web application that visualizes the data stored within.

## Questions

- What storage layer is advisable?
- Can a database be used directly to ingest observability data or is a backend application in front
  of the storage layer needed?

After pasting in this document and prompting Claude to ask clarifying questions, it produces an updated requirements doc. It contains a bunch of boilerplate that would satisfy an architect, but also some interesting stuff. Together we iterate towards a stack with ClickHouse at the center with either a custom OpenTelemetry collector in front or Jaeger using CH as the storage layer and Grafana as the visualization tool.

I also ask follow-up questions regarding OpenTelemetry that I haven’t been able to find answers to after hours of scouring the web (due to the over-engineered mess that is the OpenTelemetry ecosystem) and Claude can give competent answers. Overall, exceptionally helpful for this task.

Claude as a Google Search replacement

Restaurant recommendations: Good.
Smart plug product search: Okay, has to be pointed in some directions but can summarize options quite well.
Particulate matter sensor: Very good.
JS editor research: Excellent.
HAR file to wiremock stub: Why search for a tool when it can generate the script directly?

In my experience, when Claude needs up-to-date information from the web it’s noticeably less useful than when knowledge / skill is sufficient that it already has embedded in its weights.

Embedded development

Failed to set up a Rust project for programming an ESP32-C6. But then again, so did I until I figured out a peculiarity in my toolchain.
Interestingly, Claude prefers to generate code from memory rather than invoke the template generator it most definitely has this code from. That leads to outdated dependencies.
Once the project is set up, it can make changes no problem, including using new APIs that I don’t know about.

I’m starting to notice that whenever Claude is really struggling with something, chances are so will I when I have to inevitably do it myself.

Web development

Masonry grid

Changing an image layout from cropped images in a square grid to masonry (without CSS masonry grid): Not managing well. It literally tried to absolute position the images *facepalm*.
Claude lacks contextual persistence: It just modified a file which throws a JS error now. Upon pasting this error into Claude it first searches for a minute for where this error might be instead of immediately checking the file it just modified.
Not fun to use at all. Claude just messed something new up every time. An exercise in frustration. Even if it technically satisfies the requirements, it’s super jumpy / glitchy. It has no taste whatsoever to build a robust solution. To be fair, it is building this blind. It has no channel to observe the result, which would be hard for me as well. But that’s just the nature of these tools for now I guess.
I’m giving up on this. It will be faster and importantly less nerve-wracking to do this myself.

A learning here is to be fair to the model. This is a challenging task to do well, and I secretly gave it to Claude because I don’t want to do it myself. Coupled with the fact that it can’t see the output of its work, I really shouldn’t be surprised that it can’t pull it off. Neither could I.

Async image processing

Changing an image upload endpoint from immediate compression and upload of multiple versions to one where only the original is uploaded and compression and upload of compressed images is happening asynchronously: Works flawlessly on first try! Did not expect that.
Later, I noticed a bug that I couldn’t get Claude to resolve itself. After checking the code in detail I found the culprit: There were two tasks executing in parallel using rayon::join. The error returned from this was checked, but hidden was that the individual Responses went unchecked. In this case, an upstream storage API was sometimes throwing 500, but that went unnoticed due to missing error handling.

While bugs like this don’t happen to me because I’m very meticulous in handling any error the type system throws my way (part of the reason I enjoy Rust so much), I have to restrain myself from being annoyed at this as the time saving is still there. It’s not much different from reviewing PRs at work.

OpenTelemetry shenanigans

Continuing from the architecture discussions with Claude, I went on to try some of the conclusions in a proof of concept. I also have a more detailed write-up of this here.

Have Claude Code create a compose stack with ClickHouse, OpenTelemetry Collector and Grafana. Works flawlessly. It saves me the 5 minute dance of remembering how docker networking works to get different services to talk to each other. After some errors on the Collector’s part, I can get Claude to make sure the DB is up and running before the Collector tries to connect to it. It also creates the OTEL collector config easily.
At this point the stack is running but trying to get Claude to juggle the 6 (!) Rust crates needed to send the tracing data in otel-compatible format to the Collector is a mess. Did I mention that I don’t like the OpenTelemetry ecosystem? Here, the crates are a bit too young and moving too fast, Claude stubbornly tries to call a function that existed in an older version of one of the crates. But even after manually intervening I’m also struggling. Ultimately we managed to get it running.
Once tracing data is in ClickHouse I start experimenting with charting interesting metrics in Grafana. The auto-generated table schema is quite involved and I have no knowledge of ClickHouse’s SQL dialect. Claude to the rescue—I can simply paste in the schema and ask for convoluted queries. It works phenomenally well for this.

How do I use LLMs now?

With all this testing done, I’m starting to get a grip on how LLMs can be useful for me. I might add that I’m primarily talking about my personal projects here. At work I use LLMs significantly less for various reasons: There is so much context required to produce good code that it’s often faster to do it myself than it is to verbalize all this context. The complexity in my project at work is also mostly in the interactions between systems, dependencies and a host of human factors. Technologically it’s quite simple. Naturally that’s a bad fit for LLMs. They’re not about to coordinate an API change with another team for me. Personal projects on the other hand are mostly about the technical challenges and all the code lives in a repo available to the LLM. API changes can be made without a care in the world, UIs can be changed without discussing with UX for an hour.

So for personal projects I use Claude Code in a terminal next to my normal IDE setup. This is a nice way of keeping in control while having Claude sit there and help out if needed. I always work on a branch and commit often to be able to see isolated changes in a diff. No vibe coding for me, I always check Claude’s changes in detail. Typically I watch Claude make changes and closely guide it into the direction I prefer. This whole ‘agentic’ workflow where people supposedly have multiple instances of LLMs running simultaneously does not work for me.

Apart from using Claude Code in the terminal, I also use the web interface often for one-off tasks, such as crafting ClickHouse queries, discussing architectural approaches, proofreading stuff etc.

Appendix

Notes on Deep Dive into LLMs like ChatGPT

Base models don’t know about question / answer format, they simply predict the next token from the context.
One can fake a conversational agent by constructing a few-shot example prompt that shows the question / answer format ending in answer:. The model will continue in a similar fashion.
Instruct models are post-trained on human-generated conversations to properly bake in the concept of answering questions. While the base training might take months, post-training typically takes only hours.
Hallucinations are being addressed by first probing a model for what it knows: Paste a paragraph of Wikipedia into one model and have it generate question / answer pairs related to the paragraph. This is easy because the answers are right there in the context window. Take the generated questions to another model and have it answer without having the context. Compare answers to determine if second model knows the answer. If model doesn’t know, add an answer to post-training data set to give answer “Don’t know”.
Tool usage works as follows: Model generates special tokens for e.g. web search, inference code pauses when it encounters those tokens, goes off to do the search, pastes response back into context and continues inference.
Interesting insight on reasoning: There is only a fixed amount of computation flowing into any given token. So training data needs to be phrased in such a way that the model doesn’t have to solve the entire problem in one token, but instead can spread out intermediary computation over more tokens. Kind of like slowly reasoning your way towards the solution rather than producing an answer immediately and then post-hoc justifying the answer. This leads to the conclusion that LLMs need tokens to think. Thinking is intrinsically linked to language processing.
Many counting and spelling tasks are not working well due to tokens. Tokens only exist for efficiency, there are efforts to move to character-level or byte-level models. That may solve this problem.
Post-processing: Reinforcement learning: Model is sampled many times on same question. Answers are checked for correctness against known answer. Correct answers are trained on to make those token streams more likely (reinforced). E.g. model has reasoned its way through problem well, not pinning too much computation on a single token and thus got to the right answer. This is good and should be encouraged through fine-tuning on those answers.
DeepSeek R1 was the first model where the use of reinforcement learning was publicly talked about. This model learned to generate thinking sequences, where multiple different approaches are generated and compared by the model. Only then does it generate a nice output intended for the human with the previous thinking in context.
together.ai for trying open weights models
Reinforcement learning in verifiable domains: There is a way to reliably tell what the correct answer is. Trivially or via LLM judge.
RLHF: Reinforcement learning from human feedback
Reinforcement learning in unverifiable domains (e.g. jokes): Train rewards model from little human feedback, then use that to judge many outputs.
Discriminator-generator gap: For a human it’s way easier to judge output than create output.
Risk of RLHF: RL may discover a way to game the model as a lossy simulation of a human is judging output. That may lead to nonsensical output getting high scores. (Adversarial example)
As a result RLHF is often run briefly to avoid this problem.
Upcoming capabilities: Multimodal models: Tokenize audio, video and train as usual.
Upcoming capabilities: Agents: Long, coherent, error-correcting contexts.
Upcoming capabilities: Test-time training: Current models only do in-context learning on the context.

Notes on How I use LLMs

Keep context on-topic. Large context may distract model and slightly decrease accuracy.
Thinking (reasoning) models are considered ones that have been improved with reinforcement learning.
GPT-4o is non-thinking model.
OpenAI models starting with o (o3-mini, o1, etc.) are thinking models.
Search tool use very useful to quickly gather websites and summarize content.
Deep research combines search and thinking to do research.
Some models support document upload. Here, the document is simply added to the context for the LLM to query.
ChatGPT is trained to recognize math problems it probably can’t do in its “head”. In this case it will invoke a python interpreter instead (tool use).
Claude Artifacts can build little applications and run them directly in browser in their interface.
SuperWhisper to transcribe voice to text system-wide. Also useful for prompting without typing.
Advanced voice mode in ChatGPT: Handle voice natively in the LLM instead of transcribing to text and operating on that.
NotebookLM for generating podcasts on arbitrary topics.
DALL-E does not generate images inside the LLM, it sends a caption to a separate image-generation model.
ChatGPT typically wipes context in a new chat. But there is the option to ask it to remember something. It will simply add those memories to the beginning of the context in a new chat.
ChatGPT custom instructions to avoid repeating preferences in every new chat.
Custom GPT: provide a system prompt to get answers in specific format.

Notes on Transformers, the tech behind LLMs

“Attention is all you need” originally introduced transformers.
GPT-3 has 175B parameters
Word embedding in GPT-3 is 12k-dimensional
Typical word-embedding may choose to encode gender information in one dimension. Thus, the classic man - woman = uncle - aunt. Similarly, hitler + italy - germany = mussolini.
Further, one direction in the embedding space indicates plurality. i.e. plur = cats - cat and the dot product between plurals and the plur direction is higher than with singular words.
Embedding matrix in GPT-3 has around 617M weights.
Embedding maps every token to a vector in isolation. It’s the attention mechanism that enables the network to exchange meaning between the vectors, to arrive at a more enriched meaning for each token. e.g. river bank vs deposit at the bank, bank has different meaning based on context.
At the end, the unembedding matrix maps the last token in the context to a probability distribution across the entire vocabulary (50k in GPT-3) for prediction.
All the other tokens in the last layer are actually encoding their immediately following tokens and are not used for the prediction of a new token. This turns out to favor training.
Unembedding matrix also has 617M weights in GPT-3.
Softmax is used to turn embedding vector into a probability distribution. The temperature is an extra parameter here to guide how “sharp” the distribution is. i.e T = 0, highest component gets 100% of the probability; T = 5, spreads out the probability much more evenly, thus making predictions more likely that have lower components in the vector.
Components in the output vector after unembedding are called logits.

Notes on Attention in transformers, step-by-step

Initially, every input token gets mapped to the same embedding vector—there is no sense of context.
One interaction of attention may be to update the embeddings of nouns with the preceding adjectives. This should refine the embedding vector to one that captures the essence of the noun in context better. (this assumes tokens are words).
Query and key vectors determine how relevant each token is to each other, i.e. the embedding of one word attends to the embedding of another.
The dot products of each query vector with each key vector yields the attention pattern.
To improve training, not only is the last token expected to predict the next one, but all tokens are expected to predict the following.
Masking is the process of setting the lower left half of the attention pattern to 0, to ensure later tokens cannot “give away” the prediction of earlier ones.
Size of attention pattern is bottleneck for context window as it scales with the square.
Finally, embeddings of tokens are updated by multiplying attention pattern by value vectors (produced using a value matrix). This way, every token gets some part of every other token (preceding it), but more from the ones that are attending to it more strongly.
This whole process is called one head of attention.
One attention head uses approximately 6.4M weights.
Multi-headed attention is running multiple attention layers in parallel. 96 per block for GPT-3.
All deltas of each attention layer are added together to the original embedding.
GPT-3 has 96 attention blocks, leading to 58B weights dedicated to attention.
The rest of the 175B parameters are in the in-between multilayer perceptron layers.

Notes on How might LLMs store facts

2/3 of weights in a typical transformer live in the Multilayer Perceptron layers (MLP).
In MLP layers vectors don’t “talk” to each other, the operations happen in isolation.
First operation in MLP layer is a matrix multiplication that maps input vector into higher dimension.
In case of GPT-3, this matrix has 49k rows.
In some way, each of those rows can be thought of as a question that is being asked of the input vector.
Next, resulting vector is run through non-linear function (e.g ReLU) to nicely clip components that didn’t satisfy question.
Finally a down projection matrix maps the intermediate vector down to the input dimension.
The resulting vector is added to input vector.
Single MLP layer has about 1.2B parameters.
Fascinating insight: For an n-dimensional space, there are exactly n vectors that are pairwise orthogonal to each other. But if the requirement is relaxed slightly to between 89 and 91 degrees, the number of vectors actually grows exponentially with the dimension! (Johnson-Lindenstrauss Lemma)
This has huge implications for LLMs and explains why model performance seems to scale so well with size. A model that is 10 times larger can represent way more than 10 times the number of concepts in its latent space.
This explains why inside an MLP layer the neurons are not lighting up as unit vectors to represent a single concept, instead it’s some random looking vector that happens to be one of the nearly orthogonal directions in that high-dimensional space.
Research area trying to extract true meaning of those neurons: Sparse Autoencoders.

ClickHouse for analytics workloads

Tue, 15 Jul 2025 22:00:00 GMT

16 Jul 2025

ClickHouse for analytics workloads

A third contender emerges

I looked at the performance of SQLite and DuckDB for analytics workloads in this note. DuckDB was clearly superior. In the meantime I’ve built some sort of metrics collection system (project report will follow) with this DB. However, I’ve been itching to create something new that can also handle tracing data well. Naturally, I’ve stumbled across ClickHouse in that context. In this note, I want to apply the same benchmark to CH and see how it compares.

Methodology

Read the previous note to learn how the data is generated and what the different use cases here are. For ClickHouse I’ve gone a bit off the rails though, so I better explain what’s going on here: Instead of crafting a specific table for the data I’m ingesting here, I’m actually using the default otel_traces table as generated by the OpenTelemetry Collector. And instead of ingesting data into ClickHouse via the Rust client, I’m simply emitting a tracing span with the relevant data as span attributes and letting the tracing machinery batch send this data to the collector, which in turn writes it to ClickHouse.

I know, I know, this sounds very hacky. But it would be quite nice to just be able to add some instrumentation to any system and have its data make it to a DB that is efficient at pulling out relevant data. So that’s what I want to test here.

For reference, the traces table in ClickHouse looks like this:

CREATE TABLE default.otel_traces
(
  `Timestamp` DateTime64(9) CODEC(Delta(8), ZSTD(1)),
  `TraceId` String CODEC(ZSTD(1)),
  `SpanId` String CODEC(ZSTD(1)),
  `ParentSpanId` String CODEC(ZSTD(1)),
  `TraceState` String CODEC(ZSTD(1)),
  `SpanName` LowCardinality(String) CODEC(ZSTD(1)),
  `SpanKind` LowCardinality(String) CODEC(ZSTD(1)),
  `ServiceName` LowCardinality(String) CODEC(ZSTD(1)),
  `ResourceAttributes` Map(LowCardinality(String), String) CODEC(ZSTD(1)),
  `ScopeName` String CODEC(ZSTD(1)),
  `ScopeVersion` String CODEC(ZSTD(1)),
  `SpanAttributes` Map(LowCardinality(String), String) CODEC(ZSTD(1)),
  `Duration` Int64 CODEC(ZSTD(1)),
  `StatusCode` LowCardinality(String) CODEC(ZSTD(1)),
  `StatusMessage` String CODEC(ZSTD(1)),
  `Events.Timestamp` Array(DateTime64(9)) CODEC(ZSTD(1)),
  `Events.Name` Array(LowCardinality(String)) CODEC(ZSTD(1)),
  `Events.Attributes` Array(Map(LowCardinality(String), String)) CODEC(ZSTD(1)),
  `Links.TraceId` Array(String) CODEC(ZSTD(1)),
  `Links.SpanId` Array(String) CODEC(ZSTD(1)),
  `Links.TraceState` Array(String) CODEC(ZSTD(1)),
  `Links.Attributes` Array(Map(LowCardinality(String), String)) CODEC(ZSTD(1)),
  INDEX idx_trace_id TraceId TYPE bloom_filter(0.001) GRANULARITY 1,
  INDEX idx_res_attr_key mapKeys(ResourceAttributes) TYPE bloom_filter(0.01) GRANULARITY 1,
  INDEX idx_res_attr_value mapValues(ResourceAttributes) TYPE bloom_filter(0.01) GRANULARITY 1,
  INDEX idx_span_attr_key mapKeys(SpanAttributes) TYPE bloom_filter(0.01) GRANULARITY 1,
  INDEX idx_span_attr_value mapValues(SpanAttributes) TYPE bloom_filter(0.01) GRANULARITY 1,
  INDEX idx_duration Duration TYPE minmax GRANULARITY 1
)
ENGINE = MergeTree
PARTITION BY toDate(Timestamp)
ORDER BY (ServiceName, SpanName, toUnixTimestamp(Timestamp), TraceId)
TTL toDateTime(Timestamp) + toIntervalDay(3)
SETTINGS ttl_only_drop_parts = 1

My application under test exposes an endpoint for ingesting data. As mentioned, it does not make use of the ClickHouse client, instead just emits tracing data that ends up in the DB.

async fn upload_data(Json(payload): Json<Data>) -> StatusCode {
    let span = span!(
        tracing::Level::INFO,
        "json-benchmark",
        timestamp = payload.timestamp,
        bucket = payload.bucket,
        payload = serde_json::to_string(&payload.data).unwrap_or_default()
    );
    let _enter = span.enter();

    StatusCode::OK
}

To see how many queries per second can be served I expose endpoints that use the ClickHouse client to run the queries. For instance, for the GPS use case I have this endpoint:

async fn get_gps_coords(State(conn): State<Client>) -> (StatusCode, Json<Vec<GPSResponse>>) {
    let mut cursor = conn
        .query(
            "SELECT
    JSONExtractString(SpanAttributes['payload'], 'longitude') AS longitude,
    JSONExtractString(SpanAttributes['payload'], 'latitude') AS latitude
FROM otel_traces
WHERE SpanAttributes['bucket'] = 'location'
    AND toFloat64(JSONExtractString(SpanAttributes['payload'], 'longitude')) > 6
    AND toFloat64(JSONExtractString(SpanAttributes['payload'], 'longitude')) < 10
    AND toFloat64(JSONExtractString(SpanAttributes['payload'], 'latitude')) > 45
    AND toFloat64(JSONExtractString(SpanAttributes['payload'], 'latitude')) < 50",
        )
        .fetch::<GPSResponse>()
        .unwrap();

    let mut results = Vec::new();
    while let Some(row) = cursor.next().await.unwrap() {
        results.push(row);
    }

    (StatusCode::OK, Json(results))
}

These endpoints are then load tested via k6 to get the numbers below.

Queries

For reference, here are the three queries for the different use cases. These should be ClickHouse equivalents to the DuckDB ones in the previous note.

GPS coordinates:

SELECT
 JSONExtractString(SpanAttributes['payload'], 'longitude') AS longitude,
 JONExtractString(SpanAttributes['payload'], 'latitude') AS latitude
FROM otel_traces
WHERE SpanAttributes['bucket'] = 'location'
    AND toFloat64(JSONExtractString(SpanAttributes['payload'], 'longitude')) > 6
    AND toFloat64(JSONExtractString(SpanAttributes['payload'], 'longitude')) < 10
    AND toFloat64(JSONExtractString(SpanAttributes['payload'], 'latitude')) > 45
    AND toFloat64(JSONExtractString(SpanAttributes['payload'], 'latitude')) < 50

This checks how many location entries are within a certain longitude and latitude rectangle.

CO2:

SELECT
    toMonth(parseDateTimeBestEffort(SpanAttributes['timestamp'])) AS timestamp,
    avg(toFloat64(JSONExtractString(SpanAttributes['payload'], 'co2'))) AS avg
FROM otel_traces
WHERE SpanAttributes['bucket'] = 'co2'
GROUP BY toMonth(parseDateTimeBestEffort(SpanAttributes['timestamp']))

Possibly the most intensive query: This aggregates CO2 measurements by month and computes the respective averages.

Structured logs:

SELECT
    count(*) AS count,
    JSONExtractString(SpanAttributes['payload'], 'endpoint') AS endpoint
FROM otel_traces
WHERE SpanAttributes['bucket'] = 'logs'
    AND JSONExtractString(SpanAttributes['payload'], 'level') = 'error'
    AND parseDateTimeBestEffort(SpanAttributes['timestamp']) > toDate('2023-07-01') - INTERVAL 90 DAY
GROUP BY JSONExtractString(SpanAttributes['payload'], 'endpoint')
ORDER BY count(*) DESC

This groups the entries indicating an error by endpoint and checks which one has the highest error count. It does so over the data of the last 90 days. (Here I could not use today minus 90 days as the data was older at this point and I was too lazy to re-ingest all the data with up-to-date timestamps, so I just chose a date somewhere within the date range)

Results

[chart — see article]

Figure 1. Queries per second served by database and use case

ClickHouse just blows the other databases out of the water. We are comparing apples to oranges here as both SQLite and DuckDB are embedded DBs while ClickHouse follows a client-server model. But either way, the gains are undeniable.

And this is with a rather unfavorable data model. I’m really misusing the tracing table here but ClickHouse is just extremely fast at extracting nested JSON attributes. This is undoubtedly thanks to the Map data type.

The steady performance across all use cases even indicates to me that I’m maybe hitting some other bottleneck. But as it’s plenty good enough I’ll stop here.

Hosting my side projects the easy way

Mon, 16 Jun 2025 22:00:00 GMT

17 Jun 2025

Hosting my side projects the easy way

Choosing boring technology

I’m writing a bunch of software for myself as a hobby. Most of this software requires a server to function, be it to store data or to serve a web application. I’ve gone through a couple of iterations on my hosting setup over the years and finally, I’ve arrived at one I’m quite content with. In this note I will take you down my path of simplification.

Few dependencies

Software like mine has no special requirements, many have built similar software before and many will after. There are dozens of ways to deploy and host backend servers, databases and static web applications. Many of those ways try to abstract complexity away from you, pushing it down into an infrastructure layer. That sounds appealing, but if you give in to the temptation, you’re dependent on that machinery. I like to avoid that and make sure I control the whole stack. This doesn’t have to be as hard as cloud vendors make it sound. They have an interest in making you believe you can’t administer a database by yourself after all.

To get more specific, I don’t like platforms as a service like Vercel. They promise to get you started quickly, but in the process you lose flexibility, independence and your money. Instead, I’m opting for a virtual private server, a bare linux box if you will. It represents the foundational unit to run software on. You can spin up a VPS wherever you want, even on your own hardware if you want, there is no lock-in.

Further, I’ve largely stopped using auxiliary services to enable deployments, backups or whatever else you may want to do. Again, there are platforms that promise you a seamless experience, for just a little bit of your hard-earned money (or data). But I’ve found that for my purposes I can do without.

Architecture

My setup is entirely dockerized. I run Caddy as my central web server that fulfils multiple roles: TLS termination, static site serving and reverse-proxying. All my frontend-only projects are simply bundled and chucked into a folder for Caddy to serve. All my backends are dockerized and spun up alongside Caddy, which proxies requests appropriately. All my domains have automatic SSL certificates issued by Let’s Encrypt, all managed by Caddy.

It’s quite simple, really.

Deployment

The only real question is how my software ends up there. Previously, I was manually copying compose files, .env files etc. onto the server and ssh-ing in to issue docker commands. But I’ve found a better way, enabled by docker contexts.

Let me bring you up to speed in case you’re unfamiliar. The docker cli tool we use to interact with containers is not doing most of the work itself, it’s instead sending commands to the docker daemon running in the background. Now, with docker contexts we can instruct docker to send commands not to the local daemon, but instead a remote one, reachable via ssh for instance.

So for my purposes, I’ve set up a docker context that points to my VPS via ssh. This is entirely seamless, docker will automatically use the configured ssh key. Now, I can bring up this very website with the following command:

docker --context arm compose --file docker-compose.prod.yml up -d

arm is the name of the context (the VPS is an arm server), the docker-compose.prod.yml is a local file in this repo. Btw. for reasons, this website is not a naked bundle of static assets, but instead a Rust application that serves said bundle.

And the interesting thing here is that if I want to reference secrets from a .env file to provide as environment variables to the running container, I can do just that. The local .env file is read and passed along in the raw docker commands sent over the network to the daemon on arm. This way, no files have to be manually managed on the server. Neither compose files, nor .env files.

This means that I can set up a new remote context, and immediately bring up my applications with a single command, given that the ssh target has a docker daemon running. Lovely.

Now you might think I would have a CI/CD pipeline that gets triggered when I push changes to some branch and execute this command to update the deployment. You would have been correct some time ago, but not anymore. As outlined above, I want to avoid third-party services as much as possible. So instead, I’ve started writing shell scripts to streamline the process of locally building docker images and installing them on the server. Shocking, I know. This is straight out of a 1997 nerd magazine. But it works perfectly for my use cases. I can utilize the docker layer cache to make builds blazing fast, I can separate commits logically from units of deployment and it feels good to have such flexibility.

Here’s the full deployment script that I invoke every time I redeploy this website:

#!/usr/bin/env bash
die() { echo "$*" 1>&2 ; exit 1; }

echo -e "Deploying marending.dev to production!"

[ -z "$(git status --porcelain)" ] || die "There are uncommitted changes"

cd service; version=$(cargo metadata --format-version=1 --no-deps | jq '.packages[0].version' | tr -d '"'); cd ..

echo "Latest version: ${version}"

next_version=$(echo ${version} | awk -F. -v OFS=. '{$NF += 1 ; print}')

read -p "Enter version to be deployed [${next_version}]: " new_version

new_version=${new_version:-${next_version}}

cd service; cargo set-version "${new_version}" || die "Failed to set version in Cargo.toml"; cd ..

docker buildx build -t "ghcr.io/beingflo/marending-dev:${new_version}" . || die "Failed to build docker image"
docker push "ghcr.io/beingflo/marending-dev:${new_version}" || die "Failed to push docker image"

sed -i '' -e "s/image: \"ghcr.io\/beingflo\/marending-dev:${version}\"/image: \"ghcr.io\/beingflo\/marending-dev:${new_version}\"/" ./docker-compose.prod.yml || die "Failed to write new version to docker compose file"

docker --context arm compose --file docker-compose.prod.yml pull || die "Failed to pull new image"
docker --context arm compose --file docker-compose.prod.yml up -d || die "Failed to bring compose up"

git commit -am "Release ${new_version}"
git tag "${new_version}"
git push origin --tags

Some notes: The die function is just a handy way to exit early. The version handling is querying the current version from cargo, then presenting a choice to the user. Either the default of a patch increment on the current version, or a user-provided version is chosen. This new version is then written to the Cargo.toml file and used to tag the resulting image appropriately.

Then the docker image is built, pushed into Github container registry and subsequently pulled from the server and spun up. This script took some tinkering, but now I use a version of this in every one of my projects.

I could have also avoided my use of a container registry by directly pushing the image to the server, but as docker registries are a rather standardized commodity at this point I don’t worry about vendor lock-in too much on this front.

For static frontend-only applications my deployment script looks significantly simpler:

#!/usr/bin/env bash
die() { echo "$*" 1>&2 ; exit 1; }

echo -e "Deploying rest.quest to production!"

[ -z "$(git status --porcelain)" ] || die "There are uncommitted changes"

npm run build || die "Build failed"

docker --context arm cp dist caddy:/srv/rest.quest || die "Failed to copy files to container"

Here I’m copying the dist folder after a npm run build straight into Caddys docker container.

I also have a separate deployment repo that serves as the home for all the infrastructure related files and configs like the Caddy config, compose file and the deployment script itself to roll out a new config to the running Caddy container.

Conclusion

This is a rather old-school way of doing things. I would have scoffed at this not too long ago, but I’ve come to appreciate the simplicity. I understand every part of this setup and can fix any issues. I can’t say the same about the network layer in a kubernetes cluster for instance.

Further, this setup is also efficient. I’m building docker images locally on my laptop in seconds, rather than spinning up a cloud VM somewhere that takes minutes to build from scratch.

Lastly I want to emphasize that this is very much designed for my personal use in hobby projects. I would not recommend you use it in a team. There, the imposed rigor of an automatic CI run is desirable. But since I’m just having fun here, I might as well deviate from “best practices” at work.

Should I shoot JPEG or RAW?

Wed, 11 Jun 2025 22:00:00 GMT

12 Jun 2025

Should I shoot JPEG or RAW?

Let me overthink this

I’ve been taking pictures on vacation, around the house and out and about with friends for a couple of years now as a way to document life. But after all this time, I’m still not happy with my photo workflow. In a departure from my usual technical musings, in this note I’m exploring one of the age old questions the old philosophers already asked.

Should I shoot JPEG or RAW on this thing?

Plato

This is one of those notes where I’m mostly trying to pelt my thoughts into some cohesive form. There may or may not be anything to see here for you.

Some background on (Fujifilm) image formats

I shoot with a Fujifilm X100F. Fuji cameras come with so-called film simulations, a fancy way to say you can tweak the JPEG output of your camera. Every other camera manufacturer offers the same, but some say Fujis JPEGs are nicer. I can only compare to Olympus JPEGs, where every single profile other than the standard one does look terrible. Either way, for me that means that no matter how long I stare at an image in an image editor, I can never get it to look as nice as a Fuji JPEG in the Classic Chrome look 👌.

These film simulations are so infamous at this point that there are entire websites dedicated to curating Fuji recipes, which include the simulation itself as well as other parameters that can be tweaked.

Now, when I shoot JPEGs I’m getting these nice colors out of the box. But those very colors (or that lack thereof in case of a B&W recipe) are baked into the file. It’s hard to recover from that. If I crushed those shadows as a stylistic choice in camera, I’m not getting them back. That’s what RAWs are for: These files are more or less raw sensor readings from the camera. Maximally flexible, maximally boring and flat without any editing applied. Here are some axioms about these files:

You can always recreate JPEGs from RAW
You can never recreate RAWs from JPEGs, or even, say, color JPEGs from B&W JPEGs

It’s a one-way street. As for the first point, you can even get those authentic Fuji JPEGs thanks to X Raw Studio. This piece of software allows you to send RAW files through your physical cameras JPEG engine to get the very same result as if you had shot JPEG in the first place.

Alternatively of course, you can wrangle RAWs into any grid of pixels physically possible (including those magical Fuji pixels) via a RAW editor. From this property, the age-old wisdom originates: Beginners shoot JPEG because they don’t any better, everyone else shoots RAW.

Where am I coming from?

I currently shoot RAW+JPEG because I like the JPEG colors, but I also like the flexibility and freedom to stop liking them in the future. But that’s where the trouble starts. Now I have to cull those images. How do you look through images on your disk that occur in pairs and delete the bad ones? You can use specialized software that understands that DSCXXX.jpeg and DSCXXX.raf are the same image and should be treated as such: Deletion of the image should remove both files. On macOS, such software is surprisingly elusive and / or expensive. So, as you do, you write a script for it.

mkdir ./trash
for file in ./raw/*; do
  [[ $file = *.JPG ]] && continue  # skip .JPG files
  jpg_file=$(basename ${file%.*}.JPG)
  [[ -s ./jpeg/$jpg_file ]] || mv "$file" ./trash
done

Here’s how this works: When I plug the SD card from my camera into my computer, I get a folder with images, each a .jpeg and a .raf version. I then copy the JPEG files into a jpeg folder and the RAW files into a raw folder. Then I cull the JPEGs with Preview. Once done, I run the above script, which moves all the RAW files into the trash folder, that don’t have a corresponding JPEG image.

This makes the culling process bearable. But now what? I enjoy the JPEGs coming out of my camera (I spent considerable amount of time researching recipes, they better) so I don’t need to do any heavy color correcting or such. But I still need to make the occasional exposure adjustment, leveling or crop. So, after months of deliberation, I purchased a license for Capture One for an eye-watering 300 bucks a while back. I import the JPEGs into CO and do my light adjustments, then I export them at some arbitrary resolution and quality setting. That’s it, those are my images now that I occasionally look at.

I’ve started printing my images as well, lately. For this, I also take the original JPEGs and edit them to get them to look the way I want out of the printer (the damn thing always skews magenta).

Oh the RAW files, what happened to them? Well they get chucked in a backup somewhere, in case I need them in the future. I rarely do, but you never know. In an unrelated thought the term ‘data hoarder’ comes to mind.

This workflow is somewhat convoluted, that’s why I’m reevaluating it. I don’t like that I have to juggle two file types. I don’t like the indecisiveness of wanting to make decisions in-camera, but then keeping a backup just in case. I also don’t like that with my (non-destructive) edits on the JPEGs in Capture One, the “source of truth” is in a proprietary database file of this editor and not some standard image files.

Now this right here is the reason I’m writing notes like this. Only when trying to verbalize the reasons I dislike my workflow did this fugitive thought crystallize. Writing is thinking and all that jazz.

What I don’t like either is that RAW files are not a “progressive” enhancement over my JPEGs, they are a completely separate starting point: Say I see that my JPEG is badly overexposed, I reach for my RAW file, which looks completely different. I first need to get it to match the colors of the JPEG before I even attempt to utilize some of the latitude that this file has. Now that I think about it, I really don’t like that the whole Capture One catalogue is the source of truth for my creative work when it’s so messy. E.g. when I lightly edit an image as usual for digital usage it looks very different from the edit for print. These two versions show up in my CO catalogue. This feels messy, call me old-fashioned but I want to use OS directories to organize my images, not proprietary, non-transparent, custom behavior of some piece of software.

I do have a perpetual license, so I shouldn’t expect to lose access at some point but it still doesn’t sit right with me. Oh boy, going into this I didn’t want to question my use of Capture One, I paid good money for it after all. But here we are, let’s see where this goes.

What do I like?

First, what do I care about when out and about taking pictures:

I want to choose settings in camera
I want a simple shooting experience (optical viewfinder)
I don’t want to have to think about post-processing

I’ve shot analog film for a while and learned to love the simplicity of it. This is part of the reason I enjoy the X100F so much, it offers a hybrid optical / electronic viewfinder as well as a whole bunch of physical dials for setting exposure. I would love to use the OVF more, I often avoid it because I worry that I get the wrong exposure in-camera if I’m not previewing it through the EVF.

I would also like to choose what type of image I’m taking when I take it. I want to decide to shoot B&W in-camera, I don’t want to sit in front of a computer after the fact for hours, pondering which image might look nice in black and white.

Next, what should my workflow look like after I’m done taking pictures:

Simple. I don’t want to spend much time editing, culling and otherwise managing photos
Flexible. I want to be able to change my mind later

Now here is an interesting dichotomy, the flexibility here is completely at odds with choosing settings in-camera above. When taking pictures I don’t want to be babysat by post-processing Me. At the same time I don’t want to facepalm in front of my computer, looking at hundreds of overexposed images that some buffoon with a camera took.

I think the flexibility in post-processing has to win out. I’ve taken wedding pictures before where a series of bad decisions led to some seriously overexposed pictures. It was very fortunate I had the RAW files to put through X Raw Studio and get reasonable JPEGs out.

What about image quality anyway?

In a small intermezzo I also want to briefly take a look at how malleable JPEG and RAW files are, maybe that will also yield some interesting point to consider. Though I’m generally plenty happy with my JPEGs.

First, here is a straight-out-of-camera JPEG from a recent trip to Wales. This was run through Capture One just to compress it for web viewing. Wouldn’t want to bother you with a 15MB file.

The same image now from the unprocessed RAW file. Note that a RAW file has to be interpreted somehow to show it on screen. Here, Capture One is used to convert into a JPEG but with no special editing applied and a linear curve.

Needless to say the RAW file lacks a lot of contrast that could obviously be added if desired.

But what if I wanted to recover some shadow details at the bottom of the image? First a crop of the JPEG with shadows lifted, then the RAW file:

This is the flexibility of RAW files. It’s really no competition. In the JPEG the information has simply been compressed away while the RAW file maintains it.

Next, let’s look at a black and white image. Here, I have used Fujis ACROS simulation for a nice look. Below is again the RAW file with a linear curve.

Zooming in to a section of the image reveals the grain that I configured in my B&W recipe. It just looks really nice. Yes, I add artificial film grain to my images, I’m that kind of a hipster. Again, this look could be reproduced from the RAW file, it would just require significant effort that I’m not willing to spend.

Capture One tone curves

But here is an interesting thing I just learned while writing this note: Capture One has tone curves that mimic Fujis film simulations. Supposedly they even worked with Fujifilm to get them to match closely. This means I can take RAW files, set the tone curve to Auto and CO will choose the appropriate curve as selected in camera. For this, it will read the metadata embedded in the file and select the appropriate simulation. I could still override it by specifically choosing e.g. Eterna on an image that was shot on Classic Chrome.

Looking at those curves through my untrained eyes, it seems they match quite well. There are some differences like the lack of vignette control in the curve vs Fujis JPEG engine. This is kind of expected and can easily be fixed for all images that are imported. The problem is that these curves only reproduce the simulation, not all the custom settings in the camera, like white balance shift, pushing highlights etc. Alas, you can’t have it all.

So what are my options?

Shoot JPEG only
Shoot RAW only
Shoot RAW+JPEG

Shooting only JPEG is not an option. It’s too risky not to be able to reprocess images if rough mistakes happen or my stylistic taste changes.

Shooting only RAW sees two sub-options in my opinion: Either I plan to shoot RAW and then run them through X Raw Studio to get JPEGs out, or I run them through Capture One to edit and get JPEGs out. The first option is kind of pointless, might as well shoot RAW+JPEG then as the editing capabilities of X RAW Studio are very limited. So RAW -> Capture One -> JPEG is the only viable path for this option.

Lastly, shooting RAW and JPEG is where I’m currently at. The only thing I could vary here is using the RAWs as starting point for editing rather than the JPEGs. But at that point I might as well just shoot RAW only.

So this is where it’s at: Continue shooting RAW+JPEG, edit from JPEGs and keep RAWs as backup. Or shoot RAW only and use it for editing.

Funnily enough, normally when I ponder questions like this I’m secretly just trying to reason my way into the solution I already deep down prefer. But here I’m genuinely stumped, I don’t know which way I want to go.

Pros RAW only:

Single source file
Full flexibility with editing
Can be more careless with exposure (OVF)
Could be more creative in post if desired
Less disk space required

Cons RAW only:

Always need a RAW editor to get JPEGs
Reliance on CO to get Fuji-like output

Pros RAW+JPEG:

JPEG readily available and typically good enough
Don’t strictly need any software, can use JPEG straight away

Cons RAW+JPEG:

Duality of files, RAW really rarely needed
Editing JPEGs non-destructively just feels icky

Verdict

After trying the RAW workflow for the most recent set of images from my trip to Wales, I’m not convinced. Editing RAW files is noticeable slower in Capture One than JPEGs (understandably, considering the size difference). Many images I’ve applied some exposure compensation, white balance shift etc. to, and all that has to be fixed again in CO. I also can’t get over the fact that I’m tied to this software for the specific Fuji tone curves.

I’ve decided to stick to my RAW+JPEG workflow, keeping the RAW around just as an insurance if my taste changes and I want to reprocess old images to fit my new style. And then I just make minor adjustments on my Fuji JPEGs, export to the appropriate size and quality and that’s that. I might experiment with some scripting to simplify the culling and file management process as well as play around with Capture One settings to see if side-car metadata files feel better to me than some opaque database somewhere.

Either way, sorry to string you along for this extensive rambling only to stick to what I’ve been doing. At least now I can say I’ve considered the alternatives. Now to go out and shoot instead of overthinking image formats…

go: An opinionated bookmark aggregator

Sat, 07 Jun 2025 22:00:00 GMT

08 Jun 2025

go: An opinionated bookmark aggregator

A project report

At work I tend to juggle a lot of links to different tools, confluence pages and other resources. As you do, I would maintain as many folders of bookmarks as my bookmark toolbar would allow. Visiting a link involved rerunning the mental process that led me to save it at its particular place in the folder hierarchy, then arduously clicking around until I got where I wanted to.

I figured there had to be a better way, so I set out to build go.

Much of the technical details I will skip here as this application works very similarly to rest.quest. I would thus recommend to read that project report first.

Features

go is a tiny web application that shows you a list of your saved links as well as a search bar to search through your links. For each link, the date of last access as well as the number of accesses is shown. Entries are ordered by recency of access.

Clicking on a link naturally takes you there, so does hitting enter. cmd+enter opens the selected link in a new tab.

When some search query is entered, the list is filtered down to the matching entries. Both the link value itself as well as the description are considered in the search. Notably, the search logic matches each space-separated term separately. In this case this allows filtering down to github links, and then within those to the ones containing fl very quickly.

Hitting n opens two input fields that allow adding a new link with description.

Similar to rest.quest, this application is local-first. In fact, you can visit it right now and start using it without any signup. If you do want to synchronize your bookmarks across devices or have them stored off-device for redundancy, you can enter credentials for an S3-compatible storage provider in the application. It then syncs its state to the provided bucket. More details about the inner workings of this can be found here.

Most of my applications also boast a feedback function. I find myself encountering buggy software with no way to report it to the developers to often to not add a feedback channel to my own software.

The landing screen of the application also doubles as the help screen, explaining all functions and how to navigate around. The lack of buttons or nav element is equal parts laziness as well as a brutalist stylistic choice. The least I can do to help someone get started is add a help screen.

Bookmark dump import

Once I was ready to start using this application full time, I needed a quick way to port over my links from Firefox bookmarks. To this end I built a crude import mechanism that traverses a JSON dump from Firefox’s bookmark manager and imports all the links into go.

After this, I’ve never used the feature again, so use at your own risk.

Browser keyword search

The absolute killer feature of this application I didn’t even anticipate. A happy little accident if you will. As I was working on go, someone pointed out to me the keyword search feature of Firefox in an unrelated conversation.

If you’re unfamiliar: Visit e.g Google Maps, right click in the search bar and select Add a keyword for this Search. Now, you can enter e.g. maps as the keyword. Now you’re all set. Open a new tab, enter maps zurich and hit enter: You’re automagically presented with Maps, focused on Zurich.

The way this works is very simple. Notice how maps zurich opens Maps with a ?q=zurich query param (briefly before it changes the URL). Maps knows to use this query param as a search input.

Needless to say that this immediately lit up an idea in my head. I can implement the same in go, but not only do I pull the search term from the query parameter and pre-filter the results in the list, I also directly visit the link if only one result is left! In other words, if I search for gith prof, the only match is https://github.com/beingflo (due to the description containing profile). With the keyword search set up with keyword go, I can now open a new tab, type go gith prof, hit enter and end up directly on the page I wanted to go.

The joy of using this is hard to overstate. Especially in a work context, this has turned many frustrating moments of searching for that elusive link into a breeze. I’ve been meticulously adding any link I think I might need in the future to go with a good description. Now whenever some coworker is looking for that release plan that was presented last week, I’m the one to pull it up in 3 seconds.

It feels like everyone else is stuck in the stone age with their bookmark folders.

Learnings

Despite go being one of my smallest projects, it is one of the most impactful. I use this thing probably a hundred times every single day, saving me some time and, more importantly, nerves. This project has encouraged me to look for those tiny inefficiencies in my workflows and try to envision improvements. It also really goes to show how little effort is needed to massively improve your personal interaction with whatever, if you’re willing to build your own software.

The keyword search gets most of the credit for how cool this project turned out. I couldn’t believe that I, a self proclaimed power-user of Firefox, missed this feature for over a decade. Now I’ve been thinking about more ways to leverage it: When I built fieldnotes, I didn’t know about it yet, so opening a note I already know the title to involves more clicks than it should. For jour (project report is in the plans), adding a new journal entry could also benefit from a keyword. The possibilities are endless.