Disk rotation with LVM: almost no downtime

Posted on by | 2 Replies

Every so often, data on the home server gets dangerously close to how much storage is available on its disks. That’s when the disk rotation dance starts. First find the disk with the lowest price per GB ― ugh, is it just me or did those prices go up… I wonder why…

Then, move the data around. The idea is to remove the oldest (and generally smallest) disk, and to move data from it to the next one in the conga line, so every mount point has more space than before. I leave some wiggle room to leave the most important data on the most recent disks, though.

In the past, I used to rsync(1) data across mount points. This required multiple steps and various periods during which no new data could be written. In the last few years, I got into the habit of using LVM volumes for all those disks. On the last dance, I realised that I could use the low-level data replication mechanisms of LVM to copy the data with almost no downtime.

tl;dr:

  • extend the Volume Group to the new disk with vgextend(8),
  • move the data across Physical Volumes with pvmove(8),
  • remove the old disk from the group with vgreduce(8),
  • and some sundries to rename the Volume Groups to match the new state.
Continue reading →

Wicking garden beds for the dry season

Posted on by | 2 Replies
A row of tall squarish garden beds. They each have a cupola with insect netting on top. Some thin pipes are visible at the back.

We have a good-sized garden, and as any (deluded) home farmer, we decided we’d grow food. We had the occasional tomato and lettuce out of it, but low-to the ground garden beds proved to be a lot of effort for a rather disappointing yield. A large part of the effort was in the perpetual need to weed out anything whose seed landed into the bed, and, adding insult to injury, having to bend down low to the ground for the privilege.

Another problem was with the watering of those precious crops during the dryer months. I already have a ram pump to lift water from the nearby creek to the top of the garden, but distributing the water was left as an exercise to the reader. Spray watering uses a lot of undirected water, and dries out to evaporation quickly. Drip watering only keeps small areas moist. What do?

In a quest for a better garden bed, we happened upon the concept of wicking beds. They are higher-from-the ground garden beds, with lower substrate able to hold a lot of water. On top of this, normal gardening soil allows to grow the prized veggies. The name comes from the mechanism that pulls the water from the lower level into the top soil, keeping the plants nice and hydrated.

It was time to build some. Connecting the overflow from the ram pump was plenty enough to keep them well watered (even too much in winter). The higher clearance from the ground also led to a massively diminished weed problem, and not having to bend down to remove the few stragglers. To top it off, I added little cupolas made of polypipe with insect netting on top, to keep the crops all for ourselves. As for the yield? Well, we had a few leeks. At least two or three. No lying!

tl;dr: We used

  • pre-fab ~1m3 bed kits and a pond liner for the structure,
  • an ag-pipe and some sand for the lower tier,
  • geotextile as a barrier before putting the garden soil and mulch on top,
  • plumbing bits and bobs to control intake, but also drainage as needed, and
  • polypipe and insect protection, which is optional, but useful.
Continue reading →

Adding 433MHz RC-switched solar lights to Home Assistant with ESPHome

Posted on by | Leave a reply
Photo of a tablet showing a Home Assistant light control, with effects. On the left of the screen is an ESP32-PoE PCB with blinkenlights and a smaller dangly PCB (the FS1000A). On the right are two identical remote controls, one is upside down, showing a bit of masking tape with a hand-written annotation that read “Front, East”

We recently got solar festoon lights for our terrace. One set didn’t cover the whole area so we quickly had to order a second one. While the lights are the same, the control box is ever so slightly different—likely from a different drop-shipment―and the main problem is that we have two identical remote controls for the lighting of a single area.

This of course calls for home automation! It turned out the remotes operate in the 433MHz ISM band, which I could confirm with my tinySA, and subsequently decode with rtl_433 and a Smartee v2 RTL2832U-based software-defined radio receiver from Nooelec.

After a bit of fiddling, I was able to determine how to encode the signal for the ESPHome Remote Transmitter component, for use with an FS1000A transmitter module. bundle separate entities sending each code into an integrated sub-device that shows up nicely in Home-Assistant, with support for adjustable brightness and effects.

tl;dr:

  • The Generic-Remote model with House Codes and Tri-State translates to an rc_switch_raw with protocol 1.
    • The Tri-States 0, Z, X, 1 encode 2 bits, respectively, 00, 01, 10, 11.
  • The ESPHome/Home Assistant component can be incrementally assembled from basic button actions triggering each code into more complex template entity such a light and select.
  • Don’t forget the antenna! (wavelength: 300/433 = 69.3cm -> quarter-wave: 17.3cm)
Continue reading →

Mikrotik RouterOS backups: /export is good, but not enough

Posted on by | Leave a reply

For… reasons… I had to restore my Mikrotik router from a backup yesterday, after a hardware reset. Again, a good opportunity to backups them live! … And of course realise it’s not seamless.

I have historically done my backups manually with /export verbose show-sensitive=true after every config change, and download the generated rsc file off the router. While this is good practice, and the recommended approach for router migration (via the /import command).

The better way to do same-device Router OS backups, that I’ll do from now on, is to use the /system/backup commands. It is also available from the Files section in the WebUI. I’ll also look at creating periodic backups, as I noticed that the files stored on the device persisted across hardware and config resets.

TIL:

Optional Docker services and dependencies

Posted on by | Leave a reply
Screenshot of a terminal. ``` [21:40:31] ~/docker-profiles$ docker compose up 4s Attaching to init-1, main-1 init-1 | init Container docker-profiles-init-1 Waiting init-1 exited with code 0 Container docker-profiles-init-1 Exited main-1 | main main-1 exited with code 0 [21:40:35] ~/docker-profiles$ docker compose --profile opt up 3s Attaching to init-1, main-1, opt-init-1 opt-init-1 | opt-init init-1 | init Container docker-profiles-init-1 Waiting Container docker-profiles-opt-init-1 Waiting Container docker-profiles-opt-init-1 Exited opt-init-1 exited with code 0 init-1 exited with code 0 Container docker-profiles-init-1 Exited main-1 | main main-1 exited with code 0 ```

Like many, docker and and compose have become my go-to tool to create software that can be conveniently deployed to production with a limited amount of headache. However, many tasks, and sometimes whole services, pertain only to the development side of the workflow, and need to stay there.

Moreover, some tasks, such as time-consuming provisioning tasks, are only on-demand one-offs. They shouldn’t run at all most of the time, but they should slot into the dependency graph correctly when needed.

tl;dr: I realised that docker compose supports profiles, which allows services to be enabled conditionally, along with the depends_on.[].required option, to ignore them when they are disabled. Profiles are also useful to package actions and triggers to run on demand, so they are not started by default.

Continue reading →

Clearing Caches

Posted on by | Leave a reply

There is always a time when storage becomes a bit tight. Once the /tmp and downloads directories are clear, what next? Caches! But how to find them?

Fortunately, there is a standard that can be used to mark directories as cache-containing: CACHEDIR.TAG. It is not uniformly used, but many tools, such as plocate or pip do place one such file in their cache directories.

This is enough to delete a bunch of files without having to worry too much about them.

$ sudo locate CACHEDIR.TAG | xargs dirname | sudo xargs -I CCDD find CCDD -mindepth 1 -not -name CACHEDIR.TAG -delete
Continue reading →

Git: ignoring temporary changes

Posted on by | 1 Reply
Screenshot of a terminal. ``` [17:36:00] ~/src/tests/assume-unchanged$ echo b > a 0s main [17:36:02] ~/src/tests/assume-unchanged$ git status 0s ✹ main On branch main Changes not staged for commit: (use "git add ..." to update what will be committed) (use "git restore ..." to discard changes in working directory) modified: a no changes added to commit (use "git add" and/or "git commit -a") [17:36:03] ~/src/tests/assume-unchanged$ git update-index --assume-unchanged a [17:36:10] ~/src/tests/assume-unchanged$ git status 0s main On branch main nothing to commit, working tree clean [17:36:12] ~/src/tests/assume-unchanged$ echo c > a 0s main [17:36:14] ~/src/tests/assume-unchanged$ git status 0s main On branch main nothing to commit, working tree clean [17:36:15] ~/src/tests/assume-unchanged$ git update-index --no-assume-unchanged a [17:36:18] ~/src/tests/assume-unchanged$ git status 0s ✹ main On branch main Changes not staged for commit: (use "git add ..." to update what will be committed) (use "git restore ..." to discard changes in working directory) modified: a no changes added to commit (use "git add" and/or "git commit -a") ```

One can use git add [-N|--intent-to-add] <PATH> to record that a new path will need to be considered in later additions to the index. But what about the other way round? How to tell git that a change SHOULD NOT be considered?

tl;dr: git update-index --assume-unchanged <PATH>

EDIT 2026-03-27: Jakub (in the comments) pointed out that there were risks of overwriting the changes with this approach, and suggested using git update-index --skip-worktree <PATH> as a safer and more appropriate approach.

Continue reading →

Fixing MariaDB socket authentication on Debian

Posted on by | Leave a reply
Screenshot from a terminal. A user enters `mysql` to get into the database shell, but doesn't have the permissions to run `show processlist;`. The user then enters `sudo mysql`, and obtains a shell where `show processlist;` runs successfully, showing that the `root` user is logged in.

As I alluded to before, I use Munin to keep an eye on my machines. It’s not the most modern piece of software, but it works well enough (and I don’t need to worry too much about it as it gets set-up automatically through my SaltStack states). Amongst other things, I use it to monitor my MySQL DBs.

Or used. For some time (way longer than I’m willing to admit without blushing), this particular set of plugins wasn’t working. The authentication was set up so the system users would be granted access to the database using the same user name, if they connected locally using the unix_socket method. This worked well, until it didn’t, and I would see the following sort of error messages in the logs instead.

2026/02/20-18:09:33 [3338788] Service 'mysql_slow' exited with status 11/0.                                                                                                                      
2026/02/20-18:09:33 [3338788] Error output from mysql_slow:                                                                                                                                      
2026/02/20-18:09:33 [3338788]   DBI connect('mysql;mysql_read_default_file=/etc/mysql/debian.cnf;mysql_connect_timeout=5','munin',...) failed: Access denied for user 'munin'@'localhost' at /et 
c/munin/plugins/mysql_slow line 1090.

tl;dr:

  • The authentication system needs to be reset with ALTER USER 'root'@'localhost' IDENTIFIED VIA unix_socket; even if it already looks like it’s the case.
  • It is important that the mysql_ munin plugin is configured so the user AND the env.mysqluser match.
Continue reading →

Fixing WordPress critical errors without email

Posted on by | 3 Replies
Wordpress error screen. “There has been a critical error.” No mention of an email having been sent.

At the time of this writing, this blog runs on WordPress. And the other day, it broke. Clearly an event to the scale of us-east-1 going down. So what happened?

There has been a critical error on this website.

Right. Most information on this topic generally indicate that an email should have been sent to the admin, but this one specifically didn’t.

tl;dr:

  • Activating WP_DEBUG mode allowed me to get PHP stacktraces in the HTML, allowing to identify the source of the issue.
  • It was due to a plugin compatibility issue; moving its directory out of the way to disable it fixed the issue.
  • Further to this:
    • The email cannot be sent from MULTISITE instances as of WordPress 6.9.
    • There are additional ways to enter recovery mode: wp-login.php?action=entered_recovery_mode, but this then requires trial-and-error to find the culprit.
Continue reading →