Repository navigation
Encrypted send to untrusted target not quite working ... pretty sure it's a simple config issue #742
Description
Encrypted send to untrusted target not quite working ...
I have a backup dataset bondi3/zrepl on a server bondi that receives laptop backups. The dataset is encrypted, and laptop sends are plain, so they're re-encrypted on bondi. For offsite backups, I want to replicate this to a backup dataset wider/zrepl on remote server widey. Should be fairly simple ...
The two laptop sub-datasets under bondi3/zrepl - both look like this
bondi3/zrepl/ryzen2/ryzen22/home/alice encryption aes-256-gcm -
bondi3/zrepl/ryzen2/ryzen22/home/alice keylocation none default
bondi3/zrepl/ryzen2/ryzen22/home/alice keyformat passphrase -
bondi3/zrepl/ryzen2/ryzen22/home/alice pbkdf2iters 350000 -
bondi3/zrepl/ryzen2/ryzen22/home/alice encryptionroot bondi3/zrepl
bondi3/zrepl/ryzen2/ryzen22/home/alice keystatus available
bondi zrepl.yml source job
- type: source
name: laptop_source
filesystems:
"bondi3/zrepl<": true
snapshotting:
type: manual
send:
encrypted: true
serve:
type: tls
listen: :8541
ca: /etc/zrepl/ca.crt
cert: /etc/zrepl/bondi.crt
key: /etc/zrepl/bondi.key
client_cns:
- widey
widey zrepl.yml pull job
- name: pull_bondi
type: pull
connect:
type: tls
address: 192.168.2.104:8541
ca: /etc/zrepl/ca.crt
cert: /etc/zrepl/widey.crt
key: /etc/zrepl/widey.key
server_cn: bondi
root_fs: wider/zrepl
interval: 1h
replication:
protection:
initial: guarantee_resumability
incremental: guarantee_resumability
recv:
properties:
override: {
"readonly": "on",
"mountpoint": "none",
"canmount": "noauto"
}
# encryption off as per encrypted-send-to-untrusted-receiver
# https://zrepl.github.io/master/configuration/sendrecvoptions.html#placeholders
placeholder:
encryption: off
pruning:
keep_sender:
- type: regex
regex: '.*'
keep_receiver:
- type: regex
negate: true
regex: '^zrepl_'
- type: grid
grid: 1x1h(keep=all) | 24x1h | 30x1d | 12x30d
regex: '^zrepl_'
Except the two laptop backups (halflapp and ryzen2) fail ... this is the status on the remote widey
bondi3/zrepl PLANNING-ERROR (step 0/0, 0 B/0 B) sender does not have any versions
bondi3/zrepl/halflapp PLANNING-ERROR (step 0/0, 0 B/0 B) sender does not have any versions
bondi3/zrepl/halflapp/halflapp PLANNING-ERROR (step 0/0, 0 B/0 B) sender does not have any versions
bondi3/zrepl/halflapp/halflapp/home PLANNING-ERROR (step 0/0, 0 B/0 B) sender does not have any versions
bondi3/zrepl/halflapp/halflapp/home/alice STEP-ERROR (step 1/1, 0 B/6.8 GiB) parent(s) failed during initial replication: [bondi3/zrepl bondi3/zrepl/halflapp/halflapp]
:
bondi3/zrepl/ryzen2 PLANNING-ERROR (step 0/0, 0 B/0 B) sender does not have any versions
bondi3/zrepl/ryzen2/ryzen22 PLANNING-ERROR (step 0/0, 0 B/0 B) sender does not have any versions
bondi3/zrepl/ryzen2/ryzen22/home PLANNING-ERROR (step 0/0, 0 B/0 B) sender does not have any versions
bondi3/zrepl/ryzen2/ryzen22/home/alice STEP-ERROR (step 1/1, 0 B/44.1 GiB) parent(s) failed during initial replication: [bondi3/zrepl bondi3/zrepl/ryzen2/ryzen22]
Activity
problame commentedon Oct 14, 2023
Sorry for the delayed reply.
I think I understand the problem.
The
bondi-side filesystemsdont' have snapshots because they're placeholder filesystem.
And so, the replication planner code bails out early here:
zrepl/replication/logic/replication_logic.go
Lines 343 to 359 in 2d8c369
Would you be willing to test a fix for this? You'd download a zrepl binary from the GitHub CI and replace your distro's zrepl binaries with it.
Also, can you try this workaround: what if you change the
filesystemson thebondi-side source job to not include the chain of placeholder filesystems?- type: source name: laptop_source filesystems: - "bondi3/zrepl<": true + "bondi3/zrepl/halflapp/halflapp/home/alice": truereplication: replicate placeholder filesystems
problame commentedon Oct 14, 2023
Untested fix in: #744
It's super low risk to try this out on your setup.
Please try it for both initial replication and also a few incremental replication runs after the initial replication.
Binaries available from CircleCI (click the appropriate
quickcheck-*job, then navigate to tab "Artifacts") https://app.circleci.com/pipelines/github/zrepl/zrepl/7429/workflows/50139be5-b0fd-4fcc-a60f-998432b24bb7Halfwalker commentedon Oct 14, 2023
Just tried the new binary on widey - no luck :( You did mean for it to be replaced on the pull/target system right ?
I picked it from
https://app.circleci.com/pipelines/github/zrepl/zrepl/7429/workflows/50139be5-b0fd-4fcc-a60f-998432b24bb7/jobs/51075/artifacts
To be sure, here are the md5sums of the original 0.6.1 (zrepl.orig) and the new test one (zrepl)
It gives the exact same error
problame commentedon Oct 14, 2023
The planner runs on the active side of the replication setup.
In your case, the active side is the pull job on widey, yeah.
Just to make sure you deployed the binary correctly: you need to
systemctl stop zrepl/usr/bin/zrepl)systemctl start zreplMy PR adds debug log messages.
Do you see any of them in the logs?
Please enable debug logging, then check
widey's logs forHalfwalker commentedon Oct 14, 2023
Yup - stopped the service, then in
/usr/binmoved oldzrepltozrepl.orig, then movedzrepl-linux-amd64tozreplHere's the syslog looking for placeholder (didn't find "sender filesystem")
problame commentedon Oct 14, 2023
Hm, that's not helpful.
Please run
on the middle node of your replication chain, i.e., the bondi host.
And just to make sure, please confirm your setup looks like this
Halfwalker commentedon Oct 14, 2023
Placeholders on bondi - only looking at the
zrepldataset, which is the one to be replicated to wideyThe setup is almost right above ...
ryzen2 pushes to bondi (since it's almost alway on - main workstation)
widey pulls from bondi (since it only turns on late at night for backups)
problame commentedon Oct 14, 2023
Ok, so, setup is
I pushed a new commit to the PR, please wait for CI to finish and download the new artifacts, the try the replication again.
Halfwalker commentedon Oct 14, 2023
Nice :) Fast work, mikey likes it.
Same CircleCI url above ? I don't know CircleCI - github-actions/gitlab-CI/Drone yes, not CircleCI
problame commentedon Oct 14, 2023
No, different URL.
You can either navigate CircleCI UI or go via the commit status in the.
(Note the screenshots selects the Go 1.20 build, but feel free to use the Go 1.21 build.
Halfwalker commentedon Oct 14, 2023
Got it. I didn't see a 1.21 version for amd64, just the 1.20, unless I'm misreading ... The others are freebsd 1.20 and 1.21.
New zrepl
Same error :( Searching for placeholder in syslog
problame commentedon Oct 14, 2023
The 1.21 one is is part of the first stage, i.e., in the group of 3 boxes, the one in the middle.
Ah, I forgot to mention: with the commit that I added, it's necessary to update the sending side as well, i.e., you need to deploy the updated binary to both
wideyandbondiHalfwalker commentedon Oct 14, 2023
Ah, OK. Done. Errors out, but different ...
problame commentedon Oct 14, 2023
Hm, yeah, the root_fs dataset
bondi3/zreplis not a zrepl-managed placeholder, so zrepl expects there to be snapshots.That is how it should be, I won't change that behavior.
The following should do the trick:
- type: source name: laptop_source filesystems: - "bondi3/zrepl": true + "bondi3/zrepl": false + "bondi3/zrepl<": trueIf that doesn't work, try
- type: source name: laptop_source filesystems: - "bondi3/zrepl": true + "bondi3/zrepl/halflapp<": true + "bondi3/zrepl/ryzen2<": trueDoc context: https://zrepl.github.io/configuration/filter_syntax.html#pattern-filter
(I haven't thought about
filesystemsfilter syntax in a while, hence my uncertainty).Please report back which one worked.
Halfwalker commentedon Oct 14, 2023
Bingo ! That was it - replicating the halflapp dataset now. I'll post here if the ryzen2 one fails
Halfwalker commentedon Oct 14, 2023
OK, the ryzen2 one seemed to backup fine, so that's good. I set up a timer to trigger backups, came back after a while to see zrepl was stuck in planning stage for over an hour.
bondi seemed to have trouble -
zpool status -vwas hanging on thebondi3pool,zfs get allwas just hanging. Did a full power-cycle (I did try turning it off and back on again), and everything seemed OK. But starting a replication from widey got stuck again, and I see spl panics in thedmesg -Toutput on bondiI don't know if it's related.
Halfwalker commentedon Oct 15, 2023
Point of interest - it's definitely the replication from ryzen2 to bondi that kicks that spl
panic- soon as it tried to replication it kill zfs on bondi. I tested with admesg -Twon bondi and the moment zrepl on ryzen2 started replication - boom, panic.problame commentedon Oct 15, 2023
Well, nice to hear that it works. But this is definitely a ZFS bug, zrepl just uses the CLI and that should never cause panics.
we may still need to work around the issue. What’s the OS and ZFS version?
Halfwalker commentedon Oct 15, 2023
Doh - just remembered, I had NOT replaced the
zreplbinary on ryzen2 ... Did that, but it still causes the panic when replication starts.bondi is ubuntu 18.04, using zfs from the jonathan repo
ryzen2 is ubuntu 22.04, also with zfs from jonathan
Panic on bondi looks like this
I think it's something in the dataset doing it ... I moved back to the original
zreplbinary, and the same thing is happening. So, going to blow away those datasets (they're backups) and start clean with the testzreplbinary.Halfwalker commentedon Oct 15, 2023
That seems to have done the trick ... Changed the bondi config to point to a new dataset. Replication from both ryzen2 and halflapp to bondi worked fine with the new binary. Replication from bondi to widey is ongoing, and seems to be working fine.
So something got mucked up in the original
bondi3/zrepldataset heirarchy, enough to cause spl to freak out and panic when trying to push an incremental stream into it.Question - can multiple
pulljobs run against the samesourcejob at the same time ?Halfwalker commentedon Oct 16, 2023
Sort of answering my own question above ... From the docs https://zrepl.github.io/quickstart/fan_out_replication.html it appears that each pull client wants its own source job on the server. But the config seems to indicate that you can have multiple client_cns in the job.
I noticed with 2 clients (widey and another one wideload) pulling at different times from bondi one or the other would start failing with missing snapshots. Which was odd, since it was only ryzen2 that was managing the pruning on bondi
Refresher ...
pruning.keep_receiverpruning.keep_senderis a no-op, whilepruning.keep_receiverdoes regular grid pruning on the target dataset on wideyExperimenting with a 2nd box wideload to ALSO pull those backups.
Kind of another question ... The pull jobs on widey and wideload had guarantee_incremental set. But I switched it to guarantee_recumability. Now seems to have a persistent error (yet replication works)
What's the right way to clean this up ?
fix: replication of placeholder filesystems (#744)
problame commentedon Sep 5, 2024
Just merged #744 which contains the fix you tested, uhm, almost a year ago.
I didn't want to merge it without a regression test, now it has one.
I assume you found some way to deal with this. If not, please open a new issue.
Halfwalker commentedon Sep 5, 2024
Heh, I've basically been running the fix from back when ;) and it's been working OK. I think I removed all the bookmarks manually to clear it up.
Looking forward to updates with zrepl