test(e2e): record ZFS datasets and LINSTOR volumes in cozyreport - #4657

[MAJOR] human-rounded used without avail cannot resolve a 450-830 MiB shortfall

exec zfs list -t all -o name,used,refer,origin,creation -r $pools prints sizes through zfs_nicenum_format, which fits every number into five characters (OpenZFS 2.3.2 lib/libzutil/zutil_nicenum.c). At or above 100 GiB that means whole GiB. Here is the function compiled from that source and run on realistic pool sizes:

$ ./nicenum $((110*G)) $((110*G+450*M)) $((110*G+830*M)) $((150*G)) $((150*G+600*M)) $((12*G)) $((12*G+450*M))
118111600640   bytes=118111600640 -> 110G
118583459840   bytes=118583459840 -> 110G
118981918720   bytes=118981918720 -> 111G
161061273600   bytes=161061273600 -> 150G
161690419200   bytes=161690419200 -> 151G
12884901888    bytes=12884901888 -> 12G
13356761088    bytes=13356761088 -> 12.4G

The container-lane pools are 200G (hack/e2e-container-up.sh:117). hack/e2e-chainsaw/_lib/run-kubernetes.sh:502-503 says persistent platform volumes leave one pool well below 90 GiB free, so that pool's root used is over 110 GiB. At that size, a pool 450 MiB over its baseline and a pool exactly at it print the same figure. Hypothesis 2 ("the pool root's used matches what the cleanup check saw") and hypothesis 3 ("the root's used is back at its baseline while storage-pools.txt reports less") both need sub-GiB resolution.

They also compare against the wrong quantity. The cleanup check records LINSTOR free_capacity in KiB (run-kubernetes.sh:512-516). LINSTOR 1.33.3, the version packages/system/linstor/Makefile builds, takes that value from zfs get available -o name,value -Hp <pool> (ZfsCommands.getZPoolFreeSize, satellite/.../zfs/utils/ZfsCommands.java:321-339). The listing has no avail column, so no row in it can be set against storage-pools.txt or against the baseline-free-kib lines in the job log. Without -p, creation is also minute-resolution local time instead of epoch seconds that line up with the job log.

Fix: exec zfs list -p -t all -o name,used,avail,refer,origin,creation -r $pools. For hypothesis 2, add zpool get -p freeing,leaked,allocated,free $pools to the same exec; it reads the asynchronous-free backlog directly instead of leaving it to inference. The stubbed zfs in the test already echoes its arguments, so pinning -p and avail in the grep -qx assertions at hack/cozyreport.bats:5125 and :5135 turns them red on the current script and green after the change.

What would change my mind: hypotheses 2 and 3 are meant to be read from the job log's KiB lines and storage-pools.txt alone, and the ZFS listing only serves hypothesis 1. In that case the description should say so.

Fixed in 2fc68f0. The listing is now zfs list -p -t all -o name,used,avail,refer,origin,creation, and the same exec runs zpool get -p freeing,leaked,allocated,free before it. The test pins both commands exactly.

[MINOR] a single end-of-job listing cannot see a transient state at the failed check

The collect step takes the listing once, at the end of the job (Collect report, if: always(), in .github/workflows/pull-requests.yaml). hack/e2e-chainsaw-run.sh keeps running the remaining suites after a failure ([ "$crc" -eq 0 ] || rc=1, then the loop goes on). So every suite that sorts after kubernetes-latest and kubernetes-previous runs before cozyreport does. A leftover dataset (hypothesis 1) survives that wait, and creation dates it. An asynchronous free that completes or a FreeCapacity value that refreshes (hypotheses 2 and 3) has usually resolved by then, and the reader sees a clean pool whichever was true. #4653 asks for exactly this case: "on this failure print zfs list ... for every data-srvN pool together with the LINSTOR resources". The fix is to record the same zfs list -p ... and zpool get -p freeing ... in the failure branch of cozy_wait_linstor_pool_baseline, next to the existing baseline-free-kib dump, while the pool is still short. The alternative is to narrow the description's claim to hypothesis 1.

Not in this PR. The failure branch of the baseline check is in run-kubernetes.sh, which #4651 is rewriting, so a dump at that moment belongs with that change. I narrowed the description instead: the end-of-job listing settles a leftover dataset, and shows hypotheses 2 and 3 only if that state still holds when the report is collected.

[MINOR] a failing zpool list is reported as "no LINSTOR data pool is visible"

In pools=$(zpool list -H -o name | grep -E ...) the pipeline's status is grep's, so a failing zpool produces an empty $pools. Causes include a missing module, /dev/zfs not exposed to the satellite, or a userland/kernel mismatch. The file then ends with a conclusion about the cluster that the script never established. zpool's own message does land one line above it, and COLLECTION-FAILED.txt is written. Still, the closing line reads as the finding and contradicts the line above it. This is the PR's own module evaluated with stub zpools:

##### zpoolfail   (zpool stub: prints "The ZFS modules cannot be auto-loaded." to stderr, exit 1)
== report/linstor/zfs/srv1.txt
The ZFS modules cannot be auto-loaded.
no LINSTOR data pool is visible from this satellite
##### nozpool     (zpool absent)
== report/linstor/zfs/srv1.txt
sh: line 1: zpool: command not found
no LINSTOR data pool is visible from this satellite

Take the list in its own step first, all=$(zpool list -H -o name) || exit $?, then filter it. A test case where zpool exits non-zero should then assert that the file does not say "no LINSTOR data pool".

Fixed in 2fc68f0. The pool list no longer comes from zpool list | grep, and a failing zpool get exits with its own status, so its message is what the file holds and nothing says the node has no pool. A test with a failing zpool stub covers it.

[MINOR] the pool set is guessed from a name pattern, so the doc's "every LINSTOR data ZFS pool" overclaims

^(linstor_)?data(-.*)?$ encodes naming that two other scripts choose (hack/e2e-container-up.sh:190 data-srvN, hack/e2e-post-install-prep.sh:71-72 data), and nothing ties the three together. If the pools are renamed, every listing silently turns into "no data pool", and the bats stub, fed the names the regex was written for, stays green. In the other direction, ZFS is not namespaced, so on the container lane any host pool whose name starts with data- is listed into the artifact. The authoritative mapping is one exec away: linstor --machine-readable sp l, which run-kubernetes.sh:512-515 already parses with jq inside the controller, carries each storage pool's backing zpool. Reading the names from there makes the doc sentence at docs/agents/e2e-testing.md:127 true. At minimum, that sentence should say "pools named data or data-*", which is what the code does.

Fixed in 2fc68f0. The pool names now come from linstor --machine-readable sp l, from StorDriver/StorPoolName (the controller renames ZPool and ZPoolThin to it on create), and each satellite lists only its own node's pools. The doc line now just says per-node ZFS datasets.

[MAJOR] exact bytes are justified by a cleanup check that no longer exists, using a LINSTOR mechanism my previous review got wrong

The comment says exact bytes are needed "because the cleanup check compares KiB of LINSTOR free capacity, which is the pool's available". hack/cozyreport.bats:5138, the body ("Sizes are exact bytes, because the cleanup check works in KiB of LINSTOR free capacity, which is the pool's available") and the commit message say the same. Both halves are wrong on the tree this merges into, and both go into main as the merge commit.

First, #4651 is merged, and cozy_wait_linstor_pool_baseline on main now compares node dataset names, not KiB.

Second, I owe a correction, because this sentence comes from my previous review. On a node with only thick pools, LINSTOR's free capacity is not the live available. In v1.33.3 (pinned in packages/system/linstor/Makefile:7), the controller fetches live space only from nodes that have thin pools. For every other node it serves the value cached at the satellite's last report:

FreeCapacityFetcherProto.assembleRequests (v1.33.3):
    Stream<Node> nodeWithThinStream = nodeStream.filter(this::hasThinPools);
    ...
    private boolean hasThinPools(Node node) {
        ... .anyMatch(kind -> kind.usesThinProvisioning() && !DeviceProviderKind.DISKLESS.equals(kind));

CtrlStorPoolListApiCallHandler (v1.33.3), node with no live reply:
    if (storageInfo == null) {
        freeCapacity = storPool.getFreeSpaceTracker().getFreeCapacityLastUpdated(peerAccCtx.get()).orElse(null);

The satellite does read zfs get available, but linstor sp l shows that figure as of the satellite's last report, which is #4651's explanation for #4653. A reader who trusts the comment would compare avail in this listing with storage-pools.txt, expect them to match, and read a mismatch as a leak. Suggested wording: exact bytes, so a leftover dataset's size can be added up and set against the avail of the pool root. Then say that linstor sp l serves a cached figure on thick-only nodes, so the two can legitimately disagree. The same fix belongs in the bats comment, the body and the commit message.

scooby87 Fixed in 2ce97d4. The script comment, the bats comment, the commit message and the body now say that exact bytes let you sum leftover datasets against the pool root's avail. They also say linstor sp l serves a cached figure on thick-only nodes, so the two can disagree. I checked hasThinPools in FreeCapacityFetcherProto at v1.33.3 before writing it.

[MINOR] one pool that cannot be opened drops the dataset listing for every pool on the node

zpool get keeps going past a pool it cannot open, and only sets the exit status. This is OpenZFS 2.3.2 cmd/zpool/zpool_iter.c, pool_list_get:

for (i = 0; i < argc; i++) {
        if ((zhp = zpool_open_canfail(g_zfs, argv[i])) != NULL) {
                if (add_pool(zhp, zlp) != 0)
                        *err = B_TRUE;
        } else {
                *err = B_TRUE;
        }
}

|| exit $? then skips zfs list entirely. On a node with one faulted or unimported pool plus a healthy one, the healthy pool's datasets are never listed. That is the broken-node run where the listing matters most. This is the module body run against a stub zpool that fails on missing-pool, with srv1 missing-pool and srv1 data-srv1/thin in the map:

== linstor/zfs/srv1.txt
zpool get -p freeing,leaked,allocated,free missing-pool data-srv1
# [cozyreport] TRUNCATED: this output ends here because the read exited 1 part way through, not because the command finished
== linstor/zfs/COLLECTION-FAILED.txt
[cozyreport] srv1.txt: the read exited 1, so its output holds only what was streamed before that.
It said:
cannot open 'missing-pool': no such pool

My previous review suggested ending on that status. That advice was for zpool list feeding a filter; it does not fit here. Run both commands and exit with the worse status, for example zpool get ... $zp; r=$?; zfs list ... "$@" || r=$?; exit $r. The existing failing-zpool test still holds, and a two-pool case where one fails can assert that the other pool's datasets are listed.

scooby87 Fixed in 2ce97d4: zfs list now runs whatever zpool get returns. I didn't use exit $r, because a non-zero exit with stdout sends cozyreport_read_exec into its TRUNCATED branch and marks a complete listing as cut short. So zpool's message and status go into the listing itself, and the read's status comes from zfs list. The failing-zpool test checks the listing is there without the marker, and a new case covers a failing zfs.

[MINOR] a failed spec.nodeName read skips the listing, and the note blames zfs-pools.txt

If the bounded .spec.nodeName read returns nothing, node falls back to the pod name (:2124, pre-existing). The pod name never matches column 1 of zfs-pools.txt, so the node gets "linstor/zfs-pools.txt names no ZFS pool for node linstor-satellite-a ... that file says whether LINSTOR answered". That file does list the node's pools. The exec targets $pod and would have worked, so the listing is lost on a run with a flaky apiserver, and the note sends the reader to a file that contradicts it. Reproduced with a stub, under dash and jq 1.6 in debian:bookworm:

=== zfs/*:
# [cozyreport] linstor/zfs-pools.txt names no ZFS pool for node linstor-satellite-a, so no dataset was listed; that file says whether LINSTOR answered
=== zfs-pools.txt:
srv1 data-srv1
srv1 tank/thin
srv2 data-srv2

Record the fallback in the note, for example "spec.nodeName of $pod could not be read, so it was not matched to a LINSTOR node". A test for this needs set +e in its subshell. The existing tests eval the module under cozytest.sh's set -eu, which hack/cozyreport.sh does not set, so a failing node=$(...) aborts the eval before this branch.

scooby87 Fixed in 2ce97d4. When spec.nodeName comes back empty, the note now reads "spec.nodeName of could not be read, so it was not matched to a LINSTOR node". The new test runs the module under set +e, as you suggested.

[MINOR] the NF == 2 guard and the jq exit status have no covering test, though the body says each guard was mutated

review-helper mutate over the three new tests, one checkout per mutation:

M4 drop NF==2 guard                  GAP: the suite stayed green without the fix
M10 // empty to // ""                GAP: the suite stayed green without the fix
M12 jq status lost (pipe to cat)     GAP: the suite stayed green without the fix
M1 M2 M3 M5 M6 M7 M8 M9 M11          covered: the suite went red

Today the jq path fails closed: non-JSON stdout from linstor -m lands in COLLECTION-FAILED.txt as "the read exited 4 ... parse error". Nothing pins that, though. Add a STUB_SP_JSON='not json' case that asserts the parse error and the per-node note. Then either add a fixture row that only NF == 2 rejects, or drop the guard, since // empty already removes a pool with no StorDriver/StorPoolName. Either way, the body's "Each guard was mutated, and only the test covering that guard went red" needs correcting.

scooby87 Fixed in 2ce97d4. I dropped NF == 2, because // empty already removes the whole interpolated row. The no-pool test now has a srv1 ZFS row with no StorPoolName, plus a not json case. // "" turns it red, and so does | cat. The body no longer says every guard was mutated. It names the four that were.

Neither this NF == 2 filter nor // empty in the jq line for zfs-pools.txt is pinned. Each mutant on its own keeps all 189 tests in hack/cozyreport.bats green, same for piping the jq line through cat. With both gone, a pool row without StorPoolName gives a blank but non-empty list, the -z check passes and zfs list -r runs with no pool argument and lists every pool on the machine, the case the comment above warns about. A fixture row without StorPoolName on a node that has a pool row, plus an sp l case returning non-JSON, would pin them. Or drop one of the two filters, either one alone keeps the blank row out.

myasnikovdaniil Fixed in 2ce97d4, with the same change as scooby87's thread on this line. NF == 2 is gone. A ZFS row without StorPoolName on a node with no other pool pins // empty, and a non-JSON sp l case pins jq's exit status.

Same wording as in the script comment. Since #4651 the check compares dataset names, not KiB of free capacity, and on thick-only nodes the figure LINSTOR reports is the cached one. Reword together with the script comment.

myasnikovdaniil Reworded in 2ce97d4, together with the script comment: exact bytes so leftover datasets can be summed against the pool root's avail, and the cached linstor sp l figure on thick-only nodes named as a legitimate difference. The body now describes what this adds on top of #4651 and no longer treats #4653 as open.

[MINOR] the comment, PR body and commit message justify exact bytes and avail with a cleanup check that no longer exists on main

The comment at 2128-2131 (and the same sentence in hack/cozyreport.bats and in the commit message/PR body) says exact bytes and avail are needed "because the cleanup check compares KiB of LINSTOR free capacity, which is the pool's available". On current main (9809b22, after #4651 merged) cozy_wait_linstor_pool_baseline compares per-node dataset NAMES, not KiB of free capacity: _cozy_linstor_pool_datasets runs zfs list -H -o name -t filesystem,volume,snapshot -r <zpool> and cozy_linstor_pool_new_datasets diffs the name lists (baseline rows are labelled baseline-dataset:). The PR branch was cut before #4651 so its own tree still carries the old KiB check, which is why this reads correct in the clone and wrong in the commit it becomes. A triager who trusts the comment will set this listing's avail against storage-pools.txt and read a legitimate cached-figure difference as a leak. The listing itself (exact bytes, avail, all dataset types, freeing) is still useful; only the stated reason is stale. Severity capped at MINOR: locus is a comment plus the commit record in test tooling, no data/upgrade/tenant consequence.

IvanHunters Fixed in 2ce97d4. The script comment, the bats comment, the commit message and the body now say exact bytes let you sum leftover datasets against the pool root's avail, and that linstor sp l serves a cached figure on thick-only nodes, so the two can disagree.

[MINOR] zpool get ... || exit $? drops the dataset listing for every pool on a node when any one pool cannot be opened

In the per-satellite exec, zpool get -p freeing,leaked,allocated,free $zp || exit $? runs before exec zfs list ... -r "$@". zpool get over several pools keeps going past a pool it cannot open and only sets a non-zero exit status, so || exit $? then skips zfs list for the healthy pools too. On a node with one faulted/unimported pool plus a healthy one, the healthy pool's datasets are never listed, which is exactly the broken-node run the listing exists for. Shell logic confirmed by reading; the zpool get partial-failure behaviour is per scooby87's OpenZFS source citation (not executed here). Run both and exit with the worse status instead. Introduced by this change. MINOR: diagnostic-completeness loss in test tooling, read-only, no data harm.

IvanHunters Fixed in 2ce97d4: zfs list now runs whatever zpool get returns. I didn't exit with the worse status, because a non-zero exit with stdout makes cozyreport_read_exec mark a complete listing as TRUNCATED. zpool's message and status go into the listing instead, and the read's status comes from zfs list. Both a failing zpool and a failing zfs have their own test.

[MINOR] a failed spec.nodeName read falls back to the pod name, so a reachable node gets the "no ZFS pool" note and the note blames zfs-pools.txt

When the bounded .spec.nodeName read returns nothing, node falls back to the pod basename (:2124, pre-existing). That pod name never matches column 1 of zfs-pools.txt, so pools is empty, the node is written the note "linstor/zfs-pools.txt names no ZFS pool for node ... that file says whether LINSTOR answered", and no zfs list runs although the satellite exec targets $pod and would have worked. The note sends the reader to a file that in fact lists the node's pools. Verified by reading. The fallback is pre-existing but the consumption that turns it into a misleading note is introduced here. MINOR: diagnostic-accuracy in test tooling.

IvanHunters Fixed in 2ce97d4. An empty spec.nodeName now gets a note that names the unread node name and not zfs-pools.txt, and a test runs that path under set +e.

[MINOR] the NF == 2 and jq // empty / jq-exit-status guards have no covering test, and the body's "each guard was mutated" claim is false for them

The two sp fixtures in the suite (LINSTOR_SP_JSON, only_srv2) are both well-formed, so no test feeds a node row that is present but missing StorDriver/StorPoolName, a row with NF != 2, or non-JSON sp l output. I removed the NF == 2 guard and the full suite stayed green (189/189, dash). So the body's statement "Each guard was mutated, and only the test covering that guard went red" does not hold for NF == 2 (and, per scooby87's mutation table, also not for // empty or the jq exit status). The guards do currently work: I confirmed jq's \(... // empty) drops the whole row when StorPoolName is missing, so a blank-but-non-empty pools (which would make zfs list -r run with no pool argument and list every pool on the machine) needs BOTH guards removed. This is a test-adequacy + body-accuracy gap, not a live defect. Add a fixture row missing StorPoolName on a node that has a pool row, and a non-JSON sp l case. MINOR: the protected failure is a noisy all-pools listing, not data loss; locus is test tooling.

IvanHunters Fixed in 2ce97d4. I dropped NF == 2, since // empty already removes the whole row. A ZFS row with no StorPoolName on a node with no other pool, plus a non-JSON sp l case, now pin // empty and jq's exit status. The body no longer claims every guard was mutated.

[MINOR] a present-but-empty StorDriver/StorPoolName defeats the no-pool guard and runs zfs list -r with no pool

A present-but-empty StorDriver/StorPoolName defeats the no-pool guard and runs zfs list -r with no pool argument, which lists every dataset on the node.

// empty only drops a row whose value is null, which is the missing-key case the test covers. For "StorDriver/StorPoolName": "" the jq line prints srv1 (trailing space), awk turns that into pools=" ", [ -z "$pools" ] is false, and sh $pools word-splits to zero arguments, so both zpool get and zfs list -r run unscoped. That is the outcome the comment at :2137-2139 and the test at :5169 say must not happen.

Reproduced under /bin/sh with a "StorDriver/StorPoolName":"" row, linstor/zfs/<node>.txt gets:

zpool get -p freeing,leaked,allocated,free
zfs list -p -t all -o name,used,avail,refer,origin,creation -r

Both run with no pool. The guard is new in this PR (absent at the merge base), so this is a gap in new code rather than a regression. Your own adjacent code in hack/e2e-chainsaw/_lib/run-kubernetes.sh:524-529 uses // "" and then checks [ -n "$_node" ] && [ -n "$_zpool" ] || return 1 per line.

I could not confirm whether LINSTOR v1.33.3 ever emits an empty StorPoolName, so the reachability is unproven; the mechanism is deterministic once the input occurs.

Fix: filter in jq with select((.props["StorDriver/StorPoolName"] // "") | length > 0), or count words (set -- $pools; [ $# -eq 0 ]) instead of testing the string, and add a "StorDriver/StorPoolName":"" row to the a node linstor names no zfs pool for test.

IvanHunters Fixed in 6b7f7f0. An empty pool name is now dropped in jq the same way a missing one is, so the guard never sees srv1 with a trailing space. The no-pool test has a srv1 row with "StorDriver/StorPoolName":"". On the previous head it fails with both unscoped commands in the listing.

[NIT] two satellite pods on one node during a rollout can overwrite the zfs listing

Two satellite pods on one node during a DaemonSet rollout map to the same $node, and cozyreport_read_exec opens the file with >, so a second Terminating pod can overwrite a good listing with its container not found stderr. This is pre-existing for <node>.tgz (acknowledged at :2117-2118) and new for the zfs listing. Non-blocking, and low probability, since the lanes do not roll satellites during collection.

IvanHunters Leaving this one as is. The <node>.tgz bundle right next to it has the same keying, so the fix belongs in how the loop names files for both. The lanes don't roll satellites during collection anyway.

[NIT] duplicate pool roots in zpool get when two pools share one zpool

${d%%/*} for tank and tank/thin on one node yields zp=" tank tank", so zpool get prints tank twice. Cosmetic.

IvanHunters Leaving it. The pool's rows are printed twice and nothing parses that output. Deduping in the satellite's shell is more code than the duplicate is worth.

[NIT] TRUNCATED marker misleads when one named dataset is already gone

When one named dataset is already gone, zfs list -r prints the rest in full and exits 1, so the read is marked TRUNCATED although the listing is complete. This is the shared reader's semantics rather than this PR, but a divergence listing is exactly where that marker misleads.

IvanHunters It reads wrong there, but that's the shared reader's rule for any non-zero exit after output. Special-casing this one read would make it disagree with the rest. The zfs message with the missing dataset's name goes to COLLECTION-FAILED.txt next to the file, so it's still clear which one was gone.

IvanHunters My answer here was wrong. OpenZFS zfs_for_each lists every argument it can open and exits 1 for the one it can't, so with a missing pool a complete listing with exit 1 is the normal case. Fixed in 2e5fd56. A non-zero exit after output is now marked EXIT N after this output, and TRUNCATED is left for the timeout.

Optional. A StorPoolName of one space passes the jq filter and gives pools=" ", which [ -z ] doesn't treat as empty, so zfs list -r runs with no pool argument, same outcome the empty name had before this push. Filtering on the awk side ($1 == n && NF == 2) drops empty and blank values in one place, and a one-space row in the no-pool fixture would pin it.

Fixed in 7f0c1d0. Blank names are now dropped only in awk, which splits the pool map on tabs, so a pool name with a space (ZFS allows one) still reaches zpool and zfs as one argument.

[NIT] zpool roots not deduplicated when two LINSTOR pools share one zpool

zpools() strips each dataset to its pool root with ${d%%/*}, so two LINSTOR pools backed by one zpool (tank/a, tank/b) pass tank tank to zpool get, which reports the pool twice. zfs list dedups its own arguments; zpool get does not. Cosmetic only; carried from an earlier round and still open.

Cosmetic, so I'm leaving it out of this PR. Filed as a follow-up in #4809, together with the empty zfs-pools.txt one.

[NIT] two satellite pods on one node overwrite the same per-node zfs listing

The per-node listing goes to linstor/zfs/<node>.txt with no per-pod suffix and cozyreport_read_exec opens it with >. Two satellite pods resolving to the same node during a DaemonSet rollout can have the Terminating pod's exec refusal overwrite a good listing. Pre-existing for the <node>.tgz bundle in the same loop; the new file inherits the shape. Carried from an earlier round, low probability.

Same node-keyed naming the <node>.tgz bundle already has. Tracked in #4743, the fix belongs in the loop so both files get it at once.

[NIT] a complete zfs list that exits 1 is marked TRUNCATED

A zfs list -r that lists everything it can and exits 1 (one named dataset already gone, exactly a divergence red-run) is marked TRUNCATED by the shared reader although the listing is complete. The real cause still lands in COLLECTION-FAILED.txt, so nothing is lost; the in-file marker overstates. Shared reader's existing non-zero-after-output semantics, not new here. Carried from an earlier round.

This is by design: the shared reader marks any non-zero exit after output as TRUNCATED, and the comment above the call says so. The real cause still lands in COLLECTION-FAILED.txt, so I'm keeping it.