Skip to content

fix(post_rules): a trailing particle attaches to the family under a declared family-first order (#365) - #466

Closed
derek73 wants to merge 3 commits into
masterfrom
fix/365-trailing-particle-attaches
Closed

derek73 wants to merge 3 commits into
masterfrom
fix/365-trailing-particle-attaches

Conversation

@derek73

@derek73 derek73 commented Aug 30, 2026

Copy link
Copy Markdown
Owner

A trailing surname particle stops being stranded as a standalone middle or given name under a declared family-first order.

Closes #365

The defect

The two family-first orders disagreed about the same input. They permute which role a trailing never-given particle falls into — GIVEN under one, MIDDLE under the other — and only the given position was inspected:

Mesnil Garcia de   FAMILY_FIRST             family 'Mesnil'  given 'Garcia'  middle 'de'
Mesnil Garcia de   FAMILY_FIRST_GIVEN_LAST  family 'Mesnil Garcia de'  given ''

Both now give family de Mesnil, given Garcia.

It is P6's rule, not P1's

The issue proposed adding MIDDLE as a third P1 site, and its own revision comment proposed dropping the other fold. Both are worse: the first loses Garcia because a particle trailed it, the second reports de as the given name, which P1 exists to prevent.

rules.md#P6 already states this rule for exactly this position — a particle ending the name has nothing to link forward to and must belong to the family written beside it — and scoped it to the comma form. Its decision record left the gap as an open parenthesis: "arguably a missing comma under a declared family-first order." This decides that. A declared family-first order names the family as a comma does, so Jong Anke de now reads as Jong, Anke de does.

The design is the issue's own third reading, mechanism included — family-first as an implied comma, with FOLDED_TAG doing the reorder. What this PR adds is the guards.

Three conditions, each the comma form read across

Each was a defect found in review before it was a condition. The comma and comma-less forms now agree row for row:

Jong, de         → family 'Jong de'     |  Jong de          FF → family 'Jong de'
Mesnil, Juan de  → family 'de Mesnil'   |  Mesnil Garcia de FF → family 'de Mesnil'

A name word must remain besides the family. Without P6's own words-to-spare test the new site was wider than the rule it narrows: Jong de attached where Jong, de declines, so two writings of one one-given-word listing disagreed about word order — and the no-given-word case decisions.md deliberately leaves open was half-decided in passing.

The family must have a base of its own. A name word merely ahead of the run is not enough. van Berg Jan de under FAMILY_FIRST lost its given name outright — 'Berg' before, empty after, family de van Berg Jan — because the leftover is partitioned into UNITS and _units chains a leading particle over every word after it, collapsing [van][Berg][Jan] into one unit that takes FAMILY. 288 of a 45,648-parse battery. P1 masks it for every never-given opening, which is why fixtures that all open with de missed it.

No suffix word stranded at the end. assign leaves a trailing Jr. in a name position when a particle follows it; take the particle away and the leftover is not the same name one word shorter, because the re-layout does not re-run assign's trailing peel. Berg Jan Jr. de reported given Jr., and every battery name where the two orders agreed before and disagreed after was this shape — the rule re-opening the divergence it exists to close. It declines rather than mirroring assign's peel by hand, which is the #424/#425 failure.

Never-given vocabulary only

The comma path takes ambiguous particles too; this one cannot. P6's own Accepted clause decides it: the comma-less FAMILY_FIRST_GIVEN_LAST form is the one format that reads the unaccented Vietnamese spelling correctly, and an ambiguous-inclusive attachment takes that format away — measured, Nguyen Thi Van would give family Van Nguyen, given Thi, against P6's pinned given Van. The words-to-spare guard does not save it, since a given word does remain.

The cost is the Dutch flagship word itself. van is ambiguous, so Beethoven Ludwig van keeps middle van where Beethoven, Ludwig van gives family van Beethoven. The two forms agree for the never-given half of the tussenvoegsel set and not for the rest. The rule says so, as an Accepted clause.

Reach

12 of 6600 parses — the 1100-name corpus × three name_order values × middle_as_family off and on — over four names, none under the default order. Two of the four are names this PR's own example lines add to the corpus, which is why the corpus is 1100 and not master's 1094.

The gate cannot see any of it. compare.py parses every name as HumanName(name) and sweeps no policy at all, so no non-default name_order behavior has ever been compared across versions. Output is unchanged at all three baselines apart from the added corpus names, none of which diffs. The counts carry a recompute recipe in decisions.md#P6 instead, under the AGENTS.md provenance rule — the comparator named as the parent commit reading the same corpus files.

The issue's headline number does not measure this shape: "464 inputs where the two orders differ" is what permuting given and middle does.

Review

Four agents. Two regressions, one missed instance of the bug being fixed, one design gap — plus a fourth finding neither the suite nor the corpus could reach: Mesnil Garcia de, Dr. still exhibited #365 verbatim. state.order is not None already excludes every comma that named a family, so testing structure is not FAMILY_COMMA as well excluded only the comma that fixed nothing. Two reviewers found that independently with different inputs.

A guard became dead code. The base and words-to-spare conditions made P1's lead site and P6's new site mutually exclusive — P1's lead fires only on a leading lone never-given particle, which by definition has no base. The FOLDED_TAG exclusion added to P1's redistribution is now unreachable (0 differences over 61,854 generated parses) and is deleted, with the exclusivity pinned instead.

Two comment defects worth naming, both of which would have misled a reader more than the code:

  • The site's headline example claimed Beethoven Ludwig van reads like its comma form. False, and it contradicted a machine-checked example line twelve lines below it in the same commit. A survivor of the rejected ambiguous-inclusive draft.
  • "Attaching first leaves that site nothing to fire on — 6 of 1094 names × 3 orders" was true of the corpus before this commit added van der to it via a new example line. corpus_rules.jsonl regenerates from rules.md, so a rule that adds examples adds corpus names and then counts them. Every count now names its corpus.

Also corrected: the order-agreement claim held only at two leftover pieces (Mesnil Garcia Carlos de disagrees at three); P6's P5 clause was unqualified where it is now comma-only; the stage header was behind the code; and the #359 release bullet described behavior this PR's bullet contradicts.

Verification

  • 6155 → 6178 passed; mypy, ruff and 232 sphinx doctests clean.
  • Gate green at all three baselines: 1100 names, 229 / 194 / 102 intentional, unexplained: 0.
  • Eleven conditions and branches mutation-verified, each killed by its own test — including the eight the test reviewer had killed silently on the first draft.
  • Reach measured against a checkout of the parent commit reading the same corpus files, not against any construction of the rule.

🤖 Generated with Claude Code

…-first order (#365)

The two family-first orders disagreed about the same input. They
permute which role a trailing never-given particle falls into --
GIVEN under one, MIDDLE under the other -- and only the given
position was inspected, so "Mesnil Garcia de" read family 'Mesnil',
given 'Garcia', middle 'de' under FAMILY_FIRST and family
'Mesnil Garcia de' with no given name at all under
FAMILY_FIRST_GIVEN_LAST.

The issue proposed adding MIDDLE as a third P1 site, and its own
later revision proposed dropping the other fold. Both are worse: the
first loses a given name because a particle trailed it, the second
reports 'de' AS the given name, which P1 exists to prevent. The
disagreement is not P1's. P6 already states this rule for exactly
this position -- a particle ending the name has nothing to link
forward to and must belong to the family written beside it -- and
scoped it to the comma form, leaving "arguably a missing comma under
a declared family-first order" as an open parenthesis in its own
decision record. That is what this decides: a declared family-first
order names the family as a comma does, so the attachment fires
there too, and "Jong Anke de" now reads as "Jong, Anke de" does.

Never-given vocabulary only, where the comma path takes ambiguous
particles too. P6's own Accepted clause is what decides it: the
comma-less FAMILY_FIRST_GIVEN_LAST form is the ONE format that reads
the unaccented Vietnamese spelling correctly, and taking ambiguous
particles here takes that format away.

Three details measurement found rather than design:
- the words-to-spare test is a BASE, not a name piece -- the wider
  reading reorders inside an all-particle name ('van der' -> 'der van')
- P6 now lands at two code sites, the no-comma one before P1 (whose
  given site would take the same run with the other reading) and the
  comma one after it (it reads the roles P1 settles)
- P1's family-first lead branch handed the attached particle straight
  back to the given slot, so it now skips FOLDED_TAG tokens

14 of 6564 parses move (1094 names x 3 orders x middle_as_family off
and on), over four names, none under the default order. The gate
cannot see any of it -- compare.py parses with default policy and
sweeps no policy at all -- so the count carries a recompute recipe in
decisions.md#P6 instead. Gate green at all three baselines.
_is_name_word carried `rules.md#R2's "a particle needs a base to
attach to"`. The citation test recognizes `rules.md#R2:` and not the
possessive form, so the quote was never compared to anything. Writing
it in the checked form made the test fail twice over: a docstring's
continuation lines do not start with `#`, so the extractor sees only
the first line and finds no closed quote; and the excerpt is from an
Accepted clause, where _statements() stops at a rule's first example.

Both are properties of the doc format rather than defects, so the
docstring says what it means in its own words and records why it
cites nothing. The rule's words are in P6's body, which is where this
predicate's caller cites.
Two regressions, one missed instance of the bug being fixed, and a
design gap that made the new site wider than the rule it extends.

(i) A NAME WORD MUST REMAIN BESIDES THE FAMILY. Without P6's own
words-to-spare test the no-comma site was WIDER than the comma site
it claims to narrow: `Jong de` attached and gave family 'de Jong'
where `Jong, de` declines and gives 'Jong de'. Two writings of one
one-given-word listing disagreed about word order, and the no-given-
word case that decisions.md leaves deliberately open was half-decided
in passing. With the test the two forms agree row for row, `Mesnil de`
stays P1's fold, and P1's example line for it is unchanged by this PR.

(ii) THE FAMILY MUST HAVE A BASE, not merely a name word ahead of the
run. `van Berg Jan de` under FAMILY_FIRST lost its given name outright
-- 'Berg' before, empty after -- because the leftover is partitioned
into units and _units chains a leading particle over every word after
it, collapsing [van][Berg][Jan] to one unit that takes FAMILY. 288 of
a 45,648-parse battery lost a given name that way. P1 masks it for
every never-given opening, which is why fixtures all opening with `de`
missed it.

(iii) NO SUFFIX WORD STRANDED AT THE END. assign leaves a trailing
`Jr.` in a name position when a particle follows it, and the re-layout
does not re-run assign's trailing peel, so `Berg Jan Jr. de` reported
given 'Jr.'. Every battery name where the two orders agreed before and
disagreed after was this shape -- the rule re-opening the divergence
it exists to close. It declines rather than mirroring assign's peel by
hand, which is the #424/#425 failure.

(iv) NO STRUCTURE TEST. `state.order is not None` already excludes
every comma that named a family, so testing the structure too excluded
only the comma that fixed nothing -- where #365's symptom survived
verbatim (`Mesnil Garcia de, Dr.`).

Reach falls from 14 parses to 12 of 6600 over four names, none under
the default order. Conditions (i) and (ii) also made the two particle
sites mutually exclusive, so the FOLDED_TAG exclusion added to P1's
redistribution is unreachable -- 0 differences over 61,854 parses --
and is deleted, with the exclusivity pinned instead.

Comments and docs corrected with it: the site's headline example
claimed `Beethoven Ludwig van` reads like its comma form, which is
false and contradicts an example line twelve lines away (`van` is
ambiguous vocabulary); "attaching first leaves that site nothing to
fire on" counted a corpus this commit then changed; the order-agreement
claim held only at two leftover pieces; P6's P5 clause and the stage
header were behind the code; and the #359 release bullet described
behavior this bullet contradicts. Every count now names its corpus.

Eleven conditions and branches mutation-verified, each killed by its
own test.
@derek73 derek73 added this to the v2.2 milestone Aug 30, 2026
@derek73 derek73 added the bug label Aug 30, 2026
@derek73 derek73 self-assigned this Aug 30, 2026
@codecov

codecov Bot commented Aug 30, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.66%. Comparing base (caaed90) to head (c0c4175).

Additional details and impacted files
@@            Coverage Diff             @@
##           master     #466      +/-   ##
==========================================
+ Coverage   98.65%   98.66%   +0.01%     
==========================================
  Files          45       45              
  Lines        3194     3219      +25     
==========================================
+ Hits         3151     3176      +25     
  Misses         43       43              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@derek73

derek73 commented Aug 30, 2026

Copy link
Copy Markdown
Owner Author

Closing unmerged. The review that hardened this PR also undermined its premise.

The rule this extends is a relaxation: a particle can only join backward when something has said the family comes first, and without a comma the parser is inserting one on no evidence but the particle's position. Jong Anke de was being read as Jong, Anke de by guess.

Three review rounds found three misparses from that guess reaching too far — a lost given name (van Berg Jan de), a post-nominal promoted into the given slot (Berg Jan Jr. de), and a missed instance of the original bug (Mesnil Garcia de, Dr.). Each was answered with another guard, four in the end. That ratio is the evidence: the feature was being fenced rather than fixed.

Two premises this PR argued from are also wrong, and both were mine:

  • "A never-given particle cannot be a name at all." False. NON_GIVEN_NAME_PARTICLES supplies a default reading, not a prohibition — measured, all 33 of them read as the given name where position forces it ("de" and "de Jr."). This was the sole argument I gave against the strict resolution, and it does not stand.
  • 1.4.0 parity. Irrelevant here: 1.4.0 has no name_order, so nothing in the family-first orders has a v1 answer to be measured against. The comparisons in this branch's release bullet were beside the point.

#365 stays open and will be resolved the strict way instead: a particle is read where the declared order puts it, and joins backward only after an explicit comma. Superseded by the design work tracked in the follow-up.

The branch fix/365-trailing-particle-attaches is kept for the measurements and the review record.

@derek73 derek73 closed this Aug 30, 2026
derek73 added a commit that referenced this pull request Sep 1, 2026
The release note claims 2.2 settles what a declared name_order means for
Latin-script names. Derek asked whether that is documented anywhere --
specifically the trailing particle, "Jong Anke de". It was not.

customize.rst's name_order section covered the basic family-first read,
the comma override, FAMILY_FIRST_GIVEN_LAST for Vietnamese, and the
LEADING never-given particle that outranks the declared order. It said
nothing about either change this release actually made: #467's trailing
particle, or #395's bound on how far a leading run reaches. rules.md#P6
and decisions.md#P1 have both, but docs/design/ is contributor
documentation and is not in the Sphinx build at all -- source_suffix is
'.rst' and it is not in the toctree -- so no user reaches it from the
published docs.

Adds a subsection built on the rule Derek stated in the #466 postmortem:
a particle joins FORWARD, and backward joining is licensed only by a
family comma or a declared family-first order, both being someone
SAYING the family came first. That makes the two spellings one fact
rather than two behaviours to memorise:

    parse("Jong, Anke de").family              -> 'de Jong'
    family_first.parse("Jong Anke de").family  -> 'de Jong'

Then why FAMILY_FIRST is the only order it arises under (it is the only
one that puts a trailing piece in a MIDDLE), the reach bound with the
default-order contrast, and where the two family-first orders diverge.

Eight new doctests, all measured before being written; the suite is 244
doctests, 0 failures. The #467 and #395 bullets now point at
:doc:`customize` for the behaviour and at decisions.md for the
reasoning, rather than only at the contributor doc.

Co-Authored-By: Claude Opus 5 <[email protected]>
@derek73
derek73 deleted the fix/365-trailing-particle-attaches branch September 1, 2026 05:21
derek73 added a commit that referenced this pull request Sep 1, 2026
Ten findings from two review agents, all in dc17090. The three that
matter share one cause: the new entry re-derived arguments
decisions.md#P6 already makes, instead of citing them -- which is the
duplication this very PR exists to delete.

- The Declined block now cites decisions.md#P6 for where "216 of 216"
  came from (a sorted(particles)[:14] slice, already recorded there as
  axis 2 in its purest form) rather than inventing "six hand-picked
  particles", which contradicted it and halved the sampling error.
- It also stopped attributing #466's failure to do/mc/vd. #473's
  "ambiguous-particle half" is the 37 non-never-given particles, and
  decisions.md#P6 records that #466 died on a never-given test that
  excluded all 37, van among them. Different partition, different
  failure.
- The recompute recipe said "any spread of families and givens". With
  F or G drawn from PARTICLES it is 0 of 70, not 67 of 70
  ("Doe, Van de" vs "Doe Van de"), so a reviewer probing particle
  behavior would have reported P6 broken. Now bounded and measured.

Also: AGENTS.md's corpus caution loses its standing count -- both the
drifted and the "corrected" wording quoted figures decisions.md#P6 had
already superseded; axis 11 is no longer classed rules.md-specific,
which contradicted its own "consults no rule statement" and would have
had a reviewer skip it on a decisions.md-only change like this one;
axes 10/11 are no longer said to come from "later arcs" (#291
predates #386); the root pointer drops its numeral; docs-review.md
stops naming a section commit 9f1d46e deleted; and the agent's
description names AGENTS.md, which the axes bind and it omitted.

Refs #473

Co-Authored-By: Claude Opus 5 <[email protected]>
ethan42 pushed a commit to SavantEnvs/python-nameparser that referenced this pull request Oct 8, 2026
…ohibition

Three sites said a never-given particle "cannot be a name at all" or
"can never be a given name", and used that to justify the readings
built on it. It is false: NON_GIVEN_NAME_PARTICLES supplies the
reading position leaves open and forbids nothing. Measured, all 33
members report given == themselves when parsed bare.

The conclusions those sites draw are right; the reason given for them
was not, and it is load-bearing. A particle OPENING the name has the
rest of the name to join forward to -- that is evidence about the
writing rather than about the word, which is why no declared
name_order contradicts it. Stating it as a property of the vocabulary
invites the opposite inference for a particle ENDING the name, where
there is nothing to join forward to and the writing is all there is.

This is the premise derek73#466 was argued from, and the second correction of
this same over-claim: decisions.md#P6 already records dropping "no
particle is a name by itself" for the same reason.

Behavior descriptions that the rest of derek73#467 will change are left alone
here; this commit moves no parse.
ethan42 pushed a commit to SavantEnvs/python-nameparser that referenced this pull request Oct 8, 2026
The 1.0s wall-clock bound failed four times across derek73#466 and derek73#474, always
on CI's 3.12 job, always between 1.01s and 1.08s, with master re-running
green each time. Three local methods disagreed with CI and with each
other: uninstrumented ~1% branch-over-parent (inside the parent's own
spread), coverage-instrumented indistinguishable, and the whole
benchmark file under --cov made the branch FASTER. The same harness
measured origin/master at 94-96ms and again at 89-92ms twenty minutes
later. A check that cannot separate a 1% change from a busy runner does
not fail safe; it fails expensively.

Call counts are deterministic. Budgets: 470 for parse(), 512 for
HumanName(), set at 408.4 and 445.4 with 15% headroom. A loose 5s
backstop stays for what call counts cannot see -- a stage that gets
slower without calling anything more.

WHAT THE OLD BOUND CAUGHT IS RECORDED, NOT LOST. Against the v2.1.0 tag,
back to back on one machine: 79.1ms -> 97.0ms for 1000 facade parses,
+23%, and 386.4 -> 441.4 calls per parse. The bound was not too tight;
the parser had grown into it, and the clock could not say so with
enough confidence to act on.

The largest contributor is a correctness decision rather than waste:
derek73#439 extracted the piece-level predicates so group and assign would
share one implementation (mechanisms.md#ONE-PREDICATE-PER-QUESTION,
which exists because two copies of "is this a title piece" drifted and
caused bugs). That turned inline expressions into ~51 calls per parse
while _group shed 8. The leaves are two membership tests each; there is
no hotspot, and recovering the cost means inlining them back at hot
sites, reintroducing the drift derek73#439 removed. Not taken -- recorded in
decisions.md#parse-cost with the per-stage and per-module numbers.

One repair was tried and reverted: collapsing post_rules' three role-
index scans into one pass saved ONE call per parse, because those sites
only run when their rule fires. Recorded so it is not re-tried, and as
the reason the budget is the instrument -- a harness with +/-10% noise
cannot tell a one-call change from a fifty-call one.

The budget binds: 80 synthetic calls per parse trips both tests with
the actual number in the message.
ethan42 pushed a commit to SavantEnvs/python-nameparser that referenced this pull request Oct 8, 2026
Axes 1-10 compare a document to another document, to its examples, or
to the implementation, so none of them can find a rule that is simply
wrong -- the failure that backed out derek73#461 and closed derek73#466 unmerged.

The worked invariant: the family-comma writing under the default order
and the comma-less writing under FAMILY_FIRST must parse alike. It
holds for 67 of the 70 particles and fails for do, mc and vd, which is
rules.md#P6's contested-membership set recovered from behavior alone.

Also corrects the axis count in three places and the claim that derek73#386
earned every axis, which axes 10 and 11 falsify.

Closes derek73#473

Co-Authored-By: Claude Opus 5 <[email protected]>
ethan42 pushed a commit to SavantEnvs/python-nameparser that referenced this pull request Oct 8, 2026
Form 4 under FAMILY_FIRST must parse as its form-2 comma twin, title
movement included. Generated over a small forked vocabulary (never-
given/ambiguous/multi-token particles, an ASCII and non-ASCII
family, doubled middles) rather than sampled, and vocabulary-blind on
purpose: decisions.md#P6 (the 2026-08-30 derek73#467 entry) measured 0 of
630 -> 603 of 630 over the whole 70-particle vocabulary, with the 27
failures being exactly the particle-and-suffix trio (vd, mc, do) -- a
real limit, not noise (rules.md#P6) -- and derek73#466, the rejected
predecessor, lost a given name outright on "van Berg Jan de" and
promoted a post-nominal into the given slot on "Berg Jan Jr. de".
decisions.md#two-input-invariants recomputes the same correspondence
a second way (a 6x6 family/given spread per particle) and supplies
this file's negative-control recipe. An earlier draft of this file
cited "0/216 -> 216/216", which decisions.md#P6 explicitly retracts
as a sliced-vocabulary artifact (`sorted(particles)[:14]`, which
happened to exclude the trio); this file's own count -- 96
parametrized pairs plus a full-vocabulary negative-control sweep --
is kept separate from and never confused with either retracted or
surviving number.

The comparison now also covers the ambiguity-kind set alongside the
seven role fields (the compare.py comparator shape), since
decisions.md#P6 (derek73#405) records that this design's own repaired
failure was exactly an ambiguity asymmetry between two writings of
one name; all 96 pairs agree on both. A second, non-parametrized test
sweeps the entire shipped particle vocabulary in the minimal pair and
asserts the disagreeing set is exactly {do, mc, vd} -- a recorded
literal per mechanisms.md#RECORDED-ROSTERS, guarded by a
mechanisms.md#VOCABULARY-OVERLAP-AS-PRECONDITION precondition that
the fixture family/given is not itself particle vocabulary -- and
resolves decisions.md#two-input-invariants' "Open: derek73#469" marker for
an executable form.

Form 5 is absent by construction -- no comma format puts the given
name last (derek73#469).

Co-Authored-By: Claude Fable 5 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Mesnil Garcia de strands the particle under FAMILY_FIRST but folds it under FAMILY_FIRST_GIVEN_LAST

1 participant