Skip to content

fix(muya): treat CJK as punctuation for strong/em flanking (#4307) - #4401

Merged
Jocs merged 2 commits into
developfrom
fix/muya-cjk-strong-flanking
Jun 8, 2026
Merged

fix(muya): treat CJK as punctuation for strong/em flanking (#4307)#4401
Jocs merged 2 commits into
developfrom
fix/muya-cjk-strong-flanking

Conversation

@Jocs

@Jocs Jocs commented Jun 8, 2026

Copy link
Copy Markdown
Member

The gap (#4307)

Strong/em delimited with ** directly against a CJK character whose inner
content is punctuation-bounded did not bold in muya (@muyajs/core),
even though the legacy packages/muyajs engine bolded it. Examples that
should bold but didn't:

  • 例子例子**"加粗"**例子例子
  • 日本語**(強調)**日本語
  • 한국어**[강조]**한국어
  • 𠀀𠀁**"加粗"**𠀀𠀁 (non-BMP, CJK Ext-B)

Root cause. CommonMark §6.2 classifies CJK ideographs / Hangul / Kana as
"other" (Lo, Letter) — neither whitespace nor punctuation. So a ** run
wrapped by CJK with punctuation-bounded content (a quote/paren/bracket on the
inner side) is neither left- nor right-flanking, and the delimiters stay
literal. This is spec-conformant (GitHub behaves the same), but CJK scripts
don't use inter-word spaces, so the rule denies emphasis to virtually any CJK
paragraph that wraps ** with punctuation. Typora, VSCode markdownlint,
Joplin — and the legacy muyajs engine MarkText shipped — all widen the
flanking check so CJK counts as a boundary.

The two paths fixed

muya has two inline-tokenization paths, and both now carry the same
additive CJK-as-punctuation widening:

  1. Static / export path (marked@16) —
    state/renderToStaticHTML → utils/marked/getHighlightHtml, plus
    getClipboardHtml, run marked.parse, whose built-in emStrong
    tokenizer applies CommonMark flanking literally. New module
    utils/marked/extensions/cjkEmStrong.ts registers a tokenizer.emStrong
    override: a faithful copy of [email protected]'s emStrong body that swaps in
    CJK-widened versions of marked's own flanking regexes
    (emStrongLDelim, emStrongRDelimAst, emStrongRDelimUnd,
    punctuation, and the unicodeAlphaNumeric "other" rule) for the
    duration of the call, then restores them in a finally so the shared
    tokenizer rules are never left mutated. CJK is folded into the punctuation
    class and removed from the alphanumeric class. No fork of marked.

  2. Live editor path (inlineRenderer) — muya's own inline lexer
    (inlineRenderer/lexer.ts + utils.ts). The CJK widening is added to
    canOpenEmphasis / canCloseEmphasis in inlineRenderer/utils.ts,
    matching the legacy packages/muyajs/lib/parser/utils.js exactly,
    including full code-point reading (lastCodePointChar /
    codePointCharAt) so the non-BMP CJK Ext-B surrogate-pair branch is live.

Both paths use the same ranges and agree on every test case.

CJK ranges (matching legacy CJK_REG)

Range Block
U+3040–U+30FF Hiragana + Katakana
U+3400–U+4DBF CJK Unified Ideographs Ext-A
U+4E00–U+9FFF CJK Unified Ideographs
U+F900–U+FAFF CJK Compatibility Ideographs
U+AC00–U+D7AF Hangul Syllables
U+FF66–U+FF9D Halfwidth Katakana
U+20000–U+2A6DF CJK Unified Ideographs Ext-B (non-BMP)

Conformance — no regression

The widening is additive: it only ever lets emphasis open/close where
CommonMark refused (around CJK), never the reverse. The CommonMark 0.31 + GFM
conformance suites are unchanged:

  • test:spec: 1347 → 1347 passed, expected-failures.json unchanged,
    zero unexpected passes (the spec corpus has no CJK at emphasis
    boundaries, so nothing flips).

Tests

This branch promotes the it.fails placeholders from #4399
(state/__tests__/strongCjkFlanking.spec.ts) to passing it cases, now that
both paths bold them, and adds:

Verify (all green)

pnpm -C packages/muya lint (0 errors) · lint:types (clean) ·
check-circular (clean) · test (511 passed) · test:spec (1347 passed).

🤖 Generated with Claude Code

@Jocs @claude
fix(muya): treat CJK as punctuation for strong/em flanking (#4307)
f238787 
Strong/em delimited with `**` directly against a CJK character whose inner
content is punctuation-bounded did not bold in muya, e.g.
`例子例子**"加粗"**例子例子`, `日本語**(強調)**日本語`, `한국어**[강조]**한국어`,
and the non-BMP `𠀀𠀁**"加粗"**𠀀𠀁`. The legacy muyajs engine bolded these.

CommonMark §6.2 classifies CJK ideographs / Hangul / Kana as "other" (Lo),
neither whitespace nor punctuation, so a `**` run wrapped by CJK with
punctuation-bounded content is not left/right-flanking and stays literal.
muya has TWO inline-tokenization paths and both carry the same CJK-as-
punctuation widening the legacy engine shipped:

- Static / export path (marked@16): a `cjkEmStrong` tokenizer override that
  rebuilds marked's emStrong flanking regexes with CJK folded into the
  punctuation class and removed from the alphanumeric class, registered in
  getHighlightHtml and getClipboardHtml. Faithful copy of marked's emStrong
  body; rules are swapped in/out per-call so the shared tokenizer rules are
  never left mutated.
- Live editor path (inlineRenderer): CJK widening added to
  canOpen/canCloseEmphasis in inlineRenderer/utils.ts, plus full code-point
  reading so the non-BMP CJK Ext-B surrogate-pair branch is live.

CJK ranges (matching legacy CJK_REG): Hiragana+Katakana U+3040–U+30FF, CJK
Ext-A U+3400–U+4DBF, CJK Unified U+4E00–U+9FFF, CJK Compatibility
U+F900–U+FAFF, Hangul Syllables U+AC00–U+D7AF, Halfwidth Katakana
U+FF66–U+FF9D, and CJK Ext-B U+20000–U+2A6DF (non-BMP).

The widening is additive — it never bolds anything CommonMark accepts as
non-emphasis — so the CommonMark 0.31 + GFM conformance suites are unchanged
(1347/1347, no unexpected passes). Promotes the #4399 `it.fails` placeholders
to passing `it` cases and adds live-editor-path coverage plus negative cases.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Copilot AI review requested due to automatic review settings June 8, 2026 08:55
Copilot AI reviewed Jun 8, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes a MarkText-specific regression in @muyajs/core where strong/emphasis delimiters (** / *) directly adjacent to CJK characters fail to open/close when the inner content is punctuation-bounded (e.g. 中文**"加粗"**中文). It restores the legacy muyajs behavior by widening the emphasis “flanking” boundary rules so CJK characters are treated as punctuation boundaries, consistently across both the static/export marked pipeline and the live editor inline lexer.

Changes:

  • Add a marked tokenizer extension that overrides emStrong with CJK-widened flanking rules for static/export and clipboard HTML rendering.
  • Update the live editor inline flanking checks to treat CJK as punctuation, with correct non-BMP (surrogate-pair) handling.
  • Promote and expand regression tests to cover both render paths, including negative cases.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
packages/muya/src/utils/marked/getHighlightHtml.ts Registers the new CJK-aware marked emStrong extension for static/export HTML.
packages/muya/src/utils/marked/getClipboardHtml.ts Registers the same CJK-aware extension to keep clipboard HTML consistent with export rendering.
packages/muya/src/utils/marked/extensions/cjkEmStrong.ts Implements a marked Tokenizer.emStrong override that treats CJK as punctuation for flanking.
packages/muya/src/state/tests/strongCjkFlanking.spec.ts Promotes prior expected-fail tests and adds coverage for both static and live-editor tokenization paths.
packages/muya/src/inlineRenderer/utils.ts Widens canOpenEmphasis/canCloseEmphasis to accept CJK as punctuation boundaries and correctly reads full code points.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread packages/muya/src/state/__tests__/strongCjkFlanking.spec.ts Outdated
];
for (const src of NEGATIVE_CASES) {
it(`does not bold: ${src}`, () => {
expect(rendersStrong(src)).toBe(false);

@Jocs Jocs Jun 8, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch — done in 7b0b312. Added a rendersEm helper mirroring rendersStrong and strengthened the static/export-path negative cases to assert NEITHER <strong> NOR <em> is rendered, so a flanking regression that surfaces as unexpected italics now fails the test. (Verified all three negative cases render as plain literal text — no <strong>/<em>.)

The live editor (inlineRenderer) negative cases already cover both: tokenizesEmphasis returns true for either a strong or em token, so its expect(...).toBe(false) already rejects both tags — only the static path had the <strong>-only gap.

Green: lint (0 errors), lint:types, check-circular, test (511 passed), test:spec (1347 passed).

@Jocs @claude
test(muya): assert negative CJK flanking cases emit neither strong no…
7b0b312 
…r em

The static/export-path negative cases in strongCjkFlanking.spec.ts only
checked for <strong> via rendersStrong. Since #4307 widens the
emphasis/strong flanking logic, a regression could surface as unexpected
<em> output while still passing a <strong>-only assertion. Add a rendersEm
helper mirroring rendersStrong and assert NEITHER tag is produced for the
negative cases.

The live editor path already covers this: tokenizesEmphasis returns true
for either a strong or em token, so its negative .toBe(false) already
rejects both. Only the static path needed strengthening.

Addresses Copilot review on #4401.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
@Jocs
Jocs merged commit 79a2926 into develop Jun 8, 2026
9 checks passed
@Jocs
Jocs deleted the fix/muya-cjk-strong-flanking branch June 8, 2026 09:52
@Jocs Jocs mentioned this pull request Jun 9, 2026
Merged
Jocs added a commit that referenced this pull request Jun 9, 2026
@Jocs @claude
test(desktop): remove redundant muyajs-engine unit specs (Phase F) (#…
b4428b3 
…4422)

These 8 specs import the legacy `muya/lib` (muyajs) alias and exercise
engine-level markdown / slug / word-extraction behavior that @muyajs/core
now owns and covers with its own suites:
  - markdown-basic / list-indentation / nested-mixed-lists / export-markdown
    -> muya commonmark.spec + gfm.spec + roundTrip.spec conformance
  - markdown-footnotes -> muya footnote.spec / footnoteHtml.spec
  - markdown-strong-cjk -> muya strongCjkFlanking.spec (#4401)
  - slugger -> muya generateGithubSlug (getTOC.spec)
  - extract-word -> muya replaceCurrentWord.spec + current-word logic

They test code the app no longer uses (the muyajs engine) and would break
when packages/muyajs is removed in Phase H. Desktop-specific unit specs
(i18n, match-electron-accelerator, native-theme) are kept.

Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot
Copilot review effort
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@Jocs