plumbing: format/config, Reject unrepresentable subsection names when encoding - #2359

Problem

git-config(1): subsection names "can contain any characters except newline and the null byte."

The encoder escapes only " and \, so either of those two characters is written raw into the section header and the result cannot be read back — by go-git or by git:

m := config.NewModules()
m.Unmarshal([]byte("[submodule \"a\\\nb\"]\n\tpath = p\n\turl = https://example.com/x.git\n"))

out, _ := m.Marshal()
// [submodule "a
// b"]
//     path = p
//     url = https://example.com/x.git

config.NewModules().Unmarshal(out) // 1:12: string not terminated

git config --file … --list on that same output: fatal: bad config line 1. With a NUL instead of the newline, go-git reports illegal character NUL and git silently truncates the name to a.

This is reachable from .gitmodules: submodule names come from the repository being operated on, and config.Modules.Marshal re-encodes them.

Fix

Return an error instead of writing a file nothing can parse.

SetConfig now marshals before it opens the config writer. The writer truncates the file, so with the old order that error left the repository with an empty config.

Escaping is not an option here. git drops a backslash before any character other than " and \, so \n reads back as the letter n and would quietly rename the subsection. git refuses to create such a key at all (error: invalid key (newline)), so the encoder does the same.

Tests

Two regression tests in encoder_test.go. Withdrawing the fix while keeping them reproduces the failure on main:

--- FAIL: TestEncoderSuite/TestEncodeNeverWritesUnreadableSubsectionHeader
    Error:    Received unexpected error: 1:12: string not terminated
    Messages: encoder wrote a config that the decoder cannot read:
              "[submodule \"a\nb\"]\n\turl = https://example.com/x.git\n"

TestSetConfigKeepsExistingConfigWhenEncodingFails covers the save path: with the old order the config is read back as "".

Rebased on main. go test ./... passes apart from the known plumbing/format/gitignore TestConformanceSuite failure (#2317). The root package and plumbing/transport/http hit network timeouts on my machine when the whole suite runs at once, and both pass when run on their own. golangci-lint reports 0 issues for both packages. Behaviour checked against git 2.50.1.

Scope

Section names and option keys have the same gap — they are written with no escaping or validation at all, so a space, tab, newline, ], " or \ in either also yields an unreadable file. Those aren't reachable from repository data the way subsection names are, so I left them out to keep this a focused fix; happy to extend it if you'd prefer them covered here.

Found with a decode/encode/decode round-trip fuzz target, not included in this PR.

AI disclosure (per AI_POLICY.md): this change was produced with an AI agent, Claude Opus 5 and Opus 5.5 via Claude Code, and both commits carry the Assisted-by: trailer. The reproduction, the git 2.50.1 comparisons, the baseline/after suite runs and the withdraw-the-fix check above were all actually executed, and the output quoted is real.