There was a discussion about the larger issue of code being ”misleadingly different from what it looks like", and the general consensus was that this should be solved in editors, linters and code review tools, rather than in Python. IMO it was this thread but might need more digging.
That doesn't mean handling of \0 can't be changed. Or of control characters in general: see the informational PEP 672 for some more details.
Perhaps all unusual control characters should be banned? Even in strings? With a warning for 2 releases, according to the backwards compatibility policy – waf would need to switch to e.g. base64.

that was the rationale for making this public, but it's different than those as it's a mishandling by the parser rather than a quirk of how unicode displays

Ugh, this is freakin amazing.

C:\Users\user>more t.py

"""
How do i print 'Hello World' in Python?

Here is how:
print('Hello World')
"""

And run it:

C:\Users\user>py t.py
Hello World

open('t.py', 'w').write("\0'''\0How do i print \'Hello World\' in Python?\n\0Here is how:\nprint(\'Hello World\')\0'''")

ideally I would want the interpreter to reject files containing null bytes as a SyntaxError (and update the ast.parse error to a SyntaxError as well)

FWIW I'd be in favor of that behavior change as a Feature. I wouldn't backport it as a bug though.

there's kinda 3 things I think:

1. convert ValueError => SyntaxError in ast.parse (probably 3.12+)
2. treat source files containing null bytes as SyntaxError (probably also 3.12+)
3. fix the thing considering it as a c string by using (char*, size_t) or whatever function (probably backportable?)

does it make sense to pursue these three things and separately?

there's kinda 3 things I think:

1. convert `ValueError` => `SyntaxError` in `ast.parse` (probably 3.12+)

Agreed, it's changing behavior, so can't backport. I like SyntaxError for this situation. The root exception isn't in ast.parse(), it's in compile(), probably even deeper.

2. treat source files containing null bytes as `SyntaxError` (probably also 3.12+)

That's the biggest hole, should probably do this first.

3. fix the thing considering it as a c string by using `(char*, size_t)` or whatever function (probably backportable?)

Depends on where that fix is (can you tell I haven't looked at the source code yet? :-). If any of the affected functions are public it's going to be more difficult.

does it make sense to pursue these three things and separately?

Likely.

2. treat source files containing null bytes as `SyntaxError` (probably also 3.12+)

That's the biggest hole, should probably do this first.

Be careful about this one. Python pre-pended to raw binary data for the executed Python to locate within the file and use (embedded zip or other data) is a common idiom that must continue to work.

Be careful about this one. Python pre-pended to raw binary data for the executed Python to locate within the file and use (embedded zip or other data) is a common idiom that must continue to work.

Are you sure that works? Unlike Unix shells, Python parses the entire source file before executing any code. How would you get Python to ignore a blob of arbitrary binary data embedded in the source code, even if \0 is accepted? If the blob contains \n characters you can't hide it behind a # comment. I suppose you could prefix it with a """ quote, if you can arrange for the file to also end in """, and you're lucky that the blob doesn't contain embedded """ sequences.

But if I had to do something like that I'd probably just embed the Python code in a bash script as a "here" document and end the bash script with exit.

Oh you're right, I guess what I've seen do that is a bash+python+data hybrid monster.

Okay then we should be safe banning \0 in files starting with 3.12.

I suspect one could probably hack something together like this unfortunately

# coding: latin1
with open(__file__, 'rb') as f:
    contents = f.read().split(b'### BINARY\n')[1]
GARB = '''\
### BINARY
(actual binary here)
### BINARY
'''

You'd still have to arrange for the actual binary not to contain the sequence ''' -- because that would end the string started at GARB =. I don't see how setting the coding to Latin-1 changes matters. (Honestly it seems you're making the same mistake as Greg.)

latin1 is to prevent a decoding error while parsing the source -- I linked an in-the-wild example of such a file in the original post

Okay, that's impressive. It looks like the blob is lightly encoded -- \n and \r are encoded using #. and #&. (I'm curious what they'd do if the blob contains one of those sequences, there doesn't seem to be a way to quote them.)

The proposed change in 3.12 will break them, but they have version checks and they can just cope with it, I don't think we need to preserve this machine-dependent quirk forever. The code looks like it had to deal with various other versioning issues already (e.g. it tries to handle Python 2 and 3!).

Closing this. Starting with Python 3.12, NUL bytes won't be allowed in source code read from files. We're not backporting this since this could be considered a feature by some.

Starting with Python 3.12, NUL bytes won't be allowed in source code read from files

It was changed by the commit aab01e3: PR #97594.

Just for reference: The problem is already mentioned in https://peps.python.org/pep-0672/#control-characters

And Pylint checks for null characters already.