Should we change PEP 484 to disable implicit Optional when default = None?

@ddfisher

PEP 484 currently says:

An optional type is also automatically assumed when the default value is
None, for example::
  def handle_employee(e: Employee = None): ...
This is equivalent to::
  def handle_employee(e: Optional[Employee] = None) -> None: ...

This was intended as saving some typing in a common case, but I've received strong feedback from some quarters that this is not consistent and a bad idea. There are other places where None is allowed but none of them automatically add Optional (to the contrary).

So far it hasn't mattered much for mypy users because mypy doesn't have Optional support, but that's soon going to change (the --strict-optional flag is becoming more reliable) so if we're going to change this, now would be a good time. Thoughts? If we don't change this soon it will probably be too late.

I'm for this change. If I recall correctly, this is the only place where the PEP suggests typecheckers understand types to be something other than what's written as the type, so I think this change would be a nice simplification.

I've also started to notice this cause a bit of user confusion. See python/typeshed#500, for example, for all the arguments to builtin functions that were accidentally marked Optional due to this.

I have seen some confusions between optional arguments and optional types, and the convention in question amplifies such confusions. Therefore I am in favour of removing it.

OK, let's bring this up on python-dev once Python 3.6 beta 1 is released
(i.e. after next weekend).

On Sat, Sep 3, 2016 at 7:32 AM, Ivan Levkivskyi [email protected]
wrote:

I have seen some confusions between optional arguments and optional types,
and the convention in question amplifies such confusions. Therefore I am in
favour of removing it.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
#275 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/ACwrMpUgUvnteOgZfJvI0g_D50N3TOthks5qmYTugaJpZM4J0K87
.

--Guido van Rossum (python.org/~guido)

IMO most of the confusion comes from the name being wrong. An argument is «optional» when the parameter has a default in python. An «optional» type is a type that you can choose to add or not (as in gradual typing). If we had Nullable[T] instead of Optional many of these confusions wold go away (and it's the name used mostly everywhere else, other than Maybe T in Haskell).

Other than that, there is an inconsistency that may require a change (I just wanted to clarify that the confusions shouldn't be part of the argument)

@dmoisset

@dmoisset: I agree that Nullable or Nonable (1 char shorter) or Maybe (3 chars shorter) would be better solutions than Optional.

On the other hand, even the short Maybe is cumbersome. Nonable parameters are a frequent case and I believe the =None is sufficiently explicit to warrant introducing an additional PEP rule to cope with it, even if that rule has the character of an exception.

I like that the Optional[] can be omitted if a None default is provided. It is less typing, reads naturally (to me) and is unambiguous.
If users want to use Optional they can since Optional[Optional[T]] == Optional[T].
As syntactic sugar goes, it seems quite sweet :)

OOI, which quarters?

I haven't had enough experience with using mypy strict optional checking to decide whether the extra verbosity would become awkward. Maybe we should wait until we have a substantial codebase that can be type checked using strict optional checking.

Why do you use the word "strict"? That suggests that there is something "loose" about the syntactic sugar. There isn't; it is precisely defined.

Strict optional checking means that mypy treats None as a regular type, and mypy enforces that you add guards like if x: foo(x + 1) for values with an Optional[] type. Without strict optional checking, None is implicitly compatible with every type, kind of like null in Java, and an Optional[X] type is equivalent to just X. It has nothing to do with syntax.

So how is it relevant to default values for parameters making the type Optional?

The reason for adopting the syntactic sugar was the worry that otherwise type annotations would be too verbose and thus maybe also harder to read. I think that the main arguments for not having the syntactic sugar are consistency and conceptual simplicity.

If we have a codebase that uses Optional[] consistently, which can be enforced by the strict optional mode of mypy, we could evaluate, at least for that codebase, whether the arguments hold water. We can get a realistic of estimate of the actual verbosity of annotating code with or without the syntactic sugar. We can perhaps also get a better understanding of whether the inconsistency is an actual problem for people who are working on the code.

It may all boil down to personal preference, but having some empirical data could be useful for the discussion. Also, it doesn't seem to me that this is an urgent thing to resolve, as this doesn't require major typing implementation changes (though get_type_hints may be affected) and it's going to be easy for a static checker to support both modes via a configuration option. I don't see a need to rush the decision, and we can first wait for tools that can perform useful "strict optional" checking to be built.

Whether it is easier to read is, as you say, personal preference, which is rarely influenced by empirical evidence 😄

Happy to wait and see what you come up with.

@JukkaL

@JukkaL I ma not sure what kinds of inconsistency you mean, but this one

x = None # type: int # here the type stays int, unlike for function params

will go away after PEP 526. I think there it is better to assume implicit Optional or not for both of these

def f(x: int = None): ...
x: int = None

but not separately for each of these.

x = None # type: int

Ah, that thing. I think that should never have been added. It is just self-contradictory.

@ilevkivskyi

@ilevkivskyi There's still the question about annotating instance variables with mutable types in the class body in pre-3.6 code. This is what mypy users can write today:

class A:
    x = None  # type: List[int]  # Can't use x = [] (list is mutable) and don't want Optional

    def __init__(self) -> None:
        self.x = []

This is a common use case and I think we need to have a reasonable way to expressing this without PEP 526 syntax.

I assume we will keep a flag in mypy, but will just invert the default. Right?

I assume we will keep a flag in mypy, but will just invert the default. Right?

Yes we should do that.

PEP 484: Do not require type checkers to treat a None default special…

Closed by python/peps#689.

What if I'm annotating function that does not accept None but has it as a marker of missed argument? E.g.:

def foo(p: int = None):
    if p is None:
        print("p not specified")

Should I use some another value to achieve this? If so, should it have type int or other one? E.g.:

class _MISSING:
    pass

def f(p: int = _MISSING):
    ...

I'd say it's not desirable to put a default value if you consider that default value is not an acceptable value. In this case, just don't put a default value. If you don't pass anything, mypy will tell you and you'll get a type error, and if you explicitely pass None, mypy will tell you it's wrong.

def foo(p: int):
    # in the function body, don't try to defend against p being not an int
    ...

What about other cases such as dataclasses.field? Default value for default and default_factory has incompatible type. See stub and specification.

mypy: Enable no_implicit_optional

PR (#1)

@ewjoachim

@ewjoachim there are these different scenarios:

# None is not one of possible values, func1 can't be called with explicit `param = None`
def func1(param: Class = None):
   param = param or Class("somethin")

# None is one of the possible values, funct2 can be called with explicit `param = None`
def func2(param: Optional[Class] = None):
   pass

# None is one of the possible values, funct3 requires param and can be called with explicit `param = None`
def func3(param: Optional[Class]):
   pass

# None is one of the possible values, funct3 can be called with `param = None`
def func4(param: Optional[Class] = obj):
   pass

i.e. default values has nothing to do with type annotation i.e. public contract of a function

Not sure I get your point, and I don't agree with "default values has nothing to do with type annotation":

This function has wrong annotations which we can tell BECAUSE the annotations and the default value are clashing:

def func(param: int = "hello"):
   pass

I was answering to:

What if I'm annotating function that does not accept None but has it as a marker of missed argument? E.g.:

def foo(p: int = None):
    if p is None:
        print("p not specified")

and I understood "does not accept None" as "if you call the function with None (or without an argument), the call is going to be rejected. And I stand by my (... 2-year-old-)opinion: if a function is written to fail if an argument is missing, it's best to not have that argument have a default value.

If you want a default value and you want to be able to detect that the default value was used, either use : Optional[...] = None if None doesn't already have another meaning or do something like:

missing = object()
def func(a: Any = missing):
	if a is missing:
		print("default")

or

class Missing:
	pass
missing = Missing()
def func(json: Union[None, list, dict, str, int, float, Missing] = missing):
	if json is missing:
		print("default")

Maybe I missed an important point ?

Fix no-implicit-optional change in mypy/pep484

@ewjoachim

Instead of treating this as a special case, it could also be understood as implicit type widening, where the default value is allowed to widen the type, e.g. from str to str | None. This works only with type inference, but PEP 484 does recommend:

Type checkers are expected to attempt to infer as much information as necessary.

So arguably, it's not really a special case, but a general policy of widening the type based on a default when the typing tools has inferred the type.

This would also work for the case of a marker object, i.e. missing in @ewjoachim's example.

While I can see some strange pitfalls such as a: int = "foo" this should perhaps not win over the arguably clean syntax offered by a: str = None (assuming that the general policy of implicit type widening would get adopted).

Navigation Menu

Repository navigation

Should we change PEP 484 to disable implicit Optional when default = None? #275

33 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Should we change PEP 484 to disable implicit Optional when default = None? #275

Description

Activity

ddfisher commented on Sep 3, 2016

ilevkivskyi commented on Sep 3, 2016

gvanrossum commented on Sep 3, 2016

dmoisset commented on Sep 5, 2016

prechelt commented on Sep 5, 2016

markshannon commented on Sep 5, 2016

JukkaL commented on Sep 5, 2016

markshannon commented on Sep 6, 2016

JukkaL commented on Sep 6, 2016

markshannon commented on Sep 6, 2016

JukkaL commented on Sep 6, 2016

markshannon commented on Sep 6, 2016

ilevkivskyi commented on Sep 6, 2016

markshannon commented on Sep 6, 2016

JukkaL commented on Sep 6, 2016

33 remaining items

ilevkivskyi commented on Jul 1, 2018

gvanrossum commented on Jul 2, 2018

gvanrossum commented on Jul 10, 2018

sproshev commented on Jan 9, 2019

ewjoachim commented on Jan 9, 2019

sproshev commented on Jan 11, 2019

keu commented on Apr 11, 2021

ewjoachim commented on Apr 12, 2021

Chador-safari-web-inspec commented on Feb 4, 2025

malthe commented on Sep 26, 2025

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions