[Feature Request]: Support TypedDict hint in Apache Beam.

What would you like to happen?

CoGroupByKey produces Dict[K, Union[...]], which is narrower than TypedDict. If the user follows the CoGroupByKey with a ParDo where the function takes TypeDict as input, e.g. ParDo(fn(element: TypedDict)), then the type check in Beam fails.

Users having types like:

myTypeDict = TypedDict(
    'myTypeDict', {
        'field_1': List[Field1Type],
        'field_2': List[Field2Type],
    })

are using less specific hints like myLessSpecificDict = Dict(str, Union[List[Field1Type], List[Field2Type]]) in order to satisfy beams typing.

We could have a first-class typehint TypeDict, which may be easier once we deprecate Python 3.7.

Issue Priority

Priority: 3 (nice-to-have improvement)

Issue Components

Update the issue title?

Done, thanks.

Why have specific code for TypedDict when you can use the more generic Mapping, which accepts not only TypedDict, but also any kind of dict?

~~Note: there’s a typo in the issue title: it should be "TypedDict", not "TypeDict".~~

The value of TypedDict over Mapping is if the different keys are expected to have different values. For example, let's say I did the following:

 to_group = {
  'ints': pcoll1,  # PCollection of (str, int)
  'strs': pcoll2,  # PCollection of (str, str)
}
grouped = to_group | beam.CoGroupByKey()
something = grouped | beam.MapTuple(map_grouped)

If using Mapping, the method would have to start:

def map_grouped(key: str, values: Mapping[str, Iterable[str | int]]):
  ints = values['ints']  # type: Iterable[int]
  strs = values['strs']  # type: Iterable[str]
  ...

since the type checker can't tell which key corresponds with which value type.

If I could instead have a TypedDict, it could be a bit cleaner:

class GroupedValues(TypedDict):
  ints: Iterable[int]
  strs: Iterable[str]

def map_grouped(key: str, values: GroupedValues):
  ints = values['ints']
  strs = values['strs']

and the type checker needs no additional help.

@jrmccluskey

cc: @jrmccluskey

Navigation Menu

Repository navigation

[Feature Request]: Support TypedDict hint in Apache Beam. #24704

What would you like to happen?

Issue Priority

Issue Components

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[Feature Request]: Support TypedDict hint in Apache Beam. #24704

Description

What would you like to happen?

Issue Priority

Issue Components

Activity

AnandInguva commented on Dec 19, 2022

tvalentyn commented on Dec 20, 2022

bfontaine commented on Mar 24, 2023

aaronsarna commented on Jul 24, 2023

tvalentyn commented on Jul 24, 2023

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions