Skip to content

xorq stats: a stat taking min/max as float fails on int columns since D8, with no DAG warning #1051

Open

Description

@paddymul

Problem

Since dac617b (ADR-001 D8, in #1040), xorq min and max return ColumnValue and keep the column's type (xorq_stats_v2.py:164, :169). Before, they were cast to float64. A stat that declares them as float now gets an int on int columns, and the input check at stat_pipeline.py:121 rejects it because isinstance(1, float) is False. Float columns still work.

The DAG normally warns when a float consumer reads a ColumnValue provider. Here it doesn't: min and max are in XorqStatPipeline.EXTERNAL_KEYS (xorq_stat_pipeline.py:178), and build_typed_dag skips the type check for external keys (typed_dag.py:58). The first sign is a runtime error on each int column.

@stat()
def spread(min: float, max: float) -> float:
    return max - min

table = xo.memtable(pa.table({"i": [1, 5, 9], "f": [1.0, 5.0, 9.0]}), name="t")
sd, errs = XorqStatPipeline(XORQ_STATS_V2 + [spread], unit_test=False).process_table(table)
sd["f"]["spread"]  # 8.0
errs               # 'spread' expects 'min' as float, but got int: 1  (column i)

No stat in buckaroo, tallyman or pydata-app declares min/max as float. Tallyman project stats can't take them at all, since the loader builds a one-parameter compute(col). Only a stat written against the Python API is affected.

Suggested fix

Any of:

  • A CHANGELOG note that min/max keep the column's type, so a stat reading them should declare ColumnValue or Any.
  • Let the input check accept an int (not a bool) where float is declared, as type checkers do under PEP 484's numeric tower.
  • Type-check an external key in build_typed_dag when a stat also provides it, so the mismatch warns at build time.

Context

Found during review of #1040. Line numbers are from the PR branch at a8df9b9.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions