Is your feature request related to a problem?
Current filter compilation logic does not apply filters intuitively and is unsafe.
Claude review 1:
compile_sql_filter then has explicit None-aware combinators:
- And(left, right): if either side is None, returns the other side. Both None → None. → unknown leaf behaves like TRUE (no
constraint).
- Or(left, right): same code path — if either side is None, returns the other side. → unknown leaf behaves like FALSE (sibling
alone carries the disjunction).
- Not(inner): if inner is None, returns None. → NOT unknown again falls out as "no constraint".
If the entire expression collapses to None, compile_sql_filter returns None and the caller (e.g. _apply_episode_filter at
episode_sqlalchemy_store.py:271-274) treats that as "no filter clause" and the WHERE is omitted.
Net effect / sharp edges
- An unknown leaf inside an AND widens the result (drops the constraint).
- An unknown leaf inside an OR narrows the result — A OR unknown becomes just A, not TRUE. So unknown is implicitly "true under
AND, false under OR" — semantically inconsistent but matches the intuition that "the field doesn't exist, so no row satisfies it."
- NOT unknown evaluates to "no constraint" too (because ~None would crash, the code short-circuits to None), which contradicts the
OR reading.
- The only feedback to the caller is a logger warning; the API does not surface a typed error for misspelled fields.
Claude review 2:
❯ does the logic make sense under a filtering system? Is it consistent with intuition?
⏺ Short answer: no — it's neither algebraically consistent nor a defensible filter semantics.
Why it isn't algebraically consistent
There's no single truth assignment for "unknown" that reproduces all the cases. Trace it:
┌────────────────────┬─────────────────┬────────────────┬──────────────────────────┐
│ Tree │ Compiled │ Effective rows │ Implied truth of unknown │
├────────────────────┼─────────────────┼────────────────┼──────────────────────────┤
│ unknown │ None → no WHERE │ all rows │ TRUE │
├────────────────────┼─────────────────┼────────────────┼──────────────────────────┤
│ A AND unknown │ A │ rows where A │ TRUE (A ∧ T = A) │
├────────────────────┼─────────────────┼────────────────┼──────────────────────────┤
│ A OR unknown │ A │ rows where A │ FALSE (A ∨ F = A) │
├────────────────────┼─────────────────┼────────────────┼──────────────────────────┤
│ NOT unknown │ None → no WHERE │ all rows │ FALSE (¬F = T) │
├────────────────────┼─────────────────┼────────────────┼──────────────────────────┤
│ A AND NOT unknown │ A │ rows where A │ FALSE │
├────────────────────┼─────────────────┼────────────────┼──────────────────────────┤
│ unknown OR unknown │ None → no WHERE │ all rows │ T or F (either) │
└────────────────────┴─────────────────┴────────────────┴──────────────────────────┘
It flips between TRUE-like and FALSE-like depending on whether a sibling survived. It also isn't 3-valued (NULL) logic: in SQL,
WHERE UNKNOWN excludes the row, so A AND UNKNOWN returns nothing — the opposite of what this code does.
Where it violates intuition
- A user expects misspellings to be loudly wrong (error) or uniformly conservative (match no rows). This code instead silently
shifts the meaning of the whole expression depending on tree shape.
- The bare-unknown case matches all rows. Intuitively, if I write WHERE bogus = 'x', I expect zero rows or an error, not "ignore
the predicate."
- NOT unknown returning "no constraint" is especially surprising: I asked to exclude something and got no exclusion.
Concrete failure modes
These aren't hypothetical — they're sharp:
1. Auth/tenant scoping silently opens up. If a caller filters tenant_id = X AND role = admin and typos tennant_id, the predicate
collapses to role = admin across all tenants. If both are unknown, every row matches. Any filter used as an access-control gate is
unsafe under this policy.
2. Negation inverts intent. NOT internal_only with internal_only misspelled returns all rows including internal ones.
3. OR narrows instead of widens. flagged OR archived with archived misspelled returns only flagged rows — the user's "either"
became "only."
4. Schema drift hides. Renaming a column makes existing filters silently match different row sets. There's only a logger warning at
WARN level — easy to miss in production.
What "consistent" filter semantics look like
Pick one and stick to it:
- Strict (the segment-store branch): unknown → ValueError. Fails fast, surfaces typos at the API boundary. Best default for a typed
filter language.
- Safe-deny: unknown leaf compiles to false(), and NOT unknown to true(). Then unknown matches nothing, A AND unknown matches
nothing, A OR unknown matches A. Defensible and predictable. Used by some search DSLs that need to tolerate evolving schemas
without breaking clients.
- 3-valued (SQL NULL): harder to implement correctly but matches DB intuition; rarely worth it for a filter compiler.
The current "drop the leaf and re-knit the tree" rule is none of these. It looks like it was written to make the tree compile under
partial schema knowledge, but it does so by changing the semantics of the surviving expression in a non-uniform way.
Recommendation
For a filtering system that's also used for authorization-shaped queries (tenant scoping, role gates, soft-delete filters), the
strict policy is the only one I'd defend. If permissive/forward-compatible behavior is genuinely needed for a subset of fields,
gate that explicitly — e.g. resolver returns ("ignore", False) and the compiler maps that to true() — so the call site opts in and
the algebra stays consistent.
Describe the solution you'd like
#1291
Filter Compilation: Design
Problems with the previous design
-
Unknown fields had no consistent boolean meaning. The resolver could signal "I don't recognize this name" by returning a sentinel, and the compiler would drop the leaf and re-knit the surrounding boolean tree. The result was that the implicit truth value of an unrecognized leaf depended on its parent — TRUE-shaped under AND, FALSE-shaped under OR, "no constraint" at the top level or under NOT. No single truth assignment to "unknown" reproduced all the cases, and the behavior matched neither classical two-valued logic nor SQL's three-valued logic.
-
Typos and schema drift were invisible. Unrecognized fields produced no observable error at the API boundary. Misspelled names and renamed columns compiled successfully and silently changed the meaning of the surrounding query.
-
Storage encoding was a binary flag. Resolvers returned (column, is_json), which conflated "is this a JSON path" with "every encoding the system might want." Adding a third encoding required either changing every resolver's signature or smuggling extra information through the column expression.
-
Boolean combinators carried implicit conventions. Sentinel-handling for "leaf disappeared" was pushed into And, Or, and Not, so the meaning of a subexpression depended on what was around it. The algebra was hard to reason about and the implementation hard to extend.
New behaviors
-
Unknown fields fail loud. The resolver returns either a resolved field (column expression plus encoding tag) or raises a typed error naming the offending field. The compiler does not catch the error. Misspellings, removed columns, and schema drift all surface before the query runs. Programmer errors are routed through the exception channel; data conditions are not.
-
Boolean combinators behave as classical two-valued operators. And, Or, and Not recurse over their children with no sentinel propagation, leaf-dropping, or tree re-knitting. The truth value of any subexpression depends only on its own structure and its operands' values, never on what's around it.
-
Encoding is a closed tag, not a boolean. The resolver returns one of a small enumeration of encoding tags alongside the column. Adding a new encoding is an additive change: define a new tag value and a new leaf compiler; existing call sites do not change.
-
Leaves are type-aware. Each encoding's leaf compiler picks the right cast for the comparison value's type. Encodings that store typed values additionally check that the stored type matches the comparison-value type before comparing, so a comparison whose value type doesn't match the storage type excludes the row rather than spuriously matching or erroring.
-
One short-circuit, ordered after resolution. An empty IN set compiles to "match no rows" — but only after the field has resolved successfully. An empty IN against an unknown field still raises. This is the only deviation from "every leaf goes through the resolver and emits a comparison," and it exists only because most backends treat empty value lists as a syntax error.
Field naming and namespaces
Field names outside the resolver's recognized set raise. To support sparse user-defined keys without losing the strict-on-typo guarantee, resolvers may carve out a namespace prefix that always resolves (canonically m. or metadata.). Keys under that prefix compile successfully even when no record has ever stored them; missing-key behavior then falls through to ordinary "comparison against absent value excludes the row" semantics at query time.
This split is the central design choice: the closed schema is treated as a contract (typos error), and the open keyspace is treated as data (missing keys exclude). The compiler does not need to know which fields are which — it only sees encoding tags — but the contract makes both halves predictable for callers.
In short
The previous design tried to be helpful by tolerating unknown fields silently. The new design refuses to. Anything that looks like a programmer error — typo, schema drift, type mismatch on a typed field — raises at compile time. Anything that looks like a data condition — missing key, absent value — follows ordinary backend semantics at query time.
Describe alternatives you've considered
Three-valued logic (rejected):
3VL is the right tool for partial data (a row whose salary is NULL), not partial queries (a caller who typed salray instead of
salary) — the first is a real-world condition the database should reason about, the second is a bug that should fail loudly. Under
3VL-at-compile-time, WHERE salray = 100000 silently returns zero rows, identical to "nobody earns that"; under strict mode it
raises and you fix the typo before it ships. Same shape for schema drift: rename a column from created_at to created_ts, and every
old WHERE created_at > ... quietly returns empty under 3VL but errors loudly under strict — the latter is the conversation you
actually want.
A 3VL-shaped reading of "unknown field name" — A AND unknown = U, etc. — is rejected because schema errors are programmer errors, not data conditions. Surfacing them as ValueError is more useful than smearing them into row-inclusion behavior.
Additional context
No response
Is your feature request related to a problem?
Current filter compilation logic does not apply filters intuitively and is unsafe.
Claude review 1:
Claude review 2:
Describe the solution you'd like
#1291
Filter Compilation: Design
Problems with the previous design
Unknown fields had no consistent boolean meaning. The resolver could signal "I don't recognize this name" by returning a sentinel, and the compiler would drop the leaf and re-knit the surrounding boolean tree. The result was that the implicit truth value of an unrecognized leaf depended on its parent — TRUE-shaped under
AND, FALSE-shaped underOR, "no constraint" at the top level or underNOT. No single truth assignment to "unknown" reproduced all the cases, and the behavior matched neither classical two-valued logic nor SQL's three-valued logic.Typos and schema drift were invisible. Unrecognized fields produced no observable error at the API boundary. Misspelled names and renamed columns compiled successfully and silently changed the meaning of the surrounding query.
Storage encoding was a binary flag. Resolvers returned
(column, is_json), which conflated "is this a JSON path" with "every encoding the system might want." Adding a third encoding required either changing every resolver's signature or smuggling extra information through the column expression.Boolean combinators carried implicit conventions. Sentinel-handling for "leaf disappeared" was pushed into
And,Or, andNot, so the meaning of a subexpression depended on what was around it. The algebra was hard to reason about and the implementation hard to extend.New behaviors
Unknown fields fail loud. The resolver returns either a resolved field (column expression plus encoding tag) or raises a typed error naming the offending field. The compiler does not catch the error. Misspellings, removed columns, and schema drift all surface before the query runs. Programmer errors are routed through the exception channel; data conditions are not.
Boolean combinators behave as classical two-valued operators.
And,Or, andNotrecurse over their children with no sentinel propagation, leaf-dropping, or tree re-knitting. The truth value of any subexpression depends only on its own structure and its operands' values, never on what's around it.Encoding is a closed tag, not a boolean. The resolver returns one of a small enumeration of encoding tags alongside the column. Adding a new encoding is an additive change: define a new tag value and a new leaf compiler; existing call sites do not change.
Leaves are type-aware. Each encoding's leaf compiler picks the right cast for the comparison value's type. Encodings that store typed values additionally check that the stored type matches the comparison-value type before comparing, so a comparison whose value type doesn't match the storage type excludes the row rather than spuriously matching or erroring.
One short-circuit, ordered after resolution. An empty
INset compiles to "match no rows" — but only after the field has resolved successfully. An emptyINagainst an unknown field still raises. This is the only deviation from "every leaf goes through the resolver and emits a comparison," and it exists only because most backends treat empty value lists as a syntax error.Field naming and namespaces
Field names outside the resolver's recognized set raise. To support sparse user-defined keys without losing the strict-on-typo guarantee, resolvers may carve out a namespace prefix that always resolves (canonically
m.ormetadata.). Keys under that prefix compile successfully even when no record has ever stored them; missing-key behavior then falls through to ordinary "comparison against absent value excludes the row" semantics at query time.This split is the central design choice: the closed schema is treated as a contract (typos error), and the open keyspace is treated as data (missing keys exclude). The compiler does not need to know which fields are which — it only sees encoding tags — but the contract makes both halves predictable for callers.
In short
The previous design tried to be helpful by tolerating unknown fields silently. The new design refuses to. Anything that looks like a programmer error — typo, schema drift, type mismatch on a typed field — raises at compile time. Anything that looks like a data condition — missing key, absent value — follows ordinary backend semantics at query time.
Describe alternatives you've considered
Three-valued logic (rejected):
A 3VL-shaped reading of "unknown field name" —
A AND unknown = U, etc. — is rejected because schema errors are programmer errors, not data conditions. Surfacing them asValueErroris more useful than smearing them into row-inclusion behavior.Additional context
No response