Repository navigation
RFC: end-to-end UDF support #2353
Description
The goal of this RFC is to expose UDFs, already supported by the compiler, to the user.
Design
- The user declares UDFs in SQL; the SQL compiler generates Rust prototypes [this is already implemented].
- These prototypes serve as guidelines for the user (in addition to documentation describing SQL-to-Rust type mappings), who is responsible for implementing the functions in Rust. Specifically, the user supplies a
udf.rsfile and audf.tomlfile, which the compiler server uses to generate a separate crate for UDFs. The reason for a separate crate is to avoid dependency clashes with the main circuit crate. This crate gets imported from the main generated Rust program.
API
- Pipeline-related endpoints (POST, PUT, PATCH) will take two additional (optional) fields: rust and toml code.
- The GET endpoint will retrieve these fields, along with compiler-generated UDF prototypes.
WebConsole
- Add two additional tabs for Rust and toml code.
- We also need a way to expose compiler-generated prototypes in the UI. The simplest way is in yet another read-only tab.
Error handling
Compilation errors in UDFs should surface as Rust compilation errors. This should work already without any changes. Line numbers should match line numbers in the Rust code, since we will use the user-provided Rust file as is.
Security
In this design we trust user's Rust code. If we want a more secure version of UDFs, we can build support for wasm-based UDFs on top of this later. The pipeline manager can implement a security policy disallowing non-privileged users to submit Rust UDFs.
Discussion
The most unpleasant part of this is keeping compiler-generated prototypes and user rust code in sync. I can imagine the UI (or better yet the backend) automatically merging changes to UDFs declared in SQL into udf.rs, but that's not trivial. Better suggestions are welcome.
Activity
lalithsuresh commentedon Aug 28, 2024
If I understand this correctly, this is probably where a Postgres style
create function blah(integer) returning integer as "udfs::foo"syntax comes in handy. The compiler can generate the prototype function blah that calls udfs::foo with the correct arguments, and the user stays focused on writing udfs::foo(). If udfs:foo is missing or the types don't line up, you get a meaningful Rust error message. This also keeps the compiler-generated code/prototypes separate from udf.rs, without any merges /into/ udf.rs.So I imagine it'll be something like:
create function blah(integer) returning integer as "udfs::foo"ryzhyk commentedon Aug 28, 2024
Unless you want Rust functions to have different names from the SQL declarations, there's no need for the
as "udfs::foo"part. Likewise,prototypes.rsmight be an unnecessary middleman, although it could actually be useful as a shim between SQL and Rust, e.g., to inject line numbers in error message or deal with NULLs automatically. Need to think about it.This doesn't solve the problem of the user having to guess (or rather derive from documentation) function signatures. For this, I was thinking of exposing compiler-generated signatures to the user through the API, so they can copy/paste the definitions, rather than making them up manually.
lalithsuresh commentedon Aug 28, 2024
Without the middleman behind-the-scenes, you'll likely have trouble with synchronization (SQL <-> udf.rs and the circular nature of it). With the middleman, the synchronization is simple: the compiler always generates prototypes.rs mechanically from the SQL, that expects functions with a particular signature. When the rust compiler kicks in, the user should hopefully get back a meaningful error message from the Rust compiler that specifies the signatures missing.
lalithsuresh commentedon Aug 28, 2024
This is definitely good UX to tell the user the signature. That said, as a first cut, you can skip it if you use the middleman and just rely on the Rust compiler's error messages.
Karakatiza666 commentedon Aug 28, 2024
Hopefully the first version of UDF-related API will have at least some way to get compiler-generated signatures. It can easily be displayed in UI as-is, and later that can be changed to a more convenient UX
ryzhyk commentedon Aug 28, 2024
Yes, this is what we're building.
lalithsuresh commentedon Oct 8, 2024
Good to close?
ryzhyk commentedon Oct 8, 2024
Not quite, working on docs + UI improvements.