ILP ingest silently drops new fields with no error — data loss, no schema evolution for timeseries collections #8

Summary

When ingesting data via ILP (InfluxDB Line Protocol), if new fields appear in later ILP lines that were not present in the initial data, NodeDB silently drops them without any error or warning. The ingest reports success, but the new fields are never stored, never appear in the schema, and are not queryable.

This means timeseries collections have a frozen schema defined by the first batch of data, with no way to evolve it over time.

Environment

• NodeDB version: v0.0.0-beta.1 (branch main, commit fd8fb8f)
• OS: Linux 6.17.0 / 8GB RAM / 4 CPU
• Ingest method: ILP over TCP on port 8086
• Collection type: Timeseries (CREATE TIMESERIES dns_mega)

Reproduction steps

1. Start NodeDB with NODEDB_PORT_ILP=8086 ./target/release/nodedb
2. Create a timeseries collection: CREATE TIMESERIES dns_mega
3. Ingest data with the original field set via ILP:

dns_mega,client_ip=10.1.2.3,qtype=A,rcode=NOERROR,cached=true,upstream=udp://8.8.8.8:53,interface=eth0,interface_ip=10.0.0.1,qname=google.com elapsed_ms=1.234 1775014418086000000

4. Later, ingest data with additional new fields (response_bytes, ttl, dnssec_validated):

dns_mega,client_ip=10.1.2.3,qtype=A,rcode=NOERROR,cached=true,upstream=udp://8.8.8.8:53,interface=eth0,interface_ip=10.0.0.1,qname=google.com elapsed_ms=1.234,response_bytes=512i,ttl=3600i,dnssec_validated=true 1775015000000000000

5. Query for the new fields.

Expected behavior

• Option A (preferred): New fields are automatically added to the schema and stored. Old rows have NULL for the new fields.
• Option B: ILP ingest returns an error or warning indicating the fields were rejected.

Actual behavior

• ILP ingest returns success with 0 errors for all 10M rows containing new fields
• New fields are completely absent from the schema:

> DESCRIBE dns_mega
{"field":"timestamp","nullable":"true","type":"TIMESTAMP"}
{"field":"cached","nullable":"true","type":"VARCHAR"}
{"field":"client_ip","nullable":"true","type":"VARCHAR"}
{"field":"interface","nullable":"true","type":"VARCHAR"}
{"field":"interface_ip","nullable":"true","type":"VARCHAR"}
{"field":"qname","nullable":"true","type":"VARCHAR"}
{"field":"qtype","nullable":"true","type":"VARCHAR"}
{"field":"rcode","nullable":"true","type":"VARCHAR"}
{"field":"upstream","nullable":"true","type":"VARCHAR"}
{"field":"elapsed_ms","nullable":"true","type":"FLOAT"}
-- NO response_bytes, ttl, or dnssec_validated

• Attempting to query the new fields gives a schema error:

> SELECT response_bytes FROM dns_mega LIMIT 5
ERROR: Schema error: No field named response_bytes.
Valid fields are dns_mega.timestamp, dns_mega.cached, dns_mega.client_ip,
dns_mega.interface, dns_mega.interface_ip, dns_mega.qname, dns_mega.qtype,
dns_mega.rcode, dns_mega.upstream, dns_mega.elapsed_ms.

> SELECT ttl FROM dns_mega LIMIT 5
ERROR: Schema error: No field named ttl.

> SELECT dnssec_validated FROM dns_mega LIMIT 5
ERROR: Schema error: No field named dnssec_validated.

• No .col files exist on disk for any of the new fields:

$ find ~/.local/share/nodedb/ts/dns_mega/ -name "response_bytes*" -o -name "ttl*" -o -name "dnssec*"
(no output — zero files)

• Even the latest flushed partition's schema.json only has the original 10 columns.

Test details

In our stress test, we ingested data in phases:

• Phase 1-2: 30M rows with original fields only (elapsed_ms)
• Phase 3: 10M rows with new fields added (response_bytes, ttl, dnssec_validated)
• Phase 4: 60M more rows, last 30M with new fields

Total: 40M rows with new fields ingested at ~665K rows/sec with 0 reported errors. All new field data was silently discarded.

Why this matters

Schema evolution is critical for timeseries/observability workloads:

• Instrumentation changes: Applications add new metrics over time (e.g., adding response_bytes to an existing DNS telemetry pipeline)
• Sparse fields: Some fields only apply to certain event types (e.g., dnssec_validated only relevant for DNS responses, not all queries)
• Gradual rollouts: New fields appear as services are updated incrementally across a fleet

InfluxDB, QuestDB, and TimescaleDB all support automatic schema evolution via ILP — new fields are added to the table schema on first appearance. This is a table-stakes feature for ILP compatibility.

Suggested behavior

When a new field appears in an ILP line:

1. Automatically add the column to the collection schema
2. Existing rows should return NULL for the new column
3. New rows should store the value normally
4. The DESCRIBE output should reflect the updated schema

At minimum, if auto-evolution is not yet implemented, the ingest should return an error or log a warning rather than silently dropping data.