Context
The follow-up to #1597 moves the vector upsert inside the segment store's write transaction, so segments commit only after their records are acknowledged by the vector store (a failed upsert rolls the segments back; a forget can only ever see links whose records exist). A bulk encode_events batch then holds one SQL transaction, and on SQLite the database writer lock, for the whole batch's upsert.
Costs that grow with batch size while the upsert is in flight:
- PostgreSQL: one pooled connection stays busy for the batch's upsert, and partition deletion (which takes the registry row for update) waits that long.
- SQLite: every writer on the database file waits for the batch's upsert, so a large import stalls other partitions' writes for the upsert's duration.
Ask
Bound the work one encode transaction spans, so that a single transaction never covers a multi-second vector upsert. Either:
- chunk bulk batches inside
encode_events, each chunk its own transaction and upsert (the caller still sees one call), or
- reject oversized batches at the API with a documented request limit.
Not decided yet: chunk size versus request limit, and whether it is a knob or a constant. This issue exists so the bound is not forgotten; the follow-up PR ships without one.
Context
The follow-up to #1597 moves the vector upsert inside the segment store's write transaction, so segments commit only after their records are acknowledged by the vector store (a failed upsert rolls the segments back; a forget can only ever see links whose records exist). A bulk
encode_eventsbatch then holds one SQL transaction, and on SQLite the database writer lock, for the whole batch's upsert.Costs that grow with batch size while the upsert is in flight:
Ask
Bound the work one encode transaction spans, so that a single transaction never covers a multi-second vector upsert. Either:
encode_events, each chunk its own transaction and upsert (the caller still sees one call), orNot decided yet: chunk size versus request limit, and whether it is a knob or a constant. This issue exists so the bound is not forgotten; the follow-up PR ships without one.