Eleven changes from @brettwooldridge, most of them found on a production system, and one H2 workaround found while testing them. Four are data-integrity fixes, three of which can end with a store that will not reopen or a query that quietly returns the wrong rows.

Security Fixes

• Reading an MVStore file no longer deserializes arbitrary classes (CWE-502, GHSA-7w7v-2j76-mqwp)
  • Every MVStore map was opened with H2's default ObjectDataType, which reads stored documents, index entries and metadata back through a plain ObjectInputStream. A database file from an untrusted source - an uploaded backup, an imported or synced .db - could make an ordinary find() instantiate any Serializable class on the classpath and run its readObject(), so a gadget class there meant remote code execution. The allowlist added for GHSA-9297-g93h-86gg covered only the legacy v1 migration path.
  • Maps are now opened with an ObjectDataType that deserializes through a JEP 290 allowlist: Nitrite's own types and JDK types (org.dizitart.no2.**;java.**). A process-wide jdk.serialFilter still applies on top. Top-level object arrays, whose elements H2 reads with its own unfiltered type, are refused; Nitrite never stores one. The on-disk format is unchanged.
  • Documents holding values of other Serializable classes now fail to read unless those classes are allowed with the new MVStoreModuleBuilder.allowedClasses(...), which takes JEP 290 patterns, e.g. allowedClasses("com.example.model.**").

Issue Fixes

• MVStore's chunk retention and versions-to-keep are left at H2's defaults (#1301, #1303)

  • MVStoreUtils.openOrCreate forced setRetentionTime(0) and setVersionsToKeep(0) on every store since 2020. With both at 0, H2 may reuse a chunk's blocks while the chunk map it writes at close still lists that chunk, and the file then refuses to open at all - read-only or not - with MVStoreException: Double mark: 394/5 ... at FreeSpaceBitSet.markUsed. Only H2's recovery mode gets past it. (h2database/h2database#2752, #4083, both open.)
  • A 24-thread soak of a document workload with a close every 20 seconds reproduced it on every run at 0/0, with and without close-time compaction, on h2-mvstore 2.4.240 and on current H2 master - and on none of the runs with either setting at its H2 default (45 s / 5), including a 1 s retention window. H2's own javadoc notes the retention window is what lets readers finish traversing a map.
  • New MVStoreModuleBuilder.retentionTime(ms) and versionsToKeep(n). Both are null by default, which leaves H2's values in place. Passing 0 restores the old behaviour, at the cost described above.
  • The file no longer shrinks the instant a chunk goes dead. Reclamation now waits out the retention window, as H2 intends. close() still compacts synchronously.

• A read no longer hands out anything the store holds (#1294)

  • A stored document on MVStore is the live object in the page, and MVStore serializes pages on a background thread that any write can start through tryCommit. Whatever a read hands back must therefore share no mutable state with it, or a caller's in-place edit is written straight into the store, bypasses the indexes, and can race the serialization into a ConcurrentModificationException and a store panic.
  • The cursor has cloned each document it yields since 4.x, but two gaps remained. Document.clone() copied the top-level map and embedded documents only, so a List, Set, Map, array, byte[], Date or Calendar inside the copy was still the instance in the store and found.get("tags").add(x) reached the page. And NitriteCollection.getById() returned the stored instance itself.
  • clone() is now a deep copy: containers and arrays are copied recursively, preserving the concrete collection class where it has a public no-arg constructor and the comparator of sorted sets and maps; Dates and Calendars are cloned; immutable values are shared, and so are values of a type the copy does not know, which the javadoc now states. getById() hands out a clone as find() does, and returns null for a missing id instead of passing null through the processor chain.
  • The cost is a structural copy per result and per write, since the write path clones too. That buys the guarantee that no caller can reach into the store by accident.

• The store catalog copies its stored name set instead of mutating it in place (#1296)

  • StoreCatalog.writeCollectionEntry wrapped the catalog document in MapMetaData, which took the Set straight out of it and added the new name to that instance. On MVStore that instance is the one held in the catalog page. Creating collections in a burst interleaved with writes - which is what a data migration does - put the serializer's HashSet.writeObject and the catalog's HashSet.add on the same set at the same time, ending in MVStoreException: Could not serialize {mapNames=[...]} and MVStore.panic, after which the store is closed and every later operation throws.
  • Observed on the first start after a Xodus-to-Nitrite migration that created eleven collections in forty milliseconds. MapMetaData now copies, so a write always puts a new set and the instance a serializer may be reading is never touched - the rule WriteOperations.update already followed by cloning a document before merging into it.

• An index scan skips documents removed between the lookup and the fetch (#1302)

  • An index scan is two steps that are not atomic under concurrent writes: the index yields the matching ids, then each document is fetched. A document removed in between came back as an (id, null) row. With a residual filter that row reached Filter.apply and threw NullPointerException from the filter's document.get(); without one the cursor handed the caller a null element. The by-id fast path had the same window between containsKey and get.
  • IndexedStream now prefetches and drops ids whose document is gone, FilteredStream treats a row without a document as a non-match, and the by-id path does one get. skip() still walks ids without fetching them, so under a concurrent remove a page boundary can still count an id whose document is gone - which is the same indeterminacy the removal itself introduces.

• clear() and dropIndex() reach every layout map an index occupies (#1295)

  • IndexManager.close(), clearAll() and dropIndexDescriptor() acted only on the map name recorded in IndexMeta, which is the classic one. That already missed the composite map of a non-unique index: after collection.clear() its rows survived, and a query on that index returned the ids of the cleared documents alongside the new ones - two live documents, four results. It also broke ChangeIdField, whose createIndex found the previous index map still populated and rebuilt over it.

• The first concurrent read of an index after a restart no longer fails dropping its legacy map (#1309, #1315)

  • The first read of an index still in a legacy layout migrates it and drops the legacy map, once per index instance. ComparableIndexer created those instances with an unsynchronized check-then-act, so threads arriving together could each get an instance of their own and each run the migration. On MVStore the second drop asked MVMap.getName() for a map that was already gone, got null, and failed with NullPointerException in NitriteMVStore.removeMap; the same race also threw from Attributes.set through NitriteMap.updateLastModifiedTime.
  • Observed on a production system on the first multi-threaded lookup after every restart, because every close before #1295 left an empty map under the legacy name for the next start to drop.
  • The indexer and the MVStore map and R-tree registries now create their entries with computeIfAbsent, so there is one index instance and one map wrapper per name. NitriteMVMap keeps the name it was opened with and acts only when its compare-and-set wins, so drop() and close() run once; removeMap ignores a null name and no longer creates an empty map only to remove it.
  • Reported against 5.3.0 in #1315: four threads making the first find on a reopened file database failed 21 of 400 finds on 5.3.0 and none with these changes. The reporter's reproduction is now Issue1315Test.

• Concurrent first reads of a map on an in-memory MVStore no longer fail inside H2 (#1311)

  • H2's ObjectDataType picks the delegate that compares serialized keys on first use, through an unsynchronized field, and SerializedObjectType.compare checks delegates by identity. Threads making a map's first key comparison together could each install their own and fail with UnsupportedOperationException: Can not compare. The code is the same in h2 2.4.240 and 2.5.250.
  • Only an in-memory MVStore was exposed. A file-backed map is settled before any reader gets it, by reading its root page at open or by estimating the memory of a write. Through the public API, 32 threads making the first reads of a one-document indexed collection failed in 1 round of 5,000 on...

Issue Fixes

MVStore no longer grows without bound under repeated updates (#1284)

Updating a document leaves its old page obsolete, so a store written to far more often than it grows accumulates chunks that are mostly dead. autoCompactFillRate(0) had disabled reclamation since #41, and the file climbed while the live data stayed small — a reported ~800MB around 100 live documents.

#41 was a cursor reading a chunk that compaction had already reclaimed (Chunk 162 no longer exists), which is why compaction was turned off rather than fixed. Every iterator and cursor now registers an MVStore version for its lifetime, so chunks it is reading cannot be collected underneath it, and compaction is safe to enable again. Abandoned iterators are released by a Cleaner and by close()/drop() on the map.

New MVStoreModuleBuilder.autoCompact, defaulting to true. Set it false for the previous behaviour.

Reclamation on close is guaranteed; reclamation while running is not. close() compacts synchronously and always shrinks the file. MVStore's background compaction runs on a one-second tick and attempts its work under a try-lock, so a writer holding the store lock makes it skip that round: how much a long-running application gets depends on its idle time and on how many cores are available to schedule the thread on. An application that holds the database open for months should not rely on the background half alone — see the File Growth and Compaction docs.

Measured over 25 live documents and 100k updates: the file held at ~213KB against 299KB and climbing, and close() left 57KB against 299KB.

Index keys normalize to double only where the conversion is exact (#1282)

DBValue folded every number to a double before using it as an index key. That is what makes Integer(5) and Double(5.0) the same key (#178), which matters on stores that compare the encoded key rather than calling compareTo — the RocksDB adapter is one.

A double stops stepping by one above 2^53. Around 8.7e17 the nearest doubles are 128 apart, so ids closer than that folded onto one key: a unique index rejected an id it had never seen, and a non-unique lookup returned rows belonging to a different id. Snowflake ids, TSIDs and ULID-style identifiers all live there.

The fold now applies only where the value survives it. Cross-type equality is unchanged for every value a double holds, which is the range #178 is about.

Improvements

Paging coverage for the plan shapes a source-level skip has to decline — an or plan, a collection with removals, and an indexed filter ordered by that same index. From #1283.

Thanks to @DarkAtra and @mfleisch, whose reports and patches this release is built on.

Full changelog: https://github.com/nitrite/nitrite-java/blob/main/CHANGELOG.md

New Features

• nitrite-bridge — inspect a running Nitrite database from a desktop client. The Nitrite adapter for the dbinspect wire protocol: collections and handed-in repositories as browsable stores, schema inferred by sampling documents and always flagged as inferred, paging over FindOptions skip/limit/orderBy, the JSON filter DSL, and watch over Nitrite's collection subscription. The engine-neutral core it plugs into is org.dizitart:dbinspect-bridge, which has no database in its dependency tree at all.

  • Row editing is behind allowWrite, and whole-store snapshot behind allowSnapshot — both false unless the embedding application asks, and absent from the reported capabilities while they are. A row is addressed by _id; an update is partial; changes: 0 means the row was not there. _id inside an update's values is refused, because Nitrite merges an update document and it would rewrite the identity of the row it just matched.
  • regex is off unless allowRegex is set, and length-capped with a nested-quantifier check when it is on. java.util.regex backtracks and a match cannot be interrupted, so the default is the mitigation that matters.
  • Every client-supplied store name is resolved against the set the adapter reported. Nitrite.getCollection creates a collection that does not exist, so an unchecked name would let a paired client make one; writes go through the same allow-list.
  • The module is deliberately absent from the reactor <modules>: dbinspect-bridge is not on Maven Central until the dbinspect release, so a build of a clean checkout would fail on it. Build it on its own, with dbinspect-bridge installed locally.

• The bridge can open a transaction (docs/PROTOCOL.md §3.1). A transaction is a second adapter over the same database rather than a mode on the first, so one connection's uncommitted documents can never reach another connection's reads. Nitrite's transaction lives above the storage engine — a TransactionStore buffers the writes and a journal replays them on commit — so this works identically on MVStore, RocksDB and in memory.

  • capabilities.transactions follows allowWrite: that flag is the permission, this reports what the engine can undo. The transactional twin reports it false, because Nitrite does not nest one.
  • listStores counts through the transaction, since a total that left out the rows the person just staged is not read-your-own-writes.

Performance

• Paging a collection no longer costs every row before the page - find(filter, skipBy(n).limit(m)) now advances the storage iterator without decoding what it passes over.
  • BoundedStream skipped by calling next() on the iterator beneath it, and that iterator is below the document layer: it deserialised every skipped document in order to throw it away. Page latency was therefore linear in the page number - about 9 µs per skipped row on both persistent stores - so a 50 000-row collection paged at 200 rows took 3 ms for the first page and 350 ms for the two-hundredth. It is an ordinary browse, not an edge case.
  • New SkippableIterator, implemented by the storage-level iterators that can do better and by nothing else. BoundedStream uses it when it is there and falls back to the loop when it is not, so a stream that has to look at a document to know whether to skip it - a collection-scan filter, a blocking sort - is unaffected and still correct.
  • MVStore descends the tree by index. An MVStore page records how many entries sit beneath it, so MVMap.getKey(n) is a O(log n) descent rather than a scan; the skip seeks with it and continues from a cursor. Measured over 50 000 rows, 200 to a page: p95 583.9 ms → 5.8 ms, and the first and last pages now cost the same 0.1 ms.
  • RocksDB steps the native iterator. There is no seek-by-index to use, but the decode is what a skipped row actually cost; advancing past one is now a memcmp inside the block the iterator is already on. p95 3.8 ms over the same 50 000 rows.
  • An index scan skips its own id set the same way, without the map lookup and decode next() would have paid for each row.

Issue Fixes

• Fix the index dirty marker not being persisted (#1281)
  • markDirty() flipped isDirty on the IndexMeta that get() returned and never put it back, so nothing told the store the entry had changed and the new value was not guaranteed to be written. Both beginIndexing() and endIndexing() go through it, so the marker was unreliable in both directions: an index left half-built by a crash could come back reading as clean, and a completed one could stay marked dirty and be rebuilt on the first write after every open.

Improvements

• A sorted, limited find no longer fetches the whole collection when the sort field is indexed. find(ALL, orderBy("createdAt", Descending).limit(20)) asked for 20 rows and cost what draining every stored document costs. SortedDocumentStream collects the entire result set before BoundedStream gets to drop 99% of it — and the cost is the decode, not the comparison, so it scales with document size as well as count. An index on the sort field bought nothing: the index was only ever used to filter, never to order, and page 50 cost exactly what page 1 cost because the work finished before the skip applied.

  When the query has no filter, one sort field, a limit, and a simple unique or non-unique index on exactly that field, the sort keys are now read from that index — which already stores them — and only the documents actually returned are fetched. Measured on MVStore over 2000 rows each carrying a 150-element list, a sorted page went from ~115 ms to ~2 ms, and stopped growing with the collection.

  The index is used only when it holds exactly one entry per stored document. A multi-valued field is indexed once per element, which is detected by a duplicate-id check and an entry-count check, and falls back to the blocking sort. Ordering — including where nulls sort and how ties break — is identical either way: the same comparator runs over keys taken from the index instead of from the documents.

  Where documents are small and cheap to read, the index walk replaces a decode that was nearly free, so a sorted page over lean rows can cost a few hundred microseconds more than before. The trade is deliberate: the loss is bounded and sub-millisecond, the win grows without bound with document size.

New API (all additive)

• FindPlan.getSortIndexDescriptor()
• NitriteIndex.readSortKeys(long) and NitriteIndexer.readSortKeys(IndexDescriptor, NitriteConfig, long) — both default-implemented to return null, so existing indexer plugins are unaffected
• DocumentSorter.compareValues(Object, Object, Collator)

Full Changelog: v5.0.0...v5.1.0

Breaking Changes

• Removed the distinct find option — FindOptions.withDistinct() (both overloads), FindOptions.distinct() and FindPlan.distinct.

  The flag never affected the result set. A find never returns the same document twice, and the or sub-plan union in ReadOperations has always applied DistinctStream unconditionally — so FindOptimizer wrote the flag onto the FindPlan and nothing ever read it back. Callers passing FindOptions.withDistinct() can drop it with no change in results.

  DistinctStream itself is unchanged and still used for the or union.

Tests

• Added CollectionOrDuplicateTest, pinning the set-union behaviour of an or filter for the no-index, one-indexed-branch and all-indexed-branch cases. nitrite-java was already correct here; the same test found real duplicate defects in nitrite-rust and nitrite-flutter, and is added to keep the three implementations honest.

Full Changelog: v4.5.0...v5.0.0

New Features

• Add an exists filter - where("field").exists() matches the documents which have the field, irrespective of its value; where("field").exists().not() matches those which do not.
  • A field explicitly set to null is present and matches. This is the case no existing filter could express: eq(null) and notEq(null) cannot tell a missing field apart from one holding null, so "has this document been given a value for this field at all" was not answerable.
  • The filter deliberately does not extend ComparableFilter and so always runs as a collection scan. A missing field and a field holding null are stored under the same null key in an index, so an index scan could not tell them apart and would disagree with a collection scan.
  • Embedded fields are addressed by their dotted path (where("address.city").exists()), the same way Document.containsField resolves them.

Issue Fixes

• Fix JVM crash (SIGSEGV) in the RocksDB backend when a query pages past the end of a collection
  • EntrySet, KeySet and ValueSet close the native RocksIterator the first time hasNext() answers false, but hasNext() is idempotent by contract and a second call reached isValid() on the freed handle. BoundedStream's skip loop exhausts the iterator and then the caller asks once more, so any find(..., skipBy(n)) whose skip exceeds the number of remaining records hit it — an ordinary paging request, not an edge case. The existing catch (AssertionError) only covered the case where assertions are enabled (-ea), which is how the tests ran but not how an application runs: without assertions the same call is a SIGSEGV in Java_org_rocksdb_RocksIterator_isValid0Jni that takes the whole process down.
  • hasNext() now returns false when the iterator no longer owns its handle, so it never touches a closed iterator and stays idempotent.

Issue Fixes

• Fix field.eq(x) / field.in(..) on an array (list) field silently matching nothing when the filter runs as a collection scan
  • Array membership is matched element-wise on the index path (arrays are indexed per element) but was matched by whole-value equality (deepEquals) on the collection-scan path, so a query's results depended on whether an index existed or was chosen by the planner. Combined with the 4.4.2 planner change (#1266) — which correctly relegates the non-winning-index filter to a collection scan — an AND of an indexed array eq and a bounded range on a second indexed field left the array eq running as a collection scan, where it matched no documents.
  • EqualsFilter and InFilter now match an array/Iterable field by element containment on the collection-scan path, mirroring applyOnIndex, so results are the same regardless of index presence.

Issue Fixes

• Fix ClassCastException when an AND filter combines an equality (or other comparable) filter on one indexed field with a bounded range on a second, differently-typed indexed field #1266
  • The query planner picked the best-matching index per field independently while scanning candidate indexes, but accumulated filters from every candidate it visited into one shared set instead of keeping only the winning index's filters. The resulting index scan filter set could carry filters from two unrelated indexes (e.g. a String-valued eq filter alongside a Long-valued range pair), which were then all applied against whichever single index the planner picked, comparing a value of the wrong type against that index's keys.
  • The planner now selects a single best-matching index descriptor and only keeps that index's own filters for the index scan; filters on any other field fall back to a post-filter (collection scan) step as before.

Security Fixes

• Fix unfiltered Java deserialization in the legacy v1 database migration path (CWE-502, GHSA-9297-g93h-86gg, CVSS 9.8)
  • Opening a file-based store runs MVStoreUtils.testForMigration(), which for a legacy v1-format file deserialized stored values through ObjectInputStream.readObject() with no class restriction. Any Serializable class on the embedding application's classpath could be instantiated, so a suitable gadget chain (e.g. from commons-collections) made a malicious .db file a remote-code-execution vector.
  • The v1-compat deserializer now enforces a JEP 290 allowlist filter that only permits Nitrite's own types and standard JDK types; any other class is rejected before its readObject/readResolve callbacks can run. Applications that open Nitrite database files from untrusted sources (e.g. "import"/"restore backup" features) should upgrade.

Issue Fixes

• Fix intermittent ConcurrentModificationException and spurious UniqueConstraintException from unique and full-text indexes on the MVStore backend (regression introduced by the #1260 index rework in 4.4.0)
  • Unique and full-text indexes still store a List<NitriteId> value per key. 4.4.0 switched that list from CopyOnWriteArrayList to a plain ArrayList, which is mutated in place after being written to the map. MVStore serializes dirty page values on a background thread, so that in-place mutation races with the serializer and threw ConcurrentModificationException (and could corrupt the id list, surfacing later as a false unique-key violation) — even under single-threaded use.
  • These index value lists are CopyOnWriteArrayList again, so each mutation swaps the backing array atomically and the background serializer always sees a stable snapshot. The composite-key layout for non-unique indexes (the actual #1260 optimization) is unchanged.

Upgrade Notes

• Non-unique single-field indexes now use a composite-key on-disk layout (one (value, id) row per entry) instead of a single growing id list per value #1260
  • The public API is unchanged. Existing databases are upgraded automatically: a legacy array-format non-unique index is rebuilt into the new layout the first time it is opened, and the old index map is dropped.
  • The storage format change is forward-only — once a database has been opened by this version, it can no longer be read by an earlier version of Nitrite. Back up before upgrading if you may need to roll back.

Performance Improvements

• Fixed performance degradation when inserting thousands of documents that share the same non-unique index key #1260
  • The old layout re-wrote (and, on persistent stores, re-serialized) an ever-growing id list on every insert, making bulk inserts O(n²). The composite-key layout makes each insert and removal an O(log n) point operation across all backends (in-memory, MVStore and RocksDB).
  • RocksDB orders keys by their serialized bytes, so the composite key uses an order-preserving encoding (correct ranges over negative numbers, variable-length strings, dates and booleans).
• Made in filter index scans look up each value directly instead of scanning every index entry, so in queries on large indexed collections are now as fast as eq instead of degrading to a full index walk #1258

Issue Fixes

• Fix DocumentSorter violating the Comparator contract when two documents both have a null sort key, which caused intermittent IllegalArgumentException: Comparison method violates its general contract! from orderBy on fields with multiple null values #1261
• Fix indexed lt/lte filters returning an empty result when the indexed field contains any null value; the forward index scan now starts from the first non-null key #1262
• Fix descending indexed lt/lte filters leaking null-valued documents into the result on reopened persistent stores (MVStore and RocksDB); stored null index keys are now normalized to the DBNull sentinel in every index navigation
• Fix the RocksDB adapter failing to round-trip the null index key through Kryo, and decoding scanned keys with the wrong type when a range scan probe and the stored key have different classes
• Fix in/notIn filters on the _id field not matching legacy String ids written by pre-4.4 databases, while eq/getById on the same rows matched #1263
• Fix eq/notEq filters on the _id field for legacy String ids: eq('_id', "3") or eq('_id', 3) threw ClassCastException in the byId fast path, and notEq/negated eq failed to exclude legacy rows during collection scans