[Project / Feature]: strata-safetensors - original NVIDIA NVFP4 weights with Strata scheduling #1858

Sharing strata-safetensors with the Strata community

strata-safetensors is our experimental native NVIDIA NVFP4 safetensors backend built on Strata and sergqwer's NVFP4 work. It loads the original main-model weights directly and keeps Strata's resident expert scheduling, CPU/GPU cooperation, MTP and session caching.

The current developer preview is validated on RTX 5090 32 GB + Ryzen 9950X3D + 96 GB RAM, Windows/CUDA. It supports a 262144-token capacity profile, INT8 KV and optional external image encoding. Model/config paths need local configuration; there is no portable one-click native release yet. Other hardware/backends are not validated.

• Implementation and startup notes
• Raw measurements, reproducible commands and limitations
• First upstream contribution: weights: add an opt-in read-only safetensors source (no engine changes) #1859, a standalone read-only source with no existing engine/default changes.

Feedback and reproducible reports from original-checkpoint users are welcome in our project issues. Integration feedback belongs here or on #1859. Below are the complete implementation summary and validation limits; the measurements are not comparisons against upstream/GGUF.

What do you want to do?

Load an original NVIDIA ModelOpt NVFP4 safetensors checkpoint directly in Strata, without first converting the main model to GGUF, while reusing Strata's hot-expert cache, CPU/GPU scheduling, prefill, MTP and session machinery.

We have a working experimental Windows/CUDA implementation in strata-safetensors, built on your Strata engine and sergqwer's NVFP4 work. The accepted engine implementation is pinned at 1a73d069; the project's main branch now contains that implementation. Would you consider staged upstream integration?

The implemented path retains the original FP4 codes and scales, BF16 projections and full draft/target heads. All 63.282 GiB of expert storage stays in CUDA-pinned system RAM, with hot GPU mirrors. After loading, expert/MTP source reads are sealed; only the ngram table is demand-read during inference (explicit disk session saves/restores are separate). Original FP8 MTP experts, main-model verification, INT8 elastic KV, prompt/session reuse and cancellation are integrated. The current vision profile still uses an external BF16 image encoder with GGUF sidecars; this is not a claim that vision loads directly from safetensors.

On one RTX 5090 32 GB + Ryzen 9950X3D + 96 GB RAM Windows machine, the latest three-round comparison against our preceding native profile, not upstream Strata or GGUF, measures median 24K prefill 2968 -> 3331 tok/s and image prefill 2741 -> 3089. First approximately-8K text prefill regresses 2688 -> 2558. A separate single-fresh-engine-per-profile long-answer screen measures aggregate warm decode 91.93 -> 100.82 tok/s; it is not a repeated estimate or a general speed claim. All samples, commands, hashes and limits are published.

Finite acceptance checks cover corresponding output/main-state equality across zero/two/four/six drafts, EOS and token caps, seeded thinking-mode sampling, A/B/A reuse, disk restore, cancellation/resume and a 262000-token capacity case mapping 262144 KV cells. Expert/MTP source-read counters remain zero after loading; these counters are requested source bytes, not physical SSD telemetry. This does not establish broad model accuracy or long-distance retrieval quality. Four full-length RAM-cached sessions are not established: a multi-checkpoint 262K snapshot can exceed the configured 4 GiB session budget.

Real SM120 FP4 Tensor Core paths exist as opt-in experiments. End-to-end measurements and numerical behavior kept the quality profile on canonical FP32 expert decode and W4A8 prefill. We have not validated HIP, SYCL, native multi-GPU or continuous request batching. The complete experimental branch also inherits NVFP4 fork changes and local deployment profiles; it is not being presented as an immediately mergeable replacement for upstream defaults.

Proposed contribution

Start with a small, independent, opt-in safetensors source PR, then separate model/layout binding from forward/scheduler integration. The first PR adds metadata validation, bounded caller-owned reads, per-family accounting and explicit sealing of resident sources. It does not modify any existing engine/backend/setup file or the root build, and needs no GPU or model download to test. Both Windows Release and Debug builds/tests pass, with the symlink fixture skipped for lack of Windows privileges; the original checkpoint metadata also parses with zero payload reads.

I searched open issues and PRs before posting. #1387 concerns higher-precision/high-VRAM model support and #1764 concerns EXL3; this proposal specifically addresses original ModelOpt NVFP4 safetensors on a single consumer GPU. Feedback on whether this source/model boundary fits Strata would help guide the later integration patches.