Repository navigation
feat(cli): fluree server run --memory for a throwaway in-memory server - #1968
Conversation
The server had no way to ask for memory storage. `storage_type_str()` reported "memory" whenever no storage path was set, but `build_direct_fluree` built file storage in `.fluree/storage` in that case, so logs and `/v1/fluree/stats` named a backend that was never built. Add `--memory` / `FLUREE_MEMORY_STORAGE` (a boolean in the same style as `--cors-enabled`), built with `FlureeBuilder::memory()`. A missing storage path still means `.fluree/storage`. `--memory` and a storage path or connection config pick the same thing, so `load_and_merge_config` settles them by source. The flag replaces a path or connection config from the environment or the config file. `FLUREE_MEMORY_STORAGE` replaces one from the config file or the environment, but yields to a `--storage-path` or `--connection-config` flag. Two flags are rejected by `validate`. Resolving in `load_and_merge_config` covers every entry point, including the CLI's background child, which re-reads the environment. `storage_type_str()` now follows the build order: proxy, memory, connection config, file.
`FlureeServerBuilder::memory()` returned the default config, which builds file storage in `.fluree/storage` under the working directory. It now sets `memory`. `new()`'s doc comment claimed memory storage too and now names the file default it actually uses.
`fluree server run` needed a `.fluree/` directory, so a CI job could not start a fresh server in one line. `fluree server run --memory` keeps every ledger in memory: it needs no project directory, writes nothing to the directory it runs in, and loses everything on exit. It serves the normal HTTP API with the usual background workers. `--memory` conflicts with `--storage-path` and `--connection-config` on the command line. It beats a storage path from `FLUREE_STORAGE_PATH`, a profile, or the config file, and the `.fluree/storage` default is not filled in. `FLUREE_MEMORY_STORAGE=true` (or `-- --memory`) also selects it. In memory mode `run` writes no `server.meta.json`, so `status`, `stop`, and CLI auto-routing do not see it. They already ignore foreground servers' pid (`run` never wrote one), so nothing breaks. `start` and `restart` refuse memory mode with an error that points to `run --memory`. A daemon keeps its pid, log, and metadata in `.fluree/`, and putting them anywhere else (the global directory, say) would change where `stop` and auto-routing look.
The storage, running, configuration, CLI and quickstart pages offered memory storage only through a connection config. They now lead with `--memory` / `FLUREE_MEMORY_STORAGE`, keep the connection-config route, and explain what memory mode skips: no `.fluree/`, no `server.meta.json` (so no CLI auto-routing), and `start` refuses it. Adds a CI example that starts a throwaway server, waits for `/health` and creates a ledger, and a Docker note for passing `--memory` through. The configuration page's "Development (Memory Storage)" example had been starting file storage and now passes `--memory`. `storage_type` in `/v1/fluree/stats` can also be `connection-config` or `proxy`.
aaj3f
left a comment
There was a problem hiding this comment.
@bplatz good catch & good fix. No arguments on design or implementation, but this is one place where I wonder whether--despite the fact that we've clearly defined precedence of, for example, runtime env vars over config--we should still possibly warn someone if a memory-only env var or arg is superseding a config in place to use local file storage, for example. That is to say, this is maybe one place where someone with local file ledgers/commits/indexes who then runs in-memory may want a small warning log (or just an emphatic info log) that their configuration for their file-stored data is being superseded by the memory option. I'm not convinced we NEED this, but just wondering if it would ever surprise anyone. Full review below (and approve regardless)
This is really nice, @bplatz — and the part I like most isn't the flag, it's that storage_type_str() now mirrors build_default_fluree instead of re-guessing it. I traced the two orders against each other (state.rs:432 for proxy, then state.rs:493-521 for memory / connection-config / file) and they match term for term, which means the old "reports memory, builds .fluree/storage" lie is gone for the single most common configuration rather than just for the new one. Putting settle_memory_storage inside load_and_merge_config rather than in the CLI is the right altitude too — it's the one funnel all three entry points share, including run_child's re-parse, so the daemon and its parent can't disagree about storage. And the clap attributes on memory are byte-identical to --cors-enabled / --indexing-enabled, so this extends the switch idiom #1944 introduces rather than inventing a second one — though see the merge-order note below, because #1944 hasn't actually landed.
Before any of the code notes, the merge-order one. This PR's base ref is main, but its head has #1944's head 741fd78be as a direct ancestor (compare/741fd78be...9d0264386 → ahead_by: 4, behind_by: 0), so merging this merges the whole of #1944 — which currently has zero reviews. Four commits of config-precedence, boolean-switch and docs work, including the big deletions in ipfs-storage.md, api/headers.md, storage.md and configuration.md, would reach main reviewed nowhere. Because the base says main rather than fix/server-config-docs, GitHub shows the union instead of the delta and nothing marks which half belongs to the other review. Either re-target the base to fix/server-config-docs (my preference — the dependency becomes explicit and the diff becomes this PR's real 14 files / +748/−70) or land #1944 first and rebase. Everything below grades the 741fd78be..HEAD delta only.
On the code itself, the one I'd most like folded in before merge is config_file.rs:1055: settle_memory_storage nulls a configured storage_path / connection_config and emits nothing, so FLUREE_MEMORY_STORAGE=true can silently replace a durable storage configuration with a volatile one. Given the Docker entrypoint is fluree server run "$@" and these docs now teach the env var as a CI idiom, the realistic shape is a compose file that still carries it next to a mounted volume — healthy server, normal responses, everything gone on restart. validate() already warns for the strictly milder --storage-path vs --connection-config case, so a diagnostic here is just matching the house idiom; note it has to route through the returned error or land after init_logging, since this runs before logging exists on both entry paths. Second, validate() refuses raft + proxy two lines from the new check but not raft + memory, which lets a Raft node keep its log on disk while its content store evaporates. Then two small ones: is_file_storage() at config.rs:1029 still carries the exact inference this PR just removed from storage_type_str() (no callers, but it's public API on a library crate), and the "restart refuses FLUREE_MEMORY_STORAGE" claim in docs/cli/server.md:69 doesn't hold on the metadata-present path through run_start_with_child_args — the env var is quietly ignored there rather than refused.
Adherence to repo commitments:
- Patterns/abstractions: ✔ Extends the
--cors-enabledoptional-value boolean idiom verbatim, resolves in the sharedload_and_merge_configfunnel every entry point already uses, and makesstorage_type_str()the single owner of the build-order question —⚠️ only thatis_file_storage()was left re-deriving it wrongly next door. - Performance (speed first, memory second): ✔ No engine or hot-path code touched;
storage_type_str()runs at startup and per/v1/fluree/stats,settle_memory_storageonce per process, and no allocation / clone / lock /.collect()lands on any per-row or per-flake path. No performance-degradation risk. - Testing: ✔ Ten new tests across unit, config-resolution and true end-to-end (real binary, empty temp dir, SIGTERM-and-restart, synchronous
/reindex), each with a stated mutation check — and I confirmed all ten appear by name asPASSin the green CItestjob rather than taking "has tests" on faith.⚠️ the only untested claim isrestartrefusing the env var and-- --memory, which is the gap in the note above. - Conventions: ✔ Self-describing subjects, substantial multi-line bodies that explain the why (the
start-refuses-memory rationale especially), storage behavior documented across five pages with the--connection-configroute kept, and the stale "Development (Memory Storage)" example that had been starting file storage actually fixed. fmt/clippy/test/testsuite-sparql/wasm32/wasm-smoke/sql-bridge all green on9d0264386.
Verified locally at branch HEAD 9d0264386: no cargo run — the worktree's build dir was cold and none of these findings turn on compilation — so this is trace-and-CI-log verification: full CI log pulled and grepped for all ten test names, all three load_and_merge_config callers traced, the restart → run_start_with_child_args → child-settle path walked end to end, MemoryNameService's Arc<RwLock> fields confirmed so the deleted thread-safety note is genuinely stale, FlureeBuilder::memory()'s indexing_config: None confirmed harmless because with_indexing_thresholds reassigns unconditionally, and the Dockerfile entrypoint read to confirm the documented docker run … --memory actually works.
Approving the code so it isn't waiting on me — but with the base re-targeted (or #1944 landed first), since as it stands a merge here carries an unreviewed PR in with it. The settle_memory_storage diagnostic and the raft guard are both a few lines, and I'd rather see them here than in a backlog issue. Happy to talk through any of these if you disagree on the substance.
| (true, true) => {} | ||
| (false, true) => config.memory = false, | ||
| (_, false) => { | ||
| config.storage_path = None; |
There was a problem hiding this comment.
fluree-db-server/src/config_file.rs:1055 — optional, but the one I care about most.
settle_memory_storage erases a configured storage path or connection config without saying so anywhere. In the (_, false) arm we set config.storage_path = None and config.connection_config = None, and nothing downstream ever mentions that a durable storage configuration was dropped — the only trace is the storage = "memory" field on one info line at startup.
The precedence itself I'm fine with, and it's in the table in the PR body. What bothers me is the direction: FLUREE_MEMORY_STORAGE=true outranks a storage path that came from the config file or from the environment, so an env var can silently turn a durable deployment into a volatile one. The Docker entrypoint is fluree server run "$@", so the concrete shape is a compose file or base image that still carries FLUREE_MEMORY_STORAGE=true from a CI recipe (which these docs now teach) alongside a mounted /var/lib/fluree and FLUREE_STORAGE_PATH — the server comes up healthy, serves normally, and loses everything on the next restart. Compare validate() two hundred lines up, which does tracing::warn!("--storage-path is ignored when --connection-config is set") for the strictly milder durable→durable case.
One wrinkle if you go looking: a bare tracing::warn! here would be swallowed. On both paths this runs before logging exists — main.rs calls load_and_merge_config at :33 and init_logging at :56, and run_foreground calls build_server_config before init_logging at :190. That's presumably why config_file.rs has no tracing/eprintln calls at all today and surfaces everything through the returned ConfigFileError. So either follow that convention (hand the discard back as a warning the caller eprintln!s, the way main.rs:38 already prints non-fatal config errors), or stash what was displaced on the config and log it right after init_logging, next to where storage_type_str() already goes out.
// e.g. in the `(_, false)` arm, recording rather than logging:
if let Some(path) = config.storage_path.take() {
config.memory_displaced = Some(format!("storage path {}", path.display()));
}Minor and non-blocking — but if you agree it's right, I'd rather see it in this PR than lost in the backlog.
There was a problem hiding this comment.
Folded in (30db74c), using the second shape you described. settle_memory_storage records what it displaced on the config (ServerConfig::memory_displaced, e.g. storage path /var/lib/fluree), since it runs before logging exists. The server logs a warn! next to its other startup warnings, which runs after init_logging for both fluree server run and the standalone binary.
memory_settles_against_other_storage_by_source now asserts what gets recorded, including nothing when nothing was configured. configuration.md and cli/server.md mention the warning.
|
|
||
| // `load_and_merge_config` settles `--memory` against a storage path or | ||
| // connection config from other sources; two flags remain a conflict. | ||
| if self.memory && (self.storage_path.is_some() || self.connection_config.is_some()) { |
There was a problem hiding this comment.
fluree-db-server/src/config.rs:1184 — optional. This is more of a question than a suggestion, but I think the answer is "add the check."
Two lines below where I'm commenting, validate() refuses raft.enabled=true with storage-access-mode=proxy (config.rs:1219-1223). Nothing refuses raft.enabled=true with --memory, and build_fluree_with_nameservice → build_direct_fluree takes the new memory branch first (state.rs:493), so the node would keep its raft log and snapshots on disk at raft_storage_path while its ledger content store lives in per-process memory. The existing path-overlap guard doesn't catch it either — it's inside if let (Some(raft_path), Some(storage_path)), and by then settle_memory_storage has nulled the storage path.
The failing shape: fluree-server --raft-enabled --raft-node-id 1 --raft-storage-path /srv/raft --raft-listen-addr … --memory starts cleanly. The nameservice replicates through raft, so a follower learns about a commit the leader wrote into its own memory and can't read it; and after a restart the replayed log points at content that evaporated. It's narrow — raft isn't a default feature and only runs in the standalone binary — but it's a new footgun this PR opens, and the guard is a four-line sibling of one that already lives right there:
if self.memory {
return Err("raft.enabled=true is incompatible with --memory (the raft log \
replicates references into a content store that must outlive the \
process)".to_string());
}I recognize this is minor and non-blocking — but if you agree it's right, I'd rather see it folded in now than tracked and forgotten.
There was a problem hiding this comment.
Added (30db74c). validate() refuses raft.enabled=true with --memory, next to the proxy check, and rejects_raft_with_memory_storage covers it. It ran under the raft feature in CI.
I also closed a sibling you didn't raise. A query peer (server_role=peer) with --memory is refused too, because a peer reads the transaction server's storage, which a per-process store can never share. In proxy mode the proxy silently won instead. memory_on_a_query_peer_does_not_validate covers that, and configuration.md lists both refusals.
| /// Get storage type string for logging | ||
| /// The storage the server builds from this config, for logs and | ||
| /// `/v1/fluree/stats`. Checked in the order `build_default_fluree` picks. | ||
| pub fn storage_type_str(&self) -> &'static str { |
There was a problem hiding this comment.
fluree-db-server/src/config.rs:1040 — optional / nit. (Real location is config.rs:1029, a few lines above this hunk.)
storage_type_str() just stopped inferring "no path means memory" — but is_file_storage() at :1029 still does exactly that: self.storage_path.is_some() && self.connection_config.is_none(), doc-commented "Check if using file storage (vs memory)". With no --storage-path the server builds file storage in .fluree/storage, and that helper says false.
Nothing in the repo calls it (nor has_connection_config() at :1034 — I grepped the workspace and both have zero callers), so nothing is broken today; but they're public API on a library crate, so an embedder gets the stale answer, and it'd be a shame to leave the same wrong inference sitting in the same impl block as the fix. Either delegate it to the one function that now knows the build order, or just delete the pair since nothing uses them:
/// Check if using file storage (vs memory)
pub fn is_file_storage(&self) -> bool {
self.storage_type_str() == "file"
}(Shown as a block rather than a suggestion because it spans several lines and the anchor is single-line — applying it as a one-click suggestion would duplicate the surrounding code.)
Your call which way — deleting them is probably the cleaner answer.
There was a problem hiding this comment.
Deleted both (30db74c). Neither had callers anywhere in the workspace, so there was nothing worth keeping.
|
|
||
| /// Start the server using pre-built child args (used by `restart` to replay | ||
| /// the original arguments from `server.meta.json`). | ||
| async fn run_start_with_child_args( |
There was a problem hiding this comment.
fluree-db-cli/src/commands/server.rs:420 — optional / accuracy.
The commit body and docs/cli/server.md:69 both say start and restart refuse memory mode "whether it comes from the flag, -- --memory, or the env var." For start that's exactly true and the e2e test proves all three. For restart I think only the explicit flag is refused: run() checks the parsed ServerAction::Restart { memory } at :128, but with a server.meta.json present run_restart hands off to run_start_with_child_args here, which never builds a ServerConfig and so has no memory check at all.
I don't think anything unsafe falls out of it — I traced it and the spawned child is rescued downstream: the replayed args always carry --storage-path or --connection-config (the original start filled the .fluree/storage default before build_child_args ran), so in the child that's a ValueSource::CommandLine value and settle_memory_storage's (false, true) arm clears memory. So restart under FLUREE_MEMORY_STORAGE=true quietly ignores the env var rather than refusing it, and restart -- --memory sends the child into validate()'s "--memory cannot be combined with --storage-path" rather than the friendly pointer at memory_needs_foreground().
Cheapest fix is probably to resolve and check before the handoff — or, if you'd rather not add a resolution pass to the restart path, soften the sentence in docs/cli/server.md:69 and the commit body to say start refuses every source and restart refuses the flag. Either is fine by me; I'd just like the docs and the code to agree.
There was a problem hiding this comment.
Good catch. It turned out worse than a silent ignore. With a daemon running, restart -- --memory went through run_restart, which stops the server first. The replayed child then failed validate() on --memory plus the saved --storage-path, so the daemon was left down.
Fixed in code rather than in the docs (30db74c). run_restart now resolves its own arguments the way start does and refuses memory mode before it stops anything. It checks this command's own arguments, because the replayed ones always name a storage path, which would otherwise outrank the env var. So docs/cli/server.md is accurate now: restart refuses the flag, -- --memory and FLUREE_MEMORY_STORAGE.
The new e2e test restart_refuses_memory_without_stopping_the_daemon starts a real daemon with fluree server start, tries restart -- --memory and restart under FLUREE_MEMORY_STORAGE=true, and checks the daemon is still healthy after each. It passes in CI.
| @@ -1,14 +1,5 @@ | |||
| //! Application state management | |||
There was a problem hiding this comment.
fluree-db-server/src/state.rs:1 — praise. Deleting the "Thread Safety Note" is correct, and I checked rather than assumed: MemoryNameService has all four of its maps behind Arc<RwLock<HashMap<..>>> (fluree-db-nameservice/src/memory.rs:27-36) and its own doc already says it suits multi-threaded runtimes, so the LocalSet-or-refactor caveat was stale rather than load-bearing. Nice catch that it was still sitting there.
There was a problem hiding this comment.
Thanks for checking it rather than taking it on trust.
| .stdin(Stdio::null()) | ||
| .stdout(Stdio::from(log_file.try_clone().unwrap())) | ||
| .stderr(Stdio::from(log_file)); | ||
| for (var, _) in std::env::vars_os() { |
There was a problem hiding this comment.
fluree-db-cli/tests/server_memory.rs:41 — praise. Redirecting HOME and TMPDIR outside the working directory and stripping every FLUREE_* / XDG_* variable off the child is what makes assert_empty(cwd) actually mean something rather than accidentally pass. This is the bit that makes the "writes nothing where it runs" claim testable, and it's worth keeping exactly as-is if this test ever gets refactored.
There was a problem hiding this comment.
Agreed. The new restart test uses the same isolation: HOME/TMPDIR outside the working directory and every FLUREE_*/XDG_* variable stripped.
aaj3f
left a comment
There was a problem hiding this comment.
(meant to approve in the above review)
…n't work - When `--memory` or `FLUREE_MEMORY_STORAGE` takes the place of a configured storage path or connection config, the server now warns at startup and names what it replaced. The settle step runs before logging starts, so it records the replacement on the config (`memory_displaced`) and the server logs it with its other startup warnings. - `validate()` refuses memory storage on a query peer, which reads the transaction server's storage, and on a Raft node, whose log would outlive the data it refers to. - `fluree server restart` resolves its own arguments and refuses memory mode before it stops anything, as `start` does. Before, with a daemon running, `restart -- --memory` stopped it and then failed to start the replacement, and `FLUREE_MEMORY_STORAGE` was silently ignored. - Removes `ServerConfig::is_file_storage` and `has_connection_config`, which had no callers; the first still assumed no path meant memory.
|
Thanks. Each inline thread has its own reply; the changes are in 30db74c, and CI is green with all four new tests passing. Merge order: resolved. #1944 has merged, so this PR's diff is now only its own commits. I also merged Warning when memory overrides file storage: added. When |
Problem
fluree server runrequires a.fluree/directory, and the only way to get memory storage was to write a JSON-LD connection config. So a CI job couldn't start a fresh server (for example, to run a SPARQL test suite against it) without setup steps.The server also reported its storage wrongly:
storage_type_str()saidmemorywhenever no storage path was set, but in that casebuild_direct_flureebuilt file storage in.fluree/storage.FlureeServerBuilder::memory()built file storage too.What this does
fluree server run --memorykeeps every ledger in memory..fluree/and writes nothing to the directory it runs in, not evenserver.meta.json.FLUREE_MEMORY_STORAGE=truealso turns it on.ServerConfig.memory(--memory/FLUREE_MEMORY_STORAGE), built withFlureeBuilder::memory(). The standalone binary's default is unchanged: with no path it still uses.fluree/storage.storage_type_str()now names the storage that actually gets built, checked in the same order the build uses:proxy,memory,connection-config,file. As a result,storage_typein/v1/fluree/statschanges in two cases:file(it saidmemory);proxy.FlureeServerBuilder::memory()now actually builds memory storage. It has no callers; this is a separate commit.ServerConfig::is_file_storageandhas_connection_config. Neither had callers, and the first still assumed no path meant memory.Precedence
--memoryand a storage path or connection config all choose the storage, so the higher-precedence source wins.load_and_merge_configsettles this once. Every entry point calls it, includingstart's background child.--memoryFLUREE_MEMORY_STORAGEFLUREE_MEMORY_STORAGE--storage-path/--connection-configflag--memory--storage-path/--connection-configflagvalidate()in the serverWhen memory replaces a configured storage path or connection config, the server logs a warning at startup naming what it replaced, so a leftover
FLUREE_MEMORY_STORAGE=truenext to a mounted volume doesn't go unnoticed. The settle step runs before logging starts, so it records the replacement on the config (memory_displaced) and the server logs it with its other startup warnings.validate()also refuses memory storage on a query peer, which reads the transaction server's storage, and on a Raft node, whose log would outlive the data it refers to.Foreground only
startandrestartrefuse memory mode, whether it comes from the flag,-- --memory, or the env var. The error points torun --memory.restartchecks before it stops anything. With a daemon running,restart -- --memoryused to stop it and then fail to start the replacement..fluree/. Putting them anywhere else would change wherestopand CLI auto-routing look.server.meta.json, CLI auto-routing doesn't reach a memory server.statusandstopalready ignored foreground servers, sincerunnever wrote a pid file, so nothing changes there.memory = true. Profiles merge keys one at a time, so a profile's storage path layered over a basememory = truewould pick the wrong storage.Background workers under memory storage
None needed disabling. These were run under memory storage:
--reindex-min-bytes 1;/reindex;As in every storage mode, index builds write scratch and cache files under the system temp dir. The configuration page says so, since memory mode is otherwise disk-free.
Docs
--memoryandFLUREE_MEMORY_STORAGEare now documented incli/server.md,operations/storage.md,operations/configuration.md,operations/running-fluree.mdandgetting-started/quickstart-server.md./health, create a ledger.docker.mdon passing--memorythrough. The Dockerfile is unchanged.admin-and-health.md: lists thestorage_typevalues.Tests
For each test below, the named change was reverted and the test failed, then the change was restored.
storage_type_str_names_the_built_storagestorage_type_strmemory_flag_parses_and_names_its_env_varmemoryas a plain switch (so--memory=falseis rejected)memory_with_a_storage_path_does_not_validatevalidatecheckmemory_settles_against_other_storage_by_sourcevalidatecheck; the env var yielding to a flag; the settle stepmemory_builder_selects_memory_storageFlureeServerBuilder::memory()delegating tonew()memory_flag_beats_every_other_storage_path(env, profile, config file).fluree/storagedefault in memory mode; not passing--memoryto the servermemory_env_yields_to_a_storage_path_flagmemory_server_serves_without_a_project_and_leaves_nothingFlureeBuilder::memory()branch;runrequiring.fluree/; the default fill; not passing the flagmemory_conflicts_with_other_storage_flagsstart_and_restart_refuse_memorystartcheck; the default fill; not passing the flagAdded in review, not yet run locally (fmt and all-features clippy are clean; CI runs them):
memory_settles_against_other_storage_by_source(extended): what memory replaces is recorded, and nothing is recorded when nothing was configured.memory_on_a_query_peer_does_not_validate.rejects_raft_with_memory_storage(raftfeature).restart_refuses_memory_without_stopping_the_daemon: with a daemon started byfluree server start,restart -- --memoryandrestartwithFLUREE_MEMORY_STORAGE=trueboth refuse, and the daemon is still healthy afterwards.The end-to-end test starts the real CLI binary in an empty temp dir with no
.fluree/. It then:/reindex;The existing
storage_path_precedencetest now uses the new resolver helper and still passes.Not verified
fluree-db-clisuites (--lib,integration,server_memory,docs_coverageand the seam tests) and thefluree-db-serverlib,grp_proxyandgrp_httpsuites. The other server test groups (grp_query,grp_policy,grp_search,grp_raft,grp_bolt, telemetry) and workspace-wide runs are left to CI.fluree-serverbinary was not run, and no Docker image was built.grp_http,encryption_rotation_http::rotation_runs_and_verifies_over_httpfails some of the time when the full binary runs. It fails at the same rate on fix(server): config precedence, boolean switches, and docs for what exists #1944's code and passes when run alone.Adjacent behavior, unchanged
An env
FLUREE_CONNECTION_CONFIGstill beats a--storage-pathflag. That is documented, but it is not source-aware the way--memorynow is.