Repository navigation
Poor performance with streams (high CPU and memory usage) #2213
Description
Describe the bug
Maybe I am using it wrong.
To support replication between nodes, I use hashicorp/memberlist. Every second, a "Push/Pull" is triggered, during which I queue a Backup/Load (incremental) on the pair of nodes.
It's a very dumb way of doing replication, there are probably better ways.
When using the default settings for the stream, I can observe a huge increase in CPU usage and memory usage. In fact, as we can see in the pprof output in the linked PR, most of the time is spent in Go's garbage collector, and traces back to BadgerDB's Stream.
If I reduce the concurrency by manually setting the number of goroutine used, performance increase (but I'm afraid that replication could become too slow during peak usage of the service (more testing needed on my side).
What are your recommendations and/or tips?
Environment
- OS: Ubuntu 25.04
- Language: Go
- Version: v4.7.0
Activity
linear commentedon Jul 17, 2025
BGR-14 Poor performance with streams (high CPU and memory usage)
hiiii5 commentedon Mar 13, 2026
I'm not sure if you are seeing this, but we see that using stream.Send the z.Buffer stays in memory too long and is 512MB in pprof. This leads to large worker sets, where each worker has a buffer of 256MB so this becomes a problem and defeats the purpose of streams by forcing single threading. Our only solution so far has been to use view transactions to worker pools.
linkdd commentedon Mar 13, 2026
That's an interesting approach. I'll look into it, thanks.
We are currently redesigning the replication layer to use a proper append-only oplog, and not rely on the "tcp push/pull" mechanism to synhronize nodes, which should help as well.
github-actions commentedon Jun 21, 2026
This issue has been stale for 60 days and will be closed automatically in 7 days. Comment to keep it open.
adityeah8969 commentedon Aug 7, 2026
High cpu / memory is coming out of parallelism on the producer side (NumGo=8) that build KVList and pushes to consumer channel. I think the stream output must be serialized in key order, which requires a single writer. This is the same pattern used by backup tools like pg_dump. So naturally there is build up of resources on the producer side. Overall this behaviour might not be an oversight, just a design choice.