Repository navigation
Extract Vector Set cleanup work channel and work set - #2020
Merged
Tiago Nápoli (tiagonapoli) merged 4 commits intoAug 5, 2026
Merged
Tiago Nápoli (tiagonapoli) merged 4 commits into
Tiago Nápoli (tiagonapoli) merged 4 commits into
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
Refactors the Vector Set cleanup pipeline by encapsulating the existing Channel-based signaling and ConcurrentDictionary-based “requested drops” tracking into two dedicated helper types, and wires these into VectorManager while keeping overall behavior intended to remain unchanged.
Changes:
- Introduces
VectorSetCleanupWorkChannel<T>to centralize the single-consumer queue/wake-up patterns used by Vector Set cleanup tasks. - Introduces
VectorSetCleanupWorkSet<TValue>to wrap the “requested drops” dictionary and the NET9 alternate-lookup branching. - Adds unit tests covering both helper types.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| test/standalone/Garnet.test.vectorset/VectorSetCleanupWorkSetTests.cs | Adds unit tests for VectorSetCleanupWorkSet<TValue> behavior (add/contains/complete/enumerate/wait). |
| test/standalone/Garnet.test.vectorset/VectorSetCleanupWorkChannelTests.cs | Adds unit tests for publish/read/completion semantics of VectorSetCleanupWorkChannel<T>. |
| libs/server/Resp/Vector/VectorManager.cs | Switches cleanup/request/drop queues and requested-drop tracking to the new helper types. |
| libs/server/Resp/Vector/VectorManager.Cleanup.cs | Updates cleanup/drop worker loops to use VectorSetCleanupWorkChannel<T> + VectorSetCleanupWorkSet<TValue>. |
| libs/server/Resp/Vector/Cleanup/VectorSetCleanupWorkSet.cs | New work-set wrapper over the “requested drops” dictionary + alternate lookup. |
| libs/server/Resp/Vector/Cleanup/VectorSetCleanupWorkChannel.cs | New work-channel wrapper + payload-free extensions (TryPublish, DrainPending). |
kevin-montrose
approved these changes
Aug 4, 2026
Pull the ad-hoc Channel and ConcurrentDictionary plumbing in the Vector Set cleanup pipeline behind two small types. VectorSetCleanupWorkChannel replaces all three queues. They differ only in how the consumer reads: the drop and mark tasks service all outstanding work in one pass, so they take the whole backlog per iteration and a redundant item costs nothing, while the cleanup scan acts on the item itself and takes one per iteration. The drop queue's payload stays null - it is a wake-up, and the work it refers to lives in requestedDrops. The type folds the three-step "complete the writer, wait for the reader, wait for the consumer" shutdown into CompleteAndWaitForConsumerTask, and does not expose the underlying reader or writer. VectorSetCleanupWorkSet wraps the keyed set of outstanding drops, moving the NET9_0_OR_GREATER alternate-lookup branching and the poll-until-absent loop out of VectorManager. Behaviour is unchanged. Co-authored-by: Copilot <[email protected]> Copilot-Session: d251734c-427c-4331-aebd-471b12411de1
Tiago Nápoli (tiagonapoli)
force-pushed
the
tiagonapoli/vector-set-cleanup-primitives
branch
from
August 4, 2026 21:23
7711248 to
5ba2f32
Compare
Vasileios Zois (vazois)
approved these changes
Aug 4, 2026
Ted Hart (TedHartMS)
approved these changes
Aug 4, 2026
Tiago Nápoli (tiagonapoli)
merged commit Aug 5, 2026
bda83e9
into
microsoft:main
316 of 317 checks passed
Tiago Nápoli (tiagonapoli)
deleted the
tiagonapoli/vector-set-cleanup-primitives
branch
August 5, 2026 17:17
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pulls the ad-hoc
ChannelandConcurrentDictionaryplumbing in the Vector Set cleanup pipeline behind two small types, and uses them inVectorManager. Behaviour is unchanged.VectorSetCleanupWorkChannel<T>Replaces all three queues. They differ only in how the consumer reads:
requestDropTaskChannelrequestCleanupTaskChannel(ulong Context, TaskCompletionSource)cleanupTaskChannelThe drop queue's items are pure wake-ups - the work lives in
requestedDrops, and one pass services all of it - so the backlog genuinely collapses andgets a home as
requestDropTaskChannel.DrainPending().The cleanup-request queue also reads everything available, but for the opposite reason: each item is distinct work (a context to mark, and a
TaskCompletionSourcea caller is waiting on), and draining it in one go lets the pass mark every context and then persistUpdateContextMetadataonce rather than once per item.The last queue acts on the item itself, so it takes one per iteration.
Two of the queues carry no payload, because the work they refer to lives elsewhere - in
requestedDropsfor the drop queue, and in the marked contexts for the cleanup queue. BothDrainPending()and the payload-freeTryPublish()are extension methods onVectorSetCleanupWorkChannel<object>rather than members, so neither can be called onrequestCleanupTaskChannel: publishing adefaulttuple there would queue a nullTaskCompletionSource, and discarding its backlog would drop contexts that need marking and strand every waiter on it.The type also folds the three-step shutdown
into
requestDropTaskChannel.CompleteAndWaitForConsumerTask(requestDropTask), and does not expose the underlyingChannelReader/ChannelWriter.VectorSetCleanupWorkSet<TValue>Wraps
requestedDrops, moving theNET9_0_OR_GREATERalternate-lookup branching and the poll-until-absent loop out ofVectorManager.DropRequestedandWaitForDiskANNIndexDropbecome one-liners overContains/WaitForCompletion.Tests
Unit tests over both types.