Skip to content

Flaky test: RespSortedSetTests.CanDoSortedSetExpireLTM races TTL against CPU-dependent LTM fill #1981

Description

Type

Bug (flaky test)

Summary

Garnet.test.RespSortedSetTests.CanDoSortedSetExpireLTM is flaky and fails intermittently in CI (observed on windows-latest, net10.0, Debug, Garnet.test.collections):

Assert.That(anObject, Is.Not.Null)
Expected: not null
But was:  null
at Garnet.test.RespSortedSetTests.CanDoSortedSetExpireLTM() in RespSortedSetTests.cs:line 2196

Root cause

The test sets a field expiration (ZEXPIRE ... 4 = 4s TTL) before filling the store with 1000 sorted sets to force larger-than-memory (LTM) disk spill. That fill issues ~1000 synchronous round-trips whose duration scales with available CPU. The 4-second TTL clock starts before the fill, so on a slow / CPU-constrained runner the fill + delay exceeds 4s and the field legitimately expires before the IsNotNull assertion.

Instrumented repro (Linux, Debug, net10.0):

CPU quota 1000-key insert Elapsed since 4s TTL set at check Result
0.5 CPU ~3119 ms ~5201 ms (> 4000) FAIL (field expired)
4 CPU ~754 ms ~2796 ms (< 4000) PASS

Under a 0.5-CPU quota it reproduces 100% of the time. Garnet's expiration/disk-spill behavior is correct — the assertion simply races the CPU-dependent setup cost.

Fix

Issue the ZEXPIRE commands after the LTM fill completes, so the TTL clock is not consumed by the fill. The expiring keys are added before the fill (so they still spill to disk); only the expiration is applied afterwards.

Environment

  • Repo: microsoft/garnet main
  • Observed on: windows-latest, net10.0, Debug, Garnet.test.collections

Activity

added 10 commits that reference this issue on Jul 24, 2026
e5c70ad
3c30a2b
d3224cf
60847a2
207ee1e
2cacc8c
09b8c5e
14efe20
d699492
f2bdf08
locked and limited conversation to collaborators on Sep 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions