Skip to content

RedissonRateLimiter discards permits returned by release() when it recalculates the currently available permits #7342

Description

@likerhythm

Redis version

7.4.10

Redisson version

4.6.1

Redisson configuration

import org.redisson.Redisson;
import org.redisson.api.RedissonClient;
import org.redisson.config.Config;

public class RedissonConfig {

    public static final RedissonClient REDISSON = new RedissonConfig().create();

    private RedissonClient create() {
        Config config = new Config();
        config.useSingleServer().setAddress("redis://127.0.0.1:6379");
        return Redisson.create(config);
    }
}

What is the Expected behavior?

Repeatedly calling tryAcquire(1) and release(1) should change the number of available permits by 1 each time.

What is the Actual behavior?

Repeatedly calling tryAcquire(1) and release(1) causes the number of available permits to drop sharply on a periodic basis.

Additional information

I designed an experiment to demonstrate this behavior.

RateLimiter configuration

  • rate=10
  • interval=1000ms

The experiment code is below. Inside the while loop, tryAcquire(1) and release(1) are called repeatedly every 100ms. After each release(1) call, the available permit count is read via the getPermit() method. getPermit() does not rely on RedissonRateLimiter.availablePermits(); instead it reads the permit count directly through RedissonClient. Once the run finished, I exported both the available permit count and the number of permits held in the sorted set as JSON and plotted them. The graph is shown below the code.

public class SimulationConfig {

    public static final int RATE = 10;

    public static final int INTERVAL_MS = 1000;

    /** Interval between permit consumption and release (ms) */
    public static final int RELEASE_DELAY_MS = 100;

    /** Total execution time (sec) */
    public static final int DURATION_SEC = 30;

    private SimulationConfig() {}
}
public class Simulator {

    public void run() throws InterruptedException {
        RateLimiterConfig config = new RateLimiterConfig();
        RRateLimiter limiter = config.create();

        long startNano = System.nanoTime();
        long endAt = System.nanoTime() + SimulationConfig.DURATION_SEC * 1_000_000_000L;

        List<long[]> samples = new ArrayList<>();

        while (System.nanoTime() < endAt) {
            limiter.tryAcquire(1);
            Thread.sleep(SimulationConfig.RELEASE_DELAY_MS);
            limiter.release(1);
            long permits = getPermit();
            long tMs = (System.nanoTime() - startNano) / 1_000_000;
            long history = config.historySize();
            samples.add(new long[]{tMs, permits, history});
            Thread.sleep(10);
        }

        writeJson(samples);
    }

    private long getPermit() {
        String valueKey = RateLimiterConfig.valueKey();
        RedissonClient redisson = RedissonConfig.REDISSON;
        RBucket<Long> value = redisson.getBucket(valueKey, LongCodec.INSTANCE);
        return value.get();
    }
}

The blue line is the number of available permits, and the green line is the number of permits held in the sorted set. As the graph shows, the available permit count drops sharply at regular intervals. The same pattern appears when reading via RedissonRateLimiter.availablePermits(). The reason I didn't include the availablePermits() results here is that availablePermits() has a separate problem of returning a value greater than rate, which I've filed as a separate issue #7329

Image

The cause of this problem is that tryAcquire() recalculates currentValue as rate - used.

+ "if released > 0 then "
    + "redis.call('zremrangebyscore', permitsName, 0, tonumber(ARGV[2]) - interval); "
    + "if tonumber(currentValue) + released > tonumber(rate) then "
        + "local values = redis.call('zrange', permitsName, 0, -1); "
        + "local used = 0; "
        + "for i, v in ipairs(values) do "
            + "local random, permits = struct.unpack('Bc0I', v);"
            + "used = used + permits;"
        + "end; "
        // currentValue is recalculated here
        + "currentValue = tonumber(rate) - used; "
    + "else "
        + "currentValue = tonumber(currentValue) + released; "
    + "end; "
    + "redis.call('set', valueName, currentValue);"
+ "end;"

Here's the scenario in which the problem occurs, based on my experiment code.

  1. Every 100ms, each tryAcquire(1) adds one permit to the sorted set. Since release(1) never touches the sorted set, the entry stays there even after release(1) returns the permit. Also, because release(1) immediately reflects the returned value into the available permit count, the available permit count is restored to 10 on every release() call.
  2. After 1 second passes, the first permit consumed expires (interval=1000ms).
  3. When expired permits exist (released > 0), tryAcquire() runs the code above.
  4. Because currentValue(=10) + released(=1) > rate(10), currentValue is recalculated based on the used value.
  5. Since one permit has expired, used=9, so recalculating as currentValue = rate - used stores currentValue = 1.
  6. This repeats every time released + currentValue > rate.

This problem can be fixed with the change below. When used + currentValue > rate, it means there are permits that were already returned by release(), so to preserve those permits we skip recalculating currentValue as rate - used.

+ "if released > 0 then "
    + "redis.call('zremrangebyscore', permitsName, 0, tonumber(ARGV[2]) - interval); "
    + "if tonumber(currentValue) + released > tonumber(rate) then "
        + "local values = redis.call('zrange', permitsName, 0, -1); "
        + "local used = 0; "
        + "for i, v in ipairs(values) do "
            + "local random, permits = struct.unpack('Bc0I', v);"
            + "used = used + permits;"
        + "end; "
        // changed here
        + "if used + tonumber(currentValue) <= tonumber(rate) then "
            + "currentValue = tonumber(rate) - used; "
        + "end; "
    + "else "
        + "currentValue = tonumber(currentValue) + released; "
    + "end; "
    + "redis.call('set', valueName, currentValue);"
+ "end;"

After applying this change and running the same experiment, I no longer observed the periodic sharp drops in the permit count.

Image

I'll open a PR that fixes this issue. Review and feedback would be much appreciated.

No activity

Activity on this issue will appear here.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions