Skip to content

3.x: preserve OVERLOADED during authentication (DRIVER-1121) - #1160

Merged
dkropachev merged 3 commits into
scylladb:scylla-3.xfrom
dkropachev:dk/DRIVER-1121-auth-overloaded
Sep 30, 2026
Merged

dkropachev merged 3 commits into
scylladb:scylla-3.xfrom
dkropachev:dk/DRIVER-1121-auth-overloaded

Conversation

@dkropachev

@dkropachev dkropachev commented Sep 29, 2026 •

Copy link
Copy Markdown

ScyllaDB will return native-protocol OVERLOADED instead of BAD_CREDENTIALS when authentication cannot proceed because the server is overloaded. Java driver 3.x currently turns every authentication-phase error into AuthenticationException, making this transient condition look like invalid credentials.

This change:

  • preserves OVERLOADED as OverloadedException after protocol-v1 CREDENTIALS and v2+ AUTH_RESPONSE;
  • keeps BAD_CREDENTIALS as AuthenticationException and excludes overload from authentication-error metrics;
  • makes initial control-connection setup advance to the next contact point after authentication overload;
  • safely releases dynamic pool-connection creation state so a later attempt can grow the pool;
  • keeps host-up and host-add recovery running when cached-statement reprepare encounters authentication overload;
  • adds regression coverage for both protocol paths, metrics, initial failover, pool retry, and recovered-host reprepare;
  • records the fix in the 3.11.5 changelog.

Java driver 4.x is not changed: its existing initializer reserves AuthenticationException for AUTH_ERROR, while other setup errors close the incomplete channel and remain eligible for node failover and reconnection.

Fixes #1159

Jira: https://scylladb.atlassian.net/browse/DRIVER-1121

Compatibility and risk

No public API or wire-format change. Applications may now observe the existing OverloadedException type where they previously saw AuthenticationException. Exhausting all initial contact points still returns NoHostAvailableException, with each overload retained in its per-host error map.

Testing

  • mvn -pl driver-core test: 723 run, 0 failures/errors, 1 skipped.
  • mvn -pl driver-core -Dtest=ConnectionAuthenticationTest test: 6 passed.
  • mvn -pl driver-core -Pshort -Dtest=HostConnectionPoolTest#should_retry_additional_connection_after_authentication_overload -DfailIfNoTests=false -Dclirr.skip=true -Danimal.sniffer.skip=true test: passed.
  • mvn -pl driver-core fmt:format: 561 files checked, 0 noncomplying.
  • git diff --check

ScyllaDB can return OVERLOADED while processing CREDENTIALS or AUTH_RESPONSE. Preserve it as OverloadedException instead of turning it into AuthenticationException, and exclude it from authentication-error metrics.

Treat that transient setup failure like a connection failure in control-connection and dynamic pool-creation paths, allowing next-host failover and later pool growth. Cover protocol v1/v4 classification, metrics, initial contact points, and pool retry.

Fixes scylladb#1159

Jira: https://scylladb.atlassian.net/browse/DRIVER-1121
@coderabbitai

coderabbitai Bot commented Sep 29, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: QUIET

Plan: Advanced

Run ID: 4949a119-f77d-4e16-87b4-de5df85d4ce7

📥 Commits

Reviewing files that changed from the base of the PR and between 4bc4aea and b6a046f.

📒 Files selected for processing (2)
  • driver-core/src/test/java/com/datastax/driver/core/ConnectionAuthenticationTest.java
  • driver-core/src/test/java/com/datastax/driver/core/HostConnectionPoolTest.java

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.


📝 Walkthrough

Walkthrough

Authentication handlers now propagate OVERLOADED responses as OverloadedException without incrementing authentication-error metrics. Control-connection setup tries remaining hosts after overloads. Query preparation returns null after an overload, and pool growth can retry after an overloaded connection attempt.

Suggested reviewers: nikagra

Priority: ➖ Normal

Change: Bug fix · Severity of issue fixed: Medium

Merge Risk: ⚪ Minimal · up to b6a04

Authentication overloads can now reach host recovery without being misclassified as bad credentials. The reviewed paths leave no established material merge risk.

Security Architecture Review

Security architecture risk: 🔵 Low · up to b6a04

The inspected paths preserve authentication failure controls while treating overload as a recoverable server condition. Failed connections are not published, and pool-growth capacity is released for later attempts. No introduced security concern was identified, but broader security coverage remains incomplete.

Retained concerns
No architecture-level concerns identified.

Security review details

Security Blast Radius

  • inferred — The changed input is a database peer's authentication-phase protocol error. Its inspected downstream effects are host-attempt selection, per-host pool capacity, and cached-statement recovery within the driver instance. The change does not demonstrate a new credential authority or cross-service access path.

Trust Boundaries and Controls

  • observed — OVERLOADED remains an initialization failure, not an authentication-success signal. Initialization failure defuncts and closes the incomplete connection; Factory.open propagates the failure, and control-connection and pool publication occur only after successful initialization. BAD_CREDENTIALS still follows the AuthenticationException path.

Resilience and Maintainability Implications

  • observed — Dynamic pool growth now releases its open-connection reservation on overload and returns FAILED. Creation-task completion releases scheduling and registration state, allowing a later attempt. Inspected interruption, phase-rejection, and shutdown-race paths also release reservations or close newly created connections, preserving failure containment.
🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 13.64% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 22 functions across 6 files. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly identifies the main change: preserving native-protocol OVERLOADED responses during authentication in the 3.x driver. It is concise and specific.
Description check ✅ Passed The description directly explains the authentication error handling change, related failover behavior, regression coverage, compatibility impact, and validation results.
Linked Issues check ✅ Passed The changes satisfy #1159. Connection preserves OVERLOADED as OverloadedException for V1 CREDENTIALS and V2+ AUTH_RESPONSE, while BAD_CREDENTIALS remains an AuthenticationException. Over…
Out of Scope Changes check ✅ Passed The changes remain within #1159. The connection handling and regression tests support overload propagation and transient connection recovery. No unrelated change is demonstrated.
  • Fix all pre-merge checks with AI
✨ Finishing Touches 💡 1
🛠️ Fix failing CI checks 💡
  • Commit to this branch
  • Create a new PR

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Host-up and host-add processing may open a connection to reprepare cached statements after cancelling the current reconnection attempt. The newly preserved OverloadedException bypassed the existing transient connection-error handling and could abort that lifecycle.

Treat authentication overload like other connection failures in prepareAllQueries so pool creation and recovery can continue, with regression coverage for cached reprepare.

Refs scylladb#1159

Jira: https://scylladb.atlassian.net/browse/DRIVER-1121

@nikagra nikagra left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Overload propagation, channel cleanup, control-connection failover, pool-slot release and reprepare all check out. Nits inline. Please remove README.md update

Comment thread changelog/README.md Outdated
Comment thread changelog/README.md Outdated
Keep the Scylla-specific fix out of the upstream changelog, and reuse the existing package-local error-response helper in the authentication regression tests.

Refs scylladb#1159

Jira: https://scylladb.atlassian.net/browse/DRIVER-1121
@dkropachev
dkropachev merged commit a5a6089 into scylladb:scylla-3.x Sep 30, 2026
11 of 13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants