Skip to content

fix(sdk): bound TCP login and credentialed connect by request deadline - #4345

Draft
jiengup wants to merge 12 commits into
apache:masterfrom
jiengup:tcp-client-hangs-forever-on-login-when-the-serve
Draft

jiengup wants to merge 12 commits into
apache:masterfrom
jiengup:tcp-client-hangs-forever-on-login-when-the-serve

Conversation

@jiengup

@jiengup jiengup commented Sep 29, 2026

Copy link
Copy Markdown
Contributor

Closes #4262
Relates to #3419

Rationale

A TCP connection can stay open while the server never replies to login. The caller then has no bound on the complete login operation, particularly when it waits for a lock, reconnects, or follows a leader redirect. This is the client-side failure reported in #4262. Released clients will still need a server-side response or disconnect when an older, incompatible login frame arrives.

This is an incremental implementation of #3419's timeout proposal. It introduces a place for each transport and request category to choose a deadline and define what expiration does. The configured deadline currently applies only to TCP username/password login, TCP personal-access-token login, and connect() when configured or remembered credentials require sign-in. Extending it to other request types and transports is follow-up work.

What changed?

TCP now has a nonzero request_timeout setting (30 seconds by default), exposed through the TCP config builder, connection string, and shared client arguments. A task-local RequestBudget gives nested login, reconnect, and redirect calls one deadline; a transport-neutral ClientRequestPolicy supplies the timeout and expiration action without making the high-level client own TCP connection state. On expiry, TCP marks the connection invalid and reconciles its stream and VSR session before reconnecting.

Deadline coverage

  • For IggyClient login and credentialed connect(), the deadline starts before acquiring the high-level client read lock. It spans the awaited login or PAT login, logout-before-relogin, leader metadata lookup and redirection, recursive sign-in, and the connect coordinator.
  • During TCP connection setup, it spans waits for another connect owner, reconnect and reestablish_after sleeps, endpoint selection, TCP dial, optional TLS handshake, automatic or remembered sign-in, and leader settlement. All attempts share the caller's deadline.
  • During a budgeted TCP request, it spans the routing and stream locks, socket write and flush, response-header and response-body reads, transient-response retry sleeps, and any permitted reconnect. The spawned send task captures the deadline explicitly; header and body reads share it rather than restarting a timer.

The configured request_timeout does not yet apply to ordinary TCP requests or to a connect() that has no sign-in credentials. Ordinary TCP requests retain the existing response-read limit, which does not cover the entire operation. QUIC, WebSocket, and HTTP do not yet supply a request policy. The deadline is observed by async polling, so a blocked executor or long synchronous section can delay it; arbitrary spawned tasks do not inherit the task-local budget. A caller-supplied outer budget takes precedence over nested configured budgets. A send worker that cannot make progress may also delay deferred stream cleanup and the next reconnect, even though the caller has received the timeout error.

Replacing the underlying client is another uncovered case. IggyClient::client() exposes a shared lock, and a caller can use its write guard to replace an HTTP ClientWrapper with a TCP one. IggyClient caches the timeout policy only when it is constructed, so the cached policy remains None after this replacement. A subsequent login_user() waits for the shared lock's read guard without starting the new TCP client's deadline; the TCP login code cannot enforce its timeout until it acquires that guard. In the regression test, a replacement TCP client configured for 50 ms still produced no login timeout after an outer 500 ms wait.

Behavior after timeout and proposed extension

For the supported TCP login and credentialed-connect paths, the caller receives RequestTimeoutOutcomeUnknown. That error deliberately does not claim the server did nothing. Expiration sets a persistent invalidation flag and tries to drop the socket and mark the client disconnected without waiting for locks held by an in-flight send. If cleanup cannot acquire a lock immediately, the send worker or the next connect() discards the old stream; reconnection resets the VSR session before a new request uses it. The expired login is not replayed across a new session after its deadline.

For future coverage of all TCP requests, the same full-operation deadline and invalidation path could be reused, but an operation whose outcome is unknown must not be automatically replayed under a new VSR session. For QUIC, WebSocket, and HTTP, each transport needs its own expiration policy before the same caller-facing timeout contract is enabled; the right cleanup scope may differ for multiplexed connections and independent HTTP requests.

Follow-up API and design questions

  • This PR provides only a client-level timeout. Support per-request timeout to TCP/QUIC/WebSocket/Http clients #3419 also proposes a per-request override such as client.login(...).with_timeout(10ms). The current async trait methods return futures directly, so fluent overrides need an API change or a new request-builder/operation-specific entry point. A prototype is deliberately excluded from this PR.
  • Is a task-local budget plus transport-specific policy the right seam? Is there a simpler end-to-end way to include every awaited call without adding timeout wrappers at multiple layers?
  • Should expiry be a hard caller deadline that cancels the awaited operation and invalidates the connection, as implemented here, or a soft deadline that reports slowness while allowing an in-flight request to finish? The spawned TCP send worker is not forcibly aborted; it retains the stream until it can clean up safely. How should either choice report an operation that may have committed?

Local Execution

  • Unit and local checks: cargo fmt --all, cargo sort --no-format --workspace, cargo clippy --all-features --all-targets -- -D warnings, cargo test -p iggy_common -p iggy, ./scripts/ci/taplo.sh --check, and commit-time hooks passed on the implementation before the latest rebase. The timeout tests cover silent peers, partial replies, stalled writes, stream and connect-owner lock waits, PAT and auto login, retry bounds, and reconnect after expiration. The timeout implementation is unchanged by the rebase, but the exact rebased HEAD has not had this full sequence or a full prek run rerun.
  • Review: team-review-lite was run, and the implementation was revised to address its findings.
  • End-to-end: Tested against a real iggy-server:0.9.0. In the no-reply case, the server was paused after the TCP connection was established.
    Scenario Client Observed result
    Server paused after TCP connect Rust SDK from an earlier upstream/master snapshot Login did not finish within 35 s.
    Server paused after TCP connect Rust SDK from the tested branch Returned RequestTimeoutOutcomeUnknown in about 30.01 s.
    Server resumed after timeout Same Rust client from the tested branch Reconnected and logged in successfully.
    Server paused after TCP connect Published Python SDK 0.9.0 Login did not finish within 35 s.
    Server paused after TCP connect Python SDK built from the tested branch Returned a timeout error in about 30 s.
    Healthy server Published Python SDK 0.9.0 Logged in successfully.
  • CI: No GitHub Actions result was available when this message was drafted. The repository's pre-merge CI must still be run and pass on the submitted commit.

AI Usage

  • Tool and scope: OpenAI Codex assisted with the timeout implementation, refactoring, tests, validation analysis.
  • Verification: The local checks and end-to-end observations are listed above; the author should review them against the final submitted commit.
  • Author confirmation before posting: I have reviewed the complete diff and can explain every change and its timeout semantics.

A silent TCP peer and repeated reconnects could leave login callers waiting without a useful terminal result. Carry one configurable deadline through TCP sign-in, reconnects, and transport cleanup so callers receive a distinct timeout with an unknown outcome after a possible write.
Passing the parsed timeout through the TCP options constructor keeps configuration assembly consistent. The test harness now accepts a timeout override and verifies both its default and custom values.
The response-read limit does not bound lock waits, reconnects, or post-login work. Start one request budget at the public login or auto-login entry, carry it through nested TCP operations, and isolate the connection when the outcome is unknown.
Client providers and Rust examples need to pass the configured login budget into TcpClientConfig. Add the shared CLI option and wire it into both construction paths.
The login deadline must start before the high-level client lock is acquired, while the shared client interface should not depend on a TCP policy type. Route the request budget through a transport policy and keep TCP expiration behavior in the TCP client.
The public login budget already bounds the connect coordinator wait. Remove its duplicate deadline path and keep the coordinator focused on sharing connection results.
Request callers currently switch between a task-local budget handle and a runner, which makes deadline propagation hard to follow. Expose one deadline lookup and one future runner, and keep timeout handling inside the module. Box budgeted futures so nested TCP login reconnects stay within the runtime thread stack.
The timeout policy currently applies only to TCP sign-in and credentialed connect. Document that boundary, verify other transports leave the policy unset, and share the reconnectable error check.
The login and connect timeout flow had accumulated nested budget wrappers around individual waits, making the control flow difficult to follow. Keep the deadline in a task-local RequestBudget and apply it at request entry points while leaving transport-specific framing safeguards in place.
A timed-out request can leave a transport task holding the stream lock. Route deadline cleanup through the TCP request policy so a busy lock still marks the connection stale, and report cleanup lock errors.
A request can expire while the stream or state lock is held, leaving the next connection attempt with stale transport and VSR session state.

Keep timeout invalidation in the TCP client, let nested budgets run one cleanup callback, and expose it through a lightweight generic request policy. Remove the extra login replay loop so reconnect retains the existing attempt behavior.
@codecov

codecov Bot commented Sep 29, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 80.78049% with 197 lines in your changes missing coverage. Please review.
✅ Project coverage is 75.25%. Comparing base (9fdb642) to head (a589ff4).

Files with missing lines Patch % Lines
core/sdk/src/tcp/tcp_client.rs 83.46% 86 Missing and 19 partials ⚠️
core/sdk/src/clients/client.rs 78.04% 24 Missing and 3 partials ⚠️
core/common/src/request_budget.rs 76.69% 22 Missing and 2 partials ⚠️
core/common/src/traits/binary_impls/users.rs 73.68% 3 Missing and 7 partials ⚠️
...e/sdk/src/clients/binary_personal_access_tokens.rs 65.38% 6 Missing and 3 partials ⚠️
.../src/traits/binary_impls/personal_access_tokens.rs 75.75% 4 Missing and 4 partials ⚠️
core/sdk/src/clients/binary_users.rs 84.61% 0 Missing and 4 partials ⚠️
core/sdk/src/clients/client_builder.rs 0.00% 4 Missing ⚠️
core/common/src/types/args/mod.rs 25.00% 1 Missing and 2 partials ⚠️
core/common/src/traits/binary_transport.rs 75.00% 1 Missing ⚠️
... and 2 more
Additional details and impacted files
@@              Coverage Diff              @@
##             master    #4345       +/-   ##
=============================================
- Coverage     87.77%   75.25%   -12.53%     
- Complexity     1575     1576        +1     
=============================================
  Files          1288     1289        +1     
  Lines        227249   222599     -4650     
  Branches     190686   186038     -4648     
=============================================
- Hits         199478   167506    -31972     
- Misses        23062    50152    +27090     
- Partials       4709     4941      +232     
Components Coverage Δ
Rust Core 73.70% <80.78%> (-15.18%) ⬇️
Java SDK 68.70% <ø> (+0.01%) ⬆️
C# SDK 77.41% <ø> (+0.01%) ⬆️
Python SDK 91.00% <ø> (ø)
PHP SDK 85.67% <ø> (ø)
Node SDK 96.55% <ø> (-0.03%) ⬇️
Go SDK 70.37% <ø> (+0.08%) ⬆️
Files with missing lines Coverage Δ
core/common/src/error/iggy_error.rs 100.00% <ø> (ø)
...ypes/configuration/tcp_config/tcp_client_config.rs 100.00% <100.00%> (ø)
...figuration/tcp_config/tcp_client_config_builder.rs 56.43% <100.00%> (-14.70%) ⬇️
...ration/tcp_config/tcp_connection_string_options.rs 82.92% <100.00%> (-3.44%) ⬇️
core/common/src/traits/binary_transport.rs 25.00% <75.00%> (+25.00%) ⬆️
core/sdk/src/client_provider.rs 43.33% <50.00%> (+0.09%) ⬆️
core/sdk/src/client_wrappers/client_wrapper.rs 83.33% <83.33%> (ø)
core/common/src/types/args/mod.rs 45.51% <25.00%> (-0.54%) ⬇️
core/sdk/src/clients/binary_users.rs 90.90% <84.61%> (-9.10%) ⬇️
core/sdk/src/clients/client_builder.rs 48.50% <0.00%> (-10.73%) ⬇️
... and 6 more

... and 448 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@justinmclean

Copy link
Copy Markdown
Member

Thanks @jiengup. CI needs two small updates:

  • Add the new --tcp-request-timeout option to the expected help text in core/integration/tests/cli/general/test_help_command.rs.
  • Regenerate the Swift golden file for the new error code:
    cargo run --manifest-path foreign/swift/Tools/golden-vectors/Cargo.toml --foreign/swift/Tests/IggyTests/Fixtures/golden.json

It also needs a rebase to fix the conflicts.

@jiengup jiengup changed the title fix(sdk): bound TCP login and credentialed connect by request deadline` fix(sdk): bound TCP login and credentialed connect by request deadline Oct 8, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

TCP client hangs forever on login when the server never replies

2 participants